Outros
Gupy

Senior data engineer (databricks & ai platforms) (4809)

Logo Keep Simple

Empresa

Keep Simple

Onde estamos

São Bernardo do Campo / SP

Vagas da empresa

Nível

Sênior

Período

Híbrido

Tipo

PJ

Salário

A combinar

Responsabilidades e atribuições

Come work for a large global financial and insurance products company!

This is your chance !!

Start a successful career in a renowned company in the international market!

Great opportunity!

Global insurance and asset management company seeks a responsible, organized, dynamic and team-oriented person

  • The Senior Data Engineer is responsible for designing, building, and optimizing scalable data platforms and pipelines that support analytics, business intelligence, machine learning (ML), and AI-driven solutions
  • This role partners with data scientists, AI engineers, architects, and business stakeholders to deliver trusted, high-quality data products using Databricks, cloud technologies, and modern data engineering practices
  • The position also serves as a technical leader in enabling enterprise AI and Generative AI initiatives through robust, secure, and governed data platforms
  • Key Responsibilities:Design, develop, and maintain scalable data pipelines and data products using Databricks, Spark, Python, and SQL
  • Build and optimize batch, streaming, and real-time data integration solutions from enterprise and third-party data sources
  • Implement Databricks Lakehouse architectures utilizing Delta Lake and Medallion design patterns
  • Develop and maintain data products that support analytics, predictive modeling, machine learning, and Generative AI applications
  • Collaborate with Data Scientists and AI Engineers to prepare, transform, and govern data for AI and ML use cases
  • Design and implement feature engineering pipelines and support ML lifecycle processes
  • Develop data solutions that support Retrieval Augmented Generation (RAG), vector search, semantic search, and LLM-based applications
  • Optimize Spark jobs, SQL workloads, and data processing frameworks for performance, scalability, and cost efficiency
  • Implement data quality, observability, lineage, governance, and monitoring capabilities
  • Ensure compliance with data privacy, security, and responsible AI standards
  • Contribute to CI/CD, Infrastructure-as-Code, and DataOps practices across the data platform
  • Mentor junior engineers and promote engineering best practices across the organization

Requisitos e qualificações

  • Required Qualifications:Bachelor's degree in Computer Science, Information Systems, Engineering, Mathematics, or a related field
  • 7+ years of experience in data engineering, ETL development, or large-scale data platform engineering
  • 3+ years of hands-on experience with Databricks and Apache Spark
  • Strong proficiency in Python, SQL, and distributed data processing frameworks
  • Experience building cloud-based data lakes, data warehouses, and Lakehouse architectures
  • Experience supporting AI, machine learning, or advanced analytics initiatives
  • Strong understanding of data modeling, data governance, and enterprise data management practices
  • Experience developing and optimizing large-scale data pipelines in AWS, Azure, or Google Cloud
  • Preferred Qualifications:Experience with Databricks Delta Lake, Unity Catalog, Delta Live Tables, MLflow, Mosaic AI, and Databricks Workflows
  • Hands-on experience supporting Generative AI, Large Language Models (LLMs), RAG architectures, vector databases, or AI-powered applications
  • Familiarity with AI frameworks such as LangChain, Semantic Kernel, OpenAI APIs, Hugging Face, or similar technologies
  • Experience with feature stores, model training pipelines, and machine learning operationalization (MLOps)
  • Experience with Kafka, Event Hub, or other streaming technologies
  • Databricks Certified Data Engineer Professional or equivalent cloud certification
  • Key Competencies:Databricks Platform Engineering
  • Apache Spark Development
  • AI & Machine Learning Data Engineering
  • Generative AI Data Solutions
  • Lakehouse Architecture
  • Data Modeling & Data Warehousing
  • Data Governance & Security
  • Data Observability & Reliability Engineering
  • Cloud Data Platforms (AWS/Azure)
  • DataOps, CI/CD & Automation
  • Technical Leadership & Mentoring
  • Success Measures:Delivery of scalable, reliable, and secure data platforms supporting analytics, AI, and business operations
  • Successful enablement of AI and Generative AI use cases through high-quality, governed data products
  • Improved data quality, pipeline reliability, and platform performance
  • Increased automation, operational efficiency, and reuse of engineering frameworks
  • Adoption of data engineering standards and best practices across development teams
  • Measurable improvements in AI/ML solution delivery speed and business value realization
  • Fluent english
Ainda não tem CNPJ? Abra seu CNPJ grátis. DUAS MENSALIDADES GRÁTIS nos planos Padrão, Multibenefícios ou Expert.
Abrir CNPJ

Benefícios de trabalhar com a gente

Modelo de contratação:PJ
Forma de atuação:Híbrido (3x por semana presencial no escritório de Pinheiros/SP)