Outros
Gupy

Senior data engineer (databricks & ai-enabled data platforms) (4811)

Logo Keep Simple

Empresa

Keep Simple

Onde estamos

São Paulo / SP

Vagas da empresa

Nível

Sênior

Período

Híbrido

Tipo

PJ

Salário

A combinar

Responsabilidades e atribuições

Come work for a large global financial and insurance products company!

This is your chance !!

Start a successful career in a renowned company in the international market!

Great opportunity!

Global insurance and asset management company seeks a responsible, organized, dynamic and team-oriented person

  • We are seeking a highly skilled Senior Data Engineer to design, build, and optimize modern data platforms that enable advanced analytics, AI/ML, and business intelligence solutions
  • The ideal candidate will possess deep expertise in Databricks, cloud-based data engineering, data architecture, and modern software engineering practices
  • This role will provide technical leadership, establish engineering standards, and drive the development of scalable, secure, and reliable data solutions supporting enterprise-wide initiatives
  • Key ResponsibilitiesData Platform Engineering:Design, develop, and maintain scalable data pipelines using Databricks, Spark, and cloud-native technologies
  • Build and optimize batch and real-time data ingestion frameworks
  • Develop reusable engineering patterns, frameworks, and accelerators for enterprise data solutions
  • Implement data quality, observability, lineage, and governance capabilities across the data ecosystem
  • Support large-scale data modernization and cloud migration initiatives
  • Databricks Engineering:Design and implement Delta Lake architectures and Medallion data models
  • Build optimized ETL/ELT pipelines using Databricks notebooks, workflows, and Delta Live Tables
  • Establish performance tuning, cost optimization, and workload management best practices
  • Leverage Unity Catalog for data governance, security, and metadata management
  • Implement CI/CD pipelines and Infrastructure-as-Code for Databricks deployments
  • Cloud Data Engineering:Design cloud-native data architectures across Azure and AWS environments
  • Integrate enterprise data sources including SQL Server, Oracle, APIs, SaaS platforms, and streaming platforms
  • Build resilient and secure data services supporting high availability and disaster recovery requirements
  • Partner with infrastructure and security teams to implement cloud security controls and compliance standards
  • AI and Advanced Analytics Enablement:Build and optimize data products supporting AI, machine learning, and generative AI workloads
  • Develop feature engineering pipelines and curated datasets for model training and inference
  • Support vector databases, semantic search, RAG (Retrieval-Augmented Generation), and LLM-powered applications
  • Collaborate with Data Scientists and AI Engineers to operationalize machine learning solutions
  • Implement MLOps and DataOps practices to support scalable AI delivery
  • Technical Leadership:Serve as a technical lead for complex data engineering initiatives
  • Define architecture standards, engineering best practices, and development guidelines
  • Mentor junior and mid-level engineers
  • Conduct architecture reviews and provide technical recommendations
  • Drive adoption of automation, observability, and platform engineering principles

Requisitos e qualificações

  • Required QualificationsEducation:Bachelor's degree in Computer Science, Information Technology, Engineering, or related discipline
  • Master's degree preferred
  • Experience8+ years of experience in data engineering, data warehousing, or data platform development
  • 5+ years of hands-on experience with Databricks and Apache Spark
  • Experience designing and implementing enterprise-scale cloud data platforms
  • Experience leading technical initiatives and mentoring engineering teams
  • Technical SkillsData Engineering:Databricks
  • Apache Spark (PySpark, Spark SQL)
  • Delta Lake
  • Delta Live Tables
  • Unity Catalog
  • Data Warehousing and Data Modeling
  • Programming:Python
  • SQL
  • Scala (preferred)
  • PowerShell or scripting automation experience
  • Cloud Platforms:Microsoft Azure
  • AWS
  • Cloud-native storage and compute services
  • Databases:SQL Server
  • Oracle
  • PostgreSQL
  • Snowflake (preferred)
  • Data Integration:Azure Data Factory
  • Databricks Workflows
  • Kafka/Event Streaming
  • REST APIs
  • DevOps & Automation:GitHub
  • Azure DevOps
  • CI/CD pipelines
  • Terraform
  • Infrastructure as Code
  • Observability:Monitoring and alerting solutions
  • Data quality frameworks
  • Performance optimization
  • Logging and troubleshooting
  • AI & Emerging Technologies:Machine Learning data pipelines
  • Feature Stores
  • MLOps
  • Generative AI concepts
  • Vector databases
  • Retrieval-Augmented Generation (RAG)
  • AI governance and responsible AI principles
  • Preferred QualificationsDatabricks Certified Data Engineer Professional
  • Azure Data Engineer Associate certification
  • AWS Data Analytics certification
  • Experience supporting regulated industries such as Insurance, Financial Services, or Healthcare
  • Experience implementing enterprise data governance programs
  • Experience integrating AI solutions into production environments
  • Key CompetenciesStrategic thinking and problem solving
  • Technical leadership
  • Architecture design
  • Stakeholder management
  • Communication and presentation skills
  • Continuous improvement mindset
  • lnnovation and AI adoption
  • Risk and compliance awareness
  • Success MeasuresDelivery of scalable, resilient, and secure data solutions
  • Adoption of enterprise engineering standards and best practices
  • Improvement in platform performance, reliability, and cost efficiency
  • Accelerated delivery of analytics and AI capabilities
  • Reduction in operational support effort through automation and observability
  • Successful mentoring and development of engineering talent
  • Fluent english
Ainda não tem CNPJ? Abra seu CNPJ grátis. DUAS MENSALIDADES GRÁTIS nos planos Padrão, Multibenefícios ou Expert.
Abrir CNPJ

Benefícios de trabalhar com a gente

Modelo de contratação:PJ
Forma de atuação:Híbrido (3x por semana presencial no escritório de Pinheiros/SP)