LOCATION- MUMBAI/ NAVI MUMBAI
We are seeking a highly skilled and delivery-focused Lead Data Engineer to design and build enterprise-grade data platforms and applications under the Durables model. This role will be responsible for leading end-to-end new implementation and development initiatives, building scalable data pipelines, and architecting cloud-native solutions on AWS.
The ideal candidate will act as a key technical leader for new builds, working closely with client stakeholders to translate business requirements into robust data engineering solutions. This is a hands-on leadership role requiring strong design capabilities, cloud expertise, and experience in delivering large-scale data engineering projects in Agile environments.
Key Responsibilities
Solution Design & Implementation
- Lead end-to-end design and development of scalable, high-performance data engineering solutions.
- Architect and implement new data pipelines, data models, and cloud-native data platforms.
- Design modular, reusable, and future-ready data frameworks for enterprise use cases.
- Work closely with stakeholders to understand requirements and convert them into technical designs.
Technical Leadership
- Provide architectural direction for new data platform development and modernization initiatives.
- Define best practices for ETL/ELT design, data modeling, and cloud-native data processing.
- Build and optimize data pipelines using AWS services (Glue, Lambda, Redshift, S3, EMR, Athena, etc.).
- Leverage PySpark, SQL, Python, and distributed processing frameworks for large-scale data processing.
- Drive engineering excellence through code quality, performance tuning, and scalable design patterns.
Agile Delivery & Execution
- Lead sprint planning, estimation, and delivery of development milestones in Agile environments.
- Ensure timely delivery of high-quality data engineering components and features.
- Collaborate with cross-functional teams including product, analytics, and infrastructure teams.
Client Engagement & Solutioning
- Act as a technical advisor for clients during solution design and implementation phases.
- Present architecture approaches, design decisions, and trade-offs to stakeholders.
- Proactively recommend innovative solutions to improve scalability, performance, and business value.
Team Leadership & Mentoring
- Lead and mentor a team of data engineers, ensuring strong technical growth and delivery discipline.
- Drive task allocation, design reviews, and technical guidance across the team.
- Build a strong engineering culture focused on ownership and continuous improvement.
Process, Quality & Governance
- Establish CI/CD pipelines, version control practices, and automated testing frameworks.
- Ensure adherence to coding standards, documentation practices, and design governance.
- Identify risks and dependencies early and implement mitigation strategies.
Required Skills & Qualifications
- 6+ years of experience in Data Engineering / Data Platform Development / Application Development.
- Strong hands-on expertise in AWS cloud services (Glue, Lambda, Redshift, S3, Athena, EMR, CloudWatch).
- Proficiency in PySpark, SQL, and Python for large-scale data processing.
- Strong experience in designing and building end-to-end data pipelines and data platforms.
- Deep understanding of distributed data processing and performance optimization.
- Proven experience leading engineering teams (3+ years), ideally in development or implementation projects.
- Strong client-facing and communication skills.
- Experience with Agile delivery methodologies.
- Familiarity with tools like JIRA, Confluence, Git, Jenkins, or CI/CD pipelines.
Good to Have
- Experience in greenfield data platform development or AWS migration projects.
- Exposure to legacy-to-cloud transformation programs.
- Strong understanding of data modeling concepts and Hive/partitioning strategies.
- Experience building real-time or near real-time data pipelines.
