Associate Director – Data Engineering (Databricks Lakehouse Architect)
Location: Chennai / Trivandrum / Remote (1 Each)
Experience: Chennai & Trivandrum - 15+ Years and Remote - 12+ Years
Notice Period: Immediate to 60 Days (90 days NP would not be entertained)
About the Role
We are seeking an experienced Associate Director – Data Engineering (Databricks Lakehouse Architect) to lead the design and delivery of modern cloud-native data platforms. This is a strategic technical leadership role for a hands-on architect who can drive enterprise-scale data transformation initiatives, build scalable Data Lakehouse solutions, and lead high-performing engineering teams. The ideal candidate will possess deep expertise in Databricks, Apache Spark/PySpark, Delta Lake, and cloud-native data engineering across Azure or AWS. In addition to strong technical capabilities, the role requires the ability to engage with clients, define solution architectures, lead engineering teams, and contribute to business growth through solutioning and technical leadership.
Key Responsibilities
Data Lakehouse Architecture
- Design and architect scalable enterprise Data Lakehouse solutions using the Databricks Lakehouse platform.
- Develop modern data architectures leveraging Delta Lake, Medallion Architecture (Bronze, Silver, Gold), and cloud-native storage technologies.
- Define enterprise data architecture standards, reusable frameworks, and engineering best practices.
- Ensure scalable, secure, and high-performance data platform implementations.
Data Engineering & Pipeline Development
- Design, build, and optimize enterprise-grade batch and real-time data pipelines.
- Lead development using Apache Spark, PySpark, and SQL for large-scale data processing.
- Optimize pipeline performance, workload execution, and cost efficiency across cloud environments.
- Establish monitoring, reliability, and operational best practices for production data platforms.
Cloud Data Platform Leadership
Microsoft Azure:
- Azure Databricks
- Azure Data Factory (ADF)
- Azure Data Lake Storage (ADLS)
- Azure Synapse Analytics
OR
Amazon Web Services (AWS):
- Databricks on AWS
- Amazon S3
- AWS Glue
- Amazon Redshift
- Amazon Athena
- Provide architectural guidance for cloud-native data platform adoption.
- Recommend best-fit cloud services based on scalability, governance, performance, and business requirements.
Data Governance & Lakehouse Management
- Implement enterprise data governance using Unity Catalog.
- Establish data quality, lineage, metadata management, and security standards.
- Drive governance frameworks ensuring compliance and controlled data access across the organization.
Solution Architecture & Technical Leadership
- Own end-to-end solution architecture, technical design, estimation, and architecture reviews.
- Lead technical discussions with customers and stakeholders to understand business requirements and translate them into scalable data solutions.
- Define engineering standards, coding practices, and architecture guidelines across multiple engagements.
- Mentor and provide technical leadership to Data Engineering teams.
Delivery Leadership
- Lead multiple enterprise data engineering initiatives from discovery through deployment and production support.
- Manage project risks, technical dependencies, delivery milestones, and solution quality.
- Collaborate with cross-functional engineering, analytics, cloud, and business teams to ensure successful project execution.
- Drive continuous improvement initiatives across engineering practices and delivery methodologies.
Client Engagement & Practice Development
- Partner with business and technology stakeholders to define enterprise data modernization roadmaps.
- Support pre-sales activities including solution design, effort estimation, proposals, technical presentations, and architecture workshops.
- Contribute to capability building, innovation initiatives, proof of concepts (PoCs), and practice development.
- Stay current with emerging trends in cloud data engineering, AI-ready architectures, and modern analytics platforms.
Required Qualifications
Mandatory Experience
- 12+ years of experience in Data Engineering, Data Architecture, or Cloud Data Platform implementation.
- Strong expertise in Databricks Lakehouse Architecture.
- Hands-on experience designing and implementing Delta Lake solutions.
- Deep expertise in Apache Spark and PySpark.
- Experience implementing Medallion Architecture (Bronze, Silver, Gold).
- Strong knowledge of enterprise data pipeline design and optimization.
- Experience with Unity Catalog and enterprise data governance.
- Hands-on expertise with either Microsoft Azure or AWS cloud data services.
- Proven experience leading technical solution design, architecture reviews, and enterprise implementations.
- Experience engaging directly with enterprise customers and senior stakeholders.
Preferred Experience
- Snowflake
- Machine Learning Engineering or MLOps
- AI-ready Data Platform Architecture
- Real-time Streaming Data Pipelines
- Data Mesh or Data Fabric concepts
- Infrastructure as Code (Terraform, ARM, CloudFormation)
- CI/CD for Data Engineering
- Cloud and Data Engineering Certifications
- Experience contributing to RFPs, PoCs, and pre-sales initiatives.
Core Technical Skills
- Databricks Lakehouse Architecture
- Delta Lake
- Apache Spark
- PySpark
- Medallion Architecture
- Data Pipeline Engineering
- Unity Catalog
- Data Governance
- Azure Databricks / Databricks on AWS
- Azure Data Factory (ADF)
- Azure Data Lake Storage (ADLS)
- Azure Synapse Analytics
- Amazon S3
- AWS Glue
- Amazon Redshift
- Amazon Athena
- SQL
- Performance Optimization
- Cloud-native Data Platforms
Leadership Competencies
- Solution Architecture
- Technical Leadership
- Customer Consulting
- Engineering Mentorship
- Delivery Governance
- Architecture Reviews
- Project Estimation & Planning
- Stakeholder Management
- Pre-sales & Solutioning
- Analytical Problem Solving
- Communication & Presentation Skills
Education
Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, Data Science, or a related discipline.
Why Join?
This is an opportunity to shape enterprise-scale cloud data transformation programs, architect next-generation Data Lakehouse platforms, and work with cutting-edge technologies including Databricks, Spark, Delta Lake, Azure, and AWS. You'll play a key role in building modern, AI-ready data ecosystems while influencing technology strategy, mentoring engineering teams, and delivering impactful business outcomes across global enterprise environments.