IT ServicesFull time
6 Nov
Lead Engineer
Qualification: Graduate/Postgraduate in Computer Science, IT, or related field
Location: Mumbai Exp: 3-5Years
+ What are we looking for? We are looking for an experienced Data Engineer to design, develop, and maintain scalable data pipelines and data platforms supporting Analytics, AI/ML, and Business Intelligence use cases.The candidate will work closely with Data Scientists, AI Engineers, Analytics teams, SAP teams, and business stakeholders to build reliable, secure, and high-performance data solutions using AWS Cloud and SAP S/4HANA.
Key Responsibilities • Design and develop scalable ETL/ELT pipelines for structured and unstructured data. • Build data ingestion and transformation pipelines using Python, SQL, and AWS Cloud services. • Develop data integration solutions between SAP S/4HANA and AWS Cloud. • Work with AWS services such as Amazon S3, AWS Glue, AWS Lambda, Amazon Athena, Amazon CloudWatch, and IAM. • Implement scalable data storage and processing architectures on AWS. • Perform data cleansing, transformation, validation, reconciliation, and data-quality checks. • Design and implement incremental data loading, CDC, watermarking, and SCD Type 1/Type 2 processes. • Optimize data pipelines for performance, scalability, reliability, and cloud cost efficiency. • Implement data governance, metadata management, data lineage, security, and access controls. • Develop monitoring and alerting mechanisms for production data pipelines. • Troubleshoot pipeline failures, data-quality issues, performance bottlenecks, and production incidents. • Implement CI/CD practices for data engineering workloads using Git, GitHub, and DevOps tools. • Collaborate with Data Scientists and AI/ML Engineers to prepare high-quality datasets and feature pipelines. • Support integration of GenAI, RAG, and Agentic AI solutions with enterprise data where required. • Document technical architecture, data flows, pipeline logic, data mappings, and operational procedures.
Required Technical Skills / Programming • Strong Python • Strong SQL • Shell/Bash scripting Data Engineering • ETL/ELT • Data ingestion and integration • Data pipelines • Data warehousing • Data modelling • Data quality and validation • Incremental loading / CDC • Watermark-based processing • SCD Type 1 & Type 2 • Batch and near-real-time data processing AWS Cloud Strong hands-on experience with: • Amazon S3 • AWS Glue • AWS Lambda • Amazon Athena • Amazon CloudWatch • AWS IAM • AWS Step Functions / workflow orchestration • AWS Event Bridge • AWS Secrets Manager • AWS CI/CD services SAP S/4HANA • Strong understanding of SAP S/4HANA data structures and business processes. • Experience extracting/integrating data from SAP S/4HANA into AWS. • Experience with OData, APIs, CDS Views, IDocs, or other relevant SAP interfaces. • Understanding of SAP master and transactional data. • Ability to work with SAP functional and technical teams for data integration requirements. • Experience handling SAP data quality, reconciliation, and incremental extraction.
AI/ML Data Engineering Exposure • The candidate should be able to support modern AI/ML workloads by:Preparing high quality datasets for ML models. • Building data pipelines for AI/ML applications. • Processing structured and unstructured enterprise data. • Supporting RAG pipelines and embedding-generation workflows. • Providing governed enterprise data to AI agents and LLM applications. • Implementing data validation and monitoring for AI applications. • Supporting integration between AWS data platforms and AI/ML solutions.
Preferred Skills • AWS Step Functions / workflow orchestration • REST API integration • SAP APIs / OData / CDS Views • SAP S/4HANA integration • Data Catalog and metadata management • Data governance and security • Docker, CI/CD • Git / GitHub / DevOps, Infrastructure as Code • Experience supporting AI/ML data pipelines • Experience with RAG, embeddings, vector databases, or Agentic AI data workflows • Experience in enterprise data migration and integration projects Key Competencies • Strong problem-solving and analytical skills. • Strong understanding of AWS data engineering architecture. • Good understanding of SAP S/4HANA data and integration. • Ability to troubleshoot production issues independently. • Strong focus on data quality, reliability, and security. • Understanding of data governance and access controls. • Good communication and stakeholder-management skills. • Ability to work in an Agile/Scrum environment. • Ability to convert business requirements into scalable technical solutions.
Qualification • Bachelor's/Master's degree in Computer Science, Information Technology, Engineering, Data Science, or a related field. • 4+ years of relevant Data Engineering experience preferred. • Hands-on experience with AWS Cloud and SAP S/4HANA integration is strongly preferred.
Success Measures The Data Engineer will be evaluated on: • Reliability and scalability of data pipelines. • Data quality and completeness. • Successful integration of SAP S/4HANA with AWS. • Pipeline performance and optimization. • Reduction in processing time and cloud infrastructure cost. • Timely resolution of production issues. • Reusability and maintainability of data engineering components. • Strong data governance and security. • Enablement of Analytics, AI/ML, and Business Intelligence use cases.