Key Requirements
• 4+ years of experience in Data Engineering or a similar role.
• Strong hands-on experience with Python, PySpark, SQL, and Databricks.
• Experience working with AI coding assistants such as Claude Code, GitHub Copilot, Codex, or Cursor.
• Familiarity with Apache Airflow for workflow orchestration and scheduling.
• Experience with Linux administration and shell scripting.
• Strong understanding of data modelling, ETL processes, and data warehousing concepts.
• Good problem-solving, analytical, communication, and collaboration skills.
• Bachelor's degree in Computer Science, Engineering, or a related field.
Responsibilities
Data Pipeline Development
• Design, develop, and maintain scalable data pipelines using Python, PySpark, and SQL.
• Ingest, process, and transform data from multiple sources.
• Implement data quality checks and monitoring to ensure pipeline reliability and data integrity.
Databricks & Data Engineering
• Use Databricks for data ingestion, ETL processing, transformation, and data analysis.
• Develop and optimize Databricks jobs and clusters for performance and cost efficiency.Work with large volumes of structured and unstructured data.
Data Warehouse & Modelling
• Build and optimize data warehouse solutions for efficient data storage and retrieval.
• Design and implement data models for analytical and reporting requirements.
• Apply best practices across ETL and data warehousing processes.
Airflow & Workflow Orchestration
• Develop, configure, and manage Airflow DAGs for data pipeline orchestration and scheduling.
• Implement error handling, monitoring, and alerting to ensure reliable workflow execution.
Linux & Shell Scripting
• Work with Linux-based environments supporting data engineering workloads.
• Troubleshoot infrastructure and server-related issues.
• Use shell scripting for automation and data engineering tasks.
AI Coding Tools
• Leverage AI coding assistants such as Claude Code, GitHub Copilot, Codex, or Cursor to improve
development productivity, code quality, and efficiency.
• Demonstrate practical experience using AI-assisted development tools in day-to-day engineering work.
Collaboration & Documentation
• Collaborate with Data Scientists, Data Analysts, and other stakeholders to understand requirements and deliver effective data solutions.
• Document data engineering processes, workflows, technical solutions, and best practices.
• Contribute to continuous improvement of data engineering practices and standards.
Ideal Candidate:
We are looking for a hands-on Data Engineer with 4+ years of experience who has strong expertise in Python, PySpark, SQL, and Databricks, along with exposure to Airflow, Linux, shell scripting, data modelling, and ETL/data warehousing. Experience with modern AI coding assistants is an important part of this requirement.
Candidates who can join immediately or within 30 days will be preferred.