Job Summary

As a Data Engineer, you will develop, maintain Data Pipelines. You will be involved in the design of data solutions for ESG. You will implement and manage cluster for streaming using technologies like Kafka, flink, Druid, NoSQL DB etc.

You will be responsible for:

• Converting existing manual & semi-automated Data Ingress/ Egress processes to automated Data pipelines
• Create data pipelines for AI/ ML using Python / Pyspark
• Full operational lifecycle of Data platform including creating a streaming platform & helping with Kafka apps development
• Implement scalable solutions to meet the ever-increasing data volumes, using big data/cloud technologies Apache Spark, Kafka, any Cloud computing etc.

Job Requirements:

• Minimum 4 years of experience in Big Data technologies
• Experience in developing data processing flows using Python/ PySpark
• Hands on experience on Data ingestion, data cleansing, ETL, Data Mart creation and exposing data for consumers
• Strong experience in Oracle / PostgreSQL
• Experience of implementing and managing large scale cluster for streaming Kafka, Flink, Druid, NoSQL DB (MongoDB) etc
• Experience with Elastic Search, Splunk, Kibana or similar technologies
• Good to have experience in Business Intelligence tool (Qlik Sense)
• Knowledge of Microservices
• Familiarity with packages such as Numpy/pandas is desirable

Qualifications:

Bachelor's degree in Computer Science, Information Technology, or a similar field (Minimum Qualification)

Python Experience should be between 4-8 years