Our client offers integrated products and services that combine cutting-edge technology with deep sector expertise. Their business units specialize in E-Governance, Energy & Utilities, Public Safety & Disaster Management, Smart Cities, Software Services, and IT Services. By leveraging digital and analytical technologies, they partner with clients to drive their digital transformation journey and deliver impactful solutions.They strive to simplify lives through innovative solutions that make a positive difference. Whether it's supporting Digital India initiatives, enabling Smart Cities, ensuring public safety, or delivering software and IT services, they are dedicated to creating meaningful impact.

Big Data Engineer
As a Big Data Engineer, you will play a crucial role in shaping the future of data-driven solutions. We are seeking individuals with a strong background in big data technologies, cloud environments, and a passion for building scalable data pipelines.

Experience: 5-8 yrs

Responsibilities:

• Design and develop innovative data pipelines using Hadoop, Hive, Presto, Spark, HQL, Elasticsearch, and YARN.
• Collaborate with cloud environments like GCP, AWS, or Azure to create scalable and efficient data solutions.
• Build robust batch and streaming data pipelines from diverse data sources, including RDBMS, NoSQL, IoT/telemetry data, and APIs.
• Implement cutting-edge streaming data ingestion pipelines using Apache Kafka and leverage cloud-based services.
• Utilize ETL tools like Apache Spark and native cloud services to streamline data integration processes.
• Integrate API data using various methods, ensuring seamless data flow.
• Work with both cloud-based and on-premises data lake and data warehouse platforms.
• Develop incremental data pipelines for NoSQL databases, ensuring data accuracy and reliability.
• Handle structured and unstructured datasets in various file formats.
• Schedule and manage job workflows using industry-leading orchestrators.
• Set up and configure essential components like IAM, data catalog, logging, and monitoring.
• Collaborate with cross-functional teams to develop insightful dashboards and visualizations.

Requirements:

• Bachelor's or master's degree in computer science, engineering, or a related field.
• 5-8 years of experience working with big data technologies and cloud environments.
• Proficiency in Python, Spark (with Scala), Pyspark, and SQL stored procedures.
• Strong knowledge of building batch and streaming data pipelines from various sources.
• Familiarity with Apache Kafka and cloud-based streaming services.
• Experience with ETL tools and cloud-based services for data integration.
• Understanding of API data integration using popular methods.
• Hands-on experience with data lakes and data warehouses.
• Proficiency in handling diverse file formats.
• Prior experience in job scheduling using orchestrators.
• Familiarity with setting up IAM, data catalog, logging, and monitoring.
• Strong analytical and problem-solving skills.

If you are passionate about working with cutting-edge technologies, have a keen eye for detail, and thrive in a collaborative environment, we want to hear from you. Apply now to embark on an exciting journey as a Big Data Engineer!