Designation: Data Engineer

• Total experience:  4+ years
• CTC :  15-20Lacs
• Location – Bhandup

Job Description:

At least 4 years of hands on experience in : 

• Working with Big data technologies like Hadoop, Hive, Presto, Spark, HQL, Elasticsearch, and yarn.
• Working with any of the cloud environments (GCP/AWS/Azure). Cloud Certification(GCP Data Engineer or AWS SA or Azure Data Engineer) shall be a plus
• Working with Python, Spark(with scala) or Pyspark, SQL stored procedures
• Building Batch and Streaming data pipelines from various data sources like RDBMS, NoSQL, IoT/telemetry data, API's to various data lake or data warehouses.
• Building Streaming data ingestion pipelines using  Apache Kafka and cloud-based services (like AWS IOT core/Kinesis/MSK, GCP Pub/sub, etc)
• Using various ETL tools like Apache Spark, and native cloud-based services like GCP Dataflow/Dataproc/Data Fusion, AWS Glue, Azure Data Factory
• API data Integration using different methods like POSTMAN, CURL, Python libraries
• Worked with various Data Lake and Data Warehouses(cloud-based(like Bigquery/Redshift/Snowflake/Synapse/S3/GCS/Blob) and on-prem open source)
• Developing an incremental data pipeline NoSQL Database (either MongoDB/AWS Dynamo DB/Azure Cosmos DB/GCP Big Table/ GCP firestore)
• Working with structured/unstructured datasets, different file formats like Avro/Parquet/Json/CSV/XML/text file
• Worked on various file formats like csv, json, parquet, avro, text files etc.
• Job Scheduling using orchestrators preferably Airflow
• Setting up IAM, Data Catalog, Logging, and monitoring using any of the cloud or open source-based services
• Developing dashboards using any BI tool(Power BI/Tableau/Qlik) shall be a plus
• Developing web crawlers like social media crawlers shall be a plus