Project Description:

ML/Cloud-based system that efficiently analyzes collected data to predict/prevent/troubleshoot system failures and performance issues in smart-devices Multi-tenancy & medium-high data volume processing Data collected from smart devices is accessed from cloud (AWS) storage and undergoes translation from the device-specific schema, file formats, etc and transformations such as selection of relevant data and features before being applied to an ML model training subsystem; qualified models are then pushed to the production environment for prediction/execution. Data handling employs scalable spark-based access. The entire processing workflow is kept in sync via pipelines defined in Airflow. The state of the entire data engineering (& ML models, training, and execution) is available via Dashboard UI.

Responsibilities:

Work closely with team members in a friendly, agile, highly collaborative and supportive professional environment
• Work with scrum master(s), business analyst(s), tech lead(s) to analyze and understand user stories in each sprint
• Create design documents or make changes to existing ones
• Complete coding & unit testing for the allotted stories
• Participate in code reviews
• Participate in progress reviews

Skills

Must have

• At least 5 years of experience developing microservices with Java (version 8 or higher required) and Spring Boot.
• Experience with Relational Databases (Postgres, MySQL) and NoSQL Databases (Mongo).
• Experience with AWS Cloud technologies: S3, EMR, EC2, Glue, Athena.
• Experience with Maven, Gradle, Jenkins CI/CD.
• Experience building data pipelines, CI/CD pipelines, and fit-for-purpose data stores.

Nice to have

• Experience with Python.
• Experience with Apache Spark (Java or Python API).
• Experience with Apache Airflow.
• Experience with Dimensional Data Modeling.
• Experience building data pipelines that process more than 1TB both in streaming and batch mode kubeFlow , Spark on Kubernetes.