Summarized Accountability:

• The data engineer will play a pivotal role in building and operationalizing data for enterprise data and analytics initiatives following industry best practices. The bulk of the data engineer's work would be building, managing, and optimizing data pipelines to support the enterprise data platform leveraged by critical data and analytics consumers.
• The newly hired data engineer will need strong collaboration skills to work with varied stakeholders within the organization. In particular, the data engineer will work closely with analytics teams and with business analysts (translators) to refine their data requirements for various data and analytics initiatives and their data consumption requirements.
• Candidates need to be very much self-driven, curious, and creative.

Key Responsibilities:

• Building, monitoring, and optimizing data pipelines
• Learning and using modern data preparation, integration, and metadata management tools and techniques
• Monitoring and remediating data quality issues
• Tracking data consumption patterns
• Monitoring schema changes
• Experience with data management disciplines is strongly preferred, including data integration, modeling, optimization, data quality, and other areas directly relevant to data engineering responsibilities and tasks.

Experience & Education:

• Bachelor’s degree in computer science or information system related field
• Minimum 3 years relevant experience
• Strong Problem-Solving and Analytical Skills
• Candidates will ideally have a combination of IT skills, data governance skills, and analytics skills with a technical or computer science degree

IS Technical Discipline:

• Must be hands-on/well-verses in Python/PySpark coding and scripting
• Strong expertise and hands-on experience with Azure Data Factory and Azure Databricks
• Databricks Lakehouse architecture experience strongly preferred
• Programming/Coding Skills with familiarity with software and system engineering design principles & standards (e.g., TDD)
• Foundational knowledge of data management architectures like Data Warehouse, Data Lake, Data Hub and the supporting processes like Data Integration, Governance, Metadata Management
• Ability to design, build and manage data pipelines for data structures encompassing data transformation, data models, data quality and observability, schemas, metadata, and job management
• Understanding of structured, semi-structured, and unstructured data sources
• Experience with data integration technologies such as ETL/ELT, data replications/CDC, message-oriented data movement, API design and access, stream data integration, and data virtualization
• Basic understanding of machine learning algorithms and approaches
• Knowledge of/proficiency using one or more popular languages and frameworks such as SQL, Python, Scala, Spark, Command Line, etc.
• Familiarity with Open-Source and Commercial (e.g., Azure Cloud Services) technology
• Ability to automate development via CICD patterns and processes
• Experience leveraging version control and repository management systems (Git experience recommended)
• Experience leveraging IDE and source code editors (e.g., Visual Studio Code)
• Data visualization (e.g., Power BI, Tableau, QlikSense)

Business Skills:

• Self-driven and action-oriented
• Ability to work in diverse, cross-functional teams
• Strong collaboration skills with business & technical personas
• Strong moderation and communication skills
• Strong presentation skills, including storytelling and other techniques to guide and inspire
• Ability to understand business & functional needs and translate into data analytics problems
• Ability to refine requirements and design and develop data deliverables accordingly
• Display drive and curiosity to understand the business process to its core and its intersections with data
• Network with internal and external partners
• Continuous learning and upskilling (conferences, publications/research, courses, meetups, online resources, etc.)
• Educating and training business and technical counterparts
• Cultivates innovation and creativity
• Familiarity with design thinking techniques and agile & lean methodologies