This job is no longer available.

You can view related vacancies or set-up an email alert notification when similar jobs are added to the website using the buttons below.

Big Data Engineer(Development & Production Support Role)

Job Description

My Client is building an on-demand entertainment platform which enables consumers to access relevant content at super-fast speeds. They setup their zones across Transport, Hospitality, Retail, Healthcare and even Public Places so that everyone can enjoy high quality, buffer-free, on-demand entertainment - without being dependent on access to 4G or broadband.

Role:Data Engineer -(Development & Production Support)

  • Collaborate with architects, business users, source (data) team members for discovering data sources and tech feasibility for fetching the same into the consolidated data environment/platform
  • Iteratively design and build core components of our Data Platform (Hortanworks Big Data implementations with ecosystem products such as Kafka, Spark Streaming, Hive, Sqoop, MySQL and <one BI tool> etc)
  • Take ownership of design & development processes, ensuring incorporation of best practices, sanity of code and versioning into different environments through tools like git etc
  • Ability to draw intelligence from structured, non structured, large and complex data sets to meet functional and non functional business requirements
  • Ability to steer the team members for unit development and troubleshooting the development/production issues
  • Proven ability to operate in mix mode of incremental development, production support and support to business for their impromptu data quests
  • Willingness to participate and provide adheareable inputs, triaging the dependencies to the quarterly (team deliverables) planning

    **** For this role, hands-on big data technologies such as Spark(Streaming), Hive, Sqoop etc and previous experience in the data team is non negotiable. Spark, kafka, Hadoop
  • Iteratively design and build core components of our Data Platform
  • Process structured and unstructured data, validate data quality, help to design automated data quality tests in Data Platform.
  • Interact with a large range of proprietary and open data sources, leveraging data in all sorts of formats including files
  • Assist in the development and support of data sets required by the business.
  • Collaborate with architects, engineers, data scientist, business team members on project planning and goals.
  • Assemble large, complex data sets that meet functional and non-functional business requirements
  • Identify, design, and implement internal process improvements - automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability
  • Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources
  • Help scale the data science models into production-level models that can handle real-time data