• Design and develop data feeds from an on-premise environment into a datalake environment in an AWS cloud environment
• Design and develop programmatic transformations of the solution, by correctly partitioning,  formatting and validating the data quality
• Design and develop programmatic transformation, combinations and calculations to populate complex datamarts based on feed from the datalake
• Provide operational support to datamart datafeeds and datamarts
• Design infrastructure required to develop and operate datalake data feeds
• Design infrastructure required to develop and operate datamarts, their user interfaces and the feeds required to populate the datalake.

Job Related Knowledge:

Creating data feeds from on-premise to AWS Cloud

Support data feeds in production on break fix basis

Creating data marts using Talend or similar ETL development tool

Manipulating data using python and pyspark

Processing data using the Hadoop paradigm particularly using EMR, AWS’s distribution of Hadoop

Devop for Big Data and Business Intelligence including automated testing and deployment

Job Related Skill:

Talend

AWS: EMR, EC2, S3

Python

PySpark or Spark

Business Intelligence Data Modelling

SQL

AWS experience

Good planning and team leading skills

Strong focus on data engineering, i.e. building pipelines, big data environments

** Please note - if you have not had any feedback within 2 weeks of your CV submission, please consider your application unsuccesful **