Data Engineer

Data EngineerGlobal Data Platforms & Operations is a Team within DHL Supply Chain – AI, Data & Analytics Centre of Excellence. The department works with state-of-the-art technologies in Data Management and Data integration. We define data reference architecture for Supply Chain globally and design, implement and support end-to-end data pipelines for our Data Analytics applications.
DHL Supply Chain is offering a unique chance to become a Data Engineer in the Global Data Platforms & Operations Team where you will be working on Data Lakehouse and Data Warehousing projects. You will be initially responsible for:
• Designing, developing and maintaining near-real time ingestion pipelines through Qlik Replicate (or alternative technology) replicating data from transactional Databases to our Data Eco-system powered by Azure Data Lake and Snowflake.
• Monitoring and supporting batch data pipelines from transactional Databases to our Data Eco-system powered by Azure Data Lake and Snowflake.
• Setting up new and monitoring of existing metrics, analyzing data, performing root cause analysis and proposing issue resolutions.
• Managing the lifecycle of all incidents, ensuring that normal service operation is restored as quickly as possible and minimizing the impact on business operations.
• Document data pipelines, data models, and data integration processes to facilitate knowledge sharing and maintain data lineage.
• Cooperate with other Data Platform & Operations team members and our stakeholders to identify and implement system and process improvements.
• Leveraging DevOps framework and CI/CD.
• Supporting and promoting Agile way of working using SCRUM framework.
What will you need?
To be successful in this role you will need to have excellent analytic skills related to working with structured, semi structured, and unstructured datasets, expert SQL knowledge and experience with relational as well as no-SQL database systems.
Furthermore, you need to have experience with Data Warehousing for large complex data sets – defining, building and optimizing data models based on use case requirements.
Ideally you also have experience with developing and/or maintaining Python/PySpark data pipelines.
You must have:
• Bachelor’s degree in Engineering/Technology;
• Minimum 2-3 years of experience in the Data Engineer role;
• Expertise using relational Database systems such as Oracle, MS/Azure SQL, MySQL, etc.;
• Expert SQL knowledge. It’s great it you have experience with Snowflake SaaS data warehouse or alternative solutions. • Practical experience developing and/or supporting CDC data pipelines – we use Qlik Replicate but any other technology is welcome!
• Experience with development and/or support of Lakehouse architectures – we use Parquet / Delta, Synapse Serverless and Databricks/Databricks SQL;
• Proficiency in Python programming and experience with PySpark libraries and APIs;
• Very good understanding of Software Development Lifecycle, source code management, code reviews, etc.;
• Experience in; managing of Incident life-cycle from ticket creation till closure (we use Service Now and JIRA)
• Experience in performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement;
• Experience in building processes supporting data transformation, data structures, metadata, dependency and workload management;
You should have:
• Experience with Data Lake/Big Data Projects implementation in Cloud (preferably MS Azure) and/or On-premise platforms:
o Cloud – Azure technology stack: ADLS Gen2, Databricks (proven experience is a big plus!), EventHub, Stream Analytics, Synapse Analytics, AKS, Key Vault;
o On Premise: Spark, HDFS, Hive, Hadoop distributions (Cloudera or MapR), Kafka, Airflow (or any other scheduler)
• Working experience with DevOps frameworkData EngineerGlobal Data Platforms & Operations is a Team within DHL Supply Chain – AI, Data & Analytics Centre of Excellence. The department works with state-of-the-art technologies in Data Management and Data integration. We define data reference architecture for Supply Chain globally and design, implement and support end-to-end data pipelines for our Data Analytics applications.
DHL Supply Chain is offering a unique chance to become a Data Engineer in the Global Data Platforms & Operations Team where you will be working on Data Lakehouse and Data Warehousing projects. You will be initially responsible for:
• Designing, developing and maintaining near-real time ingestion pipelines through Qlik Replicate (or alternative technology) replicating data from transactional Databases to our Data Eco-system powered by Azure Data Lake and Snowflake.
• Monitoring and supporting batch data pipelines from transactional Databases to our Data Eco-system powered by Azure Data Lake and Snowflake.
• Setting up new and monitoring of existing metrics, analyzing data, performing root cause analysis and proposing issue resolutions.
• Managing the lifecycle of all incidents, ensuring that normal service operation is restored as quickly as possible and minimizing the impact on business operations.
• Document data pipelines, data models, and data integration processes to facilitate knowledge sharing and maintain data lineage.
• Cooperate with other Data Platform & Operations team members and our stakeholders to identify and implement system and process improvements.
• Leveraging DevOps framework and CI/CD.
• Supporting and promoting Agile way of working using SCRUM framework.
What will you need?
To be successful in this role you will need to have excellent analytic skills related to working with structured, semi structured, and unstructured datasets, expert SQL knowledge and experience with relational as well as no-SQL database systems.
Furthermore, you need to have experience with Data Warehousing for large complex data sets – defining, building and optimizing data models based on use case requirements.
Ideally you also have experience with developing and/or maintaining Python/PySpark data pipelines.
You must have:
• Bachelor’s degree in Engineering/Technology;
• Minimum 2-3 years of experience in the Data Engineer role;
• Expertise using relational Database systems such as Oracle, MS/Azure SQL, MySQL, etc.;
• Expert SQL knowledge. It’s great it you have experience with Snowflake SaaS data warehouse or alternative solutions. • Practical experience developing and/or supporting CDC data pipelines – we use Qlik Replicate but any other technology is welcome!
• Experience with development and/or support of Lakehouse architectures – we use Parquet / Delta, Synapse Serverless and Databricks/Databricks SQL;
• Proficiency in Python programming and experience with PySpark libraries and APIs;
• Very good understanding of Software Development Lifecycle, source code management, code reviews, etc.;
• Experience in; managing of Incident life-cycle from ticket creation till closure (we use Service Now and JIRA)
• Experience in performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement;
• Experience in building processes supporting data transformation, data structures, metadata, dependency and workload management;
You should have:
• Experience with Data Lake/Big Data Projects implementation in Cloud (preferably MS Azure) and/or On-premise platforms:
o Cloud – Azure technology stack: ADLS Gen2, Databricks (proven experience is a big plus!), EventHub, Stream Analytics, Synapse Analytics, AKS, Key Vault;
o On Premise: Spark, HDFS, Hive, Hadoop distributions (Cloudera or MapR), Kafka, Airflow (or any other scheduler)
• Working experience with DevOps frameworkData EngineerGlobal Data Platforms & Operations is a Team within DHL Supply Chain – AI, Data & Analytics Centre of Excellence. The department works with state-of-the-art technologies in Data Management and Data integration. We define data reference architecture for Supply Chain globally and design, implement and support end-to-end data pipelines for our Data Analytics applications.
DHL Supply Chain is offering a unique chance to become a Data Engineer in the Global Data Platforms & Operations Team where you will be working on Data Lakehouse and Data Warehousing projects. You will be initially responsible for:
• Designing, developing and maintaining near-real time ingestion pipelines through Qlik Replicate (or alternative technology) replicating data from transactional Databases to our Data Eco-system powered by Azure Data Lake and Snowflake.
• Monitoring and supporting batch data pipelines from transactional Databases to our Data Eco-system powered by Azure Data Lake and Snowflake.
• Setting up new and monitoring of existing metrics, analyzing data, performing root cause analysis and proposing issue resolutions.
• Managing the lifecycle of all incidents, ensuring that normal service operation is restored as quickly as possible and minimizing the impact on business operations.
• Document data pipelines, data models, and data integration processes to facilitate knowledge sharing and maintain data lineage.
• Cooperate with other Data Platform & Operations team members and our stakeholders to identify and implement system and process improvements.
• Leveraging DevOps framework and CI/CD.
• Supporting and promoting Agile way of working using SCRUM framework.
What will you need?
To be successful in this role you will need to have excellent analytic skills related to working with structured, semi structured, and unstructured datasets, expert SQL knowledge and experience with relational as well as no-SQL database systems.
Furthermore, you need to have experience with Data Warehousing for large complex data sets – defining, building and optimizing data models based on use case requirements.
Ideally you also have experience with developing and/or maintaining Python/PySpark data pipelines.
You must have:
• Bachelor’s degree in Engineering/Technology;
• Minimum 2-3 years of experience in the Data Engineer role;
• Expertise using relational Database systems such as Oracle, MS/Azure SQL, MySQL, etc.;
• Expert SQL knowledge. It’s great it you have experience with Snowflake SaaS data warehouse or alternative solutions. • Practical experience developing and/or supporting CDC data pipelines – we use Qlik Replicate but any other technology is welcome!
• Experience with development and/or support of Lakehouse architectures – we use Parquet / Delta, Synapse Serverless and Databricks/Databricks SQL;
• Proficiency in Python programming and experience with PySpark libraries and APIs;
• Very good understanding of Software Development Lifecycle, source code management, code reviews, etc.;
• Experience in; managing of Incident life-cycle from ticket creation till closure (we use Service Now and JIRA)
• Experience in performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement;
• Experience in building processes supporting data transformation, data structures, metadata, dependency and workload management;
You should have:
• Experience with Data Lake/Big Data Projects implementation in Cloud (preferably MS Azure) and/or On-premise platforms:
o Cloud – Azure technology stack: ADLS Gen2, Databricks (proven experience is a big plus!), EventHub, Stream Analytics, Synapse Analytics, AKS, Key Vault;
o On Premise: Spark, HDFS, Hive, Hadoop distributions (Cloudera or MapR), Kafka, Airflow (or any other scheduler)
• Working experience with DevOps framework