الوصف الوظيفي
We are seeking a Data Engineer with 2 to 4 years of experience, specializing in SQL, Python, Delta Lake (Delta Tables), and ETL development on Databricks.
She/he would work in a Data Mesh architecture, with opportunities to design and manage data products.
Knowledge of the Data Mesh framework and its principles would be beneficial.
Key Responsibilities:
Design, develop, and maintain ETL pipelines using Python, Spark and Databricks, with a focus on Delta Lake for real-time and batch data processing.
Ensure efficient data integration from multiple sources and maintain the reliability of data flows, working closely with cross-functional teams.
Work within a Data Mesh architecture, contributing to the creation and management of data products.
Optimize performance and scalability of data pipelines, ensuring high-quality data delivery.
Collaborate with data scientists, analysts, and other stakeholders to ensure the successful implementation of data requirements.
Communicate technical challenges and solutions effectively across different teams, ensuring alignment and smooth project delivery.
Ensure data security, governance, and quality best practices are followed throughout the data lifecycle.
Qualifications:
2-4 years of experience in data engineering, with a strong focus on:
Spark SQL
Python(PySpark) for ETL and data processing
Delta Lake (Delta Tables) for managing large-scale data systems.
Databricks for developing and managing data workflows.
Solid understanding of ETL development and data pipeline optimization.
Proven ability to communicate technical challenges and solutions effectively across cross-functional teams.
Experience working with multiple data sources and integrating them into cohesive data systems.
Familiarity with Azure cloud services (e.g., Azure Data Lake, Azure Databricks).
Strong knowledge of SQL for data manipulation and optimization.
Experience working in a Data Mesh architecture is a plus.
Understanding of data products and their lifecycle is beneficial.
Ability to work both independently and in collaboration with various teams.
Nice to Have:
Experience with CI/CD pipelines for data engineering.
Familiarity with data governance, security best practices, and compliance standards.
Exposure to data warehousing and streaming data (e.g., Kafka, Event Hubs).
