Project description
We are hiring for projects in the Middle East. There are many opportunities in the region. Our team consists of frontend and backend developers, data analysts and data scientists, architects, analysts and project managers, and Databricks Data engineers. As a Lead Data Engineer, you will manage the development, implementation and maintenance of data infrastructure for modern data plarform.
Responsibilities
Platform Design: Collaborate with cross-functional teams to refine and design the architecture of our data platform.
Platform Optimization: Continuously enhance the performance, scalability, and reliability of the data platform to meet evolving business requirements.
Platform Maintenance: Implement monitoring, alerting, and maintenance processes to ensure the availability and health of the data platform.
Data Governance: Establish and enforce data governance best practices to uphold data quality, security, and compliance standards.
Platform Integration: Integrate new technologies seamlessly into the data platform ecosystem, ensuring compatibility and interoperability.
Collaboration: Work closely with data engineers, data scientists, analysts, and business stakeholders to comprehend requirements and offer technical expertise and support.
Documentation: Create and maintain comprehensive documentation for the data platform architecture, processes, and workflows.
Mentorship: Provide guidance and mentorship to junior members of the data engineering team, nurturing their growth and development.
Design, develop, and maintain data pipelines to ingest and transform raw data into Silver and Gold tables using Databricks.
Build reusable data pipelines that can be utilized by application development team
Collaborate with data consumers and other stakeholders to understand data requirements and deliver high-quality data solutions.
Implement declarative ETL pipelines using Databricks, Delta Lake, and other relevant tools to ensure data is accurate, timely, and accessible.
Optimize data models and pipelines for performance, scalability, and reliability.
Model data from various sources, ensuring consistency and quality in the Silver layer.
Curate data assets in the Silver layer to support data science and machine learning applications.
Build and maintain the Gold layer to provide clean, aggregated, and business-ready data for reporting and analytics.
Ensure data governance and security best practices are followed.
Well versed with databricks fine tuning techniques for huge data sets
Document data processes, models, and pipelines for transparency and knowledge sharing.
Well versed with Devops practices for continuous integration and deployment
Writing Code, review pull requests, provide constructive feedback and maintain high code quality standards
Maintain a deep understanding of data engineering principles, best practices and emerging technologies to drive engineering initiatives
Skills
Must have
Overall 5-7 years of experience in data engineering and transformation on Cloud
3+ Years of Very Strong Experience in Azure Data Engineering, Databricks
Expertise in supporting/developing lakehouse workloads at enterprise level
Experience in pyspark is required
developing and deploying the workloads to run on the Spark distributed computing
Candidate must possess at least a Graduate or bachelor's degree in Computer Science/Information Technology, Engineering (Computer/Telecommunication) or equivalent.
Cloud deployment: Preferably Microsoft azure
Experience in implementing the platform and application monitoring using Cloud native tools
Nice to have
0-2 0-2 2-5 5+ This field is required
Under the terms of your specific consent or to perform our obligations under a contract with you, as applicable, we, Luxoft Holding Inc. will manually and electronically process your personal data, specifically your first name, last name, phone number, e-mail address and other data you provide us through this form.
Within this context, we process personal data only for the specific purpose(s) indicated in the individual consent language or other notices provided below.
We will – insofar as reasonably necessary for the purpose you have agreed to and within the scope of applicable laws – transfer your personal data to other entities within the Luxoft Group and to the group of third party recipients listed in our Privacy Notice. Such Recipients can be located outside the European Union (EU) and/or the European Economic Area (EEA) (“Third Countries”). The Third Countries concerned, e.g. the USA, may not have the level of data protection that you enjoy e.g. under the GDPR. This can result in disadvantages such as an impeded enforcement of data subjects’ rights, a lack of control over further processing and access by state authorities. You may only have limited legal remedies against this. Insofar our transfer of your personal data to recipients in Third Countries is not covered by an adequacy decision of the EU Commission, we achieve an adequate level of data protection as further detailed out in our Privacy Notice.
With your consent, we personalise marketing communications to you by way of carrying out marketing research analysis, analysing the surfing-behaviour of our website visitors and to adjust it to their detected tendencies, as well as to plan more efficient future marketing activities. This personalised marketing does not include any automated decision-making activities.
Further information on how we process personal data in general is available in our . You may withdraw any given consent at any time. The withdrawal of your consent(s) will not affect the lawfulness of processing before its withdrawal. For any request in this context, please e-mail us at: DPO@luxoft.com.
Before uploading CV or any other information to this website, to learn more about your obligations and restrictions arising from the use of this website, please read our .
