Generic Accountability
Developing and deploying data engineering workloads in Databricks on the Azure cloud infrastructure, managing and monitoring performance of these workloads, and ensuring security, availability, and scalability of cloud resources.
Specific Accountability
Work with the business lines to understand the data ingestion and cloud data engineering requirements
Work with the data modellers to translate the same into cloud/on-prem data engineering workloads design and then develop the same into appropriate engineering language – Databricks pipelines and ADF pipelines.
Very strong hands-on skills and at least 3 years experience in developing databricks and Azure Data Factory pipelines
Follow the principle of medallion architecture (3 layers of data in Azure data lake) of getting the data in right data zones basis the requirement
Work and collaborate in multi-disciplinary Agile teams, adopting Agile spirit, methodology and tools
Help code and scale applications in a multi-cloud environment integrated with on-premise
Work with the platform lead to spot out and remediate the potential operational risks of the platform
Minimum Qualification
Overall 5-7 years of experience in data
engineering and transformation on Cloud
3+ Years of Very Strong Experience in Azure
Data Engineering, Databricks
Expertise in supporting/developing Data
warehouse workloads at enterprise level
Experience in pyspark is required – developing
and deploying the workloads to run on the Spark distributed computing
Candidate
must possess at least a Graduate or bachelor’s degree in Computer
Science/Information Technology, Engineering (Computer/Telecommunication) or
equivalent.
Cloud
deployment: Preferably Microsoft azure
Experience
in implementing the platform and application monitoring using Cloud native
tools