Job Description
Designed and developed end-to-end ETL data engineering pipeline Implemented Medallion architecture as the design pattern using Data Lakehouse with Delta Lake. Ingested and stored multi-source data into Azure Data Lake Storage (ADLS) Gen2, improving data accessibility. Processed and transformed raw data using Azure Data Factory, Azure Databricks, and Pyspark. Used Databricks as an analytics platform to transform and move data across layers by creating delta tables. Used Azure services such as Azure SQL, Azure Key Vault, and Azure Synapse Analytics. Integrated Azure DevOps for Cl/CD pipelines, automating the deployment and data engineering workflows. Collaborated with data scientists and analysts on data transformations and analytics.
