Job Description
Apt Resources is searching for a talented BIG Data Engineer to join a leading Digital Engineering Company based in Abu Dhabi. The ideal candidate will have a strong background in batch data processing, the development and maintenance of scalable ETL/ELT pipelines, and expertise with cloud platforms like Azure. This role focuses on designing and building high-performance data solutions to meet business needs, ensuring efficient data processing, and upholding high data quality standards.
Key Responsibilities:
- Design, develop, and maintain scalable batch ETL/ELT pipelines using Azure Data Factory and other Azure services (such as Azure Table Storage, Azure Data Lake, etc.).
- Collaborate closely with data architects and cross-functional teams to understand business requirements and translate them into technical data solutions.
- Optimize data processing pipelines for performance, reliability, and cost-effectiveness.
- Perform data integration, cleansing, and validation to ensure high data quality and consistency across systems.
- Monitor, troubleshoot, and resolve issues related to batch processing jobs to ensure timely and accurate data delivery.
- Ensure compliance with data governance and security standards, and implement best practices in data management.
- Contribute to the continuous improvement of data infrastructure, implementing new tools and technologies to support evolving business needs.
- Work on integrating data from various sources, ensuring seamless data flows into storage and processing systems.
