Job Description
Job
Overview:We are seeking a talented and motivated Data Engineer with a minimum 3 years of experience to join our dynamic team. The ideal candidate will possess a strong background in designing, building, and maintaining scalable data pipelines and databases. You will work closely with engineering teams and product owner to translate requirements into resilient, secure, and high-performance software. Key responsibilities include designing, coding, testing, and implementing new or existing software products based on specifications. A crucial aspect of this role is a commitment to continuous learning and adaptability to evolving technologies and market trends.
Key
Responsibilities:- Data Pipeline Development:
- Design, build, and maintain robust and scalable data pipelines to process and transform large volumes of data from various sources.
- Implement ETL (Extract, Transform, Load) processes to ensure data is accurately and efficiently integrated into data warehouses or data lakes.
Database
Management:- Develop, build and optimize database queries and schema designs to enhance performance and support analytical needs.
Data
Integration:- Collaborate with cross-functional teams to gather data requirements and integrate data from diverse sources including APIs, flat files, and databases.
- Ensure data consistency, accuracy, and integrity across various systems.Data Quality and Monitoring:
- Implement data quality checks and monitoring processes to identify and resolve data issues proactively.
- Develop and maintain data documentation and metadata for transparency and reproducibility.
Performance Tuning:- Monitor and tune the performance of data pipelines and databases to ensure optimal operation and scalability.
- Troubleshoot and resolve data-related issues in a timely manner.
Collaboration and
Communication:- Work closely with various stakeholders to understand their data needs and provide appropriate solutions.
- Communicate technical concepts and solutions effectively to both technical and non-technical team members.
Continuous Learning and
Adaptation:- Stay current with emerging technologies, trends, and best practices in data engineering.
- Demonstrate a proactive attitude towards learning new tools and technologies, and adapt quickly to changes in the market and industry requirements.
Qualifications:- Education:
- Bachelor’s degree in Computer Science, IT, Engineering or a related field
Experience:- 3+ hands-on experience in data engineering
- Proficiency with data warehousing solutions and ETL tools.
Technical Skills:- Proficient in SQL: Extensive experience in writing complex queries, stored procedures, and functions in SQL and Azure SQL Managed Instance (SQL-MI).
- Data Pipeline Orchestration: Hands-on experience with data pipeline orchestration tools such as SSIS, Azure Data Factory, and Azure Synapse.
- Knowledge of Microsoft Fabric: Familiarity with end-to-end analytics and data platform Microsoft Fabric
- Programming Skills: Proficiency in Python and PySpark for data manipulation and analysis.
- Azure DevOps: Experience with Azure DevOps for version control, continuous integration, and deployment.
- Repository Management: Strong practical experience with code repositories such as Azure Repos and Bitbucket.
- Database Concepts: Solid understanding of Relational Database Management Systems (RDBMS) and data warehousing concepts.
- Cloud Platforms: Experience working with Azure – Azure Virtual Machines, Azure Storage Account, Azure Key Vault
Nice to
have:- Data Visualization: Knowledge / familiarity with data visualization tools like Power BI to create insightful and interactive reports.
- Programming Knowledge: Familiarity with additional programming languages such as
.NET, Java, or front-end technologies like React/JavaScript is a plus.
- Project Management & Issue Tracking: Familiarity/exposure in using Jira - creating/managing stories, sub-tasks, tasks, bugs
Soft
Skills:- Excellent problem-solving skills and attention to detail.
- Strong analytical and organizational abilities.
- Ability to work effectively both independently and as part of a team.
- Effective communication skills to interact with various stakeholders.
- Willingness to learn new technologies and adapt to rapid changes in the data engineering landscape.
