Job Description
Role : Mid level/Senior Data Engineer
Experience : 6-8 Years
Mandatory Skill: Python, Astronomer, Apache Airflow
Location: Hyderabad
Notice Period: less than 60 Days
Job Description:
Skills: Apache Airflow, Astronomer, Python, SQL, Kafka
Responsibilities:
- Data Pipeline Development: Design, build, and maintain robust and scalable data pipelines using Apache Airflow and Astronomer to automate complex Business processes.
- Workflow Orchestration: Collaborate with Automation engineers, and business stakeholders to develop and manage complex Business workflows
- Performance Optimization: Continuously monitor the performance of Airflow DAGs and optimize them for efficiency, reliability, and scalability.
- Python Development: Write clean, maintainable, and efficient Python code for data processing, transformation, and analysis tasks.
- Best Practices: Establish and promote best practices for workflow orchestration, data pipeline design, and coding standards within the team.
- Testing and Maintenance: Implement testing strategies for data pipelines and workflows, ensuring high availability and minimizing errors.
- Documentation: Create and maintain comprehensive documentation for data pipelines, workflows, and architectural designs for team knowledge sharing and onboarding.
- Mentorship: Provide mentorship and guidance to junior developers and team members on best practices, tools, and data engineering concepts.
- Tool Integration: Stay updated on emerging technologies and platforms related to data engineering, and advocate for tool adoption and process enhancements.
Qualifications:
- Expert knowledge of Apache Airflow, including DAG creation, scheduling, and debugging.
- Proficiency with Astronomer for deploying and managing Airflow applications.
- Strong programming skills in Python, with experience in developing data processing applications and libraries.
- Familiarity with Azure cloud platform and services related to data processing and storage.
- Good understanding on distributed systems and experience building real-time integrations with Kafka
- Experience with version control (e.g., Git) and CI/CD practices.
- Knowledge of containerization technologies (e.g., Docker) and orchestration tools (e.g., Kubernetes) is preferred.
- Strong analytical and troubleshooting skills, with the ability to work independently as well as part of a collaborative team.
- Excellent communication skills, able to articulate technical concepts to both technical and non-technical stakeholders
Preferable
- 6+ years of professional software development experience
- 4+ years of Python, DAG and Airflow development
- 2+ years of Cloud Experience; preferably with Amazon Web Services (AWS)
- 2+ year of experience working in an Agile-based development environment using Agile concepts such as Continuous Integration (CI/CD) and Test-Driven Development (TDD)
- Strong experience with monitoring and observability and building scalable services
- Understanding of various database solutions (SQL, NoSQL)
- Knowledge of Azure, GCP, AWS
