Technical Lead

Role : Mid level/Senior Data Engineer

Experience : 6-8 Years

Mandatory Skill: Python, Astronomer, Apache Airflow

Location: Hyderabad

Notice Period:  less than 60 Days

Job Description:

Skills:  Apache Airflow, Astronomer, Python, SQL, Kafka

Responsibilities:

• Data Pipeline Development: Design, build, and maintain robust and scalable data pipelines using Apache Airflow and Astronomer to automate complex Business processes.
• Workflow Orchestration: Collaborate with Automation engineers, and business stakeholders to develop and manage complex Business workflows
• Performance Optimization: Continuously monitor the performance of Airflow DAGs and optimize them for efficiency, reliability, and scalability.
• Python Development: Write clean, maintainable, and efficient Python code for data processing, transformation, and analysis tasks.
• Best Practices: Establish and promote best practices for workflow orchestration, data pipeline design, and coding standards within the team.
• Testing and Maintenance: Implement testing strategies for data pipelines and workflows, ensuring high availability and minimizing errors.
• Documentation: Create and maintain comprehensive documentation for data pipelines, workflows, and architectural designs for team knowledge sharing and onboarding.
• Mentorship: Provide mentorship and guidance to junior developers and team members on best practices, tools, and data engineering concepts.
• Tool Integration: Stay updated on emerging technologies and platforms related to data engineering, and advocate for tool adoption and process enhancements.

Qualifications:

• Expert knowledge of Apache Airflow, including DAG creation, scheduling, and debugging.
• Proficiency with Astronomer for deploying and managing Airflow applications.
• Strong programming skills in Python, with experience in developing data processing applications and libraries.
• Familiarity with Azure cloud platform and services related to data processing and storage.
• Good understanding on distributed systems and experience building real-time integrations with Kafka
• Experience with version control (e.g., Git) and CI/CD practices.
• Knowledge of containerization technologies (e.g., Docker) and orchestration tools (e.g., Kubernetes) is preferred.
• Strong analytical and troubleshooting skills, with the ability to work independently as well as part of a collaborative team.
• Excellent communication skills, able to articulate technical concepts to both technical and non-technical stakeholders

Preferable

• 6+ years of professional software development experience
• 4+ years of Python, DAG and Airflow development
• 2+ years of Cloud Experience; preferably with Amazon Web Services (AWS)
• 2+ year of experience working in an Agile-based development environment using Agile concepts such as Continuous Integration (CI/CD) and Test-Driven Development (TDD)
• Strong experience with monitoring and observability and building scalable services
• Understanding of various database solutions (SQL, NoSQL)
• Knowledge of Azure, GCP, AWS