Lead Data Engineer

Job Description

Position Overview: We are seeking a skilled and experienced Lead Data Engineer to join our growing data team. This role will involve designing, building, and maintaining scalable data pipelines and ETL processes. You will lead a team of data engineers and collaborate with data scientists, analysts, and stakeholders across the organization to deliver solutions that drive business insights.

Key Responsibilities:

• Lead the design, development, and implementation of data pipelines and ETL workflows using Matillion and Databricks.
• Architect and implement data solutions in AWS, including but not limited to Amazon S3, AWS Glue, Amazon Redshift, and Amazon RDS.
• Manage and optimize workflow orchestration using Apache Airflow to ensure reliable data processing and job scheduling.
• Collaborate with cross-functional teams to gather requirements and translate them into scalable data architecture and process designs.
• Mentor and guide junior data engineers, providing technical leadership and fostering a culture of continuous improvement and innovation.
• Ensure data quality and integrity by implementing best practices in data governance and validation.
• Monitor performance, troubleshoot issues, and optimize data systems for efficiency and scalability.
• Stay abreast of industry trends and emerging technologies to ensure continuous improvement of the data engineering practices.

Qualifications:

• Bachelor’s degree in Computer Science or related Engineering, or a related field.
• 10+ years of experience in data engineering or related roles, with a proven track record of leading technical teams.
• Extensive experience with AWS services, particularly in data storage, data processing, and database management.
• Hands-on experience with Matillion for ETL processes and data integration.
• Proficient in using Databricks for data analysis and processing with Apache Spark.
• Strong knowledge of Apache Airflow for workflow orchestration and scheduling.
• Experience with SQL and data modeling principles; familiarity with NoSQL databases is a plus.
• Excellent problem-solving skills and ability to work in a fast-paced, agile environment.
• Strong communication skills to effectively interact with technical and non-technical stakeholders.

Preferred Skills:

• Experience with Python, Pyspark, and SQL for data processing and scripting.
• Experience in AWS services,Databricks,Airflow & Matillion
• Familiarity with data visualization tools such as Power BI.
• Knowledge of machine learning practices and frameworks.

Current Employees apply HERE

Current Contingent Workers apply HERE

Search Firm Representatives Please Read Carefully 
Merck & Co., Inc., Rahway, NJ, USA, also known as Merck Sharp & Dohme LLC, Rahway, NJ, USA, does not accept unsolicited assistance from search firms for employment opportunities. All CVs / resumes submitted by search firms to any employee at our company without a valid written search agreement in place for this position will be deemed the sole property of our company.  No fee will be paid in the event a candidate is hired by our company as a result of an agency referral where no pre-existing agreement is in place. Where agency agreements are in place, introductions are position specific. Please, no phone calls or emails. 

Employee Status:

Regular

Relocation:

VISA Sponsorship:

Travel Requirements:

Flexible Work Arrangements:

Hybrid

Shift:

Valid Driving License:

Hazardous Material(s):

Required Skills:

Business Intelligence (BI), Database Administration, Data Engineering, Data Management, Data Modeling, Data Visualization, Information Management, Information Technology (IT) Infrastructure, Network Infrastructures, Software Development

Preferred Skills:

Job Posting End Date:

01/30/2025

*A job posting is effective until 11:59:59PM on the day BEFORE the listed job posting end date. Please ensure you apply to a job posting no later than the day BEFORE the job posting end date.