Role Objectives As a Senior Data Engineer, you will be responsible for designing, building, and maintaining a self-serve data platform used by other data professionals. Your expertise in infrastructure and software engineering will be crucial in providing services, tooling, and environments that enable efficient data ingestion, integration, storage, transformation, and pipelines orchestration. You will work on exciting projects such as the metrics platform, experimentation platform, ML platform, real-time data ingestion and processing engine and more!
Roles & Responsibilities:
Collaborate on the architecting of the data platform and technologies to address emerging needs and account for future scale.
Deploy data platform different components.
Onboard data team on using the platform.
Monitor data platform components.
Develop internal data tooling that enable new capabilities or streamline existing workflows.
Work on adding new data pipelines either batch or real time, fulfilling latency SLOs.
Develop data quality and governance initiatives to enhance the visibility of data issues due to wrong data entry/system bugs/etc.
Advocate for software engineering best practices among the data team.
REQUIRED EDUCATION, KNOWLEDGE, AND SKILLS
3-5 years of experience in relevant domains including software engineering/DevOps/architecture.
Strong expertise in Python and SQL, experience with Java/Scala is a plus.
Ability to write clean, well designed code bases.
Knowledge of ETL/ELT concepts.
Strong experience with applying software engineering practices.
Experience handling data infrastructure.
Experience building near real time data pipelines.
Deep understanding with the inner workings of databases (Eg. Indexes, Row-based vs Columnar based databases, Partitioning data, ...etc.)
Experience with DevOps processes and technologies including CI/CD, Terraform, Kubernetes, GitHub, etc.
Having a structured organized mindset.
Experience with handling document-based databases (ex. MongoDB) for analytics.
Experience with any major Cloud provider. Experience with GCP is a big plus.
Experience with Airflow, Dagster, or any data orchestration tool.
