Business Data Technologies (BDT) makes it easier for teams across Amazon to produce, store, catalog, secure, move, and analyze data at massive scale. Our managed solutions combine standard AWS tooling, open-source products, and custom services to free teams from worrying about the complexities of operating at Amazon scale. This lets BDT customers move beyond the engineering and operational burden associated with managing and scaling platforms, and instead focus on scaling the value they can glean from their data, both for their customers and their teams.
We own the one of the biggest (largest) data lakes for Amazon where 1000’s of Amazon teams can search, share, and store EB (Exabytes) of data in a secure and seamless way; using our solutions, teams around the world can schedule/process millions of workloads on a daily basis. We provide enterprise solutions that focus on compliance, security, integrity, and cost efficiency of operating and managing EBs of Amazon data.
Key job responsibilities
CORE RESPONSIBILITIES:
· Be hands-on with ETL to build data pipelines to support automated reporting
· Interface with other technology teams to extract, transform, and load data from a wide variety of data sources
· Implement data structures using best practices in data modeling, ETL/ELT processes, and SQL, Redshift.
· Model data and metadata for ad-hoc and pre-built reporting
· Interface with business customers, gathering requirements and delivering complete reporting solutions
· Build robust and scalable data integration (ETL) pipelines using SQL, Python and Spark.
· Build and deliver high quality data sets to support business analyst, data scientists, and customer reporting needs.
· Continually improve ongoing reporting and analysis processes, automating or simplifying self-service support for customers
· Participate in strategic & tactical planning discussions
A day in the life
As a Data Engineer, you will be working with cross-functional partners from Science, Product, SDEs, Operations and leadership to translate raw data into actionable insights for stakeholders, empowering them to make data-driven decisions. Some of the key activities include:
Crafting the Data Flow: Design and build data pipelines, the backbone of our data ecosystem.
Ensure the integrity of the data journey by implementing robust data quality checks and monitoring processes.
Architect for Insights: Translate complex business requirements into efficient data models that optimize data analysis and reporting. Automate data processing tasks to streamline workflows and improve efficiency.
Become a data detective! ensuring data availability and performance
- 1+ years of data engineering experience
- Experience with SQL
- Experience with data modeling, warehousing and building ETL pipelines
- Experience with one or more query language (e.g., SQL, PL/SQL, DDL, MDX, HiveQL, SparkSQL, Scala)
- Experience with one or more scripting language (e.g., Python, KornShell)
- Experience as a Data Engineer or in a similar role
- Experience with big data technologies such as: Hadoop, Hive, Spark, EMR
- Experience as a data engineer or related specialty (e.g., software engineer, business intelligence engineer, data scientist) with a track record of manipulating, processing, and extracting value from large datasets