Data Architect

As a Data Architect, you are required to:

Design & develop technical solutions which combine disparate information  to create meaningful insights for business, using Big-data architectures

• Build and analyze large, structured and unstructured databases based on scalable cloud infrastructures
• Develop prototypes and proof of concepts using multiple data-sources and big-data technologies
• Process, manage, extract and cleanse data to apply Data Analytics in a meaningful way
• Design and develop scalable end-to-end data pipelines for batch and stream processing
• Regularly scan the Data Analytics landscape to stay up to date with latest technologies, techniques, tools and methods in this field
• Stay curious and enthusiastic about using related technologies to solve problems and enthuse others to see the benefit in business domain

Qualification:

•  Bachelor's or Master's in Computer Science & Engineering, or equivalent. Professional Degree in Data Engineering / Analytics is desirable.

Experience level: 

• Minimum 8 years in software development with at least 2 - 3 years hands-on experience in the area of Big-data / Data Engineering.

Desired Knowledge & Experience:
Data Engineer - Big Data Developer

• Spark: Spark 3.x, RDD/DataFrames/SQL, Batch/Structured Streaming
• Knowing Spark internals: Catalyst/Tungsten/Photon
• Databricks: Workflows, SQL Warehouses/Endpoints, DLT, Pipelines, Unity, Autoloader
• IDE: IntelliJ/Pycharm, Git, Azure Devops, Github Copilot
• Test: pytest, Great Expectations
• CI/CD Yaml Azure Pipelines, Continuous Delivery, Acceptance Testing
• Big Data Design: Lakehouse/Medallion Architecture, Parquet/Delta, Partitioning, Distribution, Data Skew, Compaction
• Languages: Python/Functional Programming (FP)
• SQL: TSQL/Spark SQL/HiveQL
• Storage: Data Lake and Big Data Storage Design

Additionally it is helpful to know basics of:

• Data Pipelines: ADF/Synapse Pipelines/Oozie/Airflow
• Languages: Scala, Java
• NoSQL: Cosmos, Mongo, Cassandra
• Cubes: SSAS (ROLAP, HOLAP, MOLAP), AAS, Tabular Model
• SQL Server: TSQL, Stored Procedures
• Hadoop: HDInsight/MapReduce/HDFS/YARN/Oozie/Hive/HBase/Ambari/Ranger/Atlas/Kafka
• Data Catalog: Azure Purview, Apache Atlas, Informatica
• Big Data Architect
• Expert: in technologies, languages and methodologies mentioned in Data Engineer - Big Data Developer
• Mentor: mentors/educates Developers in technologies, languages and methodologies mentioned in Data Engineer - Big Data Developer
• Architecture Styles: Lakehouse, Lambda, Kappa, Delta, Data Lake, Data Mesh, Data Fabric, Data Warehouses (e.g. Data Vault)
• Application Architecture: Microservices, NoSql, Kubernetes, Cloud-native
• Experience: Many years of experience with all kinds of technology in the evolution of data platforms (Data Warehouse -> Hadoop -> Big Data -> Cloud -> Data Mesh)
• Certification: Architect certification (e.g. Siemens Certified Software Architect or iSAQB CPSA)

Required Soft-skills & Other Capabilities:

• Excellent communication skills, in order to explain your work to people who don't understand the mechanics behind data analysis
• Great attention to detail and the ability to solve complex business problems
• Drive and the resilience to try new ideas, if the first ones don't work
• Good planning and organizational skills
• Collaborative approach to sharing ideas and finding solutions
• Ability to work independently and also in a global team environment.