Job Description
As a Data Architect, you are required to:Design & develop technical solutions which combine disparate information to create meaningful insights for business, using Big-data architectures
- Build and analyze large, structured and unstructured databases based on scalable cloud infrastructures
- Develop prototypes and proof of concepts using multiple data-sources and big-data technologies
- Process, manage, extract and cleanse data to apply Data Analytics in a meaningful way
- Design and develop scalable end-to-end data pipelines for batch and stream processing
- Regularly scan the Data Analytics landscape to stay up to date with latest technologies, techniques, tools and methods in this field
- Stay curious and enthusiastic about using related technologies to solve problems and enthuse others to see the benefit in business domain
Qualification:
- Bachelor's or Master's in Computer Science & Engineering, or equivalent. Professional Degree in Data Engineering / Analytics is desirable.
Experience level:
- Minimum 8 years in software development with at least 2 - 3 years hands-on experience in the area of Big-data / Data Engineering.
Desired Knowledge & Experience:Data Engineer - Big Data Developer
- Spark: Spark 3.x, RDD/DataFrames/SQL, Batch/Structured Streaming
- Knowing Spark internals: Catalyst/Tungsten/Photon
- Databricks: Workflows, SQL Warehouses/Endpoints, DLT, Pipelines, Unity, Autoloader
- IDE: IntelliJ/Pycharm, Git, Azure Devops, Github Copilot
- Test: pytest, Great Expectations
- CI/CD Yaml Azure Pipelines, Continuous Delivery, Acceptance Testing
- Big Data Design: Lakehouse/Medallion Architecture, Parquet/Delta, Partitioning, Distribution, Data Skew, Compaction
- Languages: Python/Functional Programming (FP)
- SQL: TSQL/Spark SQL/HiveQL
- Storage: Data Lake and Big Data Storage Design
Additionally it is helpful to know basics of:- Data Pipelines: ADF/Synapse Pipelines/Oozie/Airflow
- Languages: Scala, Java
- NoSQL: Cosmos, Mongo, Cassandra
- Cubes: SSAS (ROLAP, HOLAP, MOLAP), AAS, Tabular Model
- SQL Server: TSQL, Stored Procedures
- Hadoop: HDInsight/MapReduce/HDFS/YARN/Oozie/Hive/HBase/Ambari/Ranger/Atlas/Kafka
- Data Catalog: Azure Purview, Apache Atlas, Informatica
- Big Data Architect
- Expert: in technologies, languages and methodologies mentioned in Data Engineer - Big Data Developer
- Mentor: mentors/educates Developers in technologies, languages and methodologies mentioned in Data Engineer - Big Data Developer
- Architecture Styles: Lakehouse, Lambda, Kappa, Delta, Data Lake, Data Mesh, Data Fabric, Data Warehouses (e.g. Data Vault)
- Application Architecture: Microservices, NoSql, Kubernetes, Cloud-native
- Experience: Many years of experience with all kinds of technology in the evolution of data platforms (Data Warehouse -> Hadoop -> Big Data -> Cloud -> Data Mesh)
- Certification: Architect certification (e.g. Siemens Certified Software Architect or iSAQB CPSA)
Required Soft-skills & Other Capabilities:- Excellent communication skills, in order to explain your work to people who don't understand the mechanics behind data analysis
- Great attention to detail and the ability to solve complex business problems
- Drive and the resilience to try new ideas, if the first ones don't work
- Good planning and organizational skills
- Collaborative approach to sharing ideas and finding solutions
- Ability to work independently and also in a global team environment.
Job Details
-
Job Location
-
Bengaluru India
-
Company Industry
-
Other Business Support Services
-
Company Type
-
Employer (Private Sector)
-
Employment Type
-
Unspecified
-
Monthly Salary Range
-
Unspecified
-
Number of Vacancies
-
Unspecified
