DevOps/MLOps Staff engineer
Location: Bangalore, India Juniper Experience and Operations (JXO) organization is incubating a Technology group to transform CX(customer experience) and EX(employee experience). The charter of this group is to discover, evaluate and leverage technology to enhance and simplify the experience of stakeholders. Knowledge and data will be central to this journey of creating a proactive and predictive support experience. The use of automation, AI and other modern technology will enable reduction of time taken to resolve issues or perform tasks. This role is part of Juniper’s strategic future of support team and will involve design of automated solutions. The role requires enabling business transformation projects using technology, review of data to enable process, systems and tool re-engineering in customer support and services. In addition, the role requires to support the enhancement of self-service, automation, omnichannel strategy by seeking solutions and drivers to achieve seamless customer experiences and increased customer loyalty.
Primary Tech skills :
MLOPs in Databricks, AWS, Python, Docker, Kubernetes, Terraform, Ansible, Prometheus, Grafana, ELK.
Secondary Tech skills :
Snowflake, Data Engineering using PySpark
Responsibilities:
Integrate, deploy and maintain key Data Engineering and ML workflows using Databricks and AWS to ensure seamless data movement from raw source to final consumption. Manage the end to end lifecycle DevOps, DataOps and ModelOps. Involve in troubleshooting and resolving integration and data-related issues.
- Set up Databricks Repos to integrate with Git and sync notebooks and source code with Databricks workspaces. Use features in Databricks for Git integration and version control.
- Create and manage CICD pipeline for smooth deployment of codes and workflows between development to stage to production environment.
- Create and manage clusters by setting up access policies as needed by data engineers and data scientists.
- Use Databricks API’s to automate and create reports on Feature Store and MLflow usage and behaviour.
- Create and manage access controls for raw and feature tables stored in Delta tables.
- Enable integration between Databricks and AWS S3 buckets by setting up right IAM policies. Set-up optimised S3 Lifecycles for data retention and storage.
- Enable monitoring of Data Engineering job workflows and automate job failures notification and fallback solutions. Optimise and suggest best practices for usage of clusters and data storage.
- Enable ingestion of streaming data into delta live tables to support realtime time based Anomaly detection models.
- Use Databricks MLflow to track model development, registry and deployment and save model artifacts like code snapshots, model parameters, metrics and other metadata.
- Use Unity Catalog to manage data and model versioning, governance and deployment. Build model drift pipelines and monitor model performance over time and enable workflows to retrain drifted models.
- Build process to automate model movement from Dev to Stage to Production and enabling A/B testing of different model versions.
- Experience in containerization technologies and orchestration frameworks , understanding of microservices architecture and deployment patterns.
- Creating Kubernetes cluster and deploying application on top of it via package manager tools like helm.
- Implement security best practices and ensure compliance with industry standards.
- Collaborate with development teams to optimize application performance and scalability.
Juniper Business Use Only
- Stay updated with emerging technologies and industry trends, and evaluate their potential impact on our infrastructure and development processes.
Qualification and Desired Experiences:
- SRE experience in building CICD pipelines, managing and owning code movements between environments, creating access controls to objects for users based on environments and roles.
- 3+ years of relevant MLOps experience in Databricks : involving GIT integration,
- setting up Access Controls in Databricks and AWS S3,
- setting up CICD pipeline for code\model deployment between environments,
- manage/maintain/monitor data pipelines and ML model performance.
- Bachelors's degree in Computer Science, Statistics, Informatics, Information Systems or another quantitative field.
- Experience with big data tools: Spark, Kafka, Spark & Kafka Streaming, Python, and Snowflake
- Working knowledge of Databricks API to automate process.
- Experience with Databricks Unity Catalog.
- Experience with Databricks Feature Store and Delta Live Tables
- Experience with Databricks MLFlow, model registry and creating endpoints for inference for realtime\batch\streaming applications.
- Experience with AWS S3 for data storage, creating S3 Lifecycles for data storage and retention.
Personal Skills:
- Ability to collaborate cross-functionally in a fast-paced environment and build sound working relationships within all levels of the organization
- Ability to handle sensitive information with keen attention to detail and accuracy. Passion for data handling ethics.
- Ability to solve complex, technical problems with creative solutions while anticipating stakeholder needs and providing assistance to meet or exceed expectations
- Able to demonstrate perseverance and resilience to overcome obstacles when presented with a complex problem.
- Assist in combining large data sets and data analysis to create optimization strategies
- Comfortable with ambiguity and uncertainty of change when assessing needs for stakeholders
- Have effective time management skills which enable you to work successfully across functions in a dynamic and solution-oriented environment while meeting deadlines
- Self-motivated and innovative; confident when working independently, but an excellent team player with a growth-oriented personality
- Will be required to routinely or customarily troubleshoot items related to applications that require independent judgement, decision-making, and unique approaches
Juniper is an Equal Opportunity workplace and Affirmative Action employer. We do not discriminate in employment decisions on the basis of race, color, religion, gender (including pregnancy), national origin, political affiliation, sexual orientation, gender identity or expression, marital status, disability, genetic information, age, veteran status, or any other applicable legally protected characteristic. All employment decisions are made on the basis of individual qualifications, merit, and business need.
We will ensure that individuals with disabilities are provided reasonable accommodation to participate in the job application or interview process, to perform essential job functions, and to receive other benefits and privileges of employment. Please contact us to request accommodation.
