Job Description
Job Summary:
The client is looking for an experienced AWS Engineer to join our IT Department to manage and optimize the Continuous Integration/Continuous Deployment (CI/CD) pipeline for machine learning (ML) solutions. The successful candidate will work closely with the Model Development team (ML Engineers), data scientists, and DevOps teams to ensure smooth deployment, scaling, and monitoring of ML models on AWS. This role requires a deep understanding of AWS cloud services, DevOps practices, and machine learning infrastructure.
Key Responsibilities:1. CI/CD Pipeline Management & Automation:- Design, implement, and maintain robust CI/CD pipelines for deploying machine learning models and solutions.
- Automate and streamline deployment processes using AWS services such as CodePipeline, CodeBuild, CodeDeploy, and CodeCommit.
- Ensure seamless integration of model training, testing, and deployment stages within the CI/CD pipeline.
- Set up and manage infrastructure as code (IaC) using tools like AWS CloudFormation or Terraform for creating scalable and reliable environments for ML applications.
- Automate deployment, scaling, and monitoring of machine learning models in AWS environments using AWS Lambda, ECS, EKS, and SageMaker.
2. AWS Cloud Services Management & Security:- Manage and configure AWS cloud services such as EC2, S3, SageMaker, Lambda, and others to support machine learning pipelines and production environments.
- Use AWS SageMaker for managing the ML lifecycle, including data preparation, training, tuning, and model deployment.
- Set up automated workflows for model retraining and versioning based on new data inputs and performance metrics.
- Ensure compliance with industry standards and internal policies regarding data privacy, security, and governance for machine learning solutions.
- Implement best practices in DevOps, including version control, code quality checks, and deployment automation using AWS services.
- Continuously improve infrastructure by staying up-to-date with new AWS features, best practices, and emerging technologies.
3. Monitoring & Optimization:- Monitor the performance of deployed ML models and pipelines using AWS CloudWatch, CloudTrail, and other monitoring tools.
- Implement automated testing, validation, and monitoring processes to ensure models perform as expected in production environments.
- Optimize costs and performance by automating resource scaling, ensuring high availability, and improving pipeline efficiency.
4. Collaboration & Support:- Collaborate with data scientists, machine learning engineers, and DevOps teams to integrate ML models into production systems.
- Provide support and troubleshooting expertise for pipeline issues, including model failures, deployment bottlenecks, and scaling problems.
- Work closely with security teams to implement best practices for security and compliance, ensuring that data and models are protected within AWS.
Key Skills & Qualifications:- Education:Bachelor’s degree in Computer Science, Information Technology, or a related field.
- Experience:5+ years of experience as an AWS Engineer, DevOps Engineer, or Cloud Engineer, with a focus on CI/CD pipelines and machine learning solutions.
- Strong expertise in AWS cloud services (S3, EC2, SageMaker, Lambda, CodePipeline, etc.).
- Experience with CI/CD tools like AWS CodePipeline, GitLab CI, or similar platforms.
- Proficiency with containerization tools such as Docker and Kubernetes for managing microservices architecture.
- Knowledge of infrastructure as code (IaC) tools such as AWS CloudFormation, Terraform, etc.
- Familiarity with machine learning frameworks (e.g., TensorFlow, PyTorch) and deployment in production environments.
- Experience in setting up and managing CI/CD pipelines for machine learning or data science solutions.
- Familiarity with version control systems like Git and deployment automation practices.
- Strong knowledge of monitoring and logging tools (e.g., AWS CloudWatch) for real-time performance tracking.
- Ability to work collaboratively with cross-functional teams, including data scientists, ML engineers, and DevOps teams.
- Strong verbal and written communication skills for documentation and knowledge sharing.
