Sr. Software Engineer - Operational readiness (Scale & Performance) (Hybrid)

As a Senior Software Engineer for the Operational Readiness team, you will play a critical role in enhancing the scalability, performance, and reliability of HashiCorp's cloud products. With at least 5 years of experience in site reliability engineering or a related field, you will lead efforts to identify performance bottlenecks, address, and mitigate operational challenges before they impact our customers. Your expertise in load testing, performance analysis, and system hardening will ensure that our services meet the highest standards of operational excellence.

Having a holistic view of enterprise and cloud systems, you will play a pivotal role in enhancing our operational resilience and maintaining the reliability of our enterprise and cloud-based products. With a focus on overall Quality you will be at the forefront of ensuring high availability and performance across HashiCorp’s offerings.

You will provide expert execution of the test plans, defining system wide strategies for product load and performance testing. You will be working on a wide variety of tools and exploring new avenues to ensure all the products meet the essential Operational readiness criteria. 

Utilize top-notch troubleshooting techniques like simulating the system with Chaos to identify, organize, and advocate for novel solutions to remediate customer impact on complex interconnected systems. 

 

• Implement best practices for system reliability, including proactive identification of potential failure points and the development of automated mitigations
• Design and execute comprehensive load testing strategies to identify performance bottlenecks and scalability limits across our cloud products
• Implement best practices and technologies to improve system resilience, ensuring high availability and fault tolerance.
• Work closely with engineering and product teams to integrate operational readiness into the development lifecycle, enhancing product stability and user satisfaction.
• Build and refine tools and frameworks for automated testing, environment simulation, and incident reproduction, reducing manual effort and increasing test coverage.
• Conduct in-depth analysis of testing results, documenting findings and making actionable recommendations for system enhancements.
• Drive Systemic Improvements to the products by introducing Chaos Testing and partnering with product development teams. 
• Share your knowledge and expertise with team members, fostering a culture of learning and continuous improvement.
• Develop and implement disaster recovery and backup strategies to ensure data integrity and system resilience.

 

• 5+ years of experience in performance engineering  systems engineering, reliability engineering or non functional testing roles with a focus on performance testing, load testing or system scalability.
• Strong programming skills in Python / Golang and exposure to scripting languages like javascript or shell script 
• Experience with version control systems such as Git. 
• Strong experience with performance testing tools like K6, Artillery, Vegeta, Locust etc and deriving key performance metrics for a product
• Proven track record of leading successful load testing and performance optimization initiatives in cloud and on-prem environments.
• Experience in creating and managing test environments for automated testing.
• Experience in creating CI/CD pipelines and maintaining quality gates for system testing.
• Understanding of monitoring and observability tools such as Datadog or Prometheus to develop dashboards indicating metrics that accurately reflect system performance and load break points and regressions.
• Exposure to cloud technologies ( AWS, Azure, Or GCP) and container technologies like Nomad or Kubernetes and/Or working in a Hybrid cloud environment.
• Effective communication and collaboration skills, capable of working with cross-functional teams and articulating technical concepts to diverse audiences.

Nice to Have:

• Experience with infrastructure as code ( Terraform ) or any HashiCorp products is a plus.
• Experience with Javascript development / using any test framework based on Java script is a plus.
• Experience in driving systemic improvements through Chaos engineering is a plus. #LI-Hybrid