Senior Site Reliability Automation Engineer
Save this job and keep your search organized
Create a free account to save jobs, create alerts and return to this listing from your dashboard.
By continuing, you agree to our Terms & Privacy Policy.
Job Overview
Our Interactive division delivers online games to players worldwide. Many development and QA teams support that business: platform services teams and multiple game studios, each often running several projects in parallel. We are modernizing how all of them build, deploy, and run software, and we need the workloads that result to be reliable, observable, and recoverable.
You will join the central Cloud & DevOps Platform team as a Senior Site Reliability Engineer. The team owns the standards, architecture, implementation, and ongoing improvement of our CI/CD, infrastructure, and Infrastructure as Code (IaC), and the centralized foundation that promotes software to stage and production environments across the globe. Those environments span AWS, other providers, and bare metal. Your focus is the reliability of the workloads running on that foundation and effective response when something goes wrong.
Our current infrastructure is largely static virtual machines monitored with Zabbix, and that monitoring will remain in place for some time. We are moving to a dynamic, container-based architecture, and we need to build out a Prometheus and Grafana solution to support it. You will work with both: keeping Zabbix-based monitoring effective for the existing estate while helping build the new observability stack.
This is an individual contributor role.
Reliability of Running Workloads
- Keep production and stage workloads healthy across a multi-account, multi-region AWS environment and across third-party hosted and bare-metal environments.
- Operate and troubleshoot Kubernetes workloads on EKS and on provider-hosted and bare-metal clusters managed centrally with Rancher.
- Operate AWS-only supporting workloads (automation, observability, internal tooling) running on ECS with Fargate.
- Support VM-based servers at third-party hosting providers that are reachable over SSH only, as they evolve toward container clusters.
- Define and track service level indicators and objectives (SLIs/SLOs) and error budgets for key services, working with the teams that own them.
- Perform capacity planning, performance analysis, and resilience reviews, including failure-mode and disaster recovery considerations for multi-region designs.
Incident Response and On-Call
- Participate in an on-call rotation and respond to production incidents, working from detection through mitigation and recovery.
- Diagnose problems across the stack: application, container, node, network, cloud service, and CI/CD or infrastructure change.
- Write blameless post-incident reviews and drive the resulting corrective actions to completion.
- Build and maintain runbooks, and use them to cut time to resolution.
- Follow and help refine standard steps for security-related incidents, in coordination with the security function.
Observability
- Build, maintain, and tune monitoring for the existing static VM estate in Zabbix, including creating monitors, triggers, and alerts and working with Zabbix data and configuration.
- Help build out a Prometheus and Grafana solution to monitor the new dynamic, container-based architecture, alongside AWS CloudWatch.
- Build and tune dashboards, alerting, and logging across both the existing and new environments.
- Design alerts around symptoms and SLOs rather than noise, and continuously reduce alert fatigue.
- Provide the visibility that dev and QA teams need to understand the health of their own workloads.
Automation and Toil Reduction
- Automate routine operational tasks and common issue resolution with Python, Bash, and similar scripting.
- Contribute to Terraform/OpenTofu and Ansible code for reliability-related infrastructure, following the team's IaC standards and review process.
- Work with the CI/CD and platform engineers so that reliability requirements are built into pipelines, golden images, and templates rather than added afterward.
- Help keep CI/CD tooling (Jenkins, Nexus, and related services) healthy and measurable as a dependency of every release.
Security (a first-class concern in everything above)
- Apply least-privilege access, patching, vulnerability response, and secrets hygiene to the systems y