Responsibilities
Serve as the primary dashboard monitor during your shift — continuously watch the GTO Operational Dashboard in Datadog, detect anomalies by correlating signals across APM, logs, metrics, synthetic tests, and Real User Monitoring, and determine whether alerts warrant an incident ticket or can be resolved through immediate investigation.
Triage and investigate production incidents — create incident tickets in JIRA Service Management, perform initial technical investigation using Datadog (traces, logs, infrastructure and application metrics), determine blast radius and likely root cause domain, and route to the correct team (Product SRE, Infrastructure SRE, or Engineering) using the smart routing model.
Own lower-severity incidents end-to-end from detection through resolution — diagnose, execute runbook procedures, and resolve without escalation where possible. Escalate promptly when an incident is unresolved within defined thresholds or requires a code-level fix.
Support the TSO Lead during major incidents as the technical right hand in the war room — surface real-time data (error rates, impact scope, deployment history, related alerts), maintain the incident ticket with live timeline entries and linked evidence, and execute mitigation actions as directed.
Draft incident communications under TSO Lead direction, including internal Slack updates, stakeholder notifications, and customer-facing status page updates (status.xsolla.com). Support clear, timely communication throughout the incident lifecycle.
During non-incident periods, analyze incident trends, recurring issues, and production bugs — compile data from Datadog, JIRA, and Slack, identify patterns, and contribute findings to regular reports for product and engineering teams.
Compile incident timelines and draft initial PIR documents for Post-Incident Review preparation. Track PIR action items post-session and flag overdue items to the TSO Lead.
Build and maintain operational automation (alert enrichment scripts, incident templates, Slack workflows, dashboard widgets) and contribute to runbook development — documenting new resolution procedures so they can be repeated by any Operations Engineer on any shift.
Conduct structured shift handoffs covering active incidents, at-risk services, upcoming deployments, and follow-up items. Participate in knowledge transfer sessions with SREs to continuously expand independent resolution capability.
Cover for the TSO Lead during vacations, absences, or emergencies — including severity classification, escalation decisions, stakeholder communications, and basic Incident Commander functions.
- Publish health reports of critical apps periodically.
- Qualifications:
- Previous experience working at a gaming company is required — you understand the pace, player expectations, live operations dynamics, and the operational demands of the gaming industry.
4+ years of experience in SRE, DevOps, production operations, NOC, or technical operations in a high-availability environment.
Strong troubleshooting and investigation skills — ability to take an alert or user-reported symptom and methodically trace it through the stack: application logs, APM traces, infrastructure metrics, database queries, and network paths.
Hands-on experience with Datadog (or equivalent observability platform: Grafana, Splunk, New Relic, Elastic) — navigating APM, building log queries, reading infrastructure dashboards, interpreting SLO burn rates, and configuring monitors and alerts.
Proficiency in at least one scripting language: Python, Go, or Bash. You will write automation scripts, build operational tooling, and work with APIs.
Clear written and verbal communication skills in English — ability to write incident tickets, investigation notes, Slack updates, shift handoff reports, status page communications, and PIR drafts that are clear, concise, and useful to both technical and non-technical audiences.
Working knowledge of Kubernetes and cloud infrastructure (GCP preferred, AWS/Azure acceptable) — understanding of pods, deployments, services, ingress, node health, and how to investigate Kubernetes-related production issues.
Understanding of SLOs, error budgets, and burn-rate alerting — knowing what a multi-window burn-rate alert means, how error budgets deplete, and how SLO breaches translate into incident severity.
Experience with incident management tooling: JIRA or JIRA Service Management, PagerDuty or OpsGenie, Slack, and Confluence.
Experience with or strong interest in AI/ML-assisted operations: anomaly detection, alert correlation, predictive monitoring, or automated remediation.
- Comfort with 24x7 shift-based operations as part of a follow-the-sun model with handoff overlaps. Weekend on-call (rotating) is required.
- Nice to Have:
- Familiarity with Datadog Service Catalog, synthetic monitoring, and RUM (Real User Monitoring).
Experience with distributed systems debugging: tracing failures across microservices, understanding cascading failures, and reading distributed traces end-to-end.
Exposure to database operations (MySQL, PostgreSQL, Redis, Kafka) at a level sufficient to investigate connection pool exhaustion, replication lag, slow queries, or queue backlogs during incidents.
Familiarity with CI/CD pipelines and deployment tooling (GitLab CI, ArgoCD, Helm) — enough to correlate recent deployments with production issues and identify rollback targets.
JIRA Service Management administration experience: workflows, automation rules, SLA timers, and queues.
ITIL Foundation certification is a plus but not required — practical experience matters more.
