Riley Tan Senior Site Reliability Engineer
San Francisco, CA • riley.tan@gmail.com • +1 415-555-0142
Profile Summary
- Senior Site Reliability Engineer with 8 years keeping checkout, order routing, and shopper dispatch services reliable for a grocery delivery marketplace, specializing in SLOs, incident command, and holiday capacity planning.
- Hands-on with languages (Go, Python), observability (Prometheus, Grafana, OpenTelemetry), orchestration (Kubernetes), infrastructure as code (Terraform), cloud (AWS), and release and chaos tooling (Argo Rollouts, Gremlin).
- Deep expertise in error budget policies, burn-rate alerting, progressive delivery, and load shedding, using blameless postmortems and game days to fix the causes behind repeat incidents.
- Works with product engineering, data platform, and customer support teams in a 24/7 on-call model, joining launch reviews, weekly SLO reviews, and incident retros.
- Raises the bar for on-call readiness and postmortem quality, and runs the monthly reliability office hours product teams book for SLO and alert reviews.
Technical Skills
- Reliability Practices:
- SLOs and SLIs, Error Budgets, Burn-Rate Alerting, Incident Command, Blameless Postmortems, Production Readiness Reviews
- Monitoring & Observability:
- Prometheus, Grafana, OpenTelemetry, Thanos, Loki
- Incident & On-Call Tools:
- PagerDuty, incident.io, Statuspage, Slack, Jira
- Kubernetes & Cloud:
- Kubernetes, Amazon EKS, Helm, Istio, AWS
- Infrastructure as Code & GitOps:
- Terraform, Argo CD, GitHub Actions, Ansible
- Release & Resilience Tools:
- Argo Rollouts, LaunchDarkly, Gremlin, AWS Fault Injection Service, k6
- Languages & Scripting:
- Go, Python, Ruby, Bash, PromQL
Education
Work Experience
- Defined SLIs and SLOs with product owners for 14 checkout and order services, with Prometheus burn-rate alerts and an error budget policy that pauses launches, lifting checkout availability from 99.82% to 99.95%.
- Led 31 SEV1 and SEV2 incidents as incident commander on a PagerDuty rotation shared by 6 teams, cutting median time to mitigate from 42 minutes to 12 minutes with rollback-first runbooks.
- Wrote or reviewed the blameless postmortem for every SEV1 and SEV2 and tracked each action item in Jira with an owner and a due date, raising on-time completion from 41% to 86% in 2 quarters.
- Instrumented 40+ Go and Ruby services with OpenTelemetry traces and built Grafana dashboards on the 4 golden signals, then deleted or retuned noisy alerts to bring pages from 34 to 9 a week.
- Measured toil at 46% of the SRE team’s on-call time, then automated certificate renewals, node drains and cache warmups with Go Kubernetes operators, bringing it down to 19%.
- Planned capacity for the Thanksgiving peak from k6 load tests at 4x normal order volume and added load shedding to order routing, holding checkout p99 latency at 340 ms through the 2025 holiday week.
- Moved 55 services to canary releases with Argo Rollouts and automatic rollback on SLI checks, cutting the change failure rate from 7.1% to 1.8%.
- Ran 18 game days with Gremlin on zone loss and slow dependencies, plus a full region failover drill for checkout, and removed 7 single points of failure before they caused an outage.
- Moved 120 threshold alerts on the scooter and e-bike telemetry pipeline to SLO-based alerts in Grafana, cutting false pages by 63%.
- Created the production readiness review every new service had to finish before launch (dashboards, runbooks, a rollback plan) and ran it for 22 launches.
- Rebuilt Kafka consumer autoscaling on Kubernetes for the telemetry pipeline, cutting ingestion lag at the evening commute peak from 9 minutes to 40 seconds.