No staging environment

Without a prod-shaped staging environment, every deploy meets production conditions for the first time in production.

Deep-dive
  • A high-throughput factory-floor pipeline (equipment -> integration server -> database, with strict transactional consistency) broke in production on more or less every deploy.
  • Code worked on local and on dev, then failed in prod each time.
  • Root cause: local and dev ran on smaller, cleaner data and looser config, never shaped like prod, and there was no third environment in between that matched prod.
  • So things local and dev hid only showed up in prod: config that differed, data volumes the dev set never reached, and timing that only appeared under real load.
  • Each deploy was the first time the code met production-shaped conditions, and the first place it met them was production.
  • Fix: add a staging environment with production-like config and data so a deploy is validated against prod-shaped conditions before it touches prod.
Code
# "works on my machine" -> build once, promote the SAME artifact
build -> [dev] -> [staging (prod-like)] -> [prod]   # same image, config via env
# add a staging env that mirrors prod; never test big changes on prod
Say it (English)
  • ·We had a factory-floor pipeline that worked on local and dev, then broke in prod on basically every deploy.
  • ·Local and dev ran on smaller, cleaner data and looser config, so they never looked like prod.
  • ·There was no environment in between that matched prod, so each deploy was the first time the code hit real config, real data volumes, and real load timing.
  • ·And the first place it hit those conditions was production, which is exactly where the incidents were.
  • ·The fix was a staging environment shaped like prod so we could catch this before it touched prod.
Push it further
  • Stand up a staging environment with production-like config and a prod-scale dataset, and make every deploy pass through it first.
  • Diff config across environments so dev's looser settings can't silently mask prod behavior.
  • Load- and volume-test against prod-shaped data before release, since the failures only appeared under real load and at real data volumes.
  • Gate prod deploys on a green staging run so production is never the first place new code meets prod conditions.