Without a prod-shaped staging environment, every deploy meets production conditions for the first time in production.
Deep-dive
- A high-throughput factory-floor pipeline (equipment -> integration server -> database, with strict transactional consistency) broke in production on more or less every deploy.
- Code worked on local and on dev, then failed in prod each time.
- Root cause: local and dev ran on smaller, cleaner data and looser config, never shaped like prod, and there was no third environment in between that matched prod.
- So things local and dev hid only showed up in prod: config that differed, data volumes the dev set never reached, and timing that only appeared under real load.
- Each deploy was the first time the code met production-shaped conditions, and the first place it met them was production.
- Fix: add a staging environment with production-like config and data so a deploy is validated against prod-shaped conditions before it touches prod.
Code
# "works on my machine" -> build once, promote the SAME artifact build -> [dev] -> [staging (prod-like)] -> [prod] # same image, config via env # add a staging env that mirrors prod; never test big changes on prod
Say it (English)
- ·We had a factory-floor pipeline that worked on local and dev, then broke in prod on basically every deploy.
- ·Local and dev ran on smaller, cleaner data and looser config, so they never looked like prod.
- ·There was no environment in between that matched prod, so each deploy was the first time the code hit real config, real data volumes, and real load timing.
- ·And the first place it hit those conditions was production, which is exactly where the incidents were.
- ·The fix was a staging environment shaped like prod so we could catch this before it touched prod.
Push it further
- Stand up a staging environment with production-like config and a prod-scale dataset, and make every deploy pass through it first.
- Diff config across environments so dev's looser settings can't silently mask prod behavior.
- Load- and volume-test against prod-shaped data before release, since the failures only appeared under real load and at real data volumes.
- Gate prod deploys on a green staging run so production is never the first place new code meets prod conditions.