Incidents

Real things that broke — what happened, the fix, and how to prevent it. Same format, plain, honest about the gaps the original posts left.

• Build inside the network
  If an artifact runs inside a closed network, build it there too — building outside turns every fix into a guess.
• Onboarding a locked network
  A device onboarding that's discovered fresh each time should be a written procedure, not improvisation.
• Silent protocol change broke flow
  When one side changes its protocol and nobody tells the middle, the integration silently stops.
• False success flag
  A status flag set on "request sent" lies; only the committed downstream result means done.
• TLS cert silently expired
  An unmonitored TLS certificate will silently expire and take production down with it.
• Hardcoded column index broke save
  Bind saves to a stable field key, not a positional column index.
• MDM lockdown at provisioning
  Lock devices down during provisioning, not after you catch someone on the wrong app.
• SELECT lock stopped factory
  An unverified production SELECT took a library cache lock and stopped factory equipment
• No staging environment
  Without a prod-shaped staging environment, every deploy meets production conditions for the first time in production.