SELECT lock stopped factory

An unverified production SELECT took a library cache lock and stopped factory equipment

Deep-dive
  • Ran one unverified SELECT against production that scanned ~20,000 rows.
  • Never checked the execution plan, never bounded the row count, never thought about where it took locks.
  • The query took a library cache lock; the floor pipeline needs strict transactional consistency, so downstream equipment at an overseas factory stopped.
  • Spent 15-20 min blaming the integration server and upstream feed before checking myself.
  • Found the lock-holding session was mine and reported it.
  • There was no pre-production review step for large SELECTs at the time.
Code
-- never run an unvetted large SELECT on prod
EXPLAIN PLAN FOR SELECT ... ;                    -- review the plan first
SELECT ... FROM t WHERE ... AND ROWNUM <= 1000;  -- bound the rows
-- read-only != safe: a long scan can hold a library-cache lock
Say it (English)
  • ·I ran a single SELECT against production without checking it first, and equipment at an overseas factory stopped.
  • ·It scanned around 20,000 rows and ended up taking a library cache lock, and the pipeline needs strict transactional consistency, so the floor downstream halted.
  • ·For about 15 to 20 minutes I looked everywhere but myself, the integration server, the upstream feed.
  • ·Then I found the session holding the lock was mine, and I reported it.
  • ·The fix was a rule: any large SELECT gets its plan, row bound, and locking checked before it hits production.
Push it further
  • Check the execution plan of any large query before running it against production.
  • Bound the number of rows a query touches instead of scanning unbounded.
  • Reason about where a statement takes locks before running it, especially against a consistency-sensitive pipeline.
  • Add a required pre-production review step for large SELECTs, since none existed.