An unverified production SELECT took a library cache lock and stopped factory equipment
Deep-dive
- Ran one unverified SELECT against production that scanned ~20,000 rows.
- Never checked the execution plan, never bounded the row count, never thought about where it took locks.
- The query took a library cache lock; the floor pipeline needs strict transactional consistency, so downstream equipment at an overseas factory stopped.
- Spent 15-20 min blaming the integration server and upstream feed before checking myself.
- Found the lock-holding session was mine and reported it.
- There was no pre-production review step for large SELECTs at the time.
Code
-- never run an unvetted large SELECT on prod EXPLAIN PLAN FOR SELECT ... ; -- review the plan first SELECT ... FROM t WHERE ... AND ROWNUM <= 1000; -- bound the rows -- read-only != safe: a long scan can hold a library-cache lock
Say it (English)
- ·I ran a single SELECT against production without checking it first, and equipment at an overseas factory stopped.
- ·It scanned around 20,000 rows and ended up taking a library cache lock, and the pipeline needs strict transactional consistency, so the floor downstream halted.
- ·For about 15 to 20 minutes I looked everywhere but myself, the integration server, the upstream feed.
- ·Then I found the session holding the lock was mine, and I reported it.
- ·The fix was a rule: any large SELECT gets its plan, row bound, and locking checked before it hits production.
Push it further
- Check the execution plan of any large query before running it against production.
- Bound the number of rows a query touches instead of scanning unbounded.
- Reason about where a statement takes locks before running it, especially against a consistency-sensitive pipeline.
- Add a required pre-production review step for large SELECTs, since none existed.