TLS cert silently expired

An unmonitored TLS certificate will silently expire and take production down with it.

Deep-dive
  • A TLS certificate secured the integration link between two manufacturing floor systems, with the upstream system pushing operational data to an integration server over it.
  • The cert had run for over a year with no certificate maintenance, then quietly hit its expiry date.
  • Once it expired, the TLS handshake failed and the data flow halted, causing a production outage on the manufacturing pipeline.
  • Root cause: there was no certificate expiry monitoring or renewal step, so it reached expiry silently in production.
  • Detection was reactive: the outage was only noticed after data stopped arriving, not before the cert lapsed.
Code
# the cert expired silently -> monitor expiry + automate renewal
echo | openssl s_client -connect host:443 2>/dev/null \
  | openssl x509 -noout -enddate          # notAfter=...
# alert if < 30 days left; renew via ACME/certbot, not by hand
Say it (English)
  • ·We had a TLS-secured link between two floor systems pushing operational data to an integration server.
  • ·The certificate had been running over a year and nobody was tracking its expiry.
  • ·When it expired the TLS handshake failed and data just stopped flowing.
  • ·We only found out after data stopped arriving, not before the cert lapsed.
  • ·The real gap was no monitoring or automated renewal on the certificate at all.
Push it further
  • Track every certificate's expiry and alert well ahead of the date so none reaches expiry silently.
  • Automate renewal where possible instead of relying on someone remembering.
  • Monitor the integration link itself so a broken handshake is caught proactively, not after data loss.