SSL Certificate Renewal Runbook

Quick Reference:

Duration: 5-15 minutes
Impact: Minimal (usually zero downtime)
Requires: Sudo access to load balancer or cert-manager admin
Severity: HIGH (expired certs = full outage)

Prerequisites

What You Need

SSH access to certificate servers or kubectl access to cluster
Let’s Encrypt API credentials (if manual renewal)
Backup certificate location documented
30+ days lead time before expiry (not 5 minutes!)

Check Current Status

# View certificate expiry
openssl s_client -connect example.com:443 -showcerts | grep -A5 "Verify return code"

# Or via Kubernetes
kubectl get certificate -n ingress-nginx
kubectl describe certificate prod-cert -n ingress-nginx

# Check expiry date specifically
openssl x509 -in /etc/ssl/certs/server.crt -noout -enddate

Automatic Renewal (Preferred)

Using cert-manager (Kubernetes)

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: prod-cert
  namespace: ingress-nginx
spec:
  secretName: prod-tls-secret
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
  - example.com
  - "*.example.com"

Cert-manager automatically:

✅ Renews 30 days before expiry
✅ Handles DNS validation
✅ Stores in Kubernetes Secret
✅ Reloads ingress automatically

Check renewal:

kubectl get events -n ingress-nginx | grep Certificate
kubectl logs -n cert-manager deploy/cert-manager | grep prod-cert

Using Certbot (Traditional)

# Automatic renewal via cron (runs daily)
0 2 * * * /usr/bin/certbot renew --quiet --no-eff-email

# Manual renewal when needed
certbot renew --force-renewal

# Renew specific domain
certbot certonly --standalone -d example.com -d "*.example.com"

After renewal:

# Reload web server
sudo systemctl reload nginx
# OR
sudo systemctl reload apache2

# Verify new certificate
openssl s_client -connect example.com:443 | grep -A2 "subject="

Manual Renewal Process

If Automatic Renewal Failed

Step 1: Stop web server (if using standalone validation)

sudo systemctl stop nginx

Step 2: Request new certificate

sudo certbot certonly \
  --standalone \
  -d example.com \
  -d "*.example.com" \
  --agree-tos \
  -m [email protected]

Step 3: Verify new certificate location

ls -lah /etc/letsencrypt/live/example.com/
# Should show: cert.pem, chain.pem, fullchain.pem, privkey.pem

Step 4: Update web server config (if using non-LE certs)

# Update nginx/apache config to point to new cert path
sudo nano /etc/nginx/nginx.conf
# Point to: /etc/letsencrypt/live/example.com/fullchain.pem
# Point to: /etc/letsencrypt/live/example.com/privkey.pem

Step 5: Test and reload

# Test configuration syntax
sudo nginx -t

# Reload gracefully
sudo systemctl reload nginx

# Verify it works
curl -I https://example.com

Troubleshooting

Issue: Rate Limit Exceeded

Symptoms: Renewal fails with “too many certificates already issued”

Solution:

# Check how many certs issued
whois domain.com | grep -i "created"

# Wait before retrying (Let's Encrypt limits):
# - 50 certs per domain per week
# - 5 duplicate certs per week

# Use staging environment for testing
certbot certonly --staging -d example.com

# After 7 days, try again

Issue: DNS Validation Failed

Symptoms: ACME challenge fails, “DNS problem”

Solution:

# Check DNS propagation
nslookup _acme-challenge.example.com

# Wait for DNS to propagate (up to 48 hours)
# Then retry

# Use manual DNS validation if needed
certbot certonly --manual -d example.com
# Follow prompts to create DNS TXT records

Issue: Certificate Not Updating

Symptoms: Old certificate still showing after renewal

Solution:

# Clear web browser cache
# (Certificates often cached for HSTS preload)

# Check if nginx/apache reloaded
ps aux | grep -i nginx | grep -i reload
# If not running, restart:
sudo systemctl restart nginx

# Check certificate timestamp
openssl s_client -connect example.com:443 -showcerts | grep notBefore

# Force immediate renewal
sudo certbot renew --force-renewal

# Check for certbot errors
sudo certbot renew -v
tail -f /var/log/letsencrypt/letsencrypt.log

Issue: Multiple Domains/Subdomains

Symptoms: Need to renew certificate with additional domains

Solution:

# List current certificates
sudo certbot certificates

# Expand existing certificate
sudo certbot certonly \
  --expand \
  -d example.com \
  -d "*.example.com" \
  -d "api.example.com"

# Or request new with all domains
sudo certbot certonly \
  --standalone \
  -d example.com \
  -d "*.example.com" \
  -d "api.example.com" \
  -d "admin.example.com"

Pre-Expiry Checklist (30 days before)

Verify automatic renewal is working
```
certbot renew --dry-run
```
Check DNS settings are correct
```
nslookup example.com
dig example.com
```

Ensure backup certificate is up-to-date

sudo cp /etc/letsencrypt/live/example.com/fullchain.pem /backups/

Document load balancer certificate paths
```
grep -r "ssl_certificate" /etc/nginx/
```

Enable monitoring/alerting for certificate expiry

# Prometheus query
ssl_certificate_not_after_seconds - time() < 2592000  # 30 days

Post-Renewal Verification

Test the certificate:

# Check certificate is valid
openssl s_client -connect example.com:443 -showcerts

# Should show:
# - subject= /CN=example.com
# - issuer= /C=US/O=Let's Encrypt/CN=R3
# - Verify return code: 0 (ok)

# Check expiry date
openssl s_client -connect example.com:443 \
  | openssl x509 -noout -enddate

# Should be ~90 days from today

Monitor after renewal:

✅ Watch application logs for SSL errors
✅ Monitor SSL certificate metrics
✅ Check that HSTS headers are present
✅ Verify no browser warnings

Common Mistakes to Avoid

❌ Don’t

Wait until certificate expires (0 day deadline = panic)
Renew manually every 90 days (set up automation!)
Use self-signed certs in production
Forget to reload web server after renewal
Store private keys in version control
Mix certificate renewal timezones

✅ Do

Set renewal reminder for 30 days before expiry
Use cert-manager or certbot with automation
Store backup certificates in secure location
Test renewal in staging first
Monitor certificate expiry with alerts
Document certificate locations and procedures

Monitoring & Alerts

Prometheus Alert Rule

- alert: CertificateExpirySoon
  expr: ssl_certificate_not_after_seconds - time() < 2592000
  for: 1h
  labels:
    severity: warning
  annotations:
    summary: "Certificate expires in {{ $value | humanizeDuration }}"

- alert: CertificateExpiryImmediate  
  expr: ssl_certificate_not_after_seconds - time() < 86400
  labels:
    severity: critical
  annotations:
    summary: "Certificate expires in less than 24 hours!"

Kubernetes Cert-Manager Alerts

# Check cert-manager reconciliation
kubectl get events -n cert-manager | grep Certificate

# Check certificate status
kubectl get cert -n ingress-nginx -o wide

# Watch for warning events
kubectl get events -A --sort-by='.lastTimestamp' | grep Warning

References

Last Updated: 2025-08-15 Owner: Platform Team Reviewed By: Security Team Next Review: 2025-12-15