Internal SSL certificate expiration
Recently, it happened that our internally used SSL certificates (which were generated by ./scripts/generate-ssl.sh
a year ago) expired.
This caused the nodes to refuse to talk to each other, eventually resulting in the log server not accepting any new messages.
The issue could be simply resolved by generating new SSL files and recreating the containers with ./scripts/generate-ssl.sh && docker compose down && docker compose up -d
.
We should add preventative measures that this doesn't easily happen again.
For example, we could automate recreation of SSL files every few months. Or, we could extend the time to live for the generated certificates.