Bliptown went down for 25 minutes. Luckily this happened at
approximately 10:30am so I was able to deal with it. The immediate
cause was found to be an empty SSL certificate chain and key. Any
connection to https://blip.town would error. Manually trying to curl
the certificate from Porkbun resulted in a timeout, which made me
think the problem with their API. I downloaded the certificate
directly from their site and installed it on beastie. Bliptown was
back online.
There was still a large backlog of outbound mail stuck inflight. After succeeding fetching the certificate locally, I remembered having recently changed the host-level firewall. The firewall, which had previously allowed all outbound traffic, now had a single rule, which only allowed ICMP traffic out, nothing else. As soon as I fixed this the backlog of emails was flushed, and calling the script to fetch the certificate succeeded.
This lead to a concentrated rewrite of daily.local that is now safer and failure tolerant.
Also implemented restoring snapshots, which was quite simple given the
amount of scaffolding I'd already done. In the snapshots page,
clicking on a snapshot and then "Restore" will restore all files of
that commit to /snapshots/<HASH>. This is almost like restoring a
version of your site available under the /snapshots path, except
that of course absolute links will still take you out.