soc.ialis.me downtime explaination Show more
I originally planned to cancel four of my servers. Since the ISP didn't do black friday and is short on stock, I decided to cancel the cancellation of two of them.
But when they cancelled the cancellation, they didn't cancel the network cancellation.
At around 1AM UTC, the internal network was cut. Proxmox (virtual machines hypervisor), when loosing all contact with theses two servers (on a cluster of three), just decided to bring down all the VMs and reboot every server. And then it became stuck because there was no LAN to contact the other servers and re-establish a cluster.