Tonight we had a series of about 12 power cuts within 30 minutes or so. Each was short (some only a few seconds) but they lead to a time consuming series of events.
First, I went downstairs and turned off all the servers to save them from the messy power supply (yes I know we need a UPS but …)
When the power seemed to be back on to stay I went to check things. The internet connection was unreliable, so I power-cycled the router. That improved things only a little. So I started looking at the IPCOP interface (our firewall), nothing obvious wrong but I noticed that I had now downloaded and installed the latest update. So I did that.
That install mentioned a command to type at the command line, so I went to SSH in, but it would not accept the password. Strange. Checked settings, tried again, still no good.
So I went to the IPCOP box in the rack cabinet and plugged in a keyboard and monitor (1st time since we moved here just over a year ago). Still would not work. I wondered about re-installing and that is when I noticed that the IPCOP machine was in the rack upside down. Oop’s. Still it has had a best uptime of 179 days (another powercut stopped it at that point) so it can’t be too bad.
I then booted as a single user, changed the password and got access again at which time everything was hunk dory.
The server is now the right way up