We’ve worked out what happened today. At 13:55 (all times UK time) one of our developers inadvertently deleted the table in the system that has the names of each customer’s system, which is referenced by the login process. People who were logged on were unaffected, but new users trying to log on got an error message. The table is held on multiple servers but as they all replicate in real time, the table was instantly deleted off all the servers.
Once we had identified the cause of the problem we could reload the deleted table on one server, check the system for data integrity, switch all the new logins to that one server and test.
At 14:32 this process was complete and all users could start logging in again.
At 15:40 the other servers were synchronised and we started spreading the logins across the multiple boxes.
During this time we had about 20 calls and emails from users who could not log in, although of course more could have been affected.
We are going to change the permissions on the database to stop developers dropping tables, and the developer involved has been sentenced to cleaning the toilets for a week!
Our apologies again to all the users who were affected.
