Everything is running smoothly again. We went offline at 16:41 CEST / 10:41AM EDT and were back online 18 minutes later at 17:00 CEST / 11:00AM EDT. While we were offline no-one was able to log in or use our service. We were also not accepting incoming email for a few minutes. Incoming email that was sent in those minutes typically gets re-sent within minutes and should all have been delivered by now. Performance for IMAP and webmail was degraded in the initial minutes after coming back online and is now back to regular levels.
Again, we are sorry for this disruption. We took precautions to eliminate exactly these kinds of issues but they were apparently not enough. We will revisit our procedures to make sure this does not happen again.
What happened was the following. We were deploying security updates today on multiple servers as part of our regular maintenance process to keep our systems up-to-date. One of the updates had an unintended side-effect on the network level on one critical server only. This unfortunately caused all vault servers to get stuck and required reboots to resolve.
Posted Aug 02, 2022 - 15:30 UTC
Monitoring
And we're back online! The upgrade issue was fixed but we had to restart vault servers. All servers are back online and users are currently logging back in. Performance will be degraded while everyone is coming back in at the same time. This normally takes between 5 to 10 minutes to settle so please expect some slowness during that time. We are monitoring the situation and expect to close the incident in a few minutes.
Posted Aug 02, 2022 - 15:07 UTC
Identified
All services are currently down due to an upgrade gone wrong. Our engineers are working to fix the issue. We expect to be back in a few minutes as services are restarted. We are very sorry for this unplanned interruption of service.
Posted Aug 02, 2022 - 14:49 UTC
This incident affected: Webmail, Mail access (IMAP), and Mail delivery (SMTP).