[Resolved] Maintenance and minor service disruption

In order to resolve an issue with one of our storage units we will upgrade its software on Friday 2023-09-29 at 09:00 CEST (03:00 AM EDT, 00:00 PDT).

This process will involve restarting the server twice within an hour, which will briefly affect some of our customers.

If you are unable to access your email during this time frame, please try again after a few minutes.

We apologize for any inconvenience caused by this maintenance.

Update 2023-10-05 12:00 CEST (06:00 AM EDT, 03:00 AM PDT)

On 2023-10-06 at 09:00 CEST (03:00 AM EDT, 00:00 AM PDT) we will perform follow-up work on the storage unit which involves phasing out disks and running integrity tests. This should not impact services, but there is a possibility it could affect performance while work is ongoing.

Update 2023-09-29 12:20 CEST (06:20 AM EDT, 03:20 AM PDT)

Email services are now being restored for all customers.

Update 2023-09-29 11:30 CEST (05:30 AM EDT, 02:30 AM PDT)

Work is progressing to restore services, and we hope to operate normally within around 30 minutes.

Update 2023-09-29 11:00 CEST (05:00 AM EDT, 02:00 AM PDT)

We are still working on hardware replacements, and in the meantime considering options to restore services for the users not directly affected by that storage unit.

Update 2023-09-29 10:15 CEST (04:15 AM EDT, 01:15 AM PDT)

During the rebuild process of the storage pool the server unfortunately encountered hardware errors. This requires us to replace hardware parts in the server, which will take around 30 minutes. We apologize for the disruption while our team is working to resolve the problem.

Update 2023-09-29 10:00 CEST (04:00 AM EDT, 01:00 AM PDT)

The software upgrade and server restarts have completed, and the storage pool is currently rebuilding. We expect operations to normalize shortly.

Update 2023-09-29 09:30 CEST (03:30 AM EDT, 00:30 AM PDT)

The upgrade is progressing but requires more server restarts than anticipated, which may prolong the process and could impact the overall performance of the service. Our system administrators are performing the upgrade as efficiently and safely as possible, and we expect all systems to return to normal operation within 20-30 minutes.