[Monitoring] Upgrade of authentication system

On Tuesday, November 23 at 09:00 CET (3 AM EST, 12 AM PST) we will be performing an upgrade of the Runbox authentication system.

This upgrade involves moving authentication to a separate and dedicated infrastructure, which will improve performance while enabling event logging and improving access control available in the Runbox user interface.

We don’t expect any noticeable disruption during this upgrade, but configuration changes to two-factor authentication settings will be disabled while work is ongoing.

UPDATE 09:30 CET: We are performing the upgrade and some services may be inaccessible for a short while.

UPDATE 10:50 CET: The upgrade is completed and login to all services should be working normally. Changes to two-factor authentication settings have been re-enabled. We will monitor the results for some time before proceeding to enable event logging and access control in the Runbox user interface.

UPDATE 15:30 CET: Some accounts are not able to log into IMAP and POP and we’re investigating what is causing this. We apologize for the inconvenience caused.

UPDATE 18:15 CET: We have located the cause for the login problems some users are experiencing and are working to fix it.

UPDATE 21:40 CET: During attempts to resolve the issue with some of the accounts we experienced compounding problems that have now been corrected. We’ll continue working to fix the original issue and apologize for any disruption you may be experiencing.

UPDATE 22:20 CET: All authentication issues appear to be resolved and access to all services is restored.

UPDATE 25.11.2021 01:15 CET: We are applying a fix to resolve a problem that affected a small number of users, and access may be limited for a short while. We apologize for any inconvenience caused.

UPDATE 25.11.2021 01:30 CET: The fix has been applied and authentication should now function correctly for all users.

UPDATE 25.11.2021 16:45 CET: We are investigating errors reported using Runbox 7 that are affecting some users.

UPDATE 25.11.2021 17:10 CET: We have located the error, which is caused by some browser user-agents sending very long strings identifying themselves and that exceeded the database field where they are being stored. We’re now deploying a fix for this issue and the problem should disappear shortly.

UPDATE 25.11.2021 22:15 CET: We’re still seeing that some users have login related problems and we’re working to investigate these.

UPDATE 26.11.2021 00:15 CET: Most accounts that have experienced problems via IMAP and POP today should now be able to access our services.

UPDATE 27-Nov-2021 14:25 CET: We are continuing to see some customers with IMAP/POP access issues. A new status post for this has been created https://status.runbox.com/2021/11/27/imap-pop-access-issues/

Posted in Service Status