There are currently some IMAP/POP issues for some customers. We are looking in to the exact cause of this, but it looks like it could be a brute force attack on our servers. We will update this post when we have further information.
UPDATE 20:00 CEST: We are working to block the brute-force attacks against our IMAP/POP services and apologize for the inconvenience caused. You may use our webmail at https://runbox.com/ to receive and send email in the meantime.
UPDATE 23:30 CEST: The situation is improving and IMAP/POP access should be normalizing for most or all accounts. Our system administration team is continuing to monitor the situation and mitigating where necessary.
UPDATE 09:20 CEST: IMAP/POP services are currently working normally and we are continuing to monitor the situation.
Record of events and mitigation efforts
A thorough review of our records and server logs indicate that the login attempts on our Dovecot proxy servers gradually increased from a normal level at 11 CEST by approximately:
135% between 12 and 13 CEST,
209% between 13 and 14 CEST,
308% between 14 and 15 CEST.
At this time our system administration team at Copyleft Solutions were alerted to IMAP/POP connection issues, ranging from slow connections to no connections at all. They proceeded to reboot the Dovecot servers and then the Dovecot proxy servers, which lead to consequential authentication and proxying issues for a short while. Subsequently an increasing number of successful connections was recorded, but the login problems mainly prevailed.
Further investigations revealed what appeared to be a targeted brute-force attack on our IMAP service that effectively denied legitimate connections from a significant number of users. Once a brute-force attack was ascertained, further analysis of server logs found a significant number of IP addresses reached our Dovecot services despite our automatic authentication based brute-force prevention systems.
Our system administration team subsequently blocked the most frequent IP addresses by adding them to the central firewall. This appeared to alleviate the situation as the login volume dropped to 246% between 17 and 18 CEST. There were however still connection timeouts to the Dovecot servers, and Runbox staff was alerted and a cooperative effort was initiated. Because staff routinely operate from different geographic locations to cover all time zones, and some at this time had varying cell phone coverage, this additionally delayed our response.
Investigations and attack mitigation continued and additional blocking of IP addresses was attempted in the firewall on the Runbox gateway servers. However, the attack volume thereafter increased to approximately 327% between 18 and 19 CEST and was sustained at approximately the same level for the next few hours.
The mail authentication service on the Dovecot proxy servers continued to experience issues, and investigations continued to determine whether these problems were caused by brute-force attacks themselves or by cascading issues in the infrastructure.
By 21 CEST the attack volume started decreasing to approximately 292% and subsequently to:
269% between 22 and 23 CEST,
200% between 23 and 24 CEST,
74% between 00 and 01 CEST the next day.
Around midnight Dovecot services were once again restarted and the situation then normalized, after which the login volume returned to a level similar to prior to the incident.