Zimbra Email Service slowdowns / unable to login & send emails

We are now satisfied that the issues encountered yesterday by some of our Zimbra Email users have been completely resolved.

IMPORTANT:

** As password changes were not getting applied yesterday, if you are experiencing normal login issues with an account, please re-set the password.

** You may find when logging into the webmail that your first attempt fails, please try again, the second attempt should succeed.

We once again apologise to those who were inconvenienced yesterday by this issue.

We believe the issue to be a cached failed LDAP connection requests generated during our earlier LDAP issues. This connectivity failure resulted in mailbox password updates not being applied.

We are now seeing mailboxes update correctly across our platform but we shall continue to monitor.

NOTE: As password changes were not getting applied, if you are experiencing normal login issues with an account, please re-set the password.

We are still receiving some reports of login issues. It appears that the common thread may be that password changes are not currently being applied. We are continuing to investigate this issue and will post an update soon.

We are continuing to receive a small number of reports about login problems or trouble resetting passwords. As of 14:30 we think these are fixed. Please email support if you are still experiencing either of these issues.

Background

Our monitoring was showing intermittent high IO load on our LDAP Master server which we speculated was causing the above problems. Our post incident investigations identified the root cause of the LDAP problem and we were planning on implementing the fix for this tonight. However because of the observed high IO, we decided to implement the fix immediately. We observed an instant reduction in IO and we expect this to resolve the password reset and login problems.

We are continuing to monitor the situation but at this time we believe that the issue (a corruption found in the main LDAP configuration) has now been resolved.

NOTE: You may find when logging into the webmail that your first attempt fails, please try again, the second attempt should succeed.

We are escalating the problem to Zimbra to understand why other LDAP servers were not used by our Zimbra platform during the problem.

Work is still ongoing to resolve this issue as our top priority. Our apologies for the continues problems for those affected.

We have identified the source of the issue causing disruption for some of our users. In most cases this has manifest in a slowing of our service, though some users may also experience issues sending emails or logging into webmail. Steps are currently being taken to resolve this issue as soon as possible. Apologies to those affected.

Several reports have been made about a slow down of our Zimbra Email service. We are currently investigating this issue.