Switch edu-ID Incident Reports

All users encountered an timeout message during the login process

Potentially every login during the time frame

Deployment of a faulty configuration on the load balancers lead to the unavailability of most Identity Provider nodes. The active nodes were not in a proper working state.

The faulty configuration of the load balancers was rolled back as soon as our monitoring system send out an alert and the mistake was noticed. 

• Prior coordination between involved teams before deploying new configuration

Most users encountered a slow login process or saw a timeout message

Potentially every login in the time frame

The combination of a temporary high load and a rolling upgrade of IdP nodes led to a degradation of the performance.

Only half of the IdP nodes were active after 10:00. The remaining nodes were not capable to handle the incoming login requests in an acceptable amount of time.

Disable the overloaded nodes and switch over to the spare nodes. Since the request peak was already over at the time, the remaining nodes could handle the load. The previously overloaded nodes were also enabled again after they have recovered.

• Review our maintenance practice. 
• Consider adding/augmenting resources
• Enhance coordination of rolling release planning with academic calendar.

Some users were not able to authenticate with MFA.

About 40% of the users which needed to login with a second factor or passkey during the incident were affected, and they had to retry later

The IdP nodes authenticate to our internal APIs using secure mTLS authentication. The renewal of involved certificates led to authentication failures on two of five IdP nodes, due to a mismatch of machine identities in the API configurations. The configuration mismatch was caused by a coincidence involving the introduction of new IdP nodes and usage of a shared client certificate on all nodes.

Rollback to the previous certificate configuration on the IdP instances, based on the documented rollback strategy

• Improve the management of internal mTLS authentication, introduce centrally managed machine certificates
• Improve error messages on the login services (the current messages mislead endusers to reset MFA, which was actually not needed)
• Further improve monitoring and alerting

edu-ID users could not create a new account.

Around 200 users were affected, and they had to retry later

The underlying cause was new version of PHP which prevented the account management to write back changes to the user database.

Rolling back to previous version of PHP.

• Improve alerting: the e2e tests did not cause an alarm during automated testing or after deployment.

Some edu-ID users (but not all) didn't receive SMS messages for around 30 minutes (10:00-10:30). SMS were not delivered in bursts.

This affected in particular the 2-step login for users without TOTP.

The SMS provider reported that "experienced unexpected network issue" caused some SMS not being sent.

The problems occurred in bursts and not for all users. Therefore, by the time the edu-ID team was made aware of the issue by a few users, the issue was already resolved.  Also, due to a monitoring problem the issue was not reported earlier.

• Automate switchover to alternative provider in case of problems.
• Monitor the monitoring system and ensure it is working properly all the time.

edu-ID users didn't receive SMS messages for around 30 minutes (10:00-10:30). This affected in particular the 2-step login for users without TOTP.

In total, there were 1793 unsent SMSes (present without delivery report in the logs) between 10:00 and 10:36 on 19.09.2024. In the same time range of the previous day, there were only 23 undelivered SMSes.

These requests are associated with 621 different mobile numbers. Thus, we can conclude that about 620 users didn't get their requested SMS for mobile verification.

The cause of the problem was a congestion in the delivery queue of our primary SMS provider.

We could switch over to our alternative SMS provicer after half an hour, such that SMS messages could be sent again. The primary SMS provider solved the problem later. We switched back to the primary provider before noon.

• Switching to the alternative SMS provider took too long. Streamline this process.
• Alert was not sent to all edu-ID staff. Extend alerting channels.
• The primary SMS provider was not aware of the issue. Switch needs a direct contact.
• Collect separate SMS metrics for each provider.
• Automate switchover to alternative provider in case of problems.

No SMS are sent to edu-ID users via an internal API. This affected in particular the 2-step login for users without TOTP. Also, no e-mails were sent via the same API e.g. to reset passwords. TOTP authentication was not affected.

In total, requests from 2687 different users failed. Thus, we can conclude that almost 2700 users saw at least one error during MFA login with SMS or mobile/email verification.

The problem was that the communication between two internal APIs failed due to an expired X.509 certificate whose automatic renewal failed.

Manually restart the acme-cert-renewal services on all nodes of the internal api for the client certificates.

• Verify and cleanup deployment of all APIs.
• include certificate expiration/renewal in monitoring.
• Implement monitoring and alerting for all critical APIs.

On Monday morning at around 8.00, start of fall semester for all Swiss universities, it was noticed the edu-ID login failed or was delayed for some users, while for others it went through smoothly.

Because not all users were affected and many users eventually managed to login after a few attempts, it is difficult to estimate the number of users. But about 200 additional support tickets, several phone calls and direct emails were retrieved by the edu-ID team and the Switch front desk. It is estimated that several thousand users were affected.

There were several factors that played a role in this issue: The many user logins (about 5x higher than in past weeks) due to semester start and the increased usage of MFA were two of them. However, the actually relevant cause was a missing index on a database table that consumed a lot of CPU in combination with the above.

Even though load tests were performed on the internal MFA API before it was enabled in Spring 2024 and even though the MFA API has been used for months without problems, this problem remained hidden until a massive number of logins by many different users triggered it.

The creation of a database index immediately solved the issue.

• Improve PostgreSQL SLIs to identify more quickly database congestions
• Check for other missing indices in database
• Improve SLIs and alerting of other critical components where possible
• Speed up notification of users. Prepare messages that can quickly be published on the IdP and the load balancer
• The incident was not displayed on https://status.switch.ch/ . Add "edu-ID login" to status page in addition to account management.
• The internal incident management had to be improvised due to team personnel changes. Improve handover processes. Conduct trainings for sucessors. Update incident management kit and checklist.