Intermittent login issues on HVCA and DSS APIs and TSA

Incident Report for GMO GlobalSign

Postmortem

An incident report for this issue is now available to download here:

GMO GlobalSign Incident Report - HTTP 503 errors returned by Atlas authentication service

Posted Oct 12, 2022 - 15:40 UTC

Resolved

All services have been restored to normal function. The investigation by the teams showed that a corrupted database index affected the performance of the system for some customers, leading to intermittent 503 errors, while others were unaffected by the issue. The index was re-created and this process completed as of 15:30 UTC, resolving the issue across all the affected customers.

Posted Oct 03, 2022 - 16:30 UTC

Update

The Infrastructure and Development teams are continuing to work the issue. At the current time a small number of customers are affected. We will continue to update.

Posted Oct 03, 2022 - 13:56 UTC

Identified

The Infrastructure team believe they have identified the issue and have put a workaround in place. Some calls to /login are taking longer than usual to complete but most customers should be seeing successful authentication now.
The team are continuing to work on the root cause and we will continue to post updates.

Posted Oct 03, 2022 - 07:14 UTC

Update

We are continuing to investigate this issue. At the current time we don't have an ETA for complete resolution, but our teams are working on it as the top priority.

Posted Oct 03, 2022 - 04:20 UTC

Investigating

We are investigating a potential issue which is giving occasional HTTP 503 ("Service busy") response to login (authentication) calls. We will provide further information as soon as we have identified the cause of the issue.

Posted Oct 03, 2022 - 01:13 UTC

This incident affected: Atlas (Document Signing Service (DSS), Certificate Issuance, Time Stamping).