The notification landed at 3:17 a.m. I read it twice, hoping the second read would say something different. It did not. Authentication service down. Severity: Critical. Affected accounts: all users. I sat up in the dark, opened my laptop, and started working.
My client is a US-based operation. Their East Coast team would be online in about five and a half hours. That sounds like a lot of time. It is not a lot of time when you do not yet know what is broken.
What the logs said
The error pattern was consistent: SAML authentication failures across every user, starting at 2:54 a.m. my time. Not intermittent. Not one account. The service was up and the servers were healthy. The authentication layer was failing.
I have seen this shape before. SAML certificates expire, and when they do, they go silently. No warning in the application, no calendar reminder from the identity provider. I pulled the certificate metadata. Expiry date: the previous evening. The certificate had been invalid for eleven hours. Every login since then had failed the signature check without telling anyone why.
Knowing what broke took eight minutes. What came next was harder.
The edge of my safe action zone
I knew the fix. Generate a new SAML certificate on the identity provider side, export the updated metadata file, import it into the service provider configuration. I had done this before in test environments. The steps were not complicated.
But the production identity provider configuration sat behind an admin account I did not hold. The client's IT lead held those credentials, and she was in Chicago, where it was just past 7 p.m.
This is the moment nobody talks about when they talk about remote IT work: enough knowledge to see the fix, enough access to do some of it, not quite enough access to do all of it safely. The temptation to improvise is real. There is usually a workaround within reach. Things that might work, or might create a second problem on top of the first.
I sat with that temptation for about four minutes. Then I made the decision to call her.
What I had ready before I dialled
Before I picked up my phone, I spent six minutes writing a tight incident summary. Not because I needed to send it before calling, but because writing it forced me to be clear in my own head before asking someone to interrupt their evening.
The summary had four parts. What is broken: SAML signing certificate expired at approximately 16:00 UTC yesterday. What the impact is: all users locked out; East Coast comes online at 09:00 ET, roughly five hours from now. What I have confirmed: servers healthy, network healthy, expired cert confirmed in the admin console. What I need: admin credentials or a delegated session to regenerate the certificate and update the service provider metadata.
No apology paragraph. No lengthy context. Just those four things. She answered on the third ring, I gave her the summary in under ninety seconds, and she had the admin session credentials shared within two minutes.
The fix
From the moment she handed me the credentials, the technical work was about twenty-five minutes. Generate the certificate, export the metadata XML, log into the service provider admin panel, import the new metadata, test with a staging account, confirm the signing chain, re-enable the normal auth flow. No drama. The logs cleared immediately. I tested against three user accounts and got clean authentication tokens back from all three.
At 4:41 a.m. I sent a short incident summary to the client's shared ops channel: the root cause, what was fixed, when it was fixed, and a note that I would write the full post-incident review later that morning. Then I closed the laptop and went back to sleep.
The East Coast team came online at 09:00 ET. As far as most of them knew, the system had always been working.
What the 3 a.m. call actually taught me
The technical part of this story is not interesting. Expired certificates are one of the most common and most preventable incidents in IT. The interesting part is the twenty-two minutes between knowing what was wrong and picking up the phone.
There is a specific kind of pride that remote IT work can produce, and it is not always useful. The pride of solving things alone. Of being the person who handles it without waking anyone. Of having the client come online in the morning to a working system and never knowing there was a problem. That version of the story is satisfying. It is also the version where I make a unprovisioned change to a production tenant at 3 a.m. and hope nothing goes sideways.
The better skill is knowing exactly where your safe action zone ends. Not the edge of your technical knowledge, but the edge of your authorized action in that specific environment. Those two lines are not the same line, and the gap between them is where a lot of quiet incidents are born.
I could diagnose the problem fully. I could not apply the fix to the production identity provider without credentials I did not hold. That gap was clear once I thought about it carefully. The instinct to find a workaround that kept me inside my current access was the instinct I had to override.
Escalation is not a failure mode. It is a judgment call. The people who are genuinely good at on-call are not the ones who never escalate. They are the ones who escalate at the right moment, with the right information, without wasting everyone's time doing it.
The IT lead in Chicago did not think less of me for calling. She had the admin session open in three minutes because we had set up the emergency access protocol together, months earlier, exactly because we knew something like this could happen outside business hours. The call worked because the preparation existed. Neither of us was improvising at 3 a.m.
The prep work that makes 3 a.m. calls survivable
Running on-call from Nairobi for a US client is genuinely different from running on-call in the same timezone as your client. The obvious difference is the hours. The less obvious difference is that you cannot rely on walking over to someone's desk. Every escalation is a call, a message, an interruption to someone's non-work hours. That cost shapes how you prepare.
A few things I now keep in order specifically because of how distributed this work is:
A documented escalation matrix with actual phone numbers. Not Slack handles. Phone numbers. At 3 a.m. when a service is down, Slack may itself be having issues. I know who to call for which category of incident, and I have called each of them at least once in a non-emergency so the first time I call is not the first time they hear my name.
An emergency access protocol agreed in advance. For any production system I support but do not hold admin credentials for, there is a documented procedure for temporary elevated access. Agreed in writing, stored in a shared location, tested by both parties. It saved about fifteen minutes on the night in question.
A short incident template filled out before every escalation call. The four-part structure described above: what is broken, what the impact is, what I have confirmed, what I need. Writing it forces clarity and cuts the time anyone spends coming up to speed while half-awake.
Certificate and token expiry tracked in a shared document. After this incident I built a simple expiry tracker: service, credential type, expiry date, owner, and a 60-day advance notice rule. A spreadsheet. Nothing sophisticated. I have not had a surprise expiry since.
The sleep at 4:52 a.m.
I went back to sleep at 4:52 a.m. It was the best sleep I had gotten in weeks, not because the problem was solved, but because I had made the right call about when to stop handling it alone.
There is a version of that night where I spend two hours trying workarounds, wake the IT lead at 5 a.m. instead of 7 p.m. her time, and hand over a partially-modified production environment. That version costs everyone more. Distributed IT work runs on trust more than most people realize. You earn it not by solving everything alone, but by knowing precisely when not to.
If you are doing remote on-call work across timezones and want to compare protocols, drop me a line.
Comments