Delays when checking out GitHub repos via SSH

Incident Report for Semaphore CI

Resolved

This incident has been resolved.
Posted May 20, 2026 - 15:32 UTC

Monitoring

A fix has been implemented, and everything should be back to normal. The previously shared snippets can be removed.
Posted May 20, 2026 - 15:02 UTC

Identified

We identified slow GitHub operations related to problems with the network, and we deployed a fix to re-route those operations to a healthier server. Some HTTPS operations started to fail because of that. To disable the fix, you should use the command:

sudo sed -i 's/^\([0-9].*github\.com\)/#\1/' /etc/hosts

And to re-enable, you can use:

sudo sed -i 's/^#\([0-9].*github\.com\)/\1/' /etc/hosts
Posted May 20, 2026 - 13:59 UTC

Monitoring

A fix has been implemented and we are monitoring the results.
Posted May 20, 2026 - 12:48 UTC

Identified

We identified slow communication between Semaphore jobs and GitHub. We are continuing to work with our upstream provider to resolve this. For impacted jobs, please replace the normal checkout command with the following:

retry --times 5 "timeout 40 bash -lc 'source ~/.toolbox/toolbox; checkout'"

And afterwards, use cd to change the directory to the cloned repository (replace with your project).
Posted May 20, 2026 - 08:44 UTC
This incident affected: Semaphore.