When cloning a repository from GitHub I sometimes only get a download rate between 50-100 KiB/sec (staying stable) while most of the time I have about 10 MiB/sec. When cloning the same repository from a different machine (= different global IP) I get full speed.
Does GitHub impose a rate limit on repository cloning? The repository in question is quite big (~100 MiB) and I clone it about twice a day.
Do you have massive binaries committed in the repos? That might do it.
Otherwise, look at optimizing your CI's behavior. Instead of:
git submodule update [--recursive]
You want:
git submodule update [--recursive] --depth 1
CI doesn't need the whole repo history, just the target state. More details here: Git shallow submodules