Intermittent connection resets from raw.githubusercontent.com on Afrihost

[grim42]

I have some software that periodically checks for updates on raw.githubusercontent.com that has worked fine for years but started failing intermittently around the start of July. I see this same thing on 3 different AfriHost+OpenServe fibre lines but not when I try the same on a separate, Vodacom connection.

I can consistently reproduce this with curl, getting intermittent connection resets approximately 10% of the time:

for i in {1..100}; do
    curl -vfsSNL -o /dev/null --retry 3 "https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh"
done

(arbitrarily testing against the Homebrew repo, but this happens for anything on raw.githubusercontent.com).

* Recv failure: Connection reset by peer
* OpenSSL SSL_connect: Connection reset by peer in connection to raw.githubusercontent.com:443
* Closing connection
curl: (35) Recv failure: Connection reset by peer

* OpenSSL SSL_read: Connection reset by peer, errno 54
* Failed receiving HTTP2 data: 56(Failure when receiving data from the peer)
* Connection #0 to host raw.githubusercontent.com left intact
curl: (56) Recv failure: Connection reset by peer

I also setup Smokeping to run this check every 5 minutes and it's happening very consistently:



This GitHub traffic appears to be served from a Fastly cache in CPT. Wondering if other folks on AfriHost are seeing the same thing?

 

[Afrigirl]

Hello.

Yes, seems like GitHub is dropping the connection, maybe because of rate limiting

 

[grim42]

Thank you.

I'm not sure about rate limiting -- unless you're saying that all AfriHost users are collectively being rate limited?

This happens consistently whether I query 10 times, 100 times, or 1000 times. The Smokeping test is separating queries by 1 second, and I've varied the delay in my adhoc testing to see if rate had any effect. The actual software that is failing as a result makes probably 10-20 queries at most, in serial over about 30 seconds.

This doesn't match what I would expect to see with rate limiting -- and the nature of the failure is a TCP RST at random points during the connection: during establishment, SSL handshake, and even truncating the HTTP response.

This traffic is being served from a Fastly CDN cache in CPT.

I found this post on Fastly's community forum which looks like it could be a similar issue. That says that Fastly uses Anycast IP addresses (which seems to be the case here) and that ECMP multipath routing could be an issue. The issue there appears to have been on the ISP side (and was resolved).

Any chance that could be applicable here?