[jkroon's problem]

It's actually even worse than most people seem to think:

http://jkroon.blogs.uls.co.za/it/security/trouble-shooting-tcp-rst-problems

They're breaking things in such a subtle way that it actually took me a few hours to realize what they're doing wrong.

I've got a good suspicion what's causing your problem. Pretty sure it's not the Content Adaptation but rather an issue with the network cache or firewalls. Should be pretty simple to fix.

Mods, can you move this over to the Vodacom sub-forum as a new thread and we can pick it up there?
 
Exactly which 'tricks' and 'shenanigans' are we talking about? :confused:
Quoting the short (English) version of TFA:
It seems that the provider has equipment on it’s network that is modifying TCP connections in some rather obscure ways, specifically, they are making adjustments to the sequence numbers on the outbound data stream...


@jkroon: You should register for the 'unrestricted' APN and see if your results are different. Apparently when you connect on 'unrestricted' you bypass Vodacom's transparent proxies.
 
Last edited:
Quoting the short (English) version of TFA:



@jkroon: You should register for the 'unrestricted' APN and see if your results are different. Apparently when you connect on 'unrestricted' you bypass Vodacom's transparent proxies.

Already sorted this out with Jaco. He saw TCP sequence renumbering that got him all excited but it's actually a (standard) security step implemented to protect users from TCP attacks.

And is according to RFC 1948.

But the actual problem Jaco had was with how the firewalls handle SACK and we've escalated this to the firewall vendor for resolution.

While on the topic of IP security, we often get these 'this or that does not work on the Vodacom network but does on network X".

Invariably when we analyse these, there is a logical reason why it does not work and we have to put workarounds in place. Read; jippo the system or standards.

This makes me wonder if VC is the only network that actually bothers to put these engines in place?

It would save us a hell of a lot of time and money to remove them but then users will be open to various threats.
 
This makes me wonder if VC is the only network that actually bothers to put these engines in place?
Does the 'unrestricted' APN bypass these engines as well?
It would save us a hell of a lot of time and money to remove them but then users will be open to various threats.
Seeing I'm paying for every byte in and out of my PC I'm thankful for the filtering. It is scary the amount of attack traffic one gets connecting with PPPoE to Telkom's ADSL network.
 
Already sorted this out with Jaco. He saw TCP sequence renumbering that got him all excited but it's actually a (standard) security step implemented to protect users from TCP attacks.

And is according to RFC 1948.

But the actual problem Jaco had was with how the firewalls handle SACK and we've escalated this to the firewall vendor for resolution.

While on the topic of IP security, we often get these 'this or that does not work on the Vodacom network but does on network X".

Invariably when we analyse these, there is a logical reason why it does not work and we have to put workarounds in place. Read; jippo the system or standards.

This makes me wonder if VC is the only network that actually bothers to put these engines in place?

It would save us a hell of a lot of time and money to remove them but then users will be open to various threats.

+1 to Vodacom
 
so what jkroon found VC new already ?

so nothing was actually found cause someone knew

about it already......interesting..lokl


+1 VC
 
Does the 'unrestricted' APN bypass these engines as well?

Nope, this is done at firewall level and the unrestricted APN will still pass through the firewalls, but with incoming ports open.

The firewalls will still check for protocols that behave out of norm (like using a known port for your own application), and in-line virus checking, etc. will still be in place as well.

But the cache is bypassed and the rules on the firewall is slightly tweaked.

So you're still protected as much as possible on the unrestricted APN but, by definition, port scanning, etc. will affect you.
 
so what jkroon found VC new already ?

so nothing was actually found cause someone knew

about it already......interesting..lokl


+1 VC

Actually a bit of both. Jaco had a problem and tried to analyse it himself (which is GREAT, I love the quality of some of the skills we have in this country).

The thing he though that could be the cause (the renumbering) is actually a feature.

But he still had a problem and it was isolated to how SACK is handled by the firewall.

So Jaco did actually discover a bug, but the other symptoms he saw was normal and nothing sinister.

Another great example how communities like myADSL help improve the network for everyone.

There are a number of forumites who often help us and it's great. Morkhans, ginngs, ic and the rest of the bunch.
 
There is only one snag. The so called security feature counts for nothing. I have (shortly after posting here and speaking with you and receiving feedback from the advanced data centre) posted a follow-up blog entry (http://jkroon.blogs.uls.co.za/it/security/vodacom-responds) which I'll partly repeat here.

Basically the quoted RFC states that a _SERVER_ should not respond with predictable sequence numbers. There is no reason why the client can't use predictable sequence numbers as the server HAS to take the sequence numbers at face value anyway. It's the RESPONSE that may not be predictable. Vodacom is modifying the initial outbound request (ie, the CLIENT sequence numbers). It DOES NOT address the issue as described in the RFC at all.

Congrats to vodacom for misinterpreting the RFC.

In addition to the fact that the implementation doesn't actually protect against the attacks described in the RFC, the RFC itself states that it's NOT an effective measure but rather an intermediate measure to assist with the protection of hosts. For those who is not going to go read the RFC, the attack described is a blind TCP injection attack that was aimed at plaintext protocols like rlogin with host trust enabled (ie, would completely trust the username as provided by the client WITHOUT requiring password authentication). Granted, this can be pulled against HTTP servers as well where authentication is done against the IP address of the incoming connection, but again, VC modifies the CLIENT sequence numbers, so they're not providing ANY protection whatsoever.

Then, secondly, you wanted to know whether other network providers bothers to switch on such measures, in a word: NO. The reasons are simple, they prefer to provide better performing networks, these kinds of modifications (having actually coded some of them myself) causes rather big performance hits both in terms of CPU requirements as well as memory usage on a per-connection basis. Just the measure that is causing this problem alone puts VC in a position where they've got a single point where ALL traffic for a specific connection MUST pass over. Ie, they're effectively causing a throughput bottleneck for themselves, by choice, with absolutely no benefit whatsoever.

The transparent HTTP proxies can be seen the same way, however, there is an advantage for VC here, it's possible to "mark" all port 80 traffic and route it differently and to still manage to get load balancing at the routing level, once the packets are past their proxies the outbound direction can follow normal load balancing procedures, and again the same on the return. Well, it's hard to visualize, but the only bottle-neck here needs to be that each individual connection needs to go to the same "proxy" in the cluster. Since the proxies themselves should already be clustered for content-sharing purposes and be performing callouts between each other this should not be too serious an issue. All in all I personally believe that transparent proxies doesn't really improve network performance all that much but actually more serve as a cost-saving for network providers such as VC. Reasoning here is simple: They can still charge you for the bandwidth but they get it much cheaper now since they don't have to pay for that bandwidth traveling over their inter-connects to other peering parties. Don't get me wrong, if an object doesn't have to be retrieved from further away that's all great, but you're adding additional overhead in order to provide this benefit. These overheads can be seen if you analyze how these proxies function:

1. connection attempt to port 80 is intercepted
2. http request is passed to intercept point
3. content cache check is done (takes time).
4. connection is spoofed (netproxy being the obvious choice here).
5. content is retrieved and passed back.

So instead of the traffic simple being routed it's passing all the way up and down a fully fledged TCP/IP stack at least once extra. All in all I myself make use of transparent proxies, get me right, I'm not against them. It should just be noted that considering the bottleneck link is between the proxy and the browser their biggest benefit isn't performance but cost saving for VC (need smaller pipes to their peers, costs less - MUCH less) which hopefully gets passed back to clients.

I've actually simply given up on this, Vodacom can go jump in a lake for all I care, I've just gotten myself some DSL at home instead. Also, issues like the DNS bugs where I often (every initial connection after the modem has been plugged in) receive 10.11.12.13 and 10.11.12.14 as DNS servers detract from the service.

Things like "in line virus checking" really does sound like the kind of crap we were up to at Netronome, which quite frankly, was stuff that was being sold to the likes of CISCO and Intel. And the bug I ran into was in fact very, very similar to a _known_ bug that was still in my code when I left the company. But we didn't do explicit sequence number offsetting so unless we were actually splicing the connections it wouldn't show, and we'd strip off SACK options anyway once we initiated splicing due to other technical reasons.

/me ends rant.
 
It's actually even worse than most people seem to think:

http://jkroon.blogs.uls.co.za/it/security/trouble-shooting-tcp-rst-problems

They're breaking things in such a subtle way that it actually took me a few hours to realize what they're doing wrong.
Is that post still available somewhere?

It is useful for showing people the issues that dodgy firewall settings can cause. (I also had the same issue at work with Cisco firewalls and the post helped a lot)

(I know this is an ancient thread, starting a new thread about this seems like it would be off-topic, since this is not about mobile data, but the TCP post) (and the chance of Jaco seeing it would be almost zero)

Archive version (but Internet Archive is blocked in many places and mostly not indexed by search engines):
https://web.archive.org/web/2013031...it/security/trouble-shooting-tcp-rst-problems
https://web.archive.org/web/2013092....blogs.uls.co.za/it/security/vodacom-responds
 
Top
Sign up to the MyBroadband newsletter
X