TENET released RFP for on-demand international bandwidth

[rpm]

TENET wants on-demand international bandwidth

TENET today released an RFP for the purchase of disaster recovery international bandwidth in case their SEACOM link goes down

 

[Kelerei]

Looking at the two cable system possibilities, and excluding cost of bandwidth for now, SAT-3 would appear on paper to be the better option for them. Running up the African west coast translates into theoretical lower latency to Europe than East African cable systems; also, since the cable is in a different geographic region, the chance of a double cable breakage is practically zero.

Of course, this probably changes when one factors cost into the equation - I'm only speculating here, but I imagine that EASSy bandwidth would be a lot cheaper than that of SAT-3. Also, due to SAT-3's age and limited capacity, it's debatable whether SAT-3 can serve TENET's redundancy needs in future. WACS would likely have been the no-brainer if it was here by now (much more potential bandwidth on offer than either SAT-3 or EASSy, and located in the same geographical area as SAT-3), but an estimated 2011 launch date is way too far into the future for TENET's intents and purposes.

 

[rpm]

Looking at the two cable system possibilities, and excluding cost of bandwidth for now, SAT-3 would appear on paper to be the better option for them. Running up the African west coast translates into theoretical lower latency to Europe than East African cable systems; also, since the cable is in a different geographic region, the chance of a double cable breakage is practically zero.

I agree that SAT-3 is the better option, but with a minimum of 1 Gbps EASSy may make more sense. There may be a few EASSy shareholders with additional capacity while I am not certain how much additional SAT-3 capacity ISPs sit with…

 

[Johand]

Uhm... I think they are approaching it wrong - this will be very expensive! I think they should rather get bandwidth on the three different cables, with each cable one half of their required capacity:

1) All three cables operating, load balanced evenly - they get a bandwidth boost.
2) Two cables operating, load balance evenly - the have all their bandwidth
3) One cable operating - they only operate at half of their bandwidth requirements.

For somebody to satisfy their "on-demand" requirement will essentially require that bandwidth to be allocated to them in any case.

 

[LazyLion]

Yeah, I also want one of those.

 

[SomeoneEls]

Uhm... I think they are approaching it wrong - this will be very expensive! I think they should rather get bandwidth on the three different cables, with each cable one half of their required capacity:

1) All three cables operating, load balanced evenly - they get a bandwidth boost.
2) Two cables operating, load balance evenly - the have all their bandwidth
3) One cable operating - they only operate at half of their bandwidth requirements.

For somebody to satisfy their "on-demand" requirement will essentially require that bandwidth to be allocated to them in any case.

They probably signed a contract to cover their needs with seacom... and can't move say a third of the bandwidth to essay and another third to SAT3
You'd most probably also get a better price buying bulk that splitting it up into 3.

Ideally the cable should negotiate the redundancy... which if I'm not mistaken Seacom is going use mainone and that newly announced cable to south america... but thats a long way off

 

[erasma]

Load balancing across these cables as suggested does not makes sense

Not sure how they would do this? How would the weighting on the traffic be decided as these links all terminate in different parts of the world. So lantency could not be used! & just using all three is not a good idea either

the idea of having a main link & back up links with on demand is a good idea



did you lot no learn anything from the MPLS article earlier?

 

[thomas7127]

No serious ISP will sell them 1Gig "backup bandwidth" at a significantly reduced rate. The ISP would have to keep 1 Gig of his upstream capacity free all the time, just in case TENET requires it. That ISP may also have parts of his capacity unprotected on Seacom and will already be short of capacity exactly at the same time Tenet utilizes the backup capacity. 1Gig is 1Gig, it does not matter if it is used or not.
Charging customers based on the transfer volume works if you have 1000s of low speed ADSL customers, but not if you have one single high capacity customer.

If any ISP really offers such "backup capacity", it would mean that he intends to disconnect other customers when Tenet requires the bandwidth. Guess which part of the customer base that would be...

@erasma: The standard Internet routing protocol BGP4 will take care of the load balancing - especially if the links terminate in different regions. By using BGP4, the Tenet routers can decide if a certain destination is closer to their connection in London or to a possible connection in India (via SAFE). The USA is equally far away from South Africa via Portugal and England as via India and Asia. From this point of view, it does not make sense that Tenet rules out SAFE upfront. They would be better suited with a backup on SAFE, this would improve their connectivity to India and Asia at all times and it would provide a good backup for connectivity to the US. Only as backup for Europe, they would have to buy additional capacity on SAT3, but 300Meg (two STM1) instead of 1Gig should suffice.

 

[ambo]

They probably signed a contract to cover their needs with seacom... and can't move say a third of the bandwidth to essay and another third to SAT3
You'd most probably also get a better price buying bulk that splitting it up into 3.

This is pretty much the case... TENET has bought and paid up front for 20 years of access to Seacom at a capacity of 10Gbps. They will not save a cent by moving part of their traffic onto another cable system.

 

[Johand]

Load balancing across these cables as suggested does not makes sense

Not sure how they would do this? How would the weighting on the traffic be decided as these links all terminate in different parts of the world. So lantency could not be used! & just using all three is not a good idea either

the idea of having a main link & back up links with on demand is a good idea



did you lot no learn anything from the MPLS article earlier?

TENET is a network and not a single data-source. I completely agree trying to "load balance" a single MPLS 10Gbps stream would be impossible They don't need to load balance at the edges of the network, but they can actually have "load balancing" in the interior of the network - e.g. routers at each location can route data to the appropriate link, with appropriate alternative routes. They can even do nifty things like QoS or even route different protocols and/or users over different cables. (e.g. torrents from students can use the cheap high latency links, research applications from research hubs can use more expensive networks low latency networks). This is done all over the internet today and is in fact the standard way of doing things MPLS is nice, but in the end of the day you can do a lot on the higher IP layer (network layer), that you can't do on the MPLS layer.

So yes - my description was maybe a bit simplistic. For example they don't need the same capacity on all cables etc - it can get very complicated and a lot a cost/redundancy trade-offs can be made. But the principle is sound.

 

[FacELesS.]

No serious ISP will sell them 1Gig "backup bandwidth" at a significantly reduced rate. The ISP would have to keep 1 Gig of his upstream capacity free all the time, just in case TENET requires it.

I can think of the following scenario in which TENET will be able to purchase this bandwidth at a cheaper price compared to a dedicated 1 Gig redundancy:
Think of it as insurance or a swap agreement to hedge the SEACOM risk, rather than backup.
So what the counter party does is sell this bandwidth insurance to TENET at X amount which is less than the price of a dedicated capacity. But the counter party sells the same bandwidth as semi-dedicated to other customers at a reduced rate, due to the condition that it can be reallocated to TENET on demand if SEACOM breaks and for that period. The counter party is happy as the reduced rate + insurance premium from TENET > normal selling price; other customers are happy with below market price bandwidth; TENET is happy with a cheaper alternative compared to dedicated bandwidth.

 

[Roman4604]

MPLS is nice, but in the end of the day you can do a lot on the higher IP layer (network layer), that you can't do on the MPLS layer.

Would have to disagree with you there.

Using BGP in combination with explicity defined MPLS-TE tunnels is way more powerful & flexible than using BGP alone, even with its dmz-link unequal cost multipath functionality. The thing that usually catches you out with BGP (in an unequal cost scenario) is that it does use IGP hop count as one of the determinants for multipath (and it can't be turned off).

 

[AndrewAlston]

Would have to disagree with you there.

Using BGP in combination with explicity defined MPLS-TE tunnels is way more powerful & flexible than using BGP alone, even with its dmz-link unequal cost multipath functionality. The thing that usually catches you out with BGP (in an unequal cost scenario) is that it does use IGP hop count as one of the determinants for multipath (and it can't be turned off).

Hey All,

Firstly, for obviously reasons, I need to state that I cannot and will not comment on the RFP itself, or anything to do with responses to it etc. However, this thread has me rather intrigued with some of the technical aspects of the discussion going on and here I can comment.

Firstly, load balancing non-equal path bandwidth irrespective if you use BGP with dmz-link or using MPLS-TE and other methods is never fun. The moment you do true packet based load balancing over non-equal paths, particularly where there can be significant variations in latency, you can end up with out of order packets. This can cause a lot of issues that aren't otherwise anticipated. As for the higher level balancing, where you are using the application layer to determine path, this requires that you actually look at the packets at a higher than layer 3 level. This is far from cost effective at the type of packet per second rates you typically run at when you're at these speeds.

Even doing equal cost packet based balancing is problematic if the circuits are dramatically different from a latency perspective. Obviously this problem goes away if you do session based balancing, but that limits the throughput of a single session to the speed of the circuit through which its routed. While this may not sound like an issue, when you start talking about extremely high speed sessions for certain applications, it can create issues.

I also have to say that I dont think MPLS and BGP should be looked at as different methods to achieve an objective as SEEMS to be the case in the thread above. The two technologies are symbiotic, have very different functions and purposes and cannot be compared in any sense to each other.

 

[Polemus]

Would have to disagree with you there.

Using BGP in combination with explicity defined MPLS-TE tunnels is way more powerful & flexible than using BGP alone, even with its dmz-link unequal cost multipath functionality. The thing that usually catches you out with BGP (in an unequal cost scenario) is that it does use IGP hop count as one of the determinants for multipath (and it can't be turned off).

WTF?? i'm a techie, and i have no idea what you just said

 

[Roman4604]

The moment you do true packet based load balancing over non-equal paths, particularly where there can be significant variations in latency, you can end up with out of order packets.

Hi Andrew,

You'd never use per-packet load balancing in the backbone (& I'm certainly not advocating it), CEF's per-session default would be best practice eliminating any occurance of out-of-order packets.

 

[Roman4604]

WTF?? i'm a techie, and i have no idea what you just said

I'll try make it as simply as possible ...

BGP is a terminal hop protocol, in other words it is not concerned about intermediate hops, only the terminal destination (intermediate hops remains the job of the igp e.g. OSPF or IS-IS via recursive lookups).

Therefore one would think if a BGP router peers with two others that hold the same equal cost external route, but the two BGP peering paths are unequal, it would still multipath (load-balance) the external route in question.

Well it doesnt even though the cost of the two recieved BGP routes is equal, since it recognises that the two peering paths are unequal. With MPLS-TE one can engineer the two paths to look equal to BGP, thus achieving proper multipathing. An added advantage is by configuring FRR backup paths for each of the primary tunnels, one can achieve failover in around 50ms without affecting BGP (and its very slow reconvergence intervals).

That as simple as I could make it.

 

[thomas7127]

I can think of the following scenario in which TENET will be able to purchase this bandwidth at a cheaper price compared to a dedicated 1 Gig redundancy:
Think of it as insurance or a swap agreement to hedge the SEACOM risk, rather than backup.
...

The model could work in theory, but I doubt that there is a real market for this kind of semi-dedicated bandwidth which will become unavailable during a SEACOM outage.

 

[Nortic]

@erasma: The standard Internet routing protocol BGP4 will take care of the load balancing - especially if the links terminate in different regions. By using BGP4, the Tenet routers can decide if a certain destination is closer to their connection in London or to a possible connection in India (via SAFE). The USA is equally far away from South Africa via Portugal and England as via India and Asia. From this point of view, it does not make sense that Tenet rules out SAFE upfront. They would be better suited with a backup on SAFE, this would improve their connectivity to India and Asia at all times and it would provide a good backup for connectivity to the US. Only as backup for Europe, they would have to buy additional capacity on SAT3, but 300Meg (two STM1) instead of 1Gig should suffice.

Great comment, I had to quote it because it is worth a second read. TENET's sub 320ms requirement is quite harsh. This leaves them with only SAT3 for now because EASSY is still using the same SEA-ME-WE cable as Seacom. Therefore in the medium term if sea-me-we dies again EASSY will be dead in the water as well. I am not convinced yet that the 10gb 20 year seacom deal of which they use 10% is tax well spent? I certainly hope they received a massive discount to make it worthwhile.

MPLS is nice, but in the end of the day you can do a lot on the higher IP layer (network layer), that you can't do on the MPLS layer.

Quite correct, but MPLS nazis will never admit it, believing the benefit out weigh the restrictions it impose.

Would have to disagree with you there.

Using BGP in combination with explicity defined MPLS-TE tunnels is way more powerful & flexible than using BGP alone, even with its dmz-link unequal cost multipath functionality. The thing that usually catches you out with BGP (in an unequal cost scenario) is that it does use IGP hop count as one of the determinants for multipath (and it can't be turned off).

Johand's point is valid, BGP was designed for internet routing, that is what it is supposed to do. BGP's dmz-link is a feature in order to load balance traffic out of the network in proportion of the external links capacity. TENET's outbound utilization is 1% of their total capacity, it does not have real relevance to the discussion. All routing protocols need to keep track of metrics, if they dont, they would be useless and you forget to mention that TE use the same IGP metric as iBGP.

To be honest I dont see the need for MPLS/TE evangelism in this discussion.

 

[ambo]

I am not convinced yet that the 10gb 20 year seacom deal of which they use 10% is tax well spent? I certainly hope they received a massive discount to make it worthwhile.

TENET doesn't spend taxpayers money

 

[Nortic]

...one would think if a BGP router peers with two others that hold the same equal cost external route, but the two BGP peering paths are unequal, it would still multipath (load-balance) the external route in question.

One would never want iBGP to multipath over unequal cost paths dynamically, hence that is why IGP cost is considered in the best path algorithm before making the multipath selection. Imagine what would happen in a national or even worse intercontinental BGP topology.

Well it doesnt even though the cost of the two recieved BGP routes is equal, since it recognises that the two peering paths are unequal. With MPLS-TE one can engineer the two paths to look equal to BGP, thus achieving proper multipathing.

As said before, TE uses the same IGP metrics by default, if the two igp paths are not the same, they are not going to be the same for TE, it will need to 'engineered' to be the same, which holds true for IP as well.

An added advantage is by configuring FRR backup paths for each of the primary tunnels, one can achieve failover in around 50ms

50ms is the sales talk figure, in production it is closer to 5-10 seconds. Consider that RESV path error messages are sent once the problem interface is detected to go down. This depends on physical layer signalling that adds additional delays. Consider that the message still have to travel from the point of failure to the point of repair which in the case of SA could be 300ms+. CPU processing of error messages and FIB changes add further delays. All these delays add up which is why 50ms is never seen in large production deployments and why redundancy technologies that work closer to the physical layer are faster. One can say that with the help of BFD FRR is more responsive but then again the same applies to BFD for IP.

...without affecting BGP (and its very slow reconvergence intervals).

You already mentioned that BGP is not concerned of the intermediate hops, therefore the same holds true without TE.