Can a layer 2 expert please grace me with their input

[TheSparrow]

https://bgp.he.net/AS37413#_graph4

There's how your ISP's ISP routes.

I fear you may have something much worse than a microwave backhaul - Openserve Metroclear.

Those are Bosal pipes into the cabinet with yellow Brady stickers. That's an expensive sentence. The kind of money thrown around by Openserve - or Neotel back in the day.

How this works is the FNO (or ISP) goes to Openserve and buys links into the Openserve cloud. There's a bacbone link that goes back their data centre and client links that go to client premises. The bandwidth is then split between many sites. Site A could have 512Mb, Site B could have 512Mb, Site C could have 1Gb and so it goes on. The cool thing about this is that there's also 512Mb between Site A and Site B for free, just set up a vlan. The problem is the pipe back to the FNO or ISP router. Back when we worked with this that was limited to something like 1Gb. If you wanted more than 1Gb, you had to bond 1Gb links. This is also hellishly expensive, so the FNO of ISP will buy the minimum amount needed. Openserve run this service over another service called Ethernet Express. So you receive an invoice for an Ethernet Express service, as well as a service for your leg of the Metroclear service.

An ISP signed us into one of those at work. They explained the costs to us and offered us a 48 month contract to get a zero install fee with Openserve. The service never delivered. In the end they installed a 60Ghz wireless back to one of their towers and their own fibre, that continues to provide us the 512Mb business service we were paying for - unlike the Metroclear where latency jumped to a point where we couldn't make phone calls when another client was downloading updates. The Nokia SAS-E switch is still in our cabinet, patiently waiting for the contract to run out, as is the very expensive 3M patch panel.

I also wanted to add here that I've seen our traffic take multiple different routes on different days. Sometimes on afrinic servers, sometimes on Frogfoot, and that was super weird to me. Would this system explain why the routes change every now and then?

 

[eddief1]

I also wanted to add here that I've seen our traffic take multiple different routes on different days. Sometimes on afrinic servers, sometimes on Frogfoot, and that was super weird to me. Would this system explain why the routes change every now and then?

@TheSparrow - No it would not.

You must remember that all the hops you see on your tracert's are your ISP's - 10.2.60.61 is already in the ISP network.

 

[belator]

@TheSparrow - No it would not.

You must remember that all the hops you see on your tracert's are your ISP's - 10.2.60.61 is already in the ISP network.

Agreed.

Please ask your ISP what their AS Number is.

Their own AS Number. Not their upstream's. Theirs.

Every ISP should have their own ASN.

 

[eddief1]

https://bgp.he.net/AS37413#_graph4

There's how your ISP's ISP routes.

I fear you may have something much worse than a microwave backhaul - Openserve Metroclear.

Those are Bosal pipes into the cabinet with yellow Brady stickers. That's an expensive sentence. The kind of money thrown around by Openserve - or Neotel back in the day.

How this works is the FNO (or ISP) goes to Openserve and buys links into the Openserve cloud. There's a bacbone link that goes back their data centre and client links that go to client premises. The bandwidth is then split between many sites. Site A could have 512Mb, Site B could have 512Mb, Site C could have 1Gb and so it goes on. The cool thing about this is that there's also 512Mb between Site A and Site B for free, just set up a vlan. The problem is the pipe back to the FNO or ISP router. Back when we worked with this that was limited to something like 1Gb. If you wanted more than 1Gb, you had to bond 1Gb links. This is also hellishly expensive, so the FNO of ISP will buy the minimum amount needed. Openserve run this service over another service called Ethernet Express. So you receive an invoice for an Ethernet Express service, as well as a service for your leg of the Metroclear service.

An ISP signed us into one of those at work. They explained the costs to us and offered us a 48 month contract to get a zero install fee with Openserve. The service never delivered. In the end they installed a 60Ghz wireless back to one of their towers and their own fibre, that continues to provide us the 512Mb business service we were paying for - unlike the Metroclear where latency jumped to a point where we couldn't make phone calls when another client was downloading updates. The Nokia SAS-E switch is still in our cabinet, patiently waiting for the contract to run out, as is the very expensive 3M patch panel.

Im surprised by the likes on this comment - In my experience Openserv ME was fantastic compared to Liquid, DFA for L2 - the sheer amounts of fibre routes they have at their disposal and hence diversity options was impressive - it just used to re-route in the background and do it's thing flawlessly for years (and so it should for the price)

@belator Most carriers would use Ethernet Express which is Point-to-point

 

[Network_Man]

I also wanted to add here that I've seen our traffic take multiple different routes on different days. Sometimes on afrinic servers, sometimes on Frogfoot, and that was super weird to me. Would this system explain why the routes change every now and then?

The backhaul (i.e Openserve, DFA, Liquid) is only Layer 2 which the FNO provides to the DC which is usually Teraco. There it gets handed off from the FNO pe switch to the ISP. The ISP is responsible for its own internal routing up until the traffic breaks out from their core. Then they are also responsible for resolving further issues down the path by reporting the issues to their upstream provider.

 

[belator]

Im surprised by the likes on this comment - In my experience Openserv ME was fantastic compared to Liquid, DFA for L2 - the sheer amounts of fibre routes they have at their disposal and hence diversity options was impressive - it just used to re-route in the background and do it's thing flawlessly for years (and so it should for the price)

@belator Most carriers would use Ethernet Express which is Point-to-point

Prior to the Openserve Metroclear we had a Neotel layer 2. That was more expensive than the Openserve. It worked well, but when something went wrong, it was chaos. Our backup wireless could easily run for a day while the ISP fought with India. In the end, our ISP learned, as soon as something's wrong, demand a Field Service Technician on site. Otherwise, the NOC in Pune just says nothing's wrong and the ticket is closed. That may have changed with Liquid's purchase.

I saw the quotes for Ethernet Express. We couldn't afford it. Simple as. The product may be good, but it's too expensive. Our ISP brought their Openserve account manager to our offices. He did a presentation on Metro Clear and it seemed to be the solution to use. It was probably what he was receiving the most commission on at the time. Since then, Openserve seem to be the only ones not involved in the race to zero.

I would not be surprised if WISPs and or small FNO's went out of business because they couldn't complete when others started dropping prices and Openserve not coming to the party. Imagine having a layer 2 to your high site in the middle of Polokwane, or your complex in Stellenbosch), paying R70K a month for it, charging your clients R900 a month for a 6Mb and then you are overbuilt and another company sells 20Mb for R500. We're all end users. Whether we manage a large data centre, or we have a 10Mb home connection, we're all price sensitive. People vote with their wallets. The only way our ISP has retained us for 12 years is by selling the layer 2 to us at cost price and adding a layer 3 markup on that.

I know for a fact (I have copies of emails and contracts) that our ME / Metroclear has a "price match" clause where Openserve were supposed to drop prices if the likes of DFA etc could offer the same product at a lower price. I also know that in our case they simply refused to implement it. Our ISP obtained competing quotes from Frogfoot and DFA for the layer 2. Both quotes were about 50% cheaper than we are paying at the moment. O/S turned around and said we weren't comparing apples with apples. The claim was the DFA was "dark fibre" - it wasn't - and the Frogfoot was PON - it wasn't. The Frogfoot was the same DFA product, just discounted thanks to Frogfoot's buying power.

As I write this, DFA are pulling fibre into our building with NTT / DiData sitting between them and our ISP. From 1 October we should be paying the same for a 2Gb service delivered on a 10Gb SFP as we are paying for a 512Kb that doesn't work properly and is actually being rerun over wireless. The only thing wrong with the 60GHz link is rain fade when it buckets down. If it wasn't for that, we'd just drop the fibre at the end of next month.

As far as I'm concerned Openserve is the last resort. If you're in a place with nothing else, take the hiding. Otherwise, there are many, better options.

 

[TheSparrow]

Prior to the Openserve Metroclear we had a Neotel layer 2. That was more expensive than the Openserve. It worked well, but when something went wrong, it was chaos. Our backup wireless could easily run for a day while the ISP fought with India. In the end, our ISP learned, as soon as something's wrong, demand a Field Service Technician on site. Otherwise, the NOC in Pune just says nothing's wrong and the ticket is closed. That may have changed with Liquid's purchase.

I saw the quotes for Ethernet Express. We couldn't afford it. Simple as. The product may be good, but it's too expensive. Our ISP brought their Openserve account manager to our offices. He did a presentation on Metro Clear and it seemed to be the solution to use. It was probably what he was receiving the most commission on at the time. Since then, Openserve seem to be the only ones not involved in the race to zero.

I would not be surprised if WISPs and or small FNO's went out of business because they couldn't complete when others started dropping prices and Openserve not coming to the party. Imagine having a layer 2 to your high site in the middle of Polokwane, or your complex in Stellenbosch), paying R70K a month for it, charging your clients R900 a month for a 6Mb and then you are overbuilt and another company sells 20Mb for R500. We're all end users. Whether we manage a large data centre, or we have a 10Mb home connection, we're all price sensitive. People vote with their wallets. The only way our ISP has retained us for 12 years is by selling the layer 2 to us at cost price and adding a layer 3 markup on that.

I know for a fact (I have copies of emails and contracts) that our ME / Metroclear has a "price match" clause where Openserve were supposed to drop prices if the likes of DFA etc could offer the same product at a lower price. I also know that in our case they simply refused to implement it. Our ISP obtained competing quotes from Frogfoot and DFA for the layer 2. Both quotes were about 50% cheaper than we are paying at the moment. O/S turned around and said we weren't comparing apples with apples. The claim was the DFA was "dark fibre" - it wasn't - and the Frogfoot was PON - it wasn't. The Frogfoot was the same DFA product, just discounted thanks to Frogfoot's buying power.

As I write this, DFA are pulling fibre into our building with NTT / DiData sitting between them and our ISP. From 1 October we should be paying the same for a 2Gb service delivered on a 10Gb SFP as we are paying for a 512Kb that doesn't work properly and is actually being rerun over wireless. The only thing wrong with the 60GHz link is rain fade when it buckets down. If it wasn't for that, we'd just drop the fibre at the end of next month.

As far as I'm concerned Openserve is the last resort. If you're in a place with nothing else, take the hiding. Otherwise, there are many, better options.

Could we make this into a new Netflix series? Like a mix of Silicon Valley and Narcos?

 

[TheSparrow]

Update on my findings:

Three other residents with different ISPs all experience bad packet loss. One resident confirmed Route Networks are using a wireless backhaul disguised as fiber.

My ISP and I found the following super weird situation:
- From my place to their exchange in rondebosch (right before the seacom point) it takes my packets 6ms.
- At this point it is joined by all their other capetown clients so I should be getting the same results.
- However, my packets get dropped at hop 4 or 5 or 6 randomly (usually one more than the other) and their other client packets don't (who didn't go through a Route Networks system).
- I tested a neighbor's internet with a different ISP and their packets also got lost on hop 5/6/7

1) This is a digital technology, how could packets be damaged by the FNO in such a way that different clients from different ISP's experience packet loss only on hops 4/5/6 and not on the first device it reaches after the FNO.

2) Is it legal to call it fiber if it's a wireless backhaul??

 

[HartsockZA]

Update on my findings:

Three other residents with different ISPs all experience bad packet loss. One resident confirmed Route Networks are using a wireless backhaul disguised as fiber.

My ISP and I found the following super weird situation:
- From my place to their exchange in rondebosch (right before the seacom point) it takes my packets 6ms.
- At this point it is joined by all their other capetown clients so I should be getting the same results.
- However, my packets get dropped at hop 4 or 5 or 6 randomly (usually one more than the other) and their other client packets don't (who didn't go through a Route Networks system).
- I tested a neighbor's internet with a different ISP and their packets also got lost on hop 5/6/7

1) This is a digital technology, how could packets be damaged by the FNO in such a way that different clients from different ISP's experience packet loss only on hops 4/5/6 and not on the first device it reaches after the FNO.

2) Is it legal to call it fiber if it's a wireless backhaul??

I have a better less BS involved route.

Get everyone that owns the units inside the complex (not the tenant) the actual owners to kick out the provider and restart an application to get proper coverage inside your building. There should be an escape clause inside the contract that the body corp signed with the provider. Usually, there is a clause that stipulates a 30-day remedial period allowing the ISP to resolve the issue but if not then out the window they go with costs. Your findings would be beneficial towards the body corp especially if it's a good complex.
It's frowned upon for your 2nd question. Usually called Air-fiber which is an actual thing wireless ptp links that simulate a fiber connection but that is a whole other matter.

Agreed.

Please ask your ISP what their AS Number is.

Their own AS Number. Not their upstream's. Theirs.

Every ISP should have their own ASN.

Every ISP should but that's like a sweet 100K plus every so often, so don't be surprised if they don't own one