Cool Ideas Fibre ISP - Feedback

[TheRoDent]

So I dont really care about having no internet tonight...

What i care about is what is actually the issue?
What is actually being done about it?
What contingencies are being put in place so it doesn't happen again...

SURELY CISP as a client of vumatel has the correct SLA's in place to answer these questions and the broadcast them to us? The client of CISP?

The universal SLA of "best effort service" is basically what it boils down to. No FNO, even Telkom provides SLA's for a consumer service other than "best effort" and "guidelines of mean time to resolve".

We get hundreds of requests for refunds from clients whose service was down for days, or weeks.

We rarely get a refund from an FNO, and if we do, then we will refund our client. It takes weeks of arguing to get a refund.

If the power is out, do you get a refund from CityPower or Eskom? I'm sorry to state it that crudely, because obviously we want to not be associated with something as basic as that, but it does kind of boil down to it...

If you want a business SLA, then the price goes up, and it is possible, but nobody wants to pay more for a better SLA backed service at home....

 

[John Tempus]

Ahem. Durbanville trenched, started a few minutes ago. All packets lost after second hop.

mtr -rc 20 154.0.1.245
Start: 2019-02-12T23:55:59+0200
HOST: xxxxxx.local Loss% Snt Last Avg Best Wrst StDev
1.|-- _gateway 0.0% 20 0.4 0.5 0.3 2.6 0.5
2.|-- 155.93.227.1 0.0% 20 6.8 14.8 2.5 38.0 8.9
3.|-- ??? 100.0 20 0.0 0.0 0.0 0.0 0.0

You are part of the chosen ones. You get to experience your fibre syncing and then get a donkey punch with no traffic.

I am still a level0 peasant without any fibre sync.

 

[image132]

@John Tempus Maybe I'm misunderstanding your conversation here but you keep asking why 2019 loadshedding is different. You ever consider that well sh*t happens?

You ever hear those stories of Eskom techs putting 440v through people's plugs and absolutely gutting the entire house, well fun fact surge protectors only work so well, nothing is full proof.

Just to be clear I'm not defending vumatel here, but these are clearly extraordinary circumstances, we've never had stage 4 loadshedding before, so who knows what havok its caused.

All I can say is just be happy you've got the internet you do. You're complaining about not having internet for 2 days.

I've been sitting with this for over 3 months.

 

[TheRoDent]

Oh I am glad you pointed out rarely having stage 4 because that was my entire point. Not entirely sure what you mean with traffic lights, light in my home ? I see no magic with traffic lights after loadshedding nor any magic with light in my home, care to elaborate what you mean?

I just meant that traffic lights are out during this load shedding, homes are without power... I get that you expect internet after power is restored, or even on UPS, but my point is that this week the entire South African infrastructure is upside down.

Sunday we had Stage 2 and no issues after power came back up. Monday we had stage 4 and hell afterwards, Today(tuesday) we had only stage 2 and it is even worse than Monday, fiber never coming back up now approach 8hrs since power returning.

So it takes no wise man to realize either stage4 destroyed Vumatel network and they are not giving us the facts or it doesn't matter if it is stage4 or stage2, the outcome is in fact even worse today after just stage2 loadshedding.

So the stage of loadshedding have no baring on the situation it seems.

Let's just agree that this entire thing is largely caused by load shedding, regardless of stage. The network is quite complex. And believe me, I have thousands of clients on Openserve, Octotel, Frogfoot and other networks having load shedding issues too. It's not just Vumatel....

So far mobile networks handled the loadshedding million times better than vumatel using my own experience.

Vodacom for me is dead during loadshedding, MTN on the otherhand is "usable" but extremely slow during loadshedding.

So, then how is Vodacom handling loadshedding better ? Sure, they are handling it better, but clearly not perfect either....

After loadshedding Vodacom work perfectly fine, MTN on the other hand is its usual crappy all over the place that I am used to whether loadshedding occurs or not.

I have no idea how Rain or Cellc LTE would perform during or after loadshedding however if they are also full of it then I will have to bite the bullet and just rip my wallet with vodacom bundles since they seem to be the absolute most reliable mobile service while fiber is being useless.

So is your internet access business critical ? If that is the case, then your plan is good. Have options, and backup.

But if the problem is not being able to Netflix or Torrent, then perhaps the unreliable fibre is still the most economical choice. Everyone dreads it when DSTV decoders go into "searching for signal" during a thunderstorm, but apparently that's OK, because it's a thunderstorm.

I don't think there's a magic bullet. There will be no single network that is 100% available to everyone's liking during loadshedding. Keeping hundreds of towers, or kerb switches UPS backed up, and diesel backed up is an abnormal situation for any network.

Yes, the mobile networks have done a better job at it since they've been at it for a long time, with towers located in weird locations, nowhere close to the density of fibre POP's, and where the design for power availability is sketchy at best.

The cost of this, however reflects in their data pricing.

 

[TheRoDent]

@John Tempus Maybe I'm misunderstanding your conversation here but you keep asking why 2019 loadshedding is different. You ever consider that well sh*t happens?

You ever hear those stories of Eskom techs putting 440v through people's plugs and absolutely gutting the entire house, well fun fact surge protectors only work so well, nothing is full proof.

Just to be clear I'm not defending vumatel here, but these are clearly extraordinary circumstances, we've never had stage 4 loadshedding before, so who knows what havok its caused.

All I can say is just be happy you've got the internet you do. You're complaining about not having internet for 2 days.

I've been sitting with this for over 3 months.

Just some feedback on this, sorry haven't PM'ed you yet. FNO is now investigating backhauls to Claremont/Rondebosch/Kenilworth.

Had a telecon this morning.

 

[John Tempus]

Yes, the mobile networks have done a better job at it since they've been at it for a long time, with towers located in weird locations, nowhere close to the density of fibre POP's, and where the design for power availability is sketchy at best.

The cost of this, however reflects in their data pricing.

This is at the core of the issue, the mobile towers actually have backup power to an acceptable degree unlike what Vumatel claim. Vumatel is making false claims that all their POPs front to backend have sustainable backup power. Backup power does not mean remaining up for a extra hour if you go by any SLA standards. Since you manage a datacenter you are well aware that if you were to estimate safety protocols into your SLA that needs to estimate potential backup power usage within a year timeframe then you would know that at any point throughout the year you might have to use 7days of backup power which involve gens/batteries and the likes due to much higher power requirements.

To keep POPs up would be managable with effective UPS's for a day or two if they dont entirely go cheap skate on it and just plan to keep it up for an hour which is as good as having no UPS at all.

So this is where we differ, a sustainable backup system would need to be able to handle at a minimum 24hrs of possible power outage and anything less is next to useless thus giving you very little to no room to manouvre in.

 

[TheRoDent]

Ahem. Durbanville trenched, started a few minutes ago. All packets lost after second hop.

mtr -rc 20 154.0.1.245
Start: 2019-02-12T23:55:59+0200
HOST: xxxxxx.local Loss% Snt Last Avg Best Wrst StDev
1.|-- _gateway 0.0% 20 0.4 0.5 0.3 2.6 0.5
2.|-- 155.93.227.1 0.0% 20 6.8 14.8 2.5 38.0 8.9
3.|-- ??? 100.0 20 0.0 0.0 0.0 0.0 0.0

Yeah, the second hop is one of the IP's we give to Vumatel that is assigned on their network as default gateway for your service.

Third hop is supposed to be us, in Bree Street, but clearly it's not making it.

 

[DJZassie]

The universal SLA of "best effort service" is basically what it boils down to. No FNO, even Telkom provides SLA's for a consumer service other than "best effort" and "guidelines of mean time to resolve".

We get hundreds of requests for refunds from clients whose service was down for days, or weeks.

We rarely get a refund from an FNO, and if we do, then we will refund our client. It takes weeks of arguing to get a refund.

If the power is out, do you get a refund from CityPower or Eskom? I'm sorry to state it that crudely, because obviously we want to not be associated with something as basic as that, but it does kind of boil down to it...

If you want a business SLA, then the price goes up, and it is possible, but nobody wants to pay more for a better SLA backed service at home....

Never said anything about a refund.

My question was, surely CISP has the SLA's with vumatel to get the answer to those questions...

As vumatel only deals with ISPs I would hazard a guess that that would mean a tier above a typical business connection...

As you are a reseller (for lack of better words)...

 

[image132]

Just some feedback on this, sorry haven't PM'ed you yet. FNO is now investigating backhauls to Claremont/Rondebosch/Kenilworth.

Had a telecon this morning.

Thanks for that.

With all the vumatel issues going on I didn't really want to divert the thread.

My point to @John Tempus was that if this is his first issue with vumatel and it required stage 4 loadshedding for it to occur. That's pretty good in my book.

 

[John Tempus]

Thanks for that.

With all the vumatel issues going on I didn't really want to divert the thread.

My point to @John Tempus was that if this is his first issue with vumatel and it required stage 4 loadshedding for it to occur. That's pretty good in my book.

Hey stage4 at least gave me 3hrs of internet on monday. Stage 2 on tuesday seem to be the atlas killer taking me offline all day so far.

 

[TheRoDent]

This is at the core of the issue, the mobile towers actually have backup power to an acceptable degree unlike what Vumatel claim. Vumatel is making false claims that all their POPs front to backend have sustainable backup power. Backup power does not mean remaining up for a extra hour if you go by any SLA standards. Since you manage a datacenter you are well aware that if you were to estimate safety protocols into your SLA that needs to estimate potential backup power usage within a year timeframe then you would know that at any point throughout the year you might have to use 7days of backup power which involve gens/batteries and the likes due to much higher power requirements.

To keep POPs up would be managable with effective UPS's for a day or two if they dont entirely go cheap skate on it and just plan to keep it up for an hour which is as good as having no UPS at all.

So this is where we differ, a sustainable backup system would need to be able to handle at a minimum 24hrs of possible power outage and anything less is next to useless thus giving you very little to no room to manouvre in.

Sure, I agree with the design principles you propose. We have gennies, UPS'es and crazy infrastructure to make sure our DC doesn't go down.

But we don't get to deploy that all on a curb. Yes, Vumatel's claims are far reaching.

The "SLA" is that there is NO "SLA". Best effort service for consumers.

I'm not saying they're the best. What I am saying is that they've done a lot. It's not perfect.
And this is not about the DHCP thing, that's just silly, and software and can be easily sorted.

But, would you be willing to pay 10-50 times more, if they had 1 week of battery backup, and a 2 hour SLA on fibre breaks?

Because that's what I pay for carrier grade fibre links...

 

[TheRoDent]

Jis, I'd love to play Apex: Legends now rather....

 

[image132]

Hey stage4 at least gave me 3hrs of internet on monday. Stage 2 on tuesday seem to be the atlas killer taking me offline all day so far.

My reading of the situation is that it is an "extraordinary event." If Vumatel's network made it through 2017 and 2018 loadshedding just fine but is broken now. Something must have failed that wasn't supposed to fail.

Anyway man hope they fix it asap for you. Sucks when you have to go outside

 

[John Tempus]

Sure, I agree with the design principles you propose. We have gennies, UPS'es and crazy infrastructure to make sure our DC doesn't go down.

But we don't get to deploy that all on a curb. Yes, Vumatel's claims are far reaching.

The "SLA" is that there is NO "SLA". Best effort service for consumers.

I'm not saying they're the best. What I am saying is that they've done a lot. It's not perfect.
And this is not about the DHCP thing, that's just silly, and software and can be easily sorted.

But, would you be willing to pay 10-50 times more, if they had 1 week of battery backup, and a 2 hour SLA on fibre breaks?

Because that's what I pay for carrier grade fibre links...

Lets get real their POPS are large enough to house a possibly 5-10KVA backup UPS. The power draw that would occur at the POPS is minimal where a 10KVA UPS would easily last 16-24hrs depending on hardware at these distribution POPs.

We are not talking bout spending millions extra, but we are talking about spending something and I think they are simply cutting corners and costs by not even doing this yet claiming they are.

They are still claiming non stop that their network does not get affected by load shedding and it is all local to CPE's at customer premise. They are still claiming having UPS on your CPE will allow you to continue using Fiber even during loadshedding. Both of these are provable lies which is why I find anything else about the network design done with performance rather than cost cutting hard to believe if they are so brazen by continuing to lie about these easy to test situations.

 

[John Tempus]

My reading of the situation is that it is an "extraordinary event." If Vumatel's network made it through 2017 and 2018 loadshedding just fine but is broken now. Something must have failed that wasn't supposed to fail.

Anyway man hope they fix it asap for you. Sucks when you have to go outside

Guess I come off a bit strong, I have internet to use for backup. I just felt I would try to be the voice in this thread for other people having same issues. Worse case if I have to move from fiber then so be it however other people might not have such easy options and without rattling the cage things tend to move even slower.

I am ultimately hoping to get a honest response from Vumatel about the situation via CISP agents. I am hoping we can get that much luck out of all the bitching.

 

[TheRoDent]

Never said anything about a refund.

My question was, surely CISP has the SLA's with vumatel to get the answer to those questions...

As vumatel only deals with ISPs I would hazard a guess that that would mean a tier above a typical business connection...

As you are a reseller (for lack of better words)...

Sure. SLA's normally go with penalties, or refunds which is kind of why I went that route in the discussion.

I could show you the SLA's but it's confidential, and eventually meaningless. The SLA provides me as an ISP with very little recourse (in terms of refunds) and the general stipulations in terms of service as to Service Level is so vague that it's not really enforceable either.

3% packet loss on a certain network ( cannot name names) is apparently perfectly acceptable. This does not equate to a good service for my clients, which is why we no longer take orders for their network.

What it boils down to, is to have a simple good human relationship with the people at the FNO.

If it gets to the point where legal details or nitty gritty matters then one is either dealing with Telkom, or a certain network which I shall not name.

 

[John Tempus]

Sure. SLA's normally go with penalties, or refunds which is kind of why I went that route in the discussion.

I could show you the SLA's but it's confidential, and eventually meaningless. The SLA provides me as an ISP with very little recourse (in terms of refunds) and the general stipulations in terms of service as to Service Level is so vague that it's not really enforceable either.

3% packet loss on a certain network ( cannot name names) is apparently perfectly acceptable. This does not equate to a good service for my clients, which is why we no longer take orders for their network.

What it boils down to, is to have a simple good human relationship with the people at the FNO.

If it gets to the point where legal details or nitty gritty matters then one is either dealing with Telkom, or a certain network which I shall not name.

Fibrehoods ? *Dang he went there*

EDIT: fibre just synced after 9hrs, BUUTT.. no traffic at all. Talk about blueballs.

 

[TheRoDent]

Lets get real their POPS are large enough to house a possibly 5-10KVA backup UPS. The power draw that would occur at the POPS is minimal where a 10KVA UPS would easily last 16-24hrs depending on hardware at these distribution POPs.

We are not talking bout spending millions extra, but we are talking about spending something and I think they are simply cutting corners and costs by not even doing this yet claiming they are.

They are still claiming non stop that their network does not get affected by load shedding and it is all local to CPE's at customer premise. They are still claiming having UPS on your CPE will allow you to continue using Fiber even during loadshedding. Both of these are provable lies which is why I find anything else about the network design done with performance rather than cost cutting hard to believe if they are so brazen by continuing to lie about these easy to test situations.

I don't disagree. As Dr. House (MD) says: "Everybody Lies".

Again, would you be willing to pay 10x the price if they provided an SLA to your satisfaction?

 

[TheRoDent]

Fibrehoods ? *Dang he went there*

EDIT: fibre just synced after 9hrs, BUUTT.. no traffic at all. Talk about blueballs.

What does a trace look like ?

 

[TheRoDent]

Fibrehoods ? *Dang he went there*

Nope, try again...