Cool Ideas Fibre ISP – Feedback Thread 2

[PBCool]

personally, i'd like you to do "like the rest of the providers" and resolve outages in under 3 days. short of that, perhaps not attacking your actual (active, paying) clients on public forums would also go a long way.

The outage mitigation started when the DDoS started, and was totally mitigated by just over 36hrs. So far less than 3 days? I am not attacking anyone at all so please don't take it that way, again just being honest and straightforward.

 

[AfricanTech]

it's starting to feel like CI responds to outages with either emotive statements about the community-feel of the brand (correctly categorised earlier as crowdfunding CI which is the "we're all in this together so don't expect us to be prepared" defence) or with claims about the unique and once-in-a-lifetime nature of each event. it gets threadbare after a while.

i think a lot of people are feeling that way, whether they articulate the way i do, and i think that's a big motivator behind the tone and anger in this thread. underlying that is an unspoken doubt that CI is capable of what they set out to be capable of. in other comms, that i haven't included here, they imply that they've been hit harder than anyone else. (it comes across really conspiratorial and doesn't inspire confidence.) i'm sure they feel that way, and i'll allow that it could be true, but to put it in context, ISPs, banks, network providers, and the general tech media have been reporting rolling waves of attacks on SA for some time. it's definitionally not unique.

i'd like them to address these things, rather than ra-ra about their being a plucky band of brothers, doggedly fighting for my liberty and access to services.

anyway.

I take all your points but you have to be fair here - the attack was an order of magnitude greater than what others have experienced - I know of another organisation alerted to a 10Gbps attack over the weekend that lasted 4 minutes - it was successfully mitigated, but, had it been of the order of 380-500 they would not have coped

 

[daelm]

I take all your points

it's all good. i've resolved it as best i can.

 

[PBCool]

No, PB. i specifically didn't say that. i said the opposite, namely that it counted in CI's favour when comparing like-for-like providers. (you're welcome to go back and read what i wrote.) what i did say is that crow-barring that into a conversation about your response to continued outages is uncalled for and comes across as manipulative.

you're providing an example of that right now.

Can you let me know where we did this? Slotting in social responsibility as an excuse for an outage isn't something we do.

 

[AfricanTech]

I think Daelm responded very eloquently on behalf of a lot of us having doubts right now.
I would just add that I am a customer for some time right now and I would hate to change, but as much as I value feedback and interaction, that can’t make up for a weekend-long outage where others have been down for far shorter.

I must add that if I assume that the rabid-defenders on this thread are not sponsored, they are unfortunately probably part of the perception of arrogance, and that is unfair as they are not controlled by you.

There is a sad reality that internet forums don't encourage measured and nuanced interaction hence the rapid polarisaton of view. Yourself and Daelm have been fairly measured but your language and tone leads me to believe that you're both inadvertently straying into the same emotive territory that you purport PBCool to have strayed into.

Speaking for myself, I am by no means a 'rabid supporter' and I understand both sides - the frustrated frothers and the frustrated 'fanbois'

 

[daelm]

Yourself and Daelm have been fairly measured but your language and tone leads me to believe that you're both inadvertently straying into the same emotive territory that you purport PBCool to have strayed into.

agree to disagree

i have a lot of concerns, they're not being addressed, and the conversation is unlikely to do so, despite my trying to be quite clear about what they are. i've cancelled, rather than go over it again and again.

 

[PBCool]

@PBCool

PB, your response was just enough to move the needle. i'm coming up on my 2-year anniversary, and i've been pretty measured in this discussion. but i'm old and life is short, which is why i'm actioning the cancellation. i assume that's via email etc? i didn't see another option in there.

I'm sorry you feel that way also being a long term customer, just responding as objective as possible. We aren't trying to cover anything up here. You would email accounts@cisp.co.za your cancellation.

 

[AfricanTech]

No, PB. i specifically didn't say that. i said the opposite, namely that it counted in CI's favour when comparing like-for-like providers. (you're welcome to go back and read what i wrote.) what i did say is that crow-barring that into a conversation about your response to continued outages is uncalled for and comes across as manipulative.

you're providing an example of that right now.

While CI speak about their schools initiative in their Philosophy as an organisation in the preamble to the Zombie explanation, it didn't feel 'crowbarred in' to me...

 

[WickedP3NGU1N]

So Daelm, while your responses have been, as you state, quite measured your reasoning and thought process to give yourself that call to action to move do not seem as measured. While some may have mitigated faster there is still the reasoning that is up in the air which still seems to be that CI weathered a sustained attack whilst the rest seemed to only have to deal with single session attacks. When it comes to a DDoS there is no magic switch that makes it go away, especially and specifically when it is a sustained and quite purposeful attack.

Regarding the CSR that CISP put into their mailings and other social interactions, it is definitely something to be proud of and I personally cannot find fault with it because most schools suffer with less than stellar internet connections and it is also a call to action for others to do the same.

Honestly the way CI replies to things does not bother me, I prefer the straight forwardness to the typical Large ISP reading list replies that permeate the industry. I am pretty straightforward myself and I am sure a lot of other people prefer the straight answers to the typical textbook responses.

I do however chuckle when people move and the grass is not greener because all the shyte they are moving to was used to fertilize the grass to make it greener. So as always, don't let the door hit you on the way out and watch that needle just in case it moves back the other way...

 

[daelm]

I'm sorry you feel that way also being a long term customer, just responding as objective as possible. We aren't trying to cover anything up here. You would email accounts@cisp.co.za your cancellation.

thanks. please check your messages - i asked what else other than what's in the mail i needed to provide.

 

[daelm]

your reasoning and thought process to give yourself that call to action to move do not seem as measured. While some may have mitigated faster there is still the reasoning that is up in the air which still seems to be that CI weathered a sustained attack whilst the rest seemed to only have to deal with single session attacks.

actually, it's been some time coming. i just didn't interact here.

 

[fogbound]

@PBCool

I'm not sure the point is well enough made around your messaging to the subscriber base.

While I and many others understand the technical difficulties, you guys were experiencing, the use of your channels to inform and update was very poor, and it is that fact coupled with the lack of access to services that has many up in arms and wanting to switch.

Whether you think that you did your best or not, it would be wise to take some lessons from a PR company or have one on call to help you manage your channels during a crisis.

Most customers who aren't tech-savvy received two SMS's.

Your announcement page was the most active but not very informative, other channels like FB & twitter were pretty cold.

Buy the non-tech savvy majority, you will be judged by your messaging and you can do a lot better.

 

[TheRoDent]

Herewith the official communications regarding this weekends incident.

65 days later - Cool Ideas - Your Fibre Guy

bit.ly

And PDF format for those that prefer the direct download: http://bit.ly/65dayspdf

This will be sent via SMS, Social and email shortly.

 

[AfricanTech]

@PBCool

I'm not sure the point is well enough made around your messaging to the subscriber base.

While I and many others understand the technical difficulties, you guys were experiencing, the use of your channels to inform and update was very poor, and it is that fact coupled with the lack of access to services that has many up in arms and wanting to switch.

Whether you think that you did your best or not, it would be wise to take some lessons from a PR company or have one on call to help you manage your channels during a crisis.

Most customers who aren't tech-savvy received two SMS's.

Your announcement page was the most active but not very informative, other channels like FB & twitter were pretty cold.

Buy the non-tech savvy majority, you will be judged by your messaging and you can do a lot better.

A trap that the overly technical fall into constantly, unfortunately, I see it all the time in my organisation - techs are so focused on fixing things that they forget that the business side needs constant reassurance that the problem is being worked on.

Your suggestion of a communications company to manage the channels is an excellent one.

 

[daelm]

@PBCool

Whether you think that you did your best or not, it would be wise to take some lessons from a PR company or have one on call to help you manage your channels during a crisis.

Most customers who aren't tech-savvy received two SMS's.

Your announcement page was the most active but not very informative, other channels like FB & twitter were pretty cold.

Buy the non-tech savvy majority, you will be judged by your messaging and you can do a lot better.

thanks, fog. that was pretty much my whole point. i'm entirely sympathetic to the technical difficulties, but my working life was crippled, and i've received very little sympathy in return, despite being promised that i was joining a "community", filled with rich and extensive and personal communication. following that, it's been suggested that my lack of overt sympathy for the difficulty they were facing was letting our nation's schools down. i find that an odd response, and one i've decided not to entertain.

 

[Mars]

Well, while it was a frustrating weekend for all of us, it must have been insanely so for all of you. Well done @TheRoDent and @PBCool.

 

[AfricanTech]

Herewith the official communications regarding this weekends incident.

65 days later - Cool Ideas - Your Fibre Guy

bit.ly

And PDF format for those that prefer the direct download: http://bit.ly/65dayspdf

This will be sent via SMS, Social and email shortly.

Damn....

I hope you don't mind me reposting the technical bits here - makes for some reading

Timeline

October 2016 - first significant DDOS attacks against Cool Ideas
• Attack: Volumetric - DNS and fragmented UDP
December 2016 - Cool Ideas implements its own UK POP in order to manage DDOS
attacks before traffic reaches South Africa.
• Hurricane Electric as the first transit carrier.
• Peering established at LINX
January 2017 - Implement first UK based DDOS detection system, with automatic
mitigation.
• Remote Triggered Blackholing
• Netflow based

February 2019 - Added additional carrier in the UK - Cogent
• Attacked blocks are mitigated via secondary carrier
March 2019 - New DDOS detection tool implemented
• Automatic RTBH based mitigation on HE and Cogent
• Netflow based

11th Sept - First Major Attack ~20-40 Gbps
• Attack: Volumetric - DNS/LDAP/Memcached and fragmented UDP
• Detected, and mitigated by internal systems
• Blackhole routing of attacked IPS
16 Sept - Implement more traffic inspection points
• Later release of DDOS detection tools.
• Netflow based

Sat-Sun - 21st and 22nd Sept - Second Major Attack ~60-80 Gbps
• Attack: Volumetric - DNS/LDAP and fragmented UDP
• Attack: SYN/ACK and family floods.
Oct 8th - Construction of scrubbing centre in UK POP
• Upgrade transit capacity to 14 times original
• Implement scrubbing devices for attacked /24ʼs
• Implement scrubbing devices for attacked /32ʼs
• Faster attack detection using SFlow
• Instead of blackholing, CISP scrubs attacked IPʼs.
Oct 14th - Nov 22nd
• Daily multi-gigabit (~20Gbps) attacks mitigated automatically with no customer impact.
Sat-Sun - 23rd Nov and 24th Nov - Third Major Attack ~320-500Gbps
• Attack spread over multiple upstreams
• Attack: Volumetric - DNS/LDAP - fragmented UDP
• Attack: Volumetric - Various SYN related attacks
• Scrubbing capacity overrun
• Hurricane Electric announces ineffective
• Revert to RTBH based mitigation
Mon 25th Nov
• Implementing specialized scrubbing facility for overflow capacity.
• Implementing NAPAfrica Cape Town port upgrades and filtering.
• Implementing NAPAfrica Johannesburg port upgrades and filtering.

 

[PBCool]

A trap that the overly technical fall into constantly, unfortunately, I see it all the time in my organisation - techs are so focused on fixing things that they forget that the business side needs constant reassurance that the problem is being worked on.

Your suggestion of a communications company to manage the channels is an excellent one.

We are very much aware of this and something been discussed about on a lengthy basis. Too much comms we get complaints, too little isn't enough. Technical vs non-technical responses etc.

It is something we are commiting to refining for our customers.

 

[AfricanTech]

thanks, fog. that was pretty much my whole point. i'm entirely sympathetic to the technical difficulties, but my working life was crippled, and i've received very little sympathy in return, despite being promised that i was joining a "community", filled with rich and extensive and personal communication. following that, it's been suggested that my lack of overt sympathy for the difficulty they were facing was letting our nation's schools down. i find that an odd response, and one i've decided not to entertain.

I find it hard to countenance that you purport that CI is playing the emotional card and you do precisely the same thing.

Quite honestly, if your work is fully dependent on working internet at home, you really should have some redundancy - not having it is quite simply negligent on your part. Said before, that the reason that I'm prepared to put up with outages like this one (outside of everything else) is that it's not mission-critical for me at home - it's an inconvenience when not available. If I was fully dependent on my livelihood, I would have backup facilities in place, the simplest of which would be to be on Openserve rather than Frogfoot.

Please don't get me wrong, I'm not intending to attack you.

 

[daelm]

Damn....

I hope you don't mind me reposting the technical bits here - makes for some reading

Timeline

October 2016 - first significant DDOS attacks against Cool Ideas
• Attack: Volumetric - DNS and fragmented UDP
December 2016 - Cool Ideas implements its own UK POP in order to manage DDOS
attacks before traffic reaches South Africa.
• Hurricane Electric as the first transit carrier.
• Peering established at LINX
January 2017 - Implement first UK based DDOS detection system, with automatic
mitigation.
• Remote Triggered Blackholing
• Netflow based

February 2019 - Added additional carrier in the UK - Cogent
• Attacked blocks are mitigated via secondary carrier
March 2019 - New DDOS detection tool implemented
• Automatic RTBH based mitigation on HE and Cogent
• Netflow based

11th Sept - First Major Attack ~20-40 Gbps
• Attack: Volumetric - DNS/LDAP/Memcached and fragmented UDP
• Detected, and mitigated by internal systems
• Blackhole routing of attacked IPS
16 Sept - Implement more traffic inspection points
• Later release of DDOS detection tools.
• Netflow based

Sat-Sun - 21st and 22nd Sept - Second Major Attack ~60-80 Gbps
• Attack: Volumetric - DNS/LDAP and fragmented UDP
• Attack: SYN/ACK and family floods.
Oct 8th - Construction of scrubbing centre in UK POP
• Upgrade transit capacity to 14 times original
• Implement scrubbing devices for attacked /24ʼs
• Implement scrubbing devices for attacked /32ʼs
• Faster attack detection using SFlow
• Instead of blackholing, CISP scrubs attacked IPʼs.
Oct 14th - Nov 22nd
• Daily multi-gigabit (~20Gbps) attacks mitigated automatically with no customer impact.
Sat-Sun - 23rd Nov and 24th Nov - Third Major Attack ~320-500Gbps
• Attack spread over multiple upstreams
• Attack: Volumetric - DNS/LDAP - fragmented UDP
• Attack: Volumetric - Various SYN related attacks
• Scrubbing capacity overrun
• Hurricane Electric announces ineffective
• Revert to RTBH based mitigation
Mon 25th Nov
• Implementing specialized scrubbing facility for overflow capacity.
• Implementing NAPAfrica Cape Town port upgrades and filtering.
• Implementing NAPAfrica Johannesburg port upgrades and filtering.

yes. i saw that. here's what it could have said:

Why should I, as a customer, be confident that you've taken steps to defend against this?

<insert answer here>

What will you be doing when this happens again? What can I expect from you and what's your commitment to that?

<insert answer here>

How are you planning to communicate with your community going forward?

<insert answer here>

How have you communicated in the past, and what are you learning from that?

<insert answer here>

What does it mean for me that you're under a targeted - and seemingly highly specific - campaign of attacks, continuing over months? Is there a reason you think that's happening?

<insert answer here>

What's your contingency plan to keep my service active? What unique and innovative steps, if any, are you taking?

<insert answer here>

What's your promise to me and how do I keep you honest to it? How are we going to hold each other accountable?

<insert answer here>