Hi Guys,
Just from an ISP perspective I'll try give you some insight as to these blacklists and RBL providers - a bit of an essay.
This is something which happens constantly. Most providers use the SpamExperts spam solution, which is currently the best and most widely used amount all hosting companies world wide. A user gets a virus on their laptop or their email password gets compromised through a trojan and then the spam emails start. 99.9% of these types of emails get stopped by the outgoing spam filters, but there is always that 0.01% chance, when the wind blows a certain direction and the moon aligns that a couple of these make it through.
These then get picked up by the users (sys admins) receiving them and get reported to spamcop/sorbs/etc.
The ISP gets notified that they are blacklisted and by this time the ISP (in most cases) has already resolved the issue with the client, either by changing credentials, suspending accounts, etc. The ISP contacts the RBL providers like sorbs (who is the absolute worst by the way) and requests to be delisted. Sorbs removes it and within an hour they receive another complaint about the same email, just from a different users, so they list you again.
The ISP contacts them and requests a delist and explaining that it's the same mail and the issue is resolved. Sorbs then responds with "No, you must remain on the blacklist for 5 days as there has been 2 reported complaints" No matter what the ISP does, these guys will not remove you. Once the 5 days is up, you request another delisting and they remove you, only to get listed again in few days time because the whole process starts again with another user and another email.
In an ideal world, an ISP would have every single one of it's customers using an anti-virus solution and have the technical skills to not click on dodgy links or know to not run outdated java or flash. In reality, they don't even know what java or flash is. Unfortunately spam filters will also always be 1 step being spammers, the same as cops are always 1 step behind crime in general.
Now I personally am happy with the blacklisting, I can see their usability, what I disagree with is these RBL providers being a law unto themselves. We have had 1 RBL provider request we pay $50 in order for them to remove us, try and pass that cost onto the customer, they simply just pack up their stuff, move to another provider and we still have to foot the bill.
We had an incident 3 months ago, where a customer sent out a newsletter through an overseas company hosted on AWS, and in his newsletter was a link to his website, which we host. Someone (or some sysadmin who intercepted the email) reported it to SpamHaus, and they decided to blacklist our server even though no actual email from this customer or his domain originated from the hosting server. To make matters worse, SpamHaus requested written confirmation from us that we have cancelled the customers services with us before they would remove our server from their blacklist.
Imagine being on the receiving end of that, imagine your hosting provider goes "Well you haven't broken any of our T&C's, you haven't broken our AUP, but we have to terminate your services because some "law unto themselves" RBL provider has told us that if we don't, they will not remove the blacklisting, meaning all other customers are affected. So Bye". Unfortunately this is the power the individuals who use these blacklists have give to these 1 man show rbl providers.
So yes, blacklists are great, but imagine your provider terminated your companies services because 1 individual in your company got a virus and unknowingly sent out spam. Who would you be upset with? The hosting company just trying to fight against these blacklists or the RBL provider who is demanding such drastic actions be taken.
The other side of it is companies who don't keep their wordpress or joomla sites up-to-date, they get malware uploaded and spam gets sent out. Once again, hosting providers and malware scanners will always be one step behind this as these scripts change on a daily basis and malware scanners rely on sys admins reporting these scripts to them. Who's responsibility is it to ensure that your site is up to date? The hosting provider or the customer? It's it's the customer, does it mean the hosting provider should automatically suspend your account if you haven't updated your site within 24 hours of a new version being released?
Another aspect regarding blacklists we have seen is what is known as cyber vandalism. For example, you go and sign up for a competitors news letter or service and once you get an email from them, you report it as spam to all these RBL lists and continue to do this week on week as the RBL providers mask who the email is sent too.
In doing this you know that you can affect your competitors business and hopefully their provider who ask them to move elsewhere. This is actually a very very common practice and is now even starting to make it's way in the ICANN domain dispute process. We have just been following an incident in the US whereby a person registered a similar domain to his competitor (ecommerce store) and then lodged an domain dispute. Long story short it resulted in his competitors domain being down and suspended for just over a week" Cyber vandalism is real and these RBL providers assist is making it very easy.
If you have made it this far into my post, please don't take it as though I am defending Web Africa. An ISP should do whatever they need to in order to get delisted. I am simply providing some insight into what is going on behind the scenes of these RBL providers.
So the question is, how does Google and AWS get around this, the short answer, they don't, they simply throw a lot of IP's at it and cycle them a lot.
For example, this is the SPF record for gmail.com:
"v=spf1 include:_netblocks.google.com include:_netblocks2.google.com include:_netblocks3.google.com ~all"
This is _netblocks.google.com - 173 312 ip addresses:
ip4:64.18.0.0/20
ip4:64.233.160.0/19
ip4:66.102.0.0/20
ip4:66.249.80.0/20
ip4:72.14.192.0/18
ip4:74.125.0.0/16
ip4:108.177.8.0/21
ip4:173.194.0.0/16
ip4:207.126.144.0/20
ip4:209.85.128.0/17
ip4:216.58.192.0/19
ip4:216.239.32.0/19
This is _netblocks2.google.com which is their ipv6 blocks - 1.536 Billion /64 IP ranges.
A /64 range has 18,446,744,073,709,551,616 IP's in it. So 18,446,744,073,709,551,616 x 1 563 000 0000 =
2001:4860:4000::/36
ip6:2404:6800:4000::/36
ip6:2607:f8b0:4000::/36
ip6:2800:3f0:4000::/36
ip6:2a00:1450:4000::/36
ip6:2c0f:fb50:4000::/36
Ignoring IPv6, they have allocated 173 000+ IP addresses for delivery of gmail.com, the moment 1 gets blacklisted they just cycle it out.
You might ask "Why can't the local ISP's just do it" - The short answer is because Afrinic is very strict in allocating out new IP's, this is why they still sitting on 40 million IP's. You have to provider network and infrastructure diagrams, documentation, peering agreements, etc and even after all of that, if they don't feel that you need them, they simply say no. Local ISP's have to justify every single IP address that they use when applying for more IP's from Afrinic. SO if you have a /16 - 65 000 IP's, you have to be able to justify every single one of them before afrinic will give you more.
Now we (domains.co.za) do this to some extent and within the limitations Afrinic imposes, but the real issue here is the power given to the RBL providers in the first place. I can strongly believe that they are one of the causes of the accelerated IPv4 space being used up so quickly.
-Dave
Please PM me your account details so I can assist.
