PSA: Afrihost ucebox.co.za has not been accepting mail properly for more than a month

MagicDude4Eva

MagicDude4Eva

Banned
Joined
Apr 2, 2008
Messages
6,477
Reaction score
40
Location
Jo'burg
If you happen to host with Afrihost and use their email infrastructure you might not be aware that their mail-infrastructure is suffering from major issues, preventing genuine mail from being consistently delivered.

Afrihost support is less than supportive and it has taken several escalation mails to get some answers (i.e. "This is only a recent error, maybe you should not cache your MX records as we fail over"). I don't think hosting customers will notice as only a portion of email gets delivered. From my own stats (for May) out of 55K emails, 17K bounced and 38K got delivered (April: out of 68K mails only 50K got delivered). A 30% rejection rate is extremely high and could severely affect your business (I have counted 770 domains hosted with Afrihost affected by this).

If you follow up with Afrihost, quote ticket #KDQ-689-72684 as a reference.
 
FWIW - no response from Afrihost. They did mention that they are loadbalancing, but all their 5 IPs report errors. Today alone more than 1000 mails remain undelivered.
 
Hi MagicDude4Eva

Sorry to hear about the issues experienced here.
I can see that our team have been actively replying to your ticket and will continue to assist where possible.
 
AfriGuy said:
Hi MagicDude4Eva

Sorry to hear about the issues experienced here.
I can see that our team have been actively replying to your ticket and will continue to assist where possible.
Click to expand...

To be honest the term "actively" is just a joke. The correspondence between your engineers and myself has not really gone anywhere. I provided detailed mail logs and explained that all 5 of your IPs are affected by that and was then told that this is a "recent" issue.

When I then started digging deeper, I found that your ucebox issue has been going on since at least April (we are not required to keep mail-logs for longer than 60 days) and I am pretty sure that the problem existed prior to April. I was then told that it must be our caching of MX records (which is also not the case). What really grinds me the most though is that your engineers knowingly lied (even your social media team on Twitter) that the issue just surfaced (where I now have inside information that your team has been struggling with your mail-infrastructure for the majority of this year).

This current incident is a deja-vue of a previous incident where Afrihost engineers misconfigured spambox which resulted in the rejection of close to 1m legitimate transactional mails. From the mail-history you will find that your engineers have not been transparent and skirted the issue. Despite my original mail including our IP list to possibly whitelist, I was told that this is not technically possible (I am not going to argue this, since any MTA-/mail-delivery engineer knows very well that this is easily possible with any MTA).

Further, your engineers made all sorts of excuses and assumptions - i.e. first there was mention that we open too many connections (not true, we open 2 SMTP connections to servers with the exception of Yahoo being 5 and Google being 20). Then the excuse came that our mail-throughput is too high - highly doubtful - we throttle at max 100msg/connection and with AH mails trickle through at such low rate that it should not be noticeable.

Lastly, your hosting division is incapable of handling normal postmaster requests (such as providing anyone with a sender limits).

Let's also not forget that on our platform we have a very small userbase using Afrihost mail-servers and yet we alone transit 50-70K mails a month with a rejection rate of 30%. None of those issues are really transparent to your clients as the sender will eventually bounce out. It needs to be said that if you offer mail-servers and host MTA-services you should at least get the bare basics right.

Even a small organisation as ours has a /27 range for dedicated mail (compared to your 5 load-balanced IP which are all broken).

TBH, I am the first one to acknowledge that technical issues happen and that we mess up in IT, but what is completely unacceptable to me is when people are dishonest and give me a run-a-round for no good reason. This then stinks of outright incompetence or lack of any integrity on your part. Sadly, I am forced to drag a simple support issue out onto Twitter and MyBB as otherwise your engineering team will have ZERO interest in properly and honestly responding to support tickets issued.

Let's not forget that you are not only impacting my customers but you are also impacting over 700 domains hosted with your business where you provide mediocre IT services.
 
bizy said:
Anyway of seeing which domains are affected?
Click to expand...

Sure - I have shared it on Pastebin via https://pastebin.com/baLVQ0w2. When restoring our archive logs I found that the issue dates back to at least March and the Pastebin includes 930 domains (with 39,094 recipients) affected. To me this number his high enough that any decent hosting provider would have taken immediate action and not be lacklustre about it for months.

FWIW: In our business we have received a good 100 tickets this month alone regarding Afrihost mail (which then eventually lead to investigating it) not being delivered. Since all of their 5 IPs are affected I would assume that not just our business is affected by sending transactional mails to Afrihost hosted customers.

Some stats (since March 2016)
- 39094 (!) recipients are affected
- 932 hosted domains affected (this is just traffic for users on our site)
- 132039 mails outbound of which only 90377 mails were delivered. 41,662 mails were not delivered due to Afrihost mail-server issues
- 459 mails bounced (some soft bounce which will eventually auto-subscribe after 3 hard-bounces in a month)
 
FWIW - Afrihost made some changes - no information has been provided by them what they have changed. I have not analysed it in detail but by the looks of it:
- they have now pinned one IP out of their pool against the MTA (spe.ucebox.co.za - 197.242.152.131).
- the other range (197.242.159.35-38) seems to have cycle through different DNS: spe.ucebox.co.za (197.242.159.35) / spe1.ucebox.co.za (197.242.159.35).
- a new domain "pseudo.ucebox.co.za" (IPs 197.242.144.5, 196.38.88.56, 207.44.188.90, 196.35.64.215) has surfaced with the error 4.4.1 (no answer from host)

Immediate bounces have now changed into transient delivery errors, but from our side we have hard bounced a large number of recipients and will not deliver mail to Afrihost anymore. We contacted most of our customers via SMS and asked them to get in touch with Afrihost directly or change their mail-provider. During the month of May we failed to deliver 21,338 order-confirmations to sellers/buyers and another 9,358 payment confirmations to sellers which resulted in delay of fulfilment and also snowballed in a large number of support tickets ("I don't get any mails from bidorbuy").

BTW: Some people have asked that I remove "sensitive" information such as IPs and domains from my post - this is really irrelevant as you are just a drill/dig/MXToolbox away from obtaining the info in the first place.
 
This is exactly the same issue I had for months when my domains were on AH servers.
Numerous tickets that blamed everyone but their services.

Got gatvol and moved all my clients and my domains away middle of last year - guess what, no more issues since.

This is exactly why i have ZERO respect and trust in AH.
 
Because reasons

If you happen to host with Afrihost and use their email infrastructure you might not be aware that their mail-infrastructure is suffering from major issues, preventing genuine mail from being consistently delivered.
Click to expand...
Let's count the weasel words: If, happen, might, aware, consistently - that's 5. So there's some kind of mail problem, but all you're telling about it is that you feel strongly, and you have statistics. However, you're not saying what the actual problem is.

If you want to demonstrate that you you are having trouble delivering mail and show that it is not your problem, you should post a log trail for an illustrative case showing the reason for non-delivery and the attempts that were made. This should preferably show your software's normal operation, not what happens when you force the queue.

Apart from all of this, what mail software are you using? (Will we laugh?) How long are your mail queues? What is the retry interval? Are your servers even able to keep up with processing your own primary queue, so that they have time to do retries, do you have available bandwidth, etc?
 
Last edited:
cfilorux said:
Let's count the weasel words: If, happen, might, aware, consistently - that's 5. So there's some kind of mail problem, but all you're telling about it is that you feel strongly, and you have statistics. However, you're not saying what the actual problem is.

If you want to demonstrate that you you are having trouble delivering mail and show that it is not your problem, you should post a log trail for an illustrative case showing the reason for non-delivery and the attempts that were made. This should preferably show your software's normal operation, not what happens when you force the queue.

Apart from all of this, what mail software are you using? (Will we laugh?) How long are your mail queues? What is the retry interval? Are your servers even able to keep up with processing your own primary queue, so that they have time to do retries, do you have available bandwidth, etc?
Click to expand...

Seriously? I have dumped the log-files to Afrihost. Our mail-infrastructure is world-class and I doubt many local companies deploy Port25 PowerMTA enterprise over a dedicated AfriNIC /27 range with a senderscore of 99 and flawless IP reputation.

On a slow day we push out 500K mails (about 20GB of mail-traffic) and we are capable of delivering over 1m mails/hour. All outbound mail is TLS encrypted and we apply SPF, DKIM and DMARC flawlessly over load-balanced virtual-MTA pools.

All of this is locally hosted on our own kit (a simple 8C / 12GB Xen-virtual) within our own cloud-infrastructure and architected/built and supported in-house.
 
cfilorux said:
.... you should post a log trail for an illustrative case showing the reason for non-delivery and the attempts that were made. ...
Click to expand...

Sound of crickets - 4036 errors today alone. For all default domains (anything but Yahoo and other high-volume recipients) we have a max of 2 connections with a max of 100 msg/connection - even a Raspberry with a Postfix could handle this. FWIW - the IPs below and "pseudo.ucebox.co.za" is new. Quite shoddy work and I am sure if you had a look at the DNS and reverse-DNS and MX records it would be an equal mess (not that it matters, as those issues have been the same since March):

Code: 
[pmta]# grep -i "ucebox" tran-2016-05-31-0000.csv  | wc -l
4036

tq,2016-05-31 19:57:05+0200,,,,,,4.4.1 (no answer from host),,pseudo.ucebox.co.za (196.38.88.56),,,,smtp,###.###.###.###,196.38.88.56,,,vmta-bidorbuy-##,,,afrihost.co.za/vmta-bidorbuy-##,
tq,2016-05-31 19:57:05+0200,,,,,,4.4.1 (no answer from host),,pseudo.ucebox.co.za (196.35.64.215),,,,smtp,###.###.###.###,196.35.64.215,,,vmta-bidorbuy-##,,,afrihost.co.za/vmta-bidorbuy-##,
tq,2016-05-31 19:57:05+0200,,,,,,4.4.1 (no answer from host),,pseudo.ucebox.co.za (207.44.188.90),,,,smtp,###.###.###.###,207.44.188.90,,,vmta-bidorbuy-##,,,afrihost.co.za/vmta-bidorbuy-##,
tq,2016-05-31 19:57:05+0200,,,,,,4.4.1 (no answer from host),,pseudo.ucebox.co.za (197.242.144.5),,,,smtp,###.###.###.###,197.242.144.5,,,vmta-bidorbuy-##,,,afrihost.co.za/vmta-bidorbuy-##,
tq,2016-05-31 19:57:29+0200,,,,,,4.3.2 (system not accepting network messages),smtp;421 spe3.ucebox.co.za: Too many concurrent SMTP connections; please try again later,mx868078.spe.ucebox.co.za (197.242.152.131)

And now I am waiting for cfilorux expert opinion on this, since he seems to be a deliverability expert.
 
MagicDude4Eva said:
Seriously? I have dumped the log-files to Afrihost. Our mail-infrastructure is world-class and I doubt many local companies deploy Port25 PowerMTA enterprise over a dedicated AfriNIC /27 range with a senderscore of 99 and flawless IP reputation.

On a slow day we push out 500K mails (about 20GB of mail-traffic) and we are capable of delivering over 1m mails/hour. All outbound mail is TLS encrypted and we apply SPF, DKIM and DMARC flawlessly over load-balanced virtual-MTA pools.

All of this is locally hosted on our own kit (a simple 8C / 12GB Xen-virtual) within our own cloud-infrastructure and architected/built and supported in-house.
Click to expand...

I have to ask the question, and seeing that you are having this problems, why haven't you rolled your own spam filter or bought an iron port or gone directly with SpamExpects.
 
Murmaider said:
I have to ask the question, and seeing that you are having this problems, why haven't you rolled your own spam filter or bought an iron port or gone directly with SpamExpects.
Click to expand...

We don't have spam issues. We simply can not deliver mail to Afrihost customers due to their mail-infrastructure not working. Those are our transactional mails going to users (i.e. payment-/order-notifications being sent to sellers) who host with Afrihost.

It is purely a deliverability issue where the receiving end (Afrihost) does not accept mails due to misconfiguration on their end. It is not specific to our MTA as it affects their whole infrastructure.

You can very well look up AH's issues on Senderbase - https://www.senderbase.org/lookup/?search_string=197.242.152.131
 
Last edited:
cee ess vee

Your system logs to .csv files ... this must be hard! How do you survive? The lines you provide say it tried a mail once over the span of for 24 seconds and hit a temporary failure in 5 parts. You're counting that as 5 errors. I think your problems may be less severe than you think. What I really don't understand his how you are sending a mail that does not have a message ID. If your mail has a message ID then your log would enable you to see the attempts at progress and eventual success or eventual failure of a particular mail.
 
Last edited:
cfilorux said:
Your system logs to .csv files ... this must be hard! How do you survive? The lines you provide say it tried a mail once over the span of for 24 seconds and hit a temporary failure in 5 parts. You're counting that as 5 errors. I think your problems may be less severe than you think. What I really don't understand his how you are sending a mail that does not have a message ID. If your mail has a message ID then your log would enable you to see the attempts at progress and eventual success or eventual failure of a particular mail.
Click to expand...

Sorry - not going to carry on with you, as you have seriously no understanding of how a high-volume MTA such as PowerMTA works. The file I dropped are just the accounting records. The delivery records are in a separate file and contain all other information. I have also anonymised records in the accounting file.

No idea why logging to a CSV is unmanageable? It is the most universal file-format which can simply be dropped to any Excel user (read= mostly non technical people) for analysis, dumped into a BI tool or as we do automatically generate daily delivery reports - since you questioned our deliverability capabilities, below is a snapshot of a campaign from yesterday with the cumulative - i.e. we can guarantee that within 30min a campaign will reach almost all customers:
delivery-stats.png

You contribute absolutely nothing to this thread - had you bothered to understand the error "4.4.1 (no answer from host)" you would have known that this is an error originating from Afrihost - i.e. the server has not responded. As it is a temporary problem, a MTA will retry and eventually fail with a 5.4.x error code. The above was an extract of a portion of today's errors. I have the insight of all mail-history to Afrihost since March and you don't. So please don't make any uneducated guesses. This thread is really for the purpose of ensuring that affected customers know about the current issues. You assume that the 5 failures listed above are for one recipient, but you are mistaken - those are various retry events in back-off/deferral mode trying to deliver (1m, 3x10m, 3x1h, 3x4h, 12h and then bounce and perm discard).

Afrihost reps have complained about my thread here and asked to remove/close it. I have asked them to be just as transparent and respond in public, which has not happened. They also refuse to provide honest answers in their support ticket which I opened over a week ago and have not responded in the last 2 days with any update.

If you have nothing constructive to say, I suggest you go back into lurk mode. Posts like yours are deeply embarrassing and lack any substance of knowledge.

FWIW - anyone can get any type of IP range from Afrinic - it is really not that difficult (our /27 range is just for mail - we actually have a /24 range and initially were offered a /22 range which we declined as it would have been a waste)
 
Last edited:
Yep, me's ignorant

Okay, so there's another file that contains detail information ... and the devil is not in that unspoken detail, but on the other end of the communications at the evil afrihost ucebox.co.za. I unreservedly accept your analysis, as I must since only you have the details of how a high-volume MTA such as PowerMTA works. I would have liked to hear more about that /27, but it's much less exciting now that it has shrunk in stature by growing to a more manageable /24
 
Last edited:
OCP said:
This is exactly the same issue I had for months when my domains were on AH servers.
Numerous tickets that blamed everyone but their services.

Got gatvol and moved all my clients and my domains away middle of last year - guess what, no more issues since.

This is exactly why i have ZERO respect and trust in AH.
Click to expand...


Hi there. Where did you move to?
 
You must log in or register to reply here.
Back
Top
Sign up to the MyBroadband newsletter
X