Cascading failure on network this morning

[The_Librarian]

Ever had a cascading failure?

I had my first. In hindsight, it was quite interesting.

What happened was that the DNS server on our DC stopped responding to queries etc. I googled for solutions, but none was fit for my case.

Restarted the server, then restarted the DNS server a couple of times. Eventually it got rid of what was stuck, and started to respond to queries once more (tested with nslookup)

Then I discovered that emails wasn't being delivered, and no incoming emails either. I tested with SMTPDiag, and Telnet, and the conclusion was that the firewall was blocking inbound/outbound SMTP



I then grabbed a spare PC, and installed Smoothwall on it. Then, on a hunch, before I swapped firewalls, I restarted the original firewall - and email transfers started to work.

Everything's back to normal now. Stress (and blood pressure) levels is also down.

But quite interesting, though.

My thoughts on this matter is probably some ne'er-do-well fondled some ports on our Smoothwall, forcefeeding the DNS server some grief which caused it to fell over and also blocked the ports.

Or something else happened - a transient power glitch, or faulty memory. We'll never know.

If the same issue happens again, I'll replace the Smoothwall with a new build and see how it goes then.

No need worrying about when it will happen again - as long as it happen when I'm back at work, and not during my leave...

[bubbatentoe]

Sounds like your average firewall failure.
DNS & email failed at the same time but you experienced it first with DNS due to the SMTP delay in delivering mails (retries).

sounds like you're having lotsa fun (so close to the holidays).
:-)