Web hosting server: Centos - need advice!

[Linux_newb]

Hi there,

I recently got a VPS with CENTOS 5.4 x86_64 and cPanel/WHM and got a company to manage the server for me.

The thing is, you have to open a ticket for the administrators and whatever you do, they do it for you... now I've asked them to do the following, but not sure what else I should/can do:

* Initial server setup
* DNS and nameserver setup
* Updated software to latest versions
* Security hardening on server
* Installed anti virus
* Changed SSH Port
* Tweaked server to avoid DDos attacks

What else do you suggest one do on the server to keep it secure and working nicely?

Any advice?

Thanks
Linux_Newb

 

[RSkeens]

Do you use APF / BFD or CSF / LFD as your firewall frontend? Is the server for one account / single site or for many?

 

[Linux_newb]

Using CSF - upgraded it to latest version too.. gonna run a few domains on the account..

 

[koffiejunkie]

The only way to avoid DDOS attacks is to not host any sites that, or sites belonging to companies that p1$$ of people. Good luck with that. On that topic, "Security hardening on server" is a pretty vague concept, and of little use if you put code on your server that's vulnerable to exploits.

 

[Linux_newb]

I hope this is not spamming... this is what they've done for the hardening:

The following is a list of software and configurations that we have now installed to secure, optimize and harden your server. The following software is not resource intensive, therefore you will not see any decrease in performance.

We've installed CHKRootKit, which is a program that looks for known signatures in trojaned system binaries, it basically detects if your system has been compomised. We've ran CHKRootKit on your system and the output returned clean.

We've installed Rootkit Hunter, which is scanning tool to find most types of exploits (backdoors, suspicious files, md5 hash comparisons, and is over 99% accurate in detecting such exploits. We've scanned your system with the Rootkit Hunter and it reported that your system is clean.

CSF Firewall has been installed.

Logwatch has been installed now. This program parses through your server's logs and reports to you via e-mail on a daily basis with tabulated information.

SIM (System Integrity Monitor) has been installed on your server now, this software checks all services 24x7 and restarts them if they are down. An e-mail is dispatched a downed service is detected and restarted.

Apache (HTTPD) web server has been optimized and secured. For extra http/php security, we can install mod_security if you would like, it is not installed by default because it can interfere with certain common functions. If you would like mod_security installed, just let us know and we'll be more than glad to install this for you. Additionally, suphp can be added for extra protection in php but suphp is less efficient and more restrictive than regular php mode, so this has to be taken into consideration before installing it. If you would like it installed, please let us know.

MySQL Server has been optimized to perform at it's best under the most common and standard environments.

System Configuration File host.conf has been secured and hardened to prevent DNS lookup poisoning and also provide protection against spoofs.

System Configuration File nsswitch.conf has been secured and hardened. We have also optimized it to perform DNS lookups more efficiently.

System Configuration File sysctl.conf has been secured and hardened to help prevent the TCP/IP stack from syn-flood attacks. It is also configured to prevet other various and similar network abuse.

All of your vulnerable directories (/tmp, /var/tmp, /dev/shm and /usr/local/apache/proxy) have been reviewed and are clean now.

/tmp and /var/tmp have been hardened and secured to prevent the execution of malicious scripts

The old archived logs files that have been rotated located in /var/log have been removed to free up space in the /var partition/directory.

MyTOP has been installed. This is an administrative console based tool for monitoring MySQL threads/processes and performance. You may have to type "mysqladmin create test" before running mytop.

We have setup a root login notification script and logger. This will send an e-mail to 'root' everytime someone logs into your server as root. Also, it will keep track of all logins in a history file located in /var/log/rootlogins

SPRI has been installed now. This program changes the priority of different processes in accordance to their level of importance. You should see at least a 5-20% decrease in the average load level of your server on average.

We have disabled the Mchat, Cgiecho, Cgiemail, Guestbook, Counter and Formmails from CPanel's system wide cgi-sys directory. The are the most commonly exploited scripts since they are in the same location on every CPanel server in the world.
!!IMPORTANT!!: If any of your users are using any of those programs, they will no longer work. If you want us to re-enable them, just let us know.

Unused programs have been disabled from the OS of your server. This reduces the chance of being compromised through software exploits on old or deprecated programs.

MultiTail has been installed and gives you the ability to tail (view realtime activity) multiple log files simultaneously.

PHPSysInfo has been installed. This is a GUI (graphical user interface) to your server's vital statistics. You can view it by going to http://0.0.0.0:2086/phpsysinfo/index.php
Replace 0.0.0.0 with your own server's IP Address. You will have to enter your root login information to gain access as it is protected under your root WHM login.

Telnet has been disabled to prevent insecure transmissions of data and passwords, SSH must be used instead of Telnet, and functions the same way.

SSH has been hardened by restricting the SSH Protocol to SSH 2. SSH will still function the same way, just more secure. If you would like your ssh port changed, or direct root login disabled, just let us know and we'll be more than glad to do this for you.

If you or your datacenter monitors your server by icmp/ping, please let us know.

Fileman (Filemanager developed by gossamer-threads.com) has been installed into WHM with root level permissions. This allows system root files to be edited in an emergency situation when SSH is not accessible. You can access Fileman by going to http://0.0.0.0:2086/fileman/fileman.cgi
Replace 0.0.0.0 with your own server's IP Address. You will have to enter your root login information to gain access as it is protected under your root WHM login.
!!IMPORTANT!! This simulates SSH access, treat it as such, do not use it unless you are familiar with SSH. Moreover, do not execute any commands you are not fluent with. As with SSH, damage can be done if Fileman is not used properly. If you are unfamiliar with SSH, do NOT use this program. It should be left in case of such an emergency.
Again, this file can only be accessed through WHM while being logged in as root.

Shell Fork Bomb/Memory Hog Protection has been enabled. Fork Bomb/Memory Hog protection will prevent users logged into a shell (ssh/telnet) from using up all the resources on the server and causing a crash.

Background Process Killer has been enabled to kill any of the following which are commonly recognized bad processes: BitchX, bnc, eggdrop, generic-sniffers, guardservices, ircd, psyBNC, ptlink and related services.

A warning message has been created for the SSH login welcome screen. Any user that logs into your server via SSH, will see a message stating 'SSH is for authorized users only and any unauthorized access will be reported to the law enforcement authorities'.

Your FTP server software has been upgraded and secured.

We've ran and simulated a basic password scan hack attempt, the results have been emailed to 'root' and a copy of the results have been saved on your server at /root/security/passwordscanner.output

Please open a separate ticket for the kernel to be checked. Also, if you would like any other security software installed, please let us know so we can review it.

* IMPORTANT * PLEASE READ * IMPORTANT *
Can we guarantee your system will be hacker proof?
No, nobody can! It is 100% impossible to make a server hacker proof, that's a fact. This is even more impossible when vulnerable freeware scripts are so commonly used (forums, bb's, guestbooks, formmails). Nevertheless, even multibillion dollar company's such as banks, government computers, credit card companies, have had server's compromised. We consider our security hardening procedures to make your system hacker resistant. The software we install secures the system without adversely effecting or hindering the normal operations of your server.

Please note that over 99% of hacks come from insecure php scripts. These insecurities in php scripts come from the programming code, and therefore there is absolutely no way search to find "all" insecure scripts.

Remote based hacks are extremely rare. If there is no weak passwords, and no insecure php scripts, you have an very rare chance of ever being hacked.

So as long as you and your users keep all of your scripts up to date, and remove any unused scripts and remove any insecure scripts, then the chances of being hacked through the most common method is greatly reduced. If you feel your users do not know how to check or are not responsible enough to keep their scripts secure, we can secure php by enabling safemode (and other similar restrictions for php such as openbasedir restriction, disabling of commonly exploited functions, phpsuexec, etc.), and this will make php much more secure. However, the downside to doing this is that it will also interfere with a lot of scripts that don't work under these restrictions. If you would like this done, just let us know and we'll be more than glad to.

[root@server ~]#

 

[koffiejunkie]

Linux_newb, skip most of it and read just bit from * IMPORTANT * onwards. I work for a big hosting company (who shall rename nameless, but let's just say we consider IBM a rival) and I know what all this means. The last bit has all the important stuff in.

You get three types of attackers:

1. Script kiddies. They generally don't know much, they find "hacker scripts" on dodgy websites, and execute it indiscriminately against a range of IP addresses. The spaghetti approach - throw everything against the wall and see what sticks. Their favourite method is password bruteforcing on ssh, smtp and pop3 mostly, and sometimes against specific control planels like webmin, CPanel and Plesk. Or admin portions of popular CMS systems - joomla, wordpres, drupal, etc.

Thing is, they are generally like the dog chasing the bus - if they ever got into your box, they wouldn't know what to do with it. Just for kicks, I left a fresh install of Debian with the password set to 'password' on-line. It took about two hours before I had two successful logins. Neither did anything. The one logged in as root, logged out, and continued to bruteforce me on the root account with other passwords. WTF? The other was of the next type.

Script kiddies who learned some code might try to deface your site. I don't see that ofen though.

2. Spammers/bot operators. They don't care about your data. They don't care about your website. They're after your bandwidth. They generally do one of two things:

a) bruteforce smtp and/or pop3, because that gives them an account to send spam with through your server, and make it look like you're the spammer.

b) execute targetted exploits against software on your server that's know to have flaws. Joomla, a lot of the time. The smarter ones will also go afer hand coded sites and look for sloppy coding to either inject addresses into a form and spam that way, abuse a script to fetch and launch a bot, or gain access the admin area of a site to upload and launch a bot. The bot can be used to send spam, relay (i.e. anonymize connections) or take part in all sorts of attacks.

3. The more sophisticated type, who tries to gain access to your application, mostly to your database, to steal credit card information or whatever else.

Of the three types, 1 and 2 are by far the most common. I've seen type 3 a handful of times in the last three years (although it's far more prevelant on our windows servers).

Your best defence against all three of these is always the following:

1. Strong, long passwords. Passwords that aren't based on a word or a name. Replacing letters with numbers ain't enough. p3n1s is not a strong password. Not long enough either Something like iX?uco\r-i3Y is a bit less likely to be guessed. And this is not just for your own passwords. This is especially true for your clients. If they have the ability to set their own passwords, check if CPanel gives you an option to enforce complexity requirements

2. For CMS systems - keep them up to date. And please don't ever do chmod 777 on everything. It's just sking for trouble. 7 is for directories and things files you need to be executable from a shell.

3. For custom written stuff, make sure you santise your input.

Now, a few notes on some of the other stuff they wrote. chkrootkit and rkhunter are useful, but really, they should be used to confirm what you already know. When they do what they're made to do, it's too late anyways.

MySQL optimisation - you can only really do this when you have a consistent load to profile your usage with. There isn't one configuration that works great for most people. There are middle-of-the-road configurations that work adequately for a variety of loads, at best.

SPRI sounds like a big placebo to me. I suspect the only reason it brings down load is because processes scheduled low on a very busy server will in effect never get executed. If your server runs out of steam, it's either bad configuration, bad code, or inadequate hardware. Figure out which and address accordingly.

Two things in particular horrified me:

PHPSysInfo has been installed. This is a GUI (graphical user interface) to your server's vital statistics. You can view it by going to http://0.0.0.0:2086/phpsysinfo/index.php
Replace 0.0.0.0 with your own server's IP Address. You will have to enter your root login information to gain access as it is protected under your root WHM login.

and

You can access Fileman by going to http://0.0.0.0:2086/fileman/fileman.cgi
Replace 0.0.0.0 with your own server's IP Address. You will have to enter your root login information to gain access as it is protected under your root WHM login.

Using your root loging for anything other than logging in as root, is a bad idea. Supplying your root credentials over plain text http is even worse. I'm not real familiar with WHM so I don't know if they can change the user, but have them at least set all these tools up with an SSL certificate, even a self signed one.

And have them disable root loging for ssh. Log in as a regular user, then type su - to become root.

 

[icyrus]

And have them disable root loging for ssh. Log in as a regular user, then type su - to become root.

Indeed, and if possible have them enable key-only login for ssh.

But most importantly, get a very good backup strategy implemented and make sure it is being adhered to.

 

[Linux_newb]

koffiejunkie,

Thank you very much for the detailed advice! If this was your box, would there be anything else you'll ask them to do? I've asked them to change the default port to something else, so if anyone scans port 22, they won't be able to login.

I've also asked them to install a few libraries (Curl, etc), ClamAV, etc.

Server is up and running and everything works great, but I would like to find out (besides security) if there is anything else that I should consider as part of server maintenance. Remember, if I want something done, I need to submit a ticket and they'll do it.

What is your view on mod_security and "secure php by enabling safemode (and other similar restrictions for php such as openbasedir restriction, disabling of commonly exploited functions, phpsuexec, etc.)" - I also read something about mod_evasive and another mod_something...

Anything else (in general) that should be done? I'm making sure all software and kernel is being kept up to date and will let them do a security check every now and then. Just wondered what else falls under regular server maintenance?

 

[Linux_newb]

Indeed, and if possible have them enable key-only login for ssh.

But most importantly, get a very good backup strategy implemented and make sure it is being adhered to.

That sounds like a plan too - well, want to make server is top notch before loading it with clients.. busy shopping for a reliable backup service - the disadvantage of not having RAID when you need it

 

[koffiejunkie]

If this was your box, would there be anything else you'll ask them to do? I've asked them to change the default port to something else, so if anyone scans port 22, they won't be able to login.
[snip]
What is your view on mod_security and "secure php by enabling safemode (and other similar restrictions for php such as openbasedir restriction, disabling of commonly exploited functions, phpsuexec, etc.)" - I also read something about mod_evasive and another mod_something...

I don't use mod_security or mod_evasive myself. And the handful of our clients who choose to use it are generally skilled enough to set it up themselves. mod_security is like a content filter in front of your applications. I can see the usefulness, but it's like a spam filter - it shoots up false positives from time to time too. If you don't have access to the logs (although it sounds like you do) it can be difficult to track down issues like that. This is really up to you.

That sounds like a plan too - well, want to make server is top notch before loading it with clients.. busy shopping for a reliable backup service - the disadvantage of not having RAID when you need it

Do they not offer RAID? If I was running my own web host business, I would not work with anything less than a RAID-1. I'll tell you why. If your drive fails, you now rely on them to rebuild your server, restore your data and make sure everything works as it should. If they don't offer this kind of services (we certainly don't, the libility and time you spend on it is just not worth it), and I assume they don't offer backup, you're going to spend a lot of time explaining to your clients why their site is down. On the other hand, if you have at least RAID-1, and one drive fails - too bad, replace it and get on with your life.

One thing to keep in mind. RAID does not equal backup. If someone accidentally delets something, or if your box gets compromised, you need backup.