Fix 403 Forbidden on newly configured CentOS 6.5 httpd server (or 13.10 Ubuntu LAMP)

jnz_00006D65

Member
Joined
Jan 12, 2014
Messages
22
3 common causes:

1. File permissions
2. You have upgraded to Apache 2.4 / wrong Directory directives in httpd.conf
3. http serving directory has wrong SELinux security context set. <-- Most common in new distro's

I'll be going slow for those still learning the basics, so bear with me.

Code:
[user@host /]# [B]echo "Commands are in bold"[/B]
Commands are in bold

Quickfixes are in GREEN if you want to skip to fixing your thang.

Each of the 3 possible fixes are in separate posts below. (to prevent my browser from going tits up.)
 

jnz_00006D65

Member
Joined
Jan 12, 2014
Messages
22
1. File/Directory permissions

This would normally be the most obvious place to start looking. And the easiest to fix. So we'll cover this first.

A quick way to eliminate:

On CentOS 6.5, with Apache 2.2, the httpd server is running as user apache.
Yours might be different. On Ubuntu it usually is www-data and on earlier versions of CentOS or other distro's it might be httpd.

To see which, run the following command:
Code:
[user@host /]# [B]cat /etc/passwd[/B]
You will see an entry similar to this:
apache:x:48:48:Apache:/var/www:/sbin/nologin

Switch to httpd user. Substitute "apache" with your server user:
Code:
[root@host /]# [B]su apache -s /bin/bash[/B]
or if you're not root:
Code:
[user@host /]# [B]sudo su apache -s /bin/bash[/B]
[sudo] password for user:_

Now you're running as apache user. Change directory to your document root / public html, from where http files are served and see if you can open a file. We'll use the cat command.

Useless info: Cat Concatenate FILE(s), or standard input, to standard output. Mostly used to output file contents to screen, as we'll be doing.

It will either work and output the contents of the file index.php (or whichever file you choose)
Code:
bash-4.1$ [B]cd /var/www/html[/B]
bash-4.1$ [B]ls[/B]
index.php
bash-4.1$ [B]cat index.php[/B]
<?php
phpinfo();
?>

Or it won't:
Code:
bash-4.1$ [B]cd /var/www/html[/B]
bash-4.1$ [B]ls[/B]
index.php
bash-4.1$ [B]cat index.php[/B]
cat: index.php: [COLOR="#B22222"]Permission denied[/COLOR]

If it worked, then this is not your problem. Skip to possible cause number 2 below.

If it did not (you got the permissions error). Proceed to fixing it:


Quick fix(skip the yelping):
Recursively change ownership and file permissions on your public_html directory (www in my example) to owner: root, group: apache with full access to owner, read and execute permissions to group, read and execute permissions to other, like so:
Code:
[root@host www]# [B]cd ..[/B]
[root@host var]#[B] ls -lh | grep www[/B]
drwxr--r-x.  6 root root 4.0K Jan  7 21:24 www
[root@host var]# [B]chown -cR root:apache www/[/B]
changed ownership of `www/html/index.php' to root:apache
changed ownership of `www/html' to root:apache
...
...
[root@host var]# [B]chmod -cR 755 www/[/B]
mode of `www/' changed to 0755 (rwxr-xr-x)
mode of `www/html' changed to 0755 (rwxr-xr-x)
...
...

Note: all parent directories must also be accessible by your webserver. (rwxr-xr-x will do)


OK, On to the fix:

Please note, I'll be executing commands as root. You can switch to root with the su command and specifying root password, or sudo su, and specifying your own password. You can also just precede every command with sudo if you wish.

Get back to your own user space if you're still running as apache by typing exit.

From your public_html (mine is called www in the example), go up/back one directory and view permissions of your public_html directory:

Code:
[root@host www]# [B]cd ..[/B]
[root@host var]#[B] ls -lh | grep www[/B]
drwxr-xr-x.  6 root root 4.0K Jan  7 21:24 www

Useless Info: grep is a command-line utility for searching plain-text data sets for lines matching a regular expression. If you omitted the | grep www part, there would be a bunch of other directories and files also listed by ls (similar to dir in windows). So to keep things clean, the output of ls was piped (|) to grep to display only what we want. The ls -lh command means "list stuff in human readable format."

The output you see above from my server is acceptable, the default on many systems and will work fine. However, if your web application at any time needs to write to the file system, it will fail. Remember the server runs as user apache which is in apache group, and nowhere are permission given to this user or its group. The only reason your visitor's browser and the server are able to read and parse these files, are because of the permissions in the last triad: r-x. This gives read and execute permissions to all.

Useless Info: File System Permissions and information:
drwxr-xr-x. 6 root root 4.0K Jan 7 21:24 www
Basically we have one letter, a file descriptor, "d", indicating that this file is a directory. After that whe have a series of 3 triads, each of which denotes the permission granted to that specific user or group for the given file or directory. So the above means:
d: it's a directory
rwx: the owner of www, root, has read, write and execute permissions
r-x: the group root has read and execute permissions
r-x: others have read and execute permissions
6: Number of hard links
root root: owner and group
4.0K: File size in human readable format (-h option used with ls)
Jan 7 21:24: Time file was lat modified
www: File/Directory name
more here

To change this to something more acceptable, I would leave the owner as root, but change the group this directory belongs to, to the server user's group, also called apache. This group can be given write permissions on a certain directory somewhere within your public_html to allow your server-side scripts write access to it.

Code:
[root@host var]# [B]chown -cR root:apache www/[/B]
changed ownership of `www/html/index.php' to root:apache
changed ownership of `www/html' to root:apache
...
...

Check the result:
Code:
[root@host var]# [B]ls -lh | grep www[/B]
drwxr-x--x.  6 root apache 4.0K Jan  7 21:24 www

As you can see, the group this file/directory belongs to, has now been changed to apache.

Your permissions are obviously not the same as mine as shown in this example, hence the permission error, so this next step is where it actually gets fixed. We need to grant these groups and owners the appropriate permissions as detailed above. Make it so:

Code:
[root@host var]# [B]chmod -cR 755 www/[/B]
mode of `www/' changed to 0755 (rwxr-xr-x)
mode of `www/html' changed to 0755 (rwxr-xr-x)
...
...

In the previous example I used octal modes (755) to specify permissions. Given the same initial set of permissions, the following would do essentially exactly the same thing:

Code:
[root@host var]# [B]chmod -cR u+wrx,g+rx,o+rx www/[/B]
mode of `www/' changed to 0755 (rwxr-xr-x)
mode of `www/html' changed to 0755 (rwxr-xr-x)
...
...

Useless Info: chmod and chown. chmod changes access permissions of file system objects. The -R option is used to recursively run through specified directory and its subdirectories, changing all file and directory (folder) permissions to our specification. The -c option tells chmod to report all changes made to the screen (sort of like a verbose output). The -cR options perform exactly the same functions for chown as for chmod. chown is used to change the owner and group of a given object.

And that be that. On to the next...
 
Last edited:

jnz_00006D65

Member
Joined
Jan 12, 2014
Messages
22
2. You have upgraded to Apache 2.4 / wrong Directory directives in httpd.conf

If you take a look at your host error log, you will see something like this:

Code:
[root@host /]#[B] tail /srv/www/logs/error_log[/B]
...
[Tue Jan 01 04:20:00 1970] [error] [client 192.168.1.199] [COLOR="#B22222"]client denied by server configuration[/COLOR]: /var/www/html/
...

Useless Info: tail prints the last 10 lines of the specified file. Or the specified number of bytes using the option -c nnnn, where nnnn is the number of bytes. Similarly, there is a head command, which you know, does the opposite. Very useful for viewing large log files. As we did in the 2nd post, the output of tail can also be piped to grep for displaying only lines containing a certain string. More here.

Now there can be many causes for this error. If you experience any of the issues described here in post 2 or 3, this line will probably be in your error log.

If it is an Apache server directive issue, the most common cause lately, is a change in Apache 2.4 from 2.2, in how access control directives are specified in your virtual host conf file.

So, if you recently did a sudo apt-get dist-upgrade on your Ubuntu server box, you may run into this problem.

To fix this issue, you have to edit your virtual host conf file and change the access directive in the Directory block to reflect these new changes. Only you will know where your virtual host configuration files are.

In CentOS, it could be anywhere but usually virtual hosts are configured in a separate file caleld vhost.conf or similar contained in /etc/httpd/conf.d/.

On Ubuntu each vhost has a separate configuration file in /etc/apache2/sites-available and /etc/apache2/sites-enabled. The files are edited in /etc/apache2/sites-available and copied over to /etc/apache2/sites-enabled. Actually, the commands a2ensite and a2dissite are used to enable a vhost configured in /etc/apache2/sites-available, and to disable a site already enabled in /etc/apache2/sites-enabled. I have a strong suspicion, though, that these commands do nothing but copy the conf files this way and that. You still have to reload the Apache server after using them. Please correct me if I'm wrong. Moving on.

A typical bareboones vhost.conf (example from CentOS, Apache 2.2) file with ip-based hosts would look something like this:

<VirtualHost 192.168.1.242:80>
ServerAdmin me@bestdomain.co.za
DocumentRoot /srv/www/html_public/
ServerName pr0n.bestdomain.co.za
ErrorLog /srv/www/logs/error_log
TransferLog /srv/www/logs/access_log
<Directory /srv/www/html_public/>
AllowOverride None
Options None
Order allow,deny
Allow from all
</Directory>
</VirtualHost>

<VirtualHost 192.168.1.222:80>
ServerAdmin me@bestdomain.co.za
DocumentRoot /srv/www2/html_public/
ServerName join.bestdomain.co.za
ErrorLog /srv/www2/logs/error_log
TransferLog /srv/www2/logs/access_log
<Directory /srv/www2/html_public/>
AllowOverride None
Options None
Order allow,deny
Allow from all
</Directory>
</VirtualHost>

If this configuration are to be used with Apache 2.4, the same 403 Forbidden error would be thrown to anyone requesting a page on any of these hosts. As you can see the access directives in the Directory blocks are specific to Apache 2.2.

Using your favourite editor (vi, nano, etc..), make the changes to your conf file as detailed in quick fix below:


Quick fix [If you've upgraded from Apache 2.2 to 2.4]:

Access control directives in your virtual host conf file for Apache 2.2 used to look like this:
Code:
<Directory /var/www/html>
  Order allow,deny
  Allow from all
</Directory>

In Apache 2.4, this needs to change to:
Code:
<Directory /var/www/html>
  Require all granted
</Directory>

Reload the server after changes.

On CentOS:
Code:
[root@host /]# [B]service httpd reload[/B]
Reloading httpd:

On Ubuntu:
Code:
[root@host /]# [B]service apache reload[/B]
Reloading apache:

#2 FIN
 
Last edited:

jnz_00006D65

Member
Joined
Jan 12, 2014
Messages
22
3. http serving directory has wrong SELinux security context set

Sooo...none of the above, hey? Enter SELinux.

From the CentOS wiki:
Security-Enhanced Linux (SELinux) is a mandatory access control (MAC) security mechanism implemented in the kernel. SELinux was first introduced in CentOS 4 and significantly enhanced in CentOS 5 and 6. These enhancements mean that content varies as to how to approach SELinux over time to solve problems.

All you 1337 mobile h4x0rz probably know, even Android has a version of SELinux running.

Useless Info: SELinux runs in 3 basic modes.
Enforcing: Default. SELinux security policy enforced on the system
Permissive: SELinux is enabled but will not enforce the security policy. Only logs actions.
Disabled: SELinux is disabled. No policies enforced, no logging or nagging.

To view current mode:
Code:
[root@host var]# [B]sestatus[/B]
SELinux status:                 enabled
SELinuxfs mount:                /selinux
Current mode:                   enforcing
Mode from config file:          enforcing
Policy version:                 24
Policy from config file:        targeted

Viewing security context of files:
Code:
[root@host var]# [B]ls -Z[/B]
drwxr-xr-x. root root   system_u:object_r:var_t:s0       cache
drwxr-x--x. root apache system_u:object_r:httpd_sys_content_t:s0 www
...
...

Security context is given in the form user:role:type:mls

Setting security context of files using chcon,
usage: chcon [OPTION]... [-u USER] [-r ROLE] [-l RANGE] [-t TYPE] FILE...
Code:
[root@host var]# [B]chcon -v --type=httpd_sys_content_t /html/index.html[/B]
context of /html/index.html changed to user_u:object_r:httpd_sys_content_t

For more information on SELinux:
https://en.wikipedia.org/wiki/Security-Enhanced_Linux
http://wiki.centos.org/HowTos/SELinux
http://selinuxproject.org/page/Main_Page

So, as you configured your dev box, setting up new hosts and services, you most probably used some non-standard directories for hosting your web apps. In CentOS, the default public_html directory is at /var/www/html. This directory will have a SELinux context of httpd_sys_content_t set. SELinux enforces this rule so that only files in this security context can be served by the Apache web server which runs under the httpd_t type domain. But you might have configured your virtual hosts to use /srv/www/html_public or similar. SeLinux does not know about these directories and so they have default security context of type default_t, var_t, etc. And that's what we need to change. (We're only skimming the top here, see the links for more detailed information of how SELinux works)

And the fix:


Quick fix:
So first, how do we see the current SELinux security context of our directory? We can use ls command again, this time with the -Z option.

Code:
[root@host www]# [B]ls -Z | grep html[/B]
drwxr-xr-x. root apache unconfined_u:object_r:var_t:s0   html

The security context is in the form user:role:type:mls. Our type needs to be set to httpd_sys_content_t and user to system_u. (-Rv for recursive operation and verbose output)

We do this using the chcon command.
Code:
[root@host www]# [B]chcon -Rv --type=httpd_sys_content_t html/[/B]
changing security context of `html/'

And check if it worked:
Code:
[root@host www]# [B]ls -Z | grep html[/B]
drwxr-xr-x. root apache unconfined_u:object_r:httpd_sys_content_t:s0 html

OK, so it worked. Next is the user. You may or may not need to do this.
Code:
[root@host www]# chcon -Rv --user=system_u html/
changing security context of `html/'

And check again:
Code:
[root@host www]# ls -Z | grep html
drwxr-xr-x. root apache system_u:object_r:httpd_sys_content_t:s0 html

And reload Apache just for kicks: service httpd reload or on Ubuntu service apache2 reload

Test, HELLO WORLD!

And of course, the final step:

Profit.
 
Last edited:
Top