Virtualmin, Apache, and Nginx Reverse Proxy

I wanted to be able to setup a reverse proxy with nginx and apache but continue using Virtualmin GPL to manage my domains. The only reason I wanted a reverse proxy is that several of the domains I host utilize a very large amount of images. I wanted a efficient way to handle these images since apache uses the same process to handle everything for the domain user and thus its memory footprint continued to grow till the process was killed. Thus I had processes using 150+ MB of memory just to serve a picture.

Anyway, Virtualmin does not support this config out of box so I had to do a bit of rigging 🙂 I figure I would share how I went about doing this.

A bit of background. Normally with this setup, you would have nginx accessible by the world and apache only accessible by the local host. But if you plan on continue allowing Virtualmin to manage your domains, you can’t do this. Unless you want to manually edit your config files each time Virtualmin creates a domain. So instead, I had to leave both open to the world.

Installing/configuring nginx

Ok, so assuming that you have a fully functioning Virtualmin and Apache2 servers, let’s install nginx. I use Ubuntu 10.04 LTS under Linode and the nginx server available in it’s ppa is a bit old. So let’s get the latest version from nginx’s ppa:

sudo add-apt-repository ppa:nginx/stable 
sudo apt-get update 
sudo apt-get install nginx

Now create the file /etc/nginx/proxy.conf and add the following to it:

proxy_redirect          off;
proxy_set_header        Host            $  host;
proxy_set_header        X-Real-IP       $  remote_addr;
proxy_set_header        X-Forwarded-For $  proxy_add_x_forwarded_for;
client_max_body_size    10m;
client_body_buffer_size 128k;
proxy_connect_timeout   90;
proxy_send_timeout      90;
proxy_read_timeout      90;
proxy_buffers           32 4k;

Those are the standard settings. You may have to adjust the numbers to best fit your site’s needs.

Open /etc/nginx/nginx.conf and edit it to match the following (adjust to your server’s needs)

user www-data www-data;
worker_processes  2;

error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;

events {
worker_connections 1024;
# multi_accept on;
}

http {
include /etc/nginx/mime.types;

access_log /var/log/nginx/access.log;

sendfile on;
#tcp_nopush on;

#keepalive_timeout 0;
keepalive_timeout 65;
tcp_nodelay on;

gzip on;
gzip_disable “MSIE [1-6].(?!.*SV1)”;

include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}

I created/edited the default host file to handle any domains that do not have a specific config file:

/etc/nginx/sites-available/default

server {
     listen 12.34.56.78:80 default;
     server_name  _;
     access_log /var/log/nginx/default.access.log;
     error_log /var/log/nginx/default.error.log;
 
     location / {
         proxy_pass http://12.34.56.78:9091;
         include /etc/nginx/proxy.conf;
    }
 }

Now create a virtual host file. I created one for each of my domains and named them like domain.com.conf so that my Virtualmin script can automatically handle creating them (see below).

So for example, create /var/nginx/sites-available/mydomain.com.conf:

server {
     listen 12.34.56.78:80;
     server_name www.mydomain.com mydomain.com foo.mydomain.com bar.mydomain.com;
     access_log /var/log/virtualmin/mydomain.com_nginx_access_log;
     error_log  /var/log/virtualmin/mydomain.com_nginx_error_log;
     location / {
         proxy_pass http://12.34.56.78:9091;
         include /etc/nginx/proxy.conf;
     }
 
     location ~* ^.+.(jpe?g|gif|png|ico|css|zip|tgz|gz|rar|bz2|doc|xls|exe|pdf|ppt|txt|tar|mp3)$   {
         expires 30d;
         root /home/mydomainuser/public_html;
     }
 }

Enable your site:

sudo ln -s /etc/nginx/sites-available/mydomain.com.conf /etc/nginx/sites-enabled/mydomain.com.conf

Of course you will replace 12.34.56.78 with your server’s IP address (if you are using a host with multiple IPs, be sure to use the correct one for this host!). Also change mydomain.com and add any other server names to the server_name directive. Now for the proxy_pass, normally this would 127.0.0.1:9091 (9091 being the port we are going to configure apache to run on). But like I said earlier, I still want Virtualmin to be fully functional without me having to manually edit the apache config files after Virtualmin created them to force apache to listen to 127.0.0.1. So I am leaving both open to the world and thus apache will listen to the host’s IP address.

Test your configs with:

nginx -t -c /etc/nginx/nginx.conf

You should get a success message. If you get errors fix it before starting nginx.

Start/restart nginx:

sudo /etc/init.d/nginx restart

Configuring Apache

We need to now configure Apache to listen to port 9091 and to tell it to use the correct IP address in the logs and in the $ _SERVER var for PHP.

Open /etc/apache2/apache2.conf and change NameVirtualHost 12.34.56.78:80 to 12.34.56.78:9091

Open /etc/apache2/ports.conf and change Listen 80 to Listen 9091

Open all the config files in /etc/apache2/sites-available and change to

Install mod_rpaf so that apache knows the true IP address of the user accessing the site:

sudo apt-get install libapache2-mod-rpaf
sudo a2enmod rpaf

Open /etc/apache2/mods-available/rpaf.conf and add your IP address(es) to the RPAFproxy_ips directive so that it looks something like this:

RPAFenable On
RPAFsethostname On
RPAFproxy_ips 127.0.0.1 12.34.56.78

Its important to add your server’s IP address(es) to the RPAFproxy_ips directive as if you do not, $ _SERVER[‘REMOTE_ADDR’] will always be your server’s IP address which can be bad for scripts that rely on this (like php based firewalls).

Reload apache:

sudo /etc/init.d/apache2 restart

Configuring Virtualmin

Now we have to make some changes to Virtualmin and its config files.

First we need to edit all the existing servers to use the new apache port. So open each file in /etc/webmin/virtual-server/domains (each file represents a server) and change web_port=80 to web_port=9091.

Now, login to Virtualmin and go to Server Settings -> Server Templates -> click on your default template -> choose Apache website from the template section dropdown -> change Port number for virtual hosts to 9091 then click Save and Next.

Restart webmin:

sudo /etc/init.d/webmin restart

Choose Log file rotation from the template section dropdown, choose Log files below for the Additional files to rotate field. Add the following to the field’s textbox:

/var/log/virtualmin/$  {DOM}_nginx_access_log
/var/log/virtualmin/$  {DOM}_nginx_error_log

Note, that you will need to manually add the nginx log files to logrotate for existing domains. You can use Webmin -> System -> Log File Rotation to do so.

Automating Virtualmin for nginx

Now I didn’t want to manually create new virtual host files for nginx each time Virtualmin created a server. So, I created a little script to do it for me.

First, I created a template file /etc/nginx/sites-available/template.conf which has the following in it:

server {
     listen {SITE_IP}:80;
     server_name www.{DOM} {DOM};
     access_log /var/log/virtualmin/{DOM}_nginx_access_log;
     error_log  /var/log/virtualmin/{DOM}_nginx_error_log;
 
     location / {
         proxy_pass http://{SITE_IP}:9091;
         include /etc/nginx/proxy.conf;
     }
 
     location ~* ^.+.(jpe?g|gif|png|ico|css|zip|tgz|gz|rar|bz2|doc|xls|exe|pdf|ppt|txt|tar|mp3)$   {
         expires 30d;
         root {HOME}/public_html;
     }
 }

Now, I created a script that Virtualmin will run after it creates/deletes/modifies a server.

/usr/local/bin/virtualmin.sh

#!/bin/sh
 NGINX_CONF_FILE="/etc/nginx/sites-available/$  {VIRTUALSERVER_DOM}.conf "
 
 if [ "$  VIRTUALSERVER_ACTION" = "CREATE_DOMAIN" ]; then
 	if [ "$  {VIRTUALSERVER_WEB}" = "1" ];
 	then
 		cp /etc/nginx/sites-available/template.conf $  NGINX_CONF_FILE
 
 		perl -pi -e "s#{DOM}#$  VIRTUALSERVER_DOM#g" $  NGINX_CONF_FILE
 		perl -pi -e "s#{SITE_IP}#$  VIRTUALSERVER_IP#g" $  NGINX_CONF_FILE
 		perl -pi -e "s#{HOME}#$  VIRTUALSERVER_HOME#g" $  NGINX_CONF_FILE
 		ln -s $  NGINX_CONF_FILE /etc/nginx/sites-enabled/$  {VIRTUALSERVER_DOM}.conf
 		/etc/init.d/nginx reload
 	fi
 
 
 elif [ "$  VIRTUALSERVER_ACTION" = "DELETE_DOMAIN" ]; then
         if [ "$  {VIRTUALSERVER_WEB}" = "1" ];
         then
 		    rm /etc/nginx/sites-enabled/$  {VIRTUALSERVER_DOM}.conf
     		rm /etc/nginx/sites-available/$  {VIRTUALSERVER_DOM}.conf
 	    	rm /var/log/virtualmin/$  {VIRTUALSERVER_DOM}_nginx_*
             /etc/init.d/nginx reload
     	fi
 
 
 elif [ "$  VIRTUALSERVER_ACTION" = "MODIFY_DOMAIN" ]; then
         if [ "$  {VIRTUALSERVER_WEB}" = "1" ];
         then
     		if [ ! -f $  NGINX_CONF_FILE ]; then
  		        cp /etc/nginx/sites-available/template.conf $  NGINX_CONF_FILE
 	        	perl -pi -e "s#{DOM}#$  VIRTUALSERVER_DOM#g" $  NGINX_CONF_FILE
 		        perl -pi -e "s#{SITE_IP}#$  VIRTUALSERVER_IP#g" $  NGINX_CONF_FILE
         		perl -pi -e "s#{HOME}#$  VIRTUALSERVER_HOME#g" $  NGINX_CONF_FILE
 		        ln -s $  NGINX_CONF_FILE /etc/nginx/sites-enabled/$  {VIRTUALSERVER_DOM}.conf
 	    	fi
     	fi
 
 	    if [ "$  VIRTUALSERVER_DOM" != "$  VIRTUALSERVER_OLDSERVER_DOM" ]; then
 	        if [ "$  {VIRTUALSERVER_WEB}" = "1" ];
         	then
 			    OLD_NGINX_CONF_FILE=/etc/nginx/sites-available/$  {VIRTUALSERVER_OLDSERVER_DOM}.conf
 			    mv $  OLD_NGINX_CONF_FILE $  NGINX_CONF_FILE
 			    rm /etc/nginx/sites-enabled/$  {VIRTUALSERVER_OLDSERVER_DOM}.conf
         		perl -pi -e "s#$  VIRTUALSERVER_OLDSERVER_DOM#$  VIRTUALSERVER_DOM#g" $  NGINX_CONF_FILE
 	        	perl -pi -e "s#$  VIRTUALSERVER_OLDSERVER_IP#$  VIRTUALSERVER_IP#g" $  NGINX_CONF_FILE
 		        perl -pi -e "s#$  VIRTUALSERVER_OLDSERVER_HOME#$  VIRTUALSERVER_HOME#g" $  NGINX_CONF_FILE
 			    ln -s /etc/nginx/sites-available/$  {VIRTUALSERVER_DOM}.conf /etc/nginx/sites-enabled/$  {VIRTUALSERVER_DOM}.conf
 	    	fi
 	    fi
 
         if [ "$  {VIRTUALSERVER_WEB}" = "1" ];
         then
 	        /etc/init.d/nginx reload
 	    fi
 fi

Now go to Virtualmin -> System Settings -> Virtualmin Configuration -> choose Actions upon server and user creation from the category dropdown and add /usr/local/bin/virtualmin-postaction.sh to “Command to run after making changes to a server.” Now Virtualmin will automatically manage the nginx host file for you.

 

Leave a Reply