Nginx Virtual hosting

Nginx is by far my favourite Web server. After years going through various Apache flavors, Sun ONE, Microsoft IIS or Lighttpd twisted Perl like configuration, Nginx simplicity was a relief.

Last week, a friend of mine asked me for some help to setup a Web server for a school project. They’re teaching pupils HTML and CSS and she wanted each of them to have their own space to test what they’ve done. There was one problem: she doesn’t know anything about system administration. So I had to setup something where she would not touch the configuration or restart Nginx.

The idea was simple: I wanted to add as many sites as I wanted without adding a configuration file or restarting Nginx.

Challenge accepted.

Things going on, I was thinking further and further, testing more and more things to know where I could go.

  1. Dynamic vhosts for static sites
  2. Dynamic vhosts for Ruby on Rails apps with Passenger
  3. Dynamic vhosts with multiple backends
  4. Dynamic vhosts with SSL support
  5. Dynamic vhosts with their own error pages (and fallback)
  6. Dynamic vhosts with separated logging
  7. Dynamic vhosts with basic auth

I setup different configurations and found out some things were possible and some other were not because of how Nginx handles some .

1. The good: dynamic vhosts for static sites

The following configuration file is a very basic setup for static Web sites. There’s no backend here, no SSL, nothing fancy, the only goal was to check the setup works (and it does).

server {
    listen 80;
    server_name _;
    
    set $site_root /data/www/$host;
    
    location / {
        root $site_root;
    }
    
    error_log  /var/log/nginx/error.log  info;
    access_log  /var/log/nginx/access.log;
}
server_name _
The `_` server_name is a _catch all_, which means it will process every query. If you want to process subdomains of your domain name, you can use `*.platyp.us` instead of `_`
$site_root
The location where sites are hosted. Using the `$host` variable allows to set that configuration directive dynamically. So `foo.bar` is hosted in `/data/www/foo.bar/`.

Note that $site_root is a user defined variables we use to avoid repeating ourselves. $host is a Nginx variable.

2. The bad: dynamic vhosts for Ruby on Rails apps with Passenger

If you want to host Ruby on Rails Web sites, Phusion Passenger mod_rails is a nice and convenient solution. I’ve been using it since the very beggining to host Publify blogs and never had anything to say about it.

I’ve reused the static site configuration, adding the minimal Passenger setup for clarity.

server {
    listen 80;
    server_name _;
    
    set $site_root /data/www/$host/public;
    set $log_root $site_root/log;
    
    location / {
        root $site_root;
        passenger_enabled on;
    }
    
    error_log  $log_root/error.log  info;
    access_log  $log_root/access.log  main;
}

2 lines differ from the first example

$site_root _
Ruby on Rails `root` lies in the application `public` directory so this is where Nginx root goes.
passenger_enabled on;
Enables `mod_rails` for the current location

Bad news, it didn’t work.

Despite being called mod_rails, Passenger does not work like its PHP counterpart. It needs to know which sites it manages at startup time to launch a Ruby worker for each of them.

3. The ugly: dynamic vhosts with multiple backends

So far, we’ve been playing with site hosted locally. This time, we’ll configure Nginx as a reverse proxy to access various application servers relying on Lua module and Redis as a backend. I’ve found the Lua configuration on Dan Sosedoff blog

So you’ll need:

  • Lua.
  • Nginx compiled with Lua support.
  • A Redis server running somewhere. If you only have one machine running your frontend server, localhost is definitely the place to be.
server {
    listen 80;
    server_name _;
    
    set $site_root /data/www/$host;
    set $log_root $site_root/logs;
    
    location / {
    
        root $site_root;
        
        if (! -d $site_root) {
            set $backend "";
            rewrite_by_lua '
                -- load global route cache into current request scope
                -- by default vars are not shared between requests
                local routes = _G.routes

                if routes == nil then
                    routes = {}
                    ngx.log(ngx.ALERT, "Route cache is empty.")
                end

                local route = routes[ngx.var.http_host]
                if route == nil then
                    local redis = require "redis"
                    local client = redis.connect("localhost", 6379)
                    route = client:get(ngx.var.host)
                end

                -- fallback to redis for lookups
                if route ~= nil then
                    ngx.var.upstream = route
                    routes[ngx.var.http_host] = route
                    _G.routes = routes
                else
                    ngx.exit(ngx.HTTP_NOT_FOUND)
                end
            ';

            proxy_buffering off;
            proxy_redirect off;
            proxy_set_header X-Real-IP  $remote_addr;
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_pass http://$backend;
            
            set $log_root "/var/log/nginx";
        }
    }
    
    error_log  $log_root/error.log  info;
    access_log  $log_root/access.log  main;
}

This one is more complicated because of the Lua part.

if (! -d $site_root)
If the site is not a static one (or a Passenger powered Ruby on Rails site), there’s no reason the directory exists, so we know we’ll have to proxy.
set $backend “”
Sets an empty var for the backend. This var is later set by the Lua script using the cache fetched from Redis. If no backend is available, then we return a 404.
proxy_pass http://$backend
Pass the query to the needed proxy.
set $log_root “/var/log/nginx”
This is a fallback to store the logs into `/var/log/nginx` if `/data/www/$host` does not exist.

Now let’s test:

$ ./redis-cli
redis> set foo.platyp.us 192.168.0.1
OK
redis> set bar.platyp.us 192.168.0.2
OK
$ curl foo.platyp.us
    Hits 192.168.0.1
$ curl bar.platyp.us
    Hits 192.168.0.1
$ curl perry.platyp.us
    Error 404

I’m not a great fan of this solution so I call it ugly.

First, it breaks my catch all philosophy as you need to map every site with a Redis record. It means no more handy wildcard.

Second, I’m not fan of the if (! -d $site_root) which makes use of Nginx rewrite module. Nginx uses stat(2) which checks the nature of the file against the Posix macro S_ISDIR so we’re adding additional system calls for every http request. It’s not a big deal, but it’s better to know how it works.

The good news is you can handle non existing sites on Nginx level as you only use proxy_pass if and only if:

  1. There’s no static Web site with that name.
  2. There’s a backend to handle it.

Nginx documentation states that if is evil and I should use try_files instead. Unfortunately, try_files won’t work the way I want.

My first tests relied on a DNS based configuration. Every Web site had a $host.internal URI pointing to the right backend. I didn’t like it for at least 3 reasons:

  1. DNS adds lots of complexity as you need to create a new A (or AAAA) record for every Web site you host to know where it should forward the requests, and you don’t always control it. Most IT will refuse to dynamically add DNS entries every time you add a new site. Adding lines in /etc/hosts is not a solution either.

  2. DNS adds network latency. The Nginx LUA module allows to caches the results so you don’t have to query Redis every time you need to map a Web site with its backend.

4. The bad: dynamic vhosts with SSL support

Thanks to TLS SNI ((RFC 6066)[https://tools.ietf.org/html/rfc6066]), you can now manage multiple certificates on the same IP address. SNI does not work with old browsers, but it’s a great alternative the IPv4 shortage if you don’t care about the minority still using Internet Explorer 6.

In a dynamic SSL or not scope, the most obvious configuration was:

server {
    listen 80;
    server_name _;
    
    set $site_root /data/www/$host;
    set $ssl_root $site_root/ssl;
    
    if (-f $ssl_root/$host.pem) {
        return 301 https://$host$request_uri;
    }

    location / {
        root $site_root;
    }

    error_log  /var/log/nginx/error.log  info;
    access_log  /var/log/nginx/access.log;
}

server {
    listen 443;
    server_name _;
    
    set $site_root /data/www/$host;
    set $ssl_root $site_root/ssl;
    
    ssl  on;
    ssl_certificate  $ssl_root/$host.pem;
    ssl_certificate_key  $ssl_root/$host.key;
    
    location / {
        root $site_root;
    }
    
    error_log  /var/log/nginx/error.log  info;
    access_log  /var/log/nginx/access.log;
}

The SSL configuration is the minimal one. If you’re looking for a more complete one, I’ve written A Bulletproof Nginx SSL Configuration you can use.

Unfortunately, dynamic vhosts with SSL won’t work. Trying to start Nginx with this configuration fails with:

nginx: [emerg] BIO_new_file("$ssl_root/$host.pem;") failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('$ssl_root/$host.pem;','r') error:2006D080:BIO routines:BIO_new_file:no such file)
nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed

There are 2 reasons for this:

  1. Nginx needs to load the whole SSL server configuration at start time, so it throws an error when the certificate or key does not exist.
  2. The Nginx SSL configuration parser does not expand user defined variables so it needs a relative or absolute path.

5. The good: dynamic vhosts with their own error pages (and fallback)

Nginx default configuration provides a handy way to manage custom error pages in http, server, location and if in location contexts. The following example is designed for a custom 404 page but it can be easily extended to any 40x or 50x pages.

server {
    listen 80;
    server_name _;
    
    set $site_root /data/www/$host;
    
    location / {
        root $site_root;
    }
    
    error_page 404 =404 /404.html;

    location /404.html {
        root $site_root/error_files;
        internal;
        
        error_page 404 =404 @fallback_404;
    }
    
    location @fallback_404 {
        root /var/www/;
        try_files /404.html =404;
        internal;
    }
    
    error_log  /var/log/nginx/error.log  info;
    access_log  /var/log/nginx/access.log;
}
error_page 404 =404 /404.html;
Tells Nginx to use `/404.html` in case of `HTTP_NOT_FOUND` with a 404 return code.
location /404.html
What happens when hitting /404.html. This is where the fun beggins.
root $site_root/error_files;
Changes the location `root` to match the Web site `error_pages` directory.
internal;
Means it’s an internal redirection so the redirect is invisible client side.
error_page 404 =404 @fallback_404;
In the `/404.html` location, the error page is in the named location `@fallback_404` and returns a 404 http code.
location @fallback_404
This is the named location used to configure the fallback 404 page. In this location, the `root` is changed to `/var/www/` so it will read files from that path instead of `$site_root`
try_files /404.html =404;
Returns `/var/www/404.html` if it exists with a 404 http code.

The most obscure part is internal. According to Nginx documentation :

Specifies that a given location can only be used for internal requests. For external requests, the client error 404 (Not Found) is returned. Internal requests are the following:

  • requests redirected by the error_page, index, random_index, and try_files directives;
  • requests redirected by the “X-Accel-Redirect” response header field from an upstream server;
  • subrequests formed by the “include virtual” command of the ngx_http_ssi_module module and by the ngx_http_addition_module module directives;
  • requests changed by the rewrite directive.

And also:

There is a limit of 10 internal redirects per request to prevent request processing cycles that can occur in incorrect configurations. If this limit is reached, the error 500 (Internal Server Error) is returned. In such cases, the “rewrite or internal redirection cycle” message can be seen in the error log.

6. The bad: dynamic vhosts with separated logging

Last thing I’ve tried to do was to dynamically separate the logs by server. I thought it would be interesting to let the users access their logs for debugging or processing purpose.

Let’s improve the first configuration.

server {
    listen 80;
    server_name _;
    
    set $site_root /data/www/$host;
    set $logging_root $site_root/logs;
    
    location / {
        root $site_root;
    }
    
    error_log  $logging_root/error.log  info;
    access_log $logging_root/access.log;
}

If this is your only server, Nginx won’t start. It won’t start because it needs to open the log files at startup, and the access_log and error_log options don’t expand variables. There’s a solution though, which is delegating the log processing to rsyslog or syslog-ng but that’s beyond what I wanted to talk about here.

7. The bad: dynamic vhosts with basic auth

server {
    listen 80;
    server_name _;
    
    set $site_root /data/www/$host;
    
    location / {
        root $site_root;
        
        if (-f $site_root/.htpasswd) {
            auth_basic "Restricted";
            auth_basic_user_file $site_root/.htpasswd;
        }
    }
    
    error_log  /var/log/nginx/error.log  info;
    access_log  /var/log/nginx/access.log;
}

Forget about it, it won’t work.

$ nginx -t
nginx: [emerg] "auth_basic" directive is not allowed here in /usr/local/etc/nginx/nginx.conf:124
nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed

The reason why it fails is because if is not part of the general configuration module as one should believe. if is part of the rewrite module and auth_basic is another module. That’s one of the reason why the Nginx community thinks if is evil.

I guess I’m done, or almost.

If I’ve been missing something or there’s way to do one of the things I’ve failed at with my “never configure, never restart” philosophy, please drop me an email frederic@t37.net, I’ll be happy to update the article.

There’s one more thing I’ve been playing with part of the week-end, and I hope it will stay buried from the man knowledge until the end of time. Any real work implementation of what I’ve been testing would certainly result in opening a door to our world for the Great Old Ones. I’ve started to implement Apache .htaccess to Nginx using Lua.

Perry the Platypus wants you to subscribe now! Even if you don't visit my site on a regular basis, you can get the latest posts delivered to you for free via Email: