Nginx-Django Webfaction
November 14, 2009 by Creart · 6 Comments
Struggling with django trying to keep memory usage low in Webfaction, I tried it all
- DEBUG = False
- Caching entire the site
- Serving the static media with apache
basically all the tips I could find in webfaction forum, till I read some people were recommending to install nginx and it will make your life easier in regards to memory. So I g00gled for anyone having this kind of setup in webfaction and this great article came up. Now all memory problems are solved.
1) First we need to create a new app in the Webfaction control panel I’m naming this for reference wf-app you can name it anything you like. Choose from the drop down menu “new app listening on port” this will give you a custom port which you will use later.
2) We need to create directories in our webapp folder and install Django and Flup (needed for the FCGI support later) so go ahead and ssh into your account and do the following.
mkdir -p bin lib/python2.5
PYTHONPATH=lib/python2.5 easy_install-2.5 -d lib/python2.5 -s bin http://www.djangoproject.com/download/1.1.1/tarball/ flup
./bin/django-admin.py startproject your_project
3) Now we need to download and untar the latest stable versions of Nginx, Nginx_UploadProgress_Module and PCRE (for the embeded perl in Nginx). Updated added Open SSL.
wget ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-7.9.tar.gz
wget http://www.openssl.org/source/openssl-0.9.8m.tar.gz (if you want SSL)
wget http://wiki.nginx.org/images/8/83/Nginx_uploadprogress_module-0.5.tar.gz
tar -xzf nginx-0.7.61.tar.gz
tar -xzf pcre-7.9.tar.gz
tar -xzf openssl-0.9.8m.tar.gz
tar -xzf Nginx_uploadprogress_module-0.5.tar.gz
4) Now lets configure, make and install Nginx
Without SSL
./configure --prefix=/home/username/webapps/wf-app/pcre
cd ../nginx-0.7.61
./configure --prefix=/home/username/webapps/wf-app/ --with-pcre=/home/username/webapps/wf-app/pcre-7.9/ --add-module=/home/username/webapps/wf-app/nginx_uploadprogress_module/
With SSL
let’s say all went well and we have no errors.
make install
At this point we should have a working install of Nginx and Django.
Now we have some issues to worry about
- What happens if the Webfaction support team ever need to reboot the server?
- What happens if our Django App crashes?
We could solve this by adding a cron job to restart our stack every few minutes but because Nginx doesn’t spawn our FCGI Django process it gets quite complex, and we have no easy way to monitor the status.
The solution as suggested here is to use supervisor to control our server processes and then make sure that we have a cron job to check and restart supervisor.
1) Now go back to webfaction control panel and create a new app listening on port to reserve a port for Supervisor to run on, and make a note of the port number for later.
2) First we need to create a start script for Django so that supervisor can monitor it, use your favorite editor to create a file called runserver.py in your django project directory
with the following content:
import sys
import os
sys.path.append('/home/username/webapps/wf-app/your_project')
sys.path.append('/home/username/webapps/wf-app/lib/python2.5/django/bin')
sys.path.append('/home/username/webapps/wf-app/lib/python2.5/django')
if __name__ == '__main__':
from flup.server.fcgi_fork import WSGIServer
from django.core.handlers.wsgi import WSGIHandler
WSGIServer(WSGIHandler(),maxSpare=1, maxChildren=1, debug=False).run()
Make it executable
3) We also need to change our django settings.py file and change the ROOT_URLCONF to:
Because supervisor can only interact with non daemonized processes we need to make a minor change to our nginx.conf file and add the following line right at the begining:
4) Time to install supervisor
5) Then create a file called supervisord.conf in your home directory with the following content replace port_number with the number you’ve got when you created the supervisor app. Make sure to replace the values that apply to your environment.
[inet_http_server] ; inet (TCP) server setings
port=127.0.0.1:your_port_number ; (ip_address:port specifier, *:port for all iface)
username=<username> ; (default is no username (open server))
password=<password> ; (default is no password (open server))
[supervisord]
logfile=/home/your_username/logs/user/supervisor/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=20MB ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=4 ; (num of main logfile rotation backups;default 10)
loglevel=debug ; (log level;default info; others: debug,warn,trace)
pidfile=/home/your_username/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false ; (start in foreground if true;default false)
minfds=1024 ; (min. avail startup file descriptors;default 1024)
minprocs=200 ; (min. avail process descriptors;default 200)
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=http://127.0.0.1:your_port_number
username=<username>
password=<password>
[program:nginx]
command=/home/your_username/webapps/wf-app/sbin/nginx -c /home/your_username/webapps/wf-app/conf/nginx.conf
autostart=true
autorestart=true
redirect_stderr=true
exitcodes=0
[fcgi-program:your_project]
socket=unix:///home/your_username/webapps/wf-app/your_project/django.sock
process_name=%(program_name)s_%(process_num)02d
numprocs=1
command = python2.5 /home/your_username/webapps/wf-app/your_project/runserver.py
environment=DJANGO_SETTINGS_MODULE=settings
autostart=true
autorestart=true
redirect_stderr=true
Assuming that all values have been replaced properly we could now start supervisor – however we would still suffer from the problem of supervisor not being started on a reboot. Let’s fix that – create a file called start_supervisor.sh in your home directory with the following content:
NAME=supervisord
SUPERVISORD=/home/your_username/bin/supervisord
SUPERVISORCTL=/home/your_username/bin/supervisorctl
PIDFILE=/home/your_username/supervisord.pid
OPTS="-c /home/your_username/supervisord.conf"
PS=$NAME
TRUE=1
FALSE=0
test -x $SUPERVISORD || exit 0
export PATH="${PATH:+$PATH:}/usr/local/bin:/usr/sbin:/sbin:/home/your_username/bin:"
isRunning(){
pidof_daemon
PID=$?
if [ $PID -gt 0 ]; then
return 1
else
return 0
fi
}
pidof_daemon() {
PIDS=`pidof -x $PS` || true
[ -e $PIDFILE ] && PIDS2=`cat $PIDFILE`
for i in $PIDS; do
if [ "$i" = "$PIDS2" ]; then
return 1
fi
done
return 0
}
start () {
echo "Starting Supervisor daemon manager..."
isRunning
isAlive=$?
if [ "${isAlive}" -eq $TRUE ]; then
echo "Supervisor is already running."
else
$SUPERVISORD $OPTS || echo "Failed...!"
echo "OK"
fi
}
stop () {
echo "Stopping Supervisor daemon manager..."
$SUPERVISORCTL shutdown || echo "Failed...!"
echo "OK"
}
case "$1" in
start)
start
;;
stop)
stop
;;
restart|reload|force-reload)
stop
start
;;
esac
exit 0
We’ll use this script to start and stop supervisor so we need to make it executable.
Now we need to add it to a cron job to make sure that it runs every few minutes (the script checks that supervisor is running so won’t start multiple instances of it) this will allow us to be back up and running a few minutes after the server comes back up.
Assuming you know the basics of VIM add the following, this will run the script every 10 min.
You should now be running on supervisor and your site should be up.
If you’d like to be able to see that status of your processes then mount the supervisor app we created at the start on a subdomain such as status.mydomain.com and then browse to that url (you’ll need to enter the user and password that you put into the supervisor.conf file)
Supervisor Screenshot
supervisor
Here is how my nginx.conf looks like, and how to add multiple sites.
worker_processes 1;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
server {
listen your_custom_app_port_number;
server_name example.com;
location / {
fastcgi_pass unix:/home/your_username/webapps/wf-app/your_project/django.sock;
fastcgi_param CONTENT_TYPE $content_type;
fastcgi_param CONTENT_LENGTH $content_length;
fastcgi_param PATH_INFO $fastcgi_script_name;
fastcgi_param QUERY_STRING $query_string;
fastcgi_param REQUEST_METHOD $request_method;
fastcgi_param SERVER_NAME $server_name;
fastcgi_param SERVER_PORT $server_port;
fastcgi_param SERVER_PROTOCOL $server_protocol;
}
}
# To use SSL you need to create another <tt>listening port app</tt>
server {
listen custom_app_listening_port;
server_name domain.com;
ssl on;
# I created the folder cert under the app and put the certificate files there.
ssl_certificate /home/username/webapps/wf-app/cert/mycert.cert;
ssl_certificate_key /home/username/webapps/wf-app/cert/mycert.key;
ssl_session_timeout 5m;
#charset koi8-r;
#access_log logs/host.access.log main;
access_log /home/username/webapps/wf-app/logs/access.log;
error_log /home/username/webapps/wf-app/logs/error.log;
location / {
# root html;
# index index.html index.htm;
fastcgi_pass unix:/home/username/webapps/wf-app/your_project/django.sock;
fastcgi_param CONTENT_TYPE $content_type;
fastcgi_param CONTENT_LENGTH $content_length;
fastcgi_param PATH_INFO $fastcgi_script_name;
fastcgi_param QUERY_STRING $query_string;
fastcgi_param REQUEST_METHOD $request_method;
fastcgi_param SERVER_NAME $server_name;
fastcgi_param SERVER_PORT $server_port;
fastcgi_param SERVER_PROTOCOL $server_protocol;
}
}
# Setting up project2, project3, project4, etc..
# Make sure to add another [fcgi-program:you_project2] with the proper values in supervisor config file
# Also don't forget to add runserver.py with the proper values as well.
# server {
# listen your_custom_app2_port_number;
# server_name example2.com;
# location / {
# fastcgi_pass unix:/home/your_username/webapps/wf-app/you_project/django.sock;
# fastcgi_param CONTENT_TYPE $content_type;
# fastcgi_param CONTENT_LENGTH $content_length;
# fastcgi_param PATH_INFO $fastcgi_script_name;
# fastcgi_param QUERY_STRING $query_string;
# fastcgi_param REQUEST_METHOD $request_method;
# fastcgi_param SERVER_NAME $server_name;
# fastcgi_param SERVER_PORT $server_port;
# fastcgi_param SERVER_PROTOCOL $server_protocol;
# }
# }
}
Thanks Richard Cooper for this tutorial, great work!
Thanks for the great article! I’m curious about your setup. You say your memory problems have been solved, but how is the performance of your site with the fastcgi stuff going on?
Thanks – I have about 4 sites running on the shared 2 plan, one of them is a satchmo store, so far I haven’t got into any issues, the performance is great taking in consideration that those are low traffic sites, I’m also using Django file cache to speed up page loading. I hope that answer your question
Great tutorial.
I need to set up multiple sites. Now when I’ve done a similar thing under Apache/mod_wsgi I only have 1 webfaction app which all the sites live under. You seem to be suggesting that I need a separate ‘custom app listening on port’ for each site. Is this correct? Can’t they all share the same port and webfaction app as they will all be managed by the same nginx instance?
I haven’t tried that, I guess you could do it – However I would recommend to separate them (custom app listening on port), is going to be easier to track, upgrade and see how your apps are behaving. The amount of memory been used by each app is minimum with this setup.
OK. I assumed there would be more memory overhead that way like there would be with Apache/mod_wsgi. I’m misunderstanding some part of all this!
Sorry to get back to you so late… your memory usage will go down big time. I had satchmo installed using apache + mod_wsgi and it will use about 90 mb with the front end and about 140mb adding the admin. Now is using about 30 -50mb all together
Also I’m using SSL and for that purpose you need to create a separate app listening port in ngingx config file.