Saturday 26 January 2013

Roll your own OpenStreetMap server

This is an Ubuntu 12.04 machine intended to be used for rendering and serving OpenStreetMap tiles


You will need to install some packages first

sudo apt-get install python-software-properties
sudo add-apt-repository ppa:kakrueger/openstreetmap
sudo apt-get update
sudo apt-get install libapache2-mod-tile

For this I'm going to mount an additional drive.
The OpenStreetMap data will use at least 300GB for the database, ~24GB for the compressed osm data file. However you'll need a more space again to store the rendered tiles. The more the better but I recommend having no less than 500GB for the data drive.

To mount the data drive use the following:
sudo mount /dev/vdb /mnt/maps_data/

let also setup auto-mount at login. Add the following to /etc/fstab
/dev/vdb /mnt/maps_data auto defaults 0 0

Downloading the planet data:
This will be about 24GB.

wget -c http://planet.openstreetmap.org/planet/planet-latest.osm.bz2
or
wget -c http://planet.openstreetmap.org/pbf/planet-latest.osm.pbf

nominatim says .pbf is preferred format but doesn't extrapolate upon this assertion.

Setting up the Database:The data base storage location needs to change so that the data is stored on the large drive. The Easiest way to do this is to copy the existing data and change the config file. Make sure to retain the file permissions, the following should work.
data_directory = '/mnt/maps_data/postgresql/9.1/main'

Change the data_directory in /etc/postgresql/9.1/main/postgresql.conf
cp -pR /var/lib/postgresql /mnt/maps_data

We should also make some changes to the config at this point to tune the database. Change the following values

shared_buffers = 768MB
work_mem = 512MB 
maintenance_work_mem = 512MB
autovacuum = off
checkpoint_segments = 20

You need to make some changes to the kernel before these settings can take effect. Add the append the following to the end of /etc/sysctl.conf

kernel.shmmax=1073741824
kernel.shmall=262144

At this point you should restart the server for these changes to take effect
sudo reboot

Exporting the data: After restarting the server check the postgresql is up and running
sudo service postgresql status

If it just says "Running clusters:" then you need to start it:
sudo service postgresql start

The export process requires a lot of memory so give the server as much ram as is available.
This may not be enough so you should temporarily add a large swap space, about 60GB to be safe, we can delete it after the export.
sudo dd if=/dev/zero of=/mnt/swapfile.swap bs=1024 count=62914560
sudo chmod 600 /mnt/swapfile.swap
sudo mkswap /mnt/swapfile.swap
sudo swapon swapfile.swap

Now to export to the database. The --slim parameter specifies that it should store temporary data in the database reducing memory usage. The -c parameter specifies the amount of temporary data it can store in the database.

This is the command I used to export the data
osm2pgsql --slim -C 15000 planet-latest.osm.bz2

Note that the export process can take upwards of 15 days and if interrupted must be started again from scratch.

Rendering tiles: We need to make sure that the rendered tile are stored on the data drive.
You could do this by changing the config files if you can find out all the places this needs to be done. Alternatively you could create a link to the data drive in place of the normal storage location.
sudo ln -s /mnt/maps_data/default /var/lib/mod_tile/default
sudo chown -R www-data:www-data /var/lib/mod_tile/default
sudo mkdir /mnt/maps_data/default
sudo chown -R www-data:www-data /mnt/maps_data/default

Now we need to render some tiles. We'll pre-render the first few zoom levels just to test
render_list -a --socket=/var/run/renderd/renderd.sock -z 0 -Z 5 -n 4

-z is the minimum zoom level rendered
-Z is the maximum zoom level rendered
Note the -n parameter is number of treads to use, adjust accordingly, this makes a big difference.


Lets cross our fingers and give it a try: modify the following (which should have been created automatically)
vim /var/www/osm/slippymap.html

Such that the following line reflects the server setup.
var newLayer = new OpenLayers.Layer.OSM("Local Tiles", "http://myserver.address.com/osm/${z}/${x}/${y}.png", {numZoomLevels: 19});
var zoom=0;

Now visit http://myserver.address.com/osm/slippymap.html in your browser. You should see be able to see a map of the world and be able to zoom in and pan around within the first 6 zoom levels without much issue or lag beyond network lag. Zooming in past zoom five you should see a noticeable to severe delay before tile start showing. This is because the server must query the database to get the data for the region you're looking at and render the tile images on the fly.

let me know how you got on.
... continue reading!