GdalRasterLayer and Ordnance Survey TIFF Files

..or any other raster files that use a colour palette.

So, having found out that you can use SharpMap's GdalRasterLayer to load and display virtual datatsets, I quickly set about trying to get it working with the tiff files provided by Ordnance Survey.

However, when I actually tried to view the data I was presented with a big black nothing.

Turns out there is...not a bug exactly, but more of an 'omission' in the GdalRasterLayer code. At the time of writing both the 0.9 release and the current trunk build of SharpMap contain a version of GdalRasterLayer that cannot handle images that use colour palettes. And yes, all the raster data from Ordnance Survey (or at least, all the tiff files) make use of them.

I eventually came across a fix in this post. However, the patch was based on an old version of GdalRasterLayer and the code had moved on significantly. After a bit of head scratching I gave in and tried my hand at porting across the changes.

Now, the code is not beautiful. It doesn't seem to be a perfect fit. Having said that, I did not want to change the original code too drastically and so it is what it is...and it even seems to work!

Anyone who is interested can pick up it up from here or download it from the original post here.

Processing Ordnance Survey Rasters with GDAL

Ordnance survey produce raster maps of the UK at several resolutions. They are provided as a set of tiles, geographically chunking the whole of the UK according to the British national grid. As with most things Ordnance survey, they seem to go out of their way to make them as hard to use in a standard GIS application as possible.

The set of raster tiles for the UK at 1:10k comes in at about 26GB. A whopping 10,000+ tiles. They do not come with any geo-referencing and are not made available in any format that includes this information (i.e Geotif). Instead, they are provided as tif, bmp or jpeg files with the option to download the geo-reference files (tfw etc) separately. To add insult to injury, it would seem that, in trying to save disk space, instead of a standard 3 band rgb tif file, what you actually get is a single band tif file with a colour palette. Its not so bad really, just another minor annoyance.

As I have mentioned before, we use a SharpMap based WPF control to render our data. This makes rendering OS tiles non-trivial for the following reasons:

1. There is a bug in GdalRasterLayer which prevents it from rendering tif files that make use of colour palettes
2. There is no dedicated layer type for rendering a raster tile index/set unless you want to serve them up over a WMS

Now, I was actually able to fix the problem in (1) and because of this (and with a better understanding of GDAL) I was able to render a raster tile set using nothing but a single GdalRasterLayer. However, maybe that's something I'll come back to in another post. For now, I want to talk about our original solution, which was to convert the tiles to ECW's using GDAL.

For 1:250k, 1:50k and 1:25k we found it was possible to create a single ECW for the whole of the UK. For 1:10k the huge number of tiles involved really put a strain on the memory. I don't know if it is achievable or not but one thing is for sure..it would take a very very long time (somewhere in the region of 1-2 weeks of solid processing). In the end, for the 1:10k stuff, we decided to create a single ECW for each top level square of the British national grid, giving us something like 50 ecw's in all. The time taken to process each of these regions varied between a few minutes and a few hours.

So, here are the steps involved to convert a set ordnance survey rasters into a single ECW:

1. Copy all the tiles at a particular resolution (say 1:250k) into their own directory. e.g C:\250k\
2. Download the geo-reference files from ordnance survey and copy them into the same directory as (1)
3. Using the fwtools shell, cd to the directory with the raster tiles and the geo-reference files in.
4. Build a virtual raster file(an xml file) of the entire set using:

   gdalbuildvrt 250k.vrt *.tif

5. Use gdal_translate on the virtual raster file to convert the entire set of tiles into a single ECW:

   gdal_translate -of ECW -co LARGE_OK=YES -expand rgb -a_srs EPSG:27700 250k.vrt 250k.ecw

Issues, workarounds and a handy bit of C#

Generally speaking, this worked well. The only real problem I faced was when dealing with the 10k national set. When trying to create a virtual raster of a top level grid square with a command like:

gdalbuildvrt SU.vrt SU*.tif

occasionally the command would fail. This would happen if the tiles in question had differing colour palettes or a differing number of bands.

The solution was to introduce an intermediate stage whereby the source rasters were 'normalised' to 3 band rgb tif files. I achieved this by using the following command on each file:

gdal_translate -co COMPRESS=DEFLATE -co TFW=YES -expand rgb HP60NE.tif Expanded\HP60NE.tif

Then, when it came to making the final ECW the -expand option had to be left out, due to the palette expansion having already happened in the previous stage.

I wrapped all of this up into a neat little C# program (mainly because I couldn't be bothered to work out the java script or batch file commands) that checked the integrity of the files, 'normalised' their palettes if necessary, built the vrt and finally, converted the raster tiles into a single ECW file, logging all the results into a separate file to record timings and errors.

The resulting tiles2ecw utility can be downloaded from here. You can either compile it into a console application or run it directly from the command line using the rather brillient cs-script.

To blog or not to blog...

I've never really understood blogging. I mean, blogs are great for finding things out but actually writing one...now why would you want to do that?

Well, i guess maybe its time to give something back. Or maybe, its just time to start recording some of the nerdy little gems that make life as a software engineer just that bit sweeter.

So here it is. A blog. Written by me. Who would have thought?

My intention is to start writing a series of posts dealing with Ordnance Survey mapping data and Open Source GIS (undoubtedly paying much attention to SharpMap along the way). Anyone who has to work with Ordnance Survey data may well understand the headache that can ensue from the vast amount of data processing and storage involved. Its often difficult to know where to start if your not buying into an enterprise GIS solution. So, if you don't want to splash out on proprietary software for dealing with all that GML...well you might just have come to the right place.