SpatiaLite data provider now in SharpMap trunk

It was nice to see that sussing out how to use those spatial indexes in SpatiaLite was not in vein. A post on bill dollins site prompted me to take a look at the recent changes to the SharpMap trunk, which now includes bills excellent SpatiaLite data provider complete with spatial index support.

Support for Labels and GDAL Rasters in SharpMap.Wpf

Having finally got around to doing some more work on SharpMap.Wpf I have at last managed to implement generic versions of both the LabelLayer and GdalRasterLayer. As with VectorLayer, a large body of the code is the same as the GDI+ specific versions however they both derive from the new SharpMap.Wpf.Layers.Layer base class which uses an IRenderContext in its overridable Render method instead of a GDI+ graphics object.

There are still a few minor issues to iron out. I haven't gotten around to supporting label rotation yet (although this should not be very difficult) and GdalRasterLayer does not yet do anything with the TransparentColor property. Still, nearly there! Just need a better demo project now.

Some screenshots below:

SharpMap.Wpf

I've been wanting to write a WPF renderer for SharpMap ever since I started using version 0.9 about a year ago. Sure, SharpMap v2 has a few early WPF implementations, but its been in beta for quite some time now.

Our existing WPF/SharpMap solution involves using the plain old GDI+ stuff and blatting the entire map into a WPF image using Interop Bitmap. It does the job quite well, so why bother doing anything more?

Well, mainly cos I wanted to! but also because I like the idea of having true WPF styles and themes. The improved brushes in WPF mean that there is scope for some really pretty maps. Also, the idea of using several WPF images to host transparent layers and animated overlays is quite appealing.

So far, I've got a WPF map renderer up and running for vector layers only. I'm currently working on WPFing the LabelLayer and the GdalRasterLayer. Maybe, when its getting to a semi-finished state i'll try and host it on CodePlex. I guess it all depends if anyone is still interested in 0.9 by then!? A few interesting points on the current state of play:

• At first I started to write a dedicated WpfVectorLayer and a corresponding WpfVectorRenderer, but I just couldn't live with the code smell. There was too much redundant duplication. In the end, I refactored the VectorLayer into a new generic class with a templatable style and introduced a new Render method that used an IRenderContext instead of a Graphics object. I now have two concrete implementations of IRenderContext, one for GDI+ and one for WPF. To this end, you can declare concrete instances of the generic VectorLayer that either use the standard VectorStyle or the new WpfVectorStyle, depending upon the type of renderer you wish to use.


• Due to the departure from the original design, all the new stuff is in a separate namespace. The 'legacy' VectorLayer has been left unchanged (yeah, so I've still ended up duplicating code, but at least moving forward you need only implement an IRenderContext)
  


• The WPF renderer uses the stream geometry classes to draw straight to a WPF DrawingContext. In addition to this, it provides methods to render the entire map, or just a single layer, to a Visual or a BitmapSource.

Here is my SharpMap.Wpf 'hello world':

GdalRasterLayer and Ordnance Survey TIFF Files

..or any other raster files that use a colour palette.

So, having found out that you can use SharpMap's GdalRasterLayer to load and display virtual datatsets, I quickly set about trying to get it working with the tiff files provided by Ordnance Survey.

However, when I actually tried to view the data I was presented with a big black nothing.

Turns out there is...not a bug exactly, but more of an 'omission' in the GdalRasterLayer code. At the time of writing both the 0.9 release and the current trunk build of SharpMap contain a version of GdalRasterLayer that cannot handle images that use colour palettes. And yes, all the raster data from Ordnance Survey (or at least, all the tiff files) make use of them.

I eventually came across a fix in this post. However, the patch was based on an old version of GdalRasterLayer and the code had moved on significantly. After a bit of head scratching I gave in and tried my hand at porting across the changes.

Now, the code is not beautiful. It doesn't seem to be a perfect fit. Having said that, I did not want to change the original code too drastically and so it is what it is...and it even seems to work!

Anyone who is interested can pick up it up from here or download it from the original post here.

Raster index datasets in SharpMap

In my previous post I eluded to the fact that it was possible to use SharpMap 'out of the box' to render raster index datasets (tiles). About a year ago I remember trying to get SharpMap to do just that and came to the conclusion that unless i wanted to use some kind of web based tile server, I would have to invest a lot of time creating a new type of layer, perhaps based loosely on the TiledWmsLayer, but specifically for loading large raster tile datasets from disk.

Turns out I didnt need to do any of that.

GdalRasterLayer supports many different raster formats. One that I was unfamiliar with until recently was its own virtual dataset format. This is basically an xml file that, among other things, can pull together several raster files and treat them as a single dataset. You can build one of these very easily by doing something like:

gdalbuildvrt rasterIndex.vrt *.tif

Now, because vrt files are just another format supported by gdal, you can load them straight in using the GdalRasterLayer like so:

GdalRasterLayer layer = new GdalRasterLayer("MyRasterIndex", "rasterIndex.vrt");

And because the GdalRasterLayer is pretty smart at what it does, it will only read the pixels that it needs, meaning it's pretty efficient in terms of its speed and memory usage.

Adventures with SpatiaLite

A while ago I came across SpatiaLite. Its particularly interesting to me as we (the company I work for) need to deploy vast amounts of mapping data on machines where it is not always possible to connect to a dedicated mapping server. For us, this means installing PostGIS onto every client and restoring 20+GB of spatial data onto each one.

I wanted to use SpatiaLite instead of PostGIS. That way, deploying the mapping data onto the client would become as simple as copying across a flat file. It sounded like the perfect tool for the job. However, as with all things, nothing is ever easy. Here are a couple of the problems I faced:

1. We deal mainly with mastermap data and use a proprietary database loader to upload it to PostGIS. This loader does not support SpatiaLite so I had to figure out how I was going to get it into the SpatiaLite format.



2. We use the SharpMap library to render our mapping data, which has extensions to support PostGIS. I would have to find or write a custom data provider if I wanted to use SpatiaLite instead.

Converting Ordnance Survey Mastermap data to SpatiaLite

First the easy (but less than satisfactory) way...

As we already had our data in PostGIS, I was able to use the latest version of fwtools to convert the existing database to SpatiaLite. This was surprisingly easy once i'd figured out all the arguments. It preserved our existing schema and was reasonably efficient. The command I used can be seen below:

ogr2ogr -f SQLite -dsco "SPATIALITE=yes" london.sqlite PG:"host=localhost user=postgres dbname=london password=postgres" mastermap

I had three main issues with this approach. Firstly, the latest version of fwtools (2.4.2) is not linked against libspatialite, meaning I was unable to create the spatial index from the command line and had to use the spatialite tools to do this as another manual step. Secondly, and more importantly, I had no way to apply change only updates (COU's) to the SpatiaLite database. I would have to apply them to PostGIS and then re-convert the whole lot. Thirdly, its just plain painful. encoding/uploading the data to PostGIS can take a long time. Converting it to SpatiaLite as a secondary stage doubles the time involved. Wouldn't it be nice if you could go straight from the original mastermap GML to the SpatiaLite database and handle COU's as well...

A Better Approach...

I'd had my eye on an open source converter for a while. After several experiments I had managed to get it translating and uploading the mastermap GML to a PostGIS database according to our schema (the tool uses a schema recommended by Ordnance Survey but, sadly, very different to the one our proprietry tool uses). Due to the foresight of the author, all this involved in the end was tweaking the xsd and xslt files (Actually that's not quite true, i had to change the source a bit as well so that it created the PostGIS tables in a slightly different way..but that's by the by).

After playing around with the 'Ogr converter' option, I finally bit the bullet and decided to have a crack at extending the tool to output directly to SpatiaLite. A few hours later (and after much reading up) I had a working converter. It could handle bulk loads, COU's and spatial index creation. It was fast and, thanks to its use of xsd/xslt, very flexible. I must take my hat of to the author of this tool for making it so extensible. Great job.

Now, just how was I going to plug this into our rendering engine? Read on...

Rendering SpatiaLite data using SharpMap

Bill dollins posted a SpatiaLite data provider on his blog a while back. Just the job. However, when I came to put it all together I was left twiddling my thumbs. It was taking an absolute age to render. The problem was that SpatiaLite does not automatically use the spatial index. Instead, you have to build a special query to make use of it...and the data provider I'd nabbed from Bill's site didn't make any use of it.

I ended up making a few changes to allow the data provider to be constructed with information about the spatial index table (yes, its implemented as a table, not a true index) and then modified the spatial query to make use of it. The main change was to the GetBoxClause method which can be seen below:

 private string _SpatialIndex;

/// <summary>
/// Name of the spatial index table
/// </summary>
public string SpatialIndex
{
get { return _SpatialIndex; }
set { _SpatialIndex = value; }
}

private string GetBoxClause(SharpMap.Geometries.BoundingBox bbox)
{
if (!string.IsNullOrEmpty(SpatialIndex))
{
StringBuilder sql = new StringBuilder("ROWID IN ( ");     
sql.Append("SELECT pkid FROM ");
sql.Append(SpatialIndex);
sql.Append(" WHERE ");
sql.AppendFormat(SharpMap.Map.numberFormat_EnUS,
"xmin < {0} AND xmax > {1} AND ymin < {2} AND ymax > {3} )",
 bbox.Max.X, bbox.Min.X, bbox.Max.Y, bbox.Min.Y);

return sql.ToString();
}

string wkt = SharpMap.Converters.WellKnownText.GeometryToWKT.Write(LineFromBbox(bbox));
return "MBRIntersects(GeomFromText('" + wkt + "')," + _GeometryColumn + ")=1";        
}

After making the changes, everything performed blisteringly fast. Perhaps even slightly faster than it did with PostGIS. So there you have it. A zero configuration GIS database for storing mastermap thats as easy to deploy as copying a file. All thanks to SpatiaLite (and a bit of work here and there).