sábado, 23 de febrero de 2013

Bulk importing into GeoServer

Some of the algorithms contained in SEXTANTE do not actually do any analysis, but they just perform some common tasks such as management ones. Calling this algorithms from SEXTANTE is interesting, because you can save time by automating those task through the SEXTANTE modeler or the batch processing interface.

We will see in this entry how to bulk import layers into GeoServer using SEXTANTE, and how to add a bit of processing along the way, to ensure that the layers we import have optimal characteristics to provide maximum performance when responding to requests.

To import a single vector layer, double click on the Import vector into GeoServer

Set the parameters of the GeoServer instance that you want to import to (and make sure GeoServer is running) , the workspace where you want to import it, and the layer to import. Click OK and the layer will be imported.

The workspace must exist. If you want to create a new one, you can use the Create workspace algorithm before importing.

Importing to GeoServer from SEXTANTE is easy but, what if you have to import all the layers in a folder? That might take time. However, using the SEXTANTE batch processing interface can make the task much easier.

Go to the toolbox and, instead of double clicking on the name of the algorithm name, select the Import vector into GeoServer algorithm, right-click on it and choose the Run as batch process option. The following dialog will be shown.

Each row represents an execution of the algorithm. That is, in this case it represents one layer being imported into GeoServer. We have to fill the table, adding as many rows as needed, to perform the bulk import.

Select the Layer to import field in the top row. That correponds to the layer to import in the first execution of the algorithm. Click on the button in the field and you will see the typical file selection dialog. Browse to the folder where all the files you want to import are found, and select them all. Not just one of them, but all of them.

When you click on OK, SEXTANTE will understand that you want to use each selected file for a different run of the algorithm, and will add the rows that are needed to accomodate them.

Filling the other rows has to be done manually, but if you want to use the same values for all the rows in a given column (you will probably be importing to the same GeoServer instance and the same workspace), you can use the following trick: enter the value in the first row and then double-click in the column name. That will cause the value to be copied to all the other rows automatically.

Once the table is filled, just click on OK and the process will import all your layers into GeoServer. Open your GeoServer admin page to check that they have been imported.

As it happens with all SEXTANTE algorithms, this one is available in the GIS Python console as well. We can get some automation by calling it from a script. As an example, the following code imports all the layers that are  loaded in the current QGIS project .

layers = QGisLayers.getVectorLayers()

for layer in layers:


    sextante.runalg("gspg:importvectorintogeoserver","http://localhost:8080/geoserver/rest","admin","geoserver", layer,"workspace")

Let's now import raster layers instead of vector ones, but adding some extra processing before importing. What we are going to do is to make sure that the raster layer has a good format (in our case GeoTiff) in terms of performance, and a correct structure (we are going to ensure that it is internally tiled and has overviews).

The first thing to do is to export the raster layer as a GeoTiff using GDAL, and using the corresponding modifiers to add inner tiles of a size of 1024 x 1024 pixels. We do this with the GDAL Translate algorithm and the options shown in the figure below:

After that, we will run the Build pyramids algorithm, and add 4 levels of overviews, using the following configuration.

The last step will be importing into GeoServer, which is trivial if we already know how to import vector layers,since the interface of the algorithm to import raster ones is almost identical.

To make things better, we can put these three steps in a simple model, using the SEXTANTE modeler:

The URL and workspace fields have been harcoded in this case, and only the store name is shown as option when running the model.

That is the most convenient way of running this workflow, since the intermediate layer should not be loaded into QGIS but just generated (we can disable this when running the Translate algorithm, but as part of the model that is just defined once, and we do not have to worry about getting unnecesary layers in the QGIS canvas.)

Since models are just like any other algorithm, they can be run on the batch processing interface, so can do a bulk import of raster layers as well, including the preparation of the corresponding files before importing.

sábado, 2 de febrero de 2013

Preparing a DEM and slope layer

In order to have more layers than can be used to create meaningful examples for the processes included in the OpenGeo Suite, I took the task of extending the Medford dataset that currently ships with the Suite and is used for most of the examples and use cases. In particular, I wanted to add a DEM and a slope layer, since the dataset has several vector layers, but lacks useful raster ones. I used quite a few SEXTANTE algorithms to do this, so I have decided to write down the process and publish it here as a case study.

The first thing I did was to look for good elevation data. The National Elevation Dataset was clearly my first choice. As it is divided in tiles, metadata shapefiles are provided, with polygons that represent the extent of each tile.

Data can be dowloaded from the National Map download website, selecting the region you want to download. The process is rather cumbersome (you have to give an email address and a download link is sent to you!!), but I ended up having two raster layers that covered the Medford area, as shown below.

These layers had two problems:

  • They covered an area that was too large for what I wanted (I was interested in a smaller region around the Medford city center, shown in brown in the image above)
  • They were in two different files. (The Medford limits fall into just one single raster layer, but, as I said, I wanted some extra area around it.

Both of them are easily solvable using SEXTANTE.

First, I create a rectangle defining the area that  want. To do it, I create a layer containing the bounding box of the layer with the limits of the Medford area, and then I buffer it, so as to have a raster layer that covers a bit more that the strictly necessary.

To calculate the bounding box , we can use the Polygon from layer extent algorithm

To buffer it, we have several alternatives. The GRASS v.buffer.distance algorithm has the option to leave square corners, which in this case it is interesting, since we are then going to use the resulting polygon to crop a raster layer, which has to be rectangular.

Here is the resulting bounding box obtained using the parameters shown above

With this layer that contains the bounding box of the raster layer I want to obtain, I can crop both of the raster layer that I downloaded, using the Clip Grid with Polygons algorithm.

Before clipping, the clipping vector layer and the raster layer to clip must be in the same CRS, since the algorithm assumes that. Originaly, they are not (the bounding box has the same CRS as the Medford limits layer, which is not the same as the downloaded DEM layers), although they appear to match, since QGIS is performing on-the-fly reprojecting before rendering, and also because both CRSs (EPSG 4326 and EPSG 4269) are rather "similar", so even without the on-the-fly reprojection everything would appear more or less "in place".

Reprojecting the clipping polygon can be done with the Reproject layer algorithm.

Now we can crop the raster layers.

Once the layers have been cropped, they can be merged using the Merge Raster Layers algorithm

A cellsize is needed for the merged layer. I used the same one of the input ones. You do not need to know it in advance before calling the algorithm. Just click on the button in the right-hand size of the text field and you will have a dialog to enter small mathematical formulas, and a list of frequently used values, among them the cellsizes and bounding coordinates of all available layers.

Note: You can save time merging first and then cropping, and you will avoid calling the Clip... algorith twice. However, if there are several layers to merge and they have a rather big size, you will end up with a large layer than it can later be difficult to process.

With that, we get the final DEM we want.

A slope layer can be computed with the Slope,Aspect,Curvature algorithm, but the DEM obtained in the last step is not suitable as input, since elevation values are in meters but cellsize is not expressed in meters (the layer uses a CRS with geographic coordinates). A new reprojection is needed. To reproject a raster layer, the GDAL Warp algorithm can be used. We reproject into a CRS with meters as units, so we can then correctly calculate the slope.

Here is the reprojected DEM.

With the new DEM, slope can now be computed.

The slope produced by the SAGA  Slope,Aspect,Curvature algorithm is expressed in radians, but degrees are a more practical and common unit. The Metric conversions algorithm will help us to do the conversion.

A similar conversion can be made using the raster calculator (whether the SAGA one or the GRASS one).
Converting to a different unit, such as percentage, can also be done with the calculator.

Reprojecting the converted slope layer back with the GDAL Warp algorithm, we get the final layer we wanted.

The reprojection processes have caused the final layer to contain data outside the bounding box we calculated in one of the first steps. This can be solved by clipping it again.

The process can be automated, as usual, using the modeler.

File conversion with SEXTANTE

All providers included in SEXTANTE can use and produce all data types supported by QGIS, even if the actual application doing the work does not support all those formats. SEXTANTE takes care of all the needed conversions, so the user doesn't have to worry about it.

This full support of many different formats can be used automate the conversion of a set of files into a given format. We will see in this post how to do it.

There are several modules specifically designed to convert/translate between formats. The GDAL and OGR providers contain modules for converting vector (ogr2ogr) and raster layer (translate) respectively. In the case of vector layers, there is also a quick trick that we can use, which does not require use any external application and does all the work directly within QGIS. The Create new layer with selected features algorithm outputs a new layer with just those features that are selected. However, if no feature is selected, it will take all of them (this is the default behaviour of SEXTANTE in case there is no selection, and it works like that for all providers), so saving a layer with no selection will create that same layer. We just have to use a different file extension, and we will get the same layer, but saved in a different file format.

The algorithm dialog is rather easy to understand, as it doesn't have much more that an input field and an output field.

Just select the layer to export and the destination file. The extension of that file will define the output file format.

This way of exporting a layer through SEXTANTE instead of using the "normal" QGIS elements (in this case the Save Selection as Vector File menu) does not make a great difference for a single layer, but in the case of having a large number of them, it will ease out work considerably. To profit from the productivity tools that are part SEXTANTE, we should run the algorithm not like we usually do, double clicking on its name, but using the batch processing interface. Right-click on the name of the algorithm and then select Execute as batch process. You will see the following dialog.

Each row represents an execution of the algorithm, so we have to add as many rows as layers we want to convert and then fill the table. It is even easier than that, because we can select  a table cell in the Input layer column, click on the edit button to bring up the file selection dialog and select a bunch of files, and SEXTANTE will take care of creating the needed rows to accomodate all the selected files.

The next thing to do is to fill the output names. Click on the upper cell in the Output layer... column and browse to the folder where you want your resulting files to be stored. We are going to convert all our files into GML files, so enter ".gml" as the destination filename (notice that the first character is a dot). Don't worry about that name, because we are going to let SEXTANTE fill the names for us. The only important thing is the extension. Click on OK and you will see a dialog like this.

Now select the values as they are shown in the above figure. This tells SEXTANTE to  fill not only the cell you have selected, but all the ones underneath, and adding in each case the filename in the Input layer field as suffix. Since the we entered en empty filename (only a file extension), the resulting names will be the same names as the input layer, but with a different extension.

The final table is ready to be executed.

Now click on OK and SEXTANTE will convert all your files.

Files generated by the batch processing interface are not loaded into QGIS, but you can go to the folder you entered to check that they have been correctly exported.