GIS & Urban Planning

I have been watching a very interesting video on the FOSS4G NA’s YouTube channel by Steve Bennett a software engineer from a company called Data 61 all about their creation and implementation of an opensource website for all geospatial data in Australia. I wasn’t aware of this website and decided to take a look at it, it’s called http://nationalmap.gov.au/. When the interface loads uses Bing as its default basemap.

It has a very impressive range of datasets available to view. A very useful feature is that where available you can download the data directly. The National Map does not host any data directly but ‘provides a federated interface to the diverse servers already operated by the various government agencies, running Geoserver, THREDDS Data Server, ArcGIS Server, CKAN, Socrata, and many more’.

To create the National Map they created a new library called ‘TerriaJS’ that makes it straightforward to build 3D web-based geospatial data explorers. The crucial point to make with TerriaJS is that it is entirely opensource and is already being used by a host of different companies both inside and outside Australia. The road that Steve and his colleagues have traversed has been long with a screenshot of his YouTube video shown below to give the timeline.

 

The search function works very well and is very responsive. I searched for and downloaded the Australian post code areas, it worked flawlessly. A screenshot of querying the post code layer is shown below.

Any query can be downloaded as either a CSV or JSON. One post code caught my attention for the sheer size of it at 1,203,979km². To put that in perspective that one post code is over twice the size of France or over 17 times the size of the Republic of Ireland.

The website has the ability to upload data. I uploaded a JSON file of Irish speed camera zones, it worked flawlessly as shown below.

I recently read Annekatrien Debien’s post on Digital Geography regarding extracting information from Sentinel-1, specifically ship data and I decided to follow the tutorial for an area of the South-West coast of Ireland.

The first step was to download a Sentinel-1 image from the Copernicus website. I downloaded an image captured on the 4th April 2016.

The next step was to open the image in the SNAP toolbox. Once it was opened, the next step was to calibrate the image.  You have to go to Radar –> Radiometric –> Calibrate as shown below.

 

To calibrate you have to select both bands VH and VV on the second tab that is displayed after you click calibrate and the parameter you need it Sigma0.

Once the calibration has run then a second product appears in the window.

Extracting Information on Ships:

The next step was to extract information on the ships.  The option is located at:

Go to Radar –> Feature Extraction –> Ocean Tools –> Ocean Object Detection

You have to make sure that the calibrated image is the one selected as shown.

 

The Sigma0_VH is the band to select. You then click run. It is quite memory and CPU intensive and took my computer 35 minutes to complete (it’s only a few months old with a lot of RAM so that may have helped). It is a very intensive process as you can see below.

The end result in the SNAP toolbox was as follows:

Individual Ships looks as follows:

As Anne has desribed on her blog, the next step was to located the log file (xml file) and import it into Excel and save as a CSV as shown below.

 

This was then imported into QGIS and a simple map was created with OSM as the background. The end result is shown below:

With a number of Irish data sets having been made available under the Creative Commons Attribution 4.0 license I thought it would be interesting to take a look at one of these, the Sites and Monuments Record (SMR) from the National Monuments Service. This falls under the Archaeological Survey of Ireland which is a unit of the National Monuments Service. It basically records any known monuments from mainly pre-dating AD 1700 and some from the post-AD 1700 period.

I very briefly queried this database online using the ESRI web-app which limits the number of visible records to less than 1000. The next step I did was to download each county’s shapefile individually (as the entire country cannot be downloaded together). Once these were downloaded as 27 different shapefiles the next task was to merge them into one. The easiest way to merged these was to use the Geospatial Data Abstraction Library (GDAL). The easiest and most convenient way to install this is to use the OSGeo4w installer that takes the hard work out of it. It is available from here. Once this is downloaded and installed (I chose the express install and only installed GDAL.

I had the 27 shapefiles in a folder on my desktop called ‘National Monuments’. I opened the OSGeo4w shell and changed the directory to the National Monuments folder and made a new directory within called ‘merged’. The following script was then run:

C:\>cd C:\Users\Donie\Desktop\National_Monuments

C:\Users\Donie\Desktop\National_Monuments>mkdir merged

C:\Users\Donie\Desktop\National_Monuments>for %f in (*.shp) do (
More? if not exist merged\merged.shp (
More? ogr2ogr -f "esri shapefile" merged\merged.shp %f) else (
More? ogr2ogr -f "esri shapefile" -update -append merged\merged.shp %f -nln Merged )
More? )

The result of the above was a merged shapefile of all 27 of the other shapefiles. As this was a huge shapefile with approximately 150,000 entries the next step was to create a PostGIS database and import the shapefile. Using pgAdmin III a postgres database was created called ‘National’ and a postgis extension was created using the following line of code:

create extension postgis;

The next step was to import the shapefile, this was completed using the PostGIS Shapefile Import/Export Manager. Only two items needed to be changed, the Spatial Reference System Identifier (SRID) was changed to 2157 (Irish Transverse Mercator) and the character encoding was changed to ‘LATIN1’.

The next step was to query the data to see what needed to be removed/edited. A simple SQL statement was run to select everything in the database:

SELECT * FROM merged;

From this 153,364 records were returned and it was clear that the column ‘classdesc’ was the column needed to find all data on ringforts. The data was then ordered using the following:

SELECT * FROM merged 
ORDER BY classdesc;

From examining the associated online data there are four types of ringforts that I need to export:

1. Ringfort – rath
2. Ringfort – cashel
3. Ringfort – unclassified
4. Enclosure

The easiest way to do this was to create a new table with just those records in it and export these as a new shapefile. The following SQL script was used to create the new table:

 

CREATE TABLE ringforts AS
SELECT classdesc
FROM merged 
WHERE classdesc IN ('Ringfort - rath', 'Ringfort - cashel', 'Enclosure', 'Ringfort - unclassified');

Once this new table was created it was exported to a GeoJSON file using the ogr2ogr. The following was the code used:

ogr2ogr -f "GeoJSON" mydata.json PG:"host=localhost user=postgres dbname=National password=xxxxxxxx" "ringforts"

This then created a GeoJson which was uploaded to CartoDB. An intensity map was created as shown below with some custom styling for the infowindows.

If you zoom to a larger scale (such as at county level) it gives a clear indication of the intensity of ringforts at that location.

https://pearoid.cartodb.com/viz/84875f8a-da75-11e5-84f4-0ecfd53eb7d3/public_map

In my next post I will calculate the townland in Ireland with the largest number of ringforts.