Author: Donal

Python-Quick Graph

I’ve never really had cause to use Matplotlib before so over the weekend I took a quick look at it and how useful it actually is for creating quick graphs. I took the electricity connections dataset (used as a measure of new homes built, however rough) and created a simple line graph. Below is the code I used and the result.

import matplotlib.pyplot as plt

years1 = ['1975', '1976', '1977', '1978', '1979', '1980', '1981', '1982', '1983', '1984', '1985', '1986', '1987', '1988',
'1989', '1990', '1991', '1992', '1993', '1994', '1995', '1996', '1997', '1998', '1999', '2000', '2001', '2002',
'2003', '2004', '2005', '2006', '2007', '2008', '2009', '2010', '2011', '2012', '2013', '2014', '2015']

scores1 = [26892, 24000, 24548, 25444, 26544, 27785, 28917, 26798, 26138, 24944, 23948, 22680, 18450, 15654, 18068, 19539,
19652, 22464, 21391, 26863, 30575, 33725, 38842, 42349, 46512, 49812, 52602, 57695, 68819, 76954, 80957, 93419, 78027,
51724, 26420, 14602, 10480, 8488, 8301, 11016, 12666]


fig, ax = plt.subplots()
ax.plot(years1, scores1, marker='o')
ax.set(xlabel='Year', ylabel='Number of Connections')
plt.title("Ireland: Electricity Connections 1975-2015", fontsize=18)
ax.grid()
plt.xticks(years1, rotation=45)
plt.margins(0.01)
plt.show()

And here is the resulting graph:

Python Plot, Electricity Connections
ESB Connections-Ireland

The main strength I’ve seen from it is that it is very easily customisable as well as being a nice addition at various stages of a complex python script in order to error check a methodology in train.

Isochrones-Ireland and Australia

I was reading Topi Tjukanov’s fantastic post yesterday on how far you can drive in one hour from European capitals. This got me thinking, wouldn’t it be interesting to compare driving distance for somewhere in Ireland and somewhere in Australia. For Ireland I chose the geographic centre and for Australia I chose my fiancée’s home city of Perth. I used Digital Geography’s post on utilising the HERE API to generate isochrones for 2, 4 and 5.5 hours driving distances. Unless I’m mistaken, the API seems to max out at 5.5 hours (as measured in seconds) so that’s why 5.5 hours was my upper limit.

Interesting but not surprising to see that you can cover almost the entire of the island of Ireland in that time but it Australia it doesn’t look like much progress at all!

Ireland-Isochrone Map

Ireland-Isochrone Map

Perth-Isochrone Map

Perth-Isochrone Map

Ireland, A Country in Motion: Methodology

I promised late last year that I’d do a blog post explaining how I created the ‘Ireland in Motion’ commuting map. Well, this is that post!

The first thing to say is, that until the ’16 census results came out it wasn’t possible (as a member of the public) to create this type of map as the Central Statistics Office (CSO) just didn’t release the data. Before now (and is still the case) in order to access the full Place of Work, School or College data (POWSCAR) you must attend a training program and sign up to be an ‘Officer of Statistics’. The deciding factor for myself was that you have to be resident in Ireland, which I am currently not. You also have to be a ‘bona fide’ researcher.

So, imagine my delight when I found out that they were releasing an aggregated anonymised dataset for the entire country! The data is aggregated by electoral divisions (ED) and county level. The POWSCAR website where the data can be downloaded is located here. There are two important caveats when talking about this data, EDs where fewer than 10 persons commuted have been excluded and records where no work, school or college were able to be geocoded have been removed. Below is an extract from the CSO’s website showing the categories available.

RESIDENCE_ED_GUID Geographic Unique Identification (GUID) Code for origin Electoral Division (ED)
RESIDENCE_CSOED CSO ED code for origin ED
RESIDENCE_CSOED_LABEL Name of origin CSO ED
RESIDENCE_COUNTY County code for origin county
RESIDENCE_COUNTY_LABEL Name of origin county
POWSC_ED_GUID GUID for destination ED
POWSC_CSOED CSO ED code for destination ED
POWSC_CSOED_LABEL Name of destination ED
POWSC_COUNTY County code for destination county
POWSC_COUNTY_LABEL Name of destination county
COUNT Number of persons commuting

The downloaded zip file when extracted was a 42MB CSV file. CSVs are an ideal format because they are supported by a huge number of programs. I knew that for the type of map I was going to create that I wanted to use create straight lines between the centroids of each ED. The basic methodology I followed was as follows:

  1.  Download CSV, inspect and clean the data (remove any extraneous records).
  2. Download the ungeneralised shapefile of the EDs (available here).
  3. Use QGIS to create the polygon centroids of each ED.
  4. Use the VLOOKUP and concatenate functions in Excel to create well-known text linestrings for the commutes between each EDs.
  5. Use python to parse the CSV file and multiply each row by the number (count) of commutes between each ED. Each row represents one commute between two EDs.
  6. Load the CSV into QGIS and save as a shapefile.
  7. Use FME to load the shapefile file into a PostGIS database.
  8. Connect database to QGIS and create the map.

Detailed Methodology:

1 CSV:

The original number of commutes in the CSV was 2, 750, 239. The following records were removed:

A. The destination was within the same ED (478,884)

B. There was no fixed place of work (174,628)

C. Work/school from home (114,189)

D. Commute to Northern Ireland (9,336)

E. Commute overseas(!) (3,531) were removed.

 

This left the grand total of 1,969, 671 Commutes to be mapped.

2  Download Ungeneralised shapefile:

The ungeneralised shapefile was downloaded from here.

3 Use QGIS to Create the Polygon Centroids:

The centroids of each polygon was quickly calculated in QGIS.

4 Vlookup and Concatenate in Excel:

The attribute table of centroids was exported to Excel and the Vlookup and Concatenate functions were used to create the linestrings for individual commutes as shown below:

1001,1002,1,-6.92771,52.83721,-6.93919,52.83783,"LINESTRING (-6.92771 52.83721, -6.93919 52.83783)"

5 Python:

A simple python script was used to multiply each line string by the count, so that each individual commute would be represented by a separate line on the map.

6 QGIS-Load CSV:

The CSV file was quickly and easily loaded into QGIS and exported as a shapefile. A better method to do this would probably have been to use FME to load the CSV directly into PostGIS and that’s something I will bear in mind for the future.

7 FME Shapefile:

FME 2017 was used to load the shapefile to PostGIS, and a simple reproject was used to get the data into Irish Transverse Mercator (EPSG 2157).

8 Connect PostGIS to QGIS:

A PostGIS layer can be added in a few clicks from within QGIS. The advantage of using PostGIS is that it will load the 1.96 million lines a lot faster than a shapefile for example, shapefiles have their uses (widely supported for example) but they are an archaic format that will hopefully go the way of the Dodo (this is already happening with the support for Geopackage in QGIS 3 for example).

 

The above is a quick overview of how I carried out the data processing for the map. It’s remarkable that almost all the software used to create the map was open-source. I’d be curious to try and do it totally open-source (replace FME with OGR and Excel with LibreOffice Calc) but as I have a home use licence for FME and Office ’16 I decided to use those.

Ireland-A Country in Motion: 1.96 Million Commutes

I was inspired by Alasdair Rae’s excellent work here, here and here to attempt something similar for Ireland. Below is the fruit of my labour over the last number of weeks. It shows 1.96 million individual commutes from every electoral division in Ireland. Commutes within a single electoral division are excluded. The data comes from the Irish Central Statistics Office, OpenStreetMap and OSI opendata. I will post in detail in the coming days on the methodology I used for completing this. For now I hope you enjoy.

Here is a pdf with much higher resolution Ireland-A Country in Motion and the same pdf in Google drive.

I will also post a very high resolution PNG over the weekend that can be used to print a high quality image.

Ireland-A Country in Motion

Ireland-A Country in Motion

 

Global Human Settlement Layer-Cork

During the course of the last week I was reading the MacKinnon report (otherwise known by its succinct title of ‘Report of the Expert Advisory Group on Local Government Arrangements in Cork’). This is the report that address whether the administrative boundaries of Cork City Council should be changed. The report recommends a large expansion of the City Council’s boundary and I put together the below GIF to quickly illustrate why this is necessary. Hopefully the expansion goes ahead, at least the new proposed boundary makes ample provision for urban growth over the next number of decades.

Cork Urban Growth

Cork Urban Growth

 

Global Human Settlement Layer-Dublin

I’ve been taking a look recently at the Global Human Settlement Layer from the the European Commission. I’ve been working on a piece to do with the physical impact of the Celtic Tiger on Ireland, but in the meantime, I’ve put together a few fun GIFs using their built up data. This uses a combination of Landsat and Sentinel scenes for four epochs: 1975, 1990, 2000 and 2014. Below is a GIF showing the growth of Dublin during this time.

Dublin Urban Growth

Dublin Urban Growth

QGIS-Generating an Isochrone

Normally with somebody asks me to undertake a quick bit of analysis it usually involves, ‘Can I see all of x features that are with 2000m of this location?’. The other day I got asked: ‘Can you give me a spreadsheet showing all locations of x within a 12 minute drive of this location?’. I thought about it and I knew for a quick piece of analysis like this I didn’t need to/have time to setup pgRouting or ESRI’s ArcGIS API for Python.

I did a quick search and found the OSM Tools plugin for QGIS and after a quick visit to OpenRouteService to get an API key and I was in business. After I installed the plugin and entered by API key I only had to click a point on the map to generate my isochrone as shown below.

OpenRouteService QGIS Dialog

OpenRouteService QGIS Dialog

The results are shown below, they are temporary layers that can be permanently saved if necessary.

Lastly, in case QGIS is giving any hassle, the isochrone can be generated online and downloaded as a geojson, test shown below for a random location in Dublin.

Online Results

Online Results

Irish Census 2016

The CSO released the results and geometry for the small areas (the largest scale available) on the 20th of July 2017 (available here). I downloaded all the data and used FME 2017 (with my shiny new home use licence) to join the geometry and CSVs and write it to PostGIS which was then brought into QGIS, definitely the quickest way to display the data. I used Google Fonts and Adobe Kuler. Below is a quick map I put together of my home county.

Unpopulated Australia

I was looking at a map highlighted on Reddit the other day from the website mapsbynik. It showed the unpopulated areas of the United States. I was going to do a similar exercise for Ireland but when I examined the census data it quickly emerged that no ‘Small Area’ (the unit with the finest spatial resolution) was constructed to have a population of zero, even the Small Areas that formed part of large national parks were considerably larger than the average small area, one assumes this was a conscious decision taken by the Central Statistics Office.

Instead I decided to focus my attention on Australia, from perusing their website, their equivalent of the Irish Small Areas was the ‘Mesh Block’ which were ‘the smallest geographic region in the Australian Statistical Geography Standard (ASGS), and the smallest geographical unit for which Census data are available’ (more information here). In 2011 there were 347, 627 mesh blocks for the country. I created a map of the unpopulated mesh blocks but it wasn’t very informative, large sections of the country were showing as being populated when in fact they weren’t.

In an effort to improve my map I continued searching their website until I found the 2011 Population Grid. The methodology of how their created their population grid is here. I then downloaded 9 shapefiles and merged them together in QGIS. I then used a definition query to select all the cells with a population of zero and created a map from this. It is a vast improvement from my initial effort. I spent some time styling my map in QGIS’s print composer. The below is the finished product. Very stark contrast between the big cities and the outback.

Unpopulated Areas of Australia

Unpopulated Areas of Australia

Farthest McDonald’s-Ireland

In my last post I downloaded all the McDonald’s in Ireland using overpass-turbo. The problem I had when I exported these was that because GeoJSON is such a flexible format, I was getting both nodes and ways. These would have had to be imported into ArcMap separately and the centroids of the ways found and merged with the points, this would have taken some time. Instead I got a great suggestion “Mmd Osm” on Youtube, that was to use ‘out centre;’ instead of:

out body;
>;
out skel qt;

This worked as promised and the centroid of both the nodes and the ways was exported. I then used OGR2OGR to convert this to a shapefile:

ogr2ogr -nlt POINT -skipfailures McDonalds_Ireland.shp McDonalds.json

I had to rename the GeoJSON file to .json for the purposes of this conversion, the last step I undertook was to transform the shapefile from WGS84 to the Irish Transverse Mercator, EPSG 2157. This was completed using the following code in OGR:

ogr2ogr -t_srs EPSG:2157 McDonalds_1shp McDonalds_Ireland.shp

OSGeo4w is in my opinion the best place to start with all things related to open source geospatial software. Now that