I was looking at this web-map this morning that shows the number of Irish living in the UK. I quickly put together an equivalent for the number of UK born individuals resident in Ireland as reported in the 2016 census.
At some stage in the second half of 2020 the Central Statistics Office released the population for each townland in Ireland for both the 2011 and 2016 censuses. They had a category for both data that has been suppressed ‘-1′ and the townlands that were unpopulated ‘0′. I think it’s a safe assumption that if you read (or have stumbled upon my blog) that you too might be interested in maps of unpopulated places. Below is the static map I posted on Twitter and Reddit and (by popular demand – read as two random people on Reddit) I’ve also included a slippy map that you can examine at your leisure.
Recently I finished reading Saroo Brierley’s book, A Long Way Home and titled Lion in the feature film. In it he recounts his extraordinary life story of getting separated from his brother at a train station whose name began with ‘B’ a few hours from home in rural India in 1987. He was five years old and his brother Guddu had taken him the few hours from home to make some extra money sweeping out trains. Guddu leaves him on a bench in the train station to get some sleep while he goes to work. The young Saroo wakes a little later but can find no trace of Guddu, in his panic he hops on a train looking for Guddu, the doors shut and he (over what he believes to be 12-15 hours) ends up in Kolkata (then Calcutta). Here, against all odds he survives on the streets and eventually ends up being adopted by (from the sounds of it) a wonderful couple in Tasmania. In his 20s he begins trying to find his mother and siblings in India by using Google Earth to pinpoint both the town he boarded the train and in turn his hometown.
His task was an unenviable one, India is the second most populous country in the world with 1,367,139,484 people (17.5% of the world’s population, Wikipedia). Below is a population density map of India to give some context as to how the population is distributed. These figures are from present day, the population of India in 1987 when Saroo became lost was 819,800,000.
The whole story piqued my interest and I wondered if we were to try the search today using the benefit of open data and open-source software could we make it more efficient. I would like to heavily caveat what is to follow by stating that I am using software and data that would not have been as extensive, complete or fully-featured when Saroo began his search in 2011.
I kept very careful notes whilst I was reading the book and below is an exhaustive list of the criteria that he would follow from his recollections from his five year old self. The criteria is both from his hometown and the town that began with ‘B’ where he got on the train that eventually took him to Kolkata.
From the initial criteria above it became clear that a number of the criteria would not be usable in replicating the search; such as that it wasn’t in the colder north of India (too subjective) or that they lived side by side with Muslims (a common occurrence in India). Below is a refined list of criteria that I am going to use in order to try and replicate the search.
Saroo’s methodology started with tracing his steps backwards from Kolkata. He knew he got on the train at a station that sounded like ‘Berampur‘ and he thought he was on the train for approximately 12-15 hours. Based on this, he consulted Indian friends of his at college about how to start searching. One friend in particular, Amreen whose father worked for the Indian Railway in New Delhi proved helpful. Her father made an educated guess that trains in India in the 1980s travelled at between 70kph-80kph. Based on this Saroo calculated that he would have travelled 1,000km in that time. He started searching methodically outwards from Kolkata to try and find the station that began with ‘B’ and hopefully, then, his hometown that was about an hour away from this.
If I was to try and recreate the search for Saroo’s hometown, I would need access to as much free geographical data as possible so I turned to OpenStreetMap and specifically the downloads available from Geofabrik. I downloaded the Protocolbuffer Binary Format (PBF) file of the entire of India. The first items I was interested in were the railway lines and railway stations of India. QGIS can load the PBF files natively but the entire of India is a bit of a stretch for it regardless of computing power available.
I used the Osmium tool to extract every railway station and line in India and then GDAL to convert them to the geopackage format. The first analysis I undertook was to follow Saroo’s and ascertain how many railway stations are within 1,000km of Howrah railway station. To give some context, OSM has listed 7,979 railway stations and 108,000km of railway line (the Wikipedia page lists 68,155 km (the discrepancy may be accounted for by in the inclusion of every siding and historic railway line etc. in the OSM data). The below map shows every railway station and line from OSM.
To give an idea of Saroo’s methodology of drawing a 1,000km buffer of Kolkata and working outwards that left him with 2,905 to search. Below is a map of all of these.
I decided not to use Saroo’s methodology of using a buffer distance from Kolkata. I had the entire OSM database for India at my disposal so I decided that the first and easiest step to undertake was to find all the railway stations that began with ‘B’ and contained ‘p’, ‘u’ and ‘r’. I used QGIS’s inbuilt python functionality, PyQGIS. I wrote a small script that would use the regular expression module to find stations that matched the above criteria.
import time, re start = time.time() # Start Message print("Program is Starting...") layer = iface.activeLayer() prov = layer.dataProvider() if layer.dataProvider().fieldNameIndex("Relevant_Rail_Stations") == -1: layer.dataProvider().addAttributes([QgsField("Criteria_Test", QVariant.String)]) layer.updateFields() pattern = '^b.*.[p].*[u].*[r]*$' # starting layer editing layer.startEditing() features = layer.getFeatures() for feat in features: Regex_Stations_Search = feat['name'] Regex_String = re.compile(pattern, re.IGNORECASE) Regex_Match = Regex_String.search(str(Regex_Stations_Search)) if Regex_Match: layer.changeAttributeValue(feat.id(), 11, "Meets_Criteria") else: layer.changeAttributeValue(feat.id(), 11, "No_Match") layer.commitChanges() iface.vectorLayerTools().stopEditing(layer) end = time.time() print("Finished running - the program took " + str((round((end - start), 2))) + " seconds")
The above script took 15 seconds to run and it narrowed down the search field from 7,779 to 91 as shown below. I don’t think it is too much of a stretch to take the various ways that he thought it might have been spelled (as below) and to then take the common letters from those spelling and narrow down the search that way.
Now that we’ve narrowed down the number of possible ‘B’-towns to 91, the next step is to try and use the hometown criteria to find the correct location. We know that Saroo’s hometown had a water tower, river, bridge, dam and a fountain in the park near the station. He believed the town’s name began with ‘G’ and sounded liked ‘Ginestlay’. I could proceed with the other criteria for the ‘B’-town however I felt that with a dam, bridge, river, fountain and water-tower there were enough unique entities that I could try at this stage to find the hometown without dedicating any further resources or time to the ‘B’-town.
The assumption that Saroo was working on from his friend Amreen’s father was that trains travelled between 70kph-80kph in the mid-80s. As processing power wouldn’t be an issue for what I was trying to do I decided to use the upper limit of 80kph for the speed of the trains. As the PBF file for the entire of India wouldn’t load correctly in QGIS I extracted all of the bridges, dams, rivers and water towers. Below are maps showing the distribution of each from OSM.
I omitted pedestrian over and underpasses at this stage as they were too prevalent to help narrow down the location. I now needed to buffer each of the possible ‘B’-towns by the distance the train would have travelled. Saroo knew that night that Guddu and he travelled for about an hour. I added a small bit of ‘fat’ to my buffer so I used 100km as the buffer distance. I then dissolved the buffers together and clipped them to the coastline. Saroo thought that he been on the train for 12-15 hours. I was thinking that it would be useful to do a buffer from Kolkata of perhaps 6 hours and omit everything inside this buffer as he knew he travelled a farther distance.
A quick GIF of this process is shown below.
Below is the final search area with the every dam, river, bridge, water-tower and train station within 100km of the ‘B’-town stations.
For the final analysis I wanted to find only the stations that had a water-tower, dam, river and bridge within 2km of the station. I picked 2km because I think that would be a reasonable distance for these features to be from a town-centre for 5 year old Saroo to walk to regularly enough to remember them distinctly. We didn’t want to include the ‘B’-town stations themselves so I simply clipped out the data for a distance of 10km from each ‘B’-town as Saroo said that Guddu and him were on the train for about an hour so 10km seems like a safe distance to clip.
The next step was to find each station that had our relevant criteria within the buffer area. For this I turned to PostGIS (I tried running the query the SQL query in QGIS but it crashed every time). I decided at the first iteration to only use—bridges, water towers and dams as my logic was that these would provide more conclusive than pedestrian over and underpasses and rivers which may have proved too common. Plus, the likelihood of a bridge being for a river was much greater than it being for a ravine, gorge etc. There were 3,223 stations within the 100km buffer of each ‘B’-town station as shown below.
The SQL query that I ran is below, this ran in about a second and returned each station that had a: bridge, water tower or dam within 2km. It narrowed down the number of possible stations from 3,223 to 24 as shown below.
CREATE VIEW Relevant_Stations AS SELECT stations_buffered.geom, , stations_buffered.name , stations_buffered.fid FROM stations_buffered INNER JOIN merged_bridge_river_dam ON ST_Intersects(stations_buffered.geom, merged_bridge_river_dam.geom) AND merged_bridge_river_dam.Feature_Ty IN ('Bridge','DAM','W_Tower') GROUP BY stations_buffered.geom, , stations_buffered.name , stations_buffered.fid HAVING COUNT(DISTINCT merged_bridge_river_dam.Feature_Ty) = 3
An example of one of the 24 stations that match the criteria is shown below.
Saroo thought that the place he was from sounded like ‘Ginestlay’ so the next step was to find all the OSM tags for ‘place’ that started with ‘G’. I used the below Osmium command to extract all of these place names:
osmium tags-filter india-latest.osm.pbf place=* -o Placenames.osm.pbf
This extracted 193,057 place names for the entire of India. To ensure that I didn’t miss anything I converted the polygons and lines to centroids and merged everything together (this took about 2 minutes in QGIS). I then filtered this by every place that began with ‘G’ and used the ‘Select by Location’ tool to narrow down the list from 24 to 12 stations. I have included images of these 12 stations below. Some of these obviously were not where Saroo was from as they formed parts of large cities (such as Lucknow) but I’ve left them in for the sake of showing the full process.
The station that we were looking for, ‘Khandwa Junction’ with the neighbourhood of ‘Ganesh Talai’ is candidate area 3 above. There are a few observations that I’d like to make regarding the above work. Firstly, I tried to stay as faithful to the criteria that Saroo had to work with. Obviously, I had read the book and I knew the answer. I hope that the above doesn’t feel reverse-engineered. I can honestly say that I didn’t look at any of the OSM data for his hometown prior to starting this post.
I’m conscious of the fact that there is a high likelihood Saroo’s own search along with the media coverage of the movie are the reason the OSM data for Khandwa Junction exists at all. However I will state that all of the above steps are very easily customisable and the criteria easy to change. The parameters could be easily change to test another theory—such as using only a ‘B’ and ‘R’ for the ‘B’-town station perhaps. If we didn’t find the answer we were looking for on the first pass we could have used a DEM to find where train lines crossed a gorge outside of a ‘B’-town station. Or we could have used Python to find all the horseshoe shaped roads outside ‘B’-town stations. Finally, we could have used the data for pedestrian overpasses and underpasses and reran the above.
What I can say with certainly is that with the OSM data available today it was very straightforward to reduce the number of stations to be searched from 2,905 (those within 1,000km of Kolkata) to 12. This would then involve only searching 0.41% of the stations that Saroo originally had to sift through. Google Earth was the best tool available to Saroo at the time but there’s no universe in which it wasn’t a brute-force, sledgehammer to crack a nut tool. I hope with the above work I have shown that it is easy to greatly reduce the number of stations to search through using some decent logic and the power of open-source GIS software and data. I also hope that if anyone else is in a bind similar to Saroo’s that the above might in some way help to demonstrate how to search for the right answer and make it home.
We all know that the majority of Australia’s population lives in the eastern states. There’s a question I’ve been thinking about with a while — if you drew a line from just north of Brisbane to just west of Adelaide, what percentage of Australia’s population (including islands and Tasmania) live below that line? Well, I had a little time on this rainy Sunday so I decided to find out. I wrangled some data out of the ABS’ Table Builder and joined it with the geopackage of the SA2 geography (I chose this as I felt it provided the right spatial granularity without being too fine) and voilà, the answer to the question very few people asked 82.3%.
I was reading Alasdair Rae’s street types blog post yesterday and I thought about doing something similar for Ireland. I didn’t have much time today so I said I’d put something quick together for Cork City. Alasdair was lucky that he could use the amazing open data from the UK’s Ordnance Survey. For Ireland, we don’t have anything similar from our national mapping agency but fear not because it’s OpenStreetMap to the rescue. In my humble opinion individuals often spend far too much time trying to get a usable output from overpass-turbo when there’s an easier way. What I did was download the .osm.pbf file for Ireland from Geofabrik. Windows Subsystem for Linux has been a lifesaver since it came along, most of the heavy lifting I can now do through it. The steps I followed are as follows:
1. Download the .osm.pbf file using wget.
2. Install the osmium-tool
3.Filter out just the ‘highways’ from the dataset and save it to a new .osm.pbf file
4. Convert the .osm.pbf file to a new shapefile ( I don’t normally use shapefile but converting to a geopackage was causing some issues in this instance).
5. Reproject the shapefile to EPSG:2157 (Irish Transverse Mercator).
I could have made this more efficient again by combining some of these actions (such as save as a new shapefile and reprojecting), I did it this way to make the process easier to explain.
Next I brought the dataset into QGIS and extracted an area just for Cork City. The main issue I then has was how to ascertain what road types occur most frequently. I exported the Cork City extract as a CSV and imported it into Excel. I then used a generic formula from the folks at exceljet to extract the last work from the ‘name’ column. I then counted the frequency of the extracted work in a pivot table to get an idea of the types I’d like to use.
The last thing I did was to create a new field using an expression to find the various road types in the ‘name’ column from the OSM data. The expression I used is the below:
CASE WHEN "name" LIKE '%Road%' then 1 WHEN "name" LIKE '%Park%' then 2 WHEN "name" LIKE '%Street%' then 3 WHEN "name" LIKE '%Avenue%' then 4 WHEN "name" LIKE '%Court%' then 5 WHEN "name" LIKE '%Drive%' then 6 WHEN "name" LIKE '%Hill%' then 7 WHEN "name" LIKE '%Lawn%' then 8 WHEN "name" LIKE '%Estate%' then 9 WHEN "name" LIKE '%Place%' then 10 WHEN "name" LIKE '%Lane%' then 11 ELSE 0 END
I then symbolised the data and exported it. The final product is below. Obviously this was a pretty quick and dirty way to do it and if I create an atlas for every major town in Ireland I will use PostGIS which will allow me to count the frequencies and create the new field with ease.
I was hiking at the weekend and it got me thinking about the unpopulated areas of Ireland. I’ve seen maps made for the 2011 census showing the square kilometres that have no usual resident population but I hadn’t seen one for the 2016 census so I put together the below. I purposefully omitted Northern Ireland because the data is nine years old. If anybody would like the replicate the below just leave a comment and I can do a YouTube tutorial or post on here on how I put it together.
A topic that I think about frequently is the physical changes Ireland has gone through over the last 10 or 20 years. I’ve covered it before in this blog but I’m going to look at it again today. I think you could ask anybody around the country where the development in their town has been during the Celtic Tiger and this current boom period (well, up until the Covid-19 outbreak anyway) and they would easily be able to tell tell you.
It’s true that we have national indicators from the Central Statistics Office identifying where the most planning permissions have been granted as well as data from the Department of Housing, Planning and Local Government on new ESB connections to show where new homes have been constructed. I was looking for something that would give an idea of the physical development that has occurred in the last 20 years.
Enter the CORINE Land Cover Inventory. It started in 1985 as a way for the European Environment Agency (based in Copenhagen) to monitor the main land uses in the Union. As a little factoid, it stands for Coordination of Information on the Environment. It contains 44 land use classes and the cell size is 100 metres. It already contains layers that show the change between different years but it didn’t contain any change layer from 2000 to 2018. I wasn’t interested in the 44 classes, in fact I was only interested in the categories shown below. I decided to omit construction sites, dump sites and a few other inconsequential categories. These are the main categories that will show development in Ireland between those years.
111 – Continuous urban fabric
112 – Discontinuous urban fabric
121 – Industrial or commercial units
122 – Road and rail networks and associated land
123 – Port areas
124 – Airports
I used the the wonderful r.reclass GRASS tool in QGIS in order to reclassify the rasters for both the year 2000 and the year 2018 to combine these categories. I then used raster algebra in order to subtract the year 2000 from the year 2018 raster in order to show the actual change. Below is a GIF showing the process for an area of North Kerry and Cork.
I create the below map in A1 so you can zoom in to get a good sense of what 18 years of development looks like in a particular county.
I redid the map showing the development up to the year 2000 as a separate colour. I’ve also added a legend.
Finally, after a few unsuccessful attempts at converting the final raster to vector I used the gdal_polygonize tool which did it seamlessly without any loss of the smaller cells. Below is the final layer as a web map that you can make full screen.
There are a few interesting area that were developed in those 18 years such as Aughinish Alumina in Co. Limerick:
There are also a few windfarms such as this one in Tyrone:
Covid-19 Cases Ireland – 26 March 2020.
Updated map for the 25th of March 2020.
I’ve been creating daily maps of Covid-19 cases in Ireland using the official Government source. I think proportional symbols and proper labelling lead to the best outcome.