I’ve made a deal with myself: one new post for every two reconstructed ones.  This could take a while…

I said “developing” there – which begs the question “how are these cities developing?”.  This isn’t just a Las Vegas-style spreading of suburbia across the desert: many of the cities I’ve visited in the past year have shanty towns, and these appear, at least from outside, to be where a lot of the city development is happening (btw, I wanted to use a less emotive word than ‘slum’ here: although it’s what Slum Dwellers International uses, there’s still a lot of negative feeling about it).  From Lagos to Guatemala to Haiti, I’ve seen dozens of homes and businesses under tin roofs looking across at smaller numbers of tower blocks, and wondered “how do these economies fit together”, and “where does it go from here”.

Good old BBC gave me a few more answers… and a few more questions (like how does the nation-based world fit with people this adaptable and informal), and case studies, Medellin and London, of both positive and negative ways that the shanty and non-shanty worlds can start to fit together.

Perhaps it\’s the way you look at it.  If you look at the Wikipedia links above, you’ll see shanty towns and slums described in very negative terms… impoverished, illegal, lack of services.   If you hang out with people who live or work in shanty towns, they’re communities and neighbours and businesses and services – and quite possibly the adaptable, informal, majority economic future of the cities they\’re part of.  Whichever way you look at it, there are a lot of people in shanty towns, and how (and what) they develop is important.

But a country doesn’t suddenly change socially because a crisis happens.  It doesn’t leap, ready-formed, into a new incarnation where the only thing that’s ever happened is the current earthquake, flood, famine etc.  Countries have histories.  Events happen, societies form, agencies and other countries, if needed, help them to develop and become resilient. And all this activity creates data.

One of the first things that a response agency does, as it goes into a crisis, is create a profile of that country.  How many people are there?  What’s it society like?  What are its existing needs?  Which crises have happened before (and how did people respond to them)?  Which NGOs are already active there?  Where is everything (towns, hospitals, clean water sources etc)?

I’ve been studying the crisis indicators from 3 organisations (UNOCHA, WWHGD, ACAPS), and the development indicators from many more (UNSTATS, UNDP, WHO, World Bank etc).  And the big questions that I’ve been asking are: what do responders going into a crisis need, where can we find that information, and how can we make that faster and easier to do.

IMHO, ACAPS has one of the best indicator lists available. Which is handy because I’m about to co-lead a Standby Task Force deployment to help make obtaining and organizing their indicator data faster and easier.  The numbers that they’re looking for are:

An SBTF team is spending the next week looking for these indicators for a specific country (DR Congo) so we can learn about how difficult that is, alternative sources of information, indicator provenance etc. whilst other teams look for demographics, crisis timelines, existing surveys and ways to generalize and automate parts of this process.  I’s going to be quite an exciting week.

References:

• Joern Birkmann (UNU) 2007, Risk and vulnerability indicators at different scales: Applicability, usefulness and policy implications
• World development indicators list

A lot of crisismapping comes down to persistence, capability and trust. Trust is a biggie: as someone (Gisli Olafssen?) said, a disaster is not the time to be handing out business cards. We need to trust the people we’re working with, the systems we’re using (within sensible limits and the occasional online equivalent of kicking the case in the right place), and we need to (again within sensible limits) trust the data we’re using. Trust in the data that happens during a crisis has been talked about a lot recently (see conversations about verification, spoofing etc). It hasn’t been discussed so much when we’re talking about crisis indicators.

First, it’s not enough just to have a number, even if it’s only used as a rough rule-of-thumb for how bad a prior situation was. We also need to know how much we can trust that number, which means knowing where the number came from: who collected the data, how big a survey it was, how accurate the numbers are likely to be.

So, in recent work, we’ve started by looking for sources. Most databanks have a list of sources somewhere in the dataset description (or for individual datapoints, in the dataset footnotes); most maps have these somewhere in the margins. If they don’t, they should, because they can tell us a lot about how much we can trust the numbers.

Take, for example the MMR, the maternal mortality rate (annual number of deaths per 100,000 live births) for the Democratic Republic of Congo in the last few years. There are several versions of this number, and here’s a quick search for them and their listed sources:

We learn several things from this.

• The numbers are estimates. It’s not always clear that this is so.
• Even on the same datasite, numbers for the same indicator can differ and aren’t always updated.
• You can sometimes use the numbers to guess at their source (e.g. the CIA figure) – but you have to be careful about generalizing this.

Maternal mortality turned out to have a very strong single source (the trends in maternal mortality reports). But for many development indicators, the numbers from different sources very rarely match, and are sometimes very wildly different. So if we want a number (or even a couple of numbers) to use, we need to do some detective work. More on this soon.

Details: I looked at 5577 csv files in the data.un.org dataset, and automatically excluded the footnotes at the end of each dataset. Data.un.org data that was excluded from the investigation were as follows: the UN interface limits downloads to 50000 rows of data, so 159 files in the set are incomplete; and 25 files were excluded because they’re in a format (multi-sheet Excel files) that needs further work to separate comments from data. In all, there are 21195188 rows of data in the remaining dataset, so much of the following work had to be automated. Every cell in the first row of each of these files was added to file “headings.txt”; every cell in the first column of each of these files (excluding the first row) was added to file “indices.txt”.

Data.un.org files excluded (because they were in Excel format) were:

• Human Development Indices: A statistical update 2011
• Indicators on Women and Men
• OECD Data
• World Tourism Data
• World Fertility Data
• World Marriage Data
• World Contraceptive Use
• Key Indicators of the Labour Market, 7th Edition
• WTI Data

My goal here was to survey the types of headers used, so I could create lists of errors, inconsistencies etc. for known index types (e.g. country names), and start the work of cross-matching indices and headers to each other.

So. The results are: the indices contain these types:

• Most files: Country, region and economic group names, with a variety of spelling errors.
• International Monetary Fund files: a code called OID. The second column (country name) from these files has also been added to the list of indices.

Headings are more varied:

• Dates/times (Jan, Feb, year, number of years etc)
• Age-related (e.g. age, age groups)
• Geographical (e.g. country, OID, national station id, station name, residence area, WMO station number, city, country of birth, country of citizenship, country or territory, country or area, reference area, area etc)
• City type
• Birth/death details (cause of death, birth weight etc)
• Number of people (e.g. number of refugees, children etc)
• Personal (gender, sex, religion, marital status)
• Nationality/language (citizenship, ethnic group, native or foreign born, language)
• Household (household size, type of household etc)
• Financial value (USD, wealth quintile, currency, SNA)
• Employment/education (occupation, industry, education, literacy)
• Trade (Commodity/ISIC rev 3, flow=import or export, etc)
• General measure words (observation value, quantity, quantity name, weight, measure etc)
• Classification words (subgroup, item, type etc)
• Metadata (Description/source/series, survey coverage, activity status, record type, reliability, variant, footnotes ““ these headings are used to refer to footnotes at the end of the file)

More detailed data should be attached to this post.

I’ve written already about how to access the data.un.org datasets from an external application and about the types of headers and indices within it. Now it’s time to make the geographical references more usable.

I looked at 5577 csv files in the data.un.org dataset, and automatically excluded the footnotes at the end of each dataset. Data.un.org data that was excluded from the investigation were as follows: the UN interface limits downloads to 50000 rows of data, so 159 files in the set are incomplete; and 25 files were excluded because they’re in a format (multi-sheet Excel files) that needs further work to separate comments from data. In all, there are 21195188 rows of data in the remaining dataset, so much of the following work had to be automated.

I collected the indices and headers from all the datasets into lists: the headers list was searched for geographical references, and the indices list was used to produce a list of corrections from the data.un.org geographical indices into both ISO standard 3166 (countrynames) and UNSTATS’ list “Country and Region Codes for Statistical Use” of region, country and economic group names on data.un.org.

Geographical references in the headers are:

• country of birth, country of citizenship, country or area, country or territory, country or territory of asylum or residence, country or territory of origin, reference area.
• OID.
• WMO station number, station name, national station id number.
• City.
• Area, residence area, city type.

The GIS naming standards found in the dataset were:

Country names: Most of the indices and country references are a close match to the UNSTATS standard or ISO3166 (as used by the World Bank etc), although country names in particular are very inconsistent in this dataset. The indices discussion below gives instructions on

• how to correct all of the country and region names to either of these standards. This applies to headers country of birth, country of citizenship, country or area, country or territory, country or territory of asylum or residence, country or territory of origin and reference area. OID: International Monetary Fund’s internal GIS standard.
• WMO station number, Station Name, National Station Id Number: World Meteorological Organisation references to meteorological stations.
• City: UNSD Demographic Statistics (code: POP) includes city names. No standard has yet been identified for these names.
• Area: UNSD Demographic Statistics (code: POP) and World Health Organisation (code: WHO) use the headings Area and Residence Area to classify the geographic extent of coverage (e.g. “Total”, “Urban”, “Rural”). UNSD Demographic Statistics also uses the heading “City type” to subclassify cities too (e.g. “City proper”, “Urban agglomeration”).

Most of the data.un.org datasets contain information that is listed by country (e.g. Yemen), region (e.g. West Africa) or economic group (e.g. Developing Regions). Looking at the placenames in the indices, we see that they are a mix of country, region and economic group names, with different spellings and formats for similar names (e.g. “Yemen”, “YEMEN”, “Yemen,Rep.”, “Yemen, Republic of” etc).

Two standards are similar to the placenames used in these files: ISO3166 and the “composition of regions” list published by data.un.org.

• ISO3166 is a widely-used standard, but contains code for countries and their subregions only (e.g. has no official lists of larger regions or economic areas) and is published as tables online and available (although without the list of withdrawn codes) in the Python library pycountry.
• The Unstats list (which ISO3166 is partially based on) contains countries, regions and economic areas, but is available only as an html table at http://unstats.un.org/unsd/methods/m49/m49regin.htm. The countries list is available as a ScraperWiki dataset which needs some editing to make it usable. The regions list has been scraped by hand for now. There are two main lists in it: the regions, subregions and countries by physical location, and the economic status (e.g. “Developing regions”, “Least developed countries”) of each country and region. These are mostly consistent, with a couple of oddities. For instance, Netherland Antilles doesn’t appear on the list of countries, but does appear on the list of small island developing states. Luckily it has a country code, so it’s been included in the list of countries.

The indices were checked against both these standards. Suggested improvements to the indices and standards include:

• Make the regions list available as a csv file online, to include withdrawn country codes, assignment dates and withdrawal dates (these are needed to match names for earlier years).
• Make the economic status list available as a csv file online.
• Lobby ISO to create a region (Africa, West Africa, North America etc.) code standard, if it doesn’t already exist.
• Lobby ISO to correct inconsistencies in the ISO countries list (e.g. republic not Republic in Bolivia’s name).
• Make a definitive statement about which GIS naming standard (ISO, UNstats etc) UN online data should attempt to adhere to.
• Change all the data.un.org datafiles to meet this standard.

Against the ISO3166 standard, the data.un.org csv index errors were:

• Withdrawn countries with no ISO3166 code: “East Timor\”, \”Czechoslovakia, Czechoslovak Socialist Republic”, \”USSR, Union of Soviet Socialist Republics\”, “Yemen, Yemen Arab Republic\”, “Yemen, Democratic, People’s Democratic Republic of\”, “Yugoslavia, Socialist Federal Republic of”, “Germany, Federal Republic of”, “German Democratic Republic”, “US Miscellaneous Pacific Islands\”, “Wake Island\”, “Serbia and Montenegro\”.
• Abbreviation, e.g. “Rep.” for “Republic”, “St.” for “Saint”, “Is.” For “Island”, “Isds” for “Islands”, “&” for “and”.
• Added markers, e.g. “+” added to the end of region names, to differentiate them from countrynames.
• Capitalisation, e.g. “YEMEN” for “Yemen”, “republic” for “Republic”, “The” for “the” and the” for “The”.
• Brackets: UNICEF in particular uses brackets “()” instead of commas in placenames
• Standards confusion: the ISO3166 labels “name” and “official_name” were both used in the same datasets (“name” is available for all countries; “official_name” is not).

Tables of other misspellings against both standards are given below. Some of these errors are the use of familiar names (e.g. Brunei, Ivory Coast, China) or issues with character translation (e.g. Cote d’Ivoire). Some names could not be resolved: remaining queries include the code for French Polynesia, whether “Christmas Is.(Aust)” is Christmas Island, whether St. Helena refers to just the island of Saint Helena, or “Saint Helena, Ascension and Tristan da Cunha” and whether Palestine and Palestinian Territories refer to “Palestinian Territory, Occupied”

Behind every global development hackathon is a team of people who act like glue, watching projects around the world and connecting up the ones that are related to each other, or can help each other or people who can just plain support each other when it’s dark and cold and they’ve discovered that there are limits to how long coffee and doughnuts can keep you awake for.

Up to now it’s down to how many connections and projects we can keep in our heads at once, and what we’ve got scribbled on paper and files all over our pcs, and how many streaming windows we can physically fit onto all our screens (because one laptop is never enough if you’re keeping tabs on a global event).

But this time we’re spoilt. We have a list of projects that’s been moderated and sorted down to a mere 61. We have spreadsheets that show which teams (There are 81 of these at the moment, and about 1000-2000 people) in the 26 camps around the world are working on those projects (and we really mean around the world when we\’re talking about the international space station). And we have a piratepad for every project, so teams around the world (e.g. Jakarta, Oxford, Stuttgart and San Francisco on the Open Data API) can leave each other notes and hints and pointers to data and code.

It’s working well. I have some favourite silly moments. Like seeing how the StartupBus moving-camps-on-a-map idea got adapted to the ISS moving across the map of camps for this hackathon. And the duck that tells when the ISS is overhead. And the tweet from a zucchini plant.

And I have to give a big nod to the New York StartupBus Crew who came out in force when Mike Caprio (this year\’s New York conductor) asked them to.

But mostly it’s the feeling that the multi-city hackathon has finally come of age. And pride that a lot of people around the world gave up their weekend to go make a difference to the world. And space. And beyond…