TV shows like Lost use mystery in a pretty smart way: giving tantalising clues as they go along and always making you want to watch more, to get some kind of satisfaction of knowing enough.
I’ve been looking into public spending and the mystery of how public spending records are stored is very similar to the TV shows.
I’ll tell you the story of the COINS database, which is a store of public spending and planning data, and you can judge for yourself how similar the story is to a gripping TV show.
The way COINS is described in documentation is that it’s a store for government spending records. In the background to reports on public expenditure, like the PESA report, and many of the Office of National statistics reports on government spending, the guidance says the reports rely heavily on COINS data.
I asked for the COINS schema, but what I got was the dimensions of the OLAP database and no description of the fields meant, so that was the first mystery. Then I requests and got the COINS training notes, but with all the screen shots and lots of the descriptions redacted, so this was the second mystery.
When I met some people from the Treasury to talk about COINS some more and they told me about the thousands of spending codes in COINS, which I then requested, but with a number of them redacted, this was the third mystery.
Then, on Friday 4th June, we got the COINS data itself, well a sample from this year, and some COINS guidance(PDF) to go with it. The guidance says:
It is possible that you won’t be able to recreate the numbers that Treasury or ONS have published. This is because:
• COINS has a single structure that is updated to reflect the latest classification of spend and organisation of government. The snapshots taken by the Treasury at defined points (e.g. to enable reports to be published) contain certain key fields, which then reference the latest structure. Any changes to the structure since the report was published may mean that it is not possible to recreate published figures;
• of the time difference between the publication of aggregated information and this release of data, with the potential that data have been updated between the differing points in time.
• Not all the data used to calculate these numbers are sourced from COINS.
So it’s not possible to check the figures against the published reports, like PESA, to check we understand the data published. Another mystery.
There is a description of useful combinations of the fields in COINS that would give accurate public spending figures from COINS — which means we are getting there.
So, what do you think? It will be satisfying when we understand all the spending!