Phew! It’s launched, and it works. And it all went surprisingly smoothly.

Just before Christmas, the Independent Parliamentary Standards Authority approached us to see if we’d be interested in producing the online part of their first consultation on MPs’ Expenses. This was exciting, to put it mildly — so we leapt in with both feet.

As is often the case with this kind of work, the deadlines were short and the team rushed. The consultation is only five weeks long, and there’s even less time to analyse the responses after it closes. With this in mind, we did our best to come up with a format that gives respondents a meaningful way to respond, while also making the responses as easy as possible to analyse.

Most of the questions have yes/no answers, and a few have options you can choose between. Each page has a box for free text responses, instead of each question — and the box is limited to 500 characters. We were a bit nervous about limiting input in this way, but hopefully it’s a good compromise. No one’s submitted anything nearly that long yet, so it hopefully people aren’t feeling constrained.

Given the number of questions, the length of the text and the constraints of the form, it was really important to make the site as intuitive as possible. The text boxes make it very obvious if you put too much in them. The final form at the end confirms that you really are finished, and the buttons are subtly differentiated using icons and colour. You’ll also find that using the sidebar menu to jump around is safe — even if you’re in the middle of writing a response. We’re using javascript to make sure the form gets submitted. And everything’s unobtrusive, accessible and standards-compliant. We even managed to squeeze in some RDFa.

The site is, of course, implemented using Wordpress. It’s a fantastic platform to work with. There’s no doubt that Wordpress’s general awesomeness is what made it possible to launch this quickly — and on time, and on budget. Everyone seems pleased with the way it’s come out (including a real user) so we’re pretty pleased!

Questions? Thoughts? Did it work well for you? Do leave a comment, and let us know what you think.

The UK’s version of data.gov, ably put together by the Cabinet Office, has just launched in private beta. We got to have a sneak peak, and it’s great!

The site is a blend of the US’s equivalent, data.gov, and Directgov | Innovate. It’s got a listing of available data packages, powered by the Comprehensive Knowledge Archive Network, and user-generated lists of apps and new ideas. This is just right: the data you need, combined with a way to promote the things you make and a place to get ideas if you’ve got itchy typing fingers but lack inspiration.

It’s not perfect. Conspicuously missing is an organised way to browse data sets: but that’s coming, along with some other tweaks and twiddlings that’ll improve the site’s usability.

The site is powered by Drupal, with packages catalogued and hosted by CKAN. Meanwhile, data.gov.uk hosts a data store powered by Talis that can scale to 100 billion triples and is hosted on Amazon EC2. The system is federated, so departments can add and control their own data, lots of which is available as RDF, with the remainder downloadable in spreadsheet form.

Speaking of spreadsheets, they’ve even written an app that departments can deploy in-house to convert spreadsheets into RDF (kudos to John Sheridan!) which makes it much easier for departments to produce structured, linked data.

This is all working now, and was put together by the team at the Cabinet Office in the last three months. This is a massive achievement, and it sounds like it’s just the beginning: they have big plans. User submissions for new datasets. Metadata to describe provenance. More data sets on the site. More data as RDF. Organised browsing for packages. Source code releases. The list goes on.

This is such an encouraging thing to see. No expensive procurement exercises for clunky, bespoke sites: instead, we have the right tools for the job, joined together. Simple things that do one job well, combined to form a more complex whole. It’s the Unix philosophy in action.

This is how all Government IT should work.

Our hearty congratulations go out to the team at the Cabinet office, with special thanks to Richard Stirling for spilling some of the beans. I had lots of questions and nitpicks, and every single one of them was answered reassuringly.

They’ve got a plan, and it’s a good one.

James Darling and I met up for a chat about some data-related stuffs this afternoon, and came across this data on average fuel prices via the Office of National Statistics. This struck us both as being very useful (any hauliers out there who want to make some a nice visualisations?) so we threw together a script to convert it into (much) more useful formats.

Check out Weekly Fuel Prices, in more formats than you can shake a stick at, here.

People have been talking about the Ordnance Survey rather a lot recently.

It’s a very strange beast. It has mountains of really useful data: electoral boundaries, postcode databases and the locations of all sorts of buildings, not to mention roads, railways and green spaces. Unfortunately, it’s a quasi-independent body, which has to pay its own way. It charges heaps of money for access to all these datasets, and makes them available under strange and deeply inappropriate licences.

This has long been a problem for civic hackers, and by implication, everyone else: if you want to make a service that needs to turn a postcode into a geographic location, you need to use their data, and most of the time, you can’t. It costs an arm and a leg.

Fortunately, it’s clear to everyone that the OS needs to be significantly reformed. They own datasets like electoral boundaries, that are fundamental to the operation of a democracy. They’ll charge you if you want to use them, even if you’re the government and you’re organising an election. That’s just nuts, and we all know it. So what to do?

Thankfully, people have been working on it. The recent Power of Information Taskforce report had lots of good things to say: Recommendation 7 was almost perfect. Liberate postcodes and boundary data. Make basic mapping data available for free to all, for modest use. Simplify licensing. Its only real downside was the absence of any mention of derived data.

Put simply, if you use an OS map to create a database of something — like postboxes, hospitals or parks — then OS share copyright in that data. In this modern age of user generated content, that position is completely unsupportable. It’s a shameless grab for intellectual property, motivated by their desire to receive extra royalties. For a private company, that might be fair enough, but it’s reprehensible for a body that exists to provide a public service.

For anyone who thinks this is merely a theoretical problem, it may be of interest that all those lovely crime maps launched by police forces across the UK are probably in violation of the OS’s licensing terms.

This, then, was the background to the latest budget. Having heard that it would contain some new announcements, we were waiting for its publication with baited breath, and indeed: it promised reform. The following day, Ordance Survey published their new commercial strategy. It is underwhelming.

There are some good changes: more data, including boundary information, will be made available through OpenSpace, their API. They’re going to revise their definition of “commercial” so that sites that use their data can carry advertising without being required to pay for licences. But that, more or less, is it. The rest of the strategy revolves around converting people using free licences into ones that become financially sustainable so they can pay. Fair enough, but hardly groundbreaking.

The real problems remain. OS still own electoral boundary data and postcode boundaries & locations. They still decide if you’re commercial, and you still have to accept their onerous licences to use their data. OS still maintain a stranglehold on any data that they consider to be derived from theirs. They’ll still charge you royalties to use that data, of at least £1000 a year. You still won’t be able to do anything with that data that’s not acceptable under their licence, like adding it to OpenStreetMap.

This new strategy is progress, but only just. It is at best a fractional improvement upon what we had before.

A lot more needs to be done.

Here’s a story from FreeOurData which is, quite frankly, incredible. The Office of National Statistics, in preparing for the next census, has found that the postcode databases offered by the Royal Mail and Ordnance Survey aren’t accurate enough for their purposes. Their solution: to build their own database. This is fair. The postcode database is not amazingly accurate, and ONS have different requirements anyway.

Unfortunately, Royal Mail and Ordnance Survey make good money from selling the postcode databases to other organisations. These datasets are very valuable: you’ve probably made use of them whenever you’ve put your postcode into a website. Royal Mail and Ordnance survey did not — apparently — like the idea of ONS making another postcode database with which they’d presumably have to compete. So, rather than take that nice dataset and do useful things with it — like giving it back to us taxpayers — the ONS have pledged to build the database, use it for the census, and then destroy it.

Postcode databases are almost a holy grail. Of all the datasets in the country, liberating the postcode database for free reuse would probably create more value than any other. The thought of spending £12m on a new, super-accurate postcode database and then destroying it is wasteful, a huge missed opportunity and to be frank, completely idiotic.

We implore you: don’t do it.

We’re terribly, fantastically pleased to announce that after a bit of wrangling, Steph Gray and I are able to release ConsultationXML as open source software under the GNU Affero license. The recent report on open source software in Government hinted that departments ought to try to release source code for the software they commission, and we’re delighted to be (we think!) the first to do so.

We’re not sure who will want to play with it yet. We hope that other departments will want to deploy and use the tool to improve their consultation offerings. It may be that people in the private sector will find some use for it. People have already used ConsultationXML for really neat things that we didn’t expect, so anything could happen: which is, of course, the point.

We ran this by the renowned geek-come-blogger-come-minister, Tom Watson, who had good things to say:

I think this is a great tool. We’ve just announced the Open Source, Open Standards and Re–Use report on the use of open source software in government, an element of which was to encourage government to contribute to the world of open source software, and this is the first practical expression of that goal.

For more information about ConsultationXML, and to download it, head over here.

The Power of Information Taskforce have been figuring out how to liberate public sector information, how to facilitate better use of the modern, social web in government, and how to support the efforts of those outside government who are doing worthy things. All in all, they’re a great bunch of people, doing great work. They’ve just published their draft report. This is notable for two reasons.

First, the report has been published using Commentariat, a Wordpress theme from the folks at DIUS that makes it easy to browse and comment on big documents. It is fantastic — it really works well — and you’d never know it was Wordpress, unless you checked. It’s a great example of Wordpress’s flexibility.

For me, this informal consultation exercise is characterised by its ease. I read the report online. When I had something to say, I could just fill out the comment box. I could read other people’s comments, which helped to clarify my own thoughts. The PoI team have been posting comments too: responding to people, thanking them for their feedback, letting us know when they’ve made changes. Brilliant. It’s a real conversation between people who genuinely want to seek out ideas. It stands in stark contrast to the process of formal consultation, which is stifled, slow, more or less one-way, and frequently happens after all the important decisions have already been made. This exercise couldn’t be more different: agile, easy, conversational and public. I have no doubt that the report will be better because of it.

Second, the report itself is great: it hits all the right boxes. Be active in other people’s networks. Make sure civil servants have decent ‘net access — it’s astonishing how many of them are so filtered that they’re almost useless. Support third sector developers. Invest in innovative new ideas, even if they’re high risk. Liberate geodata from burdensome licensing and fees. There are 25 recommendations in all, and they’re all great.

If you have some time, read the beta report and leave some comments. It’s a great document, and great opportunity to get your thoughts in front of people who will listen.

I was at the UKGovWebBarcamp last weekend, and among the talks I attended was one by the Directgov Innovate team. This team has been recently formed, and is a really good development. In their own words:

Directgov have created the innovate.direct.gov.uk developer network to inform the greater developer community about available resources, to provide a platform to connect with one another, and to showcase new ideas with the aim of supporting and encouraging innovation.

Over time we will provide content feeds and API’s allowing people to develop new and interesting ideas and applications for use by the greater community.

Among my questions was: what will this team actually do? I was very glad to hear that they plan to develop new sites, make APIs, make data available to people and create a community of developers who are interested in this field. Great. This is just what’s needed. What’s even better is that they’ve already delivered on that promise.

Yesterday, Tom Watson tweeted that we should have a site where people can check to see if their school is closed. Brian Hoadley and Paul Clarke at Directgov took up the challenge, and just a couple of days later, launched a new site: SchoolClosures.org.uk.

It’s pretty rough around the edges: there doesn’t seem to be much RSS support, and there’s no access to the underlying data, and — well — it doesn’t tell you whether your school is closed… but it is still useful, and it’s very impressive that it appeared so quickly, and with such little prompting.

Kudos to all involved — this is a fantastic and very encouraging start.

Simon Dickson picks up an interesting story from the BBC’s Editors’ blog about official releases of statistics.

Usually, when the Department of Children, Schools & Families releases new statistics, they’re given to the media in advance. The media need this lead time to be able to format all their articles and tables and make sure everything is correct and works properly: this is fair enough, given that they’re the data they’re working with lives in lots of Excel spreadsheets, with multiple sections, differing layouts and everything else you really don’t want if you’re tasked with this kind of job.

Given what they have to work with, the BBC’s anger is understandable, but perhaps misplaced: why are we still dealing with bunch of spreadsheets in the information age? Why isn’t there an API that allows this data to be queried, or at the very least, a standard data format that doesn’t change from year to year, and doesn’t reply on proprietary technologies that are hard to work with?

An API or standard data format would allow media organisations to write code which generates the statistics they need every year. They wouldn’t have to create new tools to be tweaked and tested every time there are new statistics. Better still, it would create a market for someone to create a tool that did this for them, saving them money. Even better than that, it would allow anyone who wants to do something innovative with these statistics to do so far more easily.

I think I’m not alone in saying that the case for releasing data properly — in reusable formats, to everyone, for free, whenever it is possible — has been made, has been heard and has been widely accepted as valid.

Why are we still fiddling with messy spreadsheets, and bemoaning the fact that we only have days to do what should take hours?