We’re terribly, fantastically pleased to announce that after a bit of wrangling, Steph Gray and I are able to release ConsultationXML as open source software under the GNU Affero license. The recent report on open source software in Government hinted that departments ought to try to release source code for the software they commission, and we’re delighted to be (we think!) the first to do so.

We’re not sure who will want to play with it yet. We hope that other departments will want to deploy and use the tool to improve their consultation offerings. It may be that people in the private sector will find some use for it. People have already used ConsultationXML for really neat things that we didn’t expect, so anything could happen: which is, of course, the point.

We ran this by the renowned geek-come-blogger-come-minister, Tom Watson, who had good things to say:

I think this is a great tool. We’ve just announced the Open Source, Open Standards and Re–Use report on the use of open source software in government, an element of which was to encourage government to contribute to the world of open source software, and this is the first practical expression of that goal.

For more information about ConsultationXML, and to download it, head over here.

Back in July, we were asked to make a prototype system for the Central Office of Information and the Cabinet Office.

For some time, they have wanted to put civil service job vacancies together in one place so people can find them more easily and reuse the data in their own applications, much as we have already done for central government consultations. Because of our experience with consultations, we were asked to make a prototype that uses scraping to gather data about job vacancies. Some fantastic work is underway to make this really easy by embedding RDFA into departmental websites: our part in this project was to get our hands on some data, check out what departmental websites are doing now and see if scraping could be a useful part of the solution.

We put a prototype together over a couple of months last year — altogether, it took about three weeks of development time — and I’m very happy to say that it’s now been unveiled, and you can play with it. Though the site is live, the data isn’t current: it’s only there as an example. These were all real vacancies once, but they may have been filled by now!

The site is fairly simple. Several departmental websites were scraped to get information about their current vacancies. We took that data, cleaned it up a bit and added it to a database that can be searched. Users can look for jobs by keyword (like ‘assistant‘), location (for example, a post code or place name), or all of the above plus salary.

If we can automatically identify the vacancy’s location, we geocode the it using RDFA on the site and GeoRSS in the Atom feed. We did this because it permits users to search for jobs by proximity to a location, and to import the feed into Google Maps and get an insta-mashup of vacancies plotted on a map — neat!

We think that the prototype has done rather well. It suffers from the same kinds of problems that systems relying on scraped data generally encounter: occasionally, data is missing, incomplete or in the wrong place. It would need some manual intervention if it were ever to become a real service. Thankfully, the work that’s happening at the moment to produce an RDFA vocabulary to define vacancies means that this approach shouldn’t be needed in the future.

We wrote up some recommendations as a result of doing this project: hopefully, we’ll be able to publish them at some point. We’ll definitely be helping to get departments on board when the time comes for them to start embedding RDFA in their web pages.

People have already started doing interesting things with ConsultationXML. I have to admit — I couldn’t be more pleased!

Richard Goodwin took PDF attachments from the London Gazette, uploaded them to ConsultationXML, got the HTML preview output and fed it into Wordle — and voila! A Wordle map of the London Gazette’s honours list was born.

Has anyone else done interesting things? Do let us know.

Over the last few months, we’ve been working with Steph Gray of the Department for Innovation, Universities and Skills on making consultation documents easier to reuse.

DIUS are doing some fantastic things with consultations. Typically, a formal consultation is a pretty tedious process: a department will write up a big PDF document, print it, send it to some people, stick it on their website and wait for people to respond. The whole process is pretty dated: it doesn’t really take advantage of the web, and is pretty inaccessible to most people.

DIUS have started to make this process better. In July last year, they launched a consultation that tried a bit harder to involve people. They used a Wordpress plugin, CommentPress, to allow people to comment on individual paragraphs in the consultation. They published a nice HTML version of the consultation document, with links and all. They even made a widget generator, so that people could embed questions from the consultation in their blogs.

Doing these things doubled the number of people who responded to the consultation, with very little extra marketing. Unfortunately, they were also pretty time consuming: turning a PDF into nice HTML is pretty labourious. They wanted to automate as much of this process as possible, to make it cheaper to deploy similar consultations in the future, and they asked us to help.

Creating all these consultation tools would be quite easy, if the data existed in a format that could easily be reused. Unfortunately, PDF is certainly not that format. It is is designed for print, and is difficult to repurpose. To make this easier, we wrote some tools to convert PDFs into very basic XML, and to allow people to extend that XML into something useful.

This human intervention is really important. It allows semantic information to be added to these documents: questions and their possible answers can be identified, and explanatory paragraphs can be linked to questions. It also allows formatting and images lost during conversion to be added back into the document, and extra formatting like links to be added.

So, with that in mind, we produced a web-based XML editor for staff in web publishing departments. The idea was to create an editor customised to the XML schema we’re using, so that people who are only just XML-literate can still use it. The editor automatically converts PDF documents to basic XML and then presents it for marking up, tweaking and generally-making-better. The result is awesome XML, usable by other tools to do neat things.

ConsultationXML is about to be deployed within DIUS, where it’ll be used by real people so we can get feedback and make it better. We’re hosting an installation here, so that you can play with it and give us your thoughts. We hope to make it better — it’s not quite finished yet — but it’s finished enough, so we’re getting it out there for people to try. It’ll be open source just as soon as the lawyers have done their thing.

Have a play with the beta ConsultationXML editor here.
Update: Steph has posted his writeup.

Simon Dickson picks up an interesting story from the BBC’s Editors’ blog about official releases of statistics.

Usually, when the Department of Children, Schools & Families releases new statistics, they’re given to the media in advance. The media need this lead time to be able to format all their articles and tables and make sure everything is correct and works properly: this is fair enough, given that they’re the data they’re working with lives in lots of Excel spreadsheets, with multiple sections, differing layouts and everything else you really don’t want if you’re tasked with this kind of job.

Given what they have to work with, the BBC’s anger is understandable, but perhaps misplaced: why are we still dealing with bunch of spreadsheets in the information age? Why isn’t there an API that allows this data to be queried, or at the very least, a standard data format that doesn’t change from year to year, and doesn’t reply on proprietary technologies that are hard to work with?

An API or standard data format would allow media organisations to write code which generates the statistics they need every year. They wouldn’t have to create new tools to be tweaked and tested every time there are new statistics. Better still, it would create a market for someone to create a tool that did this for them, saving them money. Even better than that, it would allow anyone who wants to do something innovative with these statistics to do so far more easily.

I think I’m not alone in saying that the case for releasing data properly — in reusable formats, to everyone, for free, whenever it is possible — has been made, has been heard and has been widely accepted as valid.

Why are we still fiddling with messy spreadsheets, and bemoaning the fact that we only have days to do what should take hours?