Archive for February, 2009
ConsultationXML is now Open Source 27th Feb 09
We’re terribly, fantastically pleased to announce that after a bit of wrangling, Steph Gray and I are able to release ConsultationXML as open source software under the GNU Affero license. The recent report on open source software in Government hinted that departments ought to try to release source code for the software they commission, and we’re delighted to be (we think!) the first to do so.
We’re not sure who will want to play with it yet. We hope that other departments will want to deploy and use the tool to improve their consultation offerings. It may be that people in the private sector will find some use for it. People have already used ConsultationXML for really neat things that we didn’t expect, so anything could happen: which is, of course, the point.
We ran this by the renowned geek-come-blogger-come-minister, Tom Watson, who had good things to say:
I think this is a great tool. We’ve just announced the Open Source, Open Standards and Re–Use report on the use of open source software in government, an element of which was to encourage government to contribute to the world of open source software, and this is the first practical expression of that goal.
For more information about ConsultationXML, and to download it, head over here.
The Twitter blackout and User Interfaces 19th Feb 09
Really good user interfaces are effortless. You understand what’s happening and what to do without thinking about it all. This means that the best, most essential bits of good user interfaces are often, by definition, the things you don’t notice at all.
This has really been brought home to me over the last couple of days by the Twitter blackout. This is a really nice idea for a protest — it’s really caught on and has no doubt brought the New Zealand government’s three-strikes shenanigans to the attention of lots more people than would otherwise have noticed.
It’s also made me realise how essential the avatars are to the UI. Without them, Twitter (or more accurately, TweetDeck) is quite a different place: I can no longer effortlessly tell who is speaking. Sometimes, when there are a few blacked out tweets in a row, it’s positively confusing. The username of the person speaking is there, but it’s quite small — and reasonably so, because it’s normally rarely needed.
What’s also interesting is that some people have obviously cottoned on to this already. One of the tweets in that screenshot is only partially blacked out — so it still gives you a bit of a hint as to the tweeter’s identity. Useful.
In any case, I expect Twitter will return to normal soon — hopefully as a result of the New Zealand government doing an about-turn on its crazy kick-people-off-the-just-internet-because-we-said-so legislation…
PS: that screenshot’s a fantastic example of Twitter at work, too. We start with Cory Doctorow croudsourcing ideas, presumably for a book. Next comes a funny meme, with which people are doing fun things. Finally, a product recommendation — the natural, honest, my-buddy-liked-this kind of recommendation, which is really the only kind that counts.
Comment on the Power of Information Taskforce's report 11th Feb 09
The Power of Information Taskforce have been figuring out how to liberate public sector information, how to facilitate better use of the modern, social web in government, and how to support the efforts of those outside government who are doing worthy things. All in all, they’re a great bunch of people, doing great work. They’ve just published their draft report. This is notable for two reasons.
First, the report has been published using Commentariat, a WordPress theme from the folks at DIUS that makes it easy to browse and comment on big documents. It is fantastic — it really works well — and you’d never know it was WordPress, unless you checked. It’s a great example of WordPress’s flexibility.
For me, this informal consultation exercise is characterised by its ease. I read the report online. When I had something to say, I could just fill out the comment box. I could read other people’s comments, which helped to clarify my own thoughts. The PoI team have been posting comments too: responding to people, thanking them for their feedback, letting us know when they’ve made changes. Brilliant. It’s a real conversation between people who genuinely want to seek out ideas. It stands in stark contrast to the process of formal consultation, which is stifled, slow, more or less one-way, and frequently happens after all the important decisions have already been made. This exercise couldn’t be more different: agile, easy, conversational and public. I have no doubt that the report will be better because of it.
Second, the report itself is great: it hits all the right boxes. Be active in other people’s networks. Make sure civil servants have decent ‘net access — it’s astonishing how many of them are so filtered that they’re almost useless. Support third sector developers. Invest in innovative new ideas, even if they’re high risk. Liberate geodata from burdensome licensing and fees. There are 25 recommendations in all, and they’re all great.
If you have some time, read the beta report and leave some comments. It’s a great document, and great opportunity to get your thoughts in front of people who will listen.
Scraping Civil Service Vacancies 5th Feb 09
Back in July, we were asked to make a prototype system for the Central Office of Information and the Cabinet Office.
For some time, they have wanted to put civil service job vacancies together in one place so people can find them more easily and reuse the data in their own applications, much as we have already done for central government consultations. Because of our experience with consultations, we were asked to make a prototype that uses scraping to gather data about job vacancies. Some fantastic work is underway to make this really easy by embedding RDFA into departmental websites: our part in this project was to get our hands on some data, check out what departmental websites are doing now and see if scraping could be a useful part of the solution.
We put a prototype together over a couple of months last year — altogether, it took about three weeks of development time — and I’m very happy to say that it’s now been unveiled, and you can play with it. Though the site is live, the data isn’t current: it’s only there as an example. These were all real vacancies once, but they may have been filled by now!
The site is fairly simple. Several departmental websites were scraped to get information about their current vacancies. We took that data, cleaned it up a bit and added it to a database that can be searched. Users can look for jobs by keyword (like ‘assistant‘), location (for example, a post code or place name), or all of the above plus salary.
If we can automatically identify the vacancy’s location, we geocode the it using RDFA on the site and GeoRSS in the Atom feed. We did this because it permits users to search for jobs by proximity to a location, and to import the feed into Google Maps and get an insta-mashup of vacancies plotted on a map — neat!
We think that the prototype has done rather well. It suffers from the same kinds of problems that systems relying on scraped data generally encounter: occasionally, data is missing, incomplete or in the wrong place. It would need some manual intervention if it were ever to become a real service. Thankfully, the work that’s happening at the moment to produce an RDFA vocabulary to define vacancies means that this approach shouldn’t be needed in the future.
We wrote up some recommendations as a result of doing this project: hopefully, we’ll be able to publish them at some point. We’ll definitely be helping to get departments on board when the time comes for them to start embedding RDFA in their web pages.
ConsultationXML: the mashups have landed 4th Feb 09
People have already started doing interesting things with ConsultationXML. I have to admit — I couldn’t be more pleased!
Richard Goodwin took PDF attachments from the London Gazette, uploaded them to ConsultationXML, got the HTML preview output and fed it into Wordle — and voila! A Wordle map of the London Gazette’s honours list was born.
Has anyone else done interesting things? Do let us know.
Innovation in Government: SchoolClosures.org.uk 3rd Feb 09
I was at the UKGovWebBarcamp last weekend, and among the talks I attended was one by the Directgov Innovate team. This team has been recently formed, and is a really good development. In their own words:
Directgov have created the innovate.direct.gov.uk developer network to inform the greater developer community about available resources, to provide a platform to connect with one another, and to showcase new ideas with the aim of supporting and encouraging innovation.
Over time we will provide content feeds and API’s allowing people to develop new and interesting ideas and applications for use by the greater community.
Among my questions was: what will this team actually do? I was very glad to hear that they plan to develop new sites, make APIs, make data available to people and create a community of developers who are interested in this field. Great. This is just what’s needed. What’s even better is that they’ve already delivered on that promise.
Yesterday, Tom Watson tweeted that we should have a site where people can check to see if their school is closed. Brian Hoadley and Paul Clarke at Directgov took up the challenge, and just a couple of days later, launched a new site: SchoolClosures.org.uk.
It’s pretty rough around the edges: there doesn’t seem to be much RSS support, and there’s no access to the underlying data, and — well — it doesn’t tell you whether your school is closed… but it is still useful, and it’s very impressive that it appeared so quickly, and with such little prompting.
Kudos to all involved — this is a fantastic and very encouraging start.
ConsultationXML: getting reusable data out of horrid PDFs 2nd Feb 09
Over the last few months, we’ve been working with Steph Gray of the Department for Innovation, Universities and Skills on making consultation documents easier to reuse.
DIUS are doing some fantastic things with consultations. Typically, a formal consultation is a pretty tedious process: a department will write up a big PDF document, print it, send it to some people, stick it on their website and wait for people to respond. The whole process is pretty dated: it doesn’t really take advantage of the web, and is pretty inaccessible to most people.
DIUS have started to make this process better. In July last year, they launched a consultation that tried a bit harder to involve people. They used a WordPress plugin, CommentPress, to allow people to comment on individual paragraphs in the consultation. They published a nice HTML version of the consultation document, with links and all. They even made a widget generator, so that people could embed questions from the consultation in their blogs.
Doing these things doubled the number of people who responded to the consultation, with very little extra marketing. Unfortunately, they were also pretty time consuming: turning a PDF into nice HTML is pretty labourious. They wanted to automate as much of this process as possible, to make it cheaper to deploy similar consultations in the future, and they asked us to help.
Creating all these consultation tools would be quite easy, if the data existed in a format that could easily be reused. Unfortunately, PDF is certainly not that format. It is is designed for print, and is difficult to repurpose. To make this easier, we wrote some tools to convert PDFs into very basic XML, and to allow people to extend that XML into something useful.
This human intervention is really important. It allows semantic information to be added to these documents: questions and their possible answers can be identified, and explanatory paragraphs can be linked to questions. It also allows formatting and images lost during conversion to be added back into the document, and extra formatting like links to be added.
So, with that in mind, we produced a web-based XML editor for staff in web publishing departments. The idea was to create an editor customised to the XML schema we’re using, so that people who are only just XML-literate can still use it. The editor automatically converts PDF documents to basic XML and then presents it for marking up, tweaking and generally-making-better. The result is awesome XML, usable by other tools to do neat things.
ConsultationXML is about to be deployed within DIUS, where it’ll be used by real people so we can get feedback and make it better. We’re hosting an installation here, so that you can play with it and give us your thoughts. We hope to make it better — it’s not quite finished yet — but it’s finished enough, so we’re getting it out there for people to try. It’ll be open source just as soon as the lawyers have done their thing.
Have a play with the beta ConsultationXML editor here.
Update: Steph has posted his writeup.