Phew! It’s launched, and it works. And it all went surprisingly smoothly.

Just before Christmas, the Independent Parliamentary Standards Authority approached us to see if we’d be interested in producing the online part of their first consultation on MPs’ Expenses. This was exciting, to put it mildly — so we leapt in with both feet.

As is often the case with this kind of work, the deadlines were short and the team rushed. The consultation is only five weeks long, and there’s even less time to analyse the responses after it closes. With this in mind, we did our best to come up with a format that gives respondents a meaningful way to respond, while also making the responses as easy as possible to analyse.

Most of the questions have yes/no answers, and a few have options you can choose between. Each page has a box for free text responses, instead of each question — and the box is limited to 500 characters. We were a bit nervous about limiting input in this way, but hopefully it’s a good compromise. No one’s submitted anything nearly that long yet, so it hopefully people aren’t feeling constrained.

Given the number of questions, the length of the text and the constraints of the form, it was really important to make the site as intuitive as possible. The text boxes make it very obvious if you put too much in them. The final form at the end confirms that you really are finished, and the buttons are subtly differentiated using icons and colour. You’ll also find that using the sidebar menu to jump around is safe — even if you’re in the middle of writing a response. We’re using javascript to make sure the form gets submitted. And everything’s unobtrusive, accessible and standards-compliant. We even managed to squeeze in some RDFa.

The site is, of course, implemented using Wordpress. It’s a fantastic platform to work with. There’s no doubt that Wordpress’s general awesomeness is what made it possible to launch this quickly — and on time, and on budget. Everyone seems pleased with the way it’s come out (including a real user) so we’re pretty pleased!

Questions? Thoughts? Did it work well for you? Do leave a comment, and let us know what you think.

James Darling and I met up for a chat about some data-related stuffs this afternoon, and came across this data on average fuel prices via the Office of National Statistics. This struck us both as being very useful (any hauliers out there who want to make some a nice visualisations?) so we threw together a script to convert it into (much) more useful formats.

Check out Weekly Fuel Prices, in more formats than you can shake a stick at, here.

mySociety have added a data dump to ScenicOrNot, the site we built for them a couple of months ago. It’s got the photos and all the votes for each of the 181,300 places that have received 3 or more votes since the site launched.

If you’re one of the many people who had something to say about the voting system that ScenicOrNot uses, we hope you might have some fun playing with the raw data! If you do make something, let us know how you get on…

Isn’t secret anymore!

MySociety have just taken the wraps off Mapumental, which is a terribly clever mapping application to help you figure out where you want to live if you have a commute (and probable more besides that). We and MySociety teamed up to build ScenicOrNot, which has produced the dataset that Mapumental consults when you move the scenicness slider.

They’ve produced a video to show the site, currently in private beta, in action:

Mapumental is really quite impressive — a significant technical accomplishment as well as very useful — so significant kudos are due to mySociety. Nicely done!

Just a quickie post — ScenicOrNot, a project we recently completed for MySociety, has been featured in BBC News Online’s Technology section.

Predictably, there’s been a big jump in player numbers. We just zoomed past 3000 players, 260,000 votes and over 50,000 places rated. Thanks Auntie!

Last week, we finished a new project. MySociety commissioned us to produce ScenicOrNot. They want to create a “database of scenicness”: something that identifies pretty places and their locations. They came to us with the idea for the site, and we put it together for them over a couple of weeks.

The idea behind ScenicOrNot is simple: it’s a game, very much akin to HotOrNot, which asks users to give photographs of places a score out of 10. There’s a photo on the site for nearly every 1km grid square of the UK, so eventually, we’ll have a dataset that we can use to give every bit of Great Britain an average score for prettiness. MySociety have a very cool plan for it, and they’ll release the data as soon as it’s useful so that other people can get cracking with their cool ideas, too.

The site is fun, easy and hopefully a little bit addictive: just the right prerequisites for a site that’s croudsourcing a new dataset. If we’ve done it right, rating pictures shouldn’t feel remotely like hard work, and by the progress we’ve seen so far — 109,000 votes cast and 10% of the country rated in just under a week — it’s not going too badly.

Please do check out the site and, as always, tell us what you think about it.

One of the problems with working with large datasets — especially when you’re scraping them — is that they don’t always work the way one might think.

We’ve recently had reports that JobcentreProPlus.com turns up jobs that aren’t close to the postcode that the user entered when they started their search. We’ve done a bit of digging, and turned up two problems. Unfortunately, neither is easily fixable.

The first problem is that JobcentrePro’s website doesn’t expose very good location data. It’s often as little as “Camden Town, London” or “Sevenoaks, Kent”. For this to be useful, we need to convert it to a latitude and longitude, so we can see if it’s near the postcode you enter when you start a search.

This process is called geocoding, and it’s an inherently error-prone process. There’s often no way to tell the the difference between places with similar names. Usually, it works well enough, but sometimes, it’ll generate a result that’s unexpected: in real terms, you see a search result for a job in Glasgow when you were searching for things in London.

There’s not a lot we can do about this. If JobcentrePlus included better geographical information in their listings — like a postcode, or a latitude/longitude — we wouldn’t have to geocode things, which would be a great improvement.

Unfortunately, in this case, it gets more complicated. The second problem is that the JobcentrePlus database (which also drives their service!) doesn’t store good location data. Sometimes the location refers to the address of the Jobcentre shop. Sometimes, it’s the agency advertising the job. Sometimes, it’s the employer’s head office, but not the actual building you’d be working in if you took the job.

In summary: the way we’re forced to gather data introduces errors, and the underlying dataset has quite a few errors to begin with.

Despite this, we still think JobcentreProPlus.com is useful. Most of the time, the job will in fact be near the jobcentre, the employer’s head office or the job agency. That’s why our “distance from postcode” field defaults to 10 miles — we’re confident that that’ll be right, most of the time.

The bottom line is that the quality of our site is completely dependent on the quality of the underlying data. Until that data is better, there’s not much we can do to improve things — but we’re not too worried. From a plain reading of search results, we think we’re doing ok. This search for stuff in London returns mostly stuff that, according to the job ad, is in London.

We think it’s good enough to be useful, and that’s really our only goal.

On Saturday I was at RewiredState. A bunch of geeks got together to build things. We wanted to show government how it’s done!

At the end of the day, we each got two minutes to present what we’d done to each other, and an assemblage of government types. People did some really cool stuff, from Rob McKinnon & co’s Compani.es, which is the website that Companies House ought to have, to a reimplementation of ActivePlaces. They scraped this multimillion pound website, got all their data, and then did with it in an afternoon what the site hasn’t managed to do with a massive budget and years of time. Great stuff. Emma Mulqueeny’s written some more about the day, and the other hacks.

Sam Smith and I got together to do a project. Given the current economic malaise, it’s quite important for people to be able to find jobs, and a little birdy turned us on to the fact that the JobCentre Plus site really isn’t good. In fact, it’s quite painful. To get any jobs out of it at all, you have to fill in 4 reasonably large forms. Once you have some jobs to look at, you can’t do anything with them. There’s no RSS, you can’t get email alerts for new jobs, and you can’t bookmark jobs you’re interested in, because their URLs don’t work properly. The next time you want to find jobs, you have to go through the whole ordeal again. Bleh.

Our task was to make this better. Sam wrote some scrapers to pull down Jobcentre’s data — which was no mean feat in itself — and I made a website to display it. It’s a bit rough and ready, but it works. You can go to www.jobcentreproplus.com, search for jobs in your area, view them, bookmark them, get email alerts, subscribe in your feed reader and use the API to search and display jobs on your own site. Everything that the real site should do and doesn’t.

We didn’t realise it at the time, but there were prizes for the hacks that the organisers liked the most. Rather suprisingly — given the very high quality of all the other projects — Sam and I won!

We’re really glad that they liked it, and we hope you will too. Have a look, and let us know what you think.

Mark Little kindly reported some bugs in the ConsultationXML distribution. The INSTALL file was missing a couple of salient details:

• ConsultationXML requires the PHP HTMLTidy extension
• The pdf directory in ConsultationXML’s root directory needs to be writeable.

We had also unwittingly left some Javascript in the codebase, which was responsible for displaying the welcome page that you’ll probably have seen in the sandbox. This isn’t supposed to be a part of the software — only the demo — so we’ve removed that.

For more information about ConsultationXML, or to download the new version, head over to the Labs page.

We’re terribly, fantastically pleased to announce that after a bit of wrangling, Steph Gray and I are able to release ConsultationXML as open source software under the GNU Affero license. The recent report on open source software in Government hinted that departments ought to try to release source code for the software they commission, and we’re delighted to be (we think!) the first to do so.

We’re not sure who will want to play with it yet. We hope that other departments will want to deploy and use the tool to improve their consultation offerings. It may be that people in the private sector will find some use for it. People have already used ConsultationXML for really neat things that we didn’t expect, so anything could happen: which is, of course, the point.

We ran this by the renowned geek-come-blogger-come-minister, Tom Watson, who had good things to say:

I think this is a great tool. We’ve just announced the Open Source, Open Standards and Re–Use report on the use of open source software in government, an element of which was to encourage government to contribute to the world of open source software, and this is the first practical expression of that goal.

For more information about ConsultationXML, and to download it, head over here.