breaking news social media search tool for journalists

Today I’ve been working on a tool – I’ve dubbed it UGC Finder
– for journalists that uses Yahoo Pipes to aggregate and filter the
results of keyword searches for tagged content and conversations in
social networks and media sharing sites.

I’m rapidly developing a serious case of Yahoo Pipes addiction but with good reason – if you’ve got a good idea it’s remarkably easy to knock together an almost working demonstration, which is what I’ve been trying to do today.

Historically, when a news story broke, journalists either went to the scene themselves or phoned one of their contacts who could go out to confirm the story and gather interviews, photos and other content.

In recent years, the "citizen journalist" model has developed, with journalists and news organisations pro-actively asking their audience to submit their accounts, photos and videos from the scene. The proliferation of cameraphones has led to a dramatic increase in the amount of such content submitted by eyewitnesses, often flooding news organisations with vast quantities of mostly unusuable quotes and photographs.

One way of dealing with this is to return to the old method of going out, finding and speaking with eyewitnesses in the places where they participate online. Or, as Jeff Jarvis brilliantly put it, to keep our antennae up wherever conversations are taking place:

"And so the key skills in a newsroom will not be to get reporters to
the scene — that will come later, after the news happens — but to have
antennae up to listen and find news reports as they happen, as people
link to what’s happening. You can’t possibly have enough reporters,
editors, producers to do that on your own. You need to have lots of
friends who’ll alert you: When I put up a link here to something I find
compelling — or even embed and broadcast it here, live — will I also
alert CNN? I don’t know. Would you? Do you have such a friendly
relationship with CNN? Maybe that will happen but that, too, is
insufficient. So you need to use every tool that’s available — the
Technorati of the live video web — to see what’s happening in the
world."

Today I’ve been working on a tool – I’ve dubbed it UGC Finder – for journalists that uses Yahoo Pipes to aggregate and filter the results of keyword searches for tagged content and conversations in social networks and media sharing sites.

I’ve set up keyword searches for:

• Explosion
• Evacuation
• Bomb

On the following services:

• Tweetscan (search of public tweets on twitter)
• Flickr (photosharing)
• Youtube (video)
• Technorati (blog search)
• Icerocket  (blog search)

I’ve truncated the content from each of these, and applied several filters, in an attempt to  find a good balance between usefulness and being drowned out by irrelevant results. I’ve set it up so that, where possible, I’ve filtered out non-unique results.

We had the chance to test UGC Finder this afternoon when there was an explosion at a pub in Leeds. It did, as I’d expected, find some reaction on twitter and livejournal but updating with new posts was a little bit on the slow side. But as an easy to use tool for journalists who might not know much about using the internet beyond sending an email and searching on google, it does seem like I’ve come up with something useful.

One shortcoming I realised when building the tool is that adding more search terms would be arduous but necessary if I ever wanted to be able to track breaking news not involving an explosion, evacuation or bomb.

With a bit of help from Chris and Ryan, I’ve been trying to come up with a way of taking the BBC News Breaking News RSS feed and extracting key words from that, then plugging those keywords into the tool so as to have new search terms created, on the fly, by our coverage of breaking stories.

I’ve managed to achieve this with a second pipe which:

• takes the BBC News RSS feed
• extracts keywords from the titles of articles
• uses those keywords in a flickr search
• output is images from flickr tagged with keywords from breaking news stories

What I’d really like to do is figure out a way of combining these so that the BBC News RSS is used to create keywords which are then plugged into the content searches I created for the first pipe above. This would create a tool which would be fairly automated and easy for any journalist to use. On top of this there could be a search box, like the one Martin Belam has used for Chipwrapper (which lets you keyword search ALL UK based news feeds), so that users could configure their own searches as well. 

You could also add location into the mix by, for example, combining the above with a geo-annotated news feed , TwitterLocal and the geofeed rss functionality of flickr.

I’ve just about hit the ceiling of my Yahoo Pipe making capabilities so here’s the idea, and above are some links to my demonstrators – now it’s over to you to make something really cool out of it. Drop me a line when you do.