Open Journalism & the Open Web

great stuff. a lot to explore

later

thank you !

thanks!

veryinteresting, thanks

thanks!

Good stuff

The sound was crystal clear this time!

Outline of assignment: 

+ Pick a subject matter you want to investigate
+ Identify a dataset or datasets that will help you formulate your story. For this exercise, only pick a dataset that is already available on the Web, e.g. via Data.gov or a state- or city-level data website. Plan:
- What you need to do to clean these data, e.g. remove columns, perform functions on other columns, remove special characters ...
- The schema you'll make to house the dataset(s) — column names, column types
- What are you doing with this data — Are you proving an existing thesis (and if so, what is it) or are you seeking to better understand the subject matter (and if so, what questions will this data answer)?
- What will your queries look like? Are you going to join multiple databases together? If so, how, and why will the results of the join be accurate and relevant? Hackers: If possible, write out a query in SQL.
- How will you express the results of your inquiry? In the text of an article? As a chart or graph? Informing a search for relevant photos and video?
- What questions won't the data answer that you want to address in your project? Who will you turn to as you start looking for those answers?

http://www.guardian.co.uk/news/datablog/2010/oct/01/data-journalism-how-to-guide

http://www.datadotgc.ca/

Or, if you're Canadian:

http://www.slideshare.net/openjournalism/data-presentation-5415254

Where to find tools for this week's assignment:http://toolkit.snd.org/ & http://help.hackshackers.com are great places to start your search.

@David -- it is a bit of a curious service; thankfully, we don't have access to it from Canada! :)

it's oesn't offer anything like a sustainable wage

I have moral issues with Mechanical Turk

http://projects.latimes.com/value-added/

@Nicholas: Yea definitely a problems there. But you are only giving a little information to each person. Especially if speed is of the essence.

right now i'm quite keen on and would appreciate feedback onhttp://canbudget.zooid.org/wiki/David_Mason/P2PU_week_1_assignmentwhich describes importing the G20/G8 budget via crowdsourcing and trying to make it presentable, participatory and reusable

Michael - True, but I'd be concerned about releasing my source materials pre-publication if I used Mechanical Turk.

@David - Awesome

You had mentioned mechanical turk earlier. Amazon has a service you can pay people to do some of that.

Hundreds of pages about fire hydrants? Oh my word... :\

a colleague and I sorted through child protection service records in Texas to identify child care centers that were consistently performing poorly -- like repeated cases of abuse, health hazards -- and what they were doing to improve/ if the state was being proactive enough and holding them accountable. I think we just used Excel and came up with a way to sort the records by certain key words

it was years ago, just have a presentation now.http://zooid.org/~vid/presentations/publicwhip/

@David -- Do you have the URL for that example? The Hansard data ?

On the subject of OCR, I guess a lot of journalists get hard copies (paper copies) of important documents, that need read by OCR for being converted to digital data? That's a question to everyone, I guess.

there's a real movement around this worth tapping into

scraped the Canadian Hansard but a year later they opened it up in a structured format.

I prefer text :)

although there was ocr version of the text available too

Steve Myers, no it was only 18 pages so a few people did it "crowdsourcing"

@david, you used Mechanical Turk for that?

we did the same thing for canbudget with *faxed* government data

what program do you use to convert out of pdf to machine readable format

Ah, good point.

Steve Myers, but the missing data becomes more evident and there are many successful cases of requesting data

Interesting point on APIs there, that they will only help you get at the info that the agency wants you to get at.

http://scraperwiki.com

http://maplight.org/

wikileaks

http://sunlightfoundation.com/

Other questions?

http://code.google.com/p/google-refine/

Steve, Sarah, Tim: great, thanks! In the queue.

I am interested in learning ways of acquiring data. You mentioned using API, can you go over how that works, and other ways to go about acquiring data?

@Mariano: Shoot me an email and I'll try to help. :)

Nick, have you used google refine/freebase gridworks at all?

I've lost my partner too... :-(

Can you describe how you'd use Socrata and what it would tell you?

:)

Folks: Please queue up your questions for Nick & post them here, or use the raise hand feature.

http://help.hackshackers.com/questions/tagged/database

@David -- Heh.

pedantic: i bet you use vim, not vi :) http://www.vim.org/

Slideshare version of Nick's presentation:http://www.slideshare.net/openjournalism/data-presentation-5415254

http://1rick.com/project/igor

Thank you, Phillip.

@Chris: If you can shoot an e-mail to me, Dani, or Sarah, we can try to help out with that. :)

Anyone lost their course partner? I have! :-/