Transparency is the New Objectivity

This phrase, attributable to David Cohn's blog post in Media Shift Idea Lab, strikes a chord in me. It labels a complex idea handily. When we were growing up, the media would report stories based on "high White House sources." It was okay that we had to take the information on faith, because the stories were "objective," meaning that they would be careful to say "Congressman Loser denies these charges." Readers could rest assured that every claim had at least two sources, principals would be reached for comment, and opposing views would be at least mentioned.

But that traditional journalistic contract has broken down in recent years, probably reflecting news media's dire economic straits more than anything. 24-hour news cycles and infotainment have turned the typical story into something of a farce.

After more than a few reporters were found to be making things up, media outlets began explaining why sources were anonymous--"because the investigation is ongoing, because she is not authorized to speak to the media." Baby steps toward transparency.

It is so ubiquitous it's hard to summon outrage, but isn't it infuriating when the TV news says President Obama spoke and shows us mute video of him speaking, but then has the gall to tell us what he said? I understand that one of the finest orators of our time needs to be summarized for a short news show, but the reporters are not only selective in their topics, they can and do add significant spin. "The president said he was confident health care reform would pass," can be given a range of nuance from sarcasm to derision.

Journalists were never objective, but the old forms made us all complicit in a charade that worked, more or less. But it's time for a better approach before all the eyeballs the media is aching to monetize are glazed over. Transparency means a link to the video of the full speech, or the PDF of the actual document. If the story is about a new website, give readers the link even though it means they might (gasp) click away from your site.

Where's Clay Shirky?

Clay Shirky, a professor at NYU, among other things, understands the coming news media better than anyone I know. He sees things clearly from the point of view of readers', communities', and civilization's needs, as opposed to most bloggers on this subject who keep trying to see how the web and the changes it brings will affect existing news media.

So, I enjoy Clay Shirky's blog (http://www.shirky.com/weblog/). But it hasn't had a post since May 31 (and the last from Mr. Shirky was April 15). I took the summer off, too, but I miss Shirky's insight.

Shirky, C. Why Micropayments Don't Work, Sept. 2003

______. Newspapers and Thinking the Unthinkable, Mar. 2009

Changes are coming

Google feels the newspaper industry's pain. Ten days ago, Google proposed a micro-payment system that would allow news publishing sites to charge their readers without burdening those readers with the need to set up payment accounts with each source they would like to read.

In a written response to the Newspaper Association of America, Google offered a single sign-on approach that would allow a reader to create an account and pay Google to visit paid news sites.

"We envision the typical scenario to be where a user pays a monthly fee for access to a wide-ranging package of premium content. One example of a "package" might be full access to the WSJ; another "package" might include the top 10 business publications. Google believes that there is real power and benefit to publishers in providing these sorts of broad, multi-publication access passes."

The same section of the document explains that Google doesn't think micro-payments would be norm for accessing content. Sure, people might pay a dime to read a specific one-off article, but more typically, people would subscribe to bundles like cable TV. My instinct says they're wrong--Everyone chafes under the cable TV industry's inflexibility--your package includes the Golf Channel whether you want it or not. I think micro-payment will skyrocket like Facebook as soon as a viable exchange is available.

Either way, this is a good development for hyper-local, community news. Your "package," if it's done right, might include the WSJ and some business sites, plus electives like your hobby and your community. You get access to premium content on TennisPro.com and the SantaMonicaObserver.com, while your neighbor's package includes CatFancy.com instead.

Google was very helpful to the Los Gatos Observer. I had to jump through some hoops (updating the sitemap for each posting, using four-digit identifiers for articles) and bug the Google News team by e-mail for a while, but once we got in the rotation, our stories were at the top or near it on any search for 'Los Gatos' and we were tied in with the Google Alerts feature. Scott Seaman, our police chief, told me his Google Alert on 'Los Gatos' had introduced him to our paper.

Google also announced AdExchange, a mechanism that elects Google as broker between ad buyers and news publishers. Again, good for publishers--it makes ad revenues easier to attract and track.

Andy Hertzfeld, one of the original Macintosh designers, is now "thinking about new ways to search for the news," according to Google Labs. Perhaps one of these ways is FastFlip, a clever use of page images (similar to Amazon's SearchInside) that allows readers to browse news articles by page. Made public this week, everyone is invited to check it out.

Also interesting is the NewsTimeline, a Google Labs project from last April. It shows a calendar-based view of a few articles. The week view only shows 3 articles per day, and the automated article tease doesn't work well. It's also slow. Click on Week (the default is Day) and it "thinks" for a long time. Automated front page layout is a known problem in this biz, and let's put it this way: If someone solves that difficult problem, NewsTimeline will benefit from that solution. Just stacking things by date is orthogonal to a satisfactory approach.

How Google Plans to Save Newspapers Niroj Chokshi, The Atlantic, 9/10/09

An iTunes for News?

David Carr, writing in the New York Times last January, suggested that someone should write an iTunes for News. The idea is related to Walter Isaacson's "How to Save Your Local Newspaper" in Time Magazine in February. Both think that Ted Nelson's mid-1960s "micro-payment" concept will fund online news dissemination, now that advertiser-supported print media seems dead.

I don't agree with this premise. When I want to possess a song, or a photo, or an e-Book, I appreciate a simple payment process like iTunes. I want; I buy. But not for news or community information.

Incoming e-mail from Aunt Sally. Click here to read the message for just .99¢.

That wouldn't work, right? Not just because you're used to free e-mail. Maybe Aunt Sally is a pain in the neck. You're not going to pay for her to nag at you. Many--maybe most--local news is someone you may not know informing you about an upcoming event, or a charity opportunity, or the latest happenings that you weren't involved in. Unless your family is participating, you wouldn't pay to read a season summary by the Little League coach, but you might read it if you came across it.

To stay generally informed, so that your life isn't constantly rocked by sudden changes, one keeps an ear to the ground. One day, you see the circus packing up and leaving town--why didn't I hear that they were coming? Because I didn't sign up for a Google alert on 'circus?' I don't like the circus that much. I don't care about circuses in general. I only care when they are coming to my town, or my hairdresser will be guest lion tamer for them. News is context-sensitive.

So, you scan that summary by the Little League coach, skimming over the long list of names you don't know. You get the gist--those kids two towns over were too much for our kids this year, but just wait. And then, near the end, the coach thanks the local electrician for putting in the new scoreboard for free, and you just paid him for work on your house, and you think, "Yeah, he's a good guy in town." You're incrementally more informed about what's going on.

That's how it works. Not for the New York Times or Time Magazine, but that's how it works at the local, community level. The way to pay for it is by assessing the local electrician and all the other businesses in town. They get the direct financial reward. Individual readers aren't going to click to pay for an article, not even two cents, and they shouldn't have to. The good news is that the cost of putting community news online is considerably cheaper than printing the New York Times. Divide it by the number of community-oriented local businesses, and the cost per business is iTunes-like.

Unlearning the past to understand the future

I'm not happy with that headline, but I wanted to summarize the key points in Clay Shirky's Here Comes Everybody, and particularly his conclusion that:

The future belongs to those who take the present for granted.

Shirky (who is 7 years younger than I am, by the way) says that older people's real-world experience is usually an advantage. Younger people, he says, "overestimate mere fads" and "make this kind of error a thousand times before they learn better." But in revolutionary times (such as after the invention of the printing press or, say, right now), experienced people make the opposite mistake.

"When a real, once-in-a-lifetime change comes along, we are at risk of regarding it as a fad." He recounts how grown-ups debated whether to allow high school students to use calculators in class when he was young. "The unspoken worry [of the grown-ups] was that, since calculators had appeared so suddenly, they might disappear just as suddenly. What none of [them] understood is that there would never again be a day when we needed to divide two seven-digit numbers on paper."

"I'm old enough to know a lot of things, just from life experience. I know that newspapers are where you get your political news and how you look for a job. I know that music comes from stores. I know that if you want to have a conversation with someone, you call them on the phone. " Shirky says he has had to unlearn these things and many others, "because those things have stopped being true."

When I started the Los Gatos Observer, I was competing with the traditional local paper, the Los Gatos Weekly Times, which was delivered every Tuesday. I posted articles 24/7 and my competitor missed the "news" by ten days or more (until they made an effort to hold the presses for Friday news). I worked hard to make my site fresh and we posted new articles at all hours. But I was constantly asked, "When does it come out?"

I don't blame people for not rushing to change how they think of things, but when you focus on a particular aspect of it, such as how we, as citizens, are informed about incidents and events in our community, the obviousness of the future that the recent revolution in communications has wrought seems impossible to miss. "The newspaper is dying," people say, reading an article about it in the newspaper. "Too bad. I like my morning newspaper."

There's nothing like riding a horse, or camping under the stars, or a number of other things that progress has taken from daily life.

The question to ask, of course, is "How can I best stay informed about my community?" I think the answer is a local, community editor and contributing neighbors who compile and categorize items (articles, photos, videos) that might be of interest.

The next question, "What is the best way to communicate these items to everyone in town?" has an obvious answer. Who in their right mind would suggest printing thousands of copies of the items on literally tons of paper, then hiring people to deposit them on every doorstep? A bundle of newspapers on the curb waiting to be distributed may evoke nostalgia, but it also represents a significant and needless environmental impact from depleted forests to burgeoning landfills.

You could e-mail these news items to everyone, or text them, but "push" media--you watch when I say you can watch--is being replaced by "pull"--opt-in marketing, TiVo, and so on. Better to post the news items on a web site and let subscribers visit when they want. Obvious, but not to some of my neighbors.

"I now spend more energy on weeding than planting," Clay Shirky writes. "Which is to say, more energy trying to forget the irrelevant than learning about the new."

Thinking about tagging

Tagging is a way to label articles and other information objects (such as events) for retrieval and clustering. A good community news web site will support searching articles, for example, and will find all articles that contain a word (or phrase). At their simplest, tags allow an article to be found even if it doesn't contain the search term.

Many times, articles will talk around a subject--the elephant in the living room. When we talk about cutting services and increasing public sector revenue, it is understood that we are talking about "budget," "city finances," and "Bud Conway," the city finance manager, even though those words might not be in the article (let's pretend). The extra words become tags so that the article can be retrieved for a searcher who would likely care.

In searching, you have precision and recall. Precision says that all search results will match the query--there will be no irrelevant articles. Recall says that all articles that match the query, even at the fuzzy edges, will be in the search results, along with a percentage of irrelevant ones. For most applications when we are searching a database of community articles, we want high recall, and tags help with that.

Tags also cluster information. The author of an article can tag it with "News" and readers can browse the News category and find the article, along with other news stories. If the author tags the article with two tags--"News" and "Schools," say--then the article appears in both categories (high recall in the browsing scenario).

Then, there is the notion of readers tagging the stories, the way Flickr lets viewers tag photos. It's a little different with photos, obviously, because the photos can't be searched without captions or tags. With articles, tagging not only potentially helps searching, but it's also a way of interacting with the article--it's a measure of popularity, or interest in an article. Both articles were viewed 1000 times, but this one had a lot more tags proposed. But do we really want readers deciding that something is "News?"

Ed Chi and Todd Mytkowicz of Xerox PARC wrote about aspects of tags recently. They explored the basic mystery of why uncontrolled reader tagging generally works. Users can tag articles with any combination of letters and numbers. Anarchy. Chaos. But, by most measures, the tag system achieves its goals. "Social tagging...[is] attempting to solve a mapping problem," they write. Users are collectively creating a map that will enable them and others like them to navigate the territory efficiently in the future.

One reason I think the potential for anarchy isn't realized is that griefers--vandals--like their mischief to be visible. If I tag a gossip story as "News" and it now appears on the Front Page, it's like spray-painting a mustache on a billboard--everyone can see how clever I am. But if I tag an article as "asdf," it is easily ignored.

Col Needham, the creator of the Internet Movie Database (imdb), has written that a few fairly basic ad hoc tweaks to the search interface greatly improved the searchability of the very large database. Finding a movie like "20,000 Leagues Under the Sea" would be difficult without synonyms like "Twenty Thousand," "20 Thousand," "20000," and so on.

Mindful of Needham's experience, I think a community news web site will need both an anarchic reader tag system alongside a more controlled tag vocabulary used by authors and editors. We'll want to suggest tags for use by authors that perhaps come pre-synonymized. One selection by an author from a drop-down list might add 3 tags. For the reader: "Propose a tag. Type it here. We'll let you know."

By the way, Chi and Mytkowicz found that entropy makes tag systems work less efficiently as they grow. Tags become less descriptive over time and "tags are becoming less meaningful in regards to providing salient navigability."

"Even with a tagging system, the navigability of the document set is becoming more challenging over time. One way for users to respond to this evolutionary pressure is to increase the number of tags they use to specify a document."

Chi and Mytkowicz find that, as the number of articles grows, users are increasing the number of tags they apply, and searchers are using more search terms. They report that Yahoo!'s average query length was 1.2 words in 1998, 2.5 words in 2004, and 3.3 words in May 2006.

Forbes.com offers a working example

Wednesday, Forbes.com posted a near-perfect example of the difference between local news and remote, national, mega-news. They posted Matt Woolsey's Home of the Week, a $6.7 million estate on 20 acres, apparently in Santa Clara County. It's not clear where the home of the week is because the headline says Los Altos, but the article says Los Gatos. I live in Los Gatos, and the photo looks like Los Altos to me.

I pointed out the error in an e-mail to Forbes yesterday, they haven't fixed it, so I think they're fair game. (I have had my suggested corrections implemented by several national news sites; Forbes is just lame.) The trouble is, Mr. Woolsey's text describes Los Gatos--he mentions the zip code, the proximity to the redwoods and the ocean, and the Spanish translation of the name. The url is http://www.forbes.com/2009/05/20/los-gatos-home-lifestyle-real-estate-home-of-the-week.html--the name Los Gatos is embedded in the url. How embarrassing if the home is really somewhere else, which I suspect it is, since I wrote a book on Los Gatos architecture and the home doesn't look familiar.

From Wikipedia, we learn that Mr. Woolsey is 28 and was born in San Francisco. He writes for Forbes (limiting himself to real estate, lifestyle, labor, baseball, transportation, and small business) but he has also appeared on CNN, CNBC, NPR, Fox News and others.

Mr. Woolsey is on the Steve Lopez track--journalists who get so big they can't respond to an e-mail and who don't stay in one place long enough to understand what they're talking about. At least Lopez still only covers Los Angeles. Woolsey is trying to cover six desks nationally.

I'm whining about this because I believe the future needs journalists, but that they must be embedded in their beat. This shoddy "Home of the Week" nonsense only works in national printed media. On the web, his story about California real estate that does such a great job filling space between Breitling ads in Forbes in Manhattan instantly reaches a guy who wrote a book about architecture in Los Gatos and he's busted.

The web keeps journalists--anyone who tries to tell the rest of us how it is--honest. I've written many wrong things in thousands of Los Gatos stories, but I corrected them as soon as I was told they were wrong. When the facts were in dispute, reader's comments presented alternative viewpoints. The articles are part of the archival record, and because they were corrected and responded to, the archives are more valuable. I'm guessing Matt Woolsey and Forbes don't think that way with respect to left-coast "Home of the Week" fluff.

UPDATE: It took my wife, Peggy, five minutes to determine that the house is in Los Gatos, after all. It's on the hillside west of Highway 17, just south of Wood Rd. She found it on realtor Michael Nevis' web page. This means that the only correction Forbes needs to make is to the headline. Oh, and the "gated community" comment has to go unless two or three homes sharing a gate qualifies as a gated community.

FURTHER UPDATE: 9/22/09 It's been 3 months. The article is still online and the headline still says Los Altos.