Thinking about tagging

Tagging is a way to label articles and other information objects (such as events) for retrieval and clustering. A good community news web site will support searching articles, for example, and will find all articles that contain a word (or phrase). At their simplest, tags allow an article to be found even if it doesn't contain the search term.

Many times, articles will talk around a subject--the elephant in the living room. When we talk about cutting services and increasing public sector revenue, it is understood that we are talking about "budget," "city finances," and "Bud Conway," the city finance manager, even though those words might not be in the article (let's pretend). The extra words become tags so that the article can be retrieved for a searcher who would likely care.

In searching, you have precision and recall. Precision says that all search results will match the query--there will be no irrelevant articles. Recall says that all articles that match the query, even at the fuzzy edges, will be in the search results, along with a percentage of irrelevant ones. For most applications when we are searching a database of community articles, we want high recall, and tags help with that.

Tags also cluster information. The author of an article can tag it with "News" and readers can browse the News category and find the article, along with other news stories. If the author tags the article with two tags--"News" and "Schools," say--then the article appears in both categories (high recall in the browsing scenario).

Then, there is the notion of readers tagging the stories, the way Flickr lets viewers tag photos. It's a little different with photos, obviously, because the photos can't be searched without captions or tags. With articles, tagging not only potentially helps searching, but it's also a way of interacting with the article--it's a measure of popularity, or interest in an article. Both articles were viewed 1000 times, but this one had a lot more tags proposed. But do we really want readers deciding that something is "News?"

Ed Chi and Todd Mytkowicz of Xerox PARC wrote about aspects of tags recently. They explored the basic mystery of why uncontrolled reader tagging generally works. Users can tag articles with any combination of letters and numbers. Anarchy. Chaos. But, by most measures, the tag system achieves its goals. "Social tagging...[is] attempting to solve a mapping problem," they write. Users are collectively creating a map that will enable them and others like them to navigate the territory efficiently in the future.

One reason I think the potential for anarchy isn't realized is that griefers--vandals--like their mischief to be visible. If I tag a gossip story as "News" and it now appears on the Front Page, it's like spray-painting a mustache on a billboard--everyone can see how clever I am. But if I tag an article as "asdf," it is easily ignored.

Col Needham, the creator of the Internet Movie Database (imdb), has written that a few fairly basic ad hoc tweaks to the search interface greatly improved the searchability of the very large database. Finding a movie like "20,000 Leagues Under the Sea" would be difficult without synonyms like "Twenty Thousand," "20 Thousand," "20000," and so on.

Mindful of Needham's experience, I think a community news web site will need both an anarchic reader tag system alongside a more controlled tag vocabulary used by authors and editors. We'll want to suggest tags for use by authors that perhaps come pre-synonymized. One selection by an author from a drop-down list might add 3 tags. For the reader: "Propose a tag. Type it here. We'll let you know."

By the way, Chi and Mytkowicz found that entropy makes tag systems work less efficiently as they grow. Tags become less descriptive over time and "tags are becoming less meaningful in regards to providing salient navigability."

"Even with a tagging system, the navigability of the document set is becoming more challenging over time. One way for users to respond to this evolutionary pressure is to increase the number of tags they use to specify a document."

Chi and Mytkowicz find that, as the number of articles grows, users are increasing the number of tags they apply, and searchers are using more search terms. They report that Yahoo!'s average query length was 1.2 words in 1998, 2.5 words in 2004, and 3.3 words in May 2006.

Forbes.com offers a working example

Wednesday, Forbes.com posted a near-perfect example of the difference between local news and remote, national, mega-news. They posted Matt Woolsey's Home of the Week, a $6.7 million estate on 20 acres, apparently in Santa Clara County. It's not clear where the home of the week is because the headline says Los Altos, but the article says Los Gatos. I live in Los Gatos, and the photo looks like Los Altos to me.

I pointed out the error in an e-mail to Forbes yesterday, they haven't fixed it, so I think they're fair game. (I have had my suggested corrections implemented by several national news sites; Forbes is just lame.) The trouble is, Mr. Woolsey's text describes Los Gatos--he mentions the zip code, the proximity to the redwoods and the ocean, and the Spanish translation of the name. The url is http://www.forbes.com/2009/05/20/los-gatos-home-lifestyle-real-estate-home-of-the-week.html--the name Los Gatos is embedded in the url. How embarrassing if the home is really somewhere else, which I suspect it is, since I wrote a book on Los Gatos architecture and the home doesn't look familiar.

From Wikipedia, we learn that Mr. Woolsey is 28 and was born in San Francisco. He writes for Forbes (limiting himself to real estate, lifestyle, labor, baseball, transportation, and small business) but he has also appeared on CNN, CNBC, NPR, Fox News and others.

Mr. Woolsey is on the Steve Lopez track--journalists who get so big they can't respond to an e-mail and who don't stay in one place long enough to understand what they're talking about. At least Lopez still only covers Los Angeles. Woolsey is trying to cover six desks nationally.

I'm whining about this because I believe the future needs journalists, but that they must be embedded in their beat. This shoddy "Home of the Week" nonsense only works in national printed media. On the web, his story about California real estate that does such a great job filling space between Breitling ads in Forbes in Manhattan instantly reaches a guy who wrote a book about architecture in Los Gatos and he's busted.

The web keeps journalists--anyone who tries to tell the rest of us how it is--honest. I've written many wrong things in thousands of Los Gatos stories, but I corrected them as soon as I was told they were wrong. When the facts were in dispute, reader's comments presented alternative viewpoints. The articles are part of the archival record, and because they were corrected and responded to, the archives are more valuable. I'm guessing Matt Woolsey and Forbes don't think that way with respect to left-coast "Home of the Week" fluff.

UPDATE: It took my wife, Peggy, five minutes to determine that the house is in Los Gatos, after all. It's on the hillside west of Highway 17, just south of Wood Rd. She found it on realtor Michael Nevis' web page. This means that the only correction Forbes needs to make is to the headline. Oh, and the "gated community" comment has to go unless two or three homes sharing a gate qualifies as a gated community.

FURTHER UPDATE: 9/22/09 It's been 3 months. The article is still online and the headline still says Los Altos.

The Wisdom of Crowds In Reverse

An article in this morning's Wall Street Journal caught my eye. Although it is not the point of the article, I was intrigued by the results of a study led by social influence researcher Robert Cialdini.

When hotels left a message asking guests to please reuse their towels to help the hotel save energy, 84% of the guests refused. When the message was, instead, "Partner with us to help the environment," almost twice as many guests, 31%, reused their towels.

When the message was changed to "Almost 75% of guests reuse towels," 44% of guests responded by reusing their towels. And when the message became even more local and specific--"75% of the guests who stayed in this room reuse towels"--49% of guests began reusing their towels.

Local and specific. "Dine out more often" or "Denny's: Real Breakfast 24/7"--these messages are easy to tune out. But to a local audience: "New desserts at the Cup & Saucer," or "Cup & Saucer Sponsors Little League" may sound corny, but unless you're just passing through, they are important stories that you want to know about. "Considering retirement, Smith may close Cup & Saucer" is a grabber.

I suspect these local messages are sticky for the same reason we want to know what guests who stayed in our hotel room did with their towels. Local people, specific places where we go or could go to spend our money--these are a degree of magnitude more important to us than some international chain's ads. National advertisers are so used to the automatic multiplier of media like television that they don't seem to realize the impact that the web will have on community, localcast, advertising.

It's a new, over-wired world

The Onion News Network reports that police were able to figure out what caused a fire at a fictional crowded college party because there were 43,000 photos taken and twittered at the party. My wife points out that this is not just funny, it's a glimpse of the future:


Police Slog Through 40,000 Insipid Party Pics To Find Cause Of Dorm Fire

Update 5/20: The morning news featured the preliminary murder trial of Johannes Mehserle, in which the witnesses were called to testify about their camera and cell phone videos of the shooting. The gunshot could be heard on some of the video and the victim's relatives reportedly reacted each time the sound was played from another viewpoint.

Dean Singleton: Dawn Breaks Over Marblehead

Dean Singleton, the news aggregator who bought up dozens of once-great newspapers in the fin-de-siecle of their existence, wrote a memo to his editors on May 8. He just figured out that the web is important, too, but he has decided that giving away news articles online devalues his dead-tree newspapers. This may sound sarcastic and hyperbolic, but it is no more than the truth.

...we continue to do an injustice to our print subscribers and create perceptions that our content has no value by putting all of our print content online for free. Not only does this erode our print circulation, it devalues the core of our business - the great local journalism we (and only we) produce on a daily basis.

Judging by the San Jose Mercury News, it is a stretch to call it "great local journalism" these days. Singleton says it's time to end the free ride, time to tell

our online audience (who don’t buy the print edition), that if you want access to all online content, you are going to have to register, and/or pay. If a non-subscriber wants the newspaper content in its entirety online, they will be directed to some sort of registration or pay vehicle (and if they are a print subscriber, they will have full access at no charge). To be clear, the brand value proposition to the consumer is that the newspaper is a product, whether in print or online, which must be paid for.

Singleton clearly thinks, in the Internet age, that the best way to disseminate news or communicate with an audience is to print the message on paper and place it on doorsteps. Wow.

To reach a younger audience, he proposes a new kind of regional news web site that will include some user-generated news and be "actively managed" to present breaking news. The key will be to differentiate from the existing newspaper.com kind of site and not just present printed newspaper content. He calls the new vision "news.com."

The most interesting proposal is a local site:

We will build a new local utility site (Local.com), which is an ecosystem of local information, resources, user content, shopping guides, and marketplaces. This site will be focused on a younger audience as well as other targeted audiences based on demographics which are attractive to our current and potential advertisers. We have the advantage of being the trusted source of for news and information in our communities and have a large base of traffic to feed into Local.com.

Local.com will leverage existing newspaper content and existing traffic, and we will add new content (such as Entertainment/Lifestyle) to target a younger audience. Central to this local site will be an aggregation of city or community sites (in the YourHub model) and marketplaces.

Local.com will be the ultimate site for people to find stuff, do stuff, and get stuff done in their local market.

This sounds okay, except that we've had the YourHub model off the San Jose Mercury News website for two years now, and it stinks. Go to the mothership web site, click on "Your City" and find the name of your city in the list. Then upload your photos and comment in the forums. Go on. What are you waiting for? You don't find this compelling in the slightest? Well, what's wrong with you?

Thanks to fellow blogger and former newspaperman Gary Scott for posting the entire memo.

Closer to the Source

I just realized why I tend to stiffen when I hear someone who has been a professional journalist for decades talk about the web as a media platform. It's odd, because I have a great respect for journalists and their profession. I seem to find myself quoting Clay Shirky at every turn these days, but I agree with him. He says we don't need newspapers any longer, but we have an urgent need for journalism.

I was raised on Time magazine and the CBS Evening News more than the daily newspaper. Walter Cronkite told us what happened, and Time followed up a few days later with a much more thorough report. We've been subscribing to the Wall Street Journal for many years now, and I read it nearly every morning, 24 hours late (we don't visit the mailbox before breakfast). I read the Los Angeles Times online edition often. But I get 100% of my news from Google News these days. I scan those headlines 10 times a day.

Through Google News, I am directed to the New York Times, the Washington Post, the BBC, sometimes CNN and others. That's how I read my news--I don't know how others do it these days.

Have you ever noticed that the news is comprehensive and authoritative only to the degree that you don't know what they're talking about? I lived near Santa Monica Airport as a kid, and we saw a lot of light plane crashes. The TV news usually got some details wrong. I worked for the actor John Carradine's son, Chris, and when his father died, every news story seemed to think he had a different number of surviving children.

So, back to getting the news. When Mr. Ghanem was blown up in Beirut, I read the New York Times account and was glad that they got every detail, every nuance, exactly right. But during the time I covered the news in little Los Gatos, I watched the other media screw up details large and small. The Bay City News Service, which supplies many outlets in the area, is virtually unknown by locals. They have a source at the county fire department, because they put every call on the wire quickly. But they had no one on the scene--they were talking to a desk jockey who wasn't on the scene, either.

One story was an overturned truck just off Highway 17, in which the driver died. Every media source in the area was telling you that this accident was slowing traffic on Highway 17 because on the map it looked close. In fact, the accident occurred at a construction site 400' above the highway. I was there. Drivers had no idea it had happened; they could not glimpse so much as a Highway Patrol vehicle. Traffic was slow because it was a summer Saturday and people were heading to the beach.

This is a long-winded way to say that I like my news from people close to the source. If it's Beirut, the New York Times is close enough. But if I want to know about a crash at Santa Monica airport, now that I live 400 miles away, I'll look for a local Santa Monica news site before I'll visit the Los Angeles Times. If I can find an Ocean Park or a Mar Vista news site or blog, that will tell me more because they are adjacent to the airport, and Santa Monica is a big city, comparatively.

So, if there was a journalist who was covering the heck out of Ocean Park and Mar Vista, that would be ideal. I enjoy Steve Lopez' columns in the LA Times, but he's too big and far-ranging for a plane crash (and now he's an author and a movie character, too). When some journalists talk about the future of the news, they seem to see themselves as Steve Lopez one day. That's what bugs me.

I think the future of news will bring us closer to the source. There is no reason to pull back and let gods-gift-to-writing explain it from media central. The location-independent web lets us zoom in and find out from someone who was right there. Sometimes that might be too close--the person might not write well, or might assume more local knowledge than we have. That's when we zoom out a level--read about Ocean Park in the LA Times, for example. But we will want the capability of zooming in really, really close.

Re-reading my explanation of why a report might be too close, I realize why I respect journalists. I said we might zoom out if the on-the-scene report was poorly written or didn't know its audience. What I'm saying is that I want more journalism so that "even" local news will be reported well.

Community Garage Sale

Our town holds a Community Garage Sale every May, coordinated by the town's excellent Community Services Director, Regina Falkner, and the volunteer Community Services Commission. When I was first appointed to that commission, I assumed that the event was some sort of communal swap meet on town property. I have since learned that I'm not alone in that initial assumption, but it is wrong nonetheless.

What it is is a single day when everyone is encouraged to have a garage sale and to register their intentions with the town. The town gives sellers a kit of literature--a nice letter, selling tips, and a piece on valuing things for the IRS--and it prints a few thousand maps to the garage sales for buyers. The maps are printed on newsprint and include a table of addresses with an attempt at classification, such as "toys," "baby things," and so forth. There are usually 50 or 60 sales, so the map gets crowded.

The town has a nice website by CivicPlus, and I can't remember if they post a PDF there. But the town pays for an ad in the local weekly dead-tree newspaper to advertise the event. Costs for the town's involvement is about $6,000. We learned this because this year the event has been canceled, "for now." [Note, the 'canceled' link is to a San Jose Mercury News story, which means the link will break at some point in the future.]

Town staff--folks like energetic Volunteer Coordinator Monica Renn--enter garage sale registration information into a spreadsheet, presumably, to produce the table listing. I'm not sure how the map is done, but the town has an impressive geographic information system (GIS) and has cadastral maps of every parcel in town available online. I would bet money that the $6,000 budget for the event is simply what they pay to print the newsprint maps and what they pay for the newspaper advertisement.

I have belabored this point because it is typical of the last-century thinking that is in the process of changing. First, the town thinks it needs to be involved "coordinating" something that will coordinate itself. Second, although the town went out of its way to avoid any mention of or support for the online-only Los Gatos Observer, the annual garage sale ad represents a "gimme" to the weekly newspaper and the printer. They won't be happy at this loss of revenue--look for an editorial opposing some other town "event" in the near future.

But is this the most efficient way to do this? Of course not, and--short of legally restricting each household's sale items--it is probably the least efficient mechanism possible. We settle on one, arbitrary day, for the convenience of the organizers. The town tries to hand out the printed maps--stacks are left at the library and town offices--but most buyers just follow the (illegal) signs on telephone poles. Of course, there is no way to advertise specific items, so vague hints like "toys" are all a buyer gets.

There are garage sales here every weekend, just like everywhere else. The town's kit of information for sellers is a drop in the ocean of information available on the web. (Google returns 25 million links for "garage sale tips.")For buyers, following the signs works on any weekend.

The web is already coordinating garage sales, thank you. It's called eBay, craigslist, and Freecycle. Clearly, it's not about spreading your junk on your driveway, it's about trading things you don't need or want anymore for a little money. Most of us have things we'd rather give away than throw away, even if we're not "save the landfill" environmentalists.

But there is a downside to existing web services. You can't sell your lawn mower on eBay, because you don't want to have to ship it somewhere. Craigslist is great, but you have to interact with strangers. I haven't used Freecycle yet--I think it's a cool concept, but Freecycle is about free, non-profit stuff, and some things are worth more than $0.

A good community news site would let neighbors post sales as events with a link to a Google map. I think members of a community should have the ability to post articles, like this blog post, with photographs. So, you should be able to associate an article about your sale that features some of the items you expect to sell. The community news site would be able to produce a map of all garage sale locations for a given date.

This works like the "self-organizing groups" that Clay Shirky describes in Here Comes Everybody, currently on my nightstand. He explains that the organizational costs of some things of value are just not worth it--Hello? The town needs to save its $6,000?--but that a service can support self-organization. He cites Flickr, where people can share photos and tag them with labels like "Mermaid Parade." And just like that, you and I can find all photographs of the Mermaid Parade, without Flickr or the dozens of photographers doing any "coordinating." It's pretty obvious that's how used stuff should find a good home.