Although it takes a while to get there: Paying for Free Web Information
NEWSPAPER publishers and other content producers have a complicated relationship with giant search engines like Google and Yahoo. They simultaneously try to curry favor with these sites, hiring people known as optimizers with magical incantations to make articles show up high on the results pages and drive traffic, all the while grumbling that maybe, perhaps, it isn’t fair for the search engines to make copies of their material — so that it can be searched or appear on aggregation sites like Google News — without compensation.
But few are willing to speak as unambiguously as Samuel Zell, the real estate developer who intends to buy the Tribune Company, did this spring at Stanford University.
“If all of the newspapers in America did not allow Google to steal their content for nothing, what would Google do?” he asked. “We have a situation today where effectively the content is being paid for by the newspapers and stolen by Google, et cetera. That can last for a short time, but it can’t last forever. I think Google and the boys understand that. We’re going to see new deals and new formulas in the media space that reflect the reality of cost benefit.”
[...] The problem for the publishers is that the big search engines are largely happy with the access they have right now. The current system relies on robots.txt, a more-than-decade-old convention that Web sites can use to block automated spiders — computer applications that crawl the Internet indexing Web pages.
But robots.txt is an all-or-nothing proposition. And publishers are in need of a hybrid solution to the fundamental challenge that has come as content has migrated online. Enter ACAP.
[...] If ACAP is intended to be a “a cudgel to force search engines to change their business models,” writes Jonathan Zittrain, an Internet law expert at Oxford University, it “won’t be successful.” Rather, he writes, “if it takes off at all, it will likely be only with a few of its more basic features. That may well be enough for the content publishers.”