The Creeping Impoverishment of Online Directories and Search Engines

How the commercialization of search engines and online directories is compromising the quality of searchable information

Joe St Sauver, Ph.D.
Director, User Services and Network Applications
joe@uoregon.edu

We all depend on online website directories and search engines to help us find Internet resources, and many of us naively assume those resources will always return objective and as- complete-as-technically-possible results.

Unfortunately, with little or no fanfare, a number of online search engines and online directories have made a fundamental decision about how they're going to do business. With that choice, they've dramatically limited the richness and completeness of the results you'll see when you do a search using their service.

What is that fundamental decision? It is the decision to charge for inclusion in their listings.

Have a Commercial Website, but Don't Want to Pay?

Too bad. If you don't pay, in many cases your site will simply become invisible, not existing in that directory or in that search engine's search results. (Some pay-for-listing search sites still offer generally well hidden no-fee submission options available on an if-we-get-around-to-it basis, but that's typically the exception, not the rule.)

Pay-to-be-included plans may also often determine how quickly your site gets crawled, indexed, and listed, or how often a site gets revisited and reindexed.

Now it is certainly true that search engine and directory companies have every right to run their businesses as they please—they do own those companies after all, and it does cost money to offer a website directory or search engine. It is also true that even search engine companies need to eventually turn a profit if they want to avoid going out of business.

However, as a search engine or directory service user, you really don't want the completeness and quality of the results of your searches to be colored by the willingness of website owners to knuckle under and pay a "listing fee."

For example, consider Yahoo. If you want to list your business website in their commercial directory, you now must pay a non-refundable annual listing fee of $299 per site. Ouch!

Online search engines and directories are betting that they have become so crucial to online success that website owners will have no option but to pay to be listed (or, as they'd probably prefer to have you think about it, by being listed you'll get more than enough incremental business to recover that cost —"just like buying advertising").

What Happened to Impartiality?

But let's think about that for a minute: if you trust a search engine or online directory to guide you to the best and most relevant search results, do you want an impartial and all-inclusive guide, or a guide who's only willing to direct you to his "friends"—i.e., those sites that have coughed up la mordida ("the bite")?

Most of us do not want our search results to be modulated by the willingness of website owners to pay to be considered!

Just a Cost of Doing Business Online?

Business website owners should also beware of simply resigning themselves to paying such fees readily, assuming that they're just another unfortunate cost of doing business.

In the short term, if online directories and search engines find they can get away with demanding a "little gift" for the privilege of being listed, staff with sharp pencils at search engine and directory sites will feel free to "tune" or "tweak" the charge model they're using.

For example, does it make sense that a small entrepreneur working from his garage and an online retailing giant should pay the same amount for equivalent online directory listings?

If being listed by search engines and online directories is truly key to online success, listing companies should be able to extract more cash from a large online client than from a tiny one.

Alternatives

Is there an alternative to ever-escalating fees? Three possibilities come to mind:

1. Federal involvement. One option would be for the federal government to regulate search engine and online directory listing practices, recognizing the oligopolistic market-making (or potentially company-crippling) power that online directory and online search entities now exercise. Alternatively, it may be time for a neutral third party like the Library of Congress to offer a web directory with listings available to anyone on a free or nonprofit basis.
2. Self-regulation. A better option would be for online directories and search engine companies to regulate themselves, offering as an alternative to paid-submission-only policies a trusted centralized no-cost submission mechanism, available to any interested website and accepted by all participating search engines and web directories. Web directories and search engine sites should also make full disclosure of any commercial relationships that influence website inclusion or exclusion, or the ordering of search results returned.
3. User power. The ultimate option, however, the true power to deal with this problem, lies with end users.
   Users need to select website directories and search engines that don't "play favorites." For example, there are excellent online directories that don't charge for listing, such as the Open Directory Project (http://dmoz.org/).

The Long Term

Over the long term, I predict a new trend will become clear: online directories and search engine sites that do not charge for inclusion will become more and more comprehensive and inclusive, and thus useful and popular, while sites that continue to charge for inclusion will begin to return fewer and fewer relevant results and simply get ignored by users in favor of less biased alternatives.

As fee-for-listing sites lose their relevance and usefulness, they will also lose market influence and the ability to demand fees for listings. The only question is whether those sites will recognize this in time to pull back to a sustainable, inclusive, fair and fully functional objective model.