Online Marketing Newsletter - February 2004

I'm off to a search engine-marketing conference at the end of the week. I'll probably send out another newsletter within a few weeks as we learn more about the still developing changes.

First, below I've listed the four major search sites on the web. They account for over 90% of the searches done in North America. (The percentage numbers show the percentage of searches done by US web surfers at that site or network of web sites in November 2003.

Google: 35%
Yahoo: 28%
AOL: 16%
MSN Search: 15%
All others: About 6%

The New Yahoo

For the past year and half most of the results in a search at Yahoo actually came from Google.

We've all been predicting that Yahoo was going to drop Google and switch to the Inktomi search engine, which Yahoo purchased last year. In our testing over the past month or so we could see results that look like they were from Inktomi being interspersed in the results of queries so this tended to confirm the change was imminent.

But we were all surprised to learn that Yahoo has begun rolling out a new search engine. It's the beginning of a rollout that will take place over the next few weeks.

Yahoo says it developed a new search engine using technology and engineering expertise it has acquired. The new Yahoo search engine robot is called Yahoo! Slurp (For those of you who like to look at web stats you may see visits by this robot in the logs).

In our testing this week since the cutover started, the new Yahoo engine looks to be very similar to Inktomi. It indexes many of the same page elements and displays very similar results as Inktomi, which leads me to believe that the new Yahoo search is largely based on Inktomi. In fact, in our testing we see that the Yahoo Search index is already quite large. We're seeing Yahoo return about 50 to 75% of the number of results that Google returns for a specific search. Based on this we believe that the Yahoo search database likely started with the Inktomi database. It's just too big to have been built in just a couple of months.

So why would Yahoo bring out a "new" search engine instead of just building on Inktomi, which it owns? I believe it was largely a competitive move. For almost two years now Yahoo has been in the awkward position of displaying results that come from another search engine, Google. Many asked the question, “Why use Yahoo when the results are mostly from Google anyway”? I believe Yahoo didn't want to replace Google with results from Inktomi since these same results are being used by many others including major competitor MSN Search (see MSN Search below).

So now we have a new search engine on the scene that we'll be learning about over the coming months.

One of the first questions of course is “how do you get into the Yahoo search engine”. In our testing most of our clients already have some number of pages indexed so you'll probably start seeing click throughs from the new Yahoo right away.
Yahoo also announced a new Paid Inclusion option. We'll be learning the details of it and any changes with the Yahoo Directory (which has had a separate cost for inclusion for a couple of years) over the next couple of weeks. Yahoo now has a confusing array of inclusion programs and rumors are circulating that there will be some consolidation soon.

Back to top

MSN and Inktomi

Until recently the “natural” results in a search at MSN Search came from the LookSmart directory with secondary results from the Inktomi search engine. In mid January MSN dropped LookSmart (i.e. for the most part as they have still been showing some results from LookSmart).

We've all been predicting that Inktomi, which already was an important search engine, would become much more important when MSN dropped LookSmart in Janaury followed by Yahoo replacing Google with Inktomi. But now that Yahoo has switched to the new Yahoo search engine instead of Inktomi the picture has changed.

Inktomi has indeed become much more important now that it provides most of the results at MSN Search (MSN Search accounts for about 15% of all web searches). But the question is how long will MSN continue to use Inktomi?

MSN has been developing its own search engine for some time now. About six weeks ago MSN stated that they would likely not go public with the new search engine until 2005.

However right after Yahoo announced that it had switched to the new Yahoo search engine, rather than Inktomi, MSN released a Beta test of its new search engine.

So don't be too surprised if MSN switches over to its own search engine much sooner than it had previously stated.

Back to top

Update on the Google

Now that Google results are no longer displayed on Yahoo, Google has lost a major part of its business. Before the switch Google results were displayed on Google, AOL, and Yahoo accounting for about 70% of all searches. Now Google provides about 50% of the search results. They're still the leader with Yahoo in second place at about 28%.

We're still watching the major changes at Google that began in mid November as they continue to roll out.

It's become farily clear that Google is incorporating the technology it acquired from its purchase of Applied Semantics, which uses artificial intelligence to examine language patterns in an attempt to determine what people are searching for. It appears that word patterns, including “Stemming” (more below) are being examined on web pages in the Google index.

Just last week a Google representative stated the following:

“The goal of a good search engine should be both to understand what a document is really about, and to understand (from a very short query) what a user really wants. And then match those things as well as possible.) Better semantic understanding helps with both those prerequisites and makes the matching easier.

So a good example is stemming. Stemming is basically SEO-neutral, because spammers can create doorway pages with word variants almost as easily as they can to optimize for a single phrase … But webmasters who never think about search engines don't bother to include word variants--they just write whatever natural text they would normally write. Stemming allows us to pull in more good documents that are near-matches. The example I like is [cert advisory]. We can give more weight to www.cert.org/advisories/ because the page has both "advisory" and "advisories" on the page, and "advisories" in the url.

Standard stemming isn't necessarily a win for quality, so we took a while and found a way to do it better.
So yes, I think semantics and document/query understanding will be more important in the future…”

Google continues to roll out these changes. Just this week we've seen significant changes in the rankings of some client pages.

A Google representative will be providing information and answering questions at the SEM conference I'll be at later this week. More to come

Back to top

George Aspland
eVision, LLC
Branford, CT