Web search engines are made of three parts:

1. User Interface, or "front-end"
   This is the part we see. Its function is to take our request for information, get the information from the database, put it in order, and send it to us.
2. Database, or "back-end"
   Here, vast amounts of information are processed and stored in carefully-organized form to make it quick to retrieve.
3. Crawlers, also known as "spiders" or "robots"
   These agents systematically move around the Web harvesting information.

Figure 1 shows the relationship between the three parts.

Figure 1

Crawlers are software programs that run on computers belonging to the search engine. (They don't actually move from one computer to another.) The job of the crawlers is to automatically harvest information. This is the general pattern of their actions:

1. Start at a page determined by humans to be useful.
2. Send the page back to the Search Engine to be processed.
3. Follow links to other pages, sending back information and following links on them.

In this way, information from millions of Web pages is sent back to the search engines.

Search engine crawlers aren't all programmed the same. There are a number of ways in which their designers try to make them better, or more specialized.

• Where they start: Crawlers are generally started on pages with lots of links for them to follow. For example, Starting a crawler on the Yahoo home page would give it links to many categories of Web sites, including Autos, Finance, Games, Groups, HotJobs, Maps, Mobile Web, Movies, Music, Personals, Real Estate, Shopping, Sports, Tech, Travel, TV, and Yellow Pages. A crawler could spend many happy hours following all those links!
• What sort of information do they bring back from each page? Some brought back only a summary, but now they almost all bring back the entire page, including images.
• How far do they go when following links? Some go to the first few pages on a site; most go as far as they can.
• What sort of site do they visit - all, or only certain kinds? The best-known search engines visit all the sites they can find. Some are specialized, searching only certain types of site, such as government sites or health-related sites. (Specialized search engines can save you time!)

The User Interface (UI) is what we see when we use a search engine. It's more than a pretty page, of course! The UI is responsible for these jobs:

• Welcome the user and make it easy to figure out how to use the system;
• Accept user input and parse it into recognizable words and commands (such as quoted strings, "+", "-", AND, OR);
• Find relevant hits in the database;
• Compute the degree of relevance of each hit;
• Sort the hits into order by relevance (and possibly slip a "sponsored link" in the right place);
• Show the list of hits to the user.

Differences in the UI are the most obvious differences between search engines. They include:

• Complexity of the starting page: which is more complex?
  • Google
  • Yahoo
  • Microsoft
• Type of search commands recognized;
• How relevance is computed:
  • Frequency of search terms in the pages
  • Number of links pointing to pages
• Order in which hits are presented
  • Relevance only: most search engines
  • Relationship of one to another: Kartoo
• Which facts are presented about each Web page: Originally, search engines presented different selections of facts about hits; now they mainly opt for simplicity, and include only:
  • Page title
  • Two lines with the context of hit words
  • URL
  • Size
  • Whether or not available in cache
• Visual style of the page and additional material presented there.

With all the differences between search engines, it's often a good idea to ask more than one. This is where the meta-search services come in. They submit your search to multiple search engines, and organize the results for you.

There are two kinds of meta services:

• on-line: they look just like a regular search engine, through a Web browser;
• desktop: these are tools that are downloaded and installed on your computer

On-Line Meta Search

• Clusty: clusters results by topic
• Dogpile: returns results from most of the standard search engines, in standard format
• Ixquick: returns results from most of the standard search engines, in standard format
• Kartoo: shows relations between topics visually
• Mamma: free on-line service from Copernic; returns results from most of the standard search engines, in standard format
• Seekz: returns results from most of the standard search engines, in standard format
• SuperCrawler: includes topic-specific directories; returns results from most of the standard search engines, in standard format
• MyWebSearch: use Ask or Yahoo (your choice); select topics from list of categories; simple listing

Desktop Meta Search

Two main kinds:

• Web search: you can usually set them up to search automatically, on a schedule, to keep you updated on topics important to you
• Hard drive search: lets you do a search of your own computer, as you would search the Web

Market for these is shrinking, as on-line services provide similar features

• Copernic: aimed at corporate desktop users; three versions of which the Basic is free
• LemmeFind: free Web search

These are increasing in popularity as hard drive capacity increases, and it get harder to find files you saved last year - or yesterday.

• Google
• Windows
• Yahoo and X1

Lots of information on the Web is not available to search engines. Why not?

Increasingly, large sites do not store information directly on the Web. Instead, they store the information in databases. People can get access to this information by going to the site's home page and typing in a question or request. The site's Web server sends the question to the database, which returns the data. The data is formatted in a helpful way that looks like a Web page, but is not accessible by following links, and so is not easily available to search engines.

Dark matter
the information on the Web that is not accessible to search engines.
This database information is informally known as dark matter, using a metaphor from astronomy. In astronomy, dark matter is the part of the universe what doesn't emit energy we can detect. Calculations have shown that the mass of the universe is much greater than what we can actually detect, so the majority must be "dark matter".

What do search engines do about dark matter? Some can be gotten by special crawlers equipped with lists of common search terms, or by closely guarded techniques. Most of it is accessible simply by going to a Web site and searching it yourself.

Since most e-commerce sites use databases but want their "dark matter" to be found, many of them have special arrangements with the major search engines to bring their content up in hit lists. For example, you can search for the title of a book, and Amazon.com will provide a link to their information about the book (hoping you'll order a copy from them!).

Have you noticed the little link that says cached at the end of many hits?

When a search engine crawls a page, it brings back the contents for indexing. Most search engines also save the page in their cache and make them available through the link in the hit entry.

Why do this? Web pages change quickly. You can't assume that a page listed in the hit list is actually there. It might have been moved to a new URL, or it might have been removed because of being old.

Sometimes, Web pages are removed because their content has caused controversy. In those cases, the cached version on a search engine may be the only way to get information about t he controversy. These caches themselves are controversial, though. Often, the people who remove a Web page want to completely prevent people from getting the information. In those cases, they may request the search engines to remove the cached pages. Should the search engines comply? That's the controversy!