19 common mistakes that prevent your Web site from showing up on search engines

Many webmasters have the problem that their Web site is not listed in search engines at all. There can be a variety of reasons that your Web site doesn't show up on search engines.

Reason #1: You are using frames.

Many search engines have problems with frames. They often only index the frameset page and not the individual frames that contain the actual content. Unfortunately, the frameset page usually doesn't have META tags, title and enough content (text) to obtain a listing on a search engine.

If you really must use frames, consider the following points:

1. Add a description of your Web site in the <noframes> area so that search engines can index that text. There you should also add a link to the homepage.

2. When a search engine indexes a frame page outside of the frameset, the visitor can be left stranded and unable to link into your site. So your individual frame pages should always contain a link back into your site.

3. Add some JavaScript to force frame pages into the frameset. This prevents visitors from inadvertently accessing an orphaned Web page. You can use the following JavaScript snippet:

---
if (top.location.href == self.location) {
top.location.href = "URL of your frame file";
}
----

Webreference.com offers a very good introduction to frames:

http://webreference.com/dev/frames/

 

Reason #2: You are using a lot of pictures on your Web site but very little text

Search engines need text to index your Web site. They cannot know what's written on your GIF or JPEG images. If you use a lot of images on your Web site, you should also create some Web pages that have a lot of text.

Some Web site promotion consultants will tell you to create so-called doorway pages. A doorway page is a Web page that contains plain text and a link to your main Web page. On that doorway page, you should describe the content of your Web site in many sentences that contain many keywords that are important for your Web site.

However, some search engines only lists Web pages if at least one remote Web page is linking to it. In that case, a doorway page will not work. Don't use doorway pages for search engine spamming! Only use doorways that have something to do with the content of your Web pages.

Note that many search engines already ignore doorway pages. For that reason, try to give your real Web site as much content (text) as possible. Fresh, continuously updated content is one of the best ways to ensure that your visitors will return again and again.

Here are 3 tips for building and distributing your content:

1. Build one page of quality content per day. Write timely, topical articles with about 250-500 words. If you aren't sure what you can write about, look in your log files which search phrases have been used to come to your site. Or use Overture's keyword suggestion tool at

http://inventory.overture.com/d/searchinventory/suggestion/

and find the core set of keywords for your topic area. Those are your subject starters.

2. Stay abreast of developments in your sector. If the big site "ABC" is coming out with product "XYZ" in autumn, then write about the product or the product sector in general and have it ready in June so that search engines can index it early. For example, all the Nintendo Game Cube sites you can find in Google today - those have been submitted 3-4 months ago.

3. Syndicate your content (along with your name and Web site URL)! Other webmasters will gladly incorporate your articles into their Web sites. Just search for "syndicate your articles", "syndicate your content" and "submit your article" on Google.

Reason #3: The submitted Web page is only a redirection.

If the Web page you submit contains a redirection to another Web site, most search engines will skip your Web site completely. Do not submit a redirection Web page.

Many webmasters tried to cheat search engines with redirection pages in the past. The search engines companies discovered that and they decided to totally skip Web pages with redirections.

Submit a real Web page that contains the product description visible to the reader.

Sometimes, you have old Web pages listed on search engines and you want them to redirect to the new Web site. There are several ways to do it:

1. You can implement a server side redirect on the old Web page, using the 301 Moved Permanently error message. This will redirect users to the new Web site, but also tells the search engines that this page has moved permanently. Some search engines will drop the page from their index, and some will eventually replace the old page with the new one without hurting your rankings.

2. You can use the META Refresh tag on the old Web page, for example <META HTTP-EQUIV=Refresh CONTENT="5; URL=http://www.seoinc.com"> tells the browser to load www.seoinc.com 5 seconds after the current document has finished loading. However, some old Web browsers don't support that tag, and some search engines penalize pages that use a refresh of a few seconds or less.

3. Instead of the META Refresh tag, you can also use JavaScript to load a new document:

<SCRIPT LANGUAGE="JavaScript">
<!--
location.replace("http://www.your-new-site.com");
-->
</SCRIPT>

Most search engine robots ignore JavaScript so this method of redirection is unlikely to be penalized.

4. You can also delete the old Web page and create a custom 404 error page. This ensures that visitors will be redirected to the new site if they click on a broken link or enter an incorrect URL. The 404 error page should contain a link to your home page and to the primary sections of your Web site.

To move to a new Web site and to keep your old search engine rankings, I recommend using one of the methods 1 or 3.

Reason #4: You have submitted your Web site too often.

If you submit your Web site more often than once a month, most search engines will consider that spamming and they will skip your site.

Spamming does not work with search engines. Most likely, it will backfire to you. More and more search engines are able to detect spam attempts and penalize or ban your page from their listings.

Sites that spam search engines degrade the value of search engine listings. As the problem grows, these sites may face the same backlash that spam mail generates. The content of most Web pages ought to be enough for search engines to determine relevancy without webmasters having to resort to repeating keywords for no reason other than to try and "beat" other Web pages. The stakes will simply keep rising, and users will also begin to hate sites that undertake these measures.

Submit your Web site to search engines and wait for 4 weeks. Then search for the URL of your site. If the search engine cannot find your site, submit your URL again.

How you can check if your Web site has been indexed:
http://www.searchenginewatch.com/webmasters/checkurl.html

How often you should resubmit your Web pages?
http://www.promoteclick.com/articles/ar22.htm

Reason #5: You overuse keywords on your Web site.

Many search engines fear to be spammed if you overuse keywords on your Web site. Do not repeat your keywords too often in your meta tags or in the body of your Web pages.

Nobody knows the magic number for the search engines but a paragraph such as the one below is not a good idea:

"Ebooks are great. I love ebooks. I've read hundreds of ebooks. You can learn much from ebooks. On my Web site you can find tons of free ebooks. When you subscribe to my newsletter on ebooks, you get two additional free ebooks."

Some years ago, you may have obtained a top ranking for the keyword "ebooks", but today the search engines will quickly ignore such nonsense and probably write it off as "spamming". It could even cause the engine spider to skip your Web site completely.

Unfortunately, search engines do not indicate on their help pages the maximum allowed number of repetitions. Some webmasters suspect this to be three, some say six. There's no way of knowing until you are penalized.

Further webmaster discussion about keywords:

http://www.webmasterworld.com/forum.cgi?ran=956&forum=8&show=60

Reason #6: Text in the background color of the web

"Color can kill your ranking!" Some Web designers, in order to get high rankings in the search engines, try to make their Web pages as keyword-rich as possible. They try to spam search engines by repeating keywords in the same color as the background color to make the text invisible to browsers and search engine spiders.

However, almost all search engines already know that trick. They will penalize or even blacklist your Web page if they determine that your page is trying to unfairly misrepresent its actual content. This tactic is commonly referred to as "spamming the search engines" or "spamdexing".

Unfortunately, the problem is that the search engines may end up penalizing Web sites which did not intend to use the hidden text trick. For example, suppose you have a Web page with a black background and a table in that page with a white background. Now suppose that you've added some black text in that table. This text will be visible to your human visitors, so in fact, the text isn't hidden. However, the
search engines can interpret this to be hidden text because they overlook the table background color.

I recommend going through all your Web pages and make sure that you haven't inadvertently made any such mistake.

And, by the way, search engines also catch on using a slightly different color than the background color to hide words, so don't use that trick.

Reason #7: Your Web pages are created dynamically.

Databases and dynamically generated Web pages are great tools to manage the contents of big Web sites. Imagine you'd have to manage the Web site contents of the New York Times without databases...

Unfortunately, dynamically generated Web pages can be a nightmare for search engine spiders because the pages don't actually exist until they are requested. A search engine spider is not going to be able to select all necessary variables on the submit page.

The exceptions are the spider programs from Google and Inktomi. They are able to index Web pages that are dynamically generated, even those that use question marks and query strings.

On the other side, AltaVista isn't able to index dynamically generated Web pages, and here's why:

http://help.altavista.com/adv_search/ast_haw_wellindexed

If you create dynamic Web pages with the help of Active
Server Pages (ASP), ColdFusion, CGI, Perl or the Apache
Server, then the following Web page offers good advice:

http://spider-food.net/dynamic-page-optimization-b.html

Reason #8: You have moved your site to a new server.

Every time you move a Web site to a Web hosting company, or when you change the domain name, the IP/DNS address of your domain name changes. While people see URLs (www.yourdomain.com), search engines often only see IP addresses.

This means that you need to re-submit your Web site to all search engines and directories when you move your Web site to a new Web hosting company or when you change your domain name. In time, search engines will realize that your IP/DNS address has changed and will remove the old IP address from their index.

It's important that you re-submit only when the move is completely finished. A good way to know when the IP address change has been updated is to upload a slightly different version of the index page to the new server. When you enter www.yourdomain.com in your Web browser and see the different version you'll know that DNS servers of your ISP (Internet Service Provider) have been updated.

If you have changed your server, resubmit your Web site URL to search engines. If you have changed your domain name, you should also change your Yahoo and Open Directory Project listings.

If you want to change your Yahoo listing, go to:

http://add.yahoo.com/fast/change

If you want to change your Open Directory Project listing:

1. Go to " http://dmoz.org/ ".

2. First locate the category in which the site appears in (search for your domain name), and go to that category.

3. Click the Update URL link at the top of the Web page.

4. Enter the URL of the Web site you would like to update.

5. A new page loads where you can make changes.

6. Click the Update URL button once you've finished with the changes.

If you change your domain name, it's likely that you lose your old visitors. Here's advice how to reduce the loss of visitors:

http://www.thesitewizard.com/news/movinghosts.shtml

Reason #9: Your Web page does not have unique IP address.

Does your Web site has a unique IP address? If not, your Web site is running the risk of getting banned from the search engines.

Human beings use domain names like yahoo.com, but network computers use IP addresses, which are numeric addresses written as four numbers, separated by periods.

Every domain name translates to a so-called IP address. For example, yahoo.com is translated to "64.58.76.225". Just enter "http://64.58.76.225/" in your Web browser and you'll go to www.yahoo.com.

Many Web hosting services don't give out unique IP addresses to their customers to save money. They assign the same IP address to multiple domain names. This means that several hundred Web sites could all be using the same IP address as your site does.

There are 3 reasons why you need a unique IP address:

1. If you're sharing an IP address with 50 other sites, you're trusting them not to over-submit or spam the search engines. When a search engine blocks an IP address, all the sites that are sharing that IP address are blocked. You could wind up being banned from the search engine.

2. If the server or the search engine spider software is misconfigured, the search engine spider may end up obtaining a Web page from another domain with the same IP address. This may mean that the other Web site gets indexed instead of yours, or your Web site will be found for the keywords which are applicable to the other site.

3. Rumor has it that having your own unique IP address may help your search engine ranking.

So when you select a Web hosting service, make sure that your domain name has a unique IP address, even if it means that you have to pay a bit more for your hosting.

Are you sharing an IP address with people you don't even know? Here's a way to test it yourself:

1. Go to " http://www.eamnesia.com/hostinfo/i.jhtml " and enter your domain name (for example, yahoo.com).

(If this URL doesn't work anymore, go to " http://www.name-space.com/search/ " and enter your domain name in the nslookup field.)

2. The result page shows you what IP address your site resolves to (for example, 64.58.76.225)

3. Copy the IP address to the clipboard.

4. Open a new window in your Web browser, enter the IP address (for example, http://64.58.76.225 ) and hit Return.

5. If your Web site appears, you have your own IP address. If another Web site or an error message appears, you probably share the IP address with others.

If you are unsure, ask your Web hosting service company if your Web site has its own IP address.

Reason #10: You are hosting your Web site at a free Web space provider.

Some search engines (e.g. AltaVista) limit the number of pages they will index from a single domain. For example, if your Web page is hosted at Geocities.com or Tripod.com, it might happen that your Web site is not listed just because the maximum page limit for that special domain name is reached.

Some search engines no longer even index pages residing on common free Web hosting services. Their complaint is that they get too many spam or low-quality submissions from free Web site domains.

However, Google is the exception. Google does index Web pages on Geocities.com and Tripod.com. Those pages also seem to have a Google PageRank of at least 3/10 because they are linked from a popular domain.

In summary, if you are serious about doing business on the Web, it helps tremendously to have your own domain name. Put yourself in the customer shoes: Would you buy from a Web site that is called "sub.free-web-space- provider.com/~category/162742/ my_business/home.htm"?

A domain name is easier to remember for your customers and it helps to build trust.

And while we're at it: A recent survey by Consumers Union found that only 29 percent of Americans trust Internet sites that sell products.

Consumer WebWatch research report on trust:
http://www.consumerwebwatch.org/news/1_abstract.htm

How to gain your customers' trust:
http://www.ecommercebase.com/article/37

 

Reason #11: Your host server was non-operational during spidering.

It can happen that your Web server is down when a search engine spider tries to index it.

If your Web site fails to respond when the search engine spider visits your site, your site will not be indexed. Even worse, if your Web site is already indexed and the search engine spider finds that your site is down, you'll often be removed from the search engine database.

It's essential to have Web space on servers that are very seldom down. Choosing a reliable Web space provider is very important for a successful online business.

At first it sounds impressive when your Web space provider promises 99% server reliability. But take a moment to calculate it. It means that 1% of the time, your potential customers cannot reach your Web site. One percent of a year means that your Web site will be down for nearly 4 days per year. That equals 4 days without sales.

As you can see, 99% reliability is not enough. You must constantly monitor your server so that you can act immediately if your server is down (e.g. call the service provider to restart the server).

Here's a free monitoring service:
http://www.internetseer.com

Another free Web site monitoring service (without ads):
http://uptime.openacs.org/uptime/

A freeware tool that can monitor the uptime of your site:
http://www.idyle.com/software/internet/host-monitor/

Monitoring service for bigger companies:
http://www.atwatch.com/

Reason #12: You don't allow robots to index your Web site.

Imagine you're a Internet marketing service company and you keep trying very hard to get a top ranking in the search engines for your customer.

Even after several weeks, the customer's Web site hasn't been listed in any search engine. Then you start to realize that the search engine spiders and robot programs cannot access the Web site because your customer blocks them (by mistake).

There are two ways to block search engine robots: a) with a simple text file in the root directory of the host server, or b) with a certain META tag in the Web pages.

a) Robots.txt

The host server might have a plain text file named "robots.txt" in the root directory. It contains rules for the search engine spiders. The rules in the robots.txt file follow the Robots Exclusion Protocol, a document designed to help Web administrators and authors of Web spiders agree on a way to navigate and catalog Web sites.

The content of the robots.txt file consists of two main commands: "User-agent" and "Disallow".

The User-agent command specifies the name of the robot for which the following commands should be applied to. You can set this to "*" to have the spidering commands applied to any robot.

The second command, "Disallow", specifies a partial URL that should not be indexed by the Web robot.

The text

---
User-agent: *
Disallow: /
---

tells all search engine spider programs to go away. If you find a text file called "robots.txt" in the root directory of the host server with the above content, you should delete it immediately. The text file says that no search engine is allowed to index your Web site.

Even if your robots.txt file don't contain the above commands, you should make sure that its syntax is correct. A robots.txt file with a faulty syntax also prevents search engine spiders to index your Web site.

To check the syntax of your robots.txt file, you can use this free tool (just enter your domain name www.domain.com): http://www.sxw.org.uk/computing/robots/check.html

b) The META ROBOTS tag

There's a second way to stop search engine robot programs to index your Web site: the META ROBOTS tag. If you find the following HTML tag in your Web pages:

---
<META NAME="robots" CONTENT="noindex,nofollow">
---

you should replace it immediately with

---
<META NAME="robots" CONTENT"="index,follow">
---

If you want all search engine spiders to index all Web pages, you can also remove the META ROBOTS tag from your Web pages.

Further information about both ways to stop search engines to index your Web site can be found at:

- http://www.wdvl.com/Location/Search/Robots.html
- http://www.ebrandmanagement.com/whitepapers/robots2.htm

Reason #13: Your Web pages require a full-fledged browser.

When search engines crawl the Web to find new Web pages, they use special software for it, called "spiders", "robots" or "crawlers".

These crawler programs don't have the functionality of full-fledged Web browsers such as Microsoft Internet Explorer or Netscape Navigator.

In fact, search engine robot programs look at your Web pages like a text browser does. They like text, text, and more text. They ignore information contained in graphic images but they can read <IMG ALT> text descriptions.

This means that search engine spider programs are not able to use Web browser technology to access your site. If your Web pages require Flash, DHTML, cookies, JavaScript, Java or passwords to access the page, then search engine spiders might not be able to index your Web site.

Therefore, it might be a good idea to test your Web pages with very old versions of Web browser applications or with the software program "Lynx", a text-only browser.

Lynx is available for download here . Here's an online version of Lynx that allows you test your Web pages with a text-only browser quickly and easily: http://www.delorie.com/web/lynxview.html .

"Simulation of a search engine spider", see how search engines see your Web site:
http://www.delorie.com/web/ses.cgi

Reason #14: Search engines could not resolve your DNS name.

There's a mistake that novice users often do. They register a domain name (for example, www.my-great-site.com), and they immediately submit the Web site URL to the search engines.

Then they wonder why the search engines didn't index their site. The answer is, they weren't able to do it.

It takes approximately 2-4 days until a domain name becomes active. All Internet access providers must update their records (DNS tables) to reflect new site locations. The process of updating DNS tables is called propagation.

Search engines must also update their DNS tables and until then, the new domain name www.my-great-site.com doesn't work.

So when you register a new domain name, you must wait about 48-72 hours until you can submit the domain name to the search engines.

 

Reason #15: Your Web site has a low link popularity.

Link popularity is becoming _the_ determining factor for a top search engine ranking.

Link popularity means the number of Web sites linking to your site. However, the quality of links is more important than the quantity of links. For instance, if the New York Times links to your site, their single link might count a lot more than 30 links from your friends' personal homepage.

By now, all top search engines use link popularity in their ranking formulas: AltaVista, Inktomi, MSN Search, HotBot. For Google, it's even the most important factor in ranking sites.

The idea behind link popularity is that other Web sites will link to your site only if you are a quality site offering quality resources. So if many Web sites link to your site, search engines come to the conclusion that your site must be very popular and deserves a high ranking.

As Google co-founder Sergey Brin said in an interview: "...a page that is pointed to by many other sites is important. In other words, external approval raises a page's ranking."

Link popularity can do a lot for your site. Not only will the most important search engines rank you higher, but links from other sites will also drive more traffic to you.

In addition, as more sites link to you, the odds increase that search engine spider programs will encounter your site more regularly so that it's less likely that they drop your site from their index.

What's the link popularity of your site?

The freeware Windows program Link Popularity Check tells you the link popularity of your Web site is and compares it with competitor sites.

Link Popularity Check is a classic freeware application: it has no nag screens, it doesn't change the system registry, it makes no unauthorized connections to the Internet and it comes with a hands-free uninstaller.

Download Link Popularity Check (freeware).

How to improve the link popularity of your Web site:

You can search the search engines for Web sites that are related to your business, find the webmaster's contact information and then solicit a reciprocal link. Do this every day and your link popularity will climb steadily but slowly. And it's very time-consuming.

Reason #16: Your Web page URL contains special characters.

Most search engines have problems indexing Web pages when their URLs contain special characters. The following special characters are known to be "search-engine-spider-stoppers":

ampersand (&)
dollar sign ($)
equals sign (=)
percent sign (%)
question mark (?)

These characters are often found in dynamically generated Web pages. They signal the search engine crawler program that there could be an infinite loop of possibilities for that page. That's why they ignore Web page URLs with the above characters.

AltaVista and Lycos explain on their help pages why they cannot index such Web pages:

http://us.altavista.com/help/search/faq_web#13
http://insite.lycos.com/searchservices/
frequently_asked_questions.asp?b=n#10
HotBot recommends that you submit your dynamic Web pages with all parameters added onto the URL (for example, "www.site.com/articles/query.asp?article=83").

Google and Inktomi utilize crawler programs that are able to index dynamically generated Web pages, even those that use question marks.

So what can you do if you have dynamically generated Web pages with special characters? If you use the Apache Server, ASP, CGI/Perl or ColdFusion, the following Web page provides some solutions:
http://spider-food.net/dynamic-page-optimization-b.html

Reason #17: Your Web site has a slow host server.

Search engine crawler programs that index Web pages don't have much time. There are approximately 2-4 billion Web pages all over the world and search engines want to index all of them.

So if the host server of your Web site has a slow connection to the Internet, you may experience that your Web site will not be indexed by the major search engines at all.

AltaVista and Google specifically mention the problem on their Web sites.

AltaVista: "If a site has a slow connection or the pages are very complex, it might time out before the crawler can index all the text."

Found at:
http://help.Altavista.com/adv_search/ast_haw_wellindexed

Google: "Your site may not have been reachable when we tried to crawl it because of network or hosting problems. When this happens, we retry multiple times, but if the site cannot be crawled, it will not be listed in our current index."

Found at:
http://www.Google.com/webmasters/2.html#A3

You may also want to limit the size of your homepage to less than 60K. It'd also benefit the still numerous users that connect to the Internet with a slow modem. For even the casual Internet user, the performance of a Web site can make the difference between pleasure and frustration.

How you can test the speed of your server:
http://web-hosting.candidinfo.com/site-response-speed.asp

 

Reason #18: Do you use the right keywords? It's important that you use the keywords on which you want to be found in the right places on your Web pages.

Imagine your Web site sells garden flowers. Your HTML source might look like this:

---
<HEAD>

<TITLE>Garden Flowers at Flower Garden Farm</TITLE>

<META NAME="keywords" CONTENT="garden flowers, flower arrangements, wildflower seeds, seed mixes">

<META NAME="description" CONTENT="Many different garden flowers available for your home, yard, garden and patio.">

</HEAD>
---

Some search engines (AltaVista, Inktomi) index the META Keywords tag, some (Google, AllTheWeb/FAST) don't.

For the search engines that index the META Keywords tag, it's important that you actually use the keywords of your META Keywords tag on your Web pages.

Some search engines nowadays consider it spam if the keywords of your META Keywords tag don't appear in the body text on your page.

Of course, you know that the most important thing for a good search engine listing is content on your Web site. Search engines need text, text and even more text to index your Web site.

For example, if you sell cars, have articles about cars on your Web site. If you offer package holidays, provide some tourist information about your destinations. Build this content and optimize the text for your target search terms.

In other words, make sure that the content on your Web pages contains the keywords that are important for your site.

And don't forget to use only those words in your META Keywords tag that appear on your Web page.

Now make the test with your homepage:

1. Go to "http://www.live-keyword-analysis.com".

2. Enter three of your META keywords in the three keyword fields.

3. Leave the ratio fields empty.

4. Copy and paste the body text of your homepage into the big edit field.

5. Click the "Update" button.

6. Now look at the ratio fields. If a keyword has a ratio of 0%, consider adding contents about that keyword on your Web page.

Reason #19: Have you waited long enough?

The Internet is growing rapidly. According to a study of the NEC Research Institute, most search engines need about 6 months to index a Web site.

Source: NUA Internet Surveys
http://www.nua.com/surveys/index.cgi?f=VS&art_id=905355013&rel=true

The data was collected when there were by far fewer Web sites than today. With today's number of Web sites, it might take even longer (although many search engines have also improved their equipment).

That means that your Web site submissions may need up to half a year until they appear in search engine listings.

The result of the study shows that it's important that you submit your Web pages the right way, right from the start. When search engines need six months to index your Web site, you should make sure that they'll index your pages correctly.