Posts Tagged ‘Robots’

SEO (Search Engine Optimization) Basics Part II

Monday, March 29th, 2010

In the second and final part of this article I will cover the remaining 5 areas that relate to solid Search Engine Ranks for particular number of key phrases. Again, this will apply to any type of website, on any industry you are targeting.
Lets review the last 5 areas:

Provide a road map of your web site, and search engines will follow it.
Called the “sitemap”, this very simple page contains links to every page on a web site and serves one very important purpose. That is to get your sub pages crawled, and into a search engine’s index.

Search Engines look for the robots.txt file, so make sure you have it.
As its name implies, this is a text file that tells robots what to and what not to index. Its content consists of so-called records. Only mention files and directories that you don’t want to be indexed. All other files will be indexed normally if they are linked on your site. Having a proper robots.txt file gives your pages the rankings they deserve. If search engines know what to do with them, then they can give them good ranking.

Submit your site to related categories in niche directories.
Internet directories have become very important because they represent an easy path for inbound links creation. Now, when you submit your site to related categories in your industry you get that extra link factor, quality. Links from these categories can have a very positive effect on the search engine rankings of your web site.

Writing articles about your site, a great way to get new content.
This will help you get good positioning on search engines and quality incoming links as well. Always keep your articles as informative as possible. Other web sites will link back to your articles if you do. Write about the topics that your visitors are interested in, and dont forget to send them to syndicate sites. They will in turn offer them to others. Just make sure that you always have a link back to your web site so that all sites who publish your article will automatically link to you.

Become an authority in your field. Visit relevant forums and blogs.
Only join forum and blogs that are relevant to your industry. Use your expertise as a site owner in your industry to give information and advice. Of course, put your URL into the signature of your posts. Being the authority is simply knowing your industry, and giving people the information they need to trust you and your site.

How Do Search Engines Work – Web Crawlers

Monday, February 1st, 2010

It is the search engines that finally bring your website to the notice of the prospective customers. Hence it is better to know how these search engines actually work and how they present information to the customer initiating a search.

There are basically two types of search engines. The first is by robots called crawlers or spiders.

Search Engines use spiders to index websites. When you submit your website pages to a search engine by completing their required submission page, the search engine spider will index your entire site. A spider is an automated program that is run by the search engine system. Spider visits a web site, read the content on the actual site, the site’s Meta tags and also follow the links that the site connects. The spider then returns all that information back to a central depository, where the data is indexed. It will visit each link you have on your website and index those sites as well. Some spiders will only index a certain number of pages on your site, so dont create a site with 500 pages!

The spider will periodically return to the sites to check for any information that has changed. The frequency with which this happens is determined by the moderators of the search engine.

A spider is almost like a book where it contains the table of contents, the actual content and the links and references for all the websites it finds during its search, and it may index up to a million pages a day.

Example: Excite, Lycos, AltaVista and Google.

When you ask a search engine to locate information, it is actually searching through the index which it has created and not actually searching the Web. Different search engines produce different rankings because not every search engine uses the same algorithm to search through the indices.

One of the things that a search engine algorithm scans for is the frequency and location of keywords on a web page, but it can also detect artificial keyword stuffing or spamdexing. Then the algorithms analyze the way that pages link to other pages in the Web. By checking how pages link to each other, an engine can both determine what a page is about, if the keywords of the linked pages are similar to the keywords on the original page.