SEO Part 8 – Robots.txt

A robots.txt is text file that resides on the root directory of the web server.  The purpose of this file is it to indicate to visiting robots and crawlers which parts of the underlying web site should be visited for the purpose of inclusion in a search engine.

More information can be obtained on the following site:

In most cases you will want the search engines to access the site and crawl all parts of the site.  An example Robots.txt file is below which indicates to ALL crawlers and robots (indicated by ‘User-agent: *’) to index the whole site apart from content under the folder ‘mySecretFolder’

User-agent: *
Disallow: /mySecretFolder/

SEO Part 7 – Page Content

In search engine terms ‘content is king’ and content for search engine spiders and crawlers is text. In Google’s webmaster help pages the following points are made in the content guidelines section.

• Create a useful, information-rich site, and write pages that clearly and accurately describe your content.

– Think about the words users would type to find your pages, and make sure that your site actually includes those words within it.
– Try to use text instead of images to display important names, content, or links. The Google crawler doesn’t recognize text contained in images.

Again these points come around to fact that content on pages should include keywords and phrases that the page is targeted towards.

An important point to remember is that content should be written for two audiences – but in this order:

1. The human audience
2. Search engine crawlers

This is why they state above that you should include words users would type to find your pages – this is the keywords we have targeted and that we know are popular terms from our research undertaken in the ‘keywords’ task.

Ranking 31st Position on Google

I (RichyP) took part in an interesting discussion over on the searchenginewatch forums the other day:Â

A web site owner was experiencing a ranking with Google that was always 31st position. It seemed whatever the search term the site always came in on page 4 of Google results. The main domain had a PR of 6 so I thought this was strange that they didn’t rank higher. Well I had a little look into this for them and discovered that they had multiple domains all pointing to the same content.  It therefore appears as though Google has penalised them via some form of duplicate content filter.

Never heard of the magic 31st position or page 4 of Google results though but an important lesson on not to serve up the same content under the banner of multiple domains.

SEO Part 5 – Keywords in Meta Tags

As with the browser title importance – the META tags should also include keywords and/or phrases. The following SEO related META tags are ones you should include in your HEAD tags.

meta name=”Title” content=”…”
meta name=”Description” content=”…”
meta name=”Keywords” content=”…”
meta name=”ROBOTS” content=”INDEX,FOLLOW”

The meta title tag should really contain the same content as the browser title tag ().

The Description tag should be a brief description of the page – it is our chance to try to influence the description of the page for crawlers. Not all robots and crawlers support it and Google is reported to create its own description when displaying pages in its results listings.

The keywords tag is now mostly ignored by search engine crawlers as it has been abused in the past by webmasters trying to rank high on certain terms. Most search engines now analyse the page and determine the keywords from the most frequently used words and phrases – although they have checks for keyword density and abuse of keywords in text can result in ranking penalties. The Meta keywords tag is sometimes useful as a way to reinforce the terms you think a page is important for (this is for the few crawlers that support it). Mentioning ‘SEO solutions’ in the keywords tag and then having this term elsewhere in the page MIGHT help boost the page a bit higher for those words. Only a few keywords should be used and all separated by commas as above. Try to avoid having too many keywords though – this should be limited to less than 10 and ideally less. Avoid repeating the same terms.

The ROBOTS tag indicates to robots and crawlers that accept META directives that they should index this page and then follow any links off this page. Of course change this if you don’t want that page indexed or links followed.

SEO Part 4 – Directory Listings

The directory site DMOZ ( is the structure behind the Google directory ( Sites listed within DMOZ generally obtain a higher pagerank (depends on category pagerank though) and are also looked upon favourably by Google as DMOZ is human edited for quality control.

You should target DMOZ categories for links to your homepage and any important pages within your site that match directly any DMOZ categories.

If you find similar sites in a category to yourselves you should target these for possible link exchanges €“ these sites will have good page ranks from being in DMOZ.

The higher the category you can get listed within in DMOZ the higher the pagerank that category has and so the more influence it has on your page rank.

Yahoo has its own directory – Inclusion within this directory is by payment and inclusion is not guaranteed. Again the higher the category you can get listed within the higher the pagerank.

SEO Part 3 – Browser Titles

The web browser title is the main identifier of the content on that page or website (if on a homepage). It is therefore important to get keywords or key phrases on the title and ensure that they are unique to that page. The more text used on the title the more diluted the description for that page becomes and therefore harder match against search term keywords.

For underlying pages on your site it is beneficial to create individual page titles relating to something specific about that page. Try to avoid having the same generic company branded title across all pages. For example a title of:

My site on web site search engine ranking improvement ideas and practices

Could be changed to any of the following:
search engine improvement
search engine improvement practices
search engine ranking ideas

Try to target the title of the page to what you and other people will search for, but proper English.

If required you can have corporate ID on the titles of pages so with Search Engine Optimisation Solutions we could have “SEOS“:

Search engine improvement – SEOS

But this would always be better at the end as the first few words are treated with more importance from a search engine point of view.

SEO Part 2 – Incoming Links

One of the biggest indicators of a popular site is how many incoming links it has. This serves as an indication that the site has value and so other webmasters and content creators have referenced it and linked to that site.

External Link Text
One very important factor is the text that external sites link to USC with. Search engines view this text as a factor on what the target page relates to. The more external sites that link to your site with similar and related keywords then the higher the chance that you will rank well for those keyword terms.

An example is as follows. If you search on Google for €œcomputers€ without the quotes you will see that Apple is currently second. If you view the apple homepage and even the source code there is no mention of the word computers in any text or meta tags. Apple has ranked well for this term because other sites link to apple with the term ‘computers’.

Apple computers search result

Although you don’t have full control over what text other sites link to you with – you can try to influence some sites into using the keywords that you have decided upon – which ultimately will help your ranking for those terms.

The power of external link text on how a web site ranks for certain terms is highlighted in Wikipedia article on ‘Google bombing’ –
This shows how external links have been used to extremes to influence the Google SERP’s.

External Link Sources
The source of external links is very important and works very closely with the link text. Google is reported to put a weighting on the relevance of a link from a site to another site if they both share a similar theme. Therefore obtaining external links to your site from other sites covering the same subject matter would be viewed as more important that links from sites covering other subjects.

Google has a patented algorithm Pagerank with which indicates the importance of a site/page. Obtaining links from high page ranking sites helps increase your sites page rank and ultimately your search engine result position (SERP). Installing the Google toolbar or by using the following site you can check a sites page rank:

Google’s explanation of pagerank is as follows:

PageRank relies on the uniquely democratic nature of the web by using its vast link structure as an indicator of an individual page’s value. In essence, Google interprets a link from page A to page B as a vote, by page A, for page B. But, Google looks at more than the sheer volume of votes, or links a page receives; it also analyzes the page that casts the vote. Votes cast by pages that are themselves “important” weigh more heavily and help to make other pages “important.”

Important, high-quality sites receive a higher PageRank, which Google remembers each time it conducts a search. Of course, important pages mean nothing to you if they don’t match your query. So, Google combines PageRank with sophisticated text-matching techniques to find pages that are both important and relevant to your search. Google goes far beyond the number of times a term appears on a page and examines all aspects of the page’s content (and the content of the pages linking to it) to determine if it’s a good match for your query.

Taken from

It is important to note that getting a few links from very high page ranking sites is more effective than getting lots of links from many low ranking sites.

Pagerank is generally higher on the homepage of a site and it degrades the further down the sites navigation tree. This is not always the case though if a page within the site has a lot of external sites linking to it. Deeply buried pages can have high page ranks if high ranking sites link to that specific page.

SEO Part 1 – Keywords

A list of keywords and phrases that is going to be targeted for SEO should be a number one priority. These keywords & phrases will then be what we you can judge the SEO effectiveness over time.

The keywords should be typical words that a user is expected to type into a search engine and find your given website. The phrases should include the keywords and maybe only a few other words.

In choosing keywords you should also consider the spelling, abbreviation, apostrophes, etc. The following table gives just a few examples:

Main Word

Alternative Words

All keywords and phrases should be checked for popularity and search volumes. The following three sites give popularity indication

Don’t automatically go for the highest traffic volume keywords. If you are a small site it may be worth targeting some niche keywords rather the top keywords. The top keywords will undoubtedly have large sites competing for them and its always better to be a “big fish in a small pond rather than a small fish in a Google ocean”. Over time as your site grows you can target the bigger keywords and compete with the big boys.

What is SEO?

SEO is a series of methods aimed at improving an internet facing web site in search engine rankings and ultimately drawing in more visitors.  It covers a wide range of activities that can range from the server configuration to the actual content on the pages and who links to them.

Optimisation commonly falls into two categories:

  • White hat SEO
  • Black hat SEO

White hat SEO are techniques that the search engines themselves allow and encourage in some circumstances whereas black hat SEO are techniques that influence search engine rankings by using frowned upon techniques that can result in a website being banned altogether from the search engine.