HTTP status codes are three digits numbers that are returned by a web server in response to a web request by a client. This code provides some general information about the status of the request to the client. The response may (or may not) also contain a descriptive text regarding the status of the request.
The IANA (Internet Assigned Numbers Authority) is responsible for maintaining the registry of accepted HTTP status codes, and is usually adhered to by all popular web server and clients.
In the context of SEO, the search engines or bots are the web clients that query the server for content that needs to be indexed. Thus using the correct response code will allow the search engines to parse, analyze and index the content correctly.
First let’s take a look at the most commonly used status codes.
- 200 (OK): This code denotes that the content is available for this request or URI and that the response contains the appropriate content. This is usually the most common response in most cases.
- 301 (Moved Permanently): This status code specifies a redirection. It tells the client that the resource has moved permanently to another location and the client should the new URI to fetch the resource.
- 302 (Temporary Redirection): This is another code for redirection. This tells the client that the server is using another URI to fetch the content but the redirection is temporary, and the client may continue to use the same URI in the future.
- 404 (Not Found): This tells the client that the resource it is requesting is missing or cannot be found. The server does not know where the resource is or if it does exist.
- 410 (Gone): This tells the client that the server understands the request, but the server knows that the resource that it pointed to does not exist anymore and is gone for ever.
There are several different scenarios where you should use these status codes correctly, so that the search engine crawlers can understand correctly the status of the content. When status codes are not designated correctly, it could cause issues with search engine indexing which can cause you to rank poorly in search results. Common issues include:
- Content not getting indexed
- Duplicate content
- Site performance issues with page loading
- Repeated crawling by bots thus increased server load
Many of the scenarios that you should pay attention to three broad categories: Success, Redirection and Not Found. If your website is working, it is quite likely that you have the success part already taken care of. We will look at the other two scenarios in detail.
Over time some of the resources in the website gets relocated to other URLs or pages. This is especially true if and when you do a website redesign. Some of the pages will have a more user friendly or appropriate URL, so will some of the images and other resources. This is almost unavoidable as you modify, enhance and update your website.
But, there may be external URLs or backlinks that drive traffic to your website using the old URLs. And obviously you don’t want to lose this traffic and it is pain to track down every single URL in the web that points to you and get it changed. This means you want to redirect traffic that uses the old URL appropriately to new pages.
Here, you will need to use the appropriate status code so that both the users (or traffic) as well as the search engines understand that the page has moved either temporarily or permanently. When used correctly, the search engines can crawl and index the new pages and content while mapping the older index and ranking to the new URL.
In most scenarios using the 301 status code for the pages are recommended, as it allows the search engines to correctly appropriate content to the new URLs. When appropriate you could use 302, if you foresee the resource “coming back” at a future time.
Not Found or No Content
It is not uncommon for pages and resources become obsolete over time. In some cases, as described above, you can use redirection to divert traffic to appropriate pages. This is mostly useful for links that drive a lot of traffic to the websites. But redirecting every single URL to pages, images, and other resources is time consuming and unnecessary.
It works best if you send the appropriate status code to the client denoting that the content is non-existent. This way the search engines can remove the content from its indexes while the browser can display appropriate error messages. The HTTP status code that you will use in this case 404. I am sure you have seen a 404 error before …
Note that redirecting all not found pages to another unrelated page or homepage is usually not a good idea.
If you have removed some pages permanently, then a status code 410 can be used as well. This will let search engines know that the pages are gone and is intentional which will reduce the number of crawling errors you would see in crawler logs for your web domain (eg: Google Webmaster Tools).
Error pages are generally a good idea. This potentially allows the user to browse to a related page and/or force the user to remain on the website. This also eliminates the display of ugly 404 pages shown by the web browsers. These are also called custom error pages.
The main issue with such pages is that they are usually served with a status code of 200 instead of 404. The status code of 200 forces the web browsers to display the error page instead of its own error page. But such a strategy usually confuses the search engine bots into thinking that the resource actually exists and the error page content needs to be indexed.
Many search engines do allow you to designate or register custom error pages using the administration interface so that such pages can be correctly identified. I suggest that you use this functionality if you use custom error pages on your site.
It is important that you return correct and appropriate status code for each and every URL on your website or domain. How you do that will vary on the configuration, architecture and the language used by your website. Your web administrator or web developer would be the best resource on how to implement this.