specify a "Vary: Accept-Encoding" header for better caching

"The following publicly cacheable, compressible resources should have a "Vary: Accept-Encoding" header."

If you are serious about your website or blog, then you must have been using different webpage analyzer tools in the quest to make it more efficient and to improve the speed.The above error/warning message is probably one of the many suggestions you might have seen from these tools. It is also one of the most common errors found by analyzers.

There are several different online tools that allow you to analyze your web pages for free. Some of the popular ones are Google Page Speed, GT Metrix, Pingdom Tools and YSlow.

One of the most common suggestion you will get is add the header "Vary: Accept-Encoding". The actual message, as noted above will looks something like this: The following publicly cacheable, compressible resources should have a "Vary: Accept-Encoding" header.

What is the Vary header?

The Vary header is relevant only in the case of cacheable content. Cacheable content usually refers to static files or files that do not change very often and definitely not with each user session or interaction. This cacheable files usually includes static files, such js, css, images or HTML files etc.

The Vary header essentially tells the client that the content that is served from the URI will vary according to the value of another request header. The syntax of the Vary header is something like this

Vary: field-name

where field-name refers to the name of the (another) request header that the content will vary by. For example, if the server is going to serve different content files based on the Accept-Encoding header, then it will look something like this

Vary: Accept-Encoding

You can use multiple field names in the header value separated by comma.

The main use-case for a Vary header is for intermediate caching proxies, if you use one. Who does not use a caching proxy these days, along with CDN anyways? These caching proxies need to know which headers to take into consideration while caching content for requests.

Usually, the caching is performed by using two fields from the request: Path and Host. When a Vary header is specified for the URL, the values in the header field is also considered while serving content from the proxy. In the above case, the proxy cache will have to cache multiple copies of content for each value of Accept-Encoding header values as well.

As you can see, the Accept-Encoding is the most used field in the Vary header. A possible value for Accept-Encoding is

Accept-Encoding: gzip, deflate, sdch

which means the clients that support compression will be served the compressed content while other will be served the inflated (or uncompressed) data.

Now this basically allows for more content to be cached and also to be cached correctly. Without the Vary header, there is a possibility that some browsers will get served the same content whether it supports compression or not.

Where/How can you add Vary Header?

If you are using Apache's mod_deflate module then this value should have been added automatically by the module. Of course, you are reading this because it did not add it and the analyzer (or somebody) is complaining about it. You might want to check out the configuration files to make sure that it is implemented correctly.

Usually this is set on the web server configuration files. In the case of Apache web servers, you can add the following to the .htaccess file

<IfModule mod_headers.c>
<FilesMatch "\.(js|css|xml|gz)$">
Header append Vary: Accept-Encoding
</FilesMatch>
</IfModule>

If you happen to use another web server, then you should be able to find corresponding functionality in your web server as well. As I mentioned above, you can add multiple header field names to the Vary header. An example of adding multiple fields is shown below:

Vary: Accept-Encoding,User-Agent,Cookie,Referer

This is very much acceptable and valid. But as you can see it defeats the whole purpose of caching. It is very unlikely that two clients will have the same set of values for all these fields and it is quite possible that the cache will never re-use the content within the time period it is cached.

If you host the website on Google App Engine then you do not have direct access to the web server configuration file. However, you can still set the Vary header in the app.yaml file for your application.

In the app.yaml, add http headers for the each of the url sections that you need the Vary header for. An example for the files served from the /images folder is :

- url: /image
static_dir: static/images
http_headers:
Vary: Accept-Encoding=

Overall, it is good practice to set the Vary header, as it helps with caching and allows your web content to be served correctly and faster thus saving both on bandwidth and server load.

Further reading : http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html