Configure varnish cache on front of LAMP server with ubuntu 16.04
Varnish Cache is a caching HTTP reverse proxy, or HTTP accelerator, which reduces the time it takes to serve content to a user. The main technique it uses is caching responses from a web or application server in memory, so future requests for the same content can be served without having to retrieve it from the web server. Performance can be improved greatly in a variety of environments, and it is especially useful when you have content-heavy dynamic web applications.
I have implemented improves version and took references from digitalocean guide
You need a LAMP or LEMP stack to implement varnish. I consider you have already have installed it.
—————– Ubuntu 14.04 varnish install ————–
Now add the Varnish GPG key to apt
curl https://repo.varnish-cache.org/ubuntu/GPG-key.txt | sudo apt-key add -
Then add the Varnish 4.0 repository to your list of apt sources:
sudo sh -c 'echo "deb https://repo.varnish-cache.org/ubuntu/ trusty varnish-4.0" >> /etc/apt/sources.list.d/varnish-cache.list'
Update apt-get and install Varnish with the following commands
sudo apt-get update sudo apt-get install varnish
—————– Ubuntu 16.04 varnish install ————–
apt-get install varnish -y
Restart varnish
service varnish restart
Ubuntu 16.04
systemctl daemon-reload systemctl restart apache2.service systemctl restart varnish.service
Check varnish is installed & what version of varnish is installed.
varnishd -V
By default, Varnish is configured to listen on port 6081 and expects your web server to be on the same server and listening on port 8080. Open a browser with port 6081 of your server.
http://DOMAIN-NAME.COM:6081
You will get this message as follow. it means varnish is working and varnish could not find the cahed file.
First, we will configure Varnish to use our LAMP as a backend.
The Varnish configuration file is located at /etc/varnish/default.vcl. Let’s backup and edit the configuration
mv /etc/varnish/default.vcl /etc/varnish/default-2015-01-28.vcl
nano /etc/varnish/default.vcl
#
# This is an example VCL file for Varnish.
#
# It does not do anything by default, delegating control to the
# builtin VCL. The builtin VCL is called when there is no explicit
# return statement.
#
# See the VCL chapters in the Users Guide at https://www.varnish-cache.org/docs/
# and http://varnish-cache.org/trac/wiki/VCLExamples for more examples.
# Marker to tell the VCL compiler that this VCL has been adapted to the
# new 4.0 format.
vcl 4.0;
# Default backend definition. Set this to point to your content server.
backend default {
# .host = "127.0.0.1";
# change the ip where your application is placed (apache or ngnix)
.host = "104.236.243.206";
# port on which application is accessible
.port = "8888";
# .probe = {
# #.url = "/"; # short easy way (GET /)
# # We prefer to only do a HEAD /
# .request =
# "HEAD / HTTP/1.1"
# "Host: localhost"
# "Connection: close";
# .interval = 5s; # check the health of each backend every 5 seconds
# .timeout = 1s; # timing out after 1 second.
# # If 3 out of the last 5 polls succeeded the backend is considered healthy, otherwise it will be marked as sick
# .window = 5;
# .threshold = 3;
# }
# .connect_timeout = 600s;
# .first_byte_timeout = 600s;
# .between_bytes_timeout = 600s;
# .max_connections = 800;
}
# Only allow purging from specific IPs
acl purge {
"localhost";
"127.0.0.1";
"104.236.243.206";
}
/*
acl editors {
# ACL to honor the "Cache-Control: no-cache" header to force a refresh but only from selected IPs
"localhost";
"127.0.0.1";
"::1";
}
*/
# This function is used when a request is send by a HTTP client (Browser)
sub vcl_recv {
set req.http.grace = 300;
# Normalize the header, remove the port (in case you're testing this on various TCP ports)
set req.http.Host = regsub(req.http.Host, ":[0-9]+", "");
# Allow purging from ACL
# if (req.method == "PURGE") {
# # If not allowed then a error 405 is returned
# if (!client.ip ~ purge) {
# return(synth(405, "This IP is not allowed to send PURGE requests."));
# }
# # If allowed, do a cache_lookup -> vlc_hit() or vlc_miss()
# return (purge);
# }
if (req.method == "PURGE") {
# If not allowed then a error 405 is returned
if (!client.ip ~ purge) {
return(synth(405, "This IP is not allowed to send PURGE requests."));
}
return (purge);
}
if (req.method == "BAN") {
# If not allowed then a error 405 is returned
if (!client.ip ~ purge) {
return(synth(405, "This IP is not allowed to send PURGE requests."));
}
# Banning with domain or ip specific urls.
# ban("req.http.host == " + req.http.host + " && req.url == " + req.url);
# Banning without domain name or ip with exact matching
# ban("req.url == " + req.url);
# Banning without domain name or ip with matching of string and regular expressions can be used.
ban("req.url ~ " + req.url);
return (purge);
}
# Post requests will not be cached
if (req.http.Authorization || req.method == "POST") {
return (pass);
}
# Remove has_js and CloudFlare/Google Analytics __* cookies.
set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(_[_a-z]+|has_js)=[^;]*", "");
# Remove a ";" prefix, if present.
set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", "");
# Did not cache the wordpress admin and login pages
# if (req.url ~ "wp-(login|admin)" || req.url ~ "preview=true") {
# return (pass);
# }
# Did not cache the admin and login pages
if ( req.url ~ "admin|xajax" ) {
return (pass);
}
# Remove the "has_js" cookie
# set req.http.Cookie = regsuball(req.http.Cookie, "has_js=[^;]+(; )?", "");
# Remove any Google Analytics based cookies
# set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(; )?", "");
# Remove the Quant Capital cookies (added by some plugin, all __qca)
# set req.http.Cookie = regsuball(req.http.Cookie, "__qc.=[^;]+(; )?", "");
# Remove the wp-settings-1 cookie
# set req.http.Cookie = regsuball(req.http.Cookie, "wp-settings-1=[^;]+(; )?", "");
# Remove the wp-settings-time-1 cookie
# set req.http.Cookie = regsuball(req.http.Cookie, "wp-settings-time-1=[^;]+(; )?", "");
# Remove the wp test cookie
# set req.http.Cookie = regsuball(req.http.Cookie, "wordpress_test_cookie=[^;]+(; )?", "");
# Are there cookies left with only spaces or that are empty?
# if (req.http.cookie ~ "^ *$") {
# unset req.http.cookie;
# }
# Remove all cookies, Let the story end once since for all ;)
unset req.http.cookie;
# Cache the following files extensions and remove cookie data from them.
if (req.url ~ "\.(css|js|png|gif|jp(e)?g|swf|ico)") {
unset req.http.cookie;
}
# Normalize Accept-Encoding header and compression
# https://www.varnish-cache.org/docs/3.0/tutorial/vary.html
if (req.http.Accept-Encoding) {
# Do no compress compressed files...
if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
unset req.http.Accept-Encoding;
} elsif (req.http.Accept-Encoding ~ "gzip") {
set req.http.Accept-Encoding = "gzip";
} elsif (req.http.Accept-Encoding ~ "deflate") {
set req.http.Accept-Encoding = "deflate";
} else {
unset req.http.Accept-Encoding;
}
}
# Check the cookies for wordpress-specific items
# if (req.http.Cookie ~ "wordpress_" || req.http.Cookie ~ "comment_") {
# return (pass);
# }
if (!req.http.cookie) {
unset req.http.cookie;
}
# Did not cache HTTP authentication and HTTP Cookie
if (req.http.Authorization || req.http.Cookie) {
# Not cacheable by default
return (pass);
}
# Cache all others requests
return (hash);
}
sub vcl_pipe {
return (pipe);
}
sub vcl_pass {
return (fetch);
}
# The data on which the hashing will take place
sub vcl_hash {
hash_data(req.url);
if (req.http.host) {
hash_data(req.http.host);
} else {
hash_data(server.ip);
}
# If the client supports compression, keep that in a different cache
if (req.http.Accept-Encoding) {
hash_data(req.http.Accept-Encoding);
}
return (lookup);
}
# This function is used when a request is sent by our backend (Nginx server)
sub vcl_backend_response {
# Remove some headers we never want to see
unset beresp.http.Server;
unset beresp.http.X-Powered-By;
# For static content strip all backend cookies
if (bereq.url ~ "\.(css|js|png|gif|jp(e?)g)|swf|ico") {
unset beresp.http.cookie;
}
# Don't store backend
# if (bereq.url ~ "wp-(login|admin)" || bereq.url ~ "preview=true") {
# set beresp.uncacheable = true;
# set beresp.ttl = 30s;
# return (deliver);
# }
# Don't store backend
if (bereq.url ~ "admin|xajax" || bereq.url ~ "preview=true") {
set beresp.uncacheable = true;
set beresp.ttl = 30s;
return (deliver);
}
# Only allow cookies to be set if we're in admin area
# if (!(bereq.url ~ "(wp-login|wp-admin|preview=true)")) {
# unset beresp.http.set-cookie;
# }
if (!(bereq.url ~ "admin|xajax")) {
unset beresp.http.set-cookie;
}
# don't cache response to posted requests or those with basic auth
if ( bereq.method == "POST" || bereq.http.Authorization ) {
set beresp.uncacheable = true;
set beresp.ttl = 120s;
return (deliver);
}
# don't cache search results
# if ( bereq.url ~ "\?s=" ){
# set beresp.uncacheable = true;
# set beresp.ttl = 120s;
# return (deliver);
# }
# only cache status ok
# if ( beresp.status != 200 ) {
# set beresp.uncacheable = true;
# set beresp.ttl = 120s;
# return (deliver);
# }
# A TTL of 2h
set beresp.ttl = 2h;
# Define the default grace period to serve cached content
set beresp.grace = 300s;
/* Remove Expires from backend, it's not long enough */
unset beresp.http.expires;
# unset beresp.http.Pragma;
/* Set the clients TTL on this object */
# set beresp.http.cache-control = "max-age=900";
/* Set how long Varnish will keep it */
set beresp.ttl = 1w;
/* marker for vcl_deliver to reset Age: */
# set beresp.http.magicmarker = "1";
return (deliver);
}
# The routine when we deliver the HTTP request to the user
# Last chance to modify headers that are sent to the client
sub vcl_deliver {
# Add a header for identifying cache hits/misses.
if (obj.hits > 0) {
set resp.http.X-Cache = "HIT - "+obj.hits;
} else {
set resp.http.x-Cache = "MISS";
}
# Remove some headers: PHP version
# unset resp.http.X-Powered-By;
set resp.http.X-Powered-By = "Publishrr.com";
# Remove some headers: Apache version & OS
# unset resp.http.Server;
set resp.http.server = "gws";
# Remove some heanders: Varnish
unset resp.http.Via;
unset resp.http.X-Varnish;
return (deliver);
}
sub vcl_init {
return (ok);
}
sub vcl_fini {
return (ok);
}
# In the event of an error, show friendlier messages.
sub vcl_backend_error {
set beresp.http.Content-Type = "text/html; charset=utf-8";
set beresp.http.Retry-After = "5";
synthetic( {"
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
<head>
<script>error_name="503 - Backend fetch failed";event_action="503 - Backend fetch failed";event_label="503 - Backend fetch failed";event_value="503";content_displaed_on_page="503 Varnish Error !";document.title=window.location.hostname + " - " + error_name;(function(i, s, o, g, r, a, m){i['GoogleAnalyticsObject']=r;i[r]=i[r] || function(){(i[r].q=i[r].q || []).push(arguments)}, i[r].l=1 * new Date();a=s.createElement(o),m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a, m)})(window, document, 'script', '//www.google-analytics.com/analytics.js', 'ga');ga('create', 'UA-39640226-2', 'auto');ga('require', 'displayfeatures');ga('require', 'linkid', 'linkid.js');ga('send', 'pageview');ga('set', 'nonInteraction', true);ga('send', 'event',{eventCategory: window.location.hostname, eventAction: event_action, eventLabel: event_label, eventValue: event_value});var dimensionValue=event_value;ga('set', 'dimension1', dimensionValue);var dimensionValue=window.location.hostname;ga('set', 'dimension2', dimensionValue);ga('send', 'exception',{'exDescription': error_name,'exFatal': true});</script>
<link href='http://maxcdn.bootstrapcdn.com/font-awesome/4.2.0/css/font-awesome.min.css' rel='stylesheet'><style>*:before, *:after{-webkit-box-sizing: border-box;-moz-box-sizing: border-box;box-sizing: border-box;}body{font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif;color: #394263;font-size: 13px;background-color: #FFF;}#error-container{padding: 120px 20px;position: relative;}#error-container .error-options{position: absolute;top: 20px;left: 20px;}#error-container h1{font-size: 96px;color: #431111;margin-bottom: 40px;}#error-container h2{color: #11103D;font-size: 24px;margin-bottom: 40px;margin-top: 80px;line-height: 1.4;}#error-container form{padding: 20px;border-radius: 3px;background: #fff;background: url(../img/template/ie8_opacity_light_10.png) repeat;background: rgba(255, 255, 255, .1);}#error-container .form-control{border-color: #fff;}.clearfix:before, .clearfix:after, .container:before, .container:after, .container-fluid:before, .container-fluid:after, .row:before, .row:after, .form-horizontal .form-group:before, .form-horizontal .form-group:after, .btn-toolbar:before, .btn-toolbar:after, .btn-group-vertical>.btn-group:before, .btn-group-vertical>.btn-group:after, .nav:before, .nav:after, .navbar:before, .navbar:after, .navbar-header:before, .navbar-header:after, .navbar-collapse:before, .navbar-collapse:after, .pager:before, .pager:after, .panel-body:before, .panel-body:after, .modal-footer:before, .modal-footer:after{content: ' ';display: table;}.col-sm-offset-2{margin-left: 16.66666667%}.col-sm-8{width: 66.66666667%}@media (min-width:768px){.col-sm-1, .col-sm-2, .col-sm-3, .col-sm-4, .col-sm-5, .col-sm-6, .col-sm-7, .col-sm-8, .col-sm-9, .col-sm-10, .col-sm-11, .col-sm-12{float: left;}.col-xs-1, .col-sm-1, .col-md-1, .col-lg-1, .col-xs-2, .col-sm-2, .col-md-2, .col-lg-2, .col-xs-3, .col-sm-3, .col-md-3, .col-lg-3, .col-xs-4, .col-sm-4, .col-md-4, .col-lg-4, .col-xs-5, .col-sm-5, .col-md-5, .col-lg-5, .col-xs-6, .col-sm-6, .col-md-6, .col-lg-6, .col-xs-7, .col-sm-7, .col-md-7, .col-lg-7, .col-xs-8, .col-sm-8, .col-md-8, .col-lg-8, .col-xs-9, .col-sm-9, .col-md-9, .col-lg-9, .col-xs-10, .col-sm-10, .col-md-10, .col-lg-10, .col-xs-11, .col-sm-11, .col-md-11, .col-lg-11, .col-xs-12, .col-sm-12, .col-md-12, .col-lg-12{position: relative;min-height: 1px;padding-left: 15px;padding-right: 15px;}.text-center{text-align: center;}*{-webkit-box-sizing: border-box;-moz-box-sizing: border-box;box-sizing: border-box;}.animation-tossing{animation-name: tossing;-webkit-animation-name: tossing;animation-duration: 2.5s;-webkit-animation-duration: 2.5s;animation-iteration-count: infinite;-webkit-animation-iteration-count: infinite;}@keyframes tossing{0%{transform: rotate(-4deg);}50%{transform: rotate(4deg);}100%{transform: rotate(-4deg);}}@-webkit-keyframes tossing{0%{-webkit-transform: rotate(-4deg);}50%{-webkit-transform: rotate(4deg);}100%{-webkit-transform: rotate(-4deg);}}.h1, .h2, .h3, .h4, .h5, .h6, h1, h2, h3, h4, h5, h6{font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-weight: 300;}a{//text-decoration: none;color: #2554B4;pointer: pointer;}.text-danger, .text-danger:hover, a.text-danger, a.text-danger:focus, a.text-danger:hover{color: #e74c3c;}/*@font-face{font-family: 'FontAwesome';src: url('../fonts/fontawesome-webfont.eot?v=4.1.0');src: url('../fonts/fontawesome-webfont.eot?#iefix&v=4.1.0') format('embedded-opentype'), url('../fonts/fontawesome-webfont.woff?v=4.1.0') format('woff'), url('../fonts/fontawesome-webfont.ttf?v=4.1.0') format('truetype'), url('../fonts/fontawesome-webfont.svg?v=4.1.0#fontawesomeregular') format('svg');font-weight: normal;font-style: normal;}*/.fa-gear:before, .fa-cog:before{content: '\f013'}.fa{display: inline-block;font-family: FontAwesome;font-style: normal;font-weight: normal;line-height: 1;-webkit-font-smoothing: antialiased;-moz-osx-font-smoothing: grayscale;}.fa-spin{-webkit-animation: spin 2s infinite linear;-moz-animation: spin 2s infinite linear;-o-animation: spin 2s infinite linear;animation: spin 2s infinite linear;}@-moz-keyframes spin{0%{-moz-transform: rotate(0deg);}100%{-moz-transform: rotate(359deg);}}@-webkit-keyframes spin{0%{-webkit-transform: rotate(0deg);}100%{-webkit-transform: rotate(359deg);}}@-o-keyframes spin{0%{-o-transform: rotate(0deg);}100%{-o-transform: rotate(359deg);}}@keyframes spin{0%{-webkit-transform: rotate(0deg);transform: rotate(0deg);}100%{-webkit-transform: rotate(359deg);transform: rotate(359deg);}</style></head>
<body>
<div id='error-container'><div class='row'><div class='col-sm-8 col-sm-offset-2 text-center'><h1 class='animation-tossing'><i class='fa fa-cog fa-spin text-danger'></i><script>document.write(content_displaed_on_page);</script></h1><h2 class='h3'>Don't worry, we'll be back.<br>Click <a href='javascript:document.location.reload(true);'>here</a> to reload the page</h2></div></div>
</body>
</html>
<!-- Comment out default error html block
<html>
<head>
<title>"} + beresp.status + " " + beresp.reason + {"</title>
</head>
<body>
<h1>Error "} + beresp.status + " " + beresp.reason + {"</h1>
<p>"} + beresp.reason + {"</p>
<h3>Guru Meditation:</h3>
<p>XID: "} + bereq.xid + {"</p>
<hr>
<p>Varnish cache server</p>
</body>
</html>
-->
"} );
return (deliver);
}
Now you need to make varnish on the frontend and apache on the backend when varnish cache is not available.
Edit varnish configuration
nano /etc/default/varnish
Change varnish listening port to 80
DAEMON_OPTS="-a :80 \
There are changes in ubuntu 15.04 and the above file will not work. please follow this guide https://www.varnish-cache.org/docs/trunk/tutorial/putting_varnish_on_port_80.html
Open apache configuration and change apache port to 8888
nano /etc/apache2/ports.conf
Restart varnish and apache
service varnish restart service apache2 restart
Now open main domain and check if varnish is serving cached version or not.
http://DOMAIN-NAME.COM
First time it will load data from apache and in reponse header you will get x-Cache: MISS
You can reload the page again and varnish will serve cached data and you will get x-Cache: HIT – 3
To make sure you can stop apache to see if varnish is serving data.
service apache2 stop
Purge one file via command line
curl -X PURGE "http://example.com/abc.html"
Purge whole domain via command line.
curl -X PURGE "http://example.com/*"
Ban one file via command line
curl -X BAN "http://example.com/abc.html"
Ban whole domain via command line.
curl -X BAN "http://example.com/*"
This is the php function which can be used to test varnish purge or use “Chrome Postman” to send the call.
function purgeURL( $hostname, $port, $purgeURL, $debug )
{
$finalURL = sprintf(
"http://%s:%d%s", $hostname, $port, $purgeURL
);
print( "Purging ${finalURL}\n" );
$curlOptionList = array(
CURLOPT_RETURNTRANSFER => true,
CURLOPT_CUSTOMREQUEST => 'PURGE',
CURLOPT_HEADER => true ,
CURLOPT_NOBODY => true,
CURLOPT_URL => $finalURL,
CURLOPT_CONNECTTIMEOUT_MS => 2000
);
$fd = false;
if( $debug == true ) {
print "\n---- Curl debug -----\n";
$fd = fopen("php://output", 'w+');
$curlOptionList[CURLOPT_VERBOSE] = true;
$curlOptionList[CURLOPT_STDERR] = $fd;
}
$curlHandler = curl_init();
curl_setopt_array( $curlHandler, $curlOptionList );
curl_exec( $curlHandler );
curl_close( $curlHandler );
if( $fd !== false ) {
fclose( $fd );
}
}
This is a very good configuration for varnish
https://github.com/mattiasgeniar/varnish-4.0-configuration-templates/blob/master/default.vcl
Varnish Book
https://www.varnish-software.com/static/book/
Varnish Documentation
https://www.varnish-cache.org/docs
Varnish 4 Toturial
https://www.varnish-cache.org/docs/4.0/tutorial/
Varnish 3,4 Configuration
http://foshttpcache.readthedocs.org/en/latest/varnish-configuration.html