Extending the Kirby page cache with Etag validation and custom flush rules

Already a very lean system to start with, Kirby CMS features a built-in “page cache” feature that stores rendered pages and serves those copies on subsequent requests. An ingenious plugin developed by the core team pointed me to an approach to significantly improve site performance even further.

Kirby is a very performant CMS to start with (see this recent benchmark test), and is already enjoyably responsive without any server-side caching in place. Yet, instead of re-rendering for every visitor, the “page cache” feature (which last year saw a thorough overhaul) allows to keep a copy of a page the first time it is requested and serve that copy to subsequent users, eliminating the need to re-render everything.

Looking at it carefully, there are several aspects where this default implementation still leaves room for granular customization. And as always, almost everything in Kirby can be modified or extended with ease.

Minimizing server load, for faster response times

Even Kirby with its light footprint inevitably puts some load on the server for every request, as the CMS has to be initialized before the system can check whether a cached copy exists. If it does, the entire launch of the CMS scaffolding was a “wasted” effort, as all that remains to be done for the system is to send the cached copy out to the user.

Earlier last year, Bastian Allgeier published a (then experimental; recently released as stable) plugin addressing this issue: aptly named “Staticache”, it overrides the core page cache mechanism to write cached pages directly into a structure of static HTML pages; essentially turning Kirby into an “on-demand static site generator”. The web server can then be configured to instantly check whether a static copy exists and only hand requests down to the CMS where that is not the case. Not a single line of PHP code has to be executed for cached pages. A brilliant idea, which rightfully gained quite an enthusiastic response.

One limitation to be aware of: there is no way to selectively serve the stored copies based on certain conditions, for example to serve cached versions to the general public only, while logged-in users always see freshly rendered copies (which may contain additional content etc.). By default, the “Staticache” approach works really well for websites that do not need to make any such distinction – which, frankly, is likely to cover a lot of sites out there already!

Idea #1: A small addition to the Staticache configuration

To utilize the Staticache plugin for sites with more specific requirements, one option would be to skip checking for static copies if certain conditions apply. While the range of conditional logic to be included in the server configuration is limited, this could be based on the presence of a session cookie by wrapping the according line in the server configuration with a conditional clause – e.g. on an Apache server:

# additional condition only applying the redirect if no session cookie is present
RewriteCond %{HTTP_COOKIE} !^.*kirby_session.*$ [NC]
# the rest of the redirect can remain as is
RewriteCond %{DOCUMENT_ROOT}/static/%{REQUEST_URI}/index.html -f [NC]
RewriteRule ^(.*) %{DOCUMENT_ROOT}/static/%{REQUEST_URI}/index.html [L]

This bypasses the static cache for any user who ever received a session cookie for this site, which works reasonably well, but may still deprive some users of potential performance benefits (presence of the cookie does not indicate a valid session). Careful evaluation is needed to see whether this strategy is suitable for a project.

Idea #2: A negligible performance penalty for more flexibility

Bastian’s underlying idea – avoiding to boot up the CMS for cached responses – can be adapted to create a slightly less direct, but at the same time significantly more flexible setup: what if, instead of directing the web server to check for cached “static” copies, we use some ultralight PHP code to do the same thing?

While writing this up, I noticed that a fork by Bruno Meilick had ventured into this direction as well, with a slightly different approach; it was also recently added to the Staticache Readme.

To do this, instead of hooking the Staticache plugin into the server configuration, we modify the index.php file in the server root. This script is in charge of bootstrapping Kirby for all requests, and we can add a “cache check” before it gets to do that (this is a simplified implementation; check the Staticache Readme for a complete example):

<?php

// with Staticache, the path of the static copy is easy to construct
$file = __DIR__ . '/static' . rtrim('/', $_SERVER['REQUEST_URI']) . '/index.html';

// if such file exists, we pipe it to the user and never proceed to load Kirby
if (file_exists($file)) {
  echo file_get_contents($file);
  exit;
}

// normally the first line in this script
require 'kirby/bootstrap.php';
…

Compared to firing up the entire Kirby core, the performance gain is measurable (albeit not massive, given Kirby’s small overall footprint, I was still positively surprised; after all, not loading an entire CMS to deal out a static cache copy does make a difference), but even more so is the increase in customization possibilities. And in my (admittedly not 100% scientific) tests, I could not detect a noticeable difference in response times to Staticache’s original configuration on the server level; such plain PHP scripts are blazing fast to execute. But with this approach, it is possible to introduce additional if-clauses and checks for certain circumstances where we may or may not want to deliver a cached copy.

Yet, this is only half the path to optimizing performance for our users.

Enabling user agents to check whether a cached copy is still up-to-date

While server-side caching strategies aim to minimize the time to process and deliver data to the user agent, the second part of an efficient setup is the browser cache: to create a snappy user experience, the goal must be to avoid any unnecessary requests and/or data traffic, while still keeping stale or expired content away from the end-user.

A commonly used, simple way to achieve this is a Cache-Control: max-age=86400 header or – as a fallback for very old user agents – an Expires header with a timestamp 24h later. This works fine for websites that do not change often, but is not suitable for any pages where serving the latest version instantly is important (any shops/e-commerce, for sure, but in principle also blog indexes, news pages etc.).

The Staticache approach actually brings a benefit that may not be obvious at first sight: since the web server is serving static files instead of dynamic script output, it can be (and most hosting servers are) configured to calculate ETag values and return Etag and Last-Modified headers. In combination with suitable Cache-Control headers (like no-cache or must-revalidate), they enable browsers to verify whether their locally cached copy is still up-to-date.

This generally works with the default Staticache setup, when the server configuration is used, but not with dynamically hi-jacking the request via index.php as outlined above (since those are again dynamic responses). But, even with the static configuration, both the Etag and Last-Modified headers are going to change constantly: Kirby’s page cache always empties the entire cache whenever a page, file, user, or site data is modified (while wasteful at first sight, there are very sound reasons for that – more about that later). That way, even pages that have not changed at all receive new “freshness attributes”, simply because the cache files were re-created.

To deal with this, the page cache mechanism has to be modified. My experimental “CachETag” (as in Cache + ETag) plugin is an exploration into dealing with that:

• It extends the page cache workflow at the point where cache copies are created (the first time a page is requested after the cache got last emptied), to calculate an MD5 hash of the HTML code and headers for the response and store it; the original, JSON-based page cache implementation already accommodates storing such additional data – which is the main reason for dropping Staticache in favor of the core page cache mechanism at this point.

• The calculated ETag hash and a suitable Cache-Control header are returned to the browser that triggered this initial caching process. (This could still be extended by a Last-Modified header based on the timestamp of the content file, but since that may have changed without the content or its rendering affected, it is more practical to only use Etag headers – as long as the checksum of the headers and payload remains the same, we can consider the page unchanged.)

• On subsequent requests, a slightly extended version of the minimialist PHP code in the root index.php file accesses the regular page cache storage and looks for a stored version – again, without ever invoking the Kirby CMS itself; just some very lightweight PHP operations. If a cached copy exists, the script can validate the stored ETag from the JSON file against the one submitted by the browser in an If-None-Match header – if they are identical, so is their HTML content and all that needs to be returned is a HTTP/1.1 304 Not Modified header. Otherwise, the newer version is returned, or – if none exists – Kirby is initialized and a new version is rendered.

Now we have a system that eliminates the overhead of loading Kirby for pages already pre-cached (the key innovation of the Staticache plugin), but with an ETag logic based on the actual content of the rendered page, unaffected by the cache copy having been re-rendered in between. While not significantly benefiting the occasional blog visitor, this can have massive benefits in UX, performance, and resource use, for navigational pages like content listings or index pages – for example websites like technical documentation or others that encourage repeat navigation via the same gateway pages. Same applies to websites that have a lot of repeat visits with rarely changing content.

Choosing suitable Cache-Control headers

With the backend side in shape, all that remains is to select the appropriate cache control headers to ensure that browsers or intermediary caches can make best use of it. For regular content pages (we are only dealing with the HTML here, image files and other assets should have their own optimized caching strategy) two options prevail:

no-cache

Cache-Control: no-cache would be a good baseline, as it – despite the slightly misleading naming – forces the browser to always ask if a newer version exists, otherwise deals out the cached version. In combination with the CachETag plugin, this means that a browser will send an HTTP request to the server when a previously cached page is accessed, but is going to display the locally cached version if the server returns a HTTP/1.1 304 Not Modified header (facilitated by the CachETag plugin).

must-revalidate

To push frontend performance even a tiny bit further, the Cache-Control: max-age=600, must-revalidate header works like no-cache in that the browser is instructed to verify whether the cached version is still current, but adding a grace period of n seconds (so max-age=600 equals 10 minutes) that the user agent will not carry out such check but instead instantly display the cached version.

This way, users may see “stale” local copies for a limited amount of time, but get an even faster experience as there are no network requests involved during the defined time period; this can work just fine for any content that does not change often. The performance gain is most noticable on pages a user may access repeatedly during a visit, such as navigational UIs, content listings or archive pages.

As a caveat, it has to be considered that this may also delay the roll-out of changed assets – careful consideration has to go into designing an appropriate caching strategy if such changes are to be expected; not least because Kirby also changes the path to content images etc. if these have been changed.

stale-while-revalidate

A third option, though with less universal browser support, is Cache-Control: max-age=1, stale-while-revalidate=59 (or any other amounts of seconds). It works like must-revalidate in that it will revalidate from the server after the time defined in max-age since storing the currently cached version has expired, but unless the stale-while-revalidate threshold has been reached still present the cached version while updating it in the background for the next request.

This is a neat way to ensure a swift user experience while keeping the cache updated. It works well for pages that are likely to be updated regularly, but where instant delivery of an updated version is not crucial. Browsers not supporting stale-while-revalidate (notably Safari, including on iOS) fall back to the max-age header alone, which allows for a fine-grained control over the minimum cache lifespan.

Overriding the “purge complete page cache” mechanism

This leaves us with one final thing to improve: as mentioned, Kirby by default empties its entire server-side page cache whenever anything is changed through the CMS admin interface. This makes sense as a CMS default, as it cannot be known what page a content change may affect, but ultimately is a wasteful process – both regarding the need to re-render unaffected pages, but also since the first users after a cache flush may experience slower loading times.

The solution is to customize the cache flush logic. The CachETag plugin provides a simple configuration switch to turn off the default global flush in the config.php file:

'cachetag.overrideFlush' => [

  // flush entire cache when saving a page of template `article`
  'article' => '/',

  // flush only path `/portfolio` and its subpages
  // when a page of template `project` is modified
  'project' => '/portfolio',

  // flush the `/clients` page and the `/portfolio` page and
  // their subpages when modifying pages with template `client`
  'client' => ['/clients', '/portfolio'],

  // flush no pages except for itself (and possible subpages)
  // when updating a page with template `privacystatement`
  'privacystatement' => false,

],

Instead of a wholesale flush whenever something is edited, a *:after hook is used to customize cache deletion based on what has been changed – depending on the template of the modified content, the according path(s) are cleared in the cache. This is certainly not useful during development, but can be a worthwhile performance optimization tool for a mature website where these logical connections between its parts are permanent.

Conclusion: One size won’t fit all

Ultimately, as this exploration illustrates, there is no universally applicable optimum when it comes to the design of a caching strategy. It all boils down to identifying the usage patterns of a website and set up the cache accordingly. For websites with no need to deal out customized renderings of pages, the Staticache approach can be all that is needed. Yet, as soon as a high degree of repeat page visits (to single pages during a session, or to the website over time) are to be expected, the added benefit of an Etag-based browser caching strategy may be worth the (almost negligible) sacrifice of not entirely “going static”.

The main point of this exploration, however, is to illustrate just how flexible it is to adjust Kirby to custom needs – in optimizing the user experience, but also resource consumption (which, no matter how small, even has an environmental component) to fit the purpose. I was truly surprised by how easy it was to hack this together – this website has been served using my CachETag experiment for over half a year now, working flawless so far.

Maximizing cache performance in Kirby CMS

An exercise in optimizing a website caching strategy

code exploration