Full Atom archive feed for my blog by Matt Hobbs (Nooshu.com)

How a Simple CSP Tweak Turned into an AI Project

2026-06-20T00:00:00Z

Introduction

In this post, I’ll tell the story of how a stray semicolon led me to build an AI-powered tool to make creating Content Security Policies (CSPs) a bit less painful for developers.

I’ve written about CSPs on this blog before:

December 2024: Securing your static website with HTTP response headers
February 2025: Configuring your Content-Security-Policy on your development environment in 11ty

To quote from my 2024 post:

The Content Security Policy (CSP) header is a security feature that protects web applications from attacks like Cross-Site Scripting (XSS) and code injection by controlling the sources from which browsers can load and execute content. If there's only a single response header you implement today, then make it the CSP Response header.

The post has a few more details, so if you are interested you can find the Content-Security-Policy (CSP) section here.

As mentioned in the post, there are already a huge number of tools available on the web for creating CSP’s, but none of them really fit my needs. So, I decided to write my own tool using cursor.ai and host it on the web for everyone to evaluate and use. Note: I've included my referral link so any readers can get 50% off their first month subscription.

It is called CSP Playground⁠. Despite the name, writing a CSP is rarely described as “fun”, and “playground” may be overselling the experience somewhat.

It seemed as sensible a name as any, though. Besides, the domain wasn’t taken and only cost £9, which is cheap enough to make almost any naming decision seem justified!

I briefly considered splashing out on a .security top-level domain (TLD), until Cloudflare Registrar reminded me that the privilege would cost only £1,515 per year.

I suspect telling my wife that we’ve cancelled our summer holiday plans so I can own a particularly niche bit of the internet would have tested the “for richer or poorer” part of our vows rather more thoroughly than intended. 😬

Problem

If any readers have tried to write a CSP “manually”, you will understand how painful it really is! Understanding the very particular syntax is challenging. Although CSP itself does not have to be written on a single line conceptually, when the header is sent over the network, it must not contain any arbitrary line breaks, else it will be classed as invalid. And don’t even get me started on the use of ;, one misplaced semicolon and your browser console will throw a very red looking hissy fit!

There are whole RFC’s dedicated to the specifics of HTTP header fields, that I won’t even pretend to understand! Linked below for a little light reading, if you are struggling to sleep!

Anyway, as I always do, I’ve wandered off on a bit of a tangent.

What I’m basically saying is writing CSP’s is complicated and frustrating (especially for a terrible typist like myself!), So why not make life easier and spend a couple of days in "AI world” and build a tool to make your life easier? You never know, it may even help others too!

Features

Form-based editor for every standard CSP directive: No hand-editing of semicolon syntax
Import from URL: Fetch a policy from headers or meta tags, or detect when none exists
Paste headers or policy: Paste a full HTTP response header block or a single Content-Security-Policy line; the CSP is extracted in the browser and used to pre-fill the form
Real-time security score: Including actionable recommendations that navigate you instantly to the exact directive
Nonce and hash helpers: Generate values and copy ready-to-paste HTML snippets
One-click export: Policy, header line, or server config for Apache, Nginx, Caddy, Cloudflare, Netlify, Vercel, and more
Report-only mode: Option to output the CSP using Content-Security-Policy-Report-Only. Allowing for testing before enforcement
MDN links and inline help: Every directive and sandbox flag links directly to the relevant MDN content
Privacy focused by design: No accounts, no tracking, and no database. Nothing is stored because there’s nowhere to store it
Runs mostly in the browser: Policy editing and paste import stay local; the only server-side processing happens when you explicitly import a URL to fetch your site’s CSP
Free and open-source: MIT licensed at https://csp-playground.dev/, code available on GitHub
Clean design Built on the fantastic utopia.fyi Fluid Design CSS grid built by Clearleft

Audit

Start from a real policy instead of a blank form. Enter a URL and CSP Playground will retrieve the CSP from the response headers or a <meta> tag. Alternatively, you can paste the headers or the policy string directly. Everything stays in the browser, with the exception of URL lookups.

Whichever approach you choose, the editor is pre-populated so you can quickly see what is being enforced.

If the lookup finds problems (duplicates, deprecated directives, odd formatting), it flags them and suggests fixes.

For sites where there is no CSP at all, the tool does not leave a user at in the lurch: it points you to a short guide on why a policy matters and how to start safely, so auditing either an existing policy or a blank slate becomes the first step toward a stronger CSP.

Build

Whether you are starting from scratch or updating an existing policy, you can work through each CSP directive using a structured form. Add sources without worrying about syntax, generate (placeholder) nonces and hashes for inline scripts and styles, and copy ready-to-use HTML snippets.

Every directive links directly to MDN, so the documentation is always one click away when you need it.

Improve

As you edit, CSP Playground scores your policy in real time and highlights improvements you can make. Recommendations are practical, and show the expected score increase, simply click on the recommendation, and it takes you straight to the relevant directive. You can also see your potential score if you applied every suggestion. This makes it a whole lot easier to focus on the changes that matter most, even if you’ve never written a CSP before.

Deploy

When you are happy with your shiny new Content Security Policy, CSP Playground generates the full header and provides config snippets for web servers and hosting platforms:

Apache
Nginx
Caddy
LiteSpeed
Microsoft IIS
Netlify
Cloudflare Pages
Vercel
Traefik
Envoy
more can be added on request, just create an issue

You can also toggle a setting so that the server setup only attaches the CSP header to HTML responses, cutting down on repeated headers across all static assets, where they aren’t required.

Lastly, you can switch to Report-Only mode to test changes before enforcement, with links to MDN guidance to help you roll out your new (or updated) CSP safely.

Result

So, the first test for the tool was the CSP for this very website, after all it’s the reason I built it! As mentioned at the start of the post, the first result of my initial iteration of the tool caught a stray. It must have been live for quite a while, as I haven’t touched the CSP since February 2025. Thankfully, it wasn’t serious, hence why it went unnoticed. The validator picked it up and changed this:

Content-Security-Policy: base-uri 'self';child-src 'self';connect-src 'self' https://challenges.cloudflare.com/;default-src 'none';img-src 'self' https://v1.indieweb-avatar.11ty.dev/;font-src 'self';form-action 'self' https://webmention.io;frame-ancestors 'none';frame-src 'self' https://player.vimeo.com/ https://www.slideshare.net/ https://www.youtube.com/ https://giscus.app/ https://challenges.cloudflare.com/;manifest-src 'self';media-src 'self';object-src 'none';script-src 'self' 'inline-speculation-rules' https://ajax.cloudflare.com https://giscus.app/ https://challenges.cloudflare.com/;script-src-elem 'self' 'inline-speculation-rules' https://ajax.cloudflare.com https://giscus.app/ https://challenges.cloudflare.com/;style-src 'self' 'unsafe-inline' https://giscus.app/;worker-src 'self';

To this:

Content-Security-Policy: base-uri 'self';child-src 'self';connect-src 'self' https://challenges.cloudflare.com/;default-src 'none';img-src 'self' https://v1.indieweb-avatar.11ty.dev/;font-src 'self';form-action 'self' https://webmention.io;frame-ancestors 'none';frame-src 'self' https://player.vimeo.com/ https://www.slideshare.net/ https://www.youtube.com/ https://giscus.app/ https://challenges.cloudflare.com/;manifest-src 'self';media-src 'self';object-src 'none';script-src 'self' 'inline-speculation-rules' https://ajax.cloudflare.com https://giscus.app/ https://challenges.cloudflare.com/;script-src-elem 'self' 'inline-speculation-rules' https://ajax.cloudflare.com https://giscus.app/ https://challenges.cloudflare.com/;style-src 'self' 'unsafe-inline' https://giscus.app/;worker-src 'self'

It took me a little while to spot the difference because the validator simply reported a "semicolon change". If you are able to notice these small changes straight away, congratulations, you’re more observant than I am!

For any readers wondering the only difference is that the trailing semicolon in the second version has been removed. It’s actually entirely optional and has no effect on how the CSP behaves.

I admit it’s hardly a groundbreaking discovery, but it was reassuring to see that the validation functionality was doing exactly what it should.

The next changes it recommended made more of an impact and improved my CSP score. The initial score was 79%. Not bad, but we can do better with only 2 minor changes:

trusted-types (TT)

I must admit I’d never heard of this directive until the tool recommended it.

It's only very recently (February) that it has been marked as Baseline in the MDN documentation due to broad cross browser support across all modern browsers.

It was actually Mozilla who were dragging their feet on this directive, but thankfully changed their stance on it back in 2023. Full support in Firefox was released in version 148 (released 24 February 2026).

If you’re curious about this directive, here’s an ELI5 explanation:

Think of an Aeroplane at an airport.
Before TT: Anybody could walk straight onto the plane.
After TT: Everybody must:
1. Go through security.
2. Have their passport checked.
3. Receive an approved boarding pass.

Only passengers (or potentially executable content in this analogy) with a valid boarding pass are allowed onto the plane.

Trusted Types are the boarding pass.

For readers who are older than 5, you can read all about the Trusted Types API in great depth on MDN.

upgrade-insecure-requests

This directive is pretty self-explanatory given the name. upgrade-insecure-requests instructs a browser to automatically upgrade HTTP requests to HTTPS before loading resources. Resources that do not support HTTPS will fail to load, although this is very rare on the modern web.

By simply adding:

trusted-types 'none'; upgrade-insecure-requests

to my CSP, it bumped the score from 79% up to 87%!

A respectable gain for what was, in the grand scheme of things, barely any work!

Adding upgrade-insecure-requests improves protection against mixed content by upgrading HTTP resources to HTTPS. Meanwhile, trusted-types 'none' does not actually enforce Trusted Types, but it does make the decision to not use them very explicit.

For the moment I have simply disabled the use of Trusted Types in the CSP, but in the future I will likely enable enforcement via the use of:

require-trusted-types-for 'script'; trusted-types default;

Once enforced this will bring the sites CSP score up to 95%! 🎉

Update 1: I tested require-trusted-types-for 'script'; with 'trusted-types default;, but it breaks a few scripts across the site, so have reverted to trusted-types 'none';. I will update once I have resolved the issues. It just goes to show how much testing you need to do when you change your site's CSP! Use Content-Security-Policy-Report-Only if in doubt!

Update 2: It turns out these errors occur because some of these scripts use legacy code, such as:

// or element.innerHTML = `<p>${text}</p>`;
element.innerHTML = '<p>' + text + '</p>';

To properly resolve these issues, you'd need to refactor the scripts to use more standard DOM manipulation methods, like:

const p = document.createElement('p');
p.textContent = text;
element.appendChild(p);

The problem comes when you don't "own" these scripts, e.g. a 3rd-party script. It's just not practical to update these scripts without the original author(s) being involved, as any new version of the script will overwrite the changes.

There is a workaround:

// run this code immediately using an Immediately Invoked Function Expression (IIFE)
(function () {
	// check see if the browser supports the trustedTypes API
	if (window.trustedTypes?.createPolicy) {
		// Create a Trusted Types policy called "default" policy for the browser to use e.g. `trusted-types default;`
		trustedTypes.createPolicy('default', {
			// allow all HTML through e.g. I trust every HTML string. (this is bad!)
			createHTML: (string) => string,
		});
	}
})();

It is critical that this code executes before the browser tries to execute innerHTML, as without the policy the code will fail. Unfortunately, this code has a major security implication: XSS vulnerabilities are no longer prevented.

Given this implication, I'm going to stick to trusted-types 'none'; for the moment, or at least until I look into using DOMPurify to "enable" XSS protection.

Update 3: I have tried a solution using ) on my preview environments, and I'm still seeing a host of little issues like missing images in some functionality, so I'll stick with trusted-types 'none'; for the moment as it's the only otion that works at the moment. It's unfortunate that it really isn't adding much in terms of site security, and it's only there for the "CSP Score", but on the positive side it's been another CSP learning curve for me! So when using the trusted-types directive, you should definetly expect issues with JavaScript that uses legacy code! It could be a while before anyone gets to roll out this directive without issues!

Update 4: I have tried a solution using DOMPurify on my preview environments, and I'm still seeing a host of little issues like missing images in some functionality. So, I'll stick with for the moment, as it's the only option that works at the moment. It's unfortunate that it really isn't adding much in terms of site security, and it's only there for the "CSP Score", but on the positive side it's been another CSP learning curve for me! So when using the trusted-types directive, you should definitely expect issues with JavaScript that uses legacy code! It could be a while before anyone gets to roll out this directive without issues!

Summary

I promised myself that I wouldn’t make this blog post an epic read this time, so fingers crossed I have kept my promise! If you happen to be interested in my “strategy” and opinion (for what it's worth) on AI assisted coding off the back of this post, I’ve wrote all about it here in my FractalAI: Generating Infinity in the Browser post from April.

Lastly, you can find CSP Playground here, and the GitHub repo for the code here. PRs and Issues welcome!

I hope you find this little tool useful! I’ve enjoyed building it and learning a little more about the world of CSPs, and Trusted Types.

As always, thanks for reading, feedback and comments are welcome. You can contact me here.

Post changelog:

20/06/26: Initial post published.
20/06/26: Reverted require-trusted-types-for 'script'; with trusted-types default; to trusted-types 'none'; because it broke a number of scripts on the site!
20/06/26: Added information about the require-trusted-types-for 'script'; workaround before 3rd-party scripts are updated to use non-legacy DOM methods. Sticking with trusted-types 'none'; for the moment.
21/06/26: I tried to resolve issues with require-trusted-types-for 'script'; with trusted-types default; by using DOMPurify, but there are still other problems that pop up, so still sticking with trusted-types 'none';. I'll chalk this one up to a learning exercise, and leave it as is for now! After all, this post was only supposed to be about CSP Playground, not the Trusted Types directive.

FractalAI: Generating Infinity in the Browser

2026-04-21T00:00:00Z

Why I Built FractalAI

Although AI has been a concept that has been discussed for the past 80 years (Thanks to Alan Turing, and many others), it’s only in the past 5 to 10 years that it has become vastly more powerful. It is currently the hot topic in pretty much every industry on the planet! Software development in particular has already changed drastically, with new tools and AI Models popping up every week. You only have to look at the Stack Overflow Developer Survey 2025 to see how common its usage is in Software development:

84% of developers are using or planning to use AI
51% of professional developers use it daily
61% of professional developers look at it favourably
35% of developers visit Stack Overflow due to AI-related issues

So like it or not, AI is here, and it is here to stay. So to keep up with "the pack” it’s probably a good idea to start learning how to use it, (and more importantly) where to be cautious!

The Constraint: 100% AI-Generated Code

To get to grips with AI and its current limits, I decided to set myself a task: build a new hobby project using only AI. All code, all functionality, and even all the design would be generated by me prompting the AI to tweak and generate the frontend code. This is an excellent way to see how AI handles CSS and general usability. For this, I use an AI Tool called Cursor. There are many tools available, but since subscribing to their pro plan, I’ve seen no real reason to change. Some of the key advantages I see over other AI tools are:

Project-wide context awareness: Cursor understands the entire codebase rather than a single file, which leads to more accurate suggestions and safer large-scale changes across interconnected systems.
Multi-file editing and refactoring: Cursor can apply consistent updates across multiple files in one action, making large refactors and cross-cutting changes significantly more efficient.
AI-driven workflows: Cursor acts as an AI agent that can plan and execute multistep development tasks rather than just offering inline code suggestions.
Context-aware integrated chat: Cursor’s built-in chat has full awareness of the repository and can directly modify code, reducing the need for manual context sharing.
Model flexibility: Cursor allows developers to switch between different AI models, enabling optimisation for performance, cost, or task-specific quality.
Optimised for complex systems: Cursor performs particularly well in large and complex codebases where deeper reasoning and architectural understanding are required.
Faster end-to-end task completion: Cursor often reduces overall development time by completing entire tasks in fewer steps, even if individual code suggestions are not the fastest.
AI-native IDE design: Cursor is built from the ground up as an AI-first environment, enabling tighter integration and more advanced workflows than plugin-based alternatives.

FYI: I’m not sponsored by Cursor, I just love how clean and simple it is too pickup and use! There are many others on the web that you can choose instead!

What FractalAI Actually Does

Well, as you may have guessed already from the name of the blog post, the project I chose to build what a fractal generator 100% built on modern browser technology. The code is all open-source and free for anyone to clone, modify and use as they wish! The code repository is on GitHub here.

The project currently comes with the following features:

Fractal Selection: Choose from 100+ fractal types via dropdown menu.
Adjustable Iterations: Control detail level (10 to 400 iterations).
35+ Colour Schemes: Classic, Fire, Ocean, Rainbow variants, Monochrome, Forest, Sunset, Purple, Cyan, Gold, Ice, Neon, Cosmic, Aurora, Coral, Autumn, Midnight, Emerald, Rose Gold, Electric, Vintage, Tropical, Galaxy, Lava, Arctic, Sakura, Volcanic, Mint, Sunrise, Steel, Prism, Mystic, Amber, and more.
Zoom and Pan: Click and drag to pan, scroll or double-click to zoom.
Real-time Parameter Adjustment: Adjust Julia set constants, scales, and other parameters in real-time.
Screenshot Capture: Save current view as PNG with EXIF metadata.
FPS Monitoring: Real-time frame rate display.
Coordinate Display: View and copy exact fractal coordinates.
Presets System: Quick-load pre-configured fractal views.
Favourites System: Save and manage favourite fractal configurations.
Share & URL State: Share fractals via URL with encoded state.

This was just a copy / paste from the README.md file in the repository, it actually comes with more features that you can read about like Machine Learning-Powered Discovery, Performance Optimizations, and Advanced Features.

I’ve been fascinated with Fractals for many years, I actually dabbled with fractal generation and Web Workers back in January 2010, scarily that’s a whole 16-years ago! That makes me feel very 👴!

The reason fractals have always fascinated me is perfectly summed up in this quote from the godfather of fractals Benoît Mandelbrot:

Bottomless wonders spring from simple rules, which are repeated without end.

I find it simply astounding that a whole infinite world can be rendered by a computer using such "simple" mathematics. For example, the iconic Mandelbrot fractal is all generated via this neat little formula:

z_{n + 1} = z_{n}^{2} + c

Where:

$z_{0} = 0$
$c \in C$ (a complex number)

A point $c$ belongs to the Mandelbrot set if the sequence does not diverge:

n \to \infty lim ∣ z_{n} ∣ \neq \to \infty

Which generates this absolutely stunning image that you can literally zoom in to forever!

The image above really doesn’t do the details of the Mandelbrot justice! So why not take a look around the original setting I used to render it here.

Why the Browser Is Surprisingly Good at Rendering Fractals

What makes the browser surprisingly good at rendering fractals is not just raw capability, but accessibility and reach. A fractal renderer written in the browser can run anywhere instantly with no installation, no setup, and no platform constraints. That is a powerful starting point. Additionaly there's JavaScript, as the language behind this project, it is embedded in every modern browser, making it one of the most widely available execution environments in the world.

This accessibility isn't theoretical, it shows up in everyday life. Just last month, while walking my 8-year-old son home from primary school, he told me about what he had been learning in "Coding Club" using his BBC micro:bit. He was genuinely excited when he said:

Daddy, I’ve been learning all about something called JavaScript!

That moment really highlighted the scale of JavaScript’s reach. It is not just a professional tool, it is something being introduced at a very young age. When I explained that it forms a large part of my own work, it reinforced how deeply embedded it is in both education and industry.

It is remarkable to think that a language first created by Brendan Eich in just 10-days has evolved into a platform capable of driving complex visualisations like fractal rendering!

This, combined with modern browser features such as high-performance JavaScript engines, GPU acceleration through WebGL, and efficient canvas APIs, the browser has become an unexpectedly capable environment for this kind of computationally intensive work.

AI as a Design Partner, Not Just a Tool

I’ve found that AI is most powerful when it is treated as a design partner rather than just a tool for generating code. While it is clearly effective at implementation, its real strength lies in collaboration. You can ask it to plan a feature, and it will respond with probing questions that help clarify intent, constraints, and direction. Those questions are not incidental, they are often the key to shaping a better outcome, giving you the opportunity to guide and refine the approach early in a project.

Because of the breadth of knowledge it draws on, the interaction feels less like issuing instructions and more like a back and forth design "conversation". It becomes a space for exploring ideas, testing assumptions, and iterating quickly before committing to an ideal solution.

This became particularly valuable in areas where I would not usually feel confident, such as design. Rather than treating that as a limitation, I leaned into AI to generate an initial direction. From there, my role shifted to refinement. Once a baseline design was in place, I could make targeted adjustments to the CSS and HTML, either through follow-up prompts or by working directly with insights gathered from the Browser DevTools.

Although I had originally set myself the constraint of not writing any code, I found that introducing precise, real-world context from DevTools significantly improved the quality of the AI’s output (more on this later in the post). By feeding in specific observations rather than abstract instructions, the responses became more relevant, more predictable, and easier to iterate on.

This is where AI moves beyond being a simple tool. It becomes a collaborator that helps shape both the design thinking and the implementation, with you firmly guiding the direction at each step.

The Architecture (Without Writing Code Myself)

This was easily the most engaging part of the process and where the most learning happened. Having the freedom to choose technologies and then observe how AI composes them into a coherent architecture is genuinely insightful, especially when you examine the reasoning behind those decisions.

When you are unsure whether a technology is a good fit for a project, you can quickly create a "throwaway" prototype and evaluate it. That shifts the role of AI from a coding assistant to a decision support tool. You are not just generating output, you are generating evidence.

Over time, this creates a feedback loop where each experiment improves your judgement. You learn what works, what to avoid, and why.

This aligns closely with core Agile principles:

Fail early to surface risk
Iterate quickly to explore options
Learn continuously to improve decisions

If you use AI for anything, use it to accelerate learning. Speed only matters when it leads to better decisions.

Machine Learning Inside the Browser

For many years, I’d wanted to have a play with machine learning (ML), but had never really had the opportunity. With the arrival of browser-based libraries like TensorFlow.js, the idea of running ML directly in the browser suddenly became far more accessible.

In this project, I saw an interesting opportunity to apply it to the “random fractal position” feature. The goal was simple on the surface. A user selects a fractal and a colour scheme, presses a button, and gets a random set of x, y, and z values to explore.

In practice, it was not that simple.

When the AI first implemented this in plain JavaScript, I ran into a fundamental problem. A large proportion of fractal space is effectively empty, which meant most “random” positions resulted in a blank or uninteresting visual. I initially tried constraining the co-ordinates to specific regions, but with over 100 fractals available, this quickly became unmanageable.

So I started thinking differently. Instead of hard-coding “good” regions, what if the system could learn what a good fractal view looks like?

That is where machine learning came in. The idea was to train a lightweight model to recognise visually interesting outputs, effectively teaching the random feature to make better choices over time. Using AI to help generate and iterate on this approach made the experimentation surprisingly fast.

Originally, I explored using TensorFlow.js, but it turned out to be far too heavy for this use case. Even compressed, it added around 450 KB of JavaScript, which is significant given the existing bundle size for all the fractal rendering functionality.

Instead, I opted for Synaptic. It is much smaller at around 45 KB when Brotli compressed, and while it has not seen recent updates, it proved more than capable for what I needed. It also keeps the overall performance profile of the application in check, which was a key consideration.

If required, this approach leaves the door open to swap Synaptic out for a more modern ML library in the future. But for now, it demonstrates something powerful. You can run meaningful machine learning directly in the browser, and use it to enhance user experience in ways that would be difficult to achieve with traditional logic alone.

As mentioned in the README.md file in the GitHub repository it is used for the following functionality:

ML-Based Scoring: Uses Synaptic.js neural network to score fractal configurations.
Hybrid Algorithm: Combines fast heuristic screening with ML refinement.
Personalised Learning: Trains on your favourites to learn your preferences.
"Surprise Me" Feature: Discover interesting new fractal zooms automatically.
Background Training: Model retrains automatically as you add favourites.
Local Storage: ML model and favourites persist in browser storage.

Note: You will find all of this functionality in the “Full-screen view” of the fractal viewer. The standard UI was starting to feel quite crowded, so that ended up being the most sensible place for it.

What Worked Surprisingly Well

Planning

Planning from the AI was excellent. This may be a Cursor-specific feature, but the follow-up questions it asked about how I wanted to approach a problem were a really nice touch. When you are unsure which direction the AI will take, this gives you useful insights early on.

Crafted Prompts

Top AI Tip: Detailed prompts made a huge difference. The more specific I was, the more reliable the output became. In some cases, I even included images or copied exact code from DevTools just to make sure the AI fully understood the context.

I really liked how Anthony Alicea explains AI in his Understanding AI-Assisted Software Development course. He visualises the AI “brain” as a cube, where each smaller blue cube represents a piece of knowledge the AI has learned. It’s a simple idea, but it makes how AI works much easier to grasp.

A visualisation from his course is shown below:

This “knowledge” could include things like:

Documents
Images
External data
Specifications
Examples
Instructions

Your job when prompting the AI is to “sail” across this ocean of knowledge and get to a favourable output on the opposite side. To get better results, you need to give the AI waypoints or "lighthouses" to direct the AI through this ocean of knowledge, your prompts are these "lighthouses". That way, you guide the path it takes, rather than leaving it to figure one out on its own.

So rather than saying:

Build me a website that renders fractals

Be more specific:

Build me a website that renders fractals. Start with a simple Mandelbrot set and use the Canvas API to draw it. I’d also like you to leverage the WebGL API in the visualisation. Make sure the code is well documented and uses modern linting tools like Stylelint and ESLint to keep it maintainable. I also want to use Vite as the local development server…

Now, you might be thinking, that’s a long prompt! It is. But if you want reliable output, you need to give the AI clear instructions to follow. Otherwise, it will just “set sail” across that vast corpus of information and produce something unfocused because it has too much freedom.

If there’s one thing I learned from this project, it’s this: spend time crafting detailed prompts. That’s how you steer the ship across the ocean of information and actually reach the result you want.

Rules

Another useful thing I learned is that you can give the AI a consistent set of rules to follow across every prompt. In Cursor, these live in the .cursor/rules directory in your repository.

In the FractalAI project, I ended up with four sets of rules:

The AI refers to these rules for every prompt, giving it consistent context and guidance. You might notice they look AI-generated, and you’d be right. It’s a bit meta, but I asked Cursor to write the rules it would follow in future prompts.

The thinking behind this was simple: it is far more likely to understand its own instructions than my rambling ones!

I’m not sure if this is specific to Cursor, or if other AIs follow a similar approach. It would be great to see some standardisation here to allow these rules to be portable across multiple AI tools in the future.

If this is already happening, I’d love to know. Please feel free to let me know if you have any information!

Writing Tests

If you are not a fan of writing tests, even though you know you should, AI helps a lot. If it can write the code, it can just as easily write the tests too.

Where a human developer might get tired of writing tests, AI doesn't. That makes it much easier to push towards high coverage and keep things consistent.

And if you enjoy that warm feeling of seeing a wall of green in your terminal's test output, this definitely helps. I’m one of those people:

Can you imagine how long it would take to get 100% coverage across all files, Statements, Branches, Functions, and Lines? That's a lot of work for one (obsessive) person.

Or, a perfect job for AI!

Refactoring

Where AI really excelled in this project was its ability to understand even the smallest details of the codebase. That makes refactoring fast and almost effortless.

For example, I initially used TensorFlow.js, but quickly realised it was too heavy and overly complex for what I needed. So I gave a simple prompt:

Could you swap out TensorFlow.js and use Synaptic instead? The page weight is too high and it’s more complex than required.

The important part here is the added context. By explaining why I wanted the change, I was building up the AI’s understanding of the project.

From just that, it learns that I care about performance and keeping things lightweight, which leads to better, more aligned outputs as the project progresses.

If you combine AI’s ability to refactor quickly with version control like Git, you are onto a winning combination.

You can easily branch and tag different versions of your project to experiment with alternative approaches or technologies. If a refactor does not work out, just roll back to a previous commit, or simply scrap the branch / tag.

One thing to watch out for if you do this is be wary of your node_modules folder. Projects tend to get a bit cranky if the dependencies are not installed correctly. And no, committing node_modules to your repo is not the answer! 🤮

An added bonus when using this technique is if you have good test coverage, you can swap out a technology and quickly run your tests to see what has broken. It makes validating a refactor fast and far less risky.

This is where AI and testing really come into their own. The AI can make sweeping changes, and your tests act as the safety net to catch anything that breaks.

Proof of concepts

At the start of my career in digital marketing, there was a familiar pattern. Most of the budget went on UX and design. By the time it reached the build phase, you would give an estimate and hear, “We don’t have the budget for that.”

Classic waterfall. Siloed teams. And not much left for actually building "the thing".

There was even a running joke that what we really needed was a big “build it” button, so the technology part cost next to nothing.

When it comes to proof of concepts, or spikes as some call them, you are essentially writing throwaway code. This is where that “build it” button starts to feel real.

With a bit of direction on the tech stack and constraints, you can have AI produce a working prototype in hours, not days. Especially when dealing with unfamiliar technologies.

What I’m really saying is that AI is great for testing ideas in code. Going back to the Agile principles I mentioned earlier in the post:

Fail early to surface risk
Iterate quickly to explore options
Learn continuously to improve decisions

These all map perfectly to AI-generated proof of concepts.

Word of warning: be careful using AI-generated code in live / production without proper validation. Make sure it goes through the usual checks like Pull Requests (PRs), Continuous Integration, and Deployment (CI/CD) gates.

If something goes wrong, you can't blame the AI (well you could try, but I doubt it would go down well!)

Rule of thumb: There should always be a human reviewing AI-generated code, especially when sensitive data is involved.

Making unfamiliar things make sense

I’ve touched on this a little in the previous sections, one of AI’s superpowers is being able to explain complex concepts to the user when required. This is especially true when it comes to software development and technologies a user has no experience of before. AI is basically an incredibly patient coach who is always willing to answer your questions, no mater how small or stupid they may be! When combined with a Code UI like VS Code, you are really working with a powerful workflow. Being able to highlight individual rows and blocks of code and get AI to explain exactly what is happening is a dream come true for anyone learning to code in 2026! I know I wish I had it when I was a junior developer, far to many years ago!

Another thing I found really useful is learning through optimisation. You might write a piece of code you understand, then ask the AI:

Are there any edge cases I have missed?
How can I make it more maintainable?
Does this code follow best practices for the framework?
How can I make it more performant?
Will this code perform well if the website is scaled up?
How can I make it easier to read for other developers?
etc…

These kinds of questions help reinforce your understanding, especially around the less obvious parts of a language.

Building momentum

Because I’m so old, I remember when Stack Overflow first appeared back in late 2008. It was a game changer for web development and beyond. A place to ask questions and get a range of answers, some useful, some less so.

I’ve never been one to answer loads of questions, but I do try to go back and share solutions when I find them.

The downside is it breaks your flow. One minute you are productive, the next you are stuck searching for answers, sometimes for hours. And if your problem is niche and not already answered, you could be left waiting days for a response.

With AI built into your code editor, you have all that knowledge and reasoning right at your fingertips. No need to leave the comfort of your IDE or go searching the web for that one insight that leads to a “Eureka” moment.

The result of using AI in this way is simple. Higher productivity, less frustration, and faster learning.

A win all round.

Where AI Fell Short (and Needed Steering)

The definition of insanity

To quote Albert Einstein:

Insanity is doing the same thing over and over again and expecting different results.

At times, this is precisely what using AI felt like. It would get stuck on a particular solution and keep coming back to it, no matter how I rephrased the prompt or tried to steer it in a different direction. It was incredibly frustrating.

The only reliable fix was to start a new chat, which meant losing all the project context, or switch to a different model in Cursor. I usually run in “auto" mode to balance cost and capability, but switching models also meant losing that built up understanding within the AI prompt.

There may be a better way to handle this without resetting everything? Again, if you know, please let me know.

Over-engineering simple problems

If there’s one thing you don’t need to ask AI to do, it’s over engineer a solution. I saw this a lot with CSS. Even simple requirements would take longer than expected and result in far more code than needed.

Take something basic like centring a <div> across all screen sizes.

This is a real example from the Benchmark panel in the top right of the FractalAI UI:

AI output

:root {
  --center-x: 50%;
  --center-y: 50%;
  --translate-center: translate(-50%, -50%);
  --container-display: flex;
  --container-align: center;
  --container-justify: center;
}

.container {
  position: relative;
  display: var(--container-display);
  align-items: var(--container-align);
  justify-content: var(--container-justify);
  height: 100vh;
  width: 100%;
}

.container::before {
  content: "";
  position: absolute;
  inset: 0;
  background: radial-gradient(circle, rgba(0,0,0,0.05), transparent);
  pointer-events: none;
}

.centered {
  position: absolute;
  top: var(--center-y);
  left: var(--center-x);
  transform: var(--translate-center);
  display: grid;
  place-items: center;
  padding: calc(1rem + 2px);
  min-width: clamp(200px, 50%, 400px);
  aspect-ratio: 1 / 1;
  backdrop-filter: blur(8px);
  box-shadow:
    0 2px 4px rgba(0,0,0,0.1),
    0 8px 16px rgba(0,0,0,0.15);
}

@supports (height: 100dvh) {
  .container {
    height: 100dvh;
  }
}

@media (min-width: 768px) {
  .centered {
    transform: translate(calc(-50% + 2px), calc(-50% + 2px));
  }
}

Why use one way to centre a <div> when you can use three!

Flexbox (align-items + justify-content)
Absolute positioning (top / left + transform)
Grid (place-items)

Additionally, it threw in CSS variables that were never reused and @supports queries where they were not needed.

I’d expect a developer with a good grasp of modern CSS to write something more like this:

Flexbox (modern)

.container {
  display: flex;
  justify-content: center;
  align-items: center;
  height: 100vh;
}

Grid (modern)

.container {
  display: grid;
  place-items: center;
  height: 100vh;
}

Look how much more maintainable and easier to debug the simpler approach is.

This is definitely something to watch for when reviewing or generating AI code.

Confidently wrong outputs

The tricky thing with AI is how convincing it sounds. Even when it's wrong.

I did not see this much in this project, but I ran into it when I worked for Government Digital Service (GDS). I asked Google Gemini for an overview of the changes from WCAG 2.1 to WCAG 2.2. The response sounded accurate, at least from my perspective as someone who understands the basics of accessibility without being an expert.

But when I shared it with our dedicated accessibility team, it did not hold up. It was full of inaccuracies and false claims.

It is a good reminder not to take AI output at face value, especially on topics you are not deeply familiar with.

Inconsistent outputs between prompts

It’s incredible how much difference a single word in a prompt can make to the output. In this example, I asked the AI to write a test for a hypothetical login function in another project I was working on.

Prompt 1

“Write a test for a login function”

AI Output:

test('login works', () => {
  expect(login('user', 'pass')).toBe(true);
});

Prompt 2

“Write a comprehensive test for a login function”

describe('login', () => {
  it('should succeed with correct credentials', () => {
    expect(login('user', 'pass')).toBe(true);
  });

  it('should fail with incorrect password', () => {
    expect(login('user', 'wrong')).toBe(false);
  });

  it('should fail with empty input', () => {
    expect(login('', '')).toBe(false);
  });
});

Just by adding “comprehensive”, the output is vastly different.

This might seem like a negative, and it is something to be aware of, but it links back to the Crafted Prompts section earlier. Prompt 1 lacks detail, and even Prompt 2 is still pretty vague.

Neither is a great example of how to get the best from AI. It just reinforces the need to be clear and specific in what you are asking for.

One extra word. An entirely different level of quality.

For every project, I work on, web performance is a priority. I see it as the foundation of accessibility, regardless of a user’s device, connection, or location.

During this project, I noticed that unless you explicitly ask for it, performance is not a priority in AI-generated code. In some cases, fractals would not render at all.

As an example, I asked Cursor to find unoptimised code in FractalAI. It highlighted this:

For non line based fractals, ProgressiveRenderer increases iterations on each requestAnimationFrame tick and calls fractalModule.render again to recreate the draw command

This is particularly problematic for the Mandelbrot, and other fractals that are shader based. Using regl, which already calls render() on every requestAnimationFrame, this approach creates a new draw command on every frame. That is extremely inefficient for both CPU and memory.

After prompting, the issue was resolved. Thanks to my visual render tests using Playwright, I could confirm that none of the fractals broke during the optimisation. Another win for visual testing!

This is also something I can automate. As mentioned earlier, Cursor’s Rules feature can enforce performance considerations. Writing a rule for this is now high-up on my to-do list.

Forgetting earlier constraints

This is a surprisingly human like trait of AI. As prompt "conversations" go on, it starts to forget earlier constraints. You often have to repeat yourself or restate key requirements it has quietly ignored.

Over time, parts of the project context seem to drift, much like people forgetting details as time passes.

Going back to the sailing analogy in Crafted Prompts, it is as if the AI loses sight of the waypoint “lighthouses” you set earlier.

Elementary mistakes

This happened more often than I expected. For such an advanced tool, the mistakes were surprisingly basic.

I saw this when asking the AI to update outdated npm packages. A simple prompt like:

Update all npm packages in this project, and run the coverage tests to ensure nothing has broken.

In Cursor, you can see what it is “thinking”, and it frequently searched for package versions from 2023 or 2024. It was not using the latest information.

It was only when I explicitly told it the current year was 2026 did it start checking for up-to-date versions!

This is a good reminder to be explicit when you need current information, especially for dependencies, libraries, and documentation.

Performance Challenges in the Browser

When I first planned this side project, I was quite naïve. My initial idea was to build a 3D fractal renderer in the browser using Three.js. It didn’t take long to realise this wasn’t going to be viable. Even on an M1 MacBook, the frame rate was so low it was effectively unusable.

This is where AI proved its value again. By quickly prototyping the idea, it became obvious in the first iteration that 3D rendering wasn’t going to work. That allowed me to pivot early to a 2D approach using an entirely different WebGL library.

Without that early validation, I could easily have spent far longer trying to optimise something that fundamentally isn’t practical right now. There simply isn’t enough available CPU and GPU power in a typical machine to support real-time 3D fractal rendering in the browser at a usable level.

The key thing to understand with fractals is the sheer volume of maths involved in rendering every single pixel. Each pixel can require hundreds, sometimes thousands, of iterations before you get a final value.

So, even at a relatively modest resolution like a 1810px × 1131px image, that is over 2 million pixels to compute for that single image. Multiply that by hundreds or thousands of calculations per pixel, and you quickly see the scale of the problem.

Without highly optimised code, the browser simply does not stand a chance of keeping up. It becomes less about the idea itself and more about the limits of the hardware running it.

Looking ahead, it is hard not to think about how this changes with more advanced GPU capabilities. If compute continues to accelerate, especially with emerging technologies, calculations at this scale could eventually take milliseconds rather than seconds.

As you zoom deeper into a fractal, the cost per pixel increases. To preserve detail and avoid visual artefacts, the renderer needs to run more iterations for each pixel. Beyond a certain point, the amount of computation required makes real-time interaction impractically slow.

There is also a more fundamental limitation around numerical precision. In the browser, JavaScript numbers use IEEE-754 64-bit double precision, which eventually cannot represent the minuscule coordinate differences needed at very high magnifications.

You can push beyond this using higher precision techniques, such as BigInt-based arithmetic or “double-double” approaches, but that additional precision comes at a significant cost. Each pixel becomes even more expensive to compute, which further impacts performance.

What this all really comes down to is simple. Rendering fractals efficiently in the browser is hard.

That being said, it is one of the quickest ways to max out your CPU and GPU, turn your computer into a heater, and simultaneously watch your electricity bill shoot up!

What Building This Changed About My View of AI

When I started this project, I was sceptical about how useful AI could be for coding. I had used tools like ChatGPT and Google Gemini for research, but never properly explored the coding side.

The term “Vibe coding” didn’t help either. If anything, it put me off. After reading Vibe code is legacy code, the last thing I wanted was to introduce more legacy code into a project. I also find the term itself a bit meaningless, especially after it was named word of the year in 2025. It felt like just another buzzword that would disappear as quickly as it arrived.

We have seen this pattern before. Technologies get hyped as “the future” and then quietly fade away. Anyone remember Google Glass from 2013??

Looking at the defintion of "vibe coding":

Vibe coding is an intuitive, fast paced way of writing code where you prioritise momentum and experimentation over upfront planning, often with the help of tools

I’m pleased to say this project has thoroughly changed my perspective on AI in software development. I still dislike the term “vibe coding”, but the practice behind it is undoubtedly what I found most useful and genuinely exciting while building FractalAI.

As mentioned earlier, if you put strong guard rails in place such as mandatory human reviews for all production PRs, along with solid performance and security practices, AI becomes a genuinely valuable tool for learning and rapid prototyping.

That said, I openly admit this is an idealistic view. In reality, there will always be individuals, companies, and bad actors willing to use AI in ways that harm the web for their own gain.

When it comes to new and exciting technology on the web, it often follows the same familiar pattern. A few people find ways to exploit it for their own benefit, usually at the expense of everyone else.

As Paula Poundstone put it best:

This is why we can’t have nice things.

The Future of AI-Generated Software

If there’s one thing I know about AI-generated software, it’s that it’s not going anywhere. The AI sector is projected to reach $539 billion in 2026, up from $391 billion in 2025. That’s a 38% increase in a single year. Longer term, it’s expected to push into the multi-trillion dollar range.

Spending alone tells the story. Global AI investment hit around $1.5 trillion in 2025, and it’s still accelerating. This isn’t hype, it’s momentum.

At the same time, the risks are evolving just as quickly. A new Claude model, Mythos, from Anthropic has reportedly identified a significant number of vulnerabilities across major operating systems. Some experts believe it shows an unprecedented ability to detect and potentially exploit security weaknesses. Whether that is true or is just scaremongering remains to be seen, but it’s telling that major financial institutions have already been given early access to the model ahead of any public release.

As I’ve said, AI is here to stay. The genie is well and truly out of the bottle.

My plan is simple. Learn as much as possible over the coming months and years. This is happening whether we like it or not, so it makes far more sense to understand its potential and how it’s being used than to ignore it.

If you don’t, you can be sure others in your respective field will be. As this goes far beyond software development. AI’s reach is expanding fast, so buckle up.

Summary

This project was an experiment in what happens when you step back from writing code and let AI take the lead. FractalAI is a fully AI-generated fractal renderer running in the browser, built to explore both the potential and the limits of modern AI-assisted development.

Along the way, it became clear that AI is far more than just a code generator. When used properly, it acts as a design partner, a rapid prototyping tool, and a way to accelerate learning. But it's not perfect. It needs direction, context, and clear constraints to produce reliable results.

The browser turned out to be a surprisingly powerful platform for this kind of work, capable of rendering complex, infinite visuals while even supporting lightweight machine learning. Combined with AI, it creates a feedback loop where ideas can be tested, refined, and understood far faster than traditional approaches.

The biggest takeaway is simple. AI is not replacing developers, it is changing how we work. If you use it well, it becomes a tool for thinking, learning, and making better decisions, not just writing code faster.

Well, there we go, yet another blog post where I’ve somehow written far more than intended! If you’ve made it this far, I’m genuinely impressed. Please accept this virtual gold star as a reward: ⭐

As always, feedback and comments are very welcome. You can contact me here.

FractalAI project

Below, you can find the links to the repository and the live site:

I’m completely open and actively encourage raising PR's and sending me feedback / issues about the project (with, or without AI!).

Post changelog:

21/04/26: Initial post published.

Using Cloudflare Workers and reCAPTCHA v3 for a Static Site Contact Form

2026-03-09T00:00:00Z

Introduction

I recently wrote a blog post that involved using a Cloudflare Worker to serve Brotli 11 compressed HTML rather than the standard uncompressed HTML file. Off the back of this post, I’ve decided to write one about my Cloudflare Worker setup that I use for my Contact Page. In order to stop the spam, I also integrated reCAPTCHA v3 into the form functionality. The form is pretty simple, it asks a visitor for their full name, email address and the message they with to send to me. The Worker then pulls together this information and sends an email to a custom email address that points to my personal email address. This allows me to easily filter messages that come from the site, which I then have the option to reply to if I so wish.

It’s worth noting that this post was heavily inspired by Sia Karamalegos’s post from 2024 about migrating from Netlify to Cloudflare. I had the absolute please of meeting Sia in Amsterdam when I spoke at Performance.now() 2023, she was MC for my talk and I really appreciated her support when I was (quite frankly) freaking out with nerves before the talk!

So why use a serverless endpoint for the contact form? Well, as I’ve mentioned many times before, this blog is a static site built using 11ty and hosted on the Pro plan of Cloudflare Pages. There simply is no backend to point the POST method too, which is required to send the email.

Previous solutions I’ve used

This isn’t the first time I’ve changed the way the contact form works on this blog. I’ve counted 5 alternatives that I have used over the years! Isn’t Git great for tracking history! I certainly wouldn’t be able to remember them all with my terrible memory! So, if you aren’t on Cloudflare, or you would rather not use a Worker for this, then there’s a list of perfectly viable alternatives below:

Netlify Forms

URL: Netlify Forms
Cost: Free+ (depending on the Netlify tier you choose)

Netlify Forms is a built-in form handling service for sites hosted on Netlify. It allows you to collect and manage form submissions from static websites without building or maintaining a backend. Submissions can be viewed in the Netlify dashboard or forwarded via email or webhook. I still think it’s a real shame that Cloudflare doesn't offer a form setup that is as easy to integrate and use as Netlify Forms, although, if it did, I wouldn’t need to be writing this blog post!

Cloudflare Pages Functions + MailChannels

URL: https://developers.cloudflare.com/pages/functions/ & https://www.mailchannels.com/
Cost: Free plan available ($10+ depending on volume-based pricing)

When I initially migrated from Netlify to Cloudflare for hosting I used this setup, that was until MailChannels sunset its free email sending service for Cloudflare Workers users (me), at that point I started looking for other solutions. As you will see below.

Formspree.io

URL: https://formspree.io/
Cost: Free tier available ($15+ a month for paid plans)

Formspree is a hosted form service that lets you collect form submissions without building your own backend server. You simply point your form at Formspree, and it handles sending submissions to your email address or other API endpoints, it comes with built-in spam protection and integrations for various workflows.

Formspark.io (previously Submit Form)

URL: https://formspark.io/
Cost: Free tier available ($9+ a month for paid plans)

Formspark is a basic hosted form backend that lets you accept and manage form submissions without the need for your own backend server. You simply submit forms to Formspark, and it stores (or forwards) the data, with options for email notifications, webhooks, and spam filtering.

Botpoison

URL: https://botpoison.com/
Cost: Free plan available ($4+ per month depending on bot verifications required)

Botpoison is an invisible anti-spam and bot prevention service for web forms that blocks automated submissions without requiring CAPTCHAs or extra steps from users. It works by analysing form interactions and applying proof-of-work challenges and reputation checks to distinguish human traffic from bots. I started using it when I was still receiving contact form spam from my Formspark.io setup.

Resend

URL: https://resend.com/
Cost: Free plan available ($20+ for paid plans)

Resend is a developer-focused email delivery service, providing a simple API and Simple Mail Transfer Protocol (SMTP) interface for sending transactional and broadcast emails from your applications or website. It handles features like deliverability, bounce tracking, suppression lists, and email analytics, so a developer doesn't need to manage email infrastructure themselves and can be integrated with many platforms. Resend is currently part of my email workflow, as I will describe later in the post.

Overview of the setup

My current setup is: Cloudflare Pages Functions (the Worker) + Google reCAPTCHA v3 + Resend for email.

A high-level view of this workflow is:

User populates the contact form with their details and message
reCAPTCHA v3 runs and adds a short-lived verification token
The form POSTs to /api/contact (via the Pages Function, that sits in functions/api/contact.js in the root of my repository
The Pages Function, then:
- verifies the token with Google
- validates the user inputs
- sends the validated inputs via email using Resend
Redirects the user to the “Thank You” page, confirming that the message has been sent.

Google reCAPTCHA v3

I have to say, out of all the things written in this post, researching about how exactly Google reCAPTCHA v3 works, was probably the most fascinating! I really love the fact that it is all invisible to the user, and it is based on a complex (but logical) scoring mechanism. There’s no need for a really frustrating visual CAPTCHA’s that don’t prove you are human, “they prove that you are American!” (I can’t take credit for this quote, that I believe goes to Terence Eden, or at least that's who I remember reading it from!)

So how exactly does reCAPTCHA v3 decide if you are a human? Well, it turns out it analyses a user's behavioural and contextual signals rather than setting a user a visual challenge, and based on those signals it creates a risk score. The biggest signal comes from how the user behaves on the page.

Behavioural analysis of a user's session

reCAPTCHA observes a number of patterns and metrics from a user's interaction with the page, these include:

Mouse movement patterns
Typing speed and rhythm
Scroll behaviour
Click timing and placement
Focus changes between fields or tabs

Us humans are a little messy when using a computer. Our mouse movements curve and wander slightly, we pause without thinking, and our typing comes with the occasional hesitation or uneven rhythm.

Skynet, on the other hand, tends to be a little too perfect. Its mouse paths are often straight, timing is predictable, and its interactions can happen far faster than any real person would manage. Those overly neat patterns are often a strong indicator that automation is at work.

Browser and device fingerprinting

Now, when I think of Browser fingerprinting I often associate it with advertising and companies tracking users across the internet. In all fairness, this is Google we are talking about, so it's very likely that they actually do this! :sad_face:. Browser and device fingerprinting is the way the reCAPTCHA script also examines the environment and device information of where the request is coming from.

Typical signals include:

Browser version and capabilities
Installed fonts and plugins
Screen resolution and device properties
WebGL and Canvas fingerprint signals
Cookie availability
JavaScript execution behaviour
Timezone of client
Language settings

You’d be amazed at how accurate these fingerprints can be. Each piece of information adds a little more entropy, meaning it becomes easier to distinguish one device from another. They are so accurate in fact I know many financial sectors or government departments use them for fraud detection.

You really don’t need to look very far to find libraries like Fingerprint that do this all for you! Look at all the information it captured about you just by clicking this link! And you thought cookie tracking was bad!

Note: There are many other fingerprinting libraries available, I’m not specifically targeting Fingerprint.

So while I disagree with browser and device fingerprinting from a privacy and tracking perspective, it does have its uses when it comes to proving that a “user” is a human and not a bot. In particular, bots often run in headless browsers or minimal environments, which look very different from real user devices.

Network and reputation signals

Due to Google’s global network, it can also delve into global reputation data. It captures:

IP reputation and history
Known bot networks or proxy usage
Data centre IP ranges
Previous abuse patterns linked to the IP or session

So much traffic flows through Googles network globally, it can detect suspicious network traffic surprisingly easily.

Action Context

Google can also score the predicted actions a user is going to make. With reCAPTCHA v3, developers call the library with an action name such as:

login
checkout
signup
comment

From its countless trillions of observations over the years, Google already knows the typical login behaviour for a human (it takes approximately 3–10 seconds). Whereas, a bot can submit hundreds of logins per minute, which is a huge red flag for the use of automation.

Scoring using Machine learning

So what does Google do with all these various signals it captures?

It feeds them into a machine learning algorithm for scoring, which will decide if whatever is interacting is a human or a bot.

Score	Meaning
0.9 to 1.0	Very likely human
0.5	Uncertain
0.0 to 0.3	Very likely bot

At this point, it is then up to your backend to decide what to do with the score it gets back. For example:

0.8+  allow request
0.4–0.8 require MFA
<0.4 block or challenge

In my case, the Cloudflare Worker is my simple serverless backend, and it contains the following code to decide if the form submit is from a human or a bot:

// Verify score (reCAPTCHA v3 returns scores between 0.0 and 1.0)
// Lower scores indicate bot-like behavior. 0.5 is a common threshold, but 0.3 allows more legitimate traffic
if (typeof verifyJson.score === "number" && verifyJson.score < 0.3) {
	return new Response(
		JSON.stringify({
			error: "CAPTCHA verification failed",
			details: `Score too low: ${verifyJson.score}`,
		}),
		{ status: 400, headers: { "content-type": "application/json" } },
	);
}

The code above is saying to reject any score that returns from Google that is under 0.3 (Very likely bot), You could of course easily change this threshold if you found it was being too aggressive with the filtering.

Technical setup

To use reCAPTCHA v3, you must first register your site in the Google reCAPTCHA admin console to obtain two keys.

The “site key” is used on the client-side and allows the browser to request a reCAPTCHA verification token for when a user performs an action. The “secret key” must remain server-side and is used to verify that token with Google.

This means storing the secret securely as an environment variable, for my setup this is done via the Cloudflare Pages build dashboard. The Cloudflare Worker then reads it from the environment variables during the build process.

Once all configured, you can also use the Google reCAPTCHA admin console to:

restrict the key to specific hostnames
configure score thresholds to control how strict the bot detection should be

The contact form (HTML and client-side)

The HTML used for the form is all pretty standard, just remember to make it as accessible as possible! Semantic markup is the key to achieving this! If you are unsure how to do this, there’s an excellent post called Creating Accessible Forms on the WebAIM.org website. The HTML for my contact form looks like this:

<!--
  Contact form: POSTs to the Cloudflare Pages Function at /api/contact.
  Uses novalidate so we can run custom client-side validation and show
  accessible error messages before and after submit.
-->
<form id="fs-frm" action="/api/contact" method="POST" aria-labelledby="page-title" novalidate>
    <!--
	  Error summary: shown when validation fails (client or server).
	  role="alert" announces to screen readers when content appears.
	  tabindex="-1" allows focus to be moved here after submit for accessibility.
	  hidden by default; JS removes the attribute to reveal and populate it.
	-->
    <div id="form-error-summary" class="form-error-summary" role="alert" tabindex="-1" hidden></div>
    <!--
	  Live region for status updates (e.g. "Sending…", success message).
	  role="status" is polite; screen readers announce changes without interrupting.
	-->
    <div id="form-status" class="form-status" role="status"></div>
    <!--
	  Honeypot (spam trap): invisible to users, attractive to bots.
	  - Wrapped in a div with a neutral class (e.g. "abc") so bots don’t skip it.
	  - aria-hidden="true" keeps it out of the accessibility tree.
	  - tabindex="-1" and autocomplete="off" reduce chance a human touches it.
	  - Obscure name so bots don’t recognise it as a honeypot; server rejects if set.
	-->
    <div class="abc" aria-hidden="true">
        <input type="checkbox" name="_its-a-trap!" tabindex="-1" autocomplete="off" />
    </div>
    <!-- Server/worker uses these hidden fields for redirect and email behaviour -->
    <input type="hidden" name="_redirect" value="/contact/thanks/" />
    <input type="hidden" name="_append" value="false" />
    <input type="hidden" name="_email.subject" value="New Message from Nooshu.com" />
    <fieldset>
        <!-- Legend is required for fieldset; visually-hidden keeps layout clean -->
        <legend class="visually-hidden">Contact form</legend>
        <div class="form-group">
            <label for="name" class="required">Full Name:</label>
            <input type="text" id="name" name="name" autocomplete="name" required />
            <span id="name-error" class="form-error" hidden></span>
        </div>
        <div class="form-group">
            <label for="email" class="required">Email:</label>
            <input type="email" id="email" name="email" autocomplete="email" placeholder="hello@example.com" required />
            <span id="email-error" class="form-error" hidden></span>
        </div>
        <div class="form-group">
            <label for="message" class="required">Message:</label>
            <textarea rows="7" id="message" name="message" required></textarea>
            <span id="message-error" class="form-error" hidden></span>
        </div>
    </fieldset>
    <!--
	  reCAPTCHA v3 token: client-side JS runs grecaptcha.execute() and sets
	  this value before submit. Server verifies the token with Google before
	  sending the email.
	-->
    <input type="hidden" name="g-recaptcha-response" value="" />
    <button type="submit">Send Message</button>
</form>

A version with fewer comments is available in this gist on GitHub (opens in new tab).

The contact form I use above has the following features:

Honeypot and reCAPTCHA to block spam

I’ve already detailed how Google reCAPTCHA works, so I won’t cover it again here, but a Honeypot is a simple first layer of protection. The basic theory behind a Honeypot is a simple form field hidden from real users but still present in the page markup. Bots that scan the source code often fill in every field they find, including this hidden one. If that field is completed, it is a strong signal that the submission came from a bot, not a human because it isn’t seen in the User Interface (UI), thus allowing the request to be rejected immediately.

Client and server validation to ensure safe input

Form validation is handled in two layers:

On the client side, the form uses novalidate with custom JavaScript to check required fields and validate the email format for the name, email, and message inputs.
On the server-side, the API performs its own validation by checking that required fields are present, enforcing max length limits for each field (name: 200 characters, email: 320 characters, message: 5000 characters), and confirming that a valid reCAPTCHA token is included with the submission.

Accessible error handling and focus management

Accessibility is built into the form by clearly associating it with the page title using aria-labelledby, providing an error summary and status regions with role=“alert" plus role=“status", also marking individual fields with aria-invalid and aria-describedby so errors are properly announced.

Inline error messages use matching IDs for assistive technologies, required fields are clearly indicated with a *, and a visually hidden legend describes the <fieldset>. When errors occur, the browser focus moves to the summary using tabindex="-1” so it can be announced immediately by the screen reader.

UX updates to keep the user informed during submission

The form includes the autocomplete attribute for the name and email fields, allowing users browser the option to suggest information already stored in the browser and pre-fill it for them. Additionally, there's a simple placeholder for the email input, letting a user know what input format is expected.

When the form is submitted, the submit button is temporarily disabled and its label changes to “Sending…” while a status message informs the user that their message is being sent. After the submission is successful, the user is then redirected to the /contact/thanks/ page, as seen in the _redirect input field, the API then returns a 303 redirect to complete the process.

Secure server side processing

The form submits a POST request to /api/contact, handled by a Cloudflare function. Hidden fields such as _redirect, _append, and _email.subject control what happens on submit.

On the server, the submission is sent as an email in both HTML and plain text formats before redirecting the user on success. The endpoint only accepts form-encoded or multipart content types. It also verifies the reCAPTCHA token using the CF-Connecting-IP header when available, and validates required fields and length limits, it then escapes HTML in the email body to ensure the content is safe.

I’d like to think this form is pretty accessible and simple to use, but if you disagree, please do let me know. How very meta! It’s always fun to learn something new!

Simplified CSP

The great thing about not having to use a third-party form service for sending contact form emails (besides the slight reduction in cost), is that it simplifies my Content-Security-Policy. This is because I’m no longer having to establish a connection to an additional third-party. The Cloudflare Worker runs on my instance of Cloudflare pages, so it is embedded in the build. And thankfully there’s no client-side interaction when using Resend, as this all happens via the Cloudflare Worker, no modifications to the CSP required! This may not seem like a big thing, but the less you can rely on a third parties the better! There are numerous examples from the history of the web, where the failure of a third party has had a massive impact on the web:

An interesting point about the Wikipedia link above is you can see how prominent these outages have become over the decades, and also the fact that the placeholder category pages for the 2030s, 2040s, 2050s, and 2060s are already listed! Oh well, those outages are for my grandkids to worry about! 👴🏼

The Cloudflare Worker (Pages Function)

So let’s get onto the final piece of the puzzle: the Cloudflare Worker code. This file sits in the root of my blog under functions/api/contact.js, this translates to the action="/api/contact” in the form code above.

// Import a small helper that escapes HTML characters.
// This prevents user-supplied content (like the message body)
// from breaking HTML or injecting markup when we render it in the email.
import { escapeHtml } from "../../_helpers/escape-html.js";

// Cloudflare Pages Functions export HTTP handlers as named exports.
// `onRequestPost` will be invoked for POST requests to this function's route.
//
// Signature:
// - `request`: the incoming HTTP Request object
// - `env`: environment bindings (secrets, KV, etc.), configured in Cloudflare
export const onRequestPost = async ({ request, env }) => {
	try {
		// 1. Accept only standard HTML form submissions
		// ------------------------------------------------
		// We support:
		// - application/x-www-form-urlencoded  (classic HTML <form method="post">)
		// - multipart/form-data                (forms that include files)
		//
		// Anything else (JSON, etc.) is rejected with 415 Unsupported Media Type.
		const contentType = request.headers.get("content-type") || "";
		if (
			!contentType.includes("application/x-www-form-urlencoded") &&
			!contentType.includes("multipart/form-data")
		) {
			return new Response(
				JSON.stringify({ error: "Unsupported content type" }),
				{
					status: 415,
					headers: { "content-type": "application/json" },
				},
			);
		}

		// Parse the incoming form body. Cloudflare's Request object
		// exposes a `.formData()` helper which returns a FormData instance.
		const form = await request.formData();

		// 2. Honeypot anti-spam field
		// ------------------------------------------------
		// A "honeypot" is an extra hidden field that humans never fill in,
		// but bots often do. If this field is present and non-empty,
		// we treat it as spam and quietly redirect away.
		if (form.get("_its-a-trap!")) {
			// Redirect back to the homepage. We build the URL from the current request
			// so it works correctly across environments (preview, production, etc.).
			const hpUrl = new URL("/", request.url);
			return Response.redirect(hpUrl.toString(), 303);
		}

		// 3. Extract and normalize form fields
		// ------------------------------------------------
		// We pull out the core fields we care about:
		// - name
		// - email
		// - message
		// - _redirect (optional; where to send the user after success)
		//
		// Every value is coerced to string and trimmed to avoid
		// issues with null/undefined and accidental leading/trailing spaces.
		const name = (form.get("name") || "").toString().trim();
		const email = (form.get("email") || "").toString().trim();
		const message = (form.get("message") || "").toString().trim();
		const redirectUrl = (
			form.get("_redirect") || "/contact/thanks/"
		).toString();

		// Basic required-field validation. If any are empty,
		// respond with a 400 Bad Request and a JSON error.
		if (!name || !email || !message) {
			return new Response(
				JSON.stringify({ error: "Missing required fields" }),
				{
					status: 400,
					headers: { "content-type": "application/json" },
				},
			);
		}

		// 4. Length limits (simple validation / abuse mitigation)
		// ------------------------------------------------
		// These caps are deliberately generous but help avoid:
		// - unrealistic payloads
		// - abuse where the form is used as a data pipe
		if (name.length > 200 || email.length > 320 || message.length > 5000) {
			return new Response(JSON.stringify({ error: "Invalid field lengths" }), {
				status: 400,
				headers: { "content-type": "application/json" },
			});
		}

		// 5. CAPTCHA verification (reCAPTCHA v3 or hCaptcha-compatible)
		// ------------------------------------------------
		// The form is expected to include a CAPTCHA token:
		// - reCAPTCHA v3: `g-recaptcha-response`
		// - (optionally) hCaptcha-style: `h-captcha-response`
		//
		// If no token is present, we immediately reject the submission.
		const recaptchaToken =
			form.get("g-recaptcha-response") || form.get("h-captcha-response");
		if (!recaptchaToken) {
			return new Response(
				JSON.stringify({ error: "reCAPTCHA token is required" }),
				{
					status: 400,
					headers: { "content-type": "application/json" },
				},
			);
		}

		// Select the secret key from Cloudflare environment variables.
		// This must be configured in the project/environment:
		// - RECAPTCHA_SECRET_KEY or
		// - RECAPTCHA_SECRET
		const secret = env.RECAPTCHA_SECRET_KEY || env.RECAPTCHA_SECRET;
		if (!secret) {
			// If the secret is missing, that's a server misconfiguration,
			// so we return a 500 Internal Server Error.
			return new Response(
				JSON.stringify({
					error:
						"Server not configured for CAPTCHA (missing RECAPTCHA_SECRET_KEY/RECAPTCHA_SECRET)",
				}),
				{ status: 500, headers: { "content-type": "application/json" } },
			);
		}

		// For better fraud detection, we also send the user's IP to reCAPTCHA.
		// Cloudflare exposes client IP via:
		// - CF-Connecting-IP header
		// - request.cf.clientAddress
		const remoteIp =
			request.headers.get("CF-Connecting-IP") ||
			request.cf?.clientAddress ||
			null;

		// Build the request body for the reCAPTCHA verification endpoint.
		// It expects application/x-www-form-urlencoded payload.
		const verifyParams = new URLSearchParams({
			secret,
			response: recaptchaToken.toString(),
		});
		if (remoteIp) {
			verifyParams.append("remoteip", remoteIp);
		}

		// POST the verification request to Google's reCAPTCHA API.
		const verifyResp = await fetch(
			"https://www.google.com/recaptcha/api/siteverify",
			{
				method: "POST",
				headers: { "content-type": "application/x-www-form-urlencoded" },
				body: verifyParams,
			},
		);

		// If Google's endpoint itself is failing (network issue or 5xx),
		// treat this as a bad gateway (502) and surface a generic error.
		if (!verifyResp.ok) {
			return new Response(
				JSON.stringify({ error: "Failed to verify reCAPTCHA" }),
				{
					status: 502,
					headers: { "content-type": "application/json" },
				},
			);
		}

		// Parse the JSON response from reCAPTCHA.
		// Example structure:
		// {
		//   success: true/false,
		//   score: 0.0-1.0,
		//   hostname: "example.com",
		//   "error-codes": [...]
		// }
		const verifyJson = await verifyResp.json();

		// 5a. Hard failure: verification not successful at all
		// ------------------------------------------------
		// We return a 400 with details about why it failed where possible.
		if (!verifyJson.success) {
			const errorCodes = verifyJson["error-codes"] || [];
			return new Response(
				JSON.stringify({
					error: "CAPTCHA verification failed",
					details:
						errorCodes.length > 0 ? errorCodes.join(", ") : "Unknown error",
				}),
				{ status: 400, headers: { "content-type": "application/json" } },
			);
		}

		// 5b. Soft failure: low reCAPTCHA v3 score
		// ------------------------------------------------
		// reCAPTCHA v3 uses scores instead of explicit "I'm not a robot" prompts.
		// Lower values are more suspicious. Here we reject anything below 0.3.
		// This threshold is a trade-off between blocking bots and not annoying users.
		if (typeof verifyJson.score === "number" && verifyJson.score < 0.3) {
			return new Response(
				JSON.stringify({
					error: "CAPTCHA verification failed",
					details: `Score too low: ${verifyJson.score}`,
				}),
				{ status: 400, headers: { "content-type": "application/json" } },
			);
		}

		// 5c. Optional hostname verification
		// ------------------------------------------------
		// The reCAPTCHA response includes a `hostname` field.
		// We check that the hostname reCAPTCHA saw matches the hostname of our request.
		// This helps prevent token reuse on other domains.
		const requestHost = new URL(request.url).hostname;
		if (
			verifyJson.hostname &&
			verifyJson.hostname !== requestHost &&
			!requestHost.endsWith(`.${verifyJson.hostname}`)
		) {
			// We allow subdomains (e.g. www.example.com vs example.com),
			// but log any unexpected mismatch to server logs for later inspection.
			// This log will show up in Cloudflare function logs.
			console.warn(
				`reCAPTCHA hostname mismatch: expected ${requestHost}, got ${verifyJson.hostname}`,
			);
		}

		// 6. Prepare email content for Resend
		// ------------------------------------------------
		// At this point, the submission has passed:
		// - basic validation
		// - anti-spam honeypot
		// - CAPTCHA checks
		//
		// Now we build an email to send to the site owner via Resend.

		// Fixed "from" address used with Resend.
		// This should be a verified sender domain for your Resend account.
		const fromEmail = "contact@example.com";

		// Allow the subject to be overridden via a hidden form field `_email.subject`,
		// otherwise fall back to a sensible default.
		const subject =
			form.get("_email.subject")?.toString() || "New Message from Nooshu.com";

		// Plain-text body, which is useful for mail clients that don't render HTML, note the use of JavaScript Template literals.
		const textBody = `New contact form submission on nooshu.com

Name: ${name}
Email: ${email}

Message:
${message}`;

		// HTML body with simple markup for better readability.
		// We escape all user-supplied fields to avoid injecting HTML or scripts.
		const htmlBody = `<h2>New contact form submission on nooshu.com</h2><p><strong>Name:</strong> ${escapeHtml(
			name,
		)}<br/><strong>Email:</strong> ${escapeHtml(email)}</p><p><strong>Message:</strong></p><pre style="white-space:pre-wrap">${escapeHtml(
			message,
		)}</pre>`;

		// Pull the Resend API key from Cloudflare environment variables.
		// This must be configured as a secret in the Pages project.
		const apiKey = env.RESEND_API_KEY;
		if (!apiKey) {
			return new Response(
				JSON.stringify({
					error: "Server not configured for email (missing RESEND_API_KEY)",
				}),
				{ status: 500, headers: { "content-type": "application/json" } },
			);
		}

		// 7. Call Resend's email API
		// ------------------------------------------------
		// We issue a POST request to Resend's /emails endpoint with:
		// - from: sender (must be a verified domain)
		// - to: recipient(s) (in this case, the site inbox)
		// - reply_to: the visitor's email address, so "Reply" in your mail client goes to them
		// - subject, text, html: the message content
		const resendResp = await fetch("https://api.resend.com/emails", {
			method: "POST",
			headers: {
				"content-type": "application/json",
				Authorization: `Bearer ${apiKey}`,
			},
			body: JSON.stringify({
				from: `Nooshu Contact <${fromEmail}>`,
				to: ["email@example.com"],
				reply_to: email,
				subject,
				text: textBody,
				html: htmlBody,
			}),
		});

		// If Resend returns a non-2xx status, we surface a 502 to the client
		// and include the raw error body for easier debugging.
		if (!resendResp.ok) {
			const text = await resendResp.text();
			return new Response(
				JSON.stringify({ error: "Email send failed", details: text }),
				{
					status: 502,
					headers: { "content-type": "application/json" },
				},
			);
		}

		// 8. Final redirect on success
		// ------------------------------------------------
		// If everything above succeeds, we redirect the user to a "thank you"
		// page (configurable via the `_redirect` field).
		//
		// We resolve `redirectUrl` relative to the current request URL
		// to avoid hard-coding the origin.
		const finalRedirect = new URL(redirectUrl, request.url);

		// 303 See Other is the canonical way to redirect after a POST,
		// telling the browser to perform a GET to the new URL.
		return Response.redirect(finalRedirect.toString(), 303);
	} catch (err) {
		// 9. Global error handler
		// ------------------------------------------------
		// If anything unexpected blows up (network issues, runtime errors, etc.),
		// we catch it here and return a generic 500 JSON response.
		//
		// The error is stringified so it's at least inspectable in logs / responses,
		// but in a real-world scenario you might want to avoid leaking details
		// to the client and instead only log them server-side.
		return new Response(
			JSON.stringify({ error: "Server error", details: String(err) }),
			{
				status: 500,
				headers: { "content-type": "application/json" },
			},
		);
	}
};

A version with fewer comments is available in this gist on GitHub (opens in new tab).

Using Cloudflare Turnstile

After I posted the blog post on Bluesky, Mastodon, and LinkedIn, I had a couple of replies back asking me why I hadn't used Cloudflare's CAPTCHA integration rather than Google's ReCAPTCHA v3. In all honesty I didn't realise Cloudflare had a competing product! So, thanks to Ryan Townsend and Paweł Grzybek for letting me know!

I'm not going to be rewriting the whole blog post, but what I will do is share the client-side JavaScript code (turnstile.js), and the contact.js file for the API that I am using to integrate with Cloudflare Turnstile. Also, any additional changes I had to make.

First the Code:

// Lightweight, dependency-free integration for Cloudflare Turnstile.
// This script:
// - Validates the contact form on the client (for UX only – server still re-validates).
// - Requests an invisible Turnstile token just before submit.
// - Injects the token into a hidden field for the server to verify.
// - Tries hard not to block the user if Turnstile is slow to load.
(function () {
  // Public Turnstile site key for this widget. This is *not* a secret.
  // The matching secret key lives on the server and is used during
  // server-side verification in the Cloudflare Pages Function.
  const SITE_KEY = "0x4AAAAAACpYvR2v9J0i1tr_";

  // The contact form we progressively enhance. If it does not exist
  // (for example, on other pages), bail out early.
  const form = document.getElementById("fs-frm");
  if (!form) return;

  // ---------------------------------------------------------------------------
  // Hidden token field management
  // ---------------------------------------------------------------------------
  // Turnstile posts its token back to the server in a field that we choose.
  // The backend expects this exact name when validating the submission.
  const tokenInputName = "cf-turnstile-response";
  let tokenInput = form.querySelector(`input[name="${tokenInputName}"]`);
  if (!tokenInput) {
    // If the hidden field was not rendered server-side, create it here.
    tokenInput = document.createElement("input");
    tokenInput.type = "hidden";
    tokenInput.name = tokenInputName;
    form.appendChild(tokenInput);
  }

  // ---------------------------------------------------------------------------
  // Accessible error + status UI helpers
  // ---------------------------------------------------------------------------
  const errorSummary = document.getElementById("form-error-summary");
  const formStatus = document.getElementById("form-status");

  // Show a high-level error message above the form and move focus there so
  // screen-reader users (and keyboard users) are notified.
  function showFormError(message) {
    if (errorSummary) {
      errorSummary.textContent = message;
      errorSummary.hidden = false;
      errorSummary.focus();
    }
  }

  // Hide and clear the error summary.
  function clearFormError() {
    if (errorSummary) {
      errorSummary.textContent = "";
      errorSummary.hidden = true;
    }
  }

  // Attach error state to a single field (ARIA attributes + inline message).
  function showFieldError(fieldId, message) {
    const field = document.getElementById(fieldId);
    const errorEl = document.getElementById(fieldId + "-error");
    if (field) {
      field.setAttribute("aria-invalid", "true");
      field.setAttribute("aria-describedby", fieldId + "-error");
    }
    if (errorEl) {
      errorEl.textContent = message;
      errorEl.hidden = false;
    }
  }

  // Clear error state from a single field.
  function clearFieldError(fieldId) {
    const field = document.getElementById(fieldId);
    const errorEl = document.getElementById(fieldId + "-error");
    if (field) {
      field.removeAttribute("aria-invalid");
      field.removeAttribute("aria-describedby");
    }
    if (errorEl) {
      errorEl.textContent = "";
      errorEl.hidden = true;
    }
  }

  // Clear all per-field and summary errors.
  function clearAllErrors() {
    clearFormError();
    ["name", "email", "message"].forEach(clearFieldError);
  }

  // Set a small, live-updating status message (polite ARIA live region).
  function setStatus(message) {
    if (formStatus) {
      formStatus.textContent = message;
    }
  }

  // ---------------------------------------------------------------------------
  // Client-side validation
  // ---------------------------------------------------------------------------
  // This mirrors the server’s validation rules, but is *only* for UX.
  // The server still validates everything again.
  function validateForm() {
    clearAllErrors();
    const errors = [];
    const nameField = form.querySelector("#name");
    const emailField = form.querySelector("#email");
    const messageField = form.querySelector("#message");

    if (!nameField.value.trim()) {
      showFieldError("name", "Please enter your full name.");
      errors.push(nameField);
    }

    if (!emailField.value.trim()) {
      showFieldError("email", "Please enter your email address.");
      errors.push(emailField);
    } else if (!/^[^\s@]+@[^\s@]+\.[^\s@]+$/.test(emailField.value.trim())) {
      showFieldError("email", "Please enter a valid email address.");
      errors.push(emailField);
    }

    if (!messageField.value.trim()) {
      showFieldError("message", "Please enter a message.");
      errors.push(messageField);
    }

    if (errors.length > 0) {
      const summary =
        errors.length === 1
          ? "There is 1 error in the form."
          : "There are " + errors.length + " errors in the form.";
      showFormError(summary);
      errors[0].focus();
      return false;
    }

    return true;
  }

  // ---------------------------------------------------------------------------
  // Turnstile widget lifecycle
  // ---------------------------------------------------------------------------
  // `widgetId` is the handle returned by turnstile.render.
  // `pendingSubmit` tracks whether we should submit once a token arrives.
  // `submitButton` lets us disable / re-enable the call-to-action button.
  let widgetId = null;
  let pendingSubmit = false;
  let submitButton = null;

  // Called by Turnstile when a token is successfully generated.
  function onTokenReceived(token) {
    tokenInput.value = token;
    // If the user was waiting for the challenge to finish, we can now submit.
    if (pendingSubmit) {
      pendingSubmit = false;
      form.submit();
    }
  }

  // Called when Turnstile hits an error or we decide the widget is unavailable.
  function onTokenError(errorCode) {
    tokenInput.value = "";
    pendingSubmit = false;
    setStatus("");
    console.error("Turnstile error:", errorCode);
    showFormError(
      "Verification failed. Please try again. If you use an ad blocker, you may need to allow Cloudflare on this site."
    );
    if (submitButton) {
      submitButton.disabled = false;
      submitButton.textContent = "Send Message";
    }
    // If we have a widget instance, reset it so the user can try again.
    if (widgetId !== null && typeof turnstile !== "undefined") {
      turnstile.reset(widgetId);
    }
  }

  // Render an invisible Turnstile widget into the placeholder div.
  // The widget itself is not visible – it will run when we call execute().
  function renderTurnstile() {
    const container = document.getElementById("turnstile-container");
    // Only render once, and only on the contact page where the container exists.
    if (!container || widgetId !== null) return;

    if (typeof turnstile !== "undefined") {
      widgetId = turnstile.render(container, {
        sitekey: SITE_KEY,
        // Invisible mode: no explicit checkbox; Cloudflare decides when
        // to prompt for a challenge, if at all.
        size: "invisible",
        // We want full control, so we manually call turnstile.execute(widgetId).
        execution: "execute",
        callback: onTokenReceived,
        "error-callback": onTokenError,
        "expired-callback": function () {
          // Expired tokens are treated as missing; the server will reject them.
          tokenInput.value = "";
        },
      });
    }
  }

  // Wait until the Turnstile script has loaded and the global `turnstile`
  // object is available. We poll for a short period so that a slow network
  // does not permanently block the user.
  function waitForTurnstile() {
    return new Promise((resolve) => {
      if (typeof turnstile !== "undefined") {
        resolve();
        return;
      }
      let attempts = 0;
      const checkInterval = setInterval(() => {
        attempts++;
        if (typeof turnstile !== "undefined") {
          clearInterval(checkInterval);
          resolve();
        } else if (attempts > 100) {
          // Give up after ~10s. At this point we will try to submit
          // without a token and let the server return a clear error.
          clearInterval(checkInterval);
          resolve();
        }
      }, 100);
    });
  }

  // As soon as the page is ready and Turnstile is (hopefully) loaded,
  // render the widget so it is ready by the time the user hits submit.
  waitForTurnstile().then(() => {
    renderTurnstile();
  });

  // ---------------------------------------------------------------------------
  // Submit flow
  // ---------------------------------------------------------------------------
  form.addEventListener("submit", async function (e) {
    // We always take over submission so we can validate + get a token first.
    e.preventDefault();

    if (!validateForm()) return;

    submitButton = form.querySelector('button[type="submit"]');
    if (submitButton) {
      submitButton.disabled = true;
      submitButton.textContent = "Sending…";
    }
    setStatus("Verifying, please wait.");

    // Make sure the Turnstile library has had a chance to load.
    await waitForTurnstile();

    // If we still do not have a widget, try to render one and give the
    // browser a short moment to paint it.
    if (widgetId === null) {
      renderTurnstile();
      await new Promise((resolve) => setTimeout(resolve, 100));
    }

    // If for some reason we already have a valid token (for example,
    // Turnstile auto-ran earlier), we can just submit the form.
    if (tokenInput.value) {
      form.submit();
      return;
    }

    // Normal path: ask Turnstile to run in invisible mode. When it finishes,
    // it will call onTokenReceived, which will submit the form if
    // `pendingSubmit` is true.
    if (widgetId !== null && typeof turnstile !== "undefined") {
      pendingSubmit = true;
      turnstile.execute(widgetId);
    } else {
      // We could not talk to Turnstile at all – surface a clear error.
      console.error("Turnstile widget not available");
      onTokenError("widget-not-available");
    }
  });
})();

A version with fewer comments is available in this gist on GitHub (opens in new tab).

Now the updates to the contact.js file that sits in functions/api/ directory.

import { escapeHtml } from '../../_helpers/escape-html.js';

/**
 * Cloudflare Pages Function that powers the contact form.
 *
 * High‑level flow (perfect for a blog diagram):
 *
 * 1. Accept a classic `<form>` POST (urlencoded or multipart).
 * 2. Validate the payload on the server (required fields + length limits).
 * 3. Verify a Cloudflare Turnstile token server‑side.
 * 4. Send a notification email via Resend (HTTPS API call).
 * 5. Redirect the user to a static “thanks” page.
 *
 * The matching HTML form does *not* need JavaScript – the worker stands on its
 * own. Client‑side JS (for validation + Turnstile widget) is purely a UX bonus.
 */
export const onRequestPost = async ({ request, env }) => {
	try {
		// ---------------------------------------------------------------------
		// 1. Only accept traditional browser form submissions
		// ---------------------------------------------------------------------
		// The contact page posts using `application/x-www-form-urlencoded`, but
		// we also support `multipart/form-data` so the handler works with
		// progressive enhancement and file inputs if they ever appear.
		const contentType = request.headers.get('content-type') || '';
		if (!contentType.includes('application/x-www-form-urlencoded') && !contentType.includes('multipart/form-data')) {
			return new Response(JSON.stringify({ error: 'Unsupported content type' }), {
				status: 415,
				headers: { 'content-type': 'application/json' },
			});
		}

		// This gives us a `FormData` instance regardless of which of the two
		// encodings the browser chose.
		const form = await request.formData();

		// ---------------------------------------------------------------------
		// 2. Honeypot – cheap, early bot filter
		// ---------------------------------------------------------------------
		// The form includes a visually hidden checkbox with this wonderfully
		// ugly name. Real users never see or tick it, but naive bots often
		// will. If it has a value, we quietly redirect back to the homepage.
		if (form.get('_its-a-trap!')) {
			const hpUrl = new URL('/', request.url);
			return Response.redirect(hpUrl.toString(), 303);
		}

		const name = (form.get('name') || '').toString().trim();
		const email = (form.get('email') || '').toString().trim();
		const message = (form.get('message') || '').toString().trim();
		const redirectUrl = (form.get('_redirect') || '/contact/thanks/').toString();

		// ---------------------------------------------------------------------
		// 3. Field‑level validation (server‑side, regardless of JS on client)
		// ---------------------------------------------------------------------
		if (!name || !email || !message) {
			return new Response(JSON.stringify({ error: 'Missing required fields' }), {
				status: 400,
				headers: { 'content-type': 'application/json' },
			});
		}

		// Upper bounds on field length keep logs and emails sane and protect
		// downstream services from unexpectedly huge payloads.
		if (name.length > 200 || email.length > 320 || message.length > 5000) {
			return new Response(JSON.stringify({ error: 'Invalid field lengths' }), {
				status: 400,
				headers: { 'content-type': 'application/json' },
			});
		}

		// ---------------------------------------------------------------------
		// 4. Verify Cloudflare Turnstile token (server‑side)
		// ---------------------------------------------------------------------
		// The front‑end Turnstile widget places its token into
		// `cf-turnstile-response`. Without a token, we *always* reject the
		// submission – even if the rest of the payload looks valid.
		const turnstileToken = form.get('cf-turnstile-response');
		if (!turnstileToken) {
			return new Response(JSON.stringify({ error: 'Turnstile token is required' }), {
				status: 400,
				headers: { 'content-type': 'application/json' },
			});
		}

		// The Turnstile secret is never exposed to the client – it lives in
		// Cloudflare Pages environment variables.
		const secret = env.TURNSTILE_SECRET_KEY;
		if (!secret) {
			return new Response(
				JSON.stringify({
					error: 'Server not configured for CAPTCHA (missing TURNSTILE_SECRET_KEY)',
				}),
				{ status: 500, headers: { 'content-type': 'application/json' } }
			);
		}

		// Including the remote IP is optional but recommended; it gives
		// Turnstile a little more context when scoring the request.
		const remoteIp = request.headers.get('CF-Connecting-IP') || request.cf?.clientAddress || null;

		const verifyBody = {
			secret,
			response: turnstileToken.toString(),
		};
		if (remoteIp) {
			verifyBody.remoteip = remoteIp;
		}

		const verifyResp = await fetch('https://challenges.cloudflare.com/turnstile/v0/siteverify', {
			method: 'POST',
			headers: { 'content-type': 'application/json' },
			body: JSON.stringify(verifyBody),
		});

		if (!verifyResp.ok) {
			return new Response(JSON.stringify({ error: 'Failed to verify Turnstile' }), {
				status: 502,
				headers: { 'content-type': 'application/json' },
			});
		}

		const verifyJson = await verifyResp.json();

		// If Turnstile says “no”, do not send an email – fail fast here.
		if (!verifyJson.success) {
			const errorCodes = verifyJson['error-codes'] || [];
			return new Response(
				JSON.stringify({
					error: 'CAPTCHA verification failed',
					details: errorCodes.length > 0 ? errorCodes.join(', ') : 'Unknown error',
				}),
				{ status: 400, headers: { 'content-type': 'application/json' } }
			);
		}

		// Optional defence‑in‑depth check: ensure the token we received was
		// minted for this hostname (or one of its subdomains).
		const requestHost = new URL(request.url).hostname;
		if (
			verifyJson.hostname &&
			verifyJson.hostname !== requestHost &&
			!requestHost.endsWith(`.${verifyJson.hostname}`)
		) {
			console.warn(`Turnstile hostname mismatch: expected ${requestHost}, got ${verifyJson.hostname}`);
		}

		// ---------------------------------------------------------------------
		// 5. Build notification email payload for Resend
		// ---------------------------------------------------------------------
		// `fromEmail` must be a sender you control and have verified with Resend.
		const fromEmail = 'email@example.com';
		const subject = form.get('_email.subject')?.toString() || 'New Message from Nooshu.com';

		const textBody = `New contact form submission on nooshu.com\n\nName: ${name}\nEmail: ${email}\n\nMessage:\n${message}`;
		const htmlBody = `<h2>New contact form submission on nooshu.com</h2><p><strong>Name:</strong> ${escapeHtml(name)}<br/><strong>Email:</strong> ${escapeHtml(email)}</p><p><strong>Message:</strong></p><pre style="white-space:pre-wrap">${escapeHtml(message)}</pre>`;

		// Resend API key comes from Cloudflare Pages environment variables,
		// never from the client or build‑time `.env`.
		const apiKey = env.RESEND_API_KEY;
		if (!apiKey) {
			return new Response(
				JSON.stringify({
					error: 'Server not configured for email (missing RESEND_API_KEY)',
				}),
				{ status: 500, headers: { 'content-type': 'application/json' } }
			);
		}

		// Call the Resend REST API. In this project we make a single,
		// straightforward `emails` call – no templates, CC/BCC, or attachments.
		const resendResp = await fetch('https://api.resend.com/emails', {
			method: 'POST',
			headers: {
				'content-type': 'application/json',
				Authorization: `Bearer ${apiKey}`,
			},
			body: JSON.stringify({
				from: `Nooshu Contact <${fromEmail}>`,
				to: ['email@example.com'],
				reply_to: email,
				subject,
				text: textBody,
				html: htmlBody,
			}),
		});

		if (!resendResp.ok) {
			const text = await resendResp.text();
			return new Response(JSON.stringify({ error: 'Email send failed', details: text }), {
				status: 502,
				headers: { 'content-type': 'application/json' },
			});
		}

		// ---------------------------------------------------------------------
		// 6. Redirect to “thanks” page
		// ---------------------------------------------------------------------
		// Using a 303 ensures the follow‑up request is a GET, which means
		// browser refreshes do not re‑POST the form.
		const finalRedirect = new URL(redirectUrl, request.url);
		return Response.redirect(finalRedirect.toString(), 303);
	} catch (err) {
		return new Response(JSON.stringify({ error: 'Server error', details: String(err) }), {
			status: 500,
			headers: { 'content-type': 'application/json' },
		});
	}
};

A version with fewer comments is available in this gist on GitHub (opens in new tab).

Amazingly, that's really the only two major changes!

The minor changes were:

Footer template: Update my footer with a link to the JavaScript above and the Turnstile JavaScript, e.g. https://challenges.cloudflare.com/turnstile/v0/api.js.
CSP: Modified my _headers file to remove Google from my CSP and add in the challenges URL (https://challenges.cloudflare.com/) to the connect-src, script-src-elem, and frame-src.
CSS: Slightly modify the CSS to remove Google ReCAPTCHA v3 specific styling (.grecaptcha-badge) and add Turnstile CSS (.turnstile-widget).
Secret Keys: RECAPTCHA_SECRET_KEY swapped out for the TURNSTILE_SECRET_KEY in the Cloudflare Pages environment variables settings page.
Contact Form HTML: HTML changes to the contact form added cf-turnstile-response hidden <input> and #turnstile-container <div>.

Why change?

So I've literally done a 180 from Google ReCAPTCHA v3 to Cloudflare Turnstile in less than a week. So why change and what are the differences? The top of my priority list that I admittedly should have considered when integrating Google ReCAPTCHA v3 is privacy! I would rather not subject any readers of the blog to any additional tracking from Google! But there are other differences too:

Area	Cloudflare Turnstile advantage over reCAPTCHA v3	Why it matters
Privacy	Turnstile is positioned as a privacy preserving alternative, and it does not rely on tracking user data across sites for ad retargeting. It also offers Ephemeral IDs, which are short-lived device identifiers that work without cookies or client-side storage.	Better fit for teams with strong privacy, GDPR, or public sector concerns.
Less operational tuning	Turnstile’s server validation is a straightforward success or failure check. By contrast, reCAPTCHA v3 returns a score and Google recommends that you review scores in the admin console and tune thresholds based on your own traffic, starting from 0.5.	Usually simpler to ship and maintain, especially for smaller teams that do not want to spend time calibrating fraud thresholds.
No CAPTCHA experience by default	Turnstile works without showing visitors a CAPTCHA and supports invisible, non-interactive, and managed modes. reCAPTCHA v3 is also frictionless, but Turnstile is explicitly built as a CAPTCHA replacement with more presentation modes for the widget itself.	More control over UX and easier to keep forms feeling clean while still validating traffic.
Can be used on any site without Cloudflare CDN	Turnstile can be embedded into any website, it does not require your traffic to go through Cloudflare or use Cloudflare’s CDN.	Easier adoption if you want the bot check without changing your hosting or edge setup.
Developer-friendly testing	Turnstile provides dummy site keys and secret keys for testing, and Cloudflare documents that these work on localhost and any development domain. Google’s reCAPTCHA FAQ says for v3 you should create a separate key for testing environments, and scores may not be accurate because v3 relies on real traffic.	Usually a smoother local dev and automated test setup.
Free plan positioning	Cloudflare documents Turnstile has a Free plan intended for personal sites, SMBs, dev and test, and most production applications. Google’s FAQ notes quota limits for non-Enterprise reCAPTCHA, and if a v3 key exceeds a monthly quota it may fail open with a static score of 0.9 for the remainder of the month.	Attractive when cost certainty matters or when you want fewer surprises as usage grows.
Extra integration option with Cloudflare security stack	Turnstile supports pre-clearance, letting it issue clearance cookies that can be used across Cloudflare protected domains.	Helpful if you already use Cloudflare WAF or bot controls and want tighter integration.

Summary

Well, that brings us to the end of another blog post, I only went off on a tangent a couple of times! So well done for sticking with my ramblings!

In the end, the goal was to keep the architecture simple while still covering the essentials. The site itself remains fully static, with a lightweight Cloudflare Worker handling form submissions. Spam protection is provided through ~~Google reCAPTCHA v3~~, Cloudflare Turnstile, while validation and accessibility patterns ensure the form behaves reliably for real users. It is a small piece of functionality, but implemented carefully it can be both robust and easy to maintain.

As always, thanks for reading, and I’d love to hear your feedback. You are welcome to contact me either via the contact form (again, very meta! 😏), or via any of the various social channels listed on the site.

Post changelog:

09/03/26: Initial post published.
13/03/26: Updates to give instructions on how to use Cloudflare Turnstile as an alternative to Google reCAPTCHA v3. Thanks again to Ryan Townsend and Paweł Grzybek.

Precompressed HTML at the Edge: Eleventy Meets Cloudflare Workers

2026-02-21T00:00:00Z

Introduction

In 2025, I wrote a series of web performance optimisation blog posts focussing on some of the key fundamental's of Frontend Web Performance:

Caching

Asset fingerprinting and the preload response header in 11ty

Summary

The blog post describes how the I enhanced web performance on my 11ty-built site by combining asset fingerprinting with the HTTP preload hint.

It explains that preload tells the browser to fetch critical resources earlier, but hashed filenames make this difficult to manage manually.

The solution was to generate preload Link headers automatically during the 11ty build. A custom script locates the fingerprinted CSS file and injects the correct preload header into the Cloudflare Pages _headers file.

This speeds up CSS delivery, removes the need for manual updates, and allows the use of long-lived Cache-Control header values such as max-age=31536000 and immutable.

Compression

Cranking Brotli up to 11 with Cloudflare Pro and 11ty.

Summary

The blog post explains how I improved the performance of my 11ty-powered blog after migrating to Cloudflare Pages by using Brotli compression at the highest level (11) for static assets. I also describes the difference between Brotli and gzip, outline how Cloudflare’s Pro plan typically applies a moderate Brotli level (4), and then show how to pre-compress JavaScript files to Brotli level 11. Lastly, serve them correctly both locally and via Cloudflare, and configure Cloudflare’s compression rules so that all assets benefit from the stronger compression to reduce file sizes and improve load performance.

Concatenation

Using an 11ty Shortcode to craft a custom CSS pipeline

Summary

While not strictly about concatenation, it covers related ideas. The post explains how I built a custom CSS pipeline for my 11ty site using a bespoke short code rather than the default bundle plugin.

It details how I preserved local live reload, added content-based fingerprinting for long-term caching, minified CSS with clean-css package, enabled Brotli compression, and used disk and memory caching to prevent unnecessary work.

The build also generates hashed filenames and matching HTML link tags, ensuring production serves fully optimised, cache friendly CSS automatically.

More compression

In this blog post, I’m going to take the compression a little further. I already have CSS and JavaScript Brotli compressed to the highest level (11) and served from Cloudflare Pages. But what about the third and final core technology of the web? Arguably the most important too… The HTML. In this post, I will look at how to compress your HTML to 11 during the 11ty build phase, and what modern technologies you need in order to make this work (it’s not as straightforward as I thought!)

Why HTML Brotli matters?

Before we dive into the details, let’s discuss why Brotli compression is important for HTML. Well, to summarise Brotli compression in one sentence:

Brotli compression reduces file size by intelligently identifying repeated patterns in data and encoding them more efficiently using a combination of modern compression techniques.

This is fantastic for anything with repeating patterns as the bytes saved over the network can be huge! The great thing about HTML is that it has a tonne of repeating patterns (e.g. Markup)! This reduction in file size could potentially equate to:

Improved Web Performance

Lower Time To First Byte impact on slow networks
Improved First Contentful Paint
Improved Largest Contentful Paint
Reduced Total Blocking Time (indirectly)

Every kilobyte saved here multiplies across your traffic volume.

At scale, cost savings add up

Let’s assume you are serving a high traffic website, say the BBC, Google, or GOV.UK. Imagine how much bandwidth could be saved by simply compressing your HTML. Any percentage saving on millions of requests per day is going to mean:

Less bandwidth
Lower CDN egress
Lower cloud costs
Lower carbon footprint

This is a win both commercially and environmentally!

Improved performance on slow and unstable mobile networks

Brotli 11 improves the web for users where they need it the most:

3G connectivity
High latency rural connections
Congested public networks
International access

HTML blocks everything else. If you shrink it aggressively, you unblock the page faster, meaning users get a better experience.

This is a real-world performance gain.

It indirectly improves Core Web Vitals

Smaller HTML means:

Faster DOM construction
Earlier CSS discovery
Earlier JS discovery
Reduced main thread idle gaps

This is especially important for server rendered pages or hybrid Server-side Rendered apps.

Compression cost is paid once for static assets

Yes, level 11 compression is expensive to compute. But this cost is paid back over time. You pay for the CPU time once. Users benefit forever assuming:

the HTML is static and cacheable at the Content Delivery Network (CDN) using long-life caching headers
you automate and pre-compress the HTML at build time

Compression Size Examples v1

Let’s have a look at a huge HTML page on the web. My go-to for this is either the W3C HTML5 Specification page OR NASA’s Astronomy Picture of the Day Archive.

Rather than choose, let’s just compress both!

W3C HTML5 Specification (Single Page)

URL: https://www.w3.org/TR/2011/WD-html5-20110405/Overview.html
Uncompressed size: 4.7 MB
Brotli (Level 11): 590 KB (~88% saving)

That’s an 88 percent reduction in bytes over the network when compressing the HTML with Brotli 11. Not bad for something that takes just over 10-seconds to run.

And yes, that’s 4.7 MB of HTML alone. It’s an absolute beast of a page.

For perspective, that would take around 12 to 15 minutes to download on a 56k modem in the late 90s. Just for the HTML. I might be showing my age here 😭

Nasa’s Astronomy Picture of the Day Archive

URL: https://apod.nasa.gov/apod/archivepix.html
Uncompressed size: 314 KB
Brotli (Level 11): 53 KB (83% saving)

We’ve taken it from roughly a third of a megabyte down to just 53 KB. That is a pretty substantial reduction!

This means faster page loads, lower data usage, and a much smoother experience for anyone on a limited data plan or dealing with patchy connections. A very positive impact, right where it matters most for users.

How is this different from my other compression posts?

So how is this different from the Brotli compression I have mentioned in the previous blog posts?

In Cranking Brotli up to 11 with Cloudflare Pro and 11ty I used a combination of:

Brotli CLI (e.g.brew install brotli)
bash scripts (compress.sh, compress-directory.sh)

In this post, I override the Cloudflare Dashboard's "Compression Rules” (which dynamically compresses HTML at Brotli level 4). The CDN is compressing the HTML on-the-fly when the user’s browser requests it.

What this blog post describes is pre-compression to Brotli 11 at 11ty build time. To do this the workflow is:

Apply Brotli level 11 pre-compression to HTML during the 11ty build on Cloudflare Pages.
_helpers/html-compression.js runs after the site is written to _site.
Each HTML file gets a matching .br file
Use the built-in Node zlib module, so no manual setup, CLI tools, scripts, Cloudflare configuration, or extra dependencies are required.
Take advantage of Cloudflare Pages Functions, which run on Cloudflare Workers. This is a great opportunity to use a modern, fast evolving platform that opens the door to powerful edge capabilities.

What’s the point?

Well, that’s a great question. As some readers may know by default Cloudflare compresses HTML at compression level 4. Now, if we compare this level to the compression level 11 above, let’s have a look at the difference:

Compression Size Examples v2

W3C HTML5 Specification (Single Page)

URL: https://www.w3.org/TR/2011/WD-html5-20110405/Overview.html
Uncompressed size: 4.7 MB
Brotli (Level 4): 729 KB (83% saving)
Compression Time (Level 4): 0.149 seconds
Brotli (Level 11): 590 KB (87% saving)
Brotli (Level 11): 11.717 seconds

Nasa’s Astronomy Picture of the Day Archive

URL: https://apod.nasa.gov/apod/archivepix.html
Uncompressed size: 314 KB
Brotli (Level 4): 66 KB (79%)
Compression Time (Level 4): 0.011 seconds
Brotli (Level 11): 53 KB (83% saving)
Brotli (Level 11): 0.743 seconds

It’s worth noting that I used Paul Calvano's fantastic Compression Tester Tool, to help with this basic analysis.

What you will likely notice is that even for large HTML files the size difference between compression level 4 and compression level 11 isn’t huge, only 12 KB in the W3C HTML5 Specification example. The real difference comes in computation time. Level 4 0.149 seconds verses 11.717 seconds! That’s a 7764% increase in time between Level 4 and Level 11! Although this is an extreme example, you can probably see why Level 11 isn’t used by Cloudflare for on-the-fly compression of HTML assets. Level 4 gives a good balence between file compression versus compression speed. I’m betting countless smart people were involved in the analysis of using Level 4 by default! When you are a company that is literally serving billions of requests per second, this decision really makes a difference in terms of processing power infrastructure and power usage!

Thankfully, the way that I have implemented level 11 compression on my blog, processing time doesn’t really matter. All the HTML is being compressed at 11ty build time. As I said above, the cost of this additional CPU time is paid back over time by users getting a better experience (even if only slightly). Furthermore, remember there’s a slight reduction in storage required on the CDN. From my perspective, if it is low effort after setup, it feels like the right move to implement it.

Problems

As I found out during implementation it isn’t just as simple as compressing the HTML to 11 and setting a static Content-Encoding header for HTML in the _headers file (trust me, I tried it!)

Problem 1: URL Path vs Actual File Path Mismatch

When serving static assets like CSS, JS, or images, there is usually a simple one-to-one relationship between the URL and the file on disk. A request to /css/site.css maps directly to _site/css/site.css. No extra logic is required because the URL path matches the file path exactly. I had no idea, but I soon found out that HTML pages behave differently. A request to / or /blog/post/ does not correspond to a literal file at that path. Instead, the server applies a convention and serves index.html inside that directory. So /blog/post/ actually maps to blog/post/index.html on disk. This mapping happens automatically when Cloudflare serves uncompressed HTML through its very efficient static asset layer.

The problem appears when serving pre-compressed Brotli files. You cannot simply request the same URL and expect the .br file to resolve. Instead, the Cloudflare Function must manually translate the directory style URL into the real file path before appending .br.

For example, / becomes /index.html.br, /blog/post/ becomes /blog/post/index.html.br, and /404.html becomes /404.html.br.

In short, HTML requires explicit path translation because the browser sees a directory style URL while the actual file stored on disk is index.html. The Cloudflare Function must bridge that gap to correctly serve the Brotli 11 compressed version, rather than the uncompressed HTML version.

It actually makes sense now that I think about it, I’ve always just taken the automatic appending of index.html to a URL Path for granted! The fact that as a user on the web doesn’t even have to think about that small detail, shows how well it works! As Dieter Rams once said:

Good design is as little design as possible.

Problem 2: A page full of Wingdings

I’m showing my age again, but for readers who don’t remember early versions of Windows (e.g. 3.1), it came bundled with a font called Wingdings. This True Type font contains many largely recognised shapes and gestures as well as some recognised world symbols.

Wingdings were an early symbol font that experimented with pictographic digital symbols, that would later lead on to ASCII emoticons like :-) & ¯\_(ツ)_/¯, which in turn would progress to the modern world of Emoji’s!

Essentially, what was happening the Cloudflare server was serving raw Brotli compressed HTML files to the browser, expecting it to understand what these (now binary, not text) files were, and how to read and understand them. I was essentially serving the HTML without the following headers:

Content-Type: text/html; charset=UTF-8
Content-Encoding: br
Vary: Accept-Encoding

Here’s a brief explanation of these headers:

Content-Encoding: br: This is telling the browser “What I’m sending you is a Brotli compressed file, you are going to need to decode it before you understand it”.
Content-Type: text/html; charset=UTF-8: This tells the browser what character set ('charset') it should use after decompression, this is essential as this is key to the browser parsing the HTML correctly.
Vary: Accept-Encoding only affects caching (e.g. Content Delivery Networks). It’s basically saying to the cache to store separate versions of this file depending on the Accept-Encoding header.

The result of the above gave me a homepage that looked like the image below (interesting but not exactly readable!):

My Approach

Step 1: Build-time compression

After the site is generated, it moves into a final preparation stage before going live.

During this stage, the system goes through all the finished HTML pages and creates highly compressed versions of them. These compressed files sit alongside the originals and are ready to be served immediately.

Because this happens as part of the 11ty build process, every release automatically includes freshly optimised files. This means the live site can deliver pages faster, with smaller file sizes and no need to compress anything on the fly (e.g. what Cloudflare does with its HTML compression to Brotli level 4).

The result is better performance for users, with no extra overhead once the site is hosted and running on Cloudflare pages.

Step 2: Cloudflare Pages Function for content negotiation

When someone visits a page on the site, a lightweight Cloudflare Function (via a Cloudflare Worker) checks whether a users browser supports modern compression.

If it does, the system serves the Brotli 11 pre compressed version of the page. This keeps file sizes small and pages loading quickly.

If the browser does not support this compression, or if a compressed version is not available, the system simply serves the standard uncompressed version of the HTML instead.

No extra processing happens at this stage. The edge layer (Cloudflare Function + Worker) is only deciding which version of the already prepared files to send. All optimisation work has already been done earlier in the 11ty build process.

The final result is fast delivery, efficient bandwidth use, and a simple, reliable build setup. Everyone wins!

Step 3: The Eleventy build uses Node.js zlib, for seamless integration

As mentioned earlier, this Brotli implementation differs from others I have used. Instead of relying on bash scripts such as compress.sh or compress-directory.sh, it uses Node.js’s built in zlib module. Because zlib is part of Node.js core, it is stable, well maintained, and requires no external dependencies. It has been available since the earliest Node.js releases, so it is a sensible default choice. I may even revisit the other build process in future and consider replacing the remaining bash scripts with a fully Node.js based approach.

Implementation

Next, let’s stop the waffling and get onto the actual implementation, I assume that’s what readers are here for after all!

Core compression utility

Here is the main compression file that is used to compress the HTML to Brotli 11:

/**
 * Core compression utility: Brotli compression for build-time pre-compression.
 *
 * This module is the single source of truth for Brotli in the project. It is used by:
 * - html-compression.js  — compresses all HTML in _site to .br (level 11)
 * - css-manipulation.js  — compresses processed CSS to .br when writing to _site
 * - js-compression.js    — compresses minified JS in _site/js to .br
 *
 * All compression happens during the Eleventy production build (postbuild phase).
 * Pre-compressed .br files are then served via content negotiation in functions/[[path]].js
 * when the client sends Accept-Encoding: br, avoiding any runtime or CDN dynamic compression.
 */

import { brotliCompressSync } from 'zlib';

/**
 * Default Brotli compression level.
 *
 * Brotli levels range from 0 to 11:
 * - 0–3: Fast, lower ratio (typical for dynamic/on-the-fly compression; e.g. CDNs often use 4).
 * - 11:  Maximum ratio, slowest; ideal for static assets compressed once at build time.
 *
 * We use 11 because compression runs only during the build, so CPU cost is paid once per
 * deploy. The resulting .br files are then served as-is with no re-compression at the edge.
 */
export const BROTLI_LEVEL = 11;

/**
 * Compress input data with Brotli.
 *
 * Uses Node's synchronous zlib API so callers can use the function in a simple, blocking
 * way during the build (no need to await). Input is normalized to a Buffer so that
 * strings (e.g. file contents read as UTF-8) and TypedArrays are supported.
 *
 * @param {Buffer | Uint8Array | string} input - Data to compress. If a string, it is
 *   encoded as UTF-8. Buffers and Uint8Array are used as-is (after copying to a Buffer
 *   when necessary via Buffer.from).
 * @param {number} [level=BROTLI_LEVEL] - Compression level 0–11. Defaults to 11 for
 *   maximum ratio. Can be overridden (e.g. via BROTLI_COMPRESSION_LEVEL in css-manipulation).
 * @returns {Buffer} - Compressed data as a Buffer. Callers typically write this to a
 *   file with a .br extension (e.g. index.html.br).
 */
export function brotliCompress(input, level = BROTLI_LEVEL) {
	const buffer = Buffer.isBuffer(input) ? input : Buffer.from(input);
	return brotliCompressSync(buffer, { level });
}

I appreciate that not everyone prefers heavily commented code, so there is also a version in this Gist (opens in new tab) with minimal comments and improved readability.

HTML compression post-build

The post-build HTML compression file:

/**
 * HTML Brotli compression (postbuild).
 *
 * This module compresses every HTML file in the Eleventy output directory (_site) to
 * Brotli level 11 and writes a corresponding .br file alongside each .html file
 * (e.g. index.html → index.html.br, blog/post/index.html → blog/post/index.html.br).
 *
 * When it runs:
 *   Only during the production postbuild phase, after Eleventy has finished writing
 *   _site. It is invoked from _config/build-events.js in the eleventy.after handler
 *   (alongside JS minification, JS Brotli, and preload header generation).
 *
 * How .br files are served:
 *   The Cloudflare Pages Function functions/[[path]].js performs content negotiation
 *   for HTML document requests. When the client sends Accept-Encoding: br, the Function
 *   fetches the pre-built .br asset (e.g. / → index.html.br) and returns it with
 *   Content-Encoding: br. Clients that do not advertise Brotli support receive the
 *   uncompressed .html from the static asset bucket. No dynamic compression runs at
 *   the edge; this step does all compression once at build time.
 *
 * Uses the core compression utility _helpers/compression.js for the actual Brotli call.
 */

import fs from 'fs';
import path from 'path';

import { brotliCompress, BROTLI_LEVEL } from './compression.js';

/**
 * Recursively find all .html files under a directory.
 *
 * Used to discover every HTML file in _site (root index.html, blog posts, portfolio
 * items, 404.html, etc.). Paths are returned as relative to _site so that we can
 * join them with siteDir for full paths and still have readable relative paths for
 * logging and error messages.
 *
 * @param {string} dir - Absolute or relative directory path to search (e.g. ./_site or a subdir).
 * @param {string[]} [acc=[]] - Accumulator array; results are pushed here during recursion.
 * @returns {string[]} Relative paths to .html files (e.g. ['index.html', 'blog/post/index.html']).
 */
function findHtmlFiles(dir, acc = []) {
	const entries = fs.readdirSync(dir, { withFileTypes: true });
	for (const entry of entries) {
		const fullPath = path.join(dir, entry.name);
		const relPath = path.relative('./_site', fullPath);
		if (entry.isDirectory()) {
			findHtmlFiles(fullPath, acc);
		} else if (entry.isFile() && entry.name.endsWith('.html')) {
			acc.push(relPath);
		}
	}
	return acc;
}

/**
 * Compress all HTML files in _site with Brotli level 11, writing .br files.
 *
 * Call this only after the Eleventy build has completed and _site is fully written.
 * For each .html file: reads content, compresses with brotliCompress (level 11),
 * writes <filename>.br next to the original. Existing .br files are skipped if their
 * mtime is >= the source .html mtime (incremental safety; in a full build all HTML is
 * usually newer, so most files are compressed). Logs counts, total bytes saved, and
 * duration; collects per-file errors without stopping the loop.
 */
export function compressHtmlFiles() {
	const startTime = Date.now();
	console.log('\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━');
	console.log('🚀 PRODUCTION POSTBUILD: Starting HTML Brotli compression');
	console.log('━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n');

	const siteDir = path.join('./_site');
	if (!fs.existsSync(siteDir)) {
		console.log('⚠️  _site directory not found, skipping HTML compression');
		return;
	}

	const htmlFiles = findHtmlFiles(siteDir);
	if (htmlFiles.length === 0) {
		console.log('⚠️  No HTML files found, skipping compression');
		return;
	}

	let compressedCount = 0;
	let skippedCount = 0;
	let totalOriginal = 0;
	let totalCompressed = 0;
	const errors = [];

	for (const relPath of htmlFiles) {
		const inputPath = path.join(siteDir, relPath);
		const outputPath = `${inputPath}.br`;

		try {
			// Skip writing if .br already exists and is not older than the source .html
			if (fs.existsSync(outputPath)) {
				const inputStats = fs.statSync(inputPath);
				const outputStats = fs.statSync(outputPath);
				if (outputStats.mtime >= inputStats.mtime) {
					skippedCount++;
					continue;
				}
			}

			const fileContent = fs.readFileSync(inputPath);
			const originalSize = fileContent.length;

			const brotliBuffer = brotliCompress(fileContent, BROTLI_LEVEL);
			const compressedSize = brotliBuffer.length;

			fs.writeFileSync(outputPath, brotliBuffer);
			compressedCount++;
			totalOriginal += originalSize;
			totalCompressed += compressedSize;
		} catch (error) {
			errors.push(`❌ ${relPath}: ${error.message}`);
		}
	}

	const totalTime = Date.now() - startTime;
	const savedPercent = totalOriginal > 0 ? ((1 - totalCompressed / totalOriginal) * 100).toFixed(1) : '0';

	console.log(`✅ Compressed ${compressedCount} HTML file(s) (${skippedCount} skipped, up-to-date)`);
	if (compressedCount > 0) {
		console.log(
			`   ${(totalOriginal / 1024).toFixed(1)} KB → ${(totalCompressed / 1024).toFixed(1)} KB (${savedPercent}% smaller)`
		);
	}
	console.log(`   Finished in ${totalTime}ms (${(totalTime / 1000).toFixed(2)}s)`);

	if (errors.length > 0) {
		console.error('\nErrors:');
		errors.forEach(e => console.error(`  ${e}`));
	}

	console.log('\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n');
}

A version with minimal comments and much improved readability is available in a Gist here (opens in new tab).

Cloudflare Pages Function for content negotiation

Here we have the Cloudflare Function file that runs in the Cloudflare Worker. It is located in the /functions directory in the root of the repository so it can be detected when Cloudflare Pages builds.

/**
 * Catch-all Cloudflare Pages Function: HTML document content negotiation.
 *
 * This Function runs for every GET and HEAD request that is not handled by a more
 * specific route (e.g. POST /api/contact is handled by functions/api/contact.js).
 * Its job is to serve pre-compressed Brotli (.br) HTML when the client supports it,
 * and otherwise pass through to the static asset bucket (uncompressed HTML).
 *
 * Pre-compressed .br files are produced at build time by _helpers/html-compression.js
 * (postbuild): every .html in _site gets a matching .br (e.g. index.html.br,
 * blog/post/index.html.br). This Function does not compress on the fly; it only
 * chooses which asset to serve and sets the correct response headers.
 *
 * Flow:
 * 1. If the request is not for an HTML document URL (see below), pass through to ASSETS.
 * 2. If the client does not send Accept-Encoding: br, pass through (serve uncompressed HTML).
 * 3. Otherwise, fetch the corresponding .br asset; if missing or error, fall back to ASSETS
 *    for the original request (uncompressed).
 * 4. If .br exists, return it with Content-Encoding: br and no-transform so nothing
 *    re-compresses the body.
 */

async function handleHtmlWithBrotli(request, env) {
	const url = new URL(request.url);
	const pathname = url.pathname;

	// Only treat as HTML document: root (/), directory-style paths ending in /, or 404
	// (Cloudflare Pages maps /404 to /404.html). All other paths (e.g. /script.js,
	// /style.css, /favicon.ico) go straight to static assets; Pages serves .br for
	// static files when available and client sends Accept-Encoding: br.
	const isHtmlDocument = pathname === '/' || pathname.endsWith('/') || pathname === '/404.html';

	if (!isHtmlDocument) {
		return env.ASSETS.fetch(request);
	}

	const acceptsBrotli = request.headers.get('Accept-Encoding')?.includes('br') ?? false;

	if (!acceptsBrotli) {
		return env.ASSETS.fetch(request);
	}

	// Map URL path to the .br file in the asset bucket. Eleventy outputs directory
	// indexes as index.html (e.g. /blog/post/ → blog/post/index.html), so the .br
	// file is <path>index.html.br.
	const brPath =
		pathname === '/' ? '/index.html.br' : pathname === '/404.html' ? '/404.html.br' : `${pathname}index.html.br`;

	const brUrl = new URL(brPath, url.origin);
	const brResponse = await env.ASSETS.fetch(brUrl.toString());

	if (!brResponse.ok) {
		return env.ASSETS.fetch(request);
	}

	// Build response from the .br body with headers that declare Brotli and prevent
	// any downstream re-compression or transformation.
	const headers = new Headers(brResponse.headers);
	headers.set('Content-Encoding', 'br');
	headers.set('Content-Type', 'text/html; charset=UTF-8');
	headers.set('Vary', 'Accept-Encoding');
	// no-transform: tells caches (including Cloudflare) not to re-encode or modify the body
	headers.set('Cache-Control', 'public, max-age=31536000, no-transform');

	return new Response(brResponse.body, {
		status: brResponse.status,
		headers,
		// encodeBody: 'manual' — we are returning the body as-is (already Brotli).
		// Without this, the Workers runtime might try to compress the response again.
		encodeBody: 'manual',
	});
}

export const onRequestGet = async ({ request, env }) => {
	return handleHtmlWithBrotli(request, env);
};

export const onRequestHead = async ({ request, env }) => {
	return handleHtmlWithBrotli(request, env);
};

A version with minimal comments and much better readability is available in a Gist here (opens in new tab).

Build lifecycle wiring

This is the main build file I use to build my 11ty blog for production on Cloudflare Pages.

/**
 * Build lifecycle wiring: registers Eleventy before/after hooks for production.
 *
 * This module is the single place where postbuild steps are scheduled. It is loaded
 * from the main Eleventy config (e.g. eleventy.config.js), which calls
 * registerBuildEvents(eleventyConfig). Handlers run only when env.isLocal is false
 * (i.e. production or preview builds); local dev builds skip all of this so that
 * _site is left as plain output and the dev server stays fast.
 *
 * Order of operations:
 * - eleventy.before: clear CSS build cache so CSS is regenerated and Brotli-compressed
 *   from scratch when needed.
 * - eleventy.after: run the production postbuild phase in a fixed order:
 *   1. generatePreloadHeaders() — write Link headers into _headers for CSS (and .br).
 *   2. minifyJavaScriptFiles()   — minify JS in _site/js (must run before Brotli).
 *   3. compressJavaScriptFiles() — Brotli-compress JS to .br.
 *   4. compressHtmlFiles()       — Brotli-compress all HTML in _site to .br.
 *   Then: destroy HTTP/HTTPS global agents to avoid hanging connections, and on
 *   production only force process.exit(0) after a short delay so the process
 *   terminates cleanly on CI/Cloudflare Pages.
 */

import env from '../_data/env.js';
import { clearCssBuildCache } from '../_helpers/css-manipulation.js';
import { generatePreloadHeaders } from '../_helpers/header-generator.js';
import { compressHtmlFiles } from '../_helpers/html-compression.js';
import { compressJavaScriptFiles } from '../_helpers/js-compression.js';
import { minifyJavaScriptFiles } from '../_helpers/js-minify.js';

/**
 * Register eleventy.before and eleventy.after handlers.
 * Only registered when !env.isLocal (production/preview).
 * @param {import("@11ty/eleventy").UserConfig} eleventyConfig
 */
export function registerBuildEvents(eleventyConfig) {
	if (env.isLocal) {
		return;
	}

	// Before build: clear any cached CSS build artifacts so this run produces fresh
	// processed and Brotli-compressed CSS when templates reference CSS.
	eleventyConfig.on('eleventy.before', () => {
		clearCssBuildCache();
	});

	eleventyConfig.on('eleventy.after', async () => {
		console.log('\n═══════════════════════════════════════════════════════════════════════════════');
		console.log('🚀 PRODUCTION POSTBUILD PHASE: Beginning postbuild operations');
		console.log('═══════════════════════════════════════════════════════════════════════════════\n');
		generatePreloadHeaders();
		await minifyJavaScriptFiles();
		compressJavaScriptFiles();
		compressHtmlFiles();
		console.log('\n═══════════════════════════════════════════════════════════════════════════════');
		console.log('✅ PRODUCTION POSTBUILD PHASE: All postbuild operations completed');
		console.log('═══════════════════════════════════════════════════════════════════════════════\n');

		// Tear down Node's default HTTP/HTTPS agents so the process can exit without
		// waiting for keep-alive connections to time out (e.g. on Cloudflare Pages CI).
		console.log('🔧 Forcing cleanup of HTTP connections and timers...');
		await new Promise(resolve => setTimeout(resolve, 100));

		try {
			const http = await import('http');
			const https = await import('https');

			if (http.globalAgent) {
				http.globalAgent.destroy();
				console.log('✅ Destroyed HTTP global agent');
			}
			if (https.globalAgent) {
				https.globalAgent.destroy();
				console.log('✅ Destroyed HTTPS global agent');
			}
		} catch (error) {
			console.warn('⚠️  Could not destroy HTTP agents:', error.message);
		}

		await new Promise(resolve => setTimeout(resolve, 50));
		console.log('✅ HTTP connection cleanup completed');

		// On production builds only, force exit after a short delay so the runner
		// (e.g. Cloudflare Pages) gets a clean exit code and doesn't hang.
		if (env.isProd) {
			console.log('🏁 Production build complete - forcing process exit in 2 seconds...');
			setTimeout(() => {
				console.log('👋 Forcing clean exit now');
				process.exit(0);
			}, 2000);
		}
	});
}

A version with minimal comments and much improved readability is available in a Gist here (opens in new tab).

Other Technical details worth mentioning

Here are 4 other small technical details worth explaining as part of the implementation:

Why encodeBody: 'manual' is required

The content we send back to the user's browser is already compressed using Brotli. It is being loaded from a file that has already been compressed in advance.

If we don't tell Cloudflare to leave it alone, it may assume the content is not compressed and try to compress it again. Compressing something that is already compressed can cause problems and may result in a broken or unreadable output, plus it is a waste of CPU time and resources.

By setting encodeBody: ‘manual', we are telling Cloudflare to send the content exactly as it is, without changing it. This ensures the pre-compressed file is delivered correctly to the user's browser.

Why Vary: Accept-Encoding and Cache-Control: no-transform matter

These 2 response headers are critical for making the pre-compression work. I've given details as to why that is below:

Vary: Accept-Encoding: This notifies browsers and CDNs that the response can change depending on what kind of compression the browser supports.

For example, if a browser says it supports Brotli, the cache will store and return the Brotli version for those requests. If another browser doesn't support Brotli, the cache will store and return a different version, such as an uncompressed version. This prevents the wrong format being sent to the wrong browser.

Cache-Control: no-transform: This informs caches and other systems between the server and the user's browser not to modify the content. It asserts that the response should not be compressed again or altered in any way.

Without this setting, a proxy might try to compress the content again, which can cause errors and waste processing power. With this header in place, the already compressed file is stored and delivered exactly as intended.

Incremental build optimisation (runtime check to skip unchanged files)

After the Eleventy build finishes, the post-build step recursively scans through the output folder, such as the _site directory, and checks each HTML file.

Before compressing a file, it checks whether a matching .br file already exists and whether it is up-to-date. If the .br file is the same age or newer than the original file, it is skipped. If the page is new or has been updated, a fresh compressed version is created.

This avoids pages that have not changed being needlessly recompressed, keeping the post build step fast. When only a few pages are updated, the need for recompression is limited.

Why we need a Cloudflare Function instead of just the `_headers` file for HTML Brotli?

1. `_headers` can only change headers, not the file itself

The _headers file lets us add or modify (but not remove) response headers. It doesn't control which file is actually sent back to the browser.

So, when someone visits / or /blog/post/, Cloudflare Pages automatically serves index.html or blog/post/index.html.

If we want to serve the Brotli version of the HTML, we need to send index.html.br instead. But the _headers file has no way to switch the file being served.

2. Setting the header alone is not enough

Even if you add Content-Encoding: br in the static _headers file, the actual file being sent would still be the normal uncompressed version of the HTML.

The browser would see this Brotli header and try to decompress the response. Since the content being sent isn't compressed, it would simply fail and the page would break.

Results

Looking at my build logs I can now see that compressing the HTML to Brotli 11 has had the following results:

📊 HTML Brotli 11 total savings: 123.4 KB (75.2% reduction)

That’s not too bad a saving considering how simple it is to set up and integrate into the 11ty build process! Thankfully, now that it’s done I can just “set it and forget it!”. Let's examine the results from the DevTools Network panel below:

DevTools Before

DevTools After

DevTools After Page Reload

Curiously, when reloading the page with DevTools open, the HTTP status code changes from 200 to 304 and the Brotli compression in the Content-Encoding: br disappears. The reason for this is because either:

The reload request doesn’t contain an Accept-Encoding: br header so the Cloudflare Worker is simply returning the uncompressed version of the HTML file as is expected.
The 304 has no response body, so there’s nothing to show as being compressed.

Difference in build times

Thanks to Thomas Steiner on Mastodon for the reminder. It completely slipped my mind to share the difference in build time before and after adding HTML Brotli compression.

Total Build time before: ~ 2 minutes 58 seconds
Total Build time after: ~ 3 minutes 16 seconds

Compressed: 485 HTML file(s) (0 skipped, up-to-date) Total time: 18.191299 seconds (about 18.19s) to Brotli compress all HTML files.

Summary

By pre-compressing this blog’s HTML during the 11ty build phase, I have reduced the number of bytes sent on each page load. While the savings are small for a low traffic site like mine, at scale across millions of users and billions of requests per day, this approach could deliver meaningful bandwidth reductions and incremental performance improvements. This is especially true where network speed and stability vary globally.

Thank you for reading, I hope you found it useful. Edge Workers are an incredibly powerful technology. I genuinely look forward to using them again in the future.

I always open to feedback and corrections. If you spot anything that needs fixing or is incorrect, please let me know. I will credit you in the post changelog below.

Post changelog:

21/02/26: Initial post published.
24/02/26: Thanks to Thomas Steiner for nudging me to add details about the difference in build times!

Asset fingerprinting and the preload response header in 11ty

2025-09-02T00:00:00Z

This blog post will be building on a number of blog posts that I wrote earlier in the year. These were the posts:

Some insights from my earlier posts may carry over here, so check them out for overlap or a fuller view of my custom CSS pipeline for 11ty.

In this post, I’ll improve performance by adding the preload technique to my blog. First, let’s look at what it is.

Preload basics

In a standard web page load, once requested, the server sends over the HTML document as well as numerous response headers too. The HTML is progressively served to the browser, and it is only when the parser encounters the standard <link src="example.css" rel="stylesheet"> link that the browser requests the CSS file from the server. Therefore, best practice is to place this CSS link as close to the top of the <head> tag as you can. This ensures that the browser sees it quickly and thus starts downloading it as soon as it can. But what if you could give the browser a "hint" as to what is coming up in the document? This is where the Preload hint functionality comes in.

The Preload hint is essentially saying to the browser:

I know you're busy doing other things at the moment, but you should also know that you absolutly will be requiring this file soon in the page load. So stick it at the top of your list to download as soon as you can.

It's important to realise that this is only a "hint", not a mandatory instruction. The browser may choose to entirely ignore it, if for example it has already parsed and discovered the file you wish for it to preload. There are 2 ways in which you can implement a preload.

1. Link in the head

This is probably the easiest way to add a preload to a website. Stick it in the <head>:

<link rel="preload" href="style.css" as="style" />
<link rel="preload" href="main.js" as="script" />

2. Preload link header

This method ensures that the browser is told about what other resources to load along with the HTML document in the form of a response header from the server. The above "Link in the head" functionality looks like this as a response header:

Link: </style.css>; rel=preload; as=style
Link: </main.js>; rel=preload; as=script

Link: </style.css>; rel=preload; as=style, </main.js>; rel=preload; as=script

Both headers give the exact same functionality, it just comes down to readability. I don't believe the single line version give any performance advantages, especially when any form of header compression is applied, e.g. hpack for HTTP/2 or qpack for HTTP/3. But please do let me know if this assumption I'm making isn't true!

In both instances above, we are telling the browser to preload the page’s CSS and JavaScript because they will be required to render the page. You may notice that I phrased it as “will be required”. This is deliberate because it is far too easy to abuse the preload functionality. If you instruct the browser to preload everything, you will likely harm web performance instead of improving it. So only preload assets that are genuinely needed for the page to render. Otherwise, you risk wasting bandwidth on unnecessary resources and slowing down the rendering process. I know Firefox warns you in the DevTools console if an asset has been preloaded but not used during a certain time period, other browsers may do this as well. So always check your browser console for similar messages.

Preload and fingerprinting

There is a small added challenge when using asset fingerprinting with the preload functionality. Since the filename of the CSS or JavaScript changes completely whenever the file contents change, you cannot simply preload index.css or main.js. They will instead be renamed to something like index-362ccd3816.css or main-2fc0e9cad0.js. These are just example names, but the important point is that the file names are unpredictable and will change with each build, assuming the content of the files change. Since nobody wants to update a preload reference by hand every time a file changes, this is where a bit of 11ty scripting magic steps in to save the day.

The Code

In order to roll this functionality into my 11ty build, I created a helper file in my _helpers directory in the root of my blog. This is called header-generator.js. Imaginative name, huh! All functionality related to the header generation will be contained within this file.

It is then imported into my eleventy.config.js like so:

import { generatePreloadHeaders } from './_helpers/header-generator.js';

Now, I only want this code to run in production, and after the 11ty build completes, so I added the following later in the config:

if (IsProduction) {
	eleventyConfig.on('eleventy.after', generatePreloadHeaders);
}

Hopefully, this code is fairly self-explanatory. I'm hooking into the eleventy.after event, which is the point at which my CSS has been Brotli compressed and fingerprinted, and the Link Header is ready to be generated and added to my Cloudflare Pages _headers file (documentation here) before the _site is built. Below is the complete header-generator.js file I am using, with detailed comments to make it easier to follow:

// standard node library imports
import fs from 'fs';
import path from 'path';

// This script generates preload headers for fingerprinted CSS files in the _site/css directory
// and adds them to the global section of the _headers file (/*) in the _site directory.
// It prefers Brotli-compressed files (e.g. *.css.br rather than *.css) if available.
export function generatePreloadHeaders() {
	// Log the start of the process
	console.log('Generating preload headers for CSS files...');
	// This is my CSS directory for my blog
	const cssDir = path.join('./_site', 'css');

	// Check if CSS directory exists
	if (!fs.existsSync(cssDir)) {
		console.log('CSS directory not found, skipping header generation');
		return;
	}

	// Find fingerprinted CSS files (both .css and .css.br). We prefer .css.br if available.
	// Fingerprinted files match the pattern index-[hash].css or index-[hash].css.br
	const cssFiles = fs.readdirSync(cssDir)
		.filter(file => {
			// Match files like index-b9fcfe85ef.css.br or index-b9fcfe85ef.css
			return file.match(/^index-[a-f0-9]{10}\.css(\.br)?$/);
		});

	// Nothing found so exit
	if (cssFiles.length === 0) {
		console.log('No fingerprinted CSS files found, skipping header generation');
		return;
	}

	// Sort to prefer *.br files over *.css files
	// (compression is done via the zlib library in another helper file)
	// both .css and .css.br files exist in the same folder with the same file hash
	// The hash is generated from the unminified and uncompressed CSS file)
	cssFiles.sort((a, b) => {
		// If a is .br and b is not, a comes first
		if (a.endsWith('.br') && !b.endsWith('.br')) return -1;
		// If b is .br and a is not, b comes first
		if (!a.endsWith('.br') && b.endsWith('.br')) return 1;
		return 0;
	});

	// Take the first (preferably .br) file
	const cssFile = cssFiles[0];

	// Construct the path for the Link header
	const cssPath = `/css/${cssFile}`;

	console.log(`Found CSS file: ${cssFile}`);
	// Now we need to read the existing _headers file, add the preload header to the global section (/*),
	// and write it back
	try {
		// Read the source headers file
		const sourceHeadersPath = path.join('./public', '_headers');
		// Set our target headers file
		const targetHeadersPath = path.join('./_site', '_headers');

		// Check if source _headers file exists
		if (!fs.existsSync(sourceHeadersPath)) {
			console.log('Source _headers file not found');
			return;
		}

		// Read the existing headers content
		let headersContent = fs.readFileSync(sourceHeadersPath, 'utf8');

		// Create the preload header
		// Note: 'nopush' prevents Cloudflare from doing HTTP/2 server push
		const preloadHeader = `  Link: <${cssPath}>; rel=preload; as=style; nopush`;

		// Find the global /* rule and add the preload header to it
		// Look for the line that just contains "/*" which is the global section
		const lines = headersContent.split('\n');
		let globalSectionIndex = -1;
		let nextSectionIndex = -1;

		// Find the global section (line that starts with just "/*")
		for (let i = 0; i < lines.length; i++) {
			if (lines[i].trim() === '/*') {
				globalSectionIndex = i;
				break;
			}
		}

		// Find the next section (line that starts with a path)
		if (globalSectionIndex !== -1) {
			for (let i = globalSectionIndex + 1; i < lines.length; i++) {
				if (lines[i].trim() !== '' && !lines[i].startsWith('  ')) {
					nextSectionIndex = i;
					break;
				}
			}
		}
		// If we found the global section, proceed to add or update the Link header
		if (globalSectionIndex !== -1) {
			// Check if a Link header already exists in the global section
			let linkHeaderExists = false;
			// Iterate through the lines in the global section
			for (let i = globalSectionIndex + 1; i < (nextSectionIndex === -1 ? lines.length : nextSectionIndex); i++) {
				// Check for the existance of the Link header
				if (lines[i].includes('Link:')) {
					// Replace existing Link header
					lines[i] = preloadHeader;
					// The Link header exists and has been updated
					linkHeaderExists = true;
					break;
				}
			}

			// If no Link header exists, add one
			if (!linkHeaderExists) {
				// Find the last header line in the global section
				let lastHeaderIndex = globalSectionIndex;
				// Iterate until the next section or end of file
				for (let i = globalSectionIndex + 1; i < (nextSectionIndex === -1 ? lines.length : nextSectionIndex); i++) {
					if (lines[i].trim() !== '' && lines[i].startsWith('  ')) {
						lastHeaderIndex = i;
					}
				}
				// Insert the Link header after the last header
				lines.splice(lastHeaderIndex + 1, 0, preloadHeader);
			}
			// rejoin the modified _headers file
			headersContent = lines.join('\n');
		} else {
			console.log('Could not find global section in _headers file');
			return;
		}

		// Write the updated headers to the _site directory before deployment to Cloudflare pages
		fs.writeFileSync(targetHeadersPath, headersContent);

		console.log(`Generated preload header: Link: <${cssPath}>; rel=preload; as=style; nopush`);

	} catch (error) {
		console.error('Error generating preload headers:', error);
	}
}

For a cleaner version without comments, I’ve uploaded the code to a Gist. Find the code Gist here (opens in new tab).

The Cloudflare `_headers` file

This setup is currently working really well, the only minor "issue" that doesn't sit right with me presently is the fact that the Link header sits on the global header path (/*) in the _headers file. This means the Link header is added to all assets served from my blog.

As far as I know, this shouldn't cause any issues, as browsers will just ignore it if on a file type that doesn't support it. But I would like to rectify this in the future.

In my testing with the Cloudflare _headers file, once a header is set in Cloudflare Pages it cannot be removed or overwritten. The only "fixes" I’ve found for this are:

Use a response header transform rule in the Cloudflare dashboard to remove the Link header from all other file types served except CSS.
Look into a Cloudflare Workers solution to examine the server responses at "the edge" and remove them that way.

I will eventually move forward with option 1 as it looks to be the most straight-forward way to do it.

I've mentioned this "minor issue" in the blog post, simply to highlight the fact about not being able to remove headers using the _headers file once they have been set. If anyone knows how to do this using only the _headers file, please let me know. I’d love to learn how.

Summary

All this is now live on this very blog, so using my custom CSS pipeline with 11ty, I now have the following happening before deployment to live:

CSS minified and Brotli compressed to level 11 (highest).
CSS Asset fingerprinting to allow for long life cache-control headers, including immutable.
Preloading of the CSS file to reduce the discovery time and improve page performance.

On a side note: This is probably one of the fastest (and shortest) blog posts I've written in a while! I knew I could do it! 🤣 I hope you found it as enjoyable to read as I did to write. As always, feedback and post corrections are welcome. Spot anything wrong? Please do let me know.

Post changelog:

02/09/25: Initial post published.

Hack to the Future - Frontend

2025-08-26T00:00:00Z

Hack to the Future - Frontend

1. Introduction

Context

So over the last few months at work, I've been conducting interviews to hire Frontend Developers for a number of new projects we have in the pipeline. It was only when looking at CV's that it struck me, a lot of these candidates weren't even born when I first started in my Web Development career! So I thought maybe developers getting into a Frontend Developer career today, may want to learn a bit about what it was like when I first started (that sentence just makes me feel old! 👴)

Looking back at “legacy” practices

Why would we want to look back on legacy best practices on the web? Other than the obvious academic and for general interest reasons?

Studying past best practices and legacy systems is crucial for understanding the evolution of technology and making informed decisions today. By examining the problems old practices were designed to solve, we gain a deeper appreciation for current best practices and avoid repeating past mistakes. As the philosopher George Santayana once said:

Those who cannot remember the past are condemned to repeat it.

This historical perspective also reveals enduring principles like progressive enhancement, which remains vital for creating accessible and resilient systems on the web.

Lessons we can apply today

For developers, understanding past methodologies is essential for properly maintaining and modernising existing systems in the future without causing critical failures. This historical knowledge will ultimately help them navigate the complexities of older codebases, to ensure they make informed decisions about how to update or replace components. Above all, reflecting on the past can help us come up with creative new ideas and prevent us from blindly following new trends. This perspective also provides a comprehensive view of how the web has evolved, grounding our current practices in a deeper understanding of the technology's history.

This process of building on past knowledge is a fundamental aspect of human progress. Just as civilizations learn from historical events to avoid repeating mistakes, developers can learn from the successes and failures of past technological eras. It's how humanity has always evolved. By building upon the accumulated wisdom and experience of those who came before us. By studying the mistakes and triumphs of the past, we improve our own work and contribute to the continuous cycle of innovation and learning that drives our entire industry forward.

2. Setting the Time Circuits to the late 90s

My first website build

In 1998, while working toward my GCSEs, I became interested in art and design, this was partly thanks to having an art teacher as my form tutor throughout secondary school. That influence, combined with the opportunity to take a double art GCSE for the same effort as a single GCSE, made the choice a pretty easy one! GCSE Art, here I come!

At the same time, I was already immersed in the emerging world of the internet, spending many hours online discovering a passion for many areas of computing and online gaming thanks to QuakeWorld Team Fortress, despite the frustration it caused at home by tying up the phone line all hours of the day, oh how I loved my US Robotics 56K modem, with its 120-150 ping! Integrated Services Digital Network (ISDN) or any form of broadband was still many years away for most people!

I was never exactly blessed with traditional artistic talent, painting, drawing, all of those art forms just wasn’t my thing. But I spotted an opportunity to combine my love of technology with the art curriculum. Back then, there were only about 2.4 million websites in existence worldwide. Most businesses and schools (including mine), were firmly offline. So, I proposed building a website for my final art project. To my surprise, my art teacher was absolutely thrilled with the idea. It turned out to be a first for the school and, as I later discovered, a first for the entire exam board too. Shock horror: I was ahead of the curve once. The curve has been safely ahead of me ever since.

I ended up creating a website for a fake record label, complete with a dreadful album cover, fictional artist, and made-up discography. Honestly, I wish I still had it! It was gloriously awful! I don’t recall much, but I remember the site used a <frameset> with three <frame> elements. The top frame displayed the logo, the left frame held the navigation menu, and the main frame was used for the page content.

The logo, by the way, was crafted in a program called 3D Text Studio (or something similar to that) that churned out spectacularly cheesy animated text like this! From a web performance perspective, that single GIF exceeded 2 MB. On a 56K modem, which was the standard connection for most users of the web at the time, that translates to a 6-minute loading time for just that GIF! Fortunately, it was never hosted online and was presented to the examiners directly from my local machine.

Long story short… the examiners loved my little website and I got a double A* Art GCSE for my effort!

So what's all this preamble leading too? Well, this is just a long-winded way to tell you (again) that I'm old… 😭

The late 90s web landscape

There have been some things I've noticed while questioning candidates in interviews recently, many candidates don't have the faintest idea of some old methodologies used in the world of Frontend, especially during the "unstable" periods of the web like the late 90s and early 00s:

Being a Frontend Developer in the late 90s was both fun in terms of innovation, but also exceedingly stressful due to the instability of the web platform! A prime example being cross-browser development. What worked in Netscape, often looked very broken in Internet Explorer (and vice versa)! And if you had clients who were looking for "pixel perfect" designs across all browsers, you were in for a bad time!

Throughout this period, a plethora of methodologies, tools, and workarounds were developed to address deficiencies in the web platform. And that’s what the rest of this post will delve into. Buckle up folks, we are about to time travel to an era when the internet started with the screeching of dial-up noises and I still had brown hair!

3. The Early Web – Layout and Design Practices

Photoshop PSDs as the “single source of truth”

Using Adobe Photoshop Documents (PSD) as a single source of design truth was a very common practice in the early days of web design. This was particularly common when design and development teams were siloed. A designer would create a PSD file that was intended to be precisely what the website would look like in the browser.

Issues

There were no considerations made for page structure, behaviour, and interactions. These fixed layout PSD's encouraged bad practices like:

Fixed page dimensions e.g. 1024px x 768px as a static canvas.
1:1 mapping of Photoshop file to web page, which was rarely achievable, especially given cross-browser inconsistencies with page rendering.
Lack of fluid or responsive design. I realise responsive design wasn't "a thing" at this time, but could it have been adopted sooner if fixed-width PSD workflows hadn't ever taken hold?
The technique was more suited to static layouts, like print design, rather than web design.
There were issues tracking interaction states like anchors with hover, active, disabled, and focus.
Dynamic content was difficult to visualise (e.g., the rendering of different lengths of text in the browser).
Poor accessibility adaptations, (e.g., increased font sizes, high-contrast modes weren’t considered in the design files).

The only way to solve many of these issues would be to create multiple PSD's to hold all these different design assumptions. And in doing so, file management and design revisions would quickly become impractical and prone to being incomplete or inconsistent.

Broken team collaboration

The use of PSD's as the single source of truth broke how teams could collaborate and innovate. This was because:

Developers would often have to interpret or translate the PSD design manually without the help of designers (e.g. due to siloed teams and strict job roles).
Changes in the design required round-trips to designers, rather than being evolved collaboratively in code.
Small team bottlenecks were common e.g. all design or development decisions needed to go through individuals rather than a whole team.
Files became outdated rapidly leading to teams working on outdated designs without realising it.
Designers often came up with designs that simply couldn't be built with the web technologies that existed at the time, especially when their designs were expected to work across different browsers.

Modern Alternatives

I'd like to think that designers using Photoshop for modern web design is a thing of the past, given the vast number of tools and techniques that are way more suited to the job than Photoshop ever was. Modern teams typically use:

Design tokens and internal component libraries as the "single source of truth".
Figma or similar tools with structured, token-aware components.
Living style guides and code-driven prototypes (e.g., Storybook).
Clear handoffs between teams using tools like zeroheight, or integrated design-to-dev platforms.

The advantage of using these modern collaboration tools enables design and development teams to share the same language and source of truth, rooted in reusable, well-tested, and accessible components.

Photoshop PSDs Summary

In the early days of my frontend career, slicing PSDs was second nature, but that workflow is now obsolete. Using Photoshop as a "single source of truth" leads to siloed teams, rigid layouts, and poor collaboration. It ignores responsiveness, accessibility, and the realities of modern web development. Today, tools like Figma, design systems, and component libraries enable faster, more inclusive, and collaborative workflows. If you’re still building from PSDs, it’s time to move on! As the web has evolved, it is imperative that we all do the same.

Frame-Based Layouts

The Frame-based layouts were introduced into browsers to solve a specific set of problems. These were:

To allow static content like navigation menus to remain in place while only the main content of the page gets updated on navigation.
To Reduce the amount of data transferred over the network, since only one part of the page would need to be loaded. This was important at the time as remember in the late 1990s and early 2000s, broadband for most people simply wasn't available. If you were very lucky (and had the money), you'd be able to get an Integrated Services Digital Network (ISDN) line installed in your home, but it was mostly online businesses that had the money (and justification) for this type of connection, even ISDN wasn’t particularly quick. Adjusted for inflation you'd be looking at £60 to £80 per month for a 0.128 Mbps connection!
To simulate a more app-like experience before JavaScript (JS) and CSS became more standardised and mature.

Example

For those curious here's an simple example of an HTML page using frames:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Frameset//EN" "http://www.w3.org/TR/html4/frameset.dtd">
<html>
	<head>
		  <title>Simple Frame Example</title>
	</head>
	<!-- Note no Body element: 2 vertical columns -->
	<!-- 30% width the menu.html document -->
	<!-- 70% width for the main content of the page -->
	<frameset cols="30%,70%">
		  <frame src="menu.html" name="menuFrame">
		  <frame src="content.html" name="contentFrame">
	</frameset>
</html>

Notes:

To use <frame> and <frameset> you needed to use a specific HTML 4.01 Frameset DOCTYPE, in the index.html file.
In my example, for a single HTML page you'd have to maintain 3 HTML files (index.html, menu.html, and content.html).
Each frame was like a mini browser window that loaded its own HTML document.

Problems

Unfortunately, there were a number of major issues with Frame-Based Layouts:

Terrible user experience: the use and navigation of frames was confusing for users, since you effectively had multiple browser panes in a single page. The URL bar would often remain static even as the content of the page changed.
Poor Accessibility: Screen readers and other assistive technology struggled to navigate frames, making it incredibly difficult for users with disabilities to understand the page content and overall page structure.
Limited Search Engine Optimisation (SEO) compatibility: Even Search engines of the day struggled to understand the index pages built within frames. This lead to poor visibility in search results, as crawlers frequently failed to understand the relationship between the different frames.
Navigation and Browser Compatibility: Because the back and forward buttons did not consistently produce the desired results, frames disrupted the navigation history, making it difficult for users to find their way around. The fact that different browser vendors weren't aligned with how frames should work in browsers lead to cross-browser issues too.
Bad for security: Frames allowed for security risks like clickjacking. This is where an attacker gets a user to interact with a page that contains malicious content without the user even realising. Modern browsers now include protections to stop these types of security issues.

Modern Alternatives

Modern CSS Layouts: Flexbox and Grid allow for responsive layouts without compromising navigation, accessibility, and SEO.
Single Page Applications (SPAs): Frameworks like React, Angular, and Vue allow developers to load page content dynamically without the need for full-page reloads. Be careful though, these libraries come with their own inherent issues if not used correctly!
Server-Side Rendering and Partial Updates: techniques like server-side includes, AJAX, or component-based rendering to update portions of a page efficiently.

Frame-Based Summary

As mentioned in the introduction at the start of this post, my first website was built using frames! I sincerely hope you never have to maintain a frame-based website! But given the enormity of the internet, it is almost certain websites exist somewhere out there, having been untouched for decades! If you do come across one remember to take a quick peek at the source code, it's like looking back in time! They once served a purpose in the early days of the web but are now considered obsolete. Their usage introduced more problems than they solved, and have been replaced with techniques that are more performant, accessible, and maintainable. Any modern website should be using semantic HTML, CSS-based layouts, and progressive enhancement.

Table-Based Layouts

In the late 1990s and early 2000s table-based layouts were a common technique for building a web page structure: A simple example of what this would look like is below:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
  <title>Table Layout Example</title>
  <!- Very simple CSS to modify the key elements in the table-based layout -->
  <style>
    body {
      font-family: Arial, sans-serif;
    }
    td {
      padding: 10px;
      border: 1px solid #ccc;
    }
    .header {
      background-color: #f2f2f2;
      text-align: center;
      font-weight: bold;
    }
    .nav {
      background-color: #e0e0e0;
      width: 200px;
      vertical-align: top;
    }
    .content {
      background-color: #ffffff;
      vertical-align: top;
    }
    .footer {
      background-color: #f2f2f2;
      text-align: center;
      font-size: 0.9em;
    }
  </style>
</head>
<body>

<table width="100%" cellspacing="0" cellpadding="0">
  <!-- Header -->
  <tr>
    <td colspan="2" class="header">
      My Table-Based Web Page
    </td>
  </tr>

  <!-- Body -->
  <tr>
    <!-- Navigation -->
    <td class="nav">
      <ul>
        <li><a href="#">Home</a></li>
        <li><a href="#">About</a></li>
        <li><a href="#">Contact</a></li>
      </ul>
    </td>

    <!-- Main Content -->
    <td class="content">
      <h2>Welcome</h2>
      <p>This layout uses an HTML table for structure, which was common before CSS-based layouts became standard.</p>
    </td>
  </tr>

  <!-- Footer -->
  <tr>
    <td colspan="2" class="footer">
      &copy; 2025 Example Company
    </td>
  </tr>
</table>

</body>
</html>

Why was it used?

At the time CSS and layout techniques were inconsistent and unstable across browsers. Developers looking for stability in cross-browser rendering turned to tables in order to do this. At the time, tables offered:

Predictable cross-browser rendering
Control over alignment, spacing, and sizing
Ability to nest elements in a grid-like structure

It was very common to see nested tables and transparent "spacer GIFs" in invisible table cells to control these layouts more precisely. You'd often find logo's, sidebars, navigations, footer, and content areas all laid out within a deeply nested HTML table in order to achieve the layout and design that was required.

Why was it so bad?

The first and hopefully most obvious point is that the <table></table> element was intended for the display of tabular data. The fact that it was used as a workaround for the lack of standardised layout techniques, shows the ingenuity of developers at the time.

Unfortunately, the use of tables for layout came with many considerable downsides, these included:

Semantics: As mentioned, tables should represent structured data, not layout. Misusing them confuses assistive technologies and harms accessibility.
Maintainability: Table-based layouts are challenging to read, modify, or scale. Small changes often require restructuring entire layouts.
Responsiveness: They are rigid and not suited to fluid or responsive design, that was to come a number of years later.
Performance: They delay rendering because browsers need to calculate the entire table layout before painting it to the page.

Is the technique still used?

There are some areas where table-based layouts may still be seen:

Legacy code bases that desperately need to be refactored, I can imagine there are many internal systems across the world where table-based layouts are still used. I’d imagine the conversation about modernising goes something like this… "If it still works, why change it?". Very short-sighted I know!
Table-based layouts are still widely used in emails due to the very limited support for CSS in email clients. It's not always the lack of support, it's the fact that many clients simply strip out any CSS in the process of rendering the email HTML.
- To give you an example of how bad it still is, from Outlook 2007+, Microsoft switched to Microsoft Word as the HTML rendering engine! And it's still in use today with Outlook 365! I did my fair share of HTML emails as a Junior Frontend Developer, the internationalised versions were the worst! Using the same table-based layouts for 19+ languages is never going to work well, especially with languages like German with their huge word length! Sorry… rant over!
They are often still used in PDF generation tools e.g. data-driven print views: invoices etc.

Modern alternatives

Modern CSS offers clean, semantic, and powerful layout tools, including:

Flexbox: One-dimensional layouts (ideal for nav bars, toolbars, etc.)
CSS Grid: Two-dimensional layouts (ideal for full-page layout and complex structures)
Media Queries: Enable responsiveness across devices
Container Queries (still an emerging technology): Context-aware layout changes.

Table-Based Summary

Table-based layouts are a throwback to a bygone era, thankfully! The years of building HTML emails has scared me for life! As they were developed during a period in which CSS was inadequate for the task. Developers had to get creative to wrestle with browser quirks, and tables were the go-to workaround. Thankfully, these days, we’ve moved on to semantic HTML and proper CSS that actually does what we need (for webpages anyway). It’s cleaner, more flexible, and maintainable, and way better for accessibility.

Quirks Mode Layouts

This topic is covered in more detail later in the blog post, but I’ll briefly mention it here for completeness.

It's important to realise that Quirks Mode Layouts weren't only limited to Internet Explorer (IE). It originated with Internet Explorer, but it was not exclusive to IE. Not only that, it later became a cross-browser convention in order to preserve the compatibility with many web pages on the internet. As that's the primary rule to consider when rolling out any new technology changes on the web. Whatever you do, "don't break the web!".

For example, if a vendor released a new browser feature that wasn't backwards compatible with earlier versions of web pages, then you have a major issue as you've just broken the web! I talk about XHTML 2.0 later in the post, as it is a prime example of a proposed technology that would have broken the web. This backwards compatibility was the sole purpose of Quirks mode. It gave modern browsers the ability to switch between:

Quirks Mode: Mimic pre-standards behaviour. Used for old, non-compliant pages.
Standards Mode: Adheres to modern web specifications (W3C and WHATWG standards).
Almost Standards Mode: The same as Standards mode only with one exception, table cell line-height rendering. This was to preserve layouts that used inline images inside HTML tables.

How were layouts triggered?

The browser decided which layout mode to use from the list above purely from the DOCTYPE used on the page. For example:

Trigger Quirks mode

This DOCTYPE will trigger Quirks mode layout:

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">

It looks valid, but it is missing the system identifier (URL) therefore it is a malformed DOCTYPE so Quirks Mode is triggered. A valid DOCTYPE is given below for comparison:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">

That missing URL in the DOCTYPE is vital. Quirks Mode would also be activated if a page did not have a DOCTYPE or was not identical to the valid DOCTYPE given above in any way. IE even had a really nasty habit of triggering Quirks Mode if any character was output in the page source before the DOCTYPE. This included invisible characters and new lines and line returns too! As you can imagine, it made debugging issues an absolute nightmare!

Almost Standards Mode

The following DOCTYPE's will trigger Almost Standards Mode:

HTML 4.01 Transitional (with full system identifier):

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">

HTML 4.01 Frameset (with full system identifier):

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Frameset//EN" "http://www.w3.org/TR/html4/frameset.dtd">

XHTML 1.0 Transitional:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

XHTML 1.0 Frameset

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Frameset//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-frameset.dtd">

Standards Mode

And lastly and most importantly for modern web development. This is the DOCTYPE you should be using to trigger standards mode in all modern browsers:

<!DOCTYPE html>

This simplified DOCTYPE was brought in as part of the HTML5 Specification after 6 years of standardisation (2008–2014).

Why was this version created?

As outlined in all the examples above, previous DOCTYPE versions were:

Long
Error-prone
Required both a public and a system identifier
Affected rendering modes (Quirks, Almost Standards, Standards)

In order to solve these issues this the new DOCTYPE:

does not reference a Document Type Definition (DTD) as (HTML5 no longer relies on SGML-based validation).
only has a single purpose: to trigger Standards Mode in all modern browsers.

Quirks Mode Summary

As we have discussed above, Quirks Mode wasn't an IE exclusive layout mode. It was introduced into all browsers in order to "not break the web". To ensure your website uses Standards Mode, use:

<!DOCTYPE html>

And remember it must be the first characters in the source code on the page!

Iframe Embeds for Layouts or Content

If you've already read the Frame-based Layouts section above, then this section will be very similar. Although both are now considered legacy techniques, they come with distinct differences.

Frameset

As I discussed earlier here's example <frameset> code:

<!-- column 1 25% width / column 2 75% width -->
<frameset cols="25%,75%">
	<frame src="nav.html">
	<frame src="main.html">
</frameset>

The <frameset> tag completely replaced the <body> tag and allowed developers to split the browser window into multiple, scrollable, resizable sections.
Each section (<frame>) loaded a separate HTML document (as seen in the code above).
This technique was intended to use them as a layout structure. e.g. different parts of the User interface (UI) came from different HTML documents.
Navigation in one frame would control the content in another frame.

Inline frames (Iframes)

These were introduced later in the HTML 4.01 Transitional specification. Example <iframe> code is found below:

<body>
	<iframe src="content.html"></iframe>
</body>

You will immediately notice the difference, using an <iframe> doesn't replace the <body> tag, it embeds an external HTML page into the original page.

The usage of iframes was to embed content into other pages, not structure the whole layout.
iframes were used to load external or isolated content within a single, self-contained HTML page.

User Experience

Both methods came with issues, but iframes were slightly better but still caused significant problems:

Embeds could be styled and resized, but remained isolated from the parent.
Navigation, SEO, and accessibility were still severely impacted when misused.

Standards and Browser Support

Framesets were deprecated and completely removed from the HTML5 specification, modern browsers either no longer support them or support is very limited without the correct DOCTYPE.

Iframes are still a part of the HTML5 specification, but they come with limited use cases, including:

Secure sandboxing
Third-party embeds

In fact, the performance of iframe embeds has recently been improved by browsers vendors adding support for loading="lazy" on iframes thus allowing the browser to delay loading until the iframe is in the viewport.

Security

Thankfully, iframes are more secure on the modern web thanks to attributes like:

Although the use of iframes still needs careful consideration for content integrity, cross-domain policies, and maintainability. They can still introduce security vulnerabilities such as:

Clickjacking
Cross-site scripting (XSS) exploits.
Cross-origin resource sharing (CORS) exploits.

Can I still use them?

Yes, iframes are still a part of the HTML5 specification, so you can still use them, but you should try to only use them in specific scenarios where required. For example, when you need to embed third-party content into a website. This is often seen in banking on the modern web, when you are paying for something online. The final step of the transaction frequently loads an iframe from your bank asking to verify your purchase, usually via some form of Multi-factor Authentication (MFA) e.g. SMS, banking app approval on your phone. While iframe embeds aren't specifically required for PCI compliance. Many online merchants tend to implement them along with 3D Secure authentication to help reduce PCI scope and exposure.

Modern alternatives

As mentioned in the Frame-Based Layouts section, there are a number of modern alternatives rather than using iframes for page layout, modularisation, and code isolation. These include:

Modern CSS Layouts: Flexbox and Grid allow for responsive layouts without compromising navigation, accessibility, and SEO.
Single Page Applications (SPAs): Frameworks like React, Angular, and Vue allow developers to load page content dynamically without the need for full-page reloads. Be careful though, these libraries come with their own inherent issues if not used correctly!
Server-Side Rendering and Partial Updates: techniques like server-side includes, AJAX, or component-based rendering to update portions of a page efficiently.

Iframe Embeds Summary

Using iframes for modularisation and isolation is now seen as a legacy approach and should generally be avoided on the modern web. That said, they still have a place for things like third-party embeds. I remember the early days of Facebook marketing, iframes were everywhere and an absolute nightmare to work with, especially when trying to get the sizing right within their UI. Urghh!

Pixel-Perfect Design

Pixel-perfect design is a now outdated design approach that was once considered the gold-standard in frontend development. The presented UI on the web page was intended to be a pixel-perfect replica of the design mockups.… If that sounds like a ridiculous and totally unworkable strategy, then I can confirm, yes it was!

What the technique means in practice

Strict visual fidelity: Developers were expected to reproduce every element of a design using the exact measurements, colours and fonts given in a static design file like a Photoshop Document file (PSD).
Close alignment with design tools: The priority while using this technique was to match the layout in the design document, no matter what. Inconsistencies across various browsers, not to mention differing screen sizes and user preferences, frequently made this an impossible task.
Used in fixed-resolution environments: The technique worked reasonably well in the era of desktop-only websites with very few fixed screen sizes (e.g. 800×600, 1024×768).

Why is it considered legacy?

Responsive design is now standard
- Modern devices now range from the size of watches, all the way up to ultra-wide monitors and TV's and everything in between. Pixel perfect design fails on both smaller and larger size screens.
Accessibility and user preferences
- Modern Accessibility-first design requires designs to be flexible in terms of layout and scaling. For example, WCAG 2.2 Success Criterion 1.4.4 expects text to be scaled up to 200% without loss of content or functionality. This simply can't be done when using the pixel-perfect design technique.
Performance and Maintainability:
- It encourages the use of hardcoded CSS rather than using scalable design systems and components.
Modern Design Tools and Systems
- Tools like Figma and Sketch now exist with scalable units, that can create responsive mockups rather than static visuals.
Design Intent Over Exact Replication
- Modern web design focuses on intent and consistency across contexts, prioritising usability and accessibility over pixel-perfect replication by leveraging newly introduced responsive browser technologies.

Can I still use it?

I hate to be the bearer of bad news, but no, you can't. It simply isn't compatible with the modern web, for all the reasons I've given above. You are better off understanding what the technique was trying to achieve and focus on modern-day alternatives.

Modern Alternatives

Use of Design tokens for spacing, colour and typography.
Modern browser layouts like CSS Grid and Flexbox.
Component libraries and Design Systems that abstract repeated patterns.
Focus on the user content, not the dimensions of the UI viewing it.

Pixel-Perfect Summary

Pixel-perfect design was always a terrible design technique that set everyone up to fail with the expectation of pixel accurate designs across browsers and devices. It served its purpose when the web was a lot simpler than it is today. A pixel-perfect design on today's web is a clear indication of an outdated aesthetic, urgently requiring a modern overhaul.

Fixed Pixel Layouts

There's a reason why I've mentioned Fixed Pixel layouts directly after Pixel-Perfect Design. They are related, but have differing concepts. They both emerged in an early era of web design, but they serve different purposes.

Pixel-perfect Design

As you will have just read above, pixel-perfect design emphasises the precise implementation of a design on a website. The output of the website should be identical to the design, down to every pixel. For this reason, it is a philosophy, not a layout strategy.

Fixed Pixel Layouts

Fixed pixel layouts are a layout strategy where all widths, heights, and positions are defined using fixed pixel values (e.g. width: 1024px). This means it the page layout doesn't adapt to different screen sizes or resolutions. They are rigid and non-responsive, and are optimised for a single resolution or screen size. Because of this Fixed Pixel Layouts break on small screen sizes (like mobiles), or large displays. They are very much associated with older websites and legacy intranet tools. If you look at early websites on the web they would often say "Best viewed with Internet Explorer (or Netscape Navigator) at 800×600 (or 1024×768)". These are prime examples of where Fixed Pixel layouts were used.

Fixed Pixel Summary

Fixed pixel layouts and pixel-perfect design are outdated. They are relics from a time when monitor sizes were uniform and browser zooming was uncommon. Pixel-perfect demands you match the design mockup exactly, which usually ends in fragile CSS and breaks the moment someone dares to increase their font size or use a high contrast mode. Fixed pixel layouts take it even further by hard-coding dimensions, making sites fall apart on anything that isn’t a desktop from 2010. Adding to their drawbacks, neither option integrates well with modern accessibility standards, responsive design principles, or current web usage patterns.

CSS Floats for Layout

CSS floats were my go-to layout technique for many years, simply because there really wasn't a viable cross-browser alternative (other than Tables, which I covered earlier in the post). Floats were originally intended to be used for wrapping text around images on a page. They weren't intended to be used for full-page layouts. At the time, designers and developers did this as a creative way to make certain layouts, not because that was the design goal of the CSS specifications.

Common issues

Float-based layouts came with a major issue: fragile designs often required “clearfix” hacks to stop float containers from collapsing.

Example clearfix hacks

Here are a few example hacks that ensured the float container would wrap around the floated elements:

Modern (Recommended):

.clearfix::after {
	content: "";
	display: table;
	clear: both;
}

Usage:

<div class="clearfix container">
	<div style="float: left;">Left</div>
	<div style="float: right;">Right</div>
</div>

Overflow Hidden (Quick Fix):

.container {
	overflow: hidden;
}

This was my personal go to for "clearfix" as it was so simple to add to the CSS. A nasty downside of this method was it would hide any content that needed to overflow outside the float container, so its usage really depended on the design requirements.

Float the Container Itself (Not Recommended):

.container {
	float: left;
	width: 100%;
}

I don't remember really using this method simply because it could interfere with other layout elements on a page and could create unexpected layout shifts during page rendering.

Using display: flow-root (Modern, Clean Alternative):

.container {
	display: flow-root;
}

This is far too modern for me, it didn't exist when I was still building UI's regularly. As, it has been stable in major browsers since January 2020. The advantage of this method is no pseudo-elements or hacks required. This is recommended for use if you are modernising code and want the cleanest approach without adding extra markup. Other float-based issues include:

Poor readability: Additional markup was required just for the container clearfix hacks. This also impacted maintainability too.
Inconsistent behaviour: Floats tended to behave differently depending on the browser in use, although not as problematic as it was, it still requires more testing to ensure cross-browser UI consistency.
Stacking issues: Aligning text, or centring horizontally and vertically, is non-trivial.

Modern Alternatives

As with most of the topics in this section of the post, there are a list of modern alternatives available for float-based layouts.

Flexbox: One-dimensional layouts (ideal for nav bars, toolbars, etc).
CSS Grid: Two-dimensional layouts (ideal for full-page layout and complex structures).
Media Queries: Enable responsiveness across devices
Container Queries (still an emerging technology): Context-aware layout changes.

Performance and Maintainability

Modern layout techniques:

Lead to cleaner and more maintainable code.
Reduce reliance on utility classes for clearing float-specific bugs.
Enhance responsiveness and accessibility by ensuring that the HTML code's structure is more predictable.

Legacy Support Considerations

Floats still appear in legacy code, so understanding them remains important for maintenance and modernisation. However, they should be avoided in greenfield projects unless strictly necessary (I'm open to suggestions if anyone has an idea as to what this necessity might be).

CSS Floats Summary

Using CSS floats for layout is now considered an anti-pattern. While historically important, floats have been replaced with modern layout systems like Flexbox and Grid, which offer cleaner, more maintainable, and more powerful solutions. In the future as the web platform evolves, newer layout techniques such as CSS Subgrid, Container Queries, and Anchor Positioning are also progressing through standardisation and will further improve layout flexibility. Avoiding floats is a key best practice when building or modernising frontend architecture.

Faux Columns

I'm not sure why the French word "Faux" was chosen for this technique, rather than just "false" or "fake", maybe to make it sound more appealing? Or more complex than it actually is? The term works though, as in English, it is used to describe something made to look like something else, which is precisely what this technique did.

What is it?

In the early days of CSS there was no reliable way to make two or more columns stretch to the same height when the content length in the columns varied. This was because floats or inline-block elements don't align their heights, so developers looked for a workaround.

How it worked

It was a clever workaround actually, it typically worked like this:

A background image was applied to the parent container of the floated columns (that designers wanted to look to be equal in height)
This background image was often a vertical gradient or a solid block of colour. This image was then repeated (repeat-y) vertically down the parent container, giving the illusion ("fake") that the columns were of equal height.
As each of the inner floated elements extended due to varying content within, the container background image extended with it. Very clever, huh!

The CSS for this "workaround" was as simple as this:

/* container with the background repeat vertically */
.container {
	background: url('faux-columns.png') repeat-y;
}
/* Note the width of this column, it is important */
.left-column {
	float: left;
	width: 200px;
}
/* Use an identical margin to the width, to "push" the right column into place next to the left column */
.right-column {
	margin-left: 200px;
}

The background image of the container would have 2 sections, one colour for the left column and another for the right column, thus tricking a users eyes into seeing the columns having equal heights!

Check out the Faux Columns A List Apart (ALA) article by Dan Cederholm from January 2004, if you are still unsure how it works, he explained it a lot better than I have!

Limitations

Unfortunately, as with all of these early CSS workarounds, there were limitations. The faux columns technique was no different. These limitations included:

Not adaptable: it required the background image to perfectly match the layout dimensions of the container, and it's inner "columns".
Unresponsive: Exactly what the word says, this only worked for fixed layouts, fluid of more dynamic layouts simply broke the illusion.
Maintenance: The technique was difficult to maintain, since any layout change required editing the CSS, background image, or both.
Poor semantics: Yes, it solved the visual presentation problem, but the underlying code wasn't semantic, the additional <div>'s just for layout purposes, inherently held no semantic meaning.

Modern Alternatives

As you would expect, there are modern alternatives that make this layout trivial:

Flexbox

With Flexbox, it really is this simple:

.container {
  display: flex;
}
.left-column, .right-column {
  flex: 1;
}

CSS Grid

In CSS grid, it's even easier as all child elements in a row are explicitly defined and align by default, no extra CSS required:

.container {
	display: grid;
	grid-template-columns: repeat(2, 1fr); /* 1fr = 1 fraction unit */
}

Can I still use it?

Errr, silly question, no not at all! Look how effortless the modern alternatives above are! Imagine how good it feels to rip out the legacy faux column code from a legacy codebase, and replace it with 1 or 2 lines of CSS!

Faux Columns Summary

The Faux Columns technique was one of those clever hacks we leaned on back when CSS didn’t give designers and developers much to work with. It did the job, but it was fragile and fiddly, and you were always one layout change away from breaking it. These days, it’s more of a historical curiosity. Flexbox and Grid have long since made it obsolete, and with newer tools like Subgrid and Container Queries coming through the standards process, we’ve moved on from trickery to browser tools that are actually built for layout.

Zoom Layouts (using CSS zoom for responsiveness)

Back when responsive design was first emerging, a technique called Zoom Layouts emerged in order to scale whole elements within a UI. This technique emerged because responsive CSS layout techniques were very limited.

Example

An simple example of this is given below:

.container {
	zoom: 0.8;
}

This CSS is easy to understand, it simply scales the entire .container by 80%.

When was it useful?

This technique was useful when you needed to shrink or enlarge an entire layout without refactoring a fixed-width design. It also came in handy when working with legacy layouts that could not adapt with fluid widths or media queries. Lastly, it was used as a workaround before widespread browser support for the transform: scale() CSS property or relative units like rem, em, %, vw, and vh.

Why is it outdated?

There are a number of reasons as to why the Zoom Layout technique is now outdated. These include:

zoom is non-standard and inconsistent: The zoom property isn't a part of any official CSS specification. It is a proprietary feature initially supported by Internet Explorer, then later Chromium-based browsers. Interestingly, Firefox and Safari have never supported the zoom property, making cross-browser layouts using the technique very tricky.
Causes Accessibility issues: zoom does affect the layout scaling, but it doesn't interact well with user-initiated zoom or accessibility scaling preferences. Thus, using this technique can create barriers for users with visual impalements that rely on native browser zooming or OS-level zooming tools.
Breaks layout semantics: zoomed elements don't always reflow correctly, for example text can reflow outside its container, images can become blurry, and form elements may not align correctly when scaled.
Modern CSS has better solutions: As with most outdated techniques in this post, modern browsers now support much better layout techniques and relative units, that make responsive design much more consistent and easier to maintain. These include Flexbox, CSS Grid and rem, em, %, vw, vh units. Along with Media Queries and container queries, this gives developers the ability to adapt individual elements proportionally, rather than resorting to scaling the entire UI.
Performance issues: The use of zoom can cause serious performance issues, especially on low-powered devices since the rescaling causes the browser to scale rasterised layers rather than reflow content natively, which increases UI repaint costs.

Can I still use it?

Seriously, only if you hate your users and love additional maintenance. In practical terms, using it would not be a responsible choice; avoid it. If you come across a critical legacy site using this approach, plan to refactor it with modern techniques. Build your layouts using CSS Grid or Flexbox for flexibility across breakpoints, implement fluid typography with clamp and viewport units, adopt container queries for component-level responsiveness, rely on viewport-based units for consistency, and always test with browser zoom and assistive technologies to ensure accessibility and adaptability for all users.

Zoom Layouts Summary

Using zoom for layout responsiveness is an outdated, non-standard technique that can compromise accessibility, compatibility, and performance. Modern responsive design principles provide far more robust, scalable, and accessible solutions.

If you require a transition approach for legacy systems still using Zoom Layouts, consider refactoring incrementally to CSS Grid and Flexbox combined with relative units like rem or percentages to modernise their responsiveness. Luckily for you, this isn’t the last time you’ll hear about the infamous proprietary zoom property, as it makes quite a few appearances later in the blog post when we dive into those classic IE layout quirks.

Nested ems instead of Pixels

Before the CSS rem unit was added to the CSS Values and Units Module Level 3, developers used the em unit as a responsive strategy to avoid fixed pixel font sizes. Having used it for years I can confirm it was a real pain in the ass to work with (pardon my French!). When using an em unit, both the CSS font size and spacing were sized relative to their immediate parent container.

For example, given this HTML:

<body>
	<div class="container">
		<p class="child">Some text here</p>
	</div>
</body>

And this CSS:

body { font-size: 1em; }
.container { font-size: 1.2em; } /* relative to body */
.child { font-size: 1.2em; } /* relative to .container, so compounded */

Can you guess what the .child font size of the text is in pixels?

Better get your Math(s) hat on, let's go through it!

Default body font size = 16px so 1em x 16px = 16px. The .container DIV is relative to the body font size so: .container font size = 1.2em x 16px = 19.2px. The .child paragraph is relative to the .container font size so: .child font size = 1.2em x 19.2px = 23.04px

That's right, that well-known font size 23.04px!

Now this is just a very basic example, imagine if you include em units for margins and paddings too! And also layer on additional nesting! Hopefully, you are starting to realise how painful em units were to use on a website, especially when the only viable alternatives were percentages (which had the same relative nesting issue and were even less intuitive to use than em), or CSS keywords e.g. font-size: small, medium, large, x-large, etc. As you can see, there weren’t a lot of viable or maintainable options in terms of responsive typography and spacing in the early responsive design era (around 2010-2013).

Why it's outdated?

Complexity and unpredictability: Nested ems lead to compounded calculations as we saw in the simple example I gave above, making sizing unpredictable in deeply nested components. A small change in a parent font size cascades unexpectedly and could completely obliterate your well-crafted layout.
Maintenance overhead: Adjusting layouts or typography with nested ems quickly creates brittle CSS and significant technical debt, especially when ems are used for spacing like margins and padding.
Inconsistent UI scales: Components may render differently in different contexts if they rely on em units, especially in large applications with diverse layout containers.

Modern Alternatives

You can utilise several modern options for nested em units. These include:

rem units for consistent global scaling relative to the root font size
Clamp-based fluid typography for responsive design, for example Utopia.fyi.
CSS custom properties (variables) for consistent, maintainable scales

Can I use them today?

You could, but I have no idea why you would! When more viable alternatives exist today like rem units for global scaling, clamp for fluid typography, and CSS variables for maintainable scales, why make life harder than it needs to be??

Nested ems Summary

Using nested em units is outdated. It adds unnecessary complexity and unpredictability. For modern responsive design you are far better off using rems for consistent global scaling, or taking advantage of the clamp CSS function if you are feeling adventurous. Lastly, you could always use modern CSS variables for more consistent and maintainable code.

Setting the browsers base font size to 62.5%

As a direct follow on from the nested em technique earlier in the post, there was an alternative that developers came up with to simplify the math(s) behind percentages (since they had the same "relative to parent" issue as em's). When developers decided to use percentages for fonts, they often set the font size on the <html> element to:

html { font-size: 62.5%; } /* default font size now 10px not 16px, due to scaling making `em` units easier to work with (base-10 rather than base-16) */
.container { font-size: 1.6em }/*16px*/
.container { font-size: 2.4em }/*24px*/
.container { font-size: 3.6em }/*36px*/

This avoided complicated fractional calculations when using em units:

Without the percentage: 1em = 16px → 24px = 1.5em.
With the percentage: 1em = 10px → 24px = 2.4em.

You still had the problem with nested elements, but that was later fixed by using rem units (root em).

Why are these techniques less common today?

It overrides user defaults: Some users may increase their base font size from 16px for accessibility reasons, hard-coding the base size to 62.5% undermines this user preference.
Modern teams work with rem: Most developers and teams now accept that 1rem = 16px and use design tokens, variables, or a spacing scale instead of forcing a base-10 (62.5% hack) mental model.
Simplicity from Modern tooling: Design systems, utility classes, and CSS variables handle sizing scales more predictably without the 62.5% hack.

Can I still use it today?

No, not really, mainly due to the list of reasons I've given above. font-size: 62.5% was merely a developer convenience hack to make 1em / 1rem equal 10px for easy math(s). Look at the short list of modern alternatives I have listed above instead.

Base font size Summary

As mentioned above, this math(s) hack for easier font sizing is no longer required on the modern web, in fact it should be avoided due to the impact it has on users who change their base font size for accessibility reasons. Look to use one of the more modern techniques mentioned in the "Why are these techniques less common today?" section above.

Fixed-Width Fonts for Responsive Text

Fixed-width fonts are better known as monospaced fonts, they allocate the same horizontal spacing to each character. For example:

.mono-spaced-font {font-family: Courier New,Courier,Lucida Sans Typewriter,Lucida Typewriter,monospace;}

The example provided above demonstrates how to render text in a monospaced font and concurrently defines a monospaced font stack for most web page implementations. The reason I say "most" is because Windows has a 99.73% support for Courier New and OSX has 95.68% support according to CSSFontStack. Which is why it is listed first in the font stack, for the less than 1% of users that don’t support it, the browser will look for Courier and so on, until it gets to the end of the font stack, where it just tells the browser to use any monospace font the system has available.

Historically, monospaced fonts were used for:

Terminal emulation.
Code editors for alignment.
Early web design, where the layout predictability was prioritised over aesthetics or responsiveness.

Why was the technique used in responsive text?

Developers and designers struggled with the web platform's limitations at the time due to a lack of suitable tools. So monospaced text was usually used for:

Consistent character spacing across browsers.
Easier text alignment in table-based layouts.
Simplifying calculations for layout sizing, since browser layout strategies were much less mature than they are today.

Why is the technique outdated?

The technique is now considered outdated for various reasons, including:

It limits design flexibility. Modern responsive design has moved on from fixed typography, as fluid typography is now possible, which is better served by proportional fonts that adapt visually to varying screen sizes and reading contexts.
Monospaced fonts are harder to read, especially for paragraphs or long text blocks. This requirement on the modern web is critical for accessibility-focused design.
Instead of outdated methods, modern CSS offers enhanced tools and support for contemporary layout techniques. Flexbox and CSS Grid, coupled with various typography scaling units like rem, em, vw, vh, and clamp(), enable more predictable and reliable layout control.
There's no performance difference between modern proportional fonts and monospaced fonts, they both have similar browser overhead, so why choose a technique that is harder to maintain and comes with a whole host of other disadvantages?

What's a modern replacement?

There are a number of modern alternatives, some of which we touched on above. These include the use of:

Fluid typography with CSS clamp() and viewport units to ensure text scales responsively across devices.
Proportional fonts with font fallback stacks to optimise readability and layout adaptability.
Only using monospaced fonts for semantic or functional reasons, not aesthetics. Code blocks and tabular data are prime examples of where monospaced fonts should be used to enhance readability of these certain areas of a website. Adventurous designers can even transform a web UI into a retro Terminal window with these elements, though readability must be carefully considered.

Can I still use it?

As stated repeatedly in this section of the blog post, while technically possible, it would be highly illogical to employ this technique. Given the numerous disadvantages outlined earlier in this section, utilising such an antiquated method on the modern web would be ill-advised.

Fixed-Width Fonts Summary

There are so many font options available to developers and designers today. There is no way you should ever use a monospaced font for anything besides sections of code, or possibly text in a data table, depending on what type of data you are wishing to display. In both of these cases, a monospaced font can enhance readability if used correctly.

4. The Plugin Era – Flash and Friends

Flash-Based Content

I distinctly remember having a conversation with a then colleague regarding iOS not supporting Flash content and how it was the beginning of the end of Adobe Flash (Flash) on the web. At the time he refused to believe it, but thankfully for the web, my prediction came true!

What was Flash-based content?

Flash was a proprietary multimedia software platform developed by Adobe. It was used to:

Deliver animations, video, and interactive content via a plugin in the web browser.
Enable rich media applications embedded in websites.
Power early interactive interfaces on the web, this was way before the web platform matured and could support these types of interactivity natively.
I, personally, remember it for flash-based advertisements of which I created many when I was first starting out in web development!

Why was it popular?

Flash was hugely popular at the time due to the fact that:

Cross-browser multimedia support was lacking on the web platform (i.e. no native support)
Advanced vector animation support
In 2005, Flash was the sole method for streaming audio and video on the web, as exemplified by YouTube's reliance on it.
Interaction was programmed through the use of ActionScript. If it sounds very familiar to JS, that's because it is, as they are both based on the ECMAScript Standard. That's a massive oversimplification, but if you are curious, read all about it on Wikipedia.
In the late 1990s there was a popular trend on the web of having completely pointless Flash intros that would load and play automatically before you entered a site. There are countless example of these intros on YouTube if you are interested!

Why was it deprecated?

There are many reasons as to why Flash is now deprecated. These include:

Flash was well-known for serious security vulnerabilities, which were often used for malicious software and browser takeovers.
Flash content often consumed significant CPU and memory, leading to poor performance and excessive battery drain on mobile devices.
As mentioned earlier, Apple refused to support Flash on iOS, citing security, performance, and stability concerns, which contributed heavily to its decline.
Adobe's proprietary Flash technology was incompatible with open web standards, hindering accessibility, interoperability, and sustainability.
Lastly, open web standards and the web platform evolved to replace Flash with native (and non-proprietary) functionality like:
- Native video and audio playback (<video> and <audio> tags)
- CSS animations and transitions
- Canvas and WebGL for interactive graphics and games
- SVG for scalable vector graphics

Flash met its end with the advent of modern web APIs, including HTML5, CSS3, and modern JS.

In 2017 Adobe announced that Flash's end of life would be in 2020. In December 2020 Adobe released the final update for Flash. By January 2021, major browsers disabled Flash by default and eventually blocked Flash content entirely.

Can I still use it?

At last! A straightforward answer to this question: No, it's impossible for you to use Flash on the modern web as Flash content in no longer supported in any modern browser. Simple! RIP Adobe (Macromedia) Flash 1996 to 2020. You won't be missed.

Modern Alternatives

As mentioned above, there are a number of native browser-based alternatives for Flash (HTML5, CSS3, Modern JS) functionality these are: - Native video and audio playback (<video> and <audio> tags) - CSS animations and transitions - Canvas and WebGL for interactive graphics and games - SVG for scalable vector graphics

Scalable Inman Flash Replacement (sIFR)

Before the introduction of the @font-face at-rule, which is defined in the CSS Fonts Module Level 4 specifications, Web Designers and Frontend Developers were desperately seeking to expand the limited number of cross-browser, and cross-operating system fonts that were available on the web. In order to achieve that, a number of workarounds and general ingenious browser hackery were built. From the name of this technique you may also be able to guess the answer for the "Can I still use it?" section!

What is sIFR?

Scalable Inman Flash Replacement (sIFR) was an creative technique that used JS and Flash together to replace HTML text elements with Flash-rendered text. This feature allowed developers to embed custom fonts directly within the Flash file. Consequently, they could modify the HTML text, and the Flash file would dynamically render the updated content.

This workaround was required at the time because there was very limited support for custom web fonts via the use of @font-face. Surprisingly, @font-face was first introduced by Microsoft in Internet Explorer 4 in 1997, using Embedded OpenType (EOT) as the font format. This was proprietary to IE, so no other browsers supported it. Since there wasn't a cross-browser way to use custom fonts, alternative techniques like sIFR emerged.

sIFR Popularity

sIFR emerged in the early to mid-2000s, with its first public version released around 2004-2005. sIFR was widely used until around 2009-2010, especially for headings and branded typography. Its popularity grew during that period due to the technique's preservation of SEO and accessibility advantages. This was because the original HTML text remained in the Document Object Model (DOM), allowing it to still be readable by search engines and assistive technology. And once setup it was simple to update the underlying text and sIFR took care of the rest. There was also the added bonus that the text remained selectable, so could be copied and pasted when needed. It sounds like a great solution, so where did it all go wrong?

Why it's outdated?

There are several reasons why the sIFR technique is now outdated. We covered the main one in the previous "Flash-Based Content" technique above:

It relies on the Adobe Flash Player browser plugin, that is now deprecated and blocked in all major browsers due to security vulnerabilities and performance issues.
It slowed down page performance by increasing page load time, due to having to download the Flash assets.
Although it was partially accessible from an HTML perspective, it certainly wasn't perfect as it introduced accessibility and compatibility issues on devices without Flash support.
Web standards came to the rescue. CSS3 brought with it native cross-browser support for @font-face for custom fonts, without the need for any browser plugins. The new standards supported Web Open Font Format (WOFF and WOFF2) font formats which are a standardised and optimised custom font format for the delivery of fonts on the web. Basically, HTML5 and CSS working together simply removed the need for plugin-based typography workarounds.

Can I still use it?

As I mentioned, above this is a pretty simple question to answer… No, not at all, the removal of Flash from all modern browsers used today guarantees that!

Modern Alternative

There's only one modern alternative that should be used on the web today: @font-face. An example of its usage is given below:

@font-face {
  font-family: "MyCustomFont";
  src: url("fonts/MyCustomFont.eot"); /* IE9 Compat Modes */
  src: url("fonts/MyCustomFont.eot?#iefix") format("embedded-opentype"), /* IE6-IE8 */
       url("fonts/MyCustomFont.woff2") format("woff2"), /* Super modern browsers */
       url("fonts/MyCustomFont.woff") format("woff"), /* Modern browsers */
       url("fonts/MyCustomFont.ttf") format("truetype"); /* Safari, Android, iOS */
  font-weight: normal;
  font-style: normal;
}

Thankfully, modern browsers widely support WOFF, thus, simplifying the above code:

@font-face {
  font-family: "MyCustomFont";
       url("fonts/MyCustomFont.woff2") format("woff2"), /* Super modern browsers */
       url("fonts/MyCustomFont.woff") format("woff"); /* modern browsers */
  font-weight: normal;
  font-style: normal;
}

In fact, any modern web browser that supports WOFF also supports WOFF2. Therefore, the code you should use today is as follows:

@font-face {
  font-family: "MyCustomFont";
       url("fonts/MyCustomFont.woff2") format("woff2");
  font-weight: normal;
  font-style: normal;
}

In all instances above you'd use the custom font like so:

.myfontclass {
	font-family: "MyCustomFont", /* other "fallback" fonts here */;
}

The browser will take care of the rest!

Note: that you should always provide a font fallback in your font-family property, just in case the font file fails to load, or it is accidentally deleted from your server. There is so much more to the use of @font-face. If you are interested in advanced topics around its usage you should definitely check out Zach Leatherman's copious amount of work on the subject over the years!

Cufón

Much like sIFR above, Cufón was created due to the lack of options when it came to using custom fonts on the web. It was popular around the same time as sIFR (late 2000s and early 2010s). It essentially was solving the same problem, but using a different cross-browser technique. Whereas sIFR used Flash, Cufón worked like so:

Fonts were converted into vector graphics, and canvas (or VML for older versions of IE) this was then used to render the text in place.
The JS then replaced the HTML text with the custom font rendered version of the text.
Since it was JS-based, there was no need for any plugins (like Flash).

Why was it used?

As previously noted with sIFR, browser support for CSS @font-face was inadequate or inconsistent at the time. Designers and developers wanted to use custom fonts for branding and stylistic reasons without users having to install a plugin or the fonts locally. Cufón was attractive because it:

Didn't require a plugin for it to work.
Provided near pixel-perfect rendering of the custom font.
Was easy to integrate with minimal JS setup.

Why is it outdated?

Modern browsers all support @font-face, a much better solution as it allows the direct use of web fonts like WOFF or WOFF2 files without the use of JS hacks.
Its usage impacted accessibility. Due to Cufón replacing text in the DOM with rendered graphics, screen readers couldn't interpret these replacements as text, thus degrading a site's accessibility.
Cufón caused web performance issues as the text replacement script was run after page load, which increased page render time, blocked interactivity and degraded the overall performance, especially on slower devices.
Although Cufón attempted to preserve the replaced text in the DOM, the results were often inconsistent, mainly because search engine crawlers at the time had inconsistent results when parsing JS-replaced content.
Cufón didn't work with responsive design, once rendered the Cufón replaced text didn't scale correctly, unless the page was reloaded at the new page size.

Can I still use it?

Although the site is still available here, and the cufon.js script is still available to download. The font generator has been taken down and is no longer maintained. So to get it working you'd need to jump through quite a few hoops! So, what I'm really trying to say is: Yes, you can, but it isn't worthwhile. Even the original author Simo Kinnunen says on the website:

Seriously, though you should be using standard web fonts by now.

Modern Alternatives

Rather than repeat myself, I'll refer you to the same section from the sIFR methodology above.

GIF Text Replacements

Although I am aware that this is not a plug-in, it seemed like the most appropriate section to include it in since we are discussing font replacement techniques. Of all the custom font techniques I've listed, this one is by far the worst in my opinion. The technique was popular in the late 1990s and into the early 2000s, when there were very few other options for using custom fonts on the web.

What is a GIF Text Replacement?

It's a self-explanatory name. In order to use a custom font, a designer would create the static asset (usually via Photoshop or similar), then the developer would cut out the text as a GIF. This image would then be used to replace the HTML text on the page with the image, in order to make it look like a custom font was in use. Example code for this technique can be seen below:

<body>
	<!-- Gif text replacement for a heading -->
	<h1>
		<img src="images/heading-text.gif" alt="Welcome to Our Website">
	</h1>
	<ul>
		<li>
			<!-- Gif text replacement for a navigation link -->
			<a href="/about.html">
				<img src="images/about-link.gif" alt="About Us">
			</a>
		</li>
	</ul>
</body>

Note: Some readers may wonder why a GIF was used instead of a PNG, since both support transparency. The main reason is that Internet Explorer offered poor support for transparent PNGs and required a complicated hack to make them work, which I will explain later in the post.

Why was it so bad?

It was time-consuming and maintenance heavy. Should the design change, then so would all the GIFs that had to be manually cut out of the design files again.
It was bad for Accessibility. Screen readers are unable to process text embedded within GIFs or images that lack meaningful alt text. The absence or outdated nature of alt text therefore created an exclusionary experience for users with visual impairments.
It was bad for SEO. Search engines could not index text within images, thus, harming discoverability, this technique relied on developers ensuring they had accurate alt-text, which isn't always the case.
It was bad for performance. At the time of its popularity, the web was going through a transition period from HTTP/1.0 to HTTP/1.1. Although HTTP/1.1 was better for TCP connections than HTTP/1.0, these TCP connections were very expensive in terms of web performance, and each of these GIF replacements required its own TCP connection, which increased page load times.
It was terrible for responsiveness. Although the responsive web was a few years away when it was popular, the difference between images and text rendering was text's ability to scale across different devices and screen sizes. Images simply couldn't do that, leading to poor rendering and pixelation on some devices.
GIF only supports 256 colours (8-bit) and for the GIF to be transparent, one of those colours would need to be transparent. So if your text had a complex colour palette, it either wouldn't work, or just look terrible.

Can I still use it?

No, it's as simple as that. It's a technique with so many negatives and so few positives, it should be confined to the interesting history of the web platform!

Modern Alternatives

Again, rather than repeat myself, I'll refer you to the same section from the sIFR methodology above.

Adobe AIR

I remember going to a conference around 2007 / 2008, and there was so much hype about Adobe AIR. It was going to be the "next big thing", due to the fact it could enable developers to create rich desktop and mobile apps using only web skills and technologies.

What was Adobe AIR?

The AIR in Adobe AIR stood for Adobe Integrated Runtime. It was a cross-platform runtime developed by Adobe. It allowed developers to use: HTML, JS, Adobe Flash, Flex, and ActionScript. All combined they could run as a standalone desktop or mobile application. It supported Windows and macOS for desktop, and later Android and iOS on mobile.

Furthermore, it also enabled:

Running Flash-based applications outside the browser.
Enabling rich multimedia, animations, and offline capabilities.

Why is it outdated?

It relied on Flash and ActionScript. With flash reaching its end of life in late 2020 due to persistent security vulnerabilities, and the momentum behind open standards like HTML5, CSS3, and JS ES6+. AIR ceased to exist due to the loss of its core technology.
A shift to modern cross-platform frameworks. The market moved towards more efficient and performant technologies like:
- React Native
- Flutter
- Electron (for desktop apps) The advantage of these frameworks is they use native components or JS runtimes, without the heavy reliance on Flash. This offered developers and users greater performance, maintainability, security, and community support.
Lack of Adobe support. Adobe handed AIR over to a subsidiary of Samsung (Harman) in June 2019for ongoing maintenance. Support is still provided by Harman, but only for enterprises with legacy applications that they still rely on, There's no active innovation or features being added to AIR by Harman.
Security concerns. As with Flash in the browser, security was always an ongoing issue, and this continued in AIR since it was the backbone to its core functionality. By continuing to build on AIR, it poses security risks and compatibility limitations with modern browsers and operating systems.
Lack of developer interest and ecosystem. Developers on the modern web tend to favour open ecosystems with an active community for support and updates. Adobe AIR’s ecosystem has completely stagnated.

Can I still use it?

As with any other Flash-based technology, I'm afraid not, it is no longer supported and even if you could, there are more modern open frameworks you could use like, React Native, Flutter, or Electron (for desktop applications). AIR is now history, and if you are using an AIR application within your digital estate, it is strongly recommended you prioritise migration, due to higher maintainability, poor security, and lack of developer availability.

Yahoo Pipes

It is easy to forget just how dominant Yahoo was on the web during the late 1990s. Before Google emerged as the leading search engine, Yahoo was one of the primary gateways to the internet. Its peak influence was between 1996 and 2000, when it played a central role in how people accessed and navigated the web. It was the default starting point for most web users due to its curated directory, news, and email services combined. It was also a technology leader on the web too, as I mention later in the blog post when I look at their extensive JS Library: Yahoo! User Interface (YUI).

I remember using Yahoo Pipes for combining my many RSS feeds at the time, it really was a fantastic visual tool for data manipulation.

What was it?

Yahoo pipes was a visual data mashup tool that allowed users to aggregate, manipulate, and filter data from around the web. It was released in 2007, and it allowed developers (and non-developers) alike to aggregate, manipulate, and filter data from around the web. It provided a drag and drop interface where users could connect various manipulation modules and join them together by creating pipes between modules. You were essentially piping data "through" the tool (hence the name!) and the manipulated date would come out the other end. It was considered highly innovative at the time, and it was used a lot for rapid prototyping.

Why is it outdated?

Have a look at the Yahoo! homepage, and you will see it is the shadow of its former self, it looks to more of a news aggregation service now than a popular search engine. This was because Yahoo decided to make a giant shift in business strategy, moving away from developer tools and open web utilities to concentrate on advertising and media products. Although it was popular with technology enthusiasts, it was never a mainstream product for Yahoo, so the operational expenses vs. usage statistics didn't align with Yahoo's business priorities. Lastly, the web evolved beyond Yahoo Pipes, and it couldn't keep pace with the changes. Modern APIs, JSON-based services, and JS frameworks allowed developers to build similar data transformations programmatically with greater flexibility. Due to all these factors Yahoo Pipes was sadly shut down in 2015.

Can I still use it today?

Nope, it was shutdown by Yahoo in 2015, with no further support or hosting by Yahoo.

Modern Alternatives

While innovative at the time, visual mashups have been replaced by:

Dedicated data transformation tools (e.g. Zapier, Integromat/Make).
Serverless functions (AWS Lambda, Azure Functions, Cloudflare Workers, and Fastly Compute@Edge) for real-time data processing.
Low-code platforms with integrated API management.

The web has also become a lot more complicated regarding Web scraping and feed aggregation. Because anti-scraping measures, authentication, and API rate limits weren't necessary when Yahoo Pipes was created, the techniques it employed didn't support the robust backend processes now required to handle these requirements. Although Yahoo Pipes was innovative at the time, it has long been discontinued and is now considered a obsolete part of web platform history.

PhoneGap / Apache Cordova

The one thing that sticks in my mind when I think about PhoneGap is when I saw a talk from one of the Nitobi engineers back in 2009 / 2010, he said something along the lines of:

We are using PhoneGap to bridge the current gap for developers in creating cross-platform mobile applications. Our goal is for it to become obsolete once native platforms fully support these capabilities.

This really impressed me at the time, spending so much time on a product with the aim for it to become obsolete.

What was PhoneGap?

PhoneGap was a mobile development framework created by a Canadian company called Nitobi in 2009, it was later acquired by Adobe in 2011. PhoneGap allowed web developers to build cross-platform mobile applications, simply, using web technologies: HTML, CSS, and JS. The applications were packaged into native containers allowing them to run as mobile apps, while also having access to device API's via JS.

Why was it required?

At the time of release, the mobile web was incredibly popular and getting bigger month on month. It's important to remember, the first version of the iPhone was released only 2-years before (June 2007). This really was an exciting time in the Web Platform's history. If you wanted to release a cross-platform application at the time and wanted to support Android, iOS, and Windows Phone, you needed developers with knowledge of multiple programming languages:

Android required Java.
iOS required Objective-C.
Windows Phone required C#.

Finding a single developer with all these skills would be incredibly hard, so to build and maintain all 3 platforms usually required a whole team of developers.

One of the main advantages of PhoneGap was that all 3 platforms had a single central codebase, which reduced development time and maintenance.

PhoneGap leveraged Cordova under the hood, which it essentially branded and wrapped for broader adoption.

Why is it outdated?

PhoneGap apps performed poorly. This was especially true for graphics-intensive, or animation-heavy interfaces. This is because PhoneGap apps ran within a WebView container rather than as a native application.
Adobe stopped supporting it. This seems to be a common theme in this blog post… Adobe ended support for PhoneGap in October 2020. At the time developers were advised to either migrate to Apache Cordova or consider other frameworks.
Alternatives evolved. As the mobile platform expanded, so did the availability of other frameworks to help developers build apps. These alternatives included:
1. React Native allowing near-native performance with JS and React paradigms.
2. Flutter enabling high-performance apps with a single Dart codebase and native compilation.
3. Progressive Web Apps (PWAs) reducing the need for wrapping web apps as native apps in many use cases.
4. Capacitor (by Ionic) providing modern native bridging with a streamlined developer experience compared to PhoneGap/Cordova.
PhoneGap's Ecosystem growth stalled. As newer frameworks were released and Adobe stopped supporting it the community moved away and PhoneGap’s plugin ecosystem stagnated.

Can I still use it?

No, there are several alternatives I have listed above that you should consider instead. PhoneGap served its initial purpose as a bridge, enabling developers to build cross-platform mobile applications. As was its mission, It became obsolete when native platforms fully incorporated these capabilities.

Microsoft Silverlight

NOTE: I never used Silverlight (I do remember it being announced) I'm just adding it to the post for completeness.

What is Silverlight?

Silverlight was a rich internet application (RIA) framework introduced by Microsoft in 2007. It was conceptually similar to Adobe Flash, designed to deliver interactive multimedia, animations, and streaming video inside the browser.

It used a subset of the .NET Framework, with applications typically written in C# or VB.NET, and presentation defined using XAML (an XML-based UI markup language). Developers could reuse existing .NET skills, which made Silverlight attractive in Microsoft-centric enterprises.

Silverlight was often used for:

Media streaming (notably Netflix in its early streaming days)
Interactive dashboards and line-of-business web apps
Cross-browser, cross-platform plugins (Windows and Mac were supported, but Linux support lagged)

Why is it considered legacy?

Plugin dependency: By the 2010s browser vendors had moved away from browser plugins in favour of newly developed web platform technologies. Plugins were often unsecure, unstable, and inaccessible.
Limited cross-platform reach: Although Silverlight was well supported on Microsoft platforms (as you would expect!), it also had support on Mac, but it had limited support on Linux (via project Moonlight), and no support on mobile devices (Android, iOS).
Rise of open web standards: HTML5, CSS3, and JavaScript rose so rapidly for audio, video and advanced graphics (via canvas). The use of plugins was no longer required.
End of support: Considering the above points, Microsoft only stopped support in October 2021. Although Browser vendors stopped a long time before: Chrome in 2015, Edge never supported it. Firefox ended support in March 2017.

Can I still use it?

Well, this is another easy one. No, you can't Microsoft have dropped support and no Modern browsers support it either.

Modern Alternatives

The answer to this is basically native web platform API's. Specifics include:

Video Streaming
- HTML5 <video> element with adaptive bitrate streaming (HLS, MPEG-DASH).
- DRM is handled via Encrypted Media Extensions (EME).
For interactive apps and dashboards:
- Modern JavaScript frameworks such as React, Angular, Vue, or Svelte.
- WebAssembly (Wasm) for near-native performance, including options like Blazor (from Microsoft) which lets you run .NET in the browser without plugins.
For graphics, animation, and UI:
- CSS3 animations and transforms for UI transitions.
- Canvas API and WebGL for 2D and 3D graphics.
- SVG for scalable vector graphics.
- WebGPU (emerging) for modern GPU-accelerated rendering.

Silverlight Summary

Silverlight is legacy because it relied on a now-obsolete plugin model, had poor cross-platform support, and was outpaced by open web standards. Today, everything Silverlight did can be done more securely and portably with HTML5, CSS, JavaScript frameworks, and WebAssembly.

Java Applets

NOTE: I never used Java Applets (although I remember them!) I'm just adding it to the post for completeness.

What is Java Applets?

Java Applets were small applications written in Java that could be embedded into web pages and run inside the browser through a special Java Plug-in (based on the NPAPI plugin architecture). Introduced in the mid-1990s, they were part of Sun Microsystems’ vision of “write once, run anywhere” – letting developers build interactive content and complex functionality that browsers of the time (pre-HTML5) could not support natively.

They were often used for:

Interactive educational content and simulations
Online games
Financial tools like mortgage calculators or trading dashboards
Enterprise intranet applications

Why is it considered legacy?

Plugin dependency: To use the Applets this required a user to install the Java Runtime Environment (JRE) plugin and keep it updated. I distinctly remember the update prompt for these nuisance updates!
Security risks: The Java plugin was a frequent target of exploits and malware, leading browsers and enterprises to actively block or disable it.
Performance and user experience: Applets often loaded slowly, had inconsistent UI integration with web pages, and required clunky permission dialogs.
Decline of NPAPI support: Browsers started phasing out NPAPI (the plugin technology Applets relied on). Chrome dropped NPAPI in 2015, Firefox dropped NPAPI in 2017 (except Flash until 2021) , and Microsoft Edge never supported NPAPI.
Official deprecation: Oracle deprecated the Java browser plugin in Java 9 (2017) and removed it entirely in later releases.

Can I still use it?

Nope! Modern browser no longer support it and Oracle stopped supporting the plugin in 2017.

Modern Alternatives

This list is basically native web platform API's. I don't want to repeat myself so refer to the Silverlight Modern Alternatives from earlier in the post.

Java Applet Summary

Java Applets are legacy because they relied on a fragile plugin model that posed significant security risks and is no longer supported by modern browsers. Today, HTML5, JavaScript, and WebAssembly provide richer, faster, and safer alternatives without requiring any plugins.

5. The JavaScript Framework Explosion

DHTML Beginnings (1997)

Dynamic HTML (DHTML) was all the rage around 1997-1998. By combining the primary web technologies: HTML, CSS, JS, and the DOM, developers realised that they could make a web page interactive and "dynamically" update the page without the need to reload the entire page. This new technique was frequently employed for animated HTML elements, such as image rollovers and dynamic navigation menus. It also provided immediate user feedback, particularly for form validation, by checking if the user had entered a valid email, for instance. If not available, then use standard HTML and CSS to show a user-friendly error message.

DHTML wasn't necessarily bad, if implemented in moderation, unfortunately it always seemed like the wild-west. What would work in Internet Explorer 4 (IE4) wouldn't always work in Netscape Navigator because Microsoft and Netscape Communications Corporation (NCC), had different levels of support for JavaScript in fact, Microsoft had their own version of the ECMAScript standard called JScript. This led to lots of maintenance headaches for developers as the solution often involving forking code for different browsers.

Secondly it turned into a bit of Copy / Paste madness. Since any developer could simply "View Source" on a web page and copy the code and add it to their own website, you often ended up with a mishmash of different interactions and animations all over a website! Thankfully, the trend eventually died out!

Can I still use it?

A web developer could technically still use DHTML on the modern web, but doing so would be strongly discouraged for any serious or production-level work. This is because in using it, a developer wouldn’t be following modern best practices like separating concerns with structured HTML, CSS and JS, building accessible and performant interfaces, using modular and maintainable code, or leveraging modern frameworks and tooling that enforce consistency, security and scalability.

Modern alternatives

There are several modern alternatives to DHTML, including:

Using Frameworks like React, Vue, or Svelte
Native browser APIs
Modern CSS techniques
Progressive enhancement and accessibility standards

Early frameworks and libraries

Prototype.js (2005)

Prototype.js version 1.0 was released in February 2005, and was initially developed to help simplify JS tasks in Ruby on Rails (RoR) projects. Its key features were a host of DOM manipulation utilities, AJAX abstraction to make XMLHttpRequest easier to handle cross-browser, Class-based inheritance in the form of a lightweight object-oriented programming (OOP) in JS. But by far its most influential feature was its shorthand DOM element selector $(), later popularised by jQuery.

I remember trying to learn and use Prototype a few times, but as a JS beginner, I found the name confusing. Especially since the prototype object sits at the heart of JavaScript, with almost the whole language hanging off it through things like property and method inheritance.

Can I still use it?

I mean, technically you could, but be warned it hasn't been updated in almost 10-years! Given the significant evolution of JS and the availability of modern, updated alternatives that leverage the latest browser JS APIs, it would not be a wise choice.

Modern Alternatives

It really depends on what a developer was using prototype.js for, as it had quite a range of functionality:

DOM manipulation

AJAX

Utility functions

Templating

Full replacement

Prototype.js was incredibly powerful. My preference would be to utilise several micro-JS libraries for specific functionalities rather than adopting an extensive framework such as React, but that's just my opinion, given the complexity of the React ecosystem.

Script.aculo.us (2005)

Script.aculo.us v1.0 was released in 2005 as an extension to Prototype.js. It built on Prototype.js by delivering a powerful set of visual effects, animations, and UI components. It also featured Drag-and-drop support out of the box, as well as sortable lists and autocompletion widgets. As with Prototype.js, Script.aculo.us popularity was partly because it was bundled with RoR, and it had widespread adoption within the "Rails" community.

I still remember Script.aculo.us for its distinctive URL, its bright, animated homepage, and how it really embodied the spirit of ‘web 2.0,’ making the web feel more alive. It's a library that left a lasting legacy, influencing later libraries like jQuery UI.

Can I still use it?

I really wouldn't recommend it, as it hasn't been updated in over 15 years! However, if you're curious to delve into some web 2.0 history, the site is still live (on HTTP, not HTTPS).

Modern Alternatives

Assuming you are only looking for a JS library for animations:

Dojo Toolkit (2005)

Dojo Toolkit was one of the first major cross-browser toolkits, released in March 2005 with version 0.1. It was developed by Alex Russell and was also maintained by a community of other developers too. It's open-source and still available on GitHub It was one of the earliest frameworks to help build rich web applications by simplifying DOM manipulation, AJAX, event handling, animations and internationalisation (i18n), among many other cutting-edge features. Not only that, it was an early advocate for Asynchronous Module Definition (AMD) and modular JS. Furthermore, it was one of the first JS libraries with strong accessibility (a11y) support. In 2018 Dojo was rewritten as Dojo 2 which supports TypeScript, reactive patterns, virtual DOM, and modern build systems. Dojo 1.x is still in use in some long-running enterprise applications on the web. The last stable release was released in 2022, which was mainly for bug fixes and security updates. New feature development has now shifted purely to Dojo 2+.

Dojo 1.x was incredibly influential for later JS libraries like jQuery, MooTools, Prototype, especially when it came to governance. It was governed by the Dojo Foundation, which later merged with the jQuery Foundation to form the JS Foundation (now part of the OpenJS Foundation).

Can I still Use it?

Version 1.x, would be a bad idea, but you could technically still use the latest version (8.0.0), although that may not be wise either, given there hasn't been a new release in over 3-years. So it's most likely better to stick with more modern framework alternatives.

Modern Alternatives

There are a number of modern alternatives you could consider, keeping in mind that Dojo was very focussed on Accessibility (A11y) and Internationalisation (i18n):

React: Maintained by: Meta (Facebook).
- A11y: Strong support, but it’s developer-driven. ARIA roles and keyboard navigation must be implemented explicitly by developers.
- i18n: Excellent ecosystem (react-intl, formatjs, lingui, etc.)
Vue.js (v3): Maintained by: Evan You and the Vue core team.
- A11y: Good defaults; still developer-led, but accessible components are emerging.
- i18n: vue-i18n is well-maintained and powerful.
Angular: Maintained by: Google
- A11y: Arguably the best among mainstream frameworks. The Angular Material team publishes a11y guidance, and many baked-in best practices exist.
- i18n: Built-in i18n support, including message extraction and compile-time translation.
Svelte / SvelteKit: Maintained by: Rich Harris and the Svelte core team
- A11y: Improving, but not as mature as React or Angular. Accessible components need to be explicitly chosen or built.
- i18n: Community libraries exist (svelte-i18n), but official support is not as comprehensive.

Yahoo! User Interface (YUI) (2006)

YUI was released in February 2006, its first release was version 2.0.0, this is because there was lots of internal development and usage within Yahoo! before it was released publicly. It was originally designed to standardise frontend development at Yahoo and provide the team with a solid cross-browser solution on which the Yahoo team could build feature-rich web applications. It contained a custom loader system that only loaded the components it needed, this was an ingenious approach in the pre-ECMAScript 6 (ES6) module era. Furthermore, it also came with a whole host of feature rich UI widgets, cross-browser abstraction, Event handling, DOM utilities, Animations, CSS Tools (which heavily influencing Normalize.css that was developed later), and it was one of the first libraries (after Dojo 1.x) to prioritise internationalisation (i18n) and accessibility, specifically Accessible Rich Internet Applications (ARIA).

What I remember most about YUI 2.x is just how huge it was! Not just the sheer number of UI modules, but the file size too: 300–350 KB minified, or 90–120 KB gzipped! This was before the widespread availability of fast broadband, and when hardware and browsers were significantly less optimised. A full build of the library could easily exceed those figures, too. This is why Yahoo also provided a combination aware CDN service to help reduce the number of requests made and bundle only the components needed. This was a practice that was way ahead of its time!

Can I still use it?

No, not really, it hasn't been updated since 2014, the reason for this is detailed in this Yahoo Engineering announcement from the time.

Modern alternatives

Given YUI's extensive nature as a framework, and to avoid repetition, I recommend referencing my notes on Prototype.js modern alternatives as an initial guide.

moo.fx (2005)

Moo.fx was a lightweight animation library designed to be unobtrusive and it also worked well with Prototype.js. It focused purely on DOM animations like height transitions and fading etc. It was part of a movement in JS to modularise the codebase for lighter and more responsive interactions. Furthermore, it was a distinct diversion away from larger and heavier animation libraries like Script.aculo.us.

I believe moo.fx was one of the first animation libraries I ever saw. Annoyingly the site hasn't been archived on archive.org. I do remember it having a simple and colourful homepage with examples of the animations you could add by only adding a tiny library to your page (3 KB).

It also had a fantastic URL too "http://moo.fx". What's so cool about the name, you may ask? Well, the .fx country code top-level domain (ccTLD) has now been rendered obsolete and is no longer available. It was originally reserved for Metropolitan France, but it was never officially delegated or made available for registration. France later adopted .fr as its official ccTLD. I can't fathom how Valerio Proietti, the creator of moo.fx, managed to register the name, but it's all true. As proof only a single record can be found on archive.org, dating back to 2007, it links to the site's robots.txt file.

Can I still use it?

I can't even find an archived copy of the homepage, let alone the library itself! So, it definitely falls into the "no, you can't still use it" category!

Modern Alternatives

Assuming we are only looking for a modern animation library, it's best to refer to the list I gave above for Script.aculo.us modern alternatives.

MooTools (2006)

The author of moo.fx, Valerio Proietti wanted more than an animation library. He aimed to develop a complete JS framework with an Object-Oriented Programming, modularity and extensibility as first-class features. Thus, MooTools was born! v1.0 was released in September 2006, and it featured some fantastic features for its lightweight size. These included a modular core, and separate components, you could also include, for extra functionality, an Advanced class system (predating the ES6 class functionality). Powerful DOM manipulation utilities, Ajax handling, effects (moo.fx), and custom events. It was both performant (for the time) and syntactically elegant. Development ceased in the mid-2010's with v1.5 (the final active release). moo.fx and MooTools hold a special place in my memory as one of the first JS libraries I learnt as a Junior developer.

Can I still use it?

Well, the website still exists here, But considering it hasn't been updated since January 2016, it's probably best to look for a modern alternative.

jQuery (core) (2006)

In 2006, a truly revolutionary JS library was released called jQuery. It was developed by the legendary John Resig. It offered a lightweight, chainable API in JS to simplify tasks like DOM traversal, and Ajax, among many other helpful tools and methods in its fantastically written API.

jQuery always prided itself on its easy-to-use API and its ability to abstract away the many cross-browser bugs related to different browser vendors implementation of JS (Mozilla) / JScript (Microsoft). It finally gave developers a "stable" API to start building JS powered websites without all the stress of the cross-browser hacks, and forking of code to make features work in every browser. With jQuery, it just worked!

I must admit, I absolutely adored jQuery (and still do)! The API was so clean and readable. The complete opposite to the DOM and the JS API! This library has saved more than just my code, it’s rescued entire projects and probably saved my sanity in the process! Especially working in Digital Marketing, as I did at the time. In those days, clients were constantly after the newest, flashiest animations, regardless of usability. It was all about chasing trends. And when the client's paying, you just nod and make that already bouncing button pulse and change colour!

Can I still use it?

Finally, I can say "Yes" to this question! Assuming you plan to use v3.x of jQuery, as the 1.x and 2.x branches are no longer supported or maintained. In fact version 4.0 is currently in beta, according to the Support page. Amazingly, almost a full 19-years after its first release, it is still under active maintenance! Not only that, according to the Web Almanac 2024, it's still the most popular JavaScript library in use on the web! Truly impressive work by the jQuery team and community!

Later era frameworks

Ext.js (2007)

Ext.js version 1.0 was released in April 2007. It was initially developed as an extension to the YUI Library before becoming a fully standalone framework. Its key features included a comprehensive suite of rich UI components, a powerful event model, and advanced layout management capabilities that far exceeded most contemporaries. It introduced a highly structured approach to building web applications, with a strong emphasis on reusable widgets and object-oriented design. But by far its most distinctive contribution was its fully integrated, desktop-like component model for the web. Something rarely seen at the time, and which set the tone for many enterprise-grade JS frameworks that followed.

I never used YUI personally, as its sheer size and breadth of functionality simply didn’t align with the kind of work I was doing at the time. As a result, Ext.js (which as mentioned above, was initially built upon YUI) wasn’t on my radar either. That being said, I’ve included it here for completeness, as it clearly played a significant role in the evolution of rich client-side application frameworks. During my research I discovered how Ext.js transformed into an enterprise-grade toolkit under the Sencha brand. Its strong emphasis on data-driven UIs distinguished it from other lightweight libraries of that period.

Can I still use it?

Yes, Ext.js is still a viable option that you can use on the modern web, if you are building a web application. It continues to be actively maintained by Sencha and even offers a React Extension, allowing for seamless integration of Ext.js components into React applications. However, be aware that Ext.js is now a paid library, with a per-year, per-developer licensing model that can be costly. While a free community version exists, it appears to have a very limited feature set.

jQuery UI (2007)

jQuery UI emerged in 2007 as an official companion library to jQuery, at a time when the JS ecosystem was fragmented and riddled with browser inconsistencies. It was developed to bring a unified, extensible suite of widgets, effects, and interactions to the web. jQuery UI offered an easy-to-integrate API that drastically lowered the barrier for implementing rich UI's, with full cross-browser compatibility. It played a crucial role in making dynamic front-end behaviour accessible to developers at all skill levels, becoming a staple in both enterprise and amateur hobbyist applications during the formative years of modern web development.

Although I was a big fan of jQuery and used it extensively across many projects, I never really had the opportunity to use jQuery UI in its entirety. When I did, it was typically for a single component, such as a date picker or drag-and-drop functionality. These components were reliable and well-supported, but required a lot of JS to function, and added complexity that I never felt was acceptable for a single feature. Especially when there were plenty of alternative micro-frameworks available, offering small, focused libraries that solved one problem well. I was far more inclined to take that modular approach than to include an entire suite of UI components unnecessarily.

One resource, I found invaluable at the time, was MicroJS. It hasn’t been updated in quite some time, but it remains a powerful illustration of how easy it is to cherry-pick the exact only the functionality you need, without burdening your page with hundreds of kilobytes of JS.

Can I still use it?

For this question, just as I did with jQuery, my answer is yes, you can still use it. It isn't updated very often, but it is still updated! The last release being in October 2024 with version 1.14.1

To put the enduring popularity of jQuery and jQuery UI into perspective, the 2024 Web Almanac reports that jQuery is still the most-used JS library on the web, appearing on 74% of pages in the dataset analysed. jQuery UI comes in fourth, with a 22% usage rate, Though described as mostly deprecated, it’s a clear reminder of how quickly modern tools become legacy software that must be maintained for decades. The latest release came nearly 18 years after jQuery UI’s first version. That’s an incredible achievement by the jQuery UI team, talk about dedication!

AngularJS (2010)

AngularJS and Angular are fundamentally different frameworks that share a name and lineage but are otherwise entirely different. AngularJS (1.x) was based on a Model-View-Controller (MVC) architecture with two-way data binding written in JS with support for ECMAScript 5 (ES5) and some ECMAScript (ES6) features. Angular (2+) was built entirely differently, it has a component-based architecture boasts stronger modularity, enabling both two-way binding and promoting unidirectional data flow. Another major difference is that Angular is written in TypeScript, a distinct superset of JS, that enables better tooling and type safety like Java and many other languages. Angular remains actively used and maintained on the modern web today.

I distinctly remember when Google released AngularJS because I was in Melbourne, Australia, working at a digital media agency. One of the Tech Directors there was raving about it, and how it was going to change Frontend development entirely. In hindsight, I agree with him, but I, personally, don’t believe it was a positive change. Single Page Apps (SPA’s) have had such a huge negative impact on Web Performance and Accessibility. Plus, a lot of the code in many of these SPA frameworks essentially reinvents functionality that’s already built into the browser and the Web Platform as a whole. Let's not overcomplicate things with framework-managed page state; we have a perfectly good back and forward buttons, thank you very much!

As you can probably tell, I’m not a big fan of SPA's. Admittedly, they have their place under some circumstances; However, I genuinely think they're excessive and unnecessarily complicate frontend web development for most applications. But I guess complex is the new simple, right? Anyway, rant over!

Can I still use it?

AngularJS, no. It is no longer actively maintained. Angular, absolutely, it is still a very popular framework on the modern web, although according to the State of JS 2024 survey, it has been overtaken by Vue.js for usage, for the 2nd year running. Frontend developers can be fickle, often chasing the next shiny framework like a kitten distracted by a dangling set of keys. It will be interesting to see if this decline in Angular usage continues in future JS surveys.

Backbone (2010)

Backbone.js was released in 2010 by Jeremy Ashkenas, who also created Underscore.js and CoffeeScript. As with AngularJS it was one of the first JS libraries that implemented the MVC architecture pattern to client-side JS. The key features of Backbone were its models, collections, views, router, events, and sync functionality that allowed it to easily communicate with RESTful API's.

I've only ever worked on a Backbone project once, and it was a rapid prototype website for a major airline based in Asia. Given the tight timelines and the client’s high design expectations, we ultimately opted for static HTML in the end, as Backbone’s complexity wasn’t advantageous for a rapid prototyping. In hindsight, had it got to the production stage, then I could see the architecture of backbone being very useful.

Can I still use it?

Backbone is nowhere near as popular as modern frameworks like React, Vue, or Angular, It is still mostly in use on legacy systems. The last version released was in April 1st 2025 v1.6.1, but looking through the releases, it seems to only get 1 update per year. According to Wappalyzer it is still in use by around 521,000 websites. The biggest of those being Atlassian. So in my opinion I'd say you should avoid using it and opt for a more popular framework with a more active community. Refer to the modern alternatives I listed for Dojo Toolkit Modern Alternatives as a starting point.

Knockout (2010)

During the early 2010s, Knockout.js was a popular JS library for building dynamic UIs using the Model-View-ViewModel (MVVM) pattern. It offered features like declarative bindings and two-way data synchronisation, which made it easier to keep the UI in sync with underlying data without manually manipulating the DOM. Its simplicity, ease of learning, and lack of required tooling, e.g. just drop in a <script> tag into an HTML page and go, this made it especially appealing at a time when frontend complexity was just beginning to accelerate. Though now largely superseded by modern frameworks, Knockout played a key role in the evolution of reactive web development.

I only used Knockout once for a client in Australia, while I lived there. It was a project that had been passed between offices because it had become a maintenance nightmare. Had it been a better planned build, it would have been a more pleasant experience, but time constraints meant a rapid delivery was prioritised over build quality.

Can I still use it?

The last release was way back in November 2019, so there's no way it should be used on a modern website build. Although, according to Wappalyzer it is still used by almost 43,000 websites. So you are very unlikely to come across it, even in legacy systems.

Modern Alternatives

If you are looking a modern MVVM framework then Vue.js, Svelte, or Aurelia are the way to go as the all fully support the MVVM architectural pattern. But if you aren't bothered about MVVM then refer to the list of modern alternatives I listed for Dojo Toolkit earlier in the post.

6. CSS Workarounds and Browser Quirks

Old CSS practices

Sliding Doors Technique

In the time before border-radius: existed in CSS, this was a legitimate way of “faking” the look of rounded borders. The technique's name reflects its implementation and purpose. (nothing to do with the 1998 film starring Gwyneth Paltrow about getting cheated on!). It was actually a rather ingenious method of creating rounded corners that expanded and contracted both in width and height of the bounding box containing the content. It was also incredibly useful for making your site navigation more interesting. If you are interested in reading all about this technique, check out "Sliding Doors of CSS" and "Sliding Doors of CSS, Part II" on alistapart.com both from October 2003!

If you are old like me, you can just jump back into full nostalgia mode by visiting the final code example here!

Can I still use it?

There's absolutely no need to use it on the modern web, simply use the border-radius CSS property, it comes with all the positives of the sliding door's technique (rounded corners), and none of the negatives (higher maintenance, additional downloads, non-semantic wrapper DIVS used only for presentation). And given its browser support on the modern web, you can go rounded corner crazy!

Image Sprites for Icons

The history of icon sprites is actually quite a fascinating one (for me at least!), since it is essentially rooted in web performance optimisation. The HTTP/1.1 specification (RFC 2616) published in June 1999, specifically says in section 8.1.4 that:

Clients that use persistent connections SHOULD limit the number of simultaneous connections that they maintain to a given server. A single-user client SHOULD NOT maintain more than 2 connections with any server or proxy.

HTTP/1.1 introduced persistent connections, a departure from the one-off connections of HTTP/1.0 (as detailed in section 8.1 of the HTTP/1.1 specification). This innovation understandably raised concerns within the Network Working Group regarding potential increases in server load across the web.

The web was becoming more and more popular every day around that time, with thousands of new websites being launched daily. Remember the Dot-com Bubble that burst in March 2000, it gives you some indication of just how popular the internet had become across the world!

You're probably thinking "what has any of this got to do with image sprites?", well, quite a lot, actually. With HTTP/1.1, every asset on a page meant opening a new TCP connection. So if your site had hundreds of icons, the browser would literally open hundreds of separate connections just to fetch them all (assuming it were allowed to)! And this is why browser connection limits were introduced into browsers. Different browsers chose different limits, but it was usually around 4 to 6 per domain. So in order to reduce the number of TCP connections to download all these image assets, a technique called Image Sprites was invented.

Image Sprites, involved compiling all your small image assets down into a single bigger background image file, then using CSS positioning (and width / height) to essentially mask the image and only show the single image you wanted to show in any particular place. It then only required a single TCP connection to download all the small images (one large image with all the icons), and you'd use some CSS magic to just reposition the background image. This technique was mostly used for image icons, and the CSS for this method would look something like this:

.icon {
  background-image: url('sprite.png');
  background-position: -20px -40px;
  width: 16px;
  height: 16px;
}

Here we have an icon that is 16px by 16px and the icon we want to show is positioned at -20px -40px on the background image. Genius! A working example of image sprites can be found here on CodePen.

Can I still use it?

Again, no need. Image sprites fell out of favour as a technique when more modern techniques like SVG sprites and webfont icons came along.

The technique was also outdated by new web performance technologies like:

But those are topics for another blog post, which I just so happen to have written a little about, so check them out (shameless self-promotion, sorry!)

Vendor Prefixes for CSS

Now I'm pretty sure some readers will be thinking, wait "vendor prefixes for CSS are old best practice??". They really aren't as they are a vital technique to stop new CSS additions from "breaking the web". By slapping on a vendor prefix to a CSS property allows browser vendors to experiment with new CSS features in isolation and then once stabilised in the CSS specifications, and supported by the major browsers, the prefix can be removed in favour of the non-prefixed version. For example, -moz-border-radius changes to border-radius. So just to clarify vendor prefixes aren't old best practice, but the way they are created within website code is. In the distant past you used to have to manually write out each of the prefixed versions to ensure maximum compatibility with all browsers. Can you imagine writing this out manually for all new CSS features you want to use! Example:

.mybox {
  -webkit-border-radius: 10px; /* Chrome, Safari, iOS Safari (early versions) */
  -moz-border-radius: 10px;    /* Firefox (pre-version 4) */
  -ms-border-radius: 10px;     /* Not officially supported, but some old syntax examples include it */
  -o-border-radius: 10px;      /* Old Opera (Presto engine, now obsolete) */
  border-radius: 10px;         /* Standard syntax */
}

What a waste of bytes and time! (I know compression pretty much fixes the bytes issue!) With the popularity of tools like Compass (a Sass Framework), LESS mixins, and Bourbon (a Sass mixin library), or even Stylus with the nib plugin, the issue started to be abstracted away from developers.

Can I still use them?

On the modern web there's simply no need to manually add prefixes. The best practice is to use PostCSS with Autoprefixer. Or if you’re using bundlers like webpack, Vite, or Parcel, they all offer out-of-the-box support for Autoprefixer. Rollup also supports Autoprefixer but requires a plugin. With the use of these tools along with Browserslist frontend developers don't even have to think about prefixes any more (once properly configured). An automation win for all developers!

Heavy Use of !important in CSS

In the early days of CSS the use of !important was a common practice for the enforcement of style overrides. It was often considered a "quick fix" for specificity issues, and used to fix visual inconsistencies in cross-browser scenarios or when using third-party CSS code. However, the overuse of !important is now widely considered a legacy practice that causes major issues with scalability, testability, and maintenance.

Can I still use it?

Yes, It can still be used in certain circumstances e.g. quick debugging, third-party overrides. But a rule of thumb I'd recommend is, if you are using !important for anything outside a small set of circumstances, you likely have bigger problems that need to be addressed first. As having predictable specificity in CSS is critical for the long term "health" of any web UI. Its use is a clear red flag that the project may be difficult to work on, as it’s a distinct frontend code smell.

That being said, If you ever do need to use it, you should clearly document the reason(s) why in the code, so future developers clearly understand why. Without clear documentation on why it is being used, a future developer is likely to remove this legacy CSS practice due to its reputation, potentially disrupting page styling elsewhere on the website.

OldIE hacks

For anyone lucky enough to have missed the joy of building websites for IE 5, 5.5, and 6 (the browser trilogy nobody asked for, released between 1999 and 2001), rest assured, that statement is absolutely, definitely not sarcastic at all. Promise! These versions of IE were just terrible browsers to work with. Microsoft in their infinite wisdom were following their own interpretation of the browser standards that were still in their infancy, and this often creating proprietary features that no other browser vendor followed or implemented.

Bundled with Windows XP, the popular IE6 unfortunately held back web progress for several years. Its standalone release for Windows 98, Windows ME, Windows NT 4.0, and Windows 2000 further cemented its widespread adoption and prolonged influence across the web. This forced Microsoft to innovate in the browser market, or face obsolescence, when a viable and revolutionary competitor finally emerged to finally move the web forwards! Initially launched in 2002 as Phoenix 0.1, a name symbolising its rise from the ashes of Netscape Navigator 9.0.0.6, this browser was renamed Firebird in early 2003, and later that same year became Firefox, the name it retains today.

And as an avid Firefox user for many years, I'm still delighted it is still around as it has its own HTML engine called Gecko. As the modern browser market is very WebKit / Blink dominated. Competition in this area is always a good thing to help drive innovation, and move the web forwards!

So, enough of the dusty old history books, what was so bad about IE6? Oh, dear, where do I even begin? It was less "web browser" and more "digital dumpster fire" with a whole host of issues and annoyances listed below. As for the "Can I still use it?" section, trust me you wouldn't want too! oldIE is thankfully about as common on the modern web as seeing a third-party script that improves web performance! Let us have a look at many oldIE issues in all their glory:

DOCTYPE fragility

If there was a single character before the DOCTYPE in IE6 it would trigger quirks mode (see the Quirks Mode Layouts section earlier in the post for a more detailed explanation). This even included newline, comment, space, or invisible byte characters. As you can imagine this was a nightmare for debugging. And if you weren't in full control of your HTML coming from the server, you were in for a bad time! Quirks mode essentially set the browser into the legacy rendering mode (mostly IE5 behaviour). If that wasn't bad enough, it would also interpret the Box model differently, causing chaos for layouts! Lastly CSS and layout behaviour would be inconsistent or in some cases, totally broken!

zoom: 1 hack

The zoom: 1 hack forced an element into the "hasLayout" rendering mode, which was a proprietary internal rendering concept in IE6 and IE7. For any element that didn't “have layout”, it could:

Collapse when floated
Break rendering of child elements
Overflow improperly
Fail to clear floats
Interrupt margin collapse

I distinctly remember using zoom: 1 a lot with float layouts, which was the main option for CSS layouts at the time. As mentioned above the element would then clear floating elements and also overflow elements correctly.

Underscore Hack

This was a pretty simple hack to target only IE6 and IE7 in quirks mode, any property prefixed by an underscore is ignored by modern browsers, but recognised by IE6 and IE7.

.selector {
	_width: 500px /* targets IE6 / IE7qm */
	width: 500px /* modern browsers */
}

Asterisk Hack

Another simple hack to target IE6 and IE7 that is ignored by modern browsers. It was especially useful for fixing box model issues.

.selector {
	*width: 100px /* targets IE6 & IE7 */
	width: 100px /* modern browsers */
}

Star HTML Hack

A slightly different hack that was added to the start of the selector rather than a property within a selector. It would only target IE6 in standards mode (not quirks mode).

html .selector { /* all browsers including IE6 */
	margin-left: 50px;
}
* html .selector { /* IE6 only - order is important */
	margin-left: 0;
}

Note: The order of the above code is important since IE6 could read both selectors but the second one "wins" due to its position in the cascade.

Child Selector hack

Another straightforward hack to target ONLY modern browsers (not IE6) was:

ul > li { /* not understood by IE6 & IE7 */
	color: rebeccapurple; /* "rebeccapurple" not understood by IE6 & IE7 */
}

IE6 & IE7 didn't understand the child selector, so it simply ignored it.

Note: the colour is also historically significant in the web community as the rebeccapurple CSS color keyword was added as a tribute to web pioneer Eric Meyer's daughter Rebecca who passed away of brain cancer at the age of six. Eric, if I’m ever lucky enough for you to see this post, please know that as a brain cancer survivor myself who has personally seen the impact this terrible disease has on friends and family, my heart truly goes out to you and your family. I am so sorry for what you’ve all been through. Rest in peace, Rebecca.

Double Margin Float Bug

This is a pretty aptly named CSS bug fix, as it does exactly what it says on the tin! Due to IE6's crazy box model interpretation it sometimes liked to double the margin on floated elements.

.floated-element {
	float:left;
	display: inline; /* fix the double margin bug in IE6 */
}

Peekaboo bug fix

IE6 sometimes threw its toys out of the pram when content changed on the page, in these cases it would either render them incorrectly or just make them disappear, hence the name peekaboo!

To fix it, you had to apply "hasLayout" to the element:

.buggy-element {
	zoom: 1; /* trigger hasLayout on the element to fix the peekaboo bug */
}

Transparent PNG fix

IE6 was unable to display the alpha channel (transparency) in PNG files, instead it just interpreted it as a grey, solid background. Combined with rounded corners on a non-solid background colour background, this was a real pain in the backside! Thankfully, IE6 supported CSS filters, allowing Microsoft to offer its proprietary AlphaImageLoader for PNG transparency. There were 2 methods to apply this filter:

.transparent-png {
	behavior: url("iepngfix.htc");
}

The .htc file contained JS logic that dynamically applied Microsoft’s proprietary filter to the PNG elements.

An alternative for background images was this:

.transparent-bg {
	background: none !important;
	filter: progid:DXImageTransform.Microsoft.AlphaImageLoader(
	src='image.png',
	sizingMethod='crop'
	);
}

Yep, how horrible is that! 🤮

But even if you solved the transparency issue your problems weren't over as its usage came with a few caveats:

AlphaImageLoader broke CSS background-position and background-repeat.
The fix didn't work on <img> tags unless additional .htc behaviour hacks were applied!
.htc files could be blocked fairly easily by companies with restrictive internet policies, so it wasn't always guaranteed to work. And if this happened to be one of your clients... well, there goes the whole design!
Lastly these hacks only worked in IE5.5 and IE6, thankfully IE7 supported transparent PNG's so these hacks had to be targeted at IE5.5 and IE6 only.

Lack of IE Developer Tools

I know the bugs above sound frustrating, but one of the biggest issues was the fact that IE6 and IE7 completely lacked any "sane" developer tools to help resolve these issues. Initially, developers had to rely only on alert() or document.write() for debugging! This was where the infamous [object Object] or [object] [object] came from when debugging JS in IE6 and IE7. If you had a JS error that you wanted to investigate, that was just about all the information you were given from alert(). Thankfully, Firebug Lite was released later by the Firebug Working Group. It wasn't developed by Mozilla and it wasn't a browser extension. It was a JS file that you could include in your page or as a bookmarklet to mimic the Firebug debugging tools. It was only later in IE8 that Microsoft included the first native developer tool in IE. You enabled it by pressing F12, it finally allowed developers to use the console functionality that we still use today. Interestingly, F12 to open the browser DevTools still remains the standard across most modern browsers!

IE Conditional Comments

Conditional comments were introduced into Internet Explorer in IE5, released in March 1999. They were introduced to Internet Explorer by Microsoft because IE had its own interpretation of HTML, CSS, and JS that diverged from W3C standards. This made it extremely difficult for Frontend developers to build consistent UI's that worked across all browsers, without resorting to complex and brittle CSS hacks as seen earlier in the post. Conditional comments allowed developers to have the best of both worlds, a fully W3C-compliant CSS file for modern browsers, and a CSS file that only applied to certain versions of Internet Explorer (that contained all the version-specific hacks!).

It's worth noting that each version of IE had its own specific hacks, so you'd often see multiple Conditional Comments in the <head> of a website after the W3C-compliant stylesheet (due to CSS loaded after the modern version taking precedence due to the cascade). The hacks were still there, but they were sectioned off into their own CSS file(s), ready to be removed when the browser was no longer used by the vast majority of site users. There were countless different ways to write conditional comments due to the version logic built into the interpreter. Note that modern browsers ignored them entirely as they were seen as standard HTML comments e.g. ().

Some example conditional comments can be found below:

<!-- Target only Internet Explorer 6 -->
<!--[if IE 6]>
  <link rel="stylesheet" href="styles-ie6.css">
<![endif]-->

<!-- Target IE 7 and lower -->
<!--[if lte IE 7]>
  <link rel="stylesheet" href="styles-ie7-and-below.css">
<![endif]-->

<!-- Target IE 8 only -->
<!--[if IE 8]>
  <link rel="stylesheet" href="styles-ie8.css">
<![endif]-->

<!-- Target IE 9 and above -->
<!--[if gte IE 9]>
  <link rel="stylesheet" href="styles-ie9-and-up.css">
<![endif]-->

<!-- Target any version of IE -->
<!--[if IE]>
  <script src="polyfills-for-ie.js"></script>
<![endif]-->

<!-- Exclude all versions of IE (i.e. target modern browsers only) -->
<!--[if !IE]> -->
  <script src="modern-browser-script.js"></script>
<!-- <![endif]-->

<!-- Combine conditions: Target IE 6 to 8 -->
<!--[if (gte IE 6)&(lte IE 8)]>
  <link rel="stylesheet" href="legacy-ie6-to-ie8.css">
<![endif]-->

Explanation of the Condition Syntax:

IE matches any version of Internet Explorer.
IE 6, IE 7, IE 8, etc. match specific versions.
lte = less than or equal to
gte = greater than or equal to
!IE = not Internet Explorer
- Note the use of the  to properly close in non-IE browsers. This was known as a downlevel-revealed comment.

As you can see from the complexity of the above examples it was fairly simple to target very specific and multiple versions of Internet Explorer. A significant drawback was the increased maintenance burden and the clutter they introduced within the sites' <head> tag.

Thankfully, Microsoft removed the parsing of conditional comments in IE10 and IE11 before they eventually introduced a whole new browser called Microsoft Edge. Edge initially used a proprietary rendering engine called EdgeHTML. However, the browser was subsequently rewritten to incorporate the same open-source engine as Google Chrome. This new version, based on Chromium 79, was released as Microsoft Edge 79 on January 15, 2020.

IE CSS Selector Limit

This issue, of all those detailed in this section, is arguably one of the most random, as well as one of the least noticeable! IE6 to IE9 had a selector limit of 4,095 selectors per stylesheet, now that may sound like a lot, but it was very straightforward to go over this limit, especially when grouping selectors. For Example:

/* This counts as a single selector */
.my-selector {
	margin: 20px;
}

That's all straightforward, but then you look at something like this:

/* This block counts as three selectors */
.button-primary, .button-secondary, .button-tertiary {
	margin: 20px;
}

Once you started grouping selectors for easier maintenance it became far too easy to hit the limit, especially on large websites. If you were a user of Bootstrap or Foundation at the time you could hit this limit unintentionally, without even knowing it!

And that brings me onto my next point: What happened when that limit was reached? Well... nothing really, IE just didn't parse any CSS beyond the 4,095 selector limit. Would it warn you that this was happening? Absolutely not!

Developers were often extremely fortunate if this issue was discovered through testing pages styled later in the stylesheet. Internet Explorer itself, however, would simply fail without any error messages or warnings.

And, to make it even more confusing: it would only impact the specific stylesheet that had exceeded the limit, not the page as a whole. For example:

<link rel="stylesheet" href="base.css">       <!-- 2000 selectors -->
<link rel="stylesheet" href="theme.css">      <!-- 4500 selectors -->
<link rel="stylesheet" href="overrides.css">  <!-- 300 selectors -->

In IE6 - IE9 this is what would happen:

base.css loads perfectly fine, it is under the limit.
theme.css only the first 4095 selectors are parsed, the rest are silently ignored.
overrides.csswould be load fully since it is under the limit.

This behaviour creates a partial styling issue. Elements relying on the theme.css file won't be styled correctly beyond the 4095 selector limit. In such cases, most pages will appear normal until a page attempts to utilise style selectors 4096 through 4500 from the theme file, at which point it will fail without warning. And of course if you were unlucky enough to be working with IE6 or IE7, then you had no developer tools to even debug the issue either!

Solution

So what was the solution? Well, with the invention preprocessors like Sass or tooling like Grunt, Gulp, or PostCSS they could automate the splitting of stylesheets at the 4065 limit.

Or another solution was to supply a simplified UI to IE browsers only and serve those CSS files via IE's Conditional Comments. But can you imagine the maintenance involved in updating multiple different stylesheets? Just for the slightest UI change!

The final approach involved reducing reliance on external stylesheets by inlining critical CSS directly into the <head> of the page, specifically for above-the-fold content (we’ll come back to that strategy later, as it’s not as relevant today). Even thinking about these different maintenance options, and their implications gives me a headache!

OldIE hacks Summary

As you can imagine CSS files around this time were a ~~bit~~ lot of a mess with all these random cross browser hacks and workarounds! Thankfully Microsoft recognised it was an issue so decided to implement conditional comments in IE5-IE9 to make this madness a little easier (in terms of organisation, not coding)

7. Markup of the Past

XHTML 1.1 and 2.0

I remember having a conversation with a friend about how he was converting his website into a new standard that had just come out, this was around 2001 and the new standard was XHTML 1.1. The most obvious difference at the time was the DOCTYPE at the top of the page source. From HTML 4.01 Strict:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
  "http://www.w3.org/TR/html4/strict.dtd">

to:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
  "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">

What did XHTML 1.1 aim to achieve?

The goal of this new standard was to modularise XHTML and enforce stricter XML compliance. Its key features were the fact that it was based on XHTML 1.0 Strict but split into modules for better reusability and extensibility. It also required documents to be well-formed XML, and it enforced stricter syntax than HTML like all tags must close, and all attributes must be quoted.

Unfortunately, for XHTML 1.1 it came with a number of limitations that doomed the specification from the start. These included, very little browser support for serving XHTML 1.1 as application/xhtml+xml, it also, more critically broke backwards compatibility in many real-world use cases. And lastly many developers continued to write XHTML but serve it as text/html which entirely defeated the point of writing XHTML in the first place!

Because it never gained wide browser support and required a very strict syntax, it eventually became obsolete and is now mostly of interest for historical or academic reasons.

What did XHTML 2.0 aim to achieve?

XHTML 2.0 was never officially released or used in any production browsers. Had it done so, this would have been the DOCTYPE:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 2.0//EN"
  "http://www.w3.org/MarkUp/DTD/xhtml2.dtd">

Development began on the specification in the early 2000s with the key goals its key goals being:

a clean break from HTML 4 and XHTML 1.x.
planned to introduce entirely new ideas like replacing <a> with <h> for links.
high up on its priority list was for it to be device-agnostic and semantically pure.

The XHTML 2.0 specification ultimately failed for the following reasons:

It lacked practical browser support and Implementation.
HTML5 (developed by WHATWG) gained real traction by improving the existing HTML and maintaining compatibility.
And the gigantic final nail in the XHTML 2.0 specification was its complete lack of backwards compatibility. It broke the entire ecosystem of existing web pages and tools.
The W3C officially halted XHTML 2.0 in July 2009 and shifted efforts to HTML5.

This then brings us onto the modern standard still in use today, and finally a DOCTYPE that was easy to remember! Look how developer-friendly it is:

<!DOCTYPE html>

It is so simple in fact it is case-insensitive and doesn't require a cumbersome URI to the Document Type Definition (DTD), which was there to define the structure, rules, and legal elements and attributes used on a page.

Inline JavaScript

There are 2 distinct methods of using inline JS in an HTML document. They both have specifications in the HTML Standard (WHATWG).

Inline Script Block

The inline block is defined in section 4.12.1 The syntax is simple and familiar. An example usage is as follows:

<!DOCTYPE html>
<html lang="en" class="no-js">
	<head>
		<meta charset="utf-8">
		<title>Title here</title>
		<script>
			document.documentElement.className = document.documentElement.className.replace('no-js', 'js');
		</script>
	</head>
	<body>
		<!-- The script can also be placed in the body too -->
	</body>
</html>

It's perfectly fine to use the inline script block in this way, and it in no way a legacy / outdated technique. But it does come with a few things worth considering before using it. Inline script blocks like this cannot be run async or defer, these attributes only apply when loading external scripts using the src attribute.

Can I still use it?

Yes, you can, but it comes with a few caveats. A script in this position in the <head> executes immediately and synchronously, and it will block page rendering until the JS code completes. So make sure you don't overload an inline script block as your website will pay the price in terms of frontend web performance.

Inline Event Handler or HTML Event Attribute

Inline event handlers are defined in section 3.2.6 and section 8.1 of the HTML Standard (WHATWG) Inline event handlers are now considered a legacy pattern in modern web development. This is due to several reasons, including the security risk posed by JS execution in the global scope, the potential for Cross-Site Scripting (XSS) vulnerabilities, and the way they clutter the code. An example of an inline event handler is below:

<!-- other page code here -->
<ul>
	<li><a onclick="alert('About clicked'); return false;" href="/about">About Us</a></li>
</ul>
<!-- other page code here -->

The simple example above captures the onclick event from the anchor, and instead of taking you to the about page which you would expect it simply brings up an alert box with the "About clicked" string.

There are other reasons why this technique is considered legacy too since they conflict with the principle of separation of concerns, and also make debugging, testing, and the implementation of accessibility best practices harder. Lastly, Content Security Policy (CSP) can also disallow inline event handlers unless explicitly allowed (another big security risk!).

Can I still use it?

No! Don't use this outdated technique to add JS interactivity to your page. Instead you should move towards external scripts and unobtrusive JavaScript. An example of which is given below:

<!-- other page code here -->
<ul>
	<li><a class="myclass" href="/about">About Us</a></li>
</ul>
<!-- other page code here -->

// assuming the element.myclass is already in the DOM
const el = document.querySelector(".myclass");
el.addEventListener('click', () => {
	// Do JS stuff here!
});

This assumes that the myclass element is already in the DOM. If it isn't document.querySelector will return null and result in a TypeError. The safest way to get around this would be to use the DOMContentLoaded event. An example of this given below:

document.addEventListener("DOMContentLoaded", () => {
  const el = document.querySelector(".myclass");
  if (el) {
    el.addEventListener("click", () => {
      // Do JS stuff here!
    });
  }
});

If you're thinking that's a lot of code just to add a click event to a single element! Then you'd be correct, hence why libraries like jQuery were so incredibly popular for event additions and basic DOM manipulation.

Document.write()

I must admit, I don't think I've every actually used Document.write() on a webpage, that's probably because I've never seen a sane reason to use it! It's a JS method provided by the browser’s DOM, that allows you to write HTML or text directly to the page. A simple example is given below:

<script>
	document.write('<h1>Hello, world!</h1>');
</script>

Wherever this code is placed in the page, it will simply output a h1 with the content of Hello, world! Now there's a reason I was so harsh on the method above, and that's because it comes with some horrible side effects. These include the following:

As with any Inline Script Block, it blocks page rendering until the content is written to the page.
It runs synchronously and can block scripts and other resources from loading efficiently.
Lastly, and this has to be the best (and most horrifying feature!). If used after the page has fully loaded, it can erase the entire DOM and replace it with whatever was passed to the method. This would be <h1>Hello, world!</h1> in the example given above.
It's important to consider the security implications of its usage as it is similar to eval()(MDN link in some ways (but not all). Both methods can enable cross-site scripting (XSS) if user input is injected without sanitisation.

When should it be used?

On the modern web the simple answer is never! It's best seen as a historical curiosity that some legacy systems may still use that haven't been modernised yet. It could also be used in elementary educational examples or demos. There are a number of much safer (and robust) modern alternatives that should be used instead. These include:

element.innerHTML (MDN Link).
element.textContent (MDN Link).
document.createElement() (MDN link) in conjunction with appendChild and insertBefore.
Modern frameworks or libraries for manipulating the DOM and updating the UI.

Fixed Viewport Meta Tags

Fixed viewport meta tags were used in early mobile responsive development. An example of what it looks like is below:

<head>
	<title>My Fixed width Mobile Site</title>
	<meta name="viewport" content="width=320, user-scalable=no">
</head>

In the example above the viewport meta tag is telling the browser that:

This website should be rendered at a fixed width of 320px
It should disable user scaling, so the layout is locked to a specific dimension, regardless of the actual screen size.

Why use this approach?

In the early days of mobile development, desktop websites were very difficult to view and interact with on small mobile screens. In order to "fix" this issue, developers often built mobile specific websites, that would sit alongside the desktop website. 320px was a popular width at the time because the first iPhone (iPhone 3G) had a 320px size screen. In order to maintain maximum control over the layout appearance on these mobile devices, developers frequently prevented users from zooming into these sites. These restrictions also helped avoid layout shifts when loading in dynamic viewport sizes e.g. device orientation (portrait vs landscape), zoom or pitch gestures, changes in browser UI elements (address bar or toolbars).

Why was this bad?

There were a number of reasons as to why this technique was bad. The number one being it was terrible for accessibility. A user with a visual impairment, on a mobile site that disabled pinch-to-zoom (user-scalable=no), had no way to read the site's content. Secondly, by dictating a screen width, you are harming both adaptability and making assumptions about a user's device. Devices come in all shapes, sizes, and pixel densities. Mobile, tablet, desktop, and every resolution in-between. There are literally an infinite number of screen dimensions possible, obviously many of those would be impractical for users beyond a certain range, but it's impossible to maintain all these fixed versions so this technique quickly become outdated. Lastly, fixed sizes can lead to performance issues, as they may cause unnecessary UI reflows and repaints when used with older layout methods such as tables or fixed-position elements.

Modern Best Practice

With the development of responsive design practices by Ethan Marcotte in 2010 (Responsive Web Design: A List Apart), fixed layouts quickly fell out of fashion, as with responsive design, you could develop a single UI that worked on all devices, no matter what their screen size or pixel density. One UI to rule them all! It comes with the huge advantages of much less maintenance for developers, and more importantly a huge usability for all users, no matter what device they are viewing a website on. The recommended viewport meta tag for modern websites is this:

<meta name="viewport" content="width=device-width, initial-scale=1">

This tells a user's browser:

to match the screen’s actual width (width=device-width).
to set the base zoom level (initial-scale=1).
to allow users to zoom the viewport (user-scalable=yes this is default if not set)

You may come across fixed layouts in legacy applications, and if you do, you should seriously consider:

Replacing fixed tags with scalable ones.
Refactoring CSS layout logic to use flexible grids, fluid typography, and media queries.
Ensuring accessibility standards are upheld, especially zooming support (up to 200% as specified in WCAG Success Criterion 1.4.4).

Web Safe Fonts Only (before @font-face)

Fonts are arguably the most crucial component of the web. Without them, there would be no content, and consequently, no internet. This fundamental importance explains the extensive nature of the CSS Fonts Module Level 4 documentation. Fonts present a challenge due to their vast variety and subjective nature. What one person finds legible, another may not.

Web-safe fonts are typefaces that are broadly supported and consistently rendered across most web browsers and operating systems, eliminating the need for users to install additional fonts. They are distinguished by three primary characteristics:

They come pre-installed on most devices across all operating systems (Windows, macOS, Linux, iOS, and Android).
They render consistently across browsers, devices, and operating systems.
If a font isn't available on a certain device, there's a viable alternative that can be used by default, this is called "fallback safety".

A common fallback font family using the font-family CSS property is displayed below:

.class {
	font-family: Arial, Helvetica Neue, Helvetica, sans-serif;
}

According to CSS Font Stack this combination of fonts is supported by 99.84% of devices on Windows, and 98.74% on Mac. Notice how it gives the browser a list of fonts, the primary choice being Arial, and if Arial isn't available then Helvetica Neue will be used, all the way down to sans-serif. This is basically saying "if none of the preceding fonts are available, then choose any sans-serif font on the device". This guarantees that a font will always be available on any device, so even though different fonts will be used depending on the operating system, the page content will still be rendered and readable for all users.

Common Web Safe Fonts

Arial
Times New Roman
Verdana
Georgia
Courier New
Trebuchet MS
Lucida Console

The issue with these fonts is that they are very limiting, especially for the Design community. Designers have very strong opinions on fonts, it is their "bread and butter", after all, so that's to be expected! For years, both developers and designers have strived to integrate all fonts, not just web-safe ones, into web development. In doing so many people on the web came up with different ways to allow them to use non-web safe fonts. These include the methods I mentioned earlier in the post:

As mentioned earlier all of these methods worked, but they had limitations, be that with Accessibility, Performance, Maintenance, Security, or SEO.

In order to mitigate thes limitations, a modern, standardised method was required for browsers to load custom fonts.

Enter @font-face

@font-face is a CSS rule that allows web developers to load custom fonts on a webpage. Unlike the methods listed above it's a native browser feature that brings typographic control to the web, while also preserving Accessibility, SEO, Maintenance, Security, and Performance (if implemented correctly).

The @font-face rule has a notable history, as it was initially implemented by Microsoft in Internet Explorer 4 in 1997. At the time it used Embedded OpenType (EOT) fonts. This was a proprietary solution by Microsoft and not a part of the CSS standard at the time, so adoption outside of Microsoft browsers was non-existent. It wasn't until the W3C developed and standardised the CSS Fonts Module Level 3, that browser support across different vendors started to improve.

Although CSS Fonts Module Level 3 work began in the early 2000s, true standardisation took time as browser vendors adopted open formats like TTF, OTF, and later WOFF and WOFF2. The CSS Fonts Module Level 3 was not released as a W3C recommendation until September 2018. The first CSS Fonts Module Level 3 working draft was published in July 2001.

Usage

So how do we actually use @font-face? Well, it's pretty straightforward:

Declaring the font

@font-face {
	font-family: 'MyFont';
	src: url('/fonts/myfont.woff2') format('woff2'),
	   url('/fonts/myfont.woff') format('woff');
	   /* other font formats here */
	font-weight: normal;
	font-style: normal;
}

Although you can define other font formats, this is no longer recommended, since the combination of WOFF and WOFF2 will cover all popular browsers. In fact, depending on your user analytics data, you may even be able to drop to only listing WOFF2, since it is now supported by 96.2% of browsers used on the internet according to Can I Use.

Using the Font

body {
	font-family: 'MyFont', sans-serif;
}

Here is where we apply the custom font to the page elements using standard CSS selectors.

IMPORTANT: note how sans-serif has also been set as a fallback e.g. later in the font list. This is best practice because we are loading an external font file to render the text on the page. If this font no longer exists on the server (or simply fails to load), users will be left with no text, since the external font is missing. This ensures that even if the custom font isn't available it will "fallback" to an appropriate web safe font.

Now there are a number of web performance points that should be considered when using web fonts, but I won't go into them here. Instead, I will point you towards Zach Leat's excellent "The Five Whys of Web Font Loading Performance" article from November 2018, and it also links to his Performance.now() conference talk on the same subject. Well worth a watch if you have a spare 46 minutes!

8. Tools and Workflow Relics

SVN (subversion, largely replaced by Git)

SVN (Subversion) is a centralised version control system that was widely used in the 2000s and early 2010s. It was the first versioning system I used at one of the digital agencies I worked at in the late 2000s. The memories that stick with me most about SVN are:

Every folder and sub-folder had an annoying .svn directory within them. This directory contains all the metadata needed by SVN to manage the versioned files.
Branching and merging in SVN was a painful experience!

Although in all honesty, I haven't used it in over a decade, so both these points may have changed and now be invalid? Actually, I doubt it for backwards compatibility with older SVN repositories.

The key word in the top paragraph above is "centralised". In a version control context that means that with SVN there's a single central repository that all version history and file management operations are built around.

In comparison, Git / GitHub are decentralised repositories. When you clone a repository onto your local machine you have the whole history of all the files, you can modify them while offline then synchronise with other developers code modifications once you're back online.

Legacy Development Practices

If SVN is being used in 2025, it could imply certain things about the codebase and teams working practices. These include:

The tooling being used is likely to be old (e.g. Eclipse plugins, shell scripts).
Continuous Integration (CI) / Continuous Development (CD) is likely to be very basic or missing entirely.
Due to the complexity of the branching and merging process in SVN, this type of workflow will likely be minimal if used at all!

Team Cultural Indicators

There are also red flags in terms of engineering culture if SVN is still being used. It typically indicates that:

The engineering team has a conservative engineering culture.
The team have a risk-averse attitude to change.
The team may have a backlog of technical debt that has accumulated over many years.
Recruitment of developers wanting to use SVN is likely to be challenging, as recent surveys indicate that SVN has 5.18% of the Version Control System (VCS) market share. It is second to Git that dominates with a 93.87% market share. This is also likely to impact retention of developers too, since Git / GitHub are the dominant tools in use in most industries (although, not all) in 2025.

What to look out for

Should you happen to encounter a project that still uses SVN for version control, you should:

Expect resistance to adopt modern workflows (e.g., GitFlow, CI/CD).
Investigate whether the tooling supports migration to Git e.g. git svn or if a full rewrite might be needed.
Evaluate whether SVN is tightly embedded in the build and deployment process.
Prepare yourself for recruitment and retention issues as mentioned above.

Migration

Assuming you've stumbled across a legacy project that uses SVN, and modernisation and migration is a goal for the project. It's worth knowing that:

SVN to Git migration tools exist (git svn or tools like SubGit), but edge cases can be painful.
You will likely need to retrain teams and completely refactor deployment automation.
Start with a small component or project to check see if modularisation is a feasible option.

Can I still use it?

TL;DR: Git / GitHub is the way to go, for modern web development best practice.

Chrome Frame

Chrome Frame was an ambitious project to bridge the gap between modern web standards and the many limitations of old IE versions (IE6 - IE8). It was released by Google as a plugin for IE in late 2009. It essentially embedded the Chrome browser engine into older IE versions, thus allowing the IE / Chrome hybrid to use modern web standards, modern JS frameworks, HTML5 features, all while still being compatible with an IE-centric environment. This proved particularly beneficial for larger enterprise organisations, which were reliant on older versions of IE due to legacy infrastructure and were unable to adopt more modern browsers.

While it sounded great in theory, unfortunately it came with a whole host of downsides, especially when it came to adoption. This included requiring admin access to machines in order to install the plugin, many companies locked-down the use of plugins due to the security risk involved in installation of 3rd-party code, lastly it introduced complexity for IT support teams and Quality Assurance (QA) teams due to the hybrid nature of the rendering engine.

Can I still use it?

No, as it was ultimately deprecated by Google in 2013, and support ended in 2014. The reasons for this were that web standards had improved and IE itself had improved with the release of IE9 and later. A significant change in browser releases was the move to "evergreen browsers". These browsers update automatically in the background, without user intervention. These browser releases were untethered from specific Operating system versions, apart from Safari being a notable exception.

Although Chrome Frame only saw limited success, it certainly helped initiate discussions on migrating from legacy browsers in large enterprise environments.

I distinctly remember when it was announced I thought it would solve all our IE problems (finally!). It was only when it was released that I realised there was no way it would be able to solve the issue because:

It was complex to install (e.g. required admin access)
The majority of users at the time using older versions of IE were likely neither technically capable nor even interested in installing it as a plugin.
1. Note: This isn’t meant to sound elitist, but at the time, most people would have likely identified the internet as the "blue ‘e’ icon" on their desktop. Outside the web development community, few knew (or cared) what a web browser was, let alone which one they used! And I'd say this statement is most likely true with the modern web too!

CSS Resets

Before we get into the details, what is a CSS reset? It's essentially a set of CSS selectors and properties that are used to "reset" all styles across browsers to a common baseline. Think of it as a solid foundation on which to build your website off. In theory if all browsers render elements identically from the start, then the site will be easier to build and maintain because all those nasty minor cross-browser CSS differences will have been dealt with, that's the theory anyway.

Just to be clear, CSS resets are still around, but they have evolved into something more forward-thinking, minimal, and only focussed on common pain points. The first CSS Reset to be released in January 2007 was Eric Meyer’s classic CSS Reset, it quickly became one of the most widely adopted resets to try to standardise styling across the modern browsers at the time, these were Internet Explorer, Firefox, and Safari. It did this by removing all margin, padding, borders, and fonts to a common baseline. It could either be included within your own CSS file, or added as a separate CSS file at the start of your CSS in the <head> tag. The order is crucial because you're establishing a standardised foundation. This allows subsequent CSS files to override these resets, either through direct duplication, leveraging the cascade, or by increasing specificity. For example:

/* Basic Reset of the body styling: Specicifity score 0,0,1 */
body {
	line-height: 1.5;
	font-family: system-ui, sans-serif;
	background: #fff;
	color: #000;
}

/* Here this selector is overriding the one above because it comes after it in the cascade: Specicifity score 0,0,1 */
body {
	font-family: "Comic Sans MS", Impact, sans-serif;
}

/* Here we are using CSS Specicifity to override the page background color: Specicifity score 0,1,1  */
body.colored-background {
	background: #ff0000;
}

This is why ordering your CSS correctly is best to do right at the start of a project. If you bring in a reset file at the end, it will either do nothing at all due to higher specificity CSS selectors before it, or it will completely undo lots of your styling, simply because it “wins” (by coming last in the cascade e.g. it’s the last CSS file loaded).

As mentioned above, CSS resets have evolved over the years. Normalize.css is a very common one in use on the modern web, as it works differently by preserving useful default browser styles and only fixing CSS styles that need to be fixed to maximise CSS consistency across modern browsers.

Other notable mentions are more modern, minimal resets that only focus on certain pain points in cross-browser rendering like box-sizing, responsive images, and font inheritance. These include Andy Bell's: A (more) Modern CSS Reset and Josh Comeau's: A Modern CSS Reset.

Kudos to the authors of CSS Resets! Their dedication makes CSS authoring significantly smoother for millions of developers across the world.

Hover-Only Interactions

Hover-only interactions are a legacy practice that suited desktop-only contexts but fail in today’s multi-device environment. An example of what a hover-only Interaction is:

.button:hover {
	background-color: #ff0000;
}

Hover-only Interactions come with the following issues:

Not accessible on touch devices: Touchscreens do not have a hover state. This means hover-only functionality becomes inaccessible on phones and tablets, leading to broken user experiences.
Not accessible on touch devices: Touchscreens do not have a hover state. This means hover-only functionality becomes inaccessible on phones and tablets, leading to broken user experiences.
Lack of fallback interaction: Many legacy implementations didn't provide alternative means (like a click or focus) to trigger the same behaviour, effectively hiding essential UI or functionality.
Keyboard accessibility problems: Hover interactions are not always accessible via keyboard unless explicitly paired with :focus or JS handling.
Poor progressive enhancement: Relying solely on hover effects often ignored the principle of progressive enhancement, especially when essential content was hidden using CSS unless hovered.
Inconsistent browser behaviour: Legacy browsers had quirks in how they handled hover states, particularly with complex layouts or when mixing JS and CSS interactions.

Modern Best Practice

UI's need to be device-agnostic and align with inclusive design principles. Hover-based interactions should be a supplementary interaction, not the primary interaction. In order to align with modern best practice you should:

Avoid hover-only interactions for essential functionality.
Use :focus alongside :hover, and consider adding :focus-visible to better support keyboard navigation.
Support click or tap events explicitly for mobile compatibility
Provide visible indicators or alternative access methods (e.g. always-visible menus on small screens)

An example in CSS is:

.button:hover,/* mouse users */
.button:focus,/* element focused via click or tab */
.button:focus-visible { /* user likely on keybord, or other assistive technology */
	 background-color: #ff0000;
}

The above can be simplified to avoid redundant styling as combining :focus and :focus-visible can sometimes cause overlapping or unnecessary duplication of visual effects. The recommended approach is to use the following as it keeps your styling clean and scoped, applying just what’s needed based on the user’s input method:

.button:hover,
.button:focus-visible {
	  background-color: #ff0000;
}

By avoiding redundant styling means you reduce:

Overlapping in CSS rules.
Maintenance complexity.
Risk of inconsistent behaviour between browsers.
Slightly less to download.

9. Legacy Web Strategies

Blackhat SEO

Blackhat SEO refers to a collection of techniques that people tried to use to manipulate search engine rankings, mostly in ways that violate search engine guidelines, especially for guidelines laid out by Google.

Intent

So why would people want to use Blackhat SEO? Well, its sole focus was prioritising rapid results over sustainable growth. Consultants selling these techniques were focussing on exploiting weaknesses in search engine algorithms, rather than creating genuine value for users. Being at the top of the Google results page was the primary goal, and companies were willing to try these techniques to get an edge on their competition. That was until search engines got wise to what was happening, and started to penalise sites that employed these techniques.

Examples

Let's look over a few outdated examples and how they worked:

Keyword stuffing: This was essentially stuffing as many keywords into a page in an attempt to trick the search engine into pushing a page to the top of the results for more search engine searches. So even if the keywords weren't at all related to the actual content, they were included anyway. Thankfully, search engines got wise to this tactic and cracked down on sites that used it.
Cloaking: This is where you show different content to search engines than you do to users. Search engines would be shown pages with detailed, keyword-rich content about a specific product in order to trick a search engine into ranking the page highly. But the page shown to users was minimal and mainly promotional with very little or no helpful information on the page related to what it was being ranked on.
Hidden text and links: This is the one I remember the most, using CSS or HTML to hide text on a page that was only intended for search engines. Think white colour text on a white background, it was that simple! It was also straightforward to spot, as you'd get pages where the scrollbar was huge, but the content on the page was very short. The overflow in the vertical direction was the hidden text that you could easily reveal by highlighting the text with your cursor!
Link farms and paid link schemes: This is where companies would create hundreds, or thousands of low-quality content linking back to a specific page, in the hope that the search engine would rank the page highly because of all the backlinks to it. There were (and most likely still are) whole business's setup that promised to get you to the top of search results by essentially spamming the web like this. If you ever had a WordPress blog without the Akismet plug-in, you'd see this rapidly! WordPress was highly vulnerable due to its support for Trackbacks and Pingbacks (XML-RPC). Have a look at any old unmaintained WordPress blog, and you are likely to see this. They were so common at one point that a new term called "splogs" (spam blogs) was created for them. Wired.com wrote a blog post all about them back in September 2006 "Spam + Blogs = Trouble".
Duplicate content: This is a simple strategy, copy another site's high-quality content and pass it off as your own. Thankfully, this now triggers de-ranking in search engines.
Automated content: Basically automating the production of low-quality and spammy content. I anticipate that this technique will see a resurgence, given the recent surge in AI tools.

"This is why we can't have nice things!" The phrase echoes in my ears as I recall all the techniques mentioned above.

Why are they Outdated?

There are a number of reasons why these techniques are no longer used:

Google and other search engines have significantly enhanced their algorithms to identify and penalise such manipulative tactics.
Modern SEO is more geared towards user-first metrics, content relevance, quality, user experience, and even web performance are now taken into account when ranking a web page.
Sites found to be using these Blackhat tactics are almost certain to be heavy penalised and may even be de-indexed completely from search engines.
Companies found to be using these tactics on the modern web are very likely to suffer reputational and credibility damage. Some sectors that are heavily regulated will likely have legal implications too.

Can I still use it?

Fortunately, these blackhat techniques are no longer effective on the web. They are detrimental to both users and the internet, yet some individuals persist in attempting to use them.

For example, Google now ranks web pages on their usability issues, one of these is called Cumulative Layout Shift (CLS). This metric measures the stability of a page while a website is loading, websites that "shift around" while loading (called "jank") aren't scored as highly as those that are more stable.

I recently saw a new SEO technique using JS that would mask an entire page with a transparent element in order to trick the browsers CLS metric into thinking the page was completely stable. After page load this element would be deleted and the page could be interacted as usual. Basically, a modern day cloaking technique aimed at improving a pages Layout Instability API score.

So yes, it still happens and "this is why we still can't have nice things!".

“Above the Fold” obsession

The concept of "above the fold" is an outdated technique. The fold's position is not fixed; rather, it varies depending on the device used to view a web page. If we take this to the extreme. Viewing a website on a desktop widescreen device vs a mobile device, there's never going to be a common "fold" in this situation. Consider the vast number of device widths, ranging from a large desktop widescreen to a mobile device—literally thousands along the x-axis. If you then factor in the viewport height (y-axis), you're looking at millions of possible viewport permutations. Past assumptions are no longer true:

User behaviour: Users scroll instinctively now. The old belief that users don’t scroll is no longer valid.
Web performance evolution: Modern performance metrics (like Largest Contentful Paint (LCP) and Interaction to Next Paint (INP)) reward real user-perceived speed, not just fast above-the-fold content.
Lazy loading and streaming: The web has moved towards prioritising meaningful content dynamically, rather than front-loading everything visible “above the fold”.

Can I still use it?

It depends. “Above the fold optimisation” is an older performance technique that focuses on rendering the visible portion of the page as quickly as possible. When used thoughtfully, it can still improve perceived load speed, especially in critical user flows. However, relying on it too heavily can narrow the focus to just a fragment of the overall experience.

Today, the more effective and sustainable approach is to optimise for end-to-end, user-centric performance. This includes not only what appears first on-screen, but also how quickly the page becomes usable and interactive. A strategy focused on delivering a consistently fast page experience will naturally improve the content visible without scrolling, regardless of the device.

Superseded compatibility approaches

Graceful Degradation

The technique of graceful degradation involves building a website to take advantage of all the modern features of a browser, and once completed add "fall backs" for browsers that don't support modern features.

Examples

An example of graceful degradation is: a developer builds a website where the main layout is using CSS Grid. But if a browser doesn't support Grid it will "fall back" to a simpler layout system like Flexbox or even a float-based layout (depending on the site's browser support requirements).

Another example is a feature-rich, JS-enhanced input form may fall back to a basic HTML form if JS is disabled or fails to load, for example, due to a poor or unstable network connection. In this case, core functionality (such as form submission) remains available, even though advanced features (like real-time validation or dynamic UI elements) are unavailable.

Why is it Legacy?

Graceful degradation is increasingly being considered a legacy approach in modern web development as it has largely been superseded by progressive enhancement which takes the opposite approach.

Why is it outdated?

This assumption of a modern baseline fails to acknowledge the true diversity of browsers and devices currently in use by users. Furthermore, comprehensive testing is challenging due to the difficulty of covering all scenarios involving older or limited browsers. Third, adding "fallbacks" further increases the complexity of an already intricate full-featured initial build. Most importantly, Graceful Degradation negatively impacts accessibility and resilience. Pages employing this technique frequently fail in low-capability environments, such as older browsers, devices, or poor connections.

Can I still use it?

There are a few scenarios where it may still be useful, these include:

Legacy enterprise environments: for example a company that mandates the use of older browsers like Internet Explorer. A notable example of this is Banks and other financial institutions in South Korea. For a country, that ranks 12th on the internet adoption rate for its citizens (97.4% it 2025), it's a pretty surprising legacy issue they are still trying to tackle!
Modernisation: If a website is in a transition phase of being modernised, and it still needs to support older browsers for a limited period.
Non-critical enhancements: If a site has non-critical enhancements like animations or media features that are optional and don't impact access to the site's core content.

What should I use instead?

Progressive Enhancement is now the preferred approach for modern web development, offering a more robust, inclusive, and future-proof way to build websites and web applications. While Graceful Degradation was a useful technique for older browsers, it has now been superseded.

Browser Sniffing

This is the practice of detecting more information about a user's browser, like its specific version number or the operating system it is currently running on. Once detected a developer can use this information to “fork” their code, e.g. make decisions as to what bug workarounds should, or shouldn't be applied. Or even tailor the user experience for a specific version of a browser. Two very common uses of this technique in the past were to redirect a user to the mobile version of the site (when mobile and desktop sites built separately), or even blocking the usage of a site on "unsupported" browsers. An example of how you'd do this in JS is below:

if (navigator.userAgent.includes('Chrome')) {
  // Apply Chrome-specific behaviour or simply block other browsers if you are feeling malicious
}

This code highlights a significant problem with browser sniffing, demonstrating why it's considered an outdated technique. In the code, the whole functionality is decided by the fact that the User-Agent variable in the browser happens to include the string "Chrome". But what happens if Google one day decides to change this string to lowercase "chrome", or even change it completely? Well, the code depending on this detection will break!

Now, you could modify the above code to tackle the case issue like so:

if(navigator.userAgent.toLowerCase().includes('chrome')){
  // Apply Chrome-specific behaviour
}

But as you can see, this has only made the code more complex and fragile.

It's also worth mentioning that this code won't do what you expect it to either, as all Chrome based browser will return true. For example:

Google Chrome
Microsoft Edge (Chromium-based)
Opera (also Chromium-based)
Brave
Vivaldi

At the time of writing each of the User-Agent strings for the above browsers contain: Chrome/115.0.0.0 (as well as other information, that I have removed for the example).

All contain "Chrome" in their User-Agent, so will all run the code.

What's worse is that Chrome on iOS will return false and not run the code. On iOS, all browsers, including what appears to be Chrome on the home screen, are actually forced to use WebKit (Safari). Consequently, "Chrome" in this instance isn't truly Chrome and is not reflected in the User-Agent String.

Other issues

Fragility isn't the only issue seen when using this technique. It can also:

add a maintenance burden for developers, as this logic will need to be updated as browsers evolve.
create browser feature mismatch, two versions of the same browser don't always support the same features.
cause accessibility risks leading to user exclusion. A user on a less-common browser or assistive technologies, could inadvertently receive a degraded experience, or even blocked completely.

Can I still use it?

Realistically, no, you should aim to avoid browser sniffing entirely and instead of asking which browser a user is using, ask what can their browser do? Essentially, you want to be detecting the features that the user's browser can support. For example, to detect if a browser supports the Service Worker API, you can do this:

if ('serviceWorker' in navigator) {
  // The users browser supports the 'serviceWorker' API, so do Service Worker stuff!
}

Browser Sniffing Summary

In summary, browser sniffing is a legacy technique that should be avoided on the modern web. In order to create a more resilient and inclusive web, you should use Feature Detection, Graceful Degradation, and Progressive Enhancement instead.

Modernizr

I was a big fan of Modernizr (with its very Web 2.0 name!). For readers who've not used or heard of it, Modernizr is a HTML5 and CSS3 feature detection library, It does this via browser feature detection, rather than browser user-agent strings, which can be unreliable and misleading. Modernizr actually tests to see if the browser being used supports a whole host of features.

It was released in 2009 at version 1.0, since then, it has had 27 releases, and 300 contributors. So how does it work, and how exactly do you use it? Here's an example of how it detects flexbox support in a user's browser.

// Adds a new test to the Modernizr object under the key 'flexbox'
Modernizr.addTest('flexbox', function () {

  // Create a new HTML div element to test CSS properties on
  var testElement = document.createElement('div');

  // Attempt to set the display property to 'flex'
  testElement.style.display = 'flex';

  // Check if the browser retains the value 'flex' for the display property
  // If supported, the style will remain 'flex'; otherwise, it may remain empty or be changed
  return testElement.style.display === 'flex';
});

And here's how you would use that in your website. There are 2 methods for how you use it:

CSS

Modernizr adds a class to the <html> element, in our case above it would be:

<html class="flexbox"></html> <!-- If the browser supports it. -->
<html class="no-flexbox"></html> <!-- If the browser doesn't supports it.

Thus, you could hang any styles you needed off these classes and tweak the CSS layout for both scenarios, flex support and no flex support.

JavaScript

The other method is to use it in JS by examining the Modernizr object:

if (Modernizr.flexbox) {
  // Flexbox is supported, do Flexbox JS stuff here
} else {
  // Flexbox is not supported, no non Flexbox JS stuff here
}

The great thing about Modernizr is that even thought it supports over 250 feature tests for HTML5, CSS3, and JS APIs, it allows developers to create custom-builds that only detect the features they intend to use. This improves performance in 2 ways:

Less JS to download, parse, then execute.
Fewer bytes sent over the network.

Use cases:

Applying fallback CSS or JS for unsupported features.
Progressive enhancement strategies.
Conditional polyfill loading.
Responsive design tweaks based on feature support (via JS)

Can I still use it?

It's still useful, but it isn't as essential as it once was, this is because:

Browser standardisation has vastly improved since 2009.
Native CSS now support feature queries (@supports).
Feature detection is often built into some JS frameworks.
Support for ancient browsers has declined.

Ironically, Modernizr, despite detecting over 250 features, does not detect JS support itself. This isn't a problem, though, because if you require this functionality it can be added via a single line of JS in the <head> of your page:

<!-- Default setup -->
<html class="no-js">
<head>
  <script>
    document.documentElement.className = document.documentElement.className.replace('no-js', 'js');
  </script>
</head>

By default, the script assumes that the browser doesn't support JS (class="no-js"). When it gets to the inline script tag, the script executes. And as this proves that JS is supported it swaps the no-js class to a js class that can then be used in CSS styling, as you would any other Modernizr CSS class as demonstrated above.

10. Tests and Standards of Yesteryear

Acid2 and Acid3 Tests

The Acid2 and Acid3 tests were really clever ways to test a browser's compliance with the ever-evolving rendering standards at the time. They were both created by the Web Standards Project (WaSP)which was founded in 1998, when the web was a battleground of two main browsers:

Microsoft (Internet Explorer 4 at the time)
Netscape (Netscape Navigator 4.05 at the time)

The Web Standards Project, aimed to promote web standards that made development simpler, more accessible, and future-proofed, by working closely with browser vendors and development toolmakers to achieve this. When the team posted their final blog post in March 2013, their mission was largely complete. It resulted in successfully getting browser vendors to support the standards set by the World Wide Web Consortium (W3C). As of 2025, the W3C remains active in setting new standards to ensure the web continues to support communication, commerce, and knowledge-sharing for all, with a strong focus on accessibility, diversity, and inclusion.

Acid2 (2005)

The Acid2 test was created to test compliance with HTML 4.01, CSS 1 & 2, and PNG rendering standards (e.g. alpha transparency). It did this by focussing on the following key areas of browser rendering:

Box model
Absolute and relative positioning
Float behaviour
Table layout
PNG alpha transparency
Data URLs

This was achieved through a highly innovative browser test: creating a simple cartoon face, similar to a smiling emoji. Depending on how compliant the browser was, influenced how well the face was rendered. It's much simpler if I just show you!

Rendering reference

This is what the output of the test is supposed to look like across all browsers.

Internet Explorer 6

This "face," rendered with IE6, appears as if the individual suffered a severe accident!

Netscape 4.8

Netscape performed similarly to IE6 at the time.

Mozilla Deer Park Alpha 2 (later Firefox 3)

Finally a browser that actually rendered a face!

There are far too many variations and versions of browsers around at the time to list here, but if you are interested in how other browsers rendered the Acid2 test check out this ancient blog post by author Mark "Tarquin" Wilton-Jones. Mark, thank you for safeguarding this significant and captivating piece of web standards history!

Acid3 (2008)

After the success of the Acid2 browser test in putting pressure on browser vendors to improve standards support in their browsers, it was decided by the WaSP team to create another browser test, this time focussing on a different set of browser technologies. These included:

DOM Level 2 and 3
ECMAScript (JS) behaviour
CSS 3 selectors
SVG rendering
Data URIs
Animation timing and rendering
WebFonts via @font-face

Rendering reference

The Acid3 test took a more traditional testing route and simply scored a browser taking the test between 1 (poor support) and 100 (perfect support).

In my opinion, the Acid3 test, while practical and easy to interpret, lacked the entertainment value of the Acid2 facial disfigurement test!

The release of the more challenging Acid3 test coincided with a surge in browser competition, particularly among Firefox, Safari, Opera, and Google Chrome (which was released later in 2008).

Scores

At the time of release (March 2008) the Acid3 scores for each major browser were as follows:

IE7: 12 / 100.
IE8 Beta 1: 18 / 100.
Firefox 2: 50 / 100.
Firefox 3 Beta 4: 71 / 100.
Opera 9.5 Beta: between 60–70 / 100.
Safari 3.1: between 75–90 / 100.
Google Chrome: Not yet released in March 2008.
Google Chrome 0.2 Beta: first release: 77 / 100.
Google Chrome 1.0: 100 / 100.

Google Chrome quickly improved its Acid3 score shortly after its initial release. This rapid improvement was mainly due to its use of the Safari WebKit engine, which already scored 75–90 out of 100 at the time.

Legacy

Neither test is maintained or relevant to the modern web, but they played a key role in pushing browser vendors toward better standards support. Today, browser compliance is measured using the Web Platform Tests (WPT), a much broader and actively maintained suite developed by the vendors themselves with input from WHATWG and W3C.

11. What Still Matters – Progressive Enhancement

Not legacy but often forgotten

Congratulations! You've made it! After discussing countless legacy approaches and techniques best left in the past, you've finally arrived at a truly timeless and Incredibly important methodology. More than two decades after Steve Champeon and Nick Finck introduced it in their talk "Inclusive Web Design For the Future” at SXSW in 2003. The Progressive Enhancement (PE) methodology remains one of the most robust and future-ready methods for modern web development.

What is Progressive Enhancement?

There's a ubiquitous diagram that is always shown whenever PE is mentioned in a blog post. And this blog post will be no different in using it, as it actually explains the concept incredibly well.

Here we have the well known Progressive Enhancement pyramid.

HTML

The HTML is at the bottom of the pyramid as it gives the website a solid foundation on which to build off. The HTML layer is the most resilient layer in the web development stack. Without the HTML there is no content, no links, no images, no website! This layer is by far the most important layer in the pyramid. Just to give you an idea of how resilient HTML is in web browsers, let's take a look at the very first website. Back on Tuesday, August 6, 1991, Sir Tim Berners-Lee, the inventor of the Web, published the very first website! Now, this statistic makes me feel ancient! The first website was published almost 34-years ago, at the time of writing! And what do you notice about the page? Well, most importantly, it still renders correctly, and the content can still be read perfectly well after all this time. If you take a peak at the page's source code, you will notice a few oddities, like:

The complete lack of a DOCTYPE, and no <head> tag.
No links to external stylesheets or JavaScript (they weren't invented yet!)
Anchors existed to link to other pages, but they had a strange NAME=[integer] attribute.
All elements were written in uppercase, e.g. <HEADER>, <BODY>, <TITLE>, <H1>.
Lack of Semantic Markup. This was to come later once the WWW had matured.

To put it into perspective, this website has been around longer than nearly half of the entire global population, that’s over 4 billion people younger than this single page on the internet! Which other digital format on the planet can boast that form of robustness and ease of accessibility? Just think about all the storage media formats that have come and gone in that time:

5.25-inch floppy disk (the disk that was actually floppy)
3.5-inch floppy disk (the 3D "save icon")
CD-ROM / CD-R / CD-RW
MiniDisc (data version)
CompactFlash (CF)
Zip disk
Jazz drive
DVD-ROM / DVD±R / DVD±RW
Blu-ray
HD DVD

The list goes on… The point is clear: if you need a long-term, reliable storage solution that just works, plain HTML on the web is hard to beat! FYI: Of course, these web assets ultimately reside on physical hardware in data centres, but that’s not the point. What matters is the resilience and accessibility the web platform offers, regardless of the underlying infrastructure.

CSS

The second layer in the pyramid is the CSS or the Cascading Style Sheets. When the World Wide Web (W3) was first invented back in the early 90s, CSS simply didn't exist. It wasn't until 1996 / mid-1997 that browsers started to support the CSS Level 1 Specification. The browsers at the time were Internet Explorer 3 and Netscape Navigator 4, both had partial (and mostly buggy implementations). Up until this point the web had been completely "naked" in terms of design. Just pages full of text, images, and the odd animated GIF. Nothing at all like the modern web we see today.

CSS constitutes the second layer of the pyramid because, frankly, it is a "nice to have." Browsers are equipped with default stylesheets (as previously discussed in the CSS Resets section), which enable HTML content to display correctly and remain readable even in the absence of a website's custom styles. A company or brand must ensure their CSS is available for browsers to download so that their website renders correctly. Without it, many users would assume the site is broken, especially given how modern websites are expected to appear. But in the unlikely event the CSS fails to load, users will still receive the HTML content in a perfectly readable format. While it may appear unappealing, it remains fully functional across all current (and all future) web browsers and assistive technologies.

The beauty of Progressive Enhancement lies in establishing a foundational layer (HTML) and then progressively adding desired features. This method ensures that if any subsequent layer fails, the underlying content and functionality remain accessible to users.

A prime example of this browser feature in action is, since April 9, 2006, CSS Naked Day has been observed. Where for a 50-hour period, website owners disable their site's CSS, allowing users to experience the semantic HTML without styling. It started as a push for web standards and semantic markup, and gave site owners an excuse to flaunt their sexy <body>. Gotta love a good HTML pun!

JavaScript

The final layer of the pyramid and the final piece of the Web stack puzzle is JS, this is the interaction layer that is added to a site last after the foundation (HTML) and design (CSS) have been added. It's difficult to believe that just three technologies, all of which have been discussed in this section, form the entirety of the web. There is truly nothing more to it than these three foundational components. Ultimately, the output of both frontend and backend development invariably consists of standard HTML, CSS, and JS. Although a multitude of tools and languages are available for web developers to use, with endless paths to choose from, they eventually all lead to identical HTML, CSS, and JS as their final output. It all comes down to using the right tool and technology for the job!

JS is deliberately placed as the final enhancement layer in the pyramid. This is not incidental. JS, while powerful, is the least resilient layer in the web stack. Its execution depends on multiple fragile components:

the network
the parser
the runtime environment
the integrity of the code itself.

A single misplaced character, for example, an errant semicolon or an undefined variable, can render entire swathes of interaction inoperable. This fragility is not a hypothetical risk. It manifests regularly across production environments all over the web, particularly where sites are heavily reliant on client-side code for core user journeys.

Progressive Enhancement Summary

The modern web has increasingly drifted away from the principles of Progressive Enhancement, often placing JS as the foundation rather than the finishing touch. Single Page Applications are a prime example, where even basic navigation and content rendering require full JS execution. This inversion of the pyramid not only risks total inoperability in degraded environments but also introduces avoidable accessibility and performance issues.

From a resilience and user experience standpoint, over-reliance on JS creates brittleness. Unlike HTML and CSS, which both degrade gracefully, JS fails noisily and catastrophically. If a CSS file fails to load, a page might look plain but will still remain usable. If a JS bundle fails, the entirety of the website's features may be lost, with little to no fallback available.

The web’s reach includes users with:

unreliable networks
older devices
constrained data plans
assistive technologies

A heavy dependence on JS frequently excludes these users or significantly worsens their experience. Progressive Enhancement is not about supporting “no JavaScript” users as a niche edge case. It’s about ensuring a robust baseline that works for everyone, every time, demonstrating empathy for all users regardless of how they access the internet.

While JS is a vital tool in a web developer’s toolkit, it must be handled with care. Its position at the top of the Progressive Enhancement pyramid reflects its power, but also its fragility. It should be used responsibly, with the awareness that its failure often leads to a broken experience. True resilience comes from building upwards from stable foundations, not downwards from brittle interactions.

Importance in government services

Having worked at GDS for 6-years, I can't tell you how many times I had to defend the frontend communities stance on Progressive Enhancement! Thankfully, it's all written in black and white in the Service Manual for all to read. However, some departments and developers found ways to work around the methodology or opted for alternative approaches. This was most likely driven by two things:

A team had made significant progress with their JS dependent service and were expressing concerns about meeting the requirements for their future service assessment(s).
Some Frontend Developers in the department were enthusiastic about adopting the latest client-side frameworks, with less emphasis on assessing their maturity or suitability for the service, and its users.

For point 1 it always amazed me that teams were able to get so far into prototyping for it to become an issue. As depressing as it may be to me, maybe the Service Manual and its guidance isn't as well-known across government as I'd like to believe?

For point 2, I 100% get it, new technology on the web is fun to play with and also to have on your CV / Resume! The real question is whether this new technology is truly the right choice for a critical public service that every UK taxpayer depends on and has a fundamental right to access?

Technology Suitability Check (Progressive Enhancement Focus):

Does the technologies core functionality allow the service to work without JS enabled?
Can the service still function reliably on low-powered or older devices when using this technology?
Is the final output from the technology easily accessible and usable with assistive technologies, regardless of the device used?
Does the technology degrade gracefully in poor network conditions, such as on a 3G connection or in rural areas?
Are all critical user journeys still functional when JS fails to load, or is blocked?

If the answer to any of the questions above is "no", then the technology probably isn't a great fit for a public service that needs to be maintained for years (or even decades!).

The last point in the list above is incredibly important:

Are all critical user journeys still functional when JS fails to load, or is blocked?

I think this is where there was a lot of misunderstanding in terms of Progressive Enhancement in government. I continually strive to highlight that a JavaScript-only journey doesn't require a direct 1-to-1 correlation with its progressively enhanced foundation. As long as for each user journey a user can complete their task, quickly and easily, then the use of JavaScript is fine.

For example, consider a feature-rich JavaScript dashboard built to enter user data into a backend database. If a simple HTML form with a submit button can achieve the same outcome (which it often can), then the dashboard is acceptable only if the HTML form provides a reliable fallback for situations where JavaScript is unavailable, such as when it fails to load due to a limited data plan, a poor connection, or a low specification device.

During a service assessment, the key question is whether the dashboard meaningfully improves data entry and user interaction, or whether it exists purely for the sake of using new technology. Adopting new tools without clear justification is not acceptable for a government service. I consider such an approach to be driven by a desire to boost a developer's CV or LinkedIn profile. E.g. CDD (CV-Driven Development) or LDD (LinkedIn-Driven Development).

12. Lessons for the Future

What these legacy practices teach us today

If there is one takeaway from this post, it is that Frontend Development has never stood still. What counts as best practice today can feel outdated or even “legacy” tomorrow. That constant state of reinvention is what first drew me in back in the late 90s, and it is what continues to excite me now. Backend development always seemed a little too steady, too predictable. Frontend, on the other hand, has always lived on the edge of change.

But what is different today is the scale of change ahead of us. With the rise of Artificial Intelligence (AI), we may be standing on the edge of a shift as significant as the birth of the modern Web itself. Just as the early internet reshaped how we live and work, the combination of AI and the Web could redefine what it even means to build, design, and interact online. The coming years are not just going to be interesting, they could mark a turning point in the history of our craft.

Applying lessons to modern frontend work

Core principle

Choose optimal solutions for the enduring parts of the stack: HTML, CSS, and JavaScript are the stable contract. Prioritise the most straightforward and maintainable approach to delivering clean, accessible HTML, efficient CSS, and lightweight, well scoped JavaScript. The further you move away from those three, the harder accessibility, maintenance, and performance become. Let the browser and the web platform do the heavy lifting.

Practical rules to work by

1. Start from the Web Platform

Prefer native elements and platform features before adding libraries. Use form controls, semantic HTML, CSS layout, media queries, inert, details and summary, dialog, fetch, URLPattern, IntersectionObserver, and Web Components where they fit.

2. Progressive enhancement as a default

Deliver meaningful HTML first, enhance with CSS, then layer JavaScript for interactivity. Critical journeys should still work when scripts fail or load slowly.

3. Ship less code

Adopt a dependency diet. Each abstraction must earn its keep through measurable value. Small utilities over frameworks by default. If a framework is chosen, configure it to output lean HTML, CSS, and JS.

4. Accessibility first, not last

Use semantic structure, proper labels, roles only when needed, visible focus, real buttons and links, reduced motion preferences, and test with keyboard and screen readers. Performance 100% is an accessibility feature.

5. Performance budgets and baselines

Set budgets for bundle size, interaction latency, and memory. Track Core Web Vitals from real users. Fail builds that exceed budgets. Optimise for first input delay, input responsiveness, and low CPU use on mid-range devices.

6. Keep the build simple

Prefer standard tooling that converges to web standards. Use the minimum build steps required. Long pipelines increase failure modes and slow iteration.

7. Design for resilience

Favour server rendering for first paint, hydrate only what is interactive, cache well, and handle partial failure gracefully. Make error states explicit.

8. Document the escape hatches

Where you choose abstractions, document how to reach the underlying HTML, CSS, and JS. Future teams should be able to debug without learning a bespoke stack.

9. Measure before you change

Add observability. Use Real User Monitoring (RUM) to guide work. Optimise the slowest real user paths, not synthetic microbenchmarks.

10. Plan for upgrades

Last, but not least, prefer tools with clear deprecation policies and migration paths. Avoid lock in. Isolate framework code behind simple boundaries so you can replace parts without rewriting the product.

A quick decision test

Can this be done with native HTML or CSS alone?
If not, can a few lines of vanilla JS do it without a dependency?
If not, does a library reduce long term cost and keep output close to the platform?
If a framework is still justified, can it produce accessible HTML by default and degrade gracefully?

Closing thought

Technologies come and go, but the contract with the browser remains. Choose the simplest path that produces high quality HTML, CSS, and JavaScript. The closer you stay to the platform, the easier your product will be to maintain, to make accessible, and to run fast at scale.

Post Summary

I have to admit, my posts always seem to take on a life of their own and end up being longer than I plan. Concise writing might be a goal for another day. If you made it all the way here, congratulations, you’ve officially joined the “end of post club”! I really hope you found this journey as enjoyable to read as it was for me to write. Revisiting these ideas was a real trip down memory lane, and it reminded me of things I hadn’t thought about in years.

Your thoughts and feedback are always welcome. If I’ve overlooked a method or technique you think deserves a mention, let me know and I’ll happily credit you in the changelog. Thanks again for sticking with me to the very end, and if you’d like to share your thoughts, you can do so here.

Post changelog:

26/08/25: Initial post published.
27/08/25: Fixed number ordering of headers.
27/08/25: Added Table of Contents for easier navigation!
28/08/25: Added Silverlight and Java Applets to the post (Thanks to an AI hallucination, regarding browser plugins from the past)
17/09/25: Thanks to Sven Kannengiesser for drawing my attention to the use of ‘here’ anchors. I have now revised them to be contextual and accessible.

Configuring your Content-Security-Policy on your development environment in 11ty

2025-02-04T00:00:00Z

This is just a short post to discuss how I improved my Content-Security-Policy (CSP), on my local development environment in 11ty. This is essentially a follow-on post from my Securing your static website with HTTP response headers I wrote last year.

Note: this post isn’t exclusive to 11ty—it’s just the static site generator I use for this site. The code can be easily adapted for any static site running on a Node.js-based server—or practically any local web server!

Why is this useful?

<rant>It's incredibly useful for me because my Cloudflare build currently takes 7 minutes and 30 seconds to complete! I've no idea why it takes this long, and as of yet I've had absolutely no response from Cloudflare on why this is! What I do know is that something happened on the 19th December 2024 between 12:33AM and 12:19PM because my Cloudflare Pages build time increased by a massive 460%! See the screenshots below for proof of that fact:

Before

Note the date, time, and git hash in the screenshot above: 12:33AM December 19, 2024, hash efd9011. Now let's compare it to the screenshot below:

After

So in this screenshot above, we have the same git hash: efd9011, but 12 hours have passed, it is now 12:19PM December 19, 2024.

Clearly my code hasn't changed because the git hash is the identical and nothing else has been committed, but for some reason my build time has increased by 460%! It's my guess that something changed on Cloudflare's side in this 12-hour period to drastically increase the build time. Even though I posted about it on the Cloudflare Discord server and the Cloudflare Community Forums, I've had no response to tell me what was changed and if it is possible to fix it. So if anyone from Cloudflare is reading, I'm begging you, please, can you investigate what changed in that 12-hours and tell me if there's something I can change to drop my build time back to 1 minute 30 seconds!</rant>

So you're probably asking: "What on earth has any of this got to do with a "Content-Security-Policy" response header? Well, a 1-minute 30-second rebuild time to update my site's CSP in the 11ty _headers file is a reasonable time to wait to test for issues. But waiting 7 minutes 20 seconds for every little change just isn't efficient or workable! Quite frankly it's incredibly frustrating. And if you've ever worked on writing a CSP before you will realise how convoluted and unreasonably lengthy they can become! There had to be a better way to do this, and there is! So let me take you through the node.js / 11ty solution I used below to make my latest CSP changes.

The code

Let's have a look at the code:

// This is my site's Content Security Policy.
// Modify this CSP, don't just copy / paste it! It will break your site!
// You can also use `var` and `let` depending on your coding syntax, they all work
const CSP = `
base-uri 'self';
child-src 'self';
connect-src 'none';
default-src 'none';
img-src 'self' https://v1.indieweb-avatar.11ty.dev/;
font-src 'self';
form-action 'self' https://webmention.io https://submit-form.com/DmOc8anHq;
frame-ancestors 'self';
frame-src 'self' https://player.vimeo.com/ https://www.slideshare.net/ https://www.youtube.com/ https://giscus.app/ https://www.google.com/;manifest-src 'self';
media-src 'self';
object-src 'none';
script-src 'self' https://ajax.cloudflare.com https://giscus.app/ https://www.google.com/ https://www.gstatic.com/;
style-src 'self' https://giscus.app/;
worker-src 'self';`.replaceAll('\n', ' ');
// This is the middleware for our 11ty development server
eleventyConfig.setServerOptions({
	middleware: [(req, res, next) => {
		if (req.url.endsWith('.html') || req.url === '/') {
			res.setHeader('Content-Type', 'text/html; charset=UTF-8');
			res.setHeader('Content-Security-Policy', CSP);
		}
		next();
	}]
});

As you will see I have formatted the CSP using template literals to make the CSP more readable, and thus easier to modify and test on my local environment. You can find the public gist for the code above here (opens in new tab).

A couple of notes on this code:

Notice how I removed newlines from the final CSP variable. Using const signals to other developers that it shouldn't change, but since strings are immutable in JavaScript, modifying it using .replace() creates a new valid CSP header string.
I've added detection for both "naked URLs" and those ending in /index.html. This is just for convenience really, as it saves having to add the index.html to URLs before the response header is sent.
Lastly, the above CSP in its current form will break 11ty's live reload functionality because the connect-src in the CSP is set to none which tells the browser to block all network requests initiated by the page.

The following network communication technologies are blocked by this directive:

In order to fix point 3 there are 2 options:

1) Modify the `connect-src` directive

This is the obvious fix for the issue:

// add the following to your CSP variable
connect-src 'self' ws: wss: http://localhost:* ws://localhost:*;

By modifying the connect-src directive, we are allowing connections on localhost via HTTP and WebSockets (which 11ty's live reload script relies on to push updates to the browser without having to reload the whole page). And we are also allowing WebSocket connections that are both encrypted (wss:) and unencrypted (ws:).

2) Modify the `default-src` directive

This is a less obvious fix for the issue:

// add the following to your CSP variable
default-src 'self' ws: wss: http://localhost:* ws://localhost:*;

The reason this works is that default-src is the "fallback" directive. So assuming your connect-src is set to self then the following ws: wss: http://localhost:* ws://localhost:*; would still be followed because the directive is falling back to default-src.

IMPORTANT: don't use default-src if your connect-src is set to none! The value of none will still block all connections and override the default-src settings.

The advantage of modifying the default-src directive is that it will be applied to other directives like script-src and img-src. So it really depends on your CSP requirements for your local development environment! For most readers, I'd guess modifying the connect-src directive will be enough. But I've added both options for completeness.

Summary

Believe it or not, I kept the blog post short this time! I hope this code snippet makes configuring your 11ty CSP a breeze! As always, thanks for reading and if you have any feedback or comments, please let me know!

Post changelog:

04/02/25: Initial post published.
09/02/25: Thanks to vrugtehagel from discord for his couple of tweaks to the const CSP code. The small changes can be seen in the gist revisions (opens in new tab) if you are interested.

Lets create a plaintext RSS feed with 11ty

2025-01-29T00:00:00Z

Before writing this blog post I had 3 types of RSS feeds for my blog posts:

In this post, I'm going to add a 4th type: Plaintext.

Origin

A few months ago I was browsing my contacts and past colleagues looking for people to follow for my blogroll page. In doing so I just so happened to remember Terence Eden who I worked with at Government Digital Service (GDS). Terence (or edent, as he is known on most social networks), is a prolific blogger! I honestly don't know where he gets the time to blog so much and work, eat, sleep, blink, and breathe! While looking at the source code of his blog for his RSS feed, I just so happened to notice he had a plaintext version of his RSS feed. At the time it was responding with a 404, so I let him know on Mastodon, and he fixed it in a matter of minutes. Once fixed, I could see how it was formatted, and I set it as a personal challenge to replicate this feature on this blog (once all other more important functionality had been added). Now you may be asking, why would you need a plaintext version? So, let's answer that question first!

Why add a plaintext RSS feed?

Other than a personal challenge, what are the reasons for adding a plaintext version of your RSS feed?

Greater syndication

Now I have no idea how people like to read my content (I'm just very glad they do!). So if there's any way I can make their lives easier by adding multiple formats of the same RSS feed, then I'm happy to do it. After all, I'm always looking for ways to make this blog more open and inclusive. Even if it's only for a handful of readers, then it's worth adding the functionality, as it really is "set it and forget it" once deployed!

Command-line feed readers

I had no idea until I started researching the topic that there are users who use their terminal window as an RSS feed reader. For this they use programs like:

newsboat: "An RSS/Atom feed reader for text terminals" — It looks to have a very active repository on GitHub with 217 forks and 3,100 stars!
Liferea: "Liferea (Linux Feed Reader), a newsreader for GTK/GNOME" — Again an active GitHub repository with 129 forks and 830+ stars. I also noticed the project is over 20 years old (looking at some of the commits)!
RSS Guard: "Feed reader (podcast player and also Gemini protocol client) which supports RSS/ATOM/JSON and many web-based feed services." — another RSS client with an active GitHub repository with 130 forks and 1,800 stars. It's not surprising really considering the number of operating systems it supports!

So if you were in any doubt that plaintext RSS feed users don't exist, then I think the numbers above prove otherwise! (before you get angry, I do realise that not all these three programs users will be plaintext users!).

Privacy and Security Concerns

You only have to look at the Web Almanac 2024's chapters on Privacy and Security to realise how important these two topics are on the modern web. So it's completely understandable that RSS users are concerned about this when they subscribe to RSS feeds. Thankfully, this is where plaintext RSS feeds excel. By using plaintext RSS feeds, users can avoid loading external content such as images or scripts, thus reducing the risk of tracking and also enhancing their privacy.

Accessibility

I've seen accessibility brought up as a plus for plaintext RSS feeds, but honestly, I’m a little sceptical. Sure, a screen reader can read the content easily enough, but when it comes to navigation and overall user experience for someone with accessibility needs, navigating a plaintext feed seems like it’d be a step backwards. Wouldn’t something like HTML or XML, with its semantic markup and built-in navigation, be a way better option? That said, I’m totally open to being wrong here—if I’ve got this all incorrect, feel free to let me know.

Data Efficiency

This one really doesn't need much discussion, really. You can't get much more minimal in terms of bandwidth usage than a plaintext RSS feed. No images, no JavaScript, no markup, just pure content. It certainly makes for an excellent source of content for users with limited internet access or those operating in low-bandwidth environments.

Customisation and Integration

Tools like RSS Fulltext Proxy and Five Filters allow for text extraction from any website to convert partial feeds into full-text. This allows for integration into any feed reader, without plugins, or additional configuration. These are exceptional options if you want to integrate a particular feed into one of your workflows.

Reliability and Readability

You literally can't get a more robust format than plaintext. It is universally supported and small. And in terms of the resulting readability, there's no cookie banners, adverts, or newsletter signup forms that pop up halfway through an article. Thankfully, none of these annoyances are possible in plaintext, so you can consume all the content without interruption. It reminds me of how the World Wide Web used to be before it was commercialised beyond recognition! Take a read of the world's first website, and you get an idea of what Sir Tim Berners-Lee had in mind when he invented it in 1989 at CERN.

The web was originally conceived and developed to meet the demand for automated information-sharing between scientists in universities and institutes around the world.

The WorldWideWeb (W3) is a wide-area hypermedia information retrieval initiative aiming to give universal access to a large universe of documents.

Anyway, enough of the "why's" let's crack on with the 11ty implementation!

The code

eleventy.config.js

First, let's modify the 11ty config file:

Here we are telling 11ty to process .txt files as template files.

eleventyConfig.addTemplateFormats("txt");

Next we need to configure 11ty and tell it how to handle .txt files with a custom handler.

eleventyConfig.addExtension("txt", {
	outputFileExtension: "txt",
	compile: async function (inputContent) {
	  return async (data) => inputContent;
	},
});

The code is fairly self-explanatory outputFileExtension is obvious, the resulting output will have a .txt extension. The compile function is a simple pass-through. It takes the txt input and returns the original content unchanged.

feed.txt.njk

Next up is the template for the plaintext feed output. I'm using Nunjucks as that's the main templating language I use across this blog, but I'm sure you'd be able to any other templating language 11ty supports too. This file sits in my /content/feed/ directory, so 11ty will process it as a njk file in the output.


---
permalink: feed/feed.txt
eleventyComputed:
  layout: null
---
# {{ metadata.title }} - {{ metadata.author.name }} - {{ metadata.description }}
## {{ metadata.fulldescription }}
URL: {{ metadata.url }}

{% for post in collections.posts | reverse -%}
{% if loop.index0 < 10 -%}
--- Start: {{ post.data.title | safe }}
Published on: {{ post.date | readableDate }}
{{ metadata.url }}{{ post.url }}

Main Content:
{{ post.templateContent | striptags(true) | decodeHtmlEntities | safe }}
{% if not loop.last %}
--- End: {{ post.data.title | safe }}



{% endif -%}
{% endif -%}
{% endfor -%}

A Gist for this template is here (opens in new tab), if you find that easier to read.

There's a fair amount going on in this template file, but it is essentially:

Setting where I want the feed to sit in my final site output (permalink)
Overriding the default layout dynamically during the build process (layout: null)
Populating the top of the feed with my basic blog metadata.
Looping through all by blog posts in reverse (newest first).
Limiting the number of posts in the feed to 10.
Outputting the parts of each post I want in the feed using Nunjucks and cleaning up the output using various filters, which I will go through next.

Code Notes:

You may be looking at this template and wondering what is going on with all the space between the logic. Well, since this template is outputting plaintext, this is all intentional spacing to make the final output more readable.
I'm using loop.index0 which is the current iteration of the loop but 0 indexed because it was the only way that seemed to work for me to limit the number of posts. I think I must have a weird collections.posts setup somewhere because regardless of what I tried, nothing worked. Likewise, I believe the native slice(0, 10) should do the same thing, but for me, the loop stopped working altogether. I tried debugging it for a couple of hours and eventually stuck with the loop.index0 setup because it actually worked. However, if anyone has any ideas on why the native slice() function on a collections.posts doesn't work, please let me know! I'd love to get to the bottom of what the issue is!
You may notice my use of the -%} in the template, this is me telling Nunjucks to trim any whitespace (spaces, tabs, newlines) directly following the tag. This was required because it was adding numerous new lines in the resulting output. It's when I discover little intricacies in Nunjucks like this, that I realise what a versatile templating language it actually is!

filters

Most of the filters used in the above template are all pretty standard to the Nunjucks language:

I only added one custom 11ty filter for this functionality.

decodeHtmlEntities

For some reason, I had some very odd character encoding issues, and the only way I could resolve the issue was to create a custom 11ty filter to search and replace them throughout the content. I looked through the default Nunjucks filters for a solution, but filters like escape and forceescape didn't work. I could have probably used Nunjucks native replace filter, but that would have cluttered up the template, so decided to move it to an 11ty filter instead:

// clean up the HTML entities in the RSS feed text
eleventyConfig.addFilter("decodeHtmlEntities", function(text) {
	return text.replace(/&([^;]+);/g, function(match, entity) {
			const entities = {
					'amp': '&',
					'apos': "'",
					'lt': '<',
					'gt': '>',
					'quot': '"',
					'nbsp': ' '
			};

			// Handle numeric entities
			if (entity[0] === '#') {
					const code = entity[1] === 'x'
							? parseInt(entity.slice(2), 16)
							: parseInt(entity.slice(1), 10);
					return String.fromCharCode(code);
			}

			return entities[entity] || match;
	});
});

It's basically just a fancy way to search and replace for multiple encoding issues it finds in the plaintext output.

Just to be clear, I wrote a really clunky version that did the same thing as this first. Basically, a chain of text.replaceAll('&', '&').replaceAll(''', "'")... and so on, I think you get the idea.

It worked fine, but then I asked ChatGPT to optimise it, and this is the result it came up with! It's certainly not as readable, but it still does the job, and it also handles numeric entities too! Off the back of this example, I think AI used correctly can be a fantastic teaching tool, especially when it comes to random little filter functions like this.

Adding the feed to your `<head>`

Once you have the feed up and running, it's time to make sure you have it visible to your users by adding it to your pages <head>. It's dead simple, just like your other feeds, only the type is set to text/plain.

<link rel="alternate" href="/feed/feed.txt" type="text/plain" title="Nooshu - Matt Hobbs blog post Plaintext Feed.">

I also have links to all my feed formats in my footer so they are easy to find. So, finally, why not look at the result of the above code now by looking at my plaintext RSS feed.

Sample output

Here's an example of the output from the code above:

# Nooshu - Matt Hobbs - Frontend web developer, turned engineering manager.
## This is the website of Matt Hobbs, who is a Frontend Engineering Manager from Oxfordshire, UK.
URL: https://nooshu.com

--- Start: The Speed Trifecta: 11ty, Brotli 11, and CSS Fingerprinting
Published on: 23 January 2025
https://nooshu.com/blog/2025/01/23/the-speed-trifecta-11ty-brotli-11-and-css-fingerprinting/

Main Content:
So recently, I have written two 11ty related blog posts:

Using an 11ty Shortcode to craft a custom CSS pipeline
Cranking Brotli up to 11 with Cloudflare Pro and 11ty

...more blogpost content here...

Post changelog:

23/01/25: Initial post published.

--- End: The Speed Trifecta: 11ty, Brotli 11, and CSS Fingerprinting

Summary

Whew! Another blog post in the books, and another shiny new feature added—this time, a working plaintext RSS feed! (That’s one more thing off the to-do list—always a win!) Thanks for stopping by! I hope you found this post interesting, maybe even useful? As always, if you’ve got thoughts, or feedback, please let me know!

Post changelog:

29/01/25: Initial post published.

The Speed Trifecta: 11ty, Brotli 11, and CSS Fingerprinting

2025-01-23T00:00:00Z

So recently, I have written two 11ty related blog posts:

If you haven't read them, no problem, they are always there if you're ever struggling to sleep!

TL;DR:

In the first post I look at how I've automated CSS fingerprinting on production (Cloudflare), resulting in the ability to use long-life Cache-Control directives like max-age=31536000 and the immutable browser hint.

In the second post, I look at how you can improve Brotli compression by manually compressing your assets to Brotli setting 11 (max) then serving them via a Cloudflare Pro plan. This gives a significant improvement in file size over the Brotli compression setting of 4 that Cloudflare uses for dynamic (on the fly) compression.

While working on both posts, it dawned on me that blending the approaches they cover would result in the ideal CSS configuration for my little blog. In this post, I'll show you exactly how I've done it. Let's get started!

Original code

So I'm going to be using the code I wrote in the CSS fingerprinting blog post I wrote recently. You can see it below, or in this Gist (opens in new tab).

import dotenv from "dotenv";
import CleanCSS from 'clean-css';
import fs from 'fs';
import crypto from 'crypto';
import path from 'path';
dotenv.config();

// create a single instance of the CleanCSS function
// to be used in file loops. Add additional optimisation settings in here.
const cleanCSS = new CleanCSS({
	level: {
		2: {
			removeDuplicateRules: true // turns on removing duplicate rules
		}
	}
});

export function manipulateCSS(eleventyConfig) {
	eleventyConfig.addShortcode("customCSS", async function(cssPath) {
		// output the file with no fingerprinting if on the dev environment
		// (allows auto-reload when the CSS is modified)
		if (process.env.ELEVENTY_ENV === 'development') {
			return `<link rel="stylesheet" href="${cssPath}">`;
		}
		// Using path.join for better cross-platform compatibility
		const inputFile = path.join('./public', cssPath);
		const outputDirectory = path.join('./_site', 'css');
		const cacheDirectory = path.join('./.cache', 'css');

		try {
			// Check if input file exists first
			if (!fs.existsSync(inputFile)) {
				console.error(`Input CSS file not found: ${inputFile}`);
				return '';
			}
			// Ensure both cache and output directories exist
			for (const dir of [cacheDirectory, outputDirectory]) {
				if (!fs.existsSync(dir)) {
					fs.mkdirSync(dir, { recursive: true });
				}
			}
			// Read the input CSS file
			const inputCSS = await fs.promises.readFile(inputFile, 'utf8');
			// Initialises a new hashing instance
			const hash = crypto.createHash('sha256')
				// Feed CSS data into the hash function
				.update(inputCSS)
				// Specify the hash should be returned as a hexadecimal string
				.digest('hex')
				// Only take the first 10 characters of the hash
				.slice(0, 10);

			// Generate our CSS Cache name
			const cacheKey = `${hash}-${cssPath.replace(/[\/\\]/g, '-')}`;
			// This is where the file will be written
			const cachePath = path.join(cacheDirectory, cacheKey);

			// store our manipulated CSS in this variable
			let processedCSS;
			// check we have a cache directory
			if (fs.existsSync(cachePath)) {
				// read the cached CSS file
				processedCSS = await fs.promises.readFile(cachePath, 'utf8');
			} else {
				// Use the memoized cleanCSS instance to minify the input CSS
				processedCSS = cleanCSS.minify(inputCSS).styles;
				await fs.promises.writeFile(cachePath, processedCSS);
			}
			// Split the input file path into its components (directory, filename, extension)
			const parsedPath = path.parse(inputFile);
			// Use path.join for output paths
			const finalFilename = path.join(outputDirectory, `${parsedPath.name}-${hash}${parsedPath.ext}`);
			// Write the optimised CSS to the final output location with the hash in the filename
			await fs.promises.writeFile(finalFilename, processedCSS);

			// path manipulation for final URL
			const hashedPath = finalFilename.replace(path.join('./_site'), '').replace(/\\/g, '/');

			// return our final link element with optimised and fingerprinted CSS.
			return `<link rel="stylesheet" href="${hashedPath}">`;
		} catch (err) {
			console.error("Error processing CSS:", err);
			return "";
		}
	});
}

Install zlib

The first thing we shall do is install the zlib package from npm, as this is what we are going to use to compress the CSS file all the way up to 11 on production.

npm install zlib

Next, we import it into our ESM code above:

import zlib from 'zlib';

Don't worry, I'm not going to go this slow all the way through the code. Feel free to skip straight to the final code if you so wish!

Setting the default compression level

This is a completely optional step, it really depends on how, and where you store all your build variables. But I'm going to be storing the Brotli Compression level I want to use on production both in the JS (as a default to fall back on if the environment variable doesn't exist), and also an environment variable on production. In the JavaScript below, set at 6 to show it can be any value between 0-11 (3 or 4 is what most CDN use for dynamic compression). Cloudflare's compression is set to 4:

// Default Brotli compression level if not set in the environment
const DEFAULT_BROTLI_COMPRESSION_LEVEL = 6;

And also in my .env file in development:

BROTLI_COMPRESSION_LEVEL=11

Now we set our compression level as a constant in our JavaScript for use later in the code:

const brotliCompressionLevel = parseInt(process.env.BROTLI_COMPRESSION_LEVEL || DEFAULT_BROTLI_COMPRESSION_LEVEL, 10);

Since our environment variable is currently set, compression will be set to 11, but if that variable isn't set, it will fall back to 6, the default value set above.

Cloudflare and Brotli

Before we go through the Brotli compression code, it's a good time to mention that Cloudflare does something automatically that is very useful for compressed files. Cloudflare Pages will automatically serve Brotli-compressed files (.br) if available, with no additional configuration required on the Cloudflare side. See the Content Compression Documentation on Cloudflare for more information.

This means we don't need to rename the compressed CSS file from index-hash.css.br back to index-hash.css for a user's browser to recognise it as a CSS file. Handy huh! Don't worry if you don't use Cloudflare Pages, I also have a version written that renames the CSS file back to index-hash-compressed.css for non-Cloudflare Pages, readers.

Important: While testing the documented method above, I could not find a way to confirm that my static compressed version was being served automatically. So in the code below I have removed this assumption and explicitly added my statically compressed version to the <link> tag. Please do let me know if I'm interpreting the documentation for this functionality wrong! As I'd love to see it in action, and also how you verify the .br file is actually being served automatically?

Brotli Compression code

Now that I've covered that bit of (potential) Cloudflare Pages automation, let's have a look at the actual compression code:

// Brotli compression

// The output filename for the compressed file
const brotliFilename = `${finalFilename}.br`;
// Only compress to Brotli if the file doesn't exist
if (!fs.existsSync(brotliFilename)) {
	// Set our zlib options here e.g. compression
	const brotliOptions = {
		level: brotliCompressionLevel // Use the level specified in the environment
	};
	// zlib does it's compression magic!
	const brotliBuffer = zlib.brotliCompressSync(Buffer.from(processedCSS), brotliOptions);
	// Write the compressed code to the output filename defined above
	await fs.promises.writeFile(brotliFilename, brotliBuffer);
}

Generate the CSS `<link>` tag

The last thing to do now is mostly the same as what I had in the original code set out above:

// path manipulation for final URL
const hashedPath = brotliFilename.replace(path.join('./_site'), '').replace(/\\/g, '/');
// return our final link element with optimised and fingerprinted CSS.
return `<link rel="stylesheet" href="${hashedPath}">`;

Only this time I'm modifying the path to the Brotli compressed filename rather than the original uncompressed CSS file! Simple!

On Development: Once added to the 11ty config as per my previous post, my development CSS looks like this:

// uncompressed CSS with no fingerprinting, auto-reload still functioning
<link rel="stylesheet" href="/css/index.css">

On Production: The CSS looks like this:

// Brotli 11 compressed, minified and fingerprinted CSS
<link rel="stylesheet" href="/css/index-b9fcfe85ef.css.br">

This is the version I have running on the site at the moment, why not view the page source and take a look.

Finishing touches

That's the bulk of the work done, but if you were to run this code on production at present you'd get a page full of naked HTML and no styling. This is because the Cloudflare Pages _headers file needs to be modified, as Cloudflare Pages is now serving my Brotli 11 compressed CSS file, not the uncompressed version we had before.

Simply add the following to your _headers file:

/css/*
Content-Encoding: br
Vary: Accept-Encoding
Content-Type: text/css

It's important to remember that this headers file is looking at the path from which the file is served on the production website! It isn't (and can't) use an extension wildcard, as Cloudflare pages doesn't support it. It took me a while to figure this out I must admit!

So for example, The following doesn't work:

*.css
Content-Encoding: br
Vary: Accept-Encoding
Content-Type: text/css

When I first migrated to Cloudflare Pages, I tried it and couldn't work out why it wasn't working. /css/* is basically saying: "Any file that resides in the css directory on production will be served with the following headers".

Lastly, remember to add the following to your Cloudflare Pages environment variables. This can be plaintext if you like, since the variable is only storing a single integer (not a secret API key):

BROTLI_COMPRESSION_LEVEL=11

Once added, you should be good to go!

Final code

As promised, below is the final code that I just described above:

import zlib from 'zlib';
import dotenv from "dotenv";
import CleanCSS from 'clean-css';
import fs from 'fs';
import crypto from 'crypto';
import path from 'path';
dotenv.config();

// An example of how you could add additional CleanCSS settings if required
const cleanCSS = new CleanCSS({
	level: {
		2: {
			removeDuplicateRules: true
		}
	}
});

// Default Brotli compression level if not set in the environment
const DEFAULT_BROTLI_COMPRESSION_LEVEL = 6;

export function manipulateCSS(eleventyConfig) {
	eleventyConfig.addShortcode("customCSS", async function(cssPath) {
		if (process.env.ELEVENTY_ENV === 'development') {
			return `<link rel="stylesheet" href="${cssPath}">`;
		}

		const inputFile = path.join('./public', cssPath);
		const outputDirectory = path.join('./_site', 'css');
		const cacheDirectory = path.join('./.cache', 'css');

		// Get compression level from the environment or use the default
		const brotliCompressionLevel = parseInt(process.env.BROTLI_COMPRESSION_LEVEL || DEFAULT_BROTLI_COMPRESSION_LEVEL, 10);

		try {
			if (!fs.existsSync(inputFile)) {
				console.error(`Input CSS file not found: ${inputFile}`);
				return '';
			}

			for (const dir of [cacheDirectory, outputDirectory]) {
				if (!fs.existsSync(dir)) {
					fs.mkdirSync(dir, { recursive: true });
				}
			}

			const inputCSS = await fs.promises.readFile(inputFile, 'utf8');
			const hash = crypto.createHash('sha256').update(inputCSS).digest('hex').slice(0, 10);

			const cacheKey = `${hash}-${cssPath.replace(/[\/\\]/g, '-')}`;
			const cachePath = path.join(cacheDirectory, cacheKey);

			let processedCSS;
			if (fs.existsSync(cachePath)) {
				processedCSS = await fs.promises.readFile(cachePath, 'utf8');
			} else {
				processedCSS = cleanCSS.minify(inputCSS).styles;
				await fs.promises.writeFile(cachePath, processedCSS);
			}

			const parsedPath = path.parse(inputFile);
			const finalFilename = path.join(outputDirectory, `${parsedPath.name}-${hash}${parsedPath.ext}`);
			await fs.promises.writeFile(finalFilename, processedCSS);

			// Brotli compression

			// The output filename for the compressed file
			const brotliFilename = `${finalFilename}.br`;

			// Only compress to Brotli if the file doesn't exist
			if (!fs.existsSync(brotliFilename)) {
				// Set our zlib options here e.g. compression
				const brotliOptions = {
					level: brotliCompressionLevel // Use the level specified in the environment
				};
				// zlib does it's compression magic!
				const brotliBuffer = zlib.brotliCompressSync(Buffer.from(processedCSS), brotliOptions);
				// Write the compressed code to the output filename defined above
				await fs.promises.writeFile(brotliFilename, brotliBuffer);
			}

			const hashedPath = brotliFilename.replace(path.join('./_site'), '').replace(/\\/g, '/');
			return `<link rel="stylesheet" href="${hashedPath}">`;
		} catch (err) {
			console.error("Error processing CSS:", err);
			return "";
		}
	});
}

And here's a Gist for the code (opens in new tab) as well!

With file renaming

Now I understand not everyone will want to serve their CSS files using the .br extension. So there's a version below that also renames the file for you back to .css, and adds the -compressed suffix to the filename as well. The rest of the code is identical to the version above:

import zlib from 'zlib';
import dotenv from "dotenv";
import CleanCSS from 'clean-css';
import fs from 'fs';
import crypto from 'crypto';
import path from 'path';
dotenv.config();

// An example of how you could add additional CleanCSS settings if required
const cleanCSS = new CleanCSS({
	level: {
		2: {
			removeDuplicateRules: true
		}
	}
});

// Default Brotli compression level if not set in the environment
const DEFAULT_BROTLI_COMPRESSION_LEVEL = 6;

export function manipulateCSS(eleventyConfig) {
	eleventyConfig.addShortcode("customCSS", async function(cssPath) {
		if (process.env.ELEVENTY_ENV === 'development') {
			return `<link rel="stylesheet" href="${cssPath}">`;
		}

		const inputFile = path.join('./public', cssPath);
		const outputDirectory = path.join('./_site', 'css');
		const cacheDirectory = path.join('./.cache', 'css');

		// Get compression level from the environment or use the default
		const brotliCompressionLevel = parseInt(process.env.BROTLI_COMPRESSION_LEVEL || DEFAULT_BROTLI_COMPRESSION_LEVEL, 10);

		try {
			if (!fs.existsSync(inputFile)) {
				console.error(`Input CSS file not found: ${inputFile}`);
				return '';
			}

			for (const dir of [cacheDirectory, outputDirectory]) {
				if (!fs.existsSync(dir)) {
					fs.mkdirSync(dir, { recursive: true });
				}
			}

			const inputCSS = await fs.promises.readFile(inputFile, 'utf8');
			const hash = crypto.createHash('sha256').update(inputCSS).digest('hex').slice(0, 10);

			const cacheKey = `${hash}-${cssPath.replace(/[\/\\]/g, '-')}`;
			const cachePath = path.join(cacheDirectory, cacheKey);

			let processedCSS;
			if (fs.existsSync(cachePath)) {
				processedCSS = await fs.promises.readFile(cachePath, 'utf8');
			} else {
				processedCSS = cleanCSS.minify(inputCSS).styles;
				await fs.promises.writeFile(cachePath, processedCSS);
			}

			const parsedPath = path.parse(inputFile);
			const finalFilename = path.join(outputDirectory, `${parsedPath.name}-${hash}${parsedPath.ext}`);
			await fs.promises.writeFile(finalFilename, processedCSS);

			// Brotli compression with renaming
			const compressedFilename = path.join(outputDirectory, `${parsedPath.name}-${hash}-compressed${parsedPath.ext}`);
			// Only compress to Brotli if the file doesn't exist
			if (!fs.existsSync(compressedFilename)) {
				// Set our zlib options here e.g. compression
				const brotliOptions = {
					level: brotliCompressionLevel
				};
				// zlib does it's compression magic!
				const brotliBuffer = zlib.brotliCompressSync(Buffer.from(processedCSS), brotliOptions);
				// Write the compressed code to the output filename defined above
				await fs.promises.writeFile(compressedFilename, brotliBuffer);
			}

			const hashedPath = compressedFilename.replace(path.join('./_site'), '').replace(/\\/g, '/');
			return `<link rel="stylesheet" href="${hashedPath}">`;
		} catch (err) {
			console.error("Error processing CSS:", err);
			return "";
		}
	});
}

The Gist for the code above is here (opens in new tab).

The above code will do the following:

On Development: It will simply output:

<link rel="stylesheet" href="/css/index.css">

Exactly as it did in the previous version.

On Production: This is the output that will be seen:

<link rel="stylesheet" href="/css/index-b9fcfe85ef-compressed.css">

In this instance, the CSS file has been Brotli compressed to 11, but it has also been renamed with the -compressed suffix and the .br extension has been removed. Technically, you don't need the suffix, but I've added it to emphasise (and remind myself) that it isn't a "standard" CSS file in plain text.

And as I mentioned in the previous CSS Shortcode blog post. You'll now be able to use long cache life headers like:

/[css]/*
Cache-Control: public, max-age=31536000, immutable

Safe in the knowledge that updating your CSS will generate a brand-new filename, effectively nullifying the existing cache. Check out the previous post if you want more details on the above _headers file code for Cloudflare.

Summary

Fantastic! You made it to the end of yet another 11ty blog post of mine, congrats! I'm not sure if there's anything else I can do to optimise my CSS delivery to users at the moment?

Maybe preloading or the Speculation Rules API? Which Cloudflare already supports, and is currently enabled on my site. This functionality is called "Speed Brain" in the Cloudflare dashboard (under the "Speed" then "Content Optimization"). It's worth noting that this functionality is currently in Beta. But I don't think I need to do anything manually to use it, since Cloudflare rolled it out to all plans (including the free plan!), back in September 2024.

But perhaps there are other optimisations I can make? Unknown, unknowns etc… As always, thanks for reading, I hope you found the post useful and if you have any feedback or comments let me know!

Post changelog:

23/01/25: Initial post published.

Using an 11ty Shortcode to craft a custom CSS pipeline

2025-01-12T00:00:00Z

I know I'm still a bit of a 11ty n00b, so I hope this isn't frowned upon in the community, but on rebuilding my blog using 11ty, I decided not to use the standard Bundle plugin that was added in v3.0.0. Instead, I decided to write a custom Shortcode to customise my CSS output. In this blog post, I will go through the code I have written in the hope it will help others and more importantly gather feedback from the community, to see if any improvements can be made to the code. Please do let me know if you have any feedback.

My Requirements

First, what am I hoping to achieve with this Shortcode solution?

Maintain the auto-reload functionality that comes with 11ty when you modify your CSS locally (this is a super helpful feature I've been a big fan of since LiveReload was released back in February 2011, and later BrowserSync).
Filename fingerprinting to allow me to use long-life Cache-Control response headers without having to worry about cache becoming outdated. (e.g. max-age=31536000, immutable)
The ability to optimise my CSS output using the excellent clean-css Node plugin. It does more than just minify! Check out all the optimisations it can help you if you haven't seen them.
Minimal impact on the 11ty build process by leveraging build caching in some way.
Set it and forget it, once it's added It requires minimal (or no ongoing maintenance), apart from updating dependencies, every so often!

The Code

I'll quit the waffling and just show you the code I currently have building my CSS for this site. This code sits in its own file called css-manipulation.js in a _helpers directory in my 11ty root (i.e. the same level as _data).

/**
 * CSS Manipulation Module for Eleventy
 *
 * This module provides CSS processing, minification, and compression functionality
 * for an Eleventy static site generator. It handles:
 * - CSS minification using CleanCSS
 * - Content-based file hashing for cache busting
 * - Brotli compression for optimized delivery
 * - Build-time caching to avoid redundant processing
 * - Concurrent request handling to prevent duplicate work
 */

// Node.js built-in modules for file system operations, path manipulation, and hashing
import crypto from "crypto"; // Used to create SHA-256 hashes of CSS content for cache busting
import fs from "fs"; // File system operations (reading/writing files, checking existence)
import path from "path"; // Cross-platform path manipulation and joining
// Node.js zlib module for Brotli compression
// Brotli is a modern compression algorithm that provides better compression ratios than gzip
import { brotliCompressSync } from "zlib";

// CleanCSS library for CSS minification and optimization
// This performs a wide range of optimizations to reduce file size
import CleanCSS from "clean-css";

// Environment configuration to check if we're in local development mode
import env from "../_data/env.js";

/**
 * CleanCSS Configuration
 *
 * CleanCSS performs multi-level CSS optimization. This configuration enables
 * comprehensive minification while maintaining CSS functionality.
 *
 * Level 1 optimizations (basic cleanups):
 * - cleanupCharsets: Remove unnecessary @charset declarations
 * - normalizeUrls: Normalize and optimize URL() references
 * - optimizeBackground: Optimize background properties
 * - optimizeBorderRadius: Shorten border-radius values
 * - optimizeFilter: Optimize filter() functions
 * - optimizeFont: Optimize font-family declarations
 * - optimizeFontWeight: Convert font-weight to numeric values
 * - optimizeOutline: Optimize outline properties
 * - removeEmpty: Remove empty CSS rules
 * - removeNegativePaddings: Remove invalid negative padding values
 * - removeQuotes: Remove unnecessary quotes from URLs and identifiers
 * - removeWhitespace: Remove unnecessary whitespace
 * - replaceMultipleZeros: Shorten multiple zero values
 * - replaceTimeUnits: Optimize time unit values (e.g., 0s → 0)
 * - replaceZeroUnits: Remove units from zero values (e.g., 0px → 0)
 * - roundingPrecision: Round numeric values to 2 decimal places
 * - selectorsSortingMethod: Sort selectors in a standard order
 * - specialComments: Remove all comments (set to "none")
 * - tidyAtRules: Clean up @-rules
 * - tidyBlockScopes: Clean up block scopes
 * - tidySelectors: Clean up and optimize selectors
 * - transform: Custom transformation function (empty, no custom transforms)
 *
 * Level 2 optimizations (advanced restructuring):
 * - mergeAdjacentRules: Combine adjacent CSS rules with same selectors
 * - mergeIntoShorthands: Convert long-form properties to shorthand
 * - mergeMedia: Combine @media rules with same conditions
 * - mergeNonAdjacentRules: Merge duplicate rules even if not adjacent
 * - mergeSemantically: Disabled to avoid potentially breaking semantic merges
 * - overrideProperties: Remove overridden properties
 * - removeEmpty: Remove empty rules at level 2
 * - reducePadding: Optimize padding values
 * - reducePositions: Optimize position values
 * - reduceTimingFunctions: Optimize animation timing functions
 * - reduceTransforms: Optimize transform functions
 * - restructureRules: Disabled to avoid aggressive restructuring that might break CSS
 * - skipProperties: Empty array means optimize all properties
 *
 * Format options (output formatting):
 * - All breaks set to false: Output as single line (no line breaks)
 * - indentBy: 0 (no indentation for minimal file size)
 * - indentWith: "space" (if indentation were enabled)
 * - All spaces set to false: Remove unnecessary spaces
 * - wrapAt: false (no line wrapping)
 *
 * Other options:
 * - inline: ["none"] - Don't inline any @import rules
 * - rebase: false - Don't rebase URLs (keep original paths)
 * - returnPromise: false - Use synchronous API (we handle async ourselves)
 */
const cleanCSS = new CleanCSS({
	level: {
		1: {
			cleanupCharsets: true,
			normalizeUrls: true,
			optimizeBackground: true,
			optimizeBorderRadius: true,
			optimizeFilter: true,
			optimizeFont: true,
			optimizeFontWeight: true,
			optimizeOutline: true,
			removeEmpty: true,
			removeNegativePaddings: true,
			removeQuotes: true,
			removeWhitespace: true,
			replaceMultipleZeros: true,
			replaceTimeUnits: true,
			replaceZeroUnits: true,
			roundingPrecision: 2,
			selectorsSortingMethod: "standard",
			specialComments: "none",
			tidyAtRules: true,
			tidyBlockScopes: true,
			tidySelectors: true,
			transform: function () {},
		},
		2: {
			mergeAdjacentRules: true,
			mergeIntoShorthands: true,
			mergeMedia: true,
			mergeNonAdjacentRules: true,
			mergeSemantically: false,
			overrideProperties: true,
			removeEmpty: true,
			reducePadding: true,
			reducePositions: true,
			reduceTimingFunctions: true,
			reduceTransforms: true,
			restructureRules: false,
			skipProperties: [],
		},
	},
	format: {
		breaks: {
			afterAtRule: false,
			afterBlockBegins: false,
			afterBlockEnds: false,
			afterComment: false,
			afterProperty: false,
			afterRuleBegins: false,
			afterRuleEnds: false,
			beforeBlockEnds: false,
			betweenSelectors: false,
		},
		indentBy: 0,
		indentWith: "space",
		spaces: {
			aroundSelectorRelation: false,
			beforeBlockBegins: false,
			beforeValue: false,
		},
		wrapAt: false,
	},
	inline: ["none"],
	rebase: false,
	returnPromise: false,
});

/**
 * Default Brotli Compression Level
 *
 * Brotli compression levels range from 0-11:
 * - 0: Fastest, least compression
 * - 11: Slowest, best compression (maximum)
 *
 * Level 11 is used by default for production builds where file size matters
 * more than compression time. This can be overridden via BROTLI_COMPRESSION_LEVEL
 * environment variable.
 */
const DEFAULT_BROTLI_COMPRESSION_LEVEL = 11;

/**
 * In-Memory Build Cache
 *
 * This Map caches CSS processing results during a single build run.
 * It serves two purposes:
 * 1. Prevents re-processing the same CSS file when multiple pages reference it
 * 2. Handles concurrent requests by storing Promises during processing
 *
 * Structure:
 * - Key: cssPath (the original CSS file path)
 * - Value: Either a Promise (if processing is in progress) or an object with:
 *   - hash: Content hash of the CSS file
 *   - processedCSS: The minified CSS string
 *   - htmlOutput: The final HTML <link> tag string
 *
 * This cache is cleared between builds (per-process, not persisted).
 */
const cssBuildCache = new Map();

/**
 * Directory Creation Tracking
 *
 * Tracks which directory combinations have been created during this build.
 * This prevents redundant fs.existsSync() and fs.mkdirSync() calls when
 * multiple CSS files are processed.
 *
 * Structure: Set of strings like "cacheDir:outputDir"
 */
const directoriesCreated = new Set();

/**
 * Main Function: manipulateCSS
 *
 * Registers a custom Eleventy shortcode that processes CSS files.
 * The shortcode can be used in templates like: 
 *
 * Processing Pipeline:
 * 1. Check if in local development mode (skip processing)
 * 2. Check in-memory cache (avoid duplicate processing)
 * 3. Read and hash the CSS file
 * 4. Check disk cache for minified CSS
 * 5. Minify CSS if not cached
 * 6. Write processed CSS with hash-based filename
 * 7. Compress with Brotli
 * 8. Return HTML <link> tag with hashed filename for cache busting
 *
 * @param {object} eleventyConfig - The Eleventy configuration object
 */
export function manipulateCSS(eleventyConfig) {
	/**
	 * Register the "customCSS" shortcode with Eleventy
	 * This makes it available in all templates as 
	 *
	 * @param {string} cssPath - The relative path to the CSS file (from /public directory)
	 * @returns {Promise<string>} - HTML string containing a <link> tag pointing to the processed CSS
	 */
	eleventyConfig.addShortcode("customCSS", async function (cssPath) {
		/**
		 * Stage 1: Local Development Short-Circuit
		 *
		 * In local development, skip all processing to speed up builds.
		 * Just return a simple <link> tag pointing to the original file.
		 * This allows for faster iteration during development.
		 */
		if (env.isLocal) {
			return `<link rel="stylesheet" href="${cssPath}">`;
		}

		/**
		 * Stage 2: In-Memory Cache Check
		 *
		 * Check if we've already processed this CSS file during this build.
		 * This handles the common case where multiple pages reference the same CSS file.
		 *
		 * The cache can contain:
		 * - A Promise: Processing is currently in progress, wait for it
		 * - An object: Processing is complete, return the cached HTML output
		 */
		if (cssBuildCache.has(cssPath)) {
			const cached = cssBuildCache.get(cssPath);
			// If it's a promise (in progress), wait for it
			// This handles concurrent requests - multiple pages calling this shortcode
			// simultaneously for the same CSS file will all wait for the same processing
			if (cached instanceof Promise) {
				return await cached;
			}
			// Otherwise return cached result immediately (already HTML string)
			// This is the fast path for subsequent page builds
			return cached.htmlOutput;
		}

		/**
		 * Stage 3: Initialize Processing
		 *
		 * Set up file paths and configuration for processing.
		 */
		// Construct full paths for input/output/cache directories
		const inputFile = path.join("./public", cssPath); // Source CSS file location
		const outputDirectory = path.join("./_site", "css"); // Where processed CSS goes (build output)
		const cacheDirectory = path.join("./.cache", "css"); // Where minified CSS cache is stored

		/**
		 * Stage 4: Get Brotli Compression Level
		 *
		 * Read compression level from environment variable or use default.
		 * This allows fine-tuning compression vs. speed trade-off per environment.
		 * Higher levels = better compression but slower processing.
		 */
		const brotliCompressionLevel = parseInt(
			process.env.BROTLI_COMPRESSION_LEVEL || DEFAULT_BROTLI_COMPRESSION_LEVEL,
			10, // Base 10 parsing
		);

		/**
		 * Stage 5: Create Processing Promise
		 *
		 * Wrap all processing in an async IIFE (Immediately Invoked Function Expression).
		 * This allows us to:
		 * 1. Handle concurrent requests by caching the Promise itself
		 * 2. Catch errors at the processing level
		 * 3. Return the same Promise to multiple concurrent callers
		 */
		const processingPromise = (async () => {
			try {
				/**
				 * Stage 5.1: Validate Input File Exists
				 *
				 * Check if the source CSS file exists before attempting to process it.
				 * If missing, log an error and return empty string (fails gracefully).
				 */
				if (!fs.existsSync(inputFile)) {
					return "";
				}

				/**
				 * Stage 5.2: Ensure Directories Exist
				 *
				 * Create output and cache directories if they don't exist.
				 * Uses a Set to track which directory combinations have been created
				 * during this build to avoid redundant checks and operations.
				 *
				 * recursive: true ensures parent directories are created if needed.
				 */
				const dirKey = `${cacheDirectory}:${outputDirectory}`;
				if (!directoriesCreated.has(dirKey)) {
					for (const dir of [cacheDirectory, outputDirectory]) {
						if (!fs.existsSync(dir)) {
							fs.mkdirSync(dir, { recursive: true });
						}
					}
					directoriesCreated.add(dirKey);
				}

				/**
				 * Stage 5.3: Read and Hash CSS File
				 *
				 * Read the source CSS file and generate a content-based hash.
				 * The hash is used for:
				 * - Cache busting (filename changes when content changes)
				 * - Disk cache key (identify if we've seen this content before)
				 *
				 * SHA-256 is used for strong collision resistance.
				 * Only first 10 characters of hash are used (sufficient for cache busting).
				 *
				 */
				const inputCSS = await fs.promises.readFile(inputFile, "utf8");
				const hash = crypto
					.createHash("sha256") // Use SHA-256 algorithm
					.update(inputCSS) // Hash the CSS content
					.digest("hex") // Get hexadecimal representation
					.slice(0, 10); // Take first 10 chars (sufficient for uniqueness)
				/**
				 * Stage 5.4: Generate Cache Key and Path
				 *
				 * Create a unique cache key combining:
				 * - The content hash (identifies file contents)
				 * - The file path (normalized to avoid path separator issues)
				 *
				 * This allows different CSS files with same content to share cache,
				 * but also handles edge cases where paths matter.
				 */
				const cacheKey = `${hash}-${cssPath.replace(/[/\\]/g, "-")}`;
				const cachePath = path.join(cacheDirectory, cacheKey);

				/**
				 * Stage 5.5: Minify CSS (or Load from Disk Cache)
				 *
				 * Check if we've already minified this exact CSS content before.
				 * Disk cache persists between builds, so unchanged CSS files don't
				 * need re-minification even after restarting the build process.
				 *
				 * If cached:
				 * - Load the pre-minified CSS from disk
				 * - Skip the expensive minification step
				 *
				 * If not cached:
				 * - Run CleanCSS minification (expensive operation)
				 * - Save result to disk cache for next time
				 */
				let processedCSS;
				if (fs.existsSync(cachePath)) {
					// Cache hit - load pre-minified CSS
					processedCSS = await fs.promises.readFile(cachePath, "utf8");
				} else {
					// Cache miss - minify and save
					processedCSS = cleanCSS.minify(inputCSS).styles;
					await fs.promises.writeFile(cachePath, processedCSS);
				}

				/**
				 * Stage 5.6: Write Processed CSS to Output Directory
				 *
				 * Write the minified CSS to the build output directory with a
				 * hash-based filename. The hash in the filename enables:
				 * - Cache busting (browser cache invalidates when content changes)
				 * - Long-term caching (files with same hash are unchanged)
				 *
				 * Filename format: original-name-hash.css
				 * Example: main-a1b2c3d4e5.css
				 *
				 * Only write if file doesn't exist to avoid redundant disk I/O
				 * (useful when multiple pages reference the same CSS).
				 */
				const parsedPath = path.parse(inputFile); // Parse original filename
				const finalFilename = path.join(
					outputDirectory,
					`${parsedPath.name}-${hash}${parsedPath.ext}`, // name-hash.css
				);
				if (!fs.existsSync(finalFilename)) {
					await fs.promises.writeFile(finalFilename, processedCSS);
				}

				/**
				 * Stage 5.7: Brotli Compression
				 *
				 * Compress the minified CSS using Brotli algorithm.
				 * Brotli provides better compression than gzip, especially for text.
				 *
				 * The .br extension indicates Brotli-compressed files.
				 * Web servers can serve these directly to browsers that support Brotli
				 * (most modern browsers do via Accept-Encoding header).
				 *
				 * Compression happens synchronously (brotliCompressSync) because:
				 * - It's fast enough for build-time processing
				 * - Simplifies error handling
				 * - Build processes typically prefer synchronous operations
				 *
				 * Only compress if file doesn't exist (avoid redundant compression).
				 */
				const brotliFilename = `${finalFilename}.br`; // Add .br extension
				if (!fs.existsSync(brotliFilename)) {
					const brotliOptions = {
						level: brotliCompressionLevel, // Compression level (0-11)
					};
					const brotliBuffer = brotliCompressSync(
						Buffer.from(processedCSS), // Convert string to Buffer
						brotliOptions,
					);
					await fs.promises.writeFile(brotliFilename, brotliBuffer);
				}

				/**
				 * Stage 5.8: Generate Final HTML Output
				 *
				 * Create the HTML <link> tag that will be inserted into the page.
				 * The path is relative to the site root and uses forward slashes
				 * (normalized for web URLs, works on all platforms).
				 *
				 * Note: The HTML references the .br file, assuming the web server
				 * can serve Brotli-compressed files when the browser supports it.
				 * If your server doesn't handle .br files, you may need to modify
				 * this to point to the uncompressed file or configure your server.
				 */
				const hashedPath = brotliFilename
					.replace(path.join("./_site"), "") // Remove build directory prefix
					.replace(/\\/g, "/"); // Normalize path separators for web URLs

				const result = `<link rel="stylesheet" href="${hashedPath}">`;

				/**
				 * Stage 5.9: Cache Result in Memory
				 *
				 * Store the processing result in the in-memory cache for subsequent
				 * calls during this build. This includes:
				 * - hash: For reference/debugging
				 * - processedCSS: In case we need the CSS string later
				 * - htmlOutput: The final HTML tag (what we return)
				 *
				 * Replace the Promise in cache with the actual result object.
				 */
				cssBuildCache.set(cssPath, { hash, processedCSS, htmlOutput: result });
				return result;
			} catch {
				/**
				 * Error Handling
				 *
				 * Catch any errors during processing and fail gracefully.
				 * Return empty string to prevent one CSS file error from breaking the entire build.
				 */
				return "";
			}
		})();

		/**
		 * Stage 6: Handle Concurrent Requests
		 *
		 * Store the processing Promise in the cache BEFORE awaiting it.
		 * This ensures that if multiple pages call this shortcode simultaneously
		 * for the same CSS file, they all get the same Promise and wait for
		 * the same processing to complete (no duplicate work).
		 *
		 * Once processing completes, the Promise in cache is replaced with
		 * the result object (see Stage 5.9).
		 */
		cssBuildCache.set(cssPath, processingPromise);
		return await processingPromise;
	});
}

It's all a little clumsy to have the code only in the blog post, so I've created a gist on GitHub here (opens in new tab) for easier reading & copying and pasting (should you wish to use or modify it yourself).

Usage

So how would you use this code in 11ty? Let's go through each of the files you'd need to modify.

eleventy.config.js

Just your standard ESM 11ty config file with an import and the execution.

// Custom CSS manipulation
import { manipulateCSS } from './_helpers/css-manipulation.js';

export default async function(eleventyConfig) {
	// All your other 11ty config above...
	// execute the CSS manipulation
	manipulateCSS(eleventyConfig);
	// All your other 11ty config below...
}

.env

It sits in the 11ty root directory and is added to .gitignore.

ELEVENTY_ENV=development

HTML template

I, personally, have my HTML <head> separate and sitting in its own Nunjucks partial, but it will work with any template setup really. This Shortcode is using Nunjucks because that's what I use on this blog. Spaces added in the Nunjucks to stop 11ty just printing the output. The index.css is my one and only CSS file I use on this blog, and it is passed into the Shortcode. It is either used unmodified (in development), or manipulated (in production).

<head>
	{ % customCSS "/css/index.css" % }
</head>

Output

For my development environment the HTML output is:

<head>
	<link rel="stylesheet" href="/css/index.css">
</head>

There's no fingerprinting until it is built on production (in my case Cloudflare Pages). Where the output looks like this:

<head>
	<link rel="stylesheet" href="/css/index-b9fcfe85ef.css">
</head>

As you can see the index.css file that contains all the CSS for this site, now has a unique 10 character hash suffix. If I were to change a single byte of data in the CSS file it will generate an entirely new hash. For those wondering, a sha256 hash using only the first 10 characters still has 1,099,511,627,776 unique combinations, so the likelihood of a collision is pretty slim!

Since the name of my CSS file is now guaranteed to have a unique name I can use the following response headers for my CSS:

_headers file

We can now start to use Cache-Control directives that maximise the time that the CSS is stored in a user's browser cache.

/[css]/*
Cache-Control: public, max-age=31536000, immutable

Here we are telling the user's browser that:

public: This response can be cached by any cache, it isn't restricted to only a private cache (like a user's browser cache). Examples of other caches are CDNs, and proxies.
max-age=31536000: 31536000 seconds is the number of seconds in a year. So, this directive tells caches that they can store and reuse this response for up to a year without the need for revalidation.
immutable: This is a hint to the browser or cache system that the content will never change for the lifetime of the cache. Even if the user reloads the page, the browser won't bother checking with the server for updates.

There's a lot involved in caching on the web, so if you want to know a lot more about this fascinating topic, check out Cache-Control for Civilians by Harry Roberts.

For those wondering about what happens to the old "orphaned" files in the cache, in this instance:

The old file version will stay in the cache for up to 1 year (until the cache duration expires).
But it's more than likely that the browser will clean up the cache before the year expiry time to free up storage space. This is more likely on devices with limited resources like older phones, since devices with limited resources will need to be more aggressive with their resource management.
In either case, this isn't anything a user has to worry about. The browsers will take care of this automatically, although a user can manually clear their browser cache should they wish too.

Summary

There we go, another 11ty blog post! We've reviewed my current CSS setup, which works well for now, but as mentioned earlier, feel free to suggest improvements or point out anything that strays from the "11ty way". As, the path to true enlightenment starts with uncovering the unseen gaps in our (my) understanding. I read that last sentence in Master Yoda's voice! As always, thanks for taking the time to read the post. I hope you found it useful and informative, and if you have any comments or feedback, please do let me know.

Post changelog:

12/01/25: Initial post published.
02/11/25: Updated post and code after shortcode was found to run for every page generation in the build process. This was inefficient and is now limited to a single run per build.

Cloudflares Mirage 2.0 broke my images on all mobile devices

2025-01-07T00:00:00Z

I'm writing this (hopefully short) blog post to warn others about the situation I found myself in the other day. As I've mentioned in my previous posts, I recently migrated everything from Jekyll and GitHub Pages to 11ty and Cloudflare Pages. This was quite a significant migration that took a while to complete! When I started the migration, I was on Cloudflare's Free plan. As I have been for several years. This is important, and I will revisit this point later!

Once the initial migration was completed, I tested the new code, design, and functionality on every device I could get my hands on, just to make sure there weren't any obvious bugs I'd missed. In doing so, I actually found a cross-browser layout bug where the latest Firefox and latest Chrome differed. I was quite surprised by this, but it shows you should always test on different browsers! Especially ones with differing rendering engines! Anyway, I'm rambling off-topic again!

By this point, I thought everything was looking pretty stable. I had the basics in place, the site was performing well, and accessible (according to Lighthouse anyway!), I'd locked it down in terms of security (another important point!).

New year, New Cloudflare Plan

Unknown to me, I'd set up an unfortunate chain of decisions that would lead to my images being broken on all mobile devices for exactly 3 weeks without me knowing (I know, a bit dramatic, and a very 1st world problem!).

In December, I'd decided to pay annually for the Pro Cloudflare plan for 2025. Unfortunately for me, what they don't mention when this happens is that certain "Pro-only" functionality is enabled with this upgrade process. One of those features happens to be a performance optimisation called Mirage 2.0. I do love the image that they have used in that blog post, and its clever play on words.

Mirage 2.0

Mirage 2.0 is Cloudflare's latest version of the image optimisation library. Mirage 1.0 was released in June 2012.

Cloudflare's Mirage 2.0 is a web performance optimisation tool focused on enhancing image delivery for mobile users by reducing page load times and bandwidth usage. It uses techniques like lazy loading and adaptive image resizing to serve images tailored to users' devices and network conditions, ensuring faster load times even on slow connections. Mirage dynamically prioritises image loading, starting with low-resolution placeholders and upgrading to higher-quality versions as needed, optimising user experience while conserving data. Integrated into Cloudflare's CDN, Mirage 2.0 provides seamless optimisation without extra effort from website operators.

The emphasis in the above paragraph is mine. Just to be clear, I had no idea Mirage 2.0 even existed until the day all my images were randomly broken on mobile devices!

Event timeline

So what exactly happened to get to this point? Well, I:

Migrated from Jekyll and GitHub Pages to 11ty and Cloudflare Pages.
Completed the site migration, optimised web performance, accessibility, and security.
Tested my shiny new site in numerous browsers (including mobile browsers).
Upgraded to the Cloudflare Pro Plan.
Mistakenly (maybe?), didn't retest the site across browsers after the upgrade (since I assumed nothing would change regarding the platform and environment!).
Made sure site still rendered perfectly on Desktop (yay!), and discovered the Pro plan even allowed me to enable Brotli compression for all assets! Winning!
Was then informed by Matteo Contrini that my images were broken on mobile off the back of announcing my new blog post on Bluesky.
Panicked, confusion… argh!

It's point 8 where the self-doubt creeps in:

"Have the images ever worked on the site??"
"I'm 100% sure I tested on mobile!"
"Surely someone would have mentioned this to me if they'd never worked??"
"Why does it work on desktop but not on mobile??"
"Why do the images work when I request the desktop site on mobile!!"
"Is it the new eleventyImageTransformPlugin? If so, surely, it would have been spotted and fixed already!"
"How on earth am I going to debug and resolve this issue?"

That list of quotes above was basically my thought process for a couple of hours after the image problem was reported!

Web Inspector for iOS

Thankfully, Matteo jumped in and helped with the final quote on the list above when he pointed me in the direction of the fantastic Web Inspector App for iOS. Once installed and enabled as a Safari extension, it allows you to easily debug web pages on iOS Safari. It comes with tabs for the:

DOM
Elements
Console
Network
Resources

In fact, if you look at the code seen in the inspector in the image, you will see how I finally worked out what was going on!

Script Injection

On upgrading to Cloudflare Pro, unbeknown to me, the Mirage functionality is automatically enabled within the Cloudflare dashboard! So every image on every page has the mirage 2.0 script injected into the DOM (but only on mobile! Remember my added emphasis in the Mirage 2.0 section earlier). Also remember how I'd locked down my blog's security using HTTP response headers. Which just so happened to include a Content Security Policy (CSP)!

Script Injection + CSP equals pain

I think most readers may realise where I'm going with this now! On the one hand, it's fantastic to know that the CSP did its job and blocked the 3rd party script from ajax.cloudflare.com! I mean, that's what it's there for, after all! But what's not so great is the fact that this injection only happens on a mobile device where there are no console errors to be seen (by default anyway). It's also worth mentioning that at this point it was still a complete guess that this was the issue, as even with the Web Inspector Extension installed. There was no usual CSP error (Content Security Policy: The page's settings blocked the loading of a resource at...) in the console, like you get in a desktop browser! So I was just hoping that this was the issue!

Disabling Mirage

So, the easiest way to resolve the issue is to disable Mirage 2.0 from the Cloudflare Dashboard, easier said than done at times! Cloudflare I adore your platform and the features you offer, but your dashboard navigation is just nuts! Please consider a major overhaul of the whole UI! Furthermore, don't hesitate to put me down as a beta tester as well!

So to disable Mirage, first select the domain you want to change, then look for the "Speed" dropdown on the left, then click the "Optimization" link:

Next, look for the "Image Optimization" tab on the page:

Lastly, scroll to the bottom of this tab, and you will find the Mirage setting you can toggle on and off:

Unsurprisingly, the setting is now turned off on my dashboard! Also note the "Beta" tag in orange next to the title…

What have we learned?

So, what have we learned from my stressful little adventure?

CSP's actually work really well for blocking third-party JavaScript code injection!
Be sure to check your website often, especially if you happen to change your Cloudflare subscription plan.
Never fully trust anyone, not even your CDN or hosting provider! Even they may choose to meddle with your site's code!
The Web Inspector App for iOS is really useful for debugging mobile Safari issues.

Steps Cloudflare can take to prevent this

So, I'm pretty sure this isn't my fault, right?? I'd like to think I've just been unlucky with having a particular setup and a unique set of circumstances. Saying that, I'm certain that Cloudflare can introduce some changes to stop this from happening to others! My recommendations would be:

Ensure that Mirage 2.0 is Opt-in only and not Opt-out. Especially for a feature that is still in Beta!
When a user upgrades their plan, consider emailing them, or pointing out the new features that are available to them with the new plan (maybe within their dashboard?)
Consider some sort of automation or logging to check to see if Mirage (or other functionality) is causing, or could cause issues if the site has a CSP in place.
Don't assume that your paying customers know everything about every product on your platform.
Finally, it's also worth mentioning to users that there have been similar image issues caused by Mirage in the past. Documented here and here. Although, admittedly, these are related to Vercel and Next.js, so a fairly different setup!

I hope that this blog post isn't coming across too harshly. As I've said before, I really love Cloudflare as a company, and the services they provide. It's just when they overstep the mark and make assumptions about the websites they are hosting!

One final point, I briefly chatting to Andy Davies from Speedcurve on Bluesky. He mentioned they (Speedcurve) recommend all clients that use Cloudflare disable both Mirage AND Rocket Loader — Source

Summary

If you made it to the end of this post, congratulations—you deserve a medal! Meanwhile, I've learned I couldn't write a short blog post if my life depended on it!

Nevertheless, I hope you found it a useful read, and it helps someone who stumbles across this completely unrelated set of circumstances! As always, thanks for reading, and if you have any feedback or comments, please do let me know!

Post changelog:

07/01/25: Initial post published.

Cranking Brotli up to 11 with Cloudflare Pro and 11ty

2025-01-05T00:00:00Z

As with the past few blog posts I've written, this post is about my migration from GitHub Pages and Jekyll to Cloudflare Pages and 11ty. Once the migration was completed, I decided to have a look at how I could optimise the web performance of my blog. As with all web performance projects I start, I begin with the basics. For me, that's the 3 C's:

For assets being delivered to your user, make sure they are properly:

Cached
Concatenated
Compressed

Before

Upon looking at the DevTools Network Panel in Firefox (my main browser of choice 😍), I could see that only my HTML document was being Brotli compressed:

Before I get into the details, let's discuss what the difference between Gzip and Brotli is. If you already know all this, feel free to skip this section.

What is Brotli compression?

Brotli is a "new" compression algorithm that was released by Google in September 2015 (hence the quotes around the new). Brotli came with the following key features:

Lossless Compression — No data is lost during compression / decompression.
High Compression Ratios — It achieves higher compression ratios compared to other popular compression algorithms used on the web, like gzip, and deflate.
Optimal for Web Performance — Brotli is specifically optimised for HTTP compression, its predefined static dictionary is optimised for patterns often found in web content. It's high compression ratios, and fast decompression, make it a perfect choice for HTTP compression.
Static and Dynamic Compression — Brotli supports both dynamic (real-time) and static (pre-compressed) data compression, offering a balance between server CPU usage and optimal file size. Dynamic compression suits real-time needs, while static compression, using the highest setting (11), maximises asset compression during the build process. Pre-compressed assets can then be delivered with minimal CPU overhead.

It's these key features among many others that have made Brotli one of the preferred compression algorithms on the modern web. According to Can I Use, as of today, 97.79% of browsers on the web support Brotli Accept-Encoding/Content-Encoding.

I, personally, love that Brotli has compression settings ranging from 0 to 11, where 0 is the fastest compression speed but the lowest compression ratio, and 11 has the highest compression ratio but requires more computational resources and time. I'd love to know if the Google developers got the idea for a compression level of 11 from the very famous scene from the film 1984 film "This Is Spinal Tap".

If you are interested in learning more about Brotli compression, here's some further reading for you:

Brotli and Cloudflare

Cloudflare has been a strong advocate for Brotli for many years, introducing support for it on September 21, 2016. Remarkably, this was just over a year after Google open-sourced Brotli on September 22, 2015, and only a few months after its formal standardisation in RFC 7932 in July 2016. Demonstrating impressive speed, Cloudflare rolled out Brotli support in just over two months, significantly ahead of one of their main competitors, Akamai, which implemented support on March 19, 2018—almost 18 months later.

At first, they were very explicit with their support, offering their users the option to toggle it on and off via the "Optimization" menu in their site dashboard:

Since then, Brotli has become so ubiquitous for web performance on the web that this toggle was removed from the dashboard in May 2024, as announced on their community forum here. It is now enabled by default for the following file types. Unfortunately, when moving from beta to live, Cloudflare decided that Brotli compression wouldn't be the default compression used on the free plan. Although, saying that, the free plan is still incredible for web performance! This sadly means that if you want all your website assets dynamically compressed (on the fly), by Cloudflare, then you will need to be on the Pro plan.

Another point to know about the Pro plan and Brotli compression is that it uses dynamic "on the fly" compression. It therefore uses a reduced compression setting. A happy medium between compression size and CPU usage happens to be around the 3 or 4 setting (out of 11). Cloudflare has chosen 4 to be the setting they use for dynamic compression. Depending on the content, Brotli's compression ratio at level 4 is approximately 2x to 4x, which is considered balanced compression. This level provides a good balance of speed and compression.

But since we are controlling the compression level on our static blog, let's crank it all the way up to 11! This is defined as maximum compression, and offers a compression ratio of approximately 4x to 7x smaller than the original file size. The use case for this level of compression is it's often used for static assets that don't change regularly, such as font files or infrequently updated files. In my case, once written, I don't plan to update the minimal amount of JavaScript in use on my blog. Hence, the JavaScript files are the perfect candidate for extreme compression. So let's get started.

Compression

Installation

The simplest way to compress your assets is by using a command-line tool in the terminal. To get started, you'll first need to install the Brotli Command-line Interface (CLI). On macOS, you can do this by either:

Using Homebrew

You may or may not have Homebrew already installed. Here I assume you haven't. To install it, you'd run the following commands in the terminal.

Install:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Ensure Homebrew is up-to-date:

brew update

Install Brotli:

brew install brotli

Verify Installation:

brotli --version

This will return the installed Brotli version. You are now ready to compress your assets using Brotli.

Using MacPorts

First, follow the MacPorts installation here.

Then update and install:

sudo port selfupdate
sudo port install brotli

Verify Installation:

brotli --version

These are just 2 ways of installing Brotli on OSX, others include:

Build from source (using Xcode).
Install via Python's pip.
Download Precompiled Binaries (via the Brotli releases page).
Use Docker.

Each method has its benefits depending on your environment and preferences. Once installed, the usage is the same for all methods listed above.

Usage

To compress a file, it's as simple as:

brotli [filename-here]

Or if you want to control the compression settings:

brotli --quality=[0-11] [filename-here]

This will output a Brotli version of the file so file.txt will be duplicated, and a new file called file.txt.br will be created. A shortcut for --quality=11 is --best:

So these commands would produce the same output:

brotli --quality=11 file.txt
brotli --best file.txt

For lots more information on the command-line options available, use:

brotli -h

Compression script — a single file

Now you can either do this manually, which is long and laborious, or we can semi-automate the task by writing a shell script (Thanks to ozcoder for the idea, and initial code!).

compress.sh

#!/bin/bash

# Check if the user provided a file name
if [ -z "$1" ]; then
	echo "Usage: $0 <filename>"
	exit 1
fi

# Get the file name from the first argument
fname="$1"

# Compress the file using Brotli at quality 11
brotli --quality=11 --output="${fname}.br" "${fname}"

# Notify the user
if [ $? -eq 0 ]; then
	echo "File compressed successfully: ${fname}.br"
else
	echo "Error occurred during compression."
	exit 1
fi

Remember to make the script executable using chmod +x compress.sh. Then you can use it like so:

./compress.sh filename.js

A Brotli compressed file (filename.js.br) will now be located in the same directory.

Compression script — whole directory

If you find manually compressing each file using the compress.sh script above tedious, then we can modify the shell script to compress all JavaScript assets in a single directory (compress-directory.sh):

compress-directory.sh

#!/bin/bash

# Check if the user provided a directory
if [ -z "$1" ]; then
	echo "Usage: $0 <directory>"
	exit 1
fi

# Get the directory from the first argument
directory="$1"

# Check if the specified directory exists
if [ ! -d "$directory" ]; then
	echo "Error: Directory '$directory' does not exist."
	exit 1
fi

# Compress all .js files in the specified directory (modify for other file types)
for file in "$directory"/*.js; do
	# Check if there are .js files in the directory
	if [ ! -e "$file" ]; then
		echo "No .js files found in '$directory'."
		exit 0
	fi

	# Compress the file using Brotli
	brotli --quality=11 --output="${file}.br" "$file"

	# Notify the user of success or failure
	if [ $? -eq 0 ]; then
		echo "Compressed: $file -> ${file}.br"
	else
		echo "Error compressing: $file"
	fi
done

echo "Compression complete."

Once either of these scripts have completed you will have one or more Brotli encoded files e.g. filename.js AND filename.js.br. You can now delete the original JavaScript file and remove the .br extension from the newly compressed file. Assuming you are using Git (which, of course, everyone does, right?), you can always retrieve the original JS file from the Git history should you need the uncompressed version again, or you can use the brotli --decompress filename.js.br command. Remember that Brotli is a Lossless compression algorithm, there's no data loss during compression and decompression, so you can always decompress to get the original file back.

My scripts are broken on localhost

So now that your JavaScript has been Brotli compressed, it's no longer in a plaintext format. Brotli compressed files are actually in a binary format, so they aren't directly editable in your code editor. These errors are in the console because presently, the local browser is expecting to receive the JavaScript in plaintext, so when it doesn't, it will let you know via the console with an error that says something like Uncaught SyntaxError: illegal character U+FFFD. Now, this is a scary looking error, but there's an easy fix. We need to tell our browser that these JavaScript files are no longer plaintext, they are Brotli compressed. To achieve that, we are going to modify our 11ty config file (e.g. eleventy.config.js) and add in some node middleware that our local 11ty development server will use.

// The rest of your eleventy.config.js code above...
eleventyConfig.setServerOptions({
	middleware: [(req, res, next) => {
		if (req.url.endsWith('.js')) {
			res.setHeader('Access-Control-Allow-Origin', '*');
			res.setHeader('Content-Encoding', 'br');
			res.setHeader('Content-Type', 'application/javascript');
			res.setHeader('Vary', 'Accept-Encoding');
		}
		next();
	}]
});
// The rest of your eleventy.config.js code below...

The code above will hopefully be fairly explanatory. It's basically saying when you serve a JavaScript file from localhost, make sure to include the above response headers. The critical one is the 'Content-Encoding', 'br' response header, which tells the browser that the files are now Brotli compressed. It then recognises that the response body has been compressed using the Brotli algorithm. The browser will then decompress the Brotli-compressed data before interpreting or rendering the content as it would do with a plaintext version.

Cloudflare Compression rules setup (Pro account)

Now that the development environment has been fixed, it's time to dive into the Cloudflare dashboard and make sure we have everything setup in there!

If you have a pro account, you will see a "Compression Rules" menu item under the "Rules" section of the navigation.

Once inside this menu, you will see an option to create a new compression rule:

You will see I've already created a "Brotli Compression" rule in the image, and it is enabled.

Within this section, you will see a long page with 2 main sections:

I added a name and just left the option on the default content types to be compressed.

Next, we need to set the compression settings we want to apply to these files being served to our users:

You will see in my settings I've been very explicit in the order of compression I would like: "Brotli, Zstandard, Gzip, Auto". Zstandard (Zstd) is generally considered better than Gzip in terms of performance and compression ratio for most use cases. Refer to this page here for more information about Zstandard.

After

Once all the settings have been updated, and we have cleared the CDN cache! We can see the final result from the network panel below:

Looking at the far-right column from the network panel, we can see that all my assets are now Brotli compressed.

Although the panel doesn't show it, the assets are being compressed using a mixture of dynamic and static compression:

HTML document — Dynamic (level 4) — by Cloudflare.
CSS file — Dynamic (level 4) — by Cloudflare.
JavaScript files — Static (level 11) — by me.
Favicon — Dynamic (level 4) — by Cloudflare.

If I wanted to, I could statically compress the favicon.ico file too, since that's never going to change. But I'll leave that to you the reader to figure out how to do that, should you wish to!

Summary

It's a real shame that you need a Cloudflare Pro account to use Brotli compression for all static assets served by Cloudflare. But their free account does give you a tremendous number of features for web performance and security out of the box, including:

HTTP/2, and HTTP/3 + QUIC.
0-RTT.
Cloudflare Workers.
DNSSEC.
Excellent cacheability of static assets.
Email security and forwarding.
Free hosting on Cloudflare Pages.
Content Delivery Network (CDN) with global caching.
Page Rules for fine-grained cache control.
Improved security.
Image optimisation.
Improved web performance.
DDoS protection.
Basic Web Application Firewall (WAF) rules.
SSL/TLS encryption (Free Universal SSL).
Bot mitigation.
IP masking via reverse proxy.
Free DNS hosting with fast resolution.
plus many other features too…

I'm genuinely interested to know if this can all be achieved on the free plan?

For example, by compressing your assets with Brotli (11), then setting your Pages _headers file to serve the 'Content-Encoding', 'br' response header along with these compressed assets. If so, I probably wouldn't recommend it as it may be against one of the Cloudflare Website and Online Services Terms of Use. And sounds like a quick way to get your account deleted by Cloudflare! If anyone tries this please do let me know! I really wish Cloudflare had a referral program, I'd probably get my Pro account paid for in no time!

And there we have it, another blog post off the back of my migration to 11ty. As always, thanks for reading, and I hope you found it useful and informative! If you have any feedback or comments, please do let me know!

Post changelog:

05/01/25: Initial post published.
05/01/25: Thanks to Barry Pollard for pointing out that I'd broken the shell scripts by adding a comment before the shebang! Doh!

Refactoring a Web Performance Snippet for Security and Best Practice

2025-01-02T00:00:00Z

In this blog post, I'm going to discuss a little Web Performance Snippet that I've seen a few WebPerf evangelists use on their websites. As you will have seen in a previous blog post, I've recently overhauled my blog, both in terms of design and also static hosting. In doing so, I've completely rewritten almost everything from my old site by either migrating it across and "cleaning it up", or simply realising the feature was no longer useful and discarding it, thus reducing technical debt in order to streamline maintenance in the future.

Security

In my last blog post, I touched on improving the security of this blog. And this is the reason for this post (in a very roundabout way!). For the past few years, since I started strengthening my Web Performance knowledge, I've had a little code snippet in my footer. It simply queries the Performance API and outputs an elementary Page Load time via the use of loadEventEnd property. Unfortunately, these are now deprecated features of the PerformanceTiming API. But as they still work, I've had no reason to change it. I'd love to say I came up with this idea and code myself, but I didn't. I actually copied this feature from Tim Kadlec several years ago, it can still be seen in the footer of his website today (bottom-right corner of his website):

Examining the code that outputs this paragraph to the page, I can see it looks like this:

window.onload = function () {
	setTimeout(function () {
		window.performance =
			window.performance ||
			window.mozPerformance ||
			window.msPerformance ||
			window.webkitPerformance ||
			{};
		var t = performance.timing || {};
		if (!t) {
			return;
		}
		var start = t.navigationStart,
			end = t.loadEventEnd;
		loadTime = (end - start) / 1000;
		var copy = document.querySelectorAll(".copy");
		copy[0].innerHTML +=
			"<p class='loaded'>This page loaded in <strong>" +
			loadTime +
			" seconds</strong>.</p>";
	}, 0);
};

And this is the code I had running for many years on my old website. I liked having this code on my website as it gave me an extremely basic statistic on how well the site loaded on any given device I happen to use (e.g. an older mobile device, or when I'm using an unstable connection in a rural area of the country). It could also be used to verify there's no obvious performance problems. For example, if I see that the metric is anywhere over 500 ms (on my laptop) then that's a good indication that something isn't performing as it should be. Over 500 ms for such a simple static page on an 2020 MacBook Air (M1) would be a real worry!

Updating the code

Unfortunately, there are a few issues with this code that wouldn't work with my newly updated website. The biggest of all being the use of the setTimeout() which is a type of eval() and is a huge security risk. You only need to look at the MDN Page for eval, and it comes with a big red warning at the top of the page:

They even have a whole section on the page as to why you shouldn't do this.

For my newly built blog, I planned to update my blog to use a fairly strict Content Security Policy (CSP). And as I intended to use the unsafe-inline CSP directive AND didn't want to go down the Nonce-based strict CSP or Hash-based strict CSP route due to the additional complexity it adds to the site. For these reasons, and also for something to do over the Christmas Holiday period, I thought I'd just refactor the code and use something more modern, like the PerformanceObserver API.

In doing so, I'd remove the need for the setTimeout code:

setTimeout(function () {}, 0);

The above code uses something called "macro-task scheduling". This is where the browser adds this function to the end of the event loop queue. In doing so, it ensures:

The load event handler has finished.
The browser has had time to populate all the performance timing values.
The performance.timing.loadEventEnd property has a valid value.

Without this macro-task scheduling code, the loadEventEnd value won't have populated by the time it is required, which would likely result in a Not-A-Number (NaN) value being displayed on the page (or alternatively a non-helpful value of 0).

Refactored Code

In the updated code below I refactored it by:

using the Optional chaining operator for detecting window.performance and window.performance.getEntriesByType.
removing the setTimeout "macro-task scheduling" code for greater security (no more eval!).
using a modern API for optimal code execution (PerformanceObserver API).
Wrapping the code in an Immediately Invoked Function Expression (IIFE) to ensure the browsers global scope isn't polluted with random functions and variables.
using Template literals (Template strings) for concatenating the final string that is injected into the DOM.
replacing the deprecated PerformanceTiming API and have migrated to the PerformanceNavigationTiming API.

<!-- HTML in my footer -->
<footer>
	<script src="/js/webperf.js" defer></script>
</footer>

/**
 * Web Performance Snippet — Page Load Time Display (webperf.js)
 *
 * Displays a simple "This page loaded in: N milliseconds." message in the
 * footer using the PerformanceNavigationTiming API. Used for quick,
 * device-level feedback on load performance (e.g. on slow connections or
 * older devices).
 *
 * Requirements: PerformanceNavigationTiming (getEntriesByType('navigation')).
 * Runs only in supporting browsers; no polyfills. No eval; safe for strict CSP.
 */
(function () {
	// Ensure the browser supports the Performance API and the method we need
	// to fetch navigation timing (PerformanceNavigationTiming entries).
	if (!window.performance?.getEntriesByType) {
		console.warn("Performance API not supported");
		return;
	}

	/**
	 * Reads navigation timing and injects the load-time message into the DOM.
	 * Uses fallbacks when duration or loadEventEnd are 0 (e.g. iOS Safari, bfcache).
	 */
	function showLoadTime() {
		const entries = performance.getEntriesByType("navigation");
		const nav = entries[0];
		if (!nav) return;

		// --- iOS Safari / WebKit 0ms fix and fallbacks ---
		// On iOS Safari (and some WebKit builds), reading timing too early can
		// yield duration === 0 or loadEventEnd === 0. For example, if we had
		// used PerformanceObserver with type "navigation", its callback can fire
		// before the load event has completed, so loadEventEnd is not yet set.
		// We avoid that by only calling showLoadTime after the load event (see
		// runAfterLoad). Here we still defensively handle any remaining 0 values
		// (e.g. bfcache restores, or timing quirks):
		// 1. Prefer nav.duration (equivalent to loadEventEnd - startTime when set).
		// 2. If duration is 0 but loadEventEnd is set, compute loadEventEnd - startTime.
		// 3. If both are 0, fall back to domContentLoadedEventEnd - startTime.
		let ms = nav.duration;
		if (!ms && nav.loadEventEnd > 0) {
			ms = nav.loadEventEnd - nav.startTime;
		}
		if (!ms && nav.domContentLoadedEventEnd > 0) {
			ms = nav.domContentLoadedEventEnd - nav.startTime;
		}
		if (typeof ms !== "number" || !(ms >= 0)) return;

		const pageLoadTime = Math.floor(ms);

		let loadTimeElement = document.getElementById("pl");
		if (!loadTimeElement) {
			loadTimeElement = document.createElement("p");
			loadTimeElement.id = "pl";
			const footer = document.querySelector("footer");
			if (footer) {
				footer.insertBefore(loadTimeElement, footer.firstChild);
			}
		}
		if (loadTimeElement) {
			loadTimeElement.textContent = `This page loaded in: ${pageLoadTime} milliseconds.`;
		}
	}

	/**
	 * Runs showLoadTime after the load event, using a single setTimeout(0) so
	 * the read happens in a subsequent macro task. This is the main fix for the
	 * iOS issue where the snippet would otherwise show "0 milliseconds":
	 * - On iOS Safari, if we read navigation timing before the load event has
	 *   fully completed, loadEventEnd (and thus duration) can still be 0.
	 * - By attaching to the window "load" event and then deferring the read
	 *   with setTimeout(showLoadTime, 0), we ensure the browser has set
	 *   loadEventEnd before we read from the Performance entry. We do not use
	 *   PerformanceObserver for the initial read for this reason.
	 */
	function runAfterLoad() {
		setTimeout(showLoadTime, 0);
	}

	if (document.readyState === "complete") {
		runAfterLoad();
	} else {
		window.addEventListener("load", runAfterLoad);
	}
})();

Here's a link to GitHub Gist (opens in new tab) for the code above (with less comments).

In refactoring Tim's code, I deliberately chose to exclude certain older browsers and their users from accessing this feature. The refactored code includes a check for the PerformanceNavigationTiming API, ensuring the web performance snippet is only added to the footer if the browser supports this feature. This prevents older browsers from displaying a partially broken footer. Interestingly, this approach aligns with a modern take on a methodology Tim already uses on his website. His website follows a progressive enhancement technique known as "cutting the mustard", a concept introduced by the BBC News web development team in April 2013!

Cutting the mustard is a technique that involves categorising browsers into two groups: older "feature browsers" with limited capabilities and modern "smart browsers" that support advanced features. By adopting a two-tiered approach to responsive web design, a core experience is delivered to all users, while additional enhancements are applied for more capable browsers. This ensures broad accessibility while providing a refined and feature-rich experience for modern devices like smartphones and larger screens. I actually used my own version of this in a previous blog post back in November 2019, when I implemented Webmentions purely in a client-side JavaScript in my blog post: "Implementing Webmentions on this blog". TL;DR: Gist here (opens in new tab).

Thankfully, most modern browsers now auto-update every 6-weeks without users even knowing, so these older browsers are becoming less and less of an "issue".

For example, let's have a look at the refactored features I mentioned above:

PerformanceObserver API

This API allows developers to asynchronously observe and collect performance-related metrics, such as resource loading times, navigation events, and custom user timing marks. It listens for entries in the Performance Timeline, filtering them by type (e.g., resource, navigation, paint) and executes a callback function whenever new entries are available. This API is useful for real-time performance monitoring (RUM) and analytics, e.g. Google Analytics.

The PerformanceObserver API was introduced into most major browsers between 2016 and 2017, Microsoft Edge was the last to introduce the API in January 2020 when they migrated over to using the Chromium browser engine, rather than EdgeHTML.

According to Can I Use, 96.64% of users globally now use a browser that supports the PerformanceObserver API.

Template Literals

Template literals in JavaScript are string literals enclosed by backticks (`) that allow for embedded expressions using the ${expression} syntax. They support multi-line strings, string interpolation, and special constructs like tagged templates for advanced processing of literals.

Template Literals were introduced into the JavaScript Language in ECMAScript 6 (ES6) which was "released" in 2015. Amazingly, most major browsers managed to implement the new specifications in the same year (2015), including iOS Safari 9.2 AND Edge 13 (EdgeHTML).

According to Can I Use, 97.97% of users globally now use a browser that supports the Template Literals language feature.

PerformanceTiming API

This API is part of the deprecated Navigation Timing API, that provides detailed timestamps for key events in the navigation and page load process, such as DNS lookup, server response, DOM processing, and resource loading. Accessible via performance.timing, it allows developers to measure metrics like page load time, Time to First Byte (TTFB), and DOM readiness. While useful for diagnosing performance bottlenecks, it has been largely replaced by the more precise and structured PerformanceNavigationTiming API, available through the performance.getEntriesByType('navigation') method.

The PerformanceTiming API is now deprecated, but it was first supported by all major browsers back in 2015. Incredibly, IE9 even supported it way back in 2011!

It's not recommended that the API be used today because at some point browser vendors may consider removing it. According to Can I Use, 97.38% of users globally now use a browser that supports the PerformanceTiming API.

PerformanceNavigationTiming API

The API is a modern, high-precision replacement for the deprecated PerformanceTiming API, offering detailed metrics on navigation and page load performance. Accessible via performance.getEntriesByType('navigation'), it provides timestamps for events like DNS lookup, server response, DOM processing, and resource loading, along with new attributes like transferSize and decodedBodySize for enhanced analysis. It simplifies performance monitoring with a single object and is ideal for modern web applications, including Single Page Apps (SPA's), making it the preferred choice for navigation performance insights.

If you know you are still using the PerformanceTiming API, it is recommended that you update to use the PerformanceNavigationTiming API. Although it's unlikely that support for the older API will be fully removed, the new API offers so much more functionality and is a much easier API to use. You may as well take advantage of the new features and functionality that is built into all modern browsers.

According to Can I Use, 96.06% of users globally now use a browser that supports the PerformanceNavigationTiming API. It was supported all the way back in Edge 12 (EdgeHTML), and was finally supported in iOS Safari in version 15.3 back in December 2021.

Data Driven Decision-Making

So when it comes to browser support for modern features like this, how is it best to approach the old browser issue? In my opinion, you should approach it as a data collection exercise. If you have data to show that your site has browsers visiting that are approaching a decade old (at the time of writing), you should absolutely continue to "cut the mustard" to support those users.

Although I believe there's a very valid counterargument too. That by supporting such old and outdated browsers you're actually harming a users' experience of the web as a whole, by allowing them to use such an outdated browser. Also, can you imagine how vulnerable a user who is still using a browser like IE9 would be on the modern web? Finally point, the web performance barrier to entry of modern JavaScript-driven websites would make the internet completely unusable for these browsers! With a Median JavaScript size of 650 KB(!) as of December 2024, even a modern desktop browser would struggle!

Summary

So there we go, another blog post related to updating nooshu.com to 11ty and the steps I took to shed the years of technical debt I'd accumulated and move to a more maintainable and forward facing blog. Big thanks to Tim Kadlec for writing the original code snippet and inspiring me to add it to my blog! As always, I hope you enjoyed reading my random ramblings, and please do let me know if you have any comments or feedback.

Post changelog:

02/01/25: Initial post published.
04/03/26: Updated the code snippet and Gist with a fix for the iOS mobile browser (Webkit), as "0 milliseconds" was often the result on page load.

Securing your static website with HTTP response headers

2024-12-28T00:00:00Z

In this post, I'm going to go into how I secured my 11ty blog (this site), using Cloudflare pages and HTTP Response headers.

If you aren't bothered about all the information below, you can just jump straight to the code.

What is a response header?

An HTTP response header is a part of the response sent from a web server to a client (like your browser) when the client requests a resource, such as a webpage, image, or data.

The response header will contain additional information (metadata) about the request being made. Response headers come in three main categories:

Information about the response, for example the status code of the resource being requested. e.g. 200 for an OK response or a status of 404 for not found.
Details about the resource, for example a resource's Content Type e.g. text/html, text/css, application/javascript^†.
Instructions for the client, for example how long to store a resource in a browser's cache (caching rules).

†: I realise that technically Content-Type isn't actually a response header, it's actually a representation header, which provides metadata about the message body, such as its length, type, and encoding, helping the recipient understand and process the data correctly. But for simplicity's sake we'll consider it to be a response header in this post.

For the rest of the blog post, we are going to focus on number 3: "Instructions for the client". As with the security HTTP response headers, we will be instructing a user's browser as to what it can, and can't do with resources on a webpage.

Why? It's a static website!

Now I know what some of you will be thinking, since it's just static HTML and only hosted on a basic web server (with nothing dynamic), surely, there's nothing to secure? I mean, it's not like a hacker can hack into the Content Management System (CMS), and steal a username and password because in most cases there isn't one! This is true, but it's important to remember, static websites can be used for anything. I'm sure plenty of developers use static websites to build transactional websites that accept sensitive user data, like their address, Social Security Number (US), National Insurance Number (UK) or Credit Card details. So really, the security you are implementing isn't to protect you, it's to protect your users.

Here is a list of reasons below:

Preventing Cross-Site Scripting (XSS) Attacks
- See the Content-Security-Policy (CSP) section for details.
Securing Sensitive User Data
- See the Strict-Transport-Security (HSTS) section for details.
Reducing Information Leakage
- See the Referrer-Policy section for details.
Isolating Contexts
- See the Cross-Origin-Opener-Policy (COOP) and Cross-Origin-Embedder-Policy (COEP) sections for details.
Blocking Unintended Features
- See the Permissions-Policy section for details.
Preventing Clickjacking
- See the X-Frame-Options section for details.
Improving Browser Behaviour
- See X-Content-Type-Options section for details.
Enhancing Trust and SEO
- A secure website builds trust with users and browsers. Search engines also rank secure websites, potentially improving your site's SEO rankings.

It's important to remember that even static websites can serve as entry points for attackers, host phishing pages, or be used in more complex attack chains.

Just show me the code

It's worth mentioning that the examples I'm giving aren't specific to Cloudflare Pages or 11ty. They can be used with any static website, or indeed any website or web server (e.g. Apache, Nginx, IIS, Tomcat, Node.js).

The code below sits in my _headers file in the public assets directory of my 11ty blog on Cloudflare Pages. There's a tonne of information in the Cloudflare documentation on headers here if you're interested.

/*
  Access-Control-Allow-Origin: https://nooshu.com
  Cache-Control: public, s-maxage=31536000, max-age=31536000
  Content-Security-Policy: base-uri 'self';child-src 'self';connect-src 'self';default-src 'none';img-src 'self' https://v1.indieweb-avatar.11ty.dev/;font-src 'self';form-action 'self' https://webmention.io https://submit-form.com/DmOc8anHq;frame-ancestors;frame-src 'self' https://player.vimeo.com/ https://www.slideshare.net/ https://www.youtube.com/ https://giscus.app/ https://www.google.com/;manifest-src 'self';media-src 'self';object-src 'none';script-src 'self' https://giscus.app/ https://www.google.com/ https://www.gstatic.com/;style-src 'self' 'unsafe-inline' https://giscus.app/;worker-src 'self';upgrade-insecure-requests;
  Cross-Origin-Opener-Policy: same-origin
  Permissions-Policy: accelerometer=(), ambient-light-sensor=(), autoplay=(), camera=(), display-capture=(), document-domain=(), encrypted-media=(), fullscreen=(), geolocation=(), gyroscope=(), magnetometer=(), microphone=(), midi=(), navigation-override=(), payment=(), picture-in-picture=(), publickey-credentials-get=(), screen-wake-lock=(), sync-xhr=(), usb=(), web-share=(), xr-spatial-tracking=()
  Referrer-Policy: strict-origin-when-cross-origin
  Cross-Origin-Resource-Policy: cross-origin
  Strict-Transport-Security: max-age=63072000; includeSubDomains; preload
  X-Content-Type-Options: nosniff
  X-DNS-Prefetch-Control: off
  X-Frame-Options: DENY
  X-Permitted-Cross-Domain-Policies: none
  Origin-Agent-Cluster: ?1

Or I've created a GitHub Gist here (opens in new tab) if that's easier to copy, paste, and modify.

Just remember to change the Content-Security-Policy (CSP) and ensure it allows any 3rd-party assets you load on your website. For this, you are specifically looking at the following CSP directives:

form-action for form actions to a third party, e.g. a contact form.
script-src to allow the execution of scripts from a third party, e.g. Google Analytics.
img-src to allow images to load from a third party, e.g. Webmention images.
frame-src to allow <iframe> embeds to load on your website e.g. YouTube, Vimeo, Twitter

Other Static website hosting

I've also created Gists for other static website hosts below, if you don't use Cloudflare pages.

Important: These versions haven't been tested, so use them at your own risk! They are a direct port from my Cloudflare Pages _headers file to whatever format the specific host uses. Please do contact me if you find any errors in any of these versions and I will update the Gist and attribute the change to you in the post changelog.

Netlify

As a _headers file (opens in new tab) or a netlify.toml file (opens in new tab).

Vercel

As a vercel.json file (opens in new tab).

GitHub Pages

GitHub Pages itself does not support custom headers directly. Use a custom domain with a proxy service like Cloudflare to inject headers.

Cloudflare Pages

Repeating myself I know, but for anyone who just quick scans looking for a Cloudflare Pages version it is here (opens in new tab).

InfinityFree

An InfinityFree version can be found here (opens in new tab).

DigitalOcean

A complete nginx.conf config for DigitalOcean version can be found here (opens in new tab).
Or if you only require the location block for Nginx that can be found here (opens in new tab).

1&1 IONOS

1&1 IONOS itself does not support custom headers directly. Use a custom domain with a proxy service like Cloudflare to inject headers.

Header Specifics

To learn what each of these response headers does, continue reading below:

Access-Control-Allow-Origin

Description

This header specifies which origins (domains) are permitted to access the resources on the server via cross-origin requests. It is primarily used in handling Cross-Origin Resource Sharing (CORS).

Example

Access-Control-Allow-Origin: *

Recommended value

Instead of using *, which allows any origin to access the resource (less secure), specify the trusted origin explicitly: Access-Control-Allow-Origin: https://yourdomain.com

Recommended value sources

Specification

W3C Specification.

Cache-Control

Description

The Cache-Control header doesn't have anything to do with security, but it's an essential header for web performance as it gives the browser instructions on how an asset is stored or cached in the browser.

Example

Cache-Control: public, max-age=31536000, immutable

Recommended value

There are too many options for this particular header. If you are interested in learning more about efficient browser caching, I recommend checking out the following sources.

Recommended value sources

Cache-Control for Civilians by Harry Roberts.
Caching best practices & max-age gotchas by Jake Archibald.
[https://alokai.com/blog/cache-control-http-header] by Jakub Pepliński
Caching Header Best Practices by Simon Hearne.

Specification

W3C Specification.

Content-Security-Policy (CSP)

Description

The Content Security Policy (CSP) header is a security feature that protects web applications from attacks like Cross-Site Scripting (XSS) and code injection by controlling the sources from which browsers can load and execute content. If there's only a single response header you implement today, then make it the CSP Response header.

Example

Here's a basic example of a CSP, I recommend checking out the resources below to ensure your website is secured with a comprehensive Content Security Policy. Content-Security-Policy: default-src 'self'; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'; img-src 'self' data:;

Recommended value

There are so many directives and settings available for a CSP, it's impossible to define a one-size fits all value. The best approach is to ensure you have one for your website. Adding a CSP to an existing website can be complicated, luckily there are tools out there to help developers, a number of which are listed below.

Recommended value sources

Specification

W3C Recommendation — CSP2.

Cross-Origin-Opener-Policy (COOP)

Description

The COOP header controls whether a document can share its browsing context group with other documents from different origins. It helps isolate your web page from potentially malicious cross-origin interactions by preventing the sharing of resources like global objects and browsing contexts.

Example

Cross-Origin-Opener-Policy: unsafe-none

Recommended value

The recommended value isolates the page entirely from other origins. Cross-Origin-Opener-Policy: same-origin

Recommended value sources

Specification

HTML Living Standard.

Permissions-Policy (formerly Feature-Policy)

Description

Enables developers to specify which browser features can be accessed by different origins, both on the main page and within embedded frames.

Example

Permissions-Policy: geolocation=(), microphone=()

Recommended value

This isn't a one size fits all header, you basically disable any browsers features that you know your site will never use. A list of valid permission directives can be found here. In the above example we are disabling the geolocation and microphone browser functionality on all origins.

Recommended value sources

Specification

W3C Editor's Draft.

Referrer-Policy

Description

The Referrer-Policy HTTP response header determines the amount of referrer data to include in the Referrer header for outgoing requests. This security policy can either be set as a response header or in the <head> of the HTML document.

Example

Referrer-Policy: origin OR <meta name="referrer" content="origin" />

Recommended value

Referrer-Policy: strict-origin-when-cross-origin OR <meta name="referrer" content="strict-origin-when-cross-origin" />

Recommended value sources

Specification

W3C's Editors Draft.

Cross-Origin-Resource-Policy (CORP)

Description

The Cross-Origin-Resource-Policy (CORP) header is a security feature that controls whether resources like images, scripts, or fonts can be accessed by other websites to prevent unauthorized cross-origin access.

Example

Cross-Origin-Resource-Policy: same-origin

Recommended value

Cross-Origin-Resource-Policy: same-site

Recommended value sources

Specification

Fetch Living Standard.

Cross-Origin-Embedder-Policy (COEP)

Description

The Cross-Origin-Embedder-Policy (COEP) header is a security feature that ensures your webpage only embeds safe, explicitly authorized resources, protecting against vulnerabilities like data leaks or Spectre attacks. It is essential for sites using advanced web features like SharedArrayBuffer or WebAssembly to enhance security and performance.

Important: I've included this header for completeness, but I currently don't have this header enabled on this site. This is because the comment technology I use Giscus, doesn't support the COEP header. I have raised an issue on their GitHub Repository here, but unfortunately, there's yet to be any comments as to when it will be supported. So be careful with this header if you embed <iframe>'s into your site as this header may break the functionality.

Example

Cross-Origin-Embedder-Policy: unsafe-none

Recommended value

Cross-Origin-Embedder-Policy: require-corp

Recommended value sources

Specification

HTML Living Standard.

Strict-Transport-Security

Description

The Strict-Transport-Security (HSTS) header is a security feature that forces web browsers to communicate only over HTTPS with a website, ensuring that all subsequent requests are secure. It prevents attackers from exploiting protocols like HTTP or performing downgrade attacks. When enabled, the header includes parameters such as max-age (the duration for which the browser should enforce HTTPS), includeSubDomains (to apply to all subdomains), and preload (to include the site in browsers' HSTS preload lists). This header significantly enhances security by protecting users from man-in-the-middle (MITM) attacks and ensuring encrypted communication.

Example

Strict-Transport-Security: max-age=86400

Recommended value

The max-age recommendation of 2-years seems to be quite a contentious topic as Google's preload list recommends 1-year (31536000), whereas OWASP recommends 2-years. There were varied opinions when I asked about it earlier in the year on the Cloudflare forums. I currently have it set to 2-years, so it can be added to Google's Preload list and conforms with OWASP's recommendation (links in the sources below). Strict-Transport-Security: max-age=63072000; includeSubDomains; preload

Recommended value sources

Specification

HTTP Strict Transport Security (HSTS).

X-Content-Type-Options

Description

The X-Content-Type-Options header is a security feature that prevents browsers from interpreting files as a different MIME type than what is specified by the server. By setting it to nosniff, it ensures that browsers strictly adhere to the declared Content-Type, reducing the risk of MIME type confusion attacks, such as executing malicious scripts disguised as other file types. This header is particularly effective in preventing cross-site scripting (XSS) and drive-by download attacks.

Example

X-Content-Type-Options: none

Recommended value

There's only a single directive with this header, but I've linked to additional resources below for further reference. X-Content-Type-Options: nosniff

Recommended value sources

Specification

Fetch Living Standard.

X-DNS-Prefetch-Control

Description

The X-DNS-Prefetch-Control header controls whether browsers should perform DNS prefetching, a performance feature that resolves domain names of links on a page before they are clicked. By setting it to on, DNS prefetching is enabled, potentially improving page load times. Conversely, setting it to off disables this behaviour, which can enhance privacy and security by preventing unnecessary DNS queries that could expose browsing behaviour or enable certain tracking methods. This header is useful for balancing performance and privacy needs.

Important: This header is currently a non-standard header, so check browser support before using.

Example

X-DNS-Prefetch-Control: on

Recommended value

X-DNS-Prefetch-Control: off

Recommended value sources

Specification

Not part of any current specification at the time of writing.

X-Frame-Options

Description

The X-Frame-Options HTTP header prevents clickjacking attacks by controlling whether a web page can be embedded in a frame or iframe. It supports three values: DENY (disallows framing completely), SAMEORIGIN (allows framing only on the same origin), and ALLOW-FROM URL (deprecated, allows framing from a specific origin). This header is being replaced by the more flexible frame-ancestors directive in Content Security Policy (CSP).

Important: This header is gradually being replaced with the (CSP) frame-ancestors directive, but implementing both while this crossover happens should cause no negative impact to your website.

Example

The ALLOW-FROM option is not widely supported and is largely replaced by the frame-ancestors directive in CSP. X-Frame-Options: ALLOW-FROM https://example.com

Recommended value

X-Frame-Options: DENY

Recommended value sources

Specification

HTML Living Standard.

X-Permitted-Cross-Domain-Policies

Description

The X-Permitted-Cross-Domain-Policies header specifies whether a browser or client can load cross-domain content, such as Flash or PDF files, from your domain. By setting it to values like none, master-only, or by-content-type, it controls the loading of potentially unsafe resources, mitigating risks such as data theft or unauthorized content execution. This header is particularly relevant for applications using plugins like Adobe Flash, providing a safeguard against cross-origin attacks and unauthorized data access.

Important: This header is currently a non-standard header, so check browser support before using.

Example

X-Permitted-Cross-Domain-Policies: by-content-type

Recommended value

X-Permitted-Cross-Domain-Policies: none

Recommended value sources

Specification

Adobe Cross Domain Policy File Specification.

Origin-Agent-Cluster

Description

The Origin-Agent-Cluster header controls whether a webpage should have its own separate "agent cluster", isolating it from other origins in terms of memory and execution contexts. By setting it to ?1, the header ensures that the page's scripts, documents, and workers operate in isolation, preventing data leakage and improving security by mitigating cross-origin side-channel attacks like Spectre. This isolation enhances security for web applications, especially in environments with sensitive data or shared hosting scenarios.

Important: This header is currently a non-standard header, so check browser support before using.

Example

Origin-Agent-Cluster: ?1

Recommended value

The value of ?1 is the structured header syntax for a boolean true value. Origin-Agent-Cluster: ?1

Recommended value sources

Specification

HTML Living Standard.

Final results

So let's have a quick look at the final results of improving the HTTP Security headers on this blog by running it through the security scanner Security Headers by Snyk.

Before

The score before came out at a pretty poor "D":

After

The score after implementing the above changes comes out at a much better "A+"!:

In summary

In this post, we've looked at what a response header is, why security response headers are important (even for static websites!), what the different implementations look like on different static hosting platforms, and also details on each of the response headers currently in use on this blog. Thanks for reading! I hope you found it interesting! And as always, if you spot any errors or have any feedback, please do let me know!

Post changelog:

28/12/24: Initial post published.
29/12/24: Added the HTML5 Boilerplate Apache Permissions Policy example (Thanks to Don Marti).

Migrating to 11ty

2024-12-22T00:00:00Z

So, the first thing to mention about this post is: It's been years in the making! As I mentioned in my previous blog post, I've hated running my site on GitHub Pages and Jekyll for years now! So when I first heard that the wonderful Zach Leatherman had built an awesome new blogging framework that was built on Node.js back in December 2017, I thought, you know I need to migrate over to that ASAP! Then you find ASAP becomes weeks, weeks become months, and before you know it, the whole world goes through a global pandemic, and you get diagnosed with grade 3 brain cancer… Or maybe that's just me! It's just over 7-years since 11ty's first release, and I've finally got around to updating this blog! Finally moving away from GitHub Pages and Ruby, and embracing the future. I've decided to go all in with Node.js, Nunjucks, and Cloudflare. And as a bonus, this includes all the lovely web performance goodies that Cloudflare and Cloudflare Pages provide.

The Catalyst

So what was the catalyst, you may ask? Well, it all starts with a man in a red hat and who has a deep, booming voice. He is very well-known in the web community as he runs 2 fantastic conferences, State of the Browser (SotB), and also London Web Standards, among other things! Of course, I'm talking about the amazing Dave Letorey! Let me tell you a short story about how I first met Dave. I was in Amsterdam to speak at Performance Now 2023. Now, I'd love to give the somewhat predictable story of how he walked up to me after my talk and asked me to speak at State of the browser 2024, but as I've since learned, Dave likes to party! So I was sitting in front of Dave on the first day of Performance Now when he introduced himself. He said he was eagerly awaiting my talk tomorrow, and he's looking for speakers for SotB 2024, and would I be interested? I immediately said yes, thanked him, and started to watch the rest of the speakers from day one, while getting increasingly nervous all day, as the quality of the speakers at Performance Now are always incredible!

I was scheduled to be the first talk on the second day. After the quality of the speakers on day 1, I knew I was going to have a tough time keeping folks interested and engaged with my talk, especially after they've only just woken up and had their breakfast!

The next morning came and as I nervously scanned the audience, but I couldn't see Dave in the audience anywhere. I just assumed he was just a late riser, and he had decided to miss the morning talks. But as I mentioned, Dave likes to party, and it turns out, Dave had got very drunk the night before and ended up god knows where at 4am in Amsterdam! He admitted this to me later in the day once he'd arrived. He was very apologetic for missing my talk, as it had been one he'd been excited to hear! Thankfully (for Dave), the talks are recorded, so he'd get to watch it at a later date!

Fast-forward a couple of months, I was just about to start a new job, and just my luck, my health took another turn for the worse, so I reluctantly had to pull out of speaking at State of the Browser 2024. It turns out Dave is an incredibly kind and giving person, as he contacted me closer to the conference date and asked if I'd like to attend the conference as his guest. I agreed immediately, as it'd been several years since my last commute into London (thanks to COVID-19 and cancer!). So that's what I did, and boy I'm glad I took him up on the offer!

State of the Browser 2024

SotB always has some incredible speakers, you only need to look at their previous speaker list to see that! I enjoyed all the outstanding talks that day, but the one that resonated with me the most was Richard Rutter's talk on "Fluid typography (and its role in design systems)". Now, I've been a big fan of typography for many years, but I've never been particularly good at design. Even so, I like to think I can recognise good design when I see it. His talk goes over various typography topics like:

It was when Richard started to discuss the tool that his company Clearleft had built called utopia.fyi, it really got me thinking and genuinely excited.

If you get the chance I'd recommend you watch Richards talk, But In short, Utopia.fyi is a conceptual approach to fluid responsive design that encourages designers and developers to create systems where elements scale proportionally and fluidly across different screen sizes, without relying on arbitrary breakpoints (media queries).

This methodology promotes minimalistic and elegant design, enhances collaboration between design and development teams, and ensures visual harmony and consistency. To support this approach, Utopia offers free tools such as type, space, and grid calculators.

It was after he demoed the tools in action that I realised it was time to throw away my old website design and be brave and attempt to create something a lot more modern and simplistic. And with the help of utopia.fyi, I may actually be able to "design" it myself!

Throwing everything away

Once I'd decided that I wanted to throw away the old design, it actually made moving forwards a lot easier. My problem with simply migrating an existing design to any new framework is that I'd most likely migrate my "bad habits" and "technical debt" along with it. It was admitting this technical debt existed that gave me the freedom and the perfect opportunity to start from scratch and focus on building something I'd be happy to maintain into the future.

This was something I never had with Jekyll. I used to dread writing and trying to publish a new blog post after not doing it for a while. There was always the feeling that GitHub may have changed "something", and I was just about to run full-force into dependency hell just to get it published once written! It's safe to say that without Richard's inspirational talk, I'd still have "Migrate nooshu.com to 11ty" very much welded to my to-do list!

Learning 11ty

11ty 2.x.x

I swear I must be one of the unluckiest people I know when it comes to timings. 11ty version 2.0.x was released in February 2023, and the beta of version 3.0.0 had been in both alpha and beta release phases since June 2024. And of course, I picked the exact week 3.0.0 final was released to start my migration over to 11ty! At the start of the week, I began building from eleventy-base-blog v8, only to realise the scale of the differences between v2 and v3 were considerable for a beginner in 11ty like me. So I ended up scrapping my v2.0.0 version in favour of starting again using the latest eleventy-base-blog v9.

11ty 3.0.0

Having never used 11ty before, I wanted to stick as close to current "best practice" in the 11ty ecosystem as I could, so figured using the "eleventy-base-blog" would be the best place to start since it's maintained by 11ty creator Zach Leatherman. You can't really get much closer to the original source than that! Luckily, most of my v2 work like the CSS and partials could be easily migrated across to v3 without any issues. It was mainly the core of 11ty that had changed. Especially, it's adoption of ECMAScript Modules (ESM) for its configuration.

Minimal Viable Website

One of the problems with starting to migrate a live website from one technology to another is: at what point do you decide that the new site is "ready" to be published? It sounds simple, but it really isn't. It's far too easy to get greedy with functionality.

I found myself looking at other people's websites and saying: "Ooh! That's cool! I need to make sure I build that functionality before I put the site live!". And that's the slippery slope to a site that will never be published because it's never perfect. So I came up with a list for a Minimum Viable Website (MVW), off the back of a conversation with Sia Karamalegos, who'd I'd had the absolute pleasure of meeting at Performance Now too.

My MVW list looked like this:

Built using 11ty v3.0.0 (duh!).
New website design using Utopia.fyi for both typography and spacing.
Hosted on Cloudflare Pages.
Optimised for accessibility and web performance.
Content migrated to only use standard Markdown for maximum portability in the future.
A simple contact form (ideally using Cloudflare Workers).
A blogroll page for the syndication of amazing content from all the wonderful content creator folks I follow.
An RSS feed in multiple formats (e.g. Atom, RSS, and JSON).
Static build automation using GitHub Actions to rebuild the blog every 12 or so hours.
Implement Webmentions for indieweb syndication.
JSON+LD for SEO and machine-readable content.
Microformats using h-card.
Giscus comments to allow readers to comment on posts should they wish to.

As you can see, the list is already getting quite long! But just to make it a little more interesting, I decided to challenge myself to:

Use a single flexible, responsive viewport that works on all devices (i.e. no media queries).
Only use plain old CSS, no Sass, Less or Stylus.
Leverage modern CSS variables wherever possible.
Utilise modern layouts like CSS Grid and Flexbox.
Minimal JavaScript, no need for any JavaScript frameworks here
Hand-coded HTML, CSS, and JavaScript all the way!
Light and dark mode functionality.
SVG's for icons and the main logo.
Lazy load as much as possible.
Use a Progressive Enhancement methodology.

Now, many readers may be wondering why I've added some of these requirements into the build, isn't it just making life harder for myself? In all honesty, it's primarily for my interest and curiosity. The web is constantly evolving and moving forwards, and there's no better place to learn something new than your own personal website! Take CSS Variables for example, I've known about them for quite a while as I'd used to use their predecessor in Sass which had a similar variable implementation, but I'd never used native, web platform "vanilla" CSS Variables. And after using them I'm so glad I did, as it's going to make maintenance in the future much simpler!

Migration challenges

Here I'll go over a few of the pain points I found during the migration:

Cleaning up my content — Part 1

I've written quite a few blog posts since 2009, when I started blogging, and in that time I've migrated the content across multiple CMS's and platforms. I actually wrote all about it in a previous blog post. It probably took me around 1.5 weeks to complete the content migration task, doing the odd hour or so in the evening. What made matters worse is that during the migration from WordPress to Jekyll and GitHub pages, literally all the HTML character encoding across all blog posts broke! Thankfully, it wasn't noticeable at all to readers, but it's annoyed the hell out of me for years as it made my text editor light up like a Christmas tree. It also made spotting actual spelling errors much more difficult. This task in itself was very much a search and replace job, so it took a while!

Cleaning up my content — Part 2

Over the lifetime of my Jekyll build, I'd implemented some unusual non-standard ways to output elements like images. Basically, a simple abstraction layer include, that I added to make my life easier (in theory). As It turns out, it really doesn't when migrating to another blogging platform!

All of my images in every blog post looked like this:

{ % include image.html src="year/month/image-name.gif" alt="Alt Text here" caption="An example Caption here" % }

But in 11ty it all had to be changed. For 11ty, I've decided to use the more standard Markdown way of adding images. So the above code needed to be transformed into this:

![Alt Text here.](./year/month/image-name.gif)

The new markup is certainly a lot cleaner and easier to remember, but the only downside I've found is there's no support for image captions (<figure> & <figcaption>). FYI: This is still an issue I've yet to solve, so if anyone has added captions to their 11ty images output using Markdown, please do let me know!

Thankfully, there's a whole range of new tools on the web currently that can ease the pain of a migration process like this. It's a buzzword that is being crowbarred into pretty much every product and company presentation in 2024: "AI"!

I've used both the free and paid for versions of ChatGPT and Google Gemini, and both of these basic models are perfect for this type of repetitive, mundane work.

My methodology, was to get the basic prompt setup at first, e.g. "I want you to convert the njk image template into the md template", then see how well it did on its own. If there were issues, prompt it again to address the issues. Then keep repeating this process. And finally, once the output was what I needed it to be, I'd give it a prompt like: "That output format is perfect, do the same for each input I give you from now on".

From then on, simply paste in the old format and let AI generate the new format for you! Even that small workflow change literally saved me hours of manual copy / pasting into the new image template format.

I did wonder if I could prompt it to do this for a whole Markdown file from a post, but the prompt became so long and unwieldy, that I eventually gave up on that idea. But I'll take what I can from AI, every minute saved with the content migration task means more time to spend on more interesting tasks.

Observations

Here are a few observations I've made during my migration to 11ty:

Flexibility

11ty is unbelievably flexible in terms of how you set it up and the templating languages it can accept. I have to admit, in some ways it's almost too flexible for a beginner. A lot of the time I found myself asking: "Okay, I can do it that way… but is that the "correct" way to do it?". To give you a prime example of where I struggled, when I looked at the template languages documentation on this page, as a beginner I was overwhelmed by the number of options available to me.

I ended up using YAML frontmatter, Markdown for content, and Nunjucks for templating. A fairly standard setup coming from "Jekyll land". But I'm sure there are hundreds if not thousands of ways to achieve the same thing with all the templates 11ty supports. And as a beginner, it's challenging to know what the advantages and disadvantages are when choosing a particular templating language path. It's almost like a single paragraph is required on complex pages like this saying: "If you are a 11ty beginner, we'd recommend using a, b, and c because of reasons x, y, and z". Then, once a beginner is on a "safe" path to success, they can build up their confidence and start to evaluate other options and iterate accordingly.

Just so you know, this is in no way a dig at 11ty, I'm just feeding this back to (hopefully) help others who also find too many options a little overwhelming at first!

Unforgiving

I've found 11ty an unforgiving beast when it comes to error handling. If there's something wrong in the code, it will let you know about it! Even down to missing images that are referenced in blog posts! I believe that this is actually a good thing, and I want to applaud Zach for building it in this way. If I make such an obvious error like omitting an image file referenced in a blog post, I would like to be informed about it so I can fix it!

Although, at times, I did feel like Samuel L. Jackson's character in the original Jurassic Park (1993) when he repeatedly enters the wrong password into Nedry's computer! But all in all, it's a good to catch these really obvious mistakes early.

If there's one feature I'd like, it's to have a little "offline prompt" that is visible in the browser once the 11ty server goes offline (e.g. errors). This is just to make it blatantly obvious that something has gone wrong in the terminal. But it really is a minor issue.

Community

I have to commend Zach for successfully building such a strong and diverse community around 11ty! It really is a "poster child" of how an open-source project should be managed and, more importantly, how it should be funded!

There are so many resources available for beginners, like:

Very comprehensive documentation.
Eleventy Discussions on GitHub.
A dedicated Discord server.
A dedicated YouTube Channel.
Multiple YouTube playlists from the community 1, 2, 3.
Andy Bell's Learn 11ty course, now open-sourced and maintained by uncenter.
Numerous plugins, both official and community contributed.
Getting started documentation.
Why you should use Eleventy.
11ty Rocks! By Stephanie Eckles.
11tybundle.dev by Bob Monsour.
Copious tutorials to keep you busy on the official site.

Honestly, I could go on, but I think you get my point!

Future Plans

So, what plans do I have for this blog in the future? I still have a list of functionality that I'd like to roll out at some point in the future, including:

Adding a plain text version of the RSS feed.
Make the source code to this blog available on GitHub (thanks for the inspiration, Sia Karamalegos).
Re-examining the use of pagination for the blog post homepage.
Built in search functionality (inspired by Tim Kadlec and WPOStats).
Add additional themes just because I can! (inspired by 11tytheme.sjoy.lol).
And finally, I have a list of around 5 or 6 smaller blog posts I want to write.

So, yes! Plenty still to be getting on with!

Thanks

Well, another blog post that turned out to be a lot longer than I first planned! Congratulations, you made it to the end! I hope you found it useful, as always, any feedback, please do let me know!

Finally, a heartfelt thank you to everyone who patiently answered my many 'n00b' questions during this migration process, including:

Aankhen.
Bob Monsour.
Bryce Wray.
Cassey Lottman.
Dave Letorey for the SotB invite.
Nicolas Hoizey.
Richard Rutter for inspiring me to rebuild, this site.
Sia Karamalegos.
Terence Eden.
Tobias Fedder.
Uncenter.

Post changelog:

22/12/24: Initial post published.

The History of Nooshu.com

2024-11-02T00:00:00Z

Context

To be clear, most (if not everyone but me) may find this post completely pointless and uninteresting. As I'm trying to get back into blogging a lot more, and also consciously understand how this blog evolved. As it certainly wasn't planned at all, as I describe my "decision-making" process (or lack of, to be honest). You will see that a lot of the time it was just me wanting to learn something new and experiment with the latest shiny new technology in the Frontend world. If you think you'd find that interesting, then please do read on! I'm hoping that future blog posts will be more useful to readers as they will likely be more technical in nature, rather than the ramblings about the past 16 years!

The beginning

This website first popped into existence on the internet on the 30th March 2009 as a WordPress 2.7.1 website. I wish I could tell you where the "Nooshu" name actually came from, but it's been years, so I have no idea! I think I may have used a random domain name generator, and it was one of the options that it suggested. The actual domain name was registered on Saturday 17th of January 2009, that's almost 16 years ago! God, I'm old!

The web in 2009

To put those 16-years into perspective in Web terms, let's examine what the web was like in 2009:

IE6 was still the most popular browser on the internet. Although IE7 was becoming more popular since it supported tabs, RSS feeds and better support for the ACID2 test.
Firefox was only on version 3.0. Firebug was king, (the precursor to all browser DevTools).
Chrome was on Version 3.0 and only available on Windows. Chrome is considered the first silently auto-updating browser with stable, beta, and dev "channels", once installed, the user doesn't have to prompt it to update in the future. Genius! Thus setting the course for the future of browser updates (apart from Safari). Nothing has changed there then, thanks Apple!
Mobile started to really take hold, Apple launched the App Store in 2008, and the first Android phones were released.
Most people were still on dial-up internet connections. Imagine having to wait minutes for a single YouTube video to load!
Twitter started to become popular, as celebrities joined the platform and the "trending topics" functionality was added.
Facebook overtook MySpace in global popularity.
jQuery was the saviour for every Frontend developer on the web, abstracting away most of the pain of cross-browser CSS & JavaScript issues! Before it, you had frameworks like Moo.fx, base2, Prototype and Script.aculo.us.
Responsive Web Design didn't exist. It wasn't until May 2010 that Ethan Marcotte published his truly monumental Responsive Web Design article on A List Apart. That article changed the web platform forever!
Frontend frameworks like Bootstrap and Zurb Foundation were yet to be released publicly, that only happened in 2011.
Everything was "Web 2.0". This term referred to the increasing interactivity and user-generated content on the web, with social media and blogs playing a central role.

I could go on, but I think you get the point I'm trying to make, a lot has changed in 16-years! So what about this website, how has it changed? Here's a brief timeline of key events in its history.

2010: WordPress + DreamHost

The last point in the list above, "Web 2.0" was most likely the catalyst for me buying the domain and setting up my own self-hosted WordPress blog on DreamHost.

In fact, looking back through my post history, the domain was purchased in 2009, but I didn't post anything at all in the first year. It wasn't until January 2010 that I posted my first blog post. And what a post it was, huh! You certainly set the web on fire with that one past-Matt, well done me!

2010 was an important time for HTML, as the HTML5 Specification was becoming a lot more mainstream. With the expiration of the XHTML 2.0 Working Group's charter, it was clear that HTML5 was the future of HTML.

And It certainly promised a lot, but the key features were: Cleaner code for developers with new semantic elements (<header>, <footer>, <nav>, <article> etc.)

Better cross-browser compatibility.
Improved performance
Easier development
Enhanced Search Engine Optimisation (SEO)
Of course, with any major addition to the web platform, older browsers that don't support new features need consideration too!

The primary rule on the web platform above all else is, remember:

don't break the web!

There are an estimated 50 billion pages on the web, most of which are very likely never going to be touched or updated ever again! So whatever you do, make sure they work with both past and future browsers.

As a side note: In my humble opinion, the web browser is up there as one of the greatest pieces of software ever created. Just think of all the errors and edge-cases it has to deal with, and it still manages to render readable pages, literally decades after they were first written and published! And in terms of human years of development effort, cooperation across continents, countries, and organisations, I'd be interested to see if anything else comes close (apart from the invention of the World Wide Web itself!).

Now, since HTML5 was a big change that introduced new elements to the platform, older browsers wouldn't recognise them. So that's when Remy Sharp created the initial JavaScript solution to this problem. He used the document.createElement(elementName) to let the CSS engine know that an element with that name exists. This advancement has a rich history that I won't go into here, but if you are interested, Paul Irish wrote an excellent blog post back in 2011 all about it!

Long story short, I also jumped on the HTML5 bandwagon and implemented the new semantic elements into my WordPress theme and also included the incredibly short DOCTYPE that I could finally remember! No more copy / pasting from a previous page templates, just look how short and simple it is!

<!DOCTYPE html>.

2013: Zurb Foundation + Responsive Design

Now, I fully admit I've always been terrible at design! I like to think that I can recognise good design when I see it. But as for creating a design from scratch, not a chance! I clearly remember looking at the CSS Zen Garden by Dave Shea in awe as to what people could create using an identical HTML structure and a single modified CSS file! Looking through the designs today, it still blows my mind at the creativity and imagination of others. There really were some groundbreaking designs in there.

And remember, CSS 2.0 was still new and far from being feature-complete. Furthermore, due to the first browser wars, browser vendors were racing to implement new features without any agreement on cross-browser specifications!

So when frameworks came out that promised to abstract away the actual design process for me, I jumped at the chance to use one! Simply modify your template to match the HTML specified in the framework, include the CSS and JavaScript, and you are done! A fully responsive website that was well tested and didn't look like the poster child for GeoCities. I'm not sure why I chose Zurb Foundation over Bootstrap. I think it was because Zurb was less popular and considered the "underdog" against the vast resources of Twitter (makers of Bootstrap).

2016: PWAs & AMP

It turned out that 2016 would be the year to experiment with new technologies. After all, that's what a personal website is for, isn't it??

Service Workers

In July, I decided to experiment with the shiny new Service Worker API. This allowed me to turn my humble little blog into a Progressive Web Application (PWA), complete with features like: Push Notifications. Native device access (Camera, GPS, etc). Offline Access. Installable to a user's device "home screen". Greater reliability over unreliable network connections. Improved performance via fine control over asset caching. Essentially, PWAs provide a native app-like experience with features like home screen installation and access to more powerful device features.

In hindsight, the functionality is all pretty overkill for a small blog with barely any readers in my own little corner of the internet. But any excuse to dabble with new technology, I guess!

Accelerated Mobile Pages (AMP)

Another technology I implemented was Accelerated Mobile Pages, or AMP for short.

For readers who've not heard of them, Accelerated Mobile Pages are stripped-down versions of web pages that load rapidly on mobile devices.

As with the Service worker API, there was no real reason to implement AMP on this site apart from for general experimentation with the technology. In the end, AMP turned into (yet another) walled garden on the internet, this time it helped keep users (and their content) within the Google domain. In fact, the web community were so concerned with AMP that hundreds of developers signed an online petition to persuade Google to make the technology more viable for the whole of the web platform (not just Google!)

2017: Bye, bye WordPress, Hello Jekyll

Having been with DreamHost and WordPress for 7-years, the web platform had gone through a huge amount of change. With the invention of the Jamstack web development architecture, it was now possible to create simple static websites using only a few build tools and host them on free services. Rather than having to use a CMS, a Database, and a Server-side language (like PHP, Ruby, .Net etc). And since I was already paying for GitHub for repository storage, why not use that for hosting my static website too? That ultimately lead me down the Jekyll route, since it's what is used by GitHub to render static HTML via their Pages functionality.

Annoyingly, I didn't write a blog post at the time about this dramatic architecture change to the site, so I'm struggling to recall all the details. But I do remember that, not being a ruby developer, I found it quite a steep learning curve at first. Especially when it came to the asset pipeline. Furthermore, I don't think I ever got used to all the dependency hell you needed to jump through if you wanted to make a small change. When GitHub had updated the version of Jekyll, they used to render the pages e.g. etc.

But once the site was set up and all working, it was a breeze to update and deploy new blog posts! And, bonus! No monthly costs for separate hosting or updating WordPress to the latest version because of zero-day security issues!

The main downside I found with Jekyll and GitHub Pages was the limitation that you could only use "GitHub approved" plugins. If you wanted to use another, less common plugin that GitHub didn't support, you're out of luck! There was probably a solution to that, but I never took the time to investigate further.

All in all, I was pleased with the setup for many years, but the dependency on Ruby in the background never sat well with me. It's an incredibly powerful language, but always seemed very overkill for generating simple static HTML files!

2018: Bye, bye AMP

Considering the uproar about AMP in 2016, I'm not sure why it took me so long for to rip it out of the site!

It was most likely that having 2 young children, a mortgage, and a job that required 4-hours commute each day! Being a fully-fledged adult basically left me with no time to do it! But, thankfully, I did manage to write a post all about my reasons for removing AMP here.

2021: Bye, bye GitHub Pages, Hello Netlify, and Cloudflare

So, 2021 turned out to be an incredibly character building year for me personally and a truly horrible time for my immediate family and friends. As our late Queen Liz said in 1992 & 2019, it was our Annus horribilis (disastrous year).

But if there is a silver lining to being diagnosed with Stage 3 brain cancer, it's that it really motivates you to do things you've been putting off for years! Looking back now, I was using work as a kind of "safety net". Since it was something I had at least a little control over in my life. I'd thrown myself well and truly into my work at Government Digital Service (GDS) and was getting more involved in the wonderful Web Performance community.

In doing so, I decided I needed much more control over the blog than what GitHub Pages could offer. The natural progression at that time was to migrate to Netlify. The company had a free tier that came with all the features I required and also supported Jekyll. So migration would be simple. Once the migration was completed, my "digital legacy" would be running on 100% free services. I even placed the site behind Cloudflare's free tier services, so I could take advantage of all the lovely web performance goodies that they have to offer! Once completed, I could then focus on my immediate future (family, operation, radiotherapy, chemotherapy, recovery etc).

2024: Bye, bye Netlify and Jekyll, Hello 11ty, and Cloudflare Pages

I've now had "stable" MRI scans for the past 3-years, which is remarkable! So it's time to look to the future and do something new with the site. Which brings us up to the present day. And this is the very blog post I've written, and you (presumably) just read, that has been published using 11ty and Cloudflare Pages. And I must admit, it feels fantastic to finally clean up years and years of technical debt and give the site a new lick of paint, I even managed to complete most of the design myself!

But I'll leave the details about the full migration from Netlify and Jekyll, to 11ty and Cloudflare Pages for another blog post.

The future

In summary, a lot can change in 16-years, both on the web and in life. Or if you prefer a cheesy one-liner: "Embrace the past, and use it to shape the future."

On that cheesy note, I'll leave you with one of my favourite quotes from C.S. Lewis:

Isn't it funny how day by day nothing changes, but when you look back everything is different?

As always, thanks for reading. And if you have any comments or feedback, either use the comment section below, or use one of the links on the contact page.

Post changelog:

02/11/24: Initial post published.

GOV.UK Cookie banner and why it won't go away

2023-12-12T00:00:00Z

This is a blog post I've been meaning to write for many years. While working at Government Digital Service (GDS) for 6 years as Head of Frontend, I had so many people ask me about the GOV.UK Cookie banner. The main complaint being "it would never go away". I was asked this question on so many platforms like email, Twitter, and even Reddit! The fact is, it has a surprisingly complex answer!

The aim of this blog post is to answer this question, so I can easily point someone in the direction of this post, should they be interested in the many technical details involved.

First and for-most, what is a cookie banner?

In simple terms a cookie banner is that annoying pop-up that you get on every website on the modern web. The one that asks you to "accept" or "decline" the storage of cookies in your browser.

Cookie banners exist primarily due to the implementation of privacy regulations such as the Privacy and Electronic Communications Regulations (PECR) in the European Union. There are similar regulations in other parts of the world like the California Consumer Privacy Act (CCPA) in the United States.

What are cookies?

Cookies are small text files that store information on a user's computer or device when they visit a website. They are used for various purposes. Some example usages are:

remembering a user's preferences
tracking user behaviour across websites
delivering targeted advertising

Why are they called "cookies"?

The term "cookie" was coined by web-browser programmer Lou Montulli in 1994. Montulli was inspired by the concept of "magic cookies" in early computer networking. These are small pieces of data that are sent between computers to help them communicate more efficiently. The term "magic cookie" was itself inspired by fortune cookies, which are small treats that contain a surprise message.

Montulli decided to use the term "cookie" for this new technology as he thought it would be easier for people to remember than a complex technical term.

Over time, the term "cookie" has become synonymous with any small piece of data that is stored on a user's computer by a website.

In this post, my focus is specifically GOV.UK (www.gov.uk) and related UK Government services e.g.

What you may notice about the GOV.UK homepage (www.gov.uk) and the services listed above, is that they all have the same design.

There are a few minor differences between pages, but the overall aim is to use the same design language across all services. This is why the GOV.UK Design System exists. To help service teams across the UK government achieve the same design quickly and easily.

It's also worth mentioning that the GOV.UK Design System also has a cookie component with lots of guidance for other service teams to use.

This consistency was a conscious decision by the design team back in 2011–2012. Its design is clean and simple and has always had accessibility & inclusivity as top priorities from the very beginning.

The design actually won the Design Museum Design of the Year Award 2013 and the D&AD 2013 Black Pencil award.

But as with every bold decision in life there are people who disagree. Like this comment from Howard Andrews who says:

Where is the government web site for grown-ups? The whole of "gov.uk" is obviously designed for primary school children and those with special educational needs.

I, personally, disagree with Howard, but as it's taxpayer money being spent everyone is entitled to express their opinion.

The point I'm getting at is that consistency across the whole user journey was chosen to simplify the site for users. Do users need to know that they are moving between services run by entirely different departments? Not really! As long as they can complete their task quickly and easily and get on with their lives, I'm sure they'll be happy.

But it's worth noting that this design consistency across services is a double-edged sword:

Pros

Gives users a stable design across the complete user journey thus reducing the cognitive load users experience.
Allows users to recognise a government website instantly.
Reduces development and design time in the public sector.
Reduces costs associated with building government services.
Builds brand loyalty and a site's trustworthiness

Cons

Changing a design across multiple departments requires more coordination.
Technical edge-cases like the cookie banner reappearing across services (a lot more on this later!)
Users may be unaware they have navigated to an entirely different site.
Design stagnation due to a rigid set of rules to follow, the design may eventually look dated.

I strongly believe the benefits of consistent branding far outweigh the drawbacks. Especially when it comes to government services that millions of people use every day.

Of all the "cons" listed above, this is the one that users notice the most!

Users may be unaware they have navigated to a completely different site

If the site design barely changes between user navigations it's no wonder a user still thinks they are on the same site. As mentioned before, this was an intentional decision. It abstracts away the complexity of government departments into a clear and consistent user journey.

There's no single source of truth when it comes to back-end technologies in UK government services. For example, www.gov.uk is built using Ruby on Rails. Whereas Universal Credit uses Java. The DVSA's Check MOT service is built using PHP. Unfortunately, the reason the cookie banner "won't go away" is exactly because the user is on an entirely different site. There are technical reasons for why this happens that I will cover now.

Browser Privacy

We start our technical explanation with how browsers have changed over recent years to protect users' privacy online. Back when web browsers were first developed, the internet was nothing like it is today. User-data capture was primitive and so was internet advertising. Unfortunately, the same can't be said about the "modern web" of 2023. Companies across the globe have realised how valuable user-data is. And a modern-day gold-rush has emerged from the access to billions of peoples private browser data. Companies like Facebook and Google were until very recently able to easily track users across the internet using specific information like:

Websites visited
Search queries
Social media interactions
Location data
Device information

From this data, companies can build an extremely accurate digital profile of users. Especially when these companies analyse this data all together (e.g. for all users). Very recently Google toyed with an idea called Federated Learning of Cohorts (FloC).

FloC was a fairly simple idea. Examine all the captured user data and look for similarities in the data. Then create a bucket (or cohort) for all these similar profiles to live in. This turned out to be a great way to track a large group of people while keeping an individual's identity private. For example, if you recently browsed websites related to motorsport (e.g. Formula 1), then you would be placed into a cohort of other people who are interested in motorsport.

This all sounds very innocent, but it's incredible how accurate these cohorts can become, when there are 10s of thousands of people in them and millions of data points to examine. These cohorts would become more and more accurate over time as they gather more data and refine the algorithms. Eventually, these companies would have such an accurate idea of who you are and what you like. They'd be able to predict what you enjoy and target you with targetted adverts that they would be fairly certain you were likely to engage with. Thankfully, the privacy community pushed back on the idea of FloC and the development was stopped in favour of a new technology called Topics.

The key differences between FloC and Topics are:

A user will be assigned to a maximum of 5 topics from a predefined list of 350.
Cohorts aren't used, instead a user is assigned to specific topics based on their browsing history.
A user can edit the topics they have been assigned to and even turn the API off completely if they don't want targetted adverts.

The Topics API was first enabled in Chrome 115, released on July 17, 2023 as an Origin Trial. On September 20, 2023 the Origin trial ended and Topics is now available for the vast majority of Chrome users.

The current future of interest-based advertising on the web is uncertain. It is possible that these technologies will be replaced by new approaches in the future.

Network isolation and HTTP Cache partitioning

The next topic to discuss is another fairly modern development for browsers. Back in the early days of browsers there was a single browser cache that was used for every site you visited. This left site visitors open to a number of security and privacy attacks such as:

Detect if a user has visited a specific site

An attacker can examine what is in the cache and infer what sites a user has visited from the cache contents. As certain resources may only be available on a specific set of sites.

Cross-site search attack (XS-Search)

An attacker can identify whether a particular search query yielded no results by examining the presence of a "no search results" image stored in the user's browser cache.

Cross-site tracking

The cache can be employed as a repository for persistent identifiers, enabling malicious actors to monitor user behaviour across multiple websites.

Thankfully, browser vendors have added security and privacy features to stop these attacks. The status of each major browser is listed below:

Chrome

Cache partitioning was first introduced in Chrome 86, which was released on October 5, 2020. Additional storage partitioning beyond the HTTP Cache was enabled from Chrome 115 which was released on October 4, 2022. See the Privacy Sandbox article here for more information.

Firefox

Cache partitioning, also known as network partitioning, was enabled by default for all users in Firefox 85, which was released on January 13, 2021. This feature was introduced as part of Firefox's efforts to improve user privacy by making it more difficult for websites to track users across different websites. The browser also supports Enhanced Tracking Protection which automatically protects a user's privacy while they browse.

Edge (v79+)

Edge usually follows Chrome closely, and at the time of writing Chrome and Edge are aligned and stable in terms of Cache partitioning. More information on Microsoft's plans for Edge and the Privacy-Preserving Ads API in 2024 can be found in this blog post published on the 5th March 2024.

Safari

Apple has been way ahead of the curve with these features. The team actually talked about these features way back in February 2013! For more information check out this post "Safari, Caching and Third-Party Resources" by good friend Andy Davies.

iOS Intelligent Tracking Prevention (ITP)

While we are on the subject of Apple and Safari, it's probably a good time to mention their default tracking protection. Apple enabled Intelligent Tracking Prevention (ITP) in Safari with the release of iOS 11 and macOS High Sierra on September 19, 2017.

ITP is used to stop sites from tracking a user across the internet. Safari automatically examines resources downloaded from a users' journey around the internet.

Machine Learning is then used on the device to classify these resources to see if they could be used for tracking purposes. For more privacy, if a user hasn't interacted with a website in the past 30 days the website data and cookies are immediately purged.

ITP 2.1

In February 2019 Apple released ITP version 2.1 for iOS 12.2 and Safari 12.1 on macOS High Sierra and Mojave. This version brought major changes for sites wishing to track users. Third-party (tracking) cookies were blocked completely. Furthermore, any existing cookies that have had no user interaction for 30 days are then purged. More importantly for the GOV.UK cookie banner, any cookie set using the document.cookie API automatically expires after only 7 days!

Implications of ITP 2.1 for GOV.UK

Although most users of iOS 12.2 and Safari 12.1 on macOS High Sierra and Mojave wouldn't have realised ITP 2.1 had been installed and enabled on their devices. This was, in fact, a considerable change for websites that used this data for decision-making purposes.

GDS and the GOV.UK team in particular, were one of those impacted by ITP 2.1. Although I've heard many complaints over the years about this, GOV.UK uses Google Analytics (GA) for its analytics. The data captured is pretty limited. For example, whole sections of GA aren't enabled internally, like "user demographics", "behaviours", and "interests" weren't available for performance analysts to see. That's not to say Google can't see that data themselves!

But this is the hand we were dealt and a legacy of the age of GOV.UK. Back in 2012 when GOV.UK was first launched an Analytics product was needed to evaluate and prove that users were using GOV.UK and which parts were being used. User privacy wasn't as important as it is today, and GA was likely the most feature rich (and easiest) to implement. In hindsight a self-hosted analytics solution should have been used like Piwik (now Motomo). Although I can't be sure if the versions at the time had all the features required by the GDS teams at the time (2007). See the additional context from Tom Loosemore below for more information on the early decisions made by GDS about tracking.

With the release and rollout of ITP 2.1 on iOS and OSX this meant the annoying cookie banner at the top of GOV.UK would appear every 7 days. This is because the cookie used to store the information that the user had already seen it would be purged every week. So a user browsing GOV.UK first thing Monday morning on an iOS device, accepts the banner then exactly 1 week later they will need to do the same. This would be pretty frustrating for the user, but unfortunately, it's completely outside any website's control. So if you're an iOS user and have ever wondered why you keep seeing the same cookie banner over and over again, this is the reason!

A workaround to the 7 day problem

Thanks to Barry Pollard for this info on a workaround. I quote:

When you drop the cookie using client side JS, you make a JS API call a back-end which reads the cookie, and then sends it back as an HTTP set-cookie header. Boom - it's no longer a JS cookie.

This is allowed AFAIK because Apple wants to block JS that isn't on the same domain as the document (i.e. most likely tracking cookies). Their intention was never to block pure 1st party cookies as I understand it.

iOS Mobile Bias

According to the monthly GOV.UK statistics I posted back in February 2023, Safari made up 31.48% of traffic in the month. And 56.21% of users were on a mobile, with the Apple iPhone being the most popular device by an absolutely huge margin of 46.98%!

Unfortunately, with the ITP change in version 2.1 there's no way of knowing how accurate these figures are. This is why I always posted a notice in the tweet thread saying:

‡ Note: since December 2019 GOV.UK requires explicit opt-in for tracking which introduces bias. This is especially true for mobile devices, since the cookie banner takes up more screen estate and is more likely to be accepted.

Which brings us back to the cookie banner design. This is what GOV.UK homepage looks like on Desktop:

This is what it looks like on Mobile:

As you can see, the cookie banner is huge on any mobile device taking up around 65% of the screen. It's also worth noting that a user is only seen by GA if they accept the cookie banner. As the cookie banner on a mobile is huge, a mobile user is more likely to "Accept" it so they can actually view the page content! This where the mobile bias comes into play with the GOV.UK analytics data.

Note: This only became an issue when the Information Commissioner's Office (ICO) ruled that tracking all users by default wasn't adhering to their explicit consent policy.

Additional Context

This is one of the reasons why I love blogging so much, it's astonishing how much you can learn from the feedback other people give you.

And it's also a great example of why you should keep some form of decision record in an organisation for when people leave and take all that precious knowledge with them! A huge thank you to Tom Loosemore for this additional context. Tom was Deputy Director, Government Digital Service, Cabinet Office, between January 2011 and October 2015.

I've directly quoted his comment he left me on LinkedIn related to this post:

Behind the scenes GDS tried very hard circa 2012 to convince the ICO that forcing cookie opt-in pop-ups would be counter productive to their laudable aims. Had they done proper user research as part of their response to PECR legislation we might not be in this mess. Policy is delivery, redux.
The GOV.UK team blogged several times about balancing the need for user analytics with privacy, with a nicely intense debate in the comments. Sadly piwik was totally unsuitable as an alternative to GA. Enough well-meaning organisations had adopted it and then regretted it for us to know it was a non-starter. See this blog post
Google would break its contract with HMG if it looked at or used the GA analytics data collected across *.gov.uk for anything other than protecting the GA service itself from attack. The options to stop Google employees (even GA support staff) accessing the *.gov.uk analytics data still seem strong enough to me. See this blog post.

The Public Suffix List

When it comes to setting cookies across domains gov.uk and service.gov.uk, there's an added complication. Both of these domains are on the Public Suffix List (PSL).

In simple terms, the PSL is a list of domain names that are considered to be effective top-level domains (eTLD). This has ramifications for setting cookies, that I will go into next.

If we wanted to stop the cookie banner appearing multiple times across a GOV.UK users journey, a cookie needs to be set to store the information that the banner has been seen and accepted. If we wanted to achieve this only using cookies, the PSL stops us from being able to do this.

As mentioned above gov.uk and service.gov.uk are in the PSL. So they are known as an effective top-level domain (eTLD). Browsers handle eTLD's differently to other domains. It so happens that Network isolation and HTTP Cache partitioning are partitioned using the following URL structure scheme://eTLD+1. There's an example below that, I hope, will explain it clearer:

For a GOV.UK user browsing from the homepage to the Universal Credit Sign In page the browser sees the following:

// ‘gov.uk'
eTLD = gov.uk
+1 = www
eTLD + 1 = https://www.gov.uk

// any other government service
eTLD = service.gov.uk
+1 = www.universal-credit
eTLD + 1 = https://www.universal-credit.service.gov.uk
OR
eTLD + 1 = https://www.check-mot.service.gov.uk/
OR
eTLD + 1 = https://www.payments.service.gov.uk/
OR
// I think you get the idea

The eTLD + 1's are the keys used for cache and network partitioning. Due to this partitioning, any cookie set on www.gov.uk can't be seen or shared on www.universal-credit.service.gov.uk or any other service domain. Thus, a user that accepted the cookie banner on the homepage will see it again in their user journey because this cookie can't persist through to the service domain.

This is the reason the cookie banner "won't go away". Each service on a user's journey, sees them as a brand-new user that needs to accept that service's own cookie banner.

Unfortunately, as we mentioned earlier service domains follow the same GOV.UK branding, so many users won't even realise they are browsing an entirely different website.

Another browser technology that looked promising and was considered to solve the shared cookie problem is called Related Website Sets (formerly known as: First-Party Sets).

First-Party Sets (now renamed Related Website Sets) is a Privacy Sandbox proposal that allows a company to declare relationships between websites they own. This enables browsers to allow limited cross-site tracking between these related sites for specific purposes, even after third-party cookies are phased out. This aims to minimise disruption to certain website functionalities while still prioritising user privacy.

At the time it was deemed unsuitable for *.gov.uk and *.service.gov.uk purposes because of the following limitations:

There was a limitation of ~150 sites included in the "set". This limit is way to small for the GOV.UK estate of sites and services.
For security and verification purposes each site in the set was required to have the same SSL certificate. This would simply be impossible to roll out and co-ordinate across all sites and services that encapsulate GOV.UK, hundreds of departments and teams would be involved in this process alone!

Because of these 2 reasons the use of First-Party Sets was quickly dismissed as a viable option to solve a the shared cookie banner across all GOV.UK sites and services. These limitations may now have completely changed as the specification has evolved. There's a great explanation video on Related Website Sets (RWS) on the Privacy Sandbox website here.

Final Thoughts

So there we have it. A whole brain dump on the GOV.UK cookie banner, GOV.UK service design and surrounding browser technologies related to privacy and security. Hopefully, you found it interesting. Maybe it answered a few questions you didn't even want to know 😂 As always if you have any questions or feedback, please do get in touch.

Post change-log:

12/12/23: Initial blog post published. Happy Birthday to me!
12/12/23: Thanks (again) to Barry Pollard for the numerous fixes to the Cross Site Tracking section of the post and a workaround for the 7 day time limit on Safari!
13/12/23: A huge thanks to Tom Loosemore for his incredibly informative LinkedIn post on the ICO and GA in the early years of GOV.UK. His update can be seen here.
13/12/23: Thanks to Joe Lanman for pointing out it's Privacy and Electronic Communications Regulations (PECR) not General Data Protection Regulation (GDPR) that is important for cookie banners.
13/12/23: Thanks to Alex Russell, for getting back to me about the status of Cache partitioning in Edge.
05/03/24: Thanks to Rowan Merewood, for the new information on what the Edge team has planned for 2024 in their latest blog post.
02/07/24: Added information about Related Website Sets (formerly known as: First-Party Sets).

Remove Subresource Integrity (SRI) from the GOV.UK assets domain

2023-10-23T00:00:00Z

Note: This is an RFC I wrote to persuade the GOV.UK Senior Technology team to remove Subresource Integrity (SRI) from the assets served from the assets domain of GOV.UK, thus enabling better performance for the HTTP/2 protocol. It was originally posted on the GOV.UK RFCs GitHub repo. It can be viewed here. I'm posting it to my blog so as to own my own work and hopefully give it more visibility. I wrote a blog post about the changes but the content was deemed too technical for the "Technology in Government blog", so I pushed for it to be published as a case study instead, which can be seen here. I also wrote about this issue on my personal blog the post is called HTTP/2 and Subresource Integrity don't always get on.

Summary

Due to the security requirements of SRI, crossorigin="anonymous" is required for these assets. This forces browsers to open a separate TCP connection to download this resource. Since TCP connections are expensive to open and maintain, this practice has a negative effect on frontend performance.

Problem

Running Web Page Test (WPT) against a number of pages across GOV.UK reveals that the browser is having to open a greater number of TCP connections due to the use or SRI, and more specifically the crossorigin="anonymous" attribute attached to the CSS and JavaScript references. This attribute is mandatory for security against cross origin policy violations. The use of the crossorigin="anonymous" forces the browser to open a brand new TCP connection to ensure that no user credentials are shared across the connection. This can be seen in the images below:

Homepage

In this example 11 TCP connections are being opened when the browser default for HTTP/1.1 in many browsers is 6. Since opening TCP connections is expensive, this will be having a negative effect on performance.

Full test can no longer be seen due to WebPageTest being sold to CatchPoint.

Past Prime Ministers page

The same issue can be seen on the Past Prime ministers page, where 13 TCP connections are opened instead of the default 6 a browser usually does. As can be seen in the bandwidth graph the connection isn't being fully utilised, this is probably due to the TCP slow start is an algorithm that each TCP connection uses.

Full test can no longer be seen due to WebPageTest being sold to CatchPoint.

HTML size

A very minor point, but adding the crossorigin="anonymous" integrity="sha256-wLi6ixZSqsrSmNdPJHUiYBh/U4tQxAwkhPfzM8vDzys=" to the 12 resource requests across a page could be adding an extra 1KB to the HTML size (although GZipping will minimise this).

HTTP/2

Will HTTP/2 solve this issue? Since HTTP/2 multiplexes assets over different streams on a single TCP connection. Unfortunately not. Requests without credentials use a separate connection as mentioned in the article here. So even when we eventually enable HTTP/2 we will have the same issue where an anonymous connection will be opened.

Proposal

I'm proposing we disable SRI for all assets from the https://assets.publishing.service.gov.uk domain. Since this is a domain under our control it is highly unlikely that a MITM attack will occur. Assuming an attacker gets that far they would most likely be able bring the whole site down anyway due to the level of access they would need. Removing SRI will then allow us to remove the mandatory crossorigin="anonymous", which will then allow the browser to open and reuse connections as they are intended (and hit the maximum of 6 connections).

For assets that do come from a third-party domain, they should still have SRI enabled for security reasons.

Enabling HTTP/2 on GOV.UK

2023-09-28T00:00:00Z

Note: This is an RFC I wrote to persuade the GOV.UK Senior Technology team to enable HTTP/2 on GOV.UK. It was originally posted on the GOV.UK RFCs GitHub repository. It can be viewed here. I'm posting it to my blog so as to own my own work and hopefully give it more visibility. The Technology in Government blog post I wrote on the work can be seen here.

Summary

Back in November 2018 we trialed the use of HTTP/2 on GOV.UK. According to quite a few sources, enabling HTTP/2 should improve web performance for users by introducing technology like multiplexed streams, HPACK header compression and stream prioritisation. Unfortunately it turned out that from our synthetic web performance testing it actually slowed the site down in many instances.

We tested 5 different page types, on multiple devices and connection speeds and examined the following performance metrics to come up with a result:

First visual change
Visually complete 95%
Last visual change
Speed index
Load time (fully loaded)

And for Lighthouse reports these metrics were examined:

First Contentful Paint
First Meaningful Paint
Speed Index
First CPU Idle
Time to Interactive

The RFC below discusses the problems with our current setup and suggests possible solutions.

Problems

1 - Sub Resource Integrity (SRI)

On GOV.UK we are using Subresource Integrity for all our CSS and JavaScript assets coming from the assets domain. The SRI specification requires that the crossorigin attribute is set to anonymous to be used with SRI resources for security reasons. This is because of data leakage from credentialed TCP connections. This requirement is forcing the browser to open a second TCP connection in 'anonymous mode' so it can download the CSS / JS from the assets domain. In doing so this is adding 100's of milliseconds of delay to the page rendering timeline. This occurs even when using the preconnect resource hint, a browser feature intended to help fix this issue.

This performance issue is occurring in both HTTP/1.1 and HTTP/2 as seen in the WebPageTest connection view waterfalls below:

HTTP/1.1

Here we can see 13 TCP connections being opened (6 'credentialed', 6 'anonymous', 1 third-party to Google Analytics). If we weren't using SRI, we could reduce this requirement down to 8 (6 'credentialed', 1 'anonymous' for fonts, 1 third-party to Google Analytics). This is the first step in improving web performance for GOV.UK users on HTTP/1.1.

HTTP/2

Below you can see the delay in the waterfall while the 2nd anonymous TCP connection is established:

And this is what it could look like if we were to remove SRI:

In the example test above on a Nexus 5 device under 3G connection speeds, we could bring the request of the CSS & JS files forwards by ~750 ms. This should speed up the whole waterfall and turn the results from the summary list above from red to green in favour of HTTP/2.

This is achieved through the use of HTTP/2 connection coalescing, which can be seen in action on GOV.UK from our trial below:

The coalesced connection is under-utilised if 'anonymous mode' is used on our static assets. There's also an impact from the fact that TCP Slow Start is in action on the delayed anonymous connection, meaning assets will download slower than they could via the single coalesced connection. So we are in the following situation: the connection used to download the HTML isn't utilised fully (by this point it will already be up to speed), and the anonymous connection downloading the critical CSS has been delayed, so it isn't up to speed yet. For best performance we should be utilising the connection that has already been established via the HTML download, and use it for other critical page assets (CSS/JS).

2 - Assets served with `Access-Control-Allow-Origin: *`

My initial thinking was that we could quickly switch the crossorigin attribute from anonymous to use-credentials for our static assets (CSS/JS). Unfortunately on examining the Fetch specification there's information in the table (5th row down) that states:

Access-Control-Expose-Headers, Access-Control-Allow-Methods, and Access-Control-Allow-Headers response headers can only use * as value when request's credentials mode is not "include".

Basically, the use of the wildcard (*) isn't allowed on a credentialed connection (use-credentials). And if it is used, the browser will block any requests and raise an error message the that looks like this:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at ‘https://assets.example.com/script.js'. (Reason: Credential is not supported if the CORS header ‘Access-Control-Allow-Origin' is ‘*').

We are currently serving all our assets from the assets domain with the Access-Control-Allow-Origin: * header, which is blocking us from making this change. The reason why we are serving assets like this is because we are using SRI and they need the crossorigin attribute for it to work.

The Heroku applications (e.g. https://government-frontend.herokuapp.com) that are created are a legitimate reason for using this header when crossorigin is present. If we were to remove the Access-Control-Allow-Origin: * header then the JS and fonts (due to their unique CORS requirements) on the Heroku previews would break. According to the Access-Control-Allow-Origin documentation, there are only 3 valid values for this header:

*
<origin>
null

The <origin> value is very important. As this is where we would ideally like to specify: *.gov.uk; *.herokuapp.com. Unfortunately this header only accepts a single origin, and it doesn't recognise wildcard values. So we are in a situation where we need to use * (for SRI) and a specific origin value (e.g https://government-frontend.herokuapp.com for every unique Heroku app URL, including individual PR's) at the same time. There's the possibility that this can be done by adjusting the VCL config on the CDN, but that then opens us out to more potential complications as listed in @kevindew's comment here. So this could complicate both the CDN configuration and local development if not properly investigated.

3 - The assets domain

Domain sharding for static assets is an anti-pattern under HTTP/2, and our current HTTP/1.1 setup isn't optimal for performance either. If our static assets weren't being served including SRI we would be able to remove the crossorigin and integrity attributes from the <script> and <link> tags. This in turn would allow the browser to correctly use HTTP/2 connection coalescing, which would minimise the impact the assets domain is having on HTTP/2 performance.

Some browser implementations of HTTP/2 connection coalescing are notoriously flaky, so this performance benefit will only be available in some browsers. Users on older versions of Safari (before 12), IE, and Edge 18 and below, all don't support coalescing, so won't benefit from this optimisation. But as we've seen above, the domain sharding isn't offering any benefits anyway. So why not remove the need for connection coalescing completely for our static assets and serve them from the origin. All browsers that support HTTP/2 will then receive the same benefits and experience of the browser using a single multiplexed TCP connection. 95.7% of the UK population use a browser that supports HTTP/2, so most of our users will benefit from this change.

Once completed our connection graph will look similar to this:

The bulk of the page assets will be downloaded on the initial connection to the origin (www.gov.uk), with a secondary anonymous connection only being opened for the fonts. That's twelve connections under our current HTTP/1.1 implementation, down to just two for HTTP/2 in many cases. So in order to fix our HTTP/1.1 and HTTP/2 performance issues we should serve static assets (CSS, JavaScript, fonts, images) all from the origin (www.gov.uk).

Summary

In order to enable HTTP/2 on the GOV.UK domain (and for it to perform well) we need to all problems listed above:

SRI requires the crossorigin attribute to be set for it to work.
The use of crossorigin='anonymous' breaks HTTP/1.1 connection reuse and HTTP/2 connection coalescing.
Switching to use crossorigin='use-credentials' isn't possible because we serve our assets with the Access-Control-Allow-Origin: * header so that the assets leveraging SRI can be used on other domains (e.g. Heroku).
Even if we serve the static assets from the origin, we will still need the crossorigin attribute because we will still be loading them with SRI enabled.

Therefore my recommendation is to remove the integrity and crossorigin attributes from our static resources (CSS, JS) and work towards serving these assets from the origin domain so we are no longer reliant on an individual browsers implementation of HTTP/2 connection coalescing. NOTE: it is possible to do these streams of work separately.

We should only serve our fonts with the Access-Control-Allow-Origin: * header (this is a requirement when using web fonts). All other assets can have this header removed (as we will have removed the crossorigin attribute). We only need to be serving the following font files:

EOT - For Internet Explorer only (6-11)
WOFF - For all modern versions of "evergreen" browsers and IE9-11.
WOFF2 - All modern browsers since ~2016. it's usage trumps the use of WOFF.

EOT files are no longer going to be served to Internet Explorer once GOV.UK has migrated fully to the Design System. So EOT can be removed in the future. There are no plans that I am aware of to create another webfont format, so we won't be needing to add any more for the foreseeable future.

SRI Note: In serving the static assets from the origin domain, SRI no longer serves any purpose, as it's a security measure intended to be used on third party resources. As all of our assets will be first party, it really isn't required and serves only to complicate our setup and impact on performance.

Serving static assets from the origin also clears a path for future web performance enhancements like HTTP/3, QUIC (Quick UDP Internet Connection) and 0-RTT. HTTP/3 has no support at the moment as it currently sits behind browser flags. But at some point in the future it will be supported and the asset domain will need to be removed for static assets anyway, to keep up with the latest protocol developments.

NOTE: As mentioned by @david-ncsc in his comment, there's probably a security benefit to keeping the uploaded files on a separate origin as defence against uploaded malicious files. We should therefore only serve the static assets (CSS, JavaScript, fonts) from the origin. All other assets can stay the same.

Proposal

MUST

Only serve our font assets with Access-Control-Allow-Origin: * header (WOFF2, WOFF, EOT). - Medium (work required to modify NGINX config here and here)
Remove the integrity and crossorigin attributes from our static resources (CSS, JS) - Easy (PR's quickly raised for all apps CSS/JS).
Serve static assets from the origin rather than the assets domain - Medium/Hard (spike already investigated by @kevindew and commented on here).

SHOULD

Review the use of Access-Control-Allow-Methods and Access-Control-Allow-Headers headers as from looking at the CORS documentation they aren't actually needed. - Easy (Code resides here).

Consequences

Browsers that support HTTP/2 will be allowed to use it to its maximum potential. Browsers that don't will still use HTTP/1.1, but will now only require 6 TCP connections rather than 12.
Both H1 and H2 users should receive improved performance due to improved TCP connection efficiencies.
Under HTTP/2 we should see only 2 TCP connections on the WebPageTest connection graph, with the vast majority of assets loading via the same connection as the HTML.

How to enable HTTP/2

Once we have made the changes how do we enable HTTP/2, and what changes need to be made to the applications?

I simply contact Fastly support and ask them to enable it on GOV.UK. This will enable it on production, integration, and the assets domains.
No changes to any infrastructure or apps need to be made, as this is a transport layer change on the CDN.

Enabling Brotli compression on GOV.UK

2023-09-28T00:00:00Z

Note: This is an RFC I wrote to persuade the GOV.UK Senior Technology team to enable Brotli compression on GOV.UK. It was originally posted on the GOV.UK RFCs GitHub repo. It can be viewed here. I'm posting it to my blog so as to own my own work and hopefully give it more visibility. A blog post was drafted about this change but unfortunatly never published, as it was considered too technical for a GDS blog post.

Summary

Brotli compression is a new compression algorithm that is available in modern browsers that can offer 10-20% better compression over gzip compression. By enabling this option on the CDN users with browsers that support it will receive smaller static files over the wire (HTML, CSS, JavaScript), and those that don't will continue to receive assets that are gzipped.

Problem

On GOV.UK we should always be looking to reduce the number of bytes a user has to download to view our pages. This is both good for web performance and data consumption on user devices. Any browser that supports WOFF2 fonts will also support Brotli compression (since this is the compression used in the fonts). According to the latest stats from caniuse.com, the Brotli Accept-Encoding/Content-Encoding functionality is supported by 95% of user browsers globally, and I expect this figure will actually be bigger if we only consider UK based users:

This compression has been enabled on both integration and staging. We saw the following savings in terms of total page bytes for HTML, CSS, and JavaScript across the frontend applications of GOV.UK:

Page	Application	Before (b)	After (b)	Diff (%)
/	frontend	259,371	248,653	-4.13
/coronavirus	info-frontend	288,650	272,687	-5.53
/browse/driving/driving-licences	collections	278,385	265,832	-4.51
/contact	feedback	242,047	232,124	-4.10
/search/all	finder-frontend	313,400	295,979	-5.56
/bank-holidays	frontend	263,686	251,555	-4.60
/national-minimum-wage-rates	government-frontend	266,699	254,073	-4.73
/info/coronavirus/business-support	info-frontend	233,133	226,382	-2.90
/licence-finder/sectors	licence-finder	242,555	234,969	-3.13
/guidance/immigration-rules	manuals-frontend	250,431	240,763	-3.86
/service-manual/service-standard	service-manual-frontend	254,529	245,881	-3.40
/additional-commodity-code/y	frontend	250,664	240,974	-3.87
/government/people/theresa-may	whitehall	277,888	265,642	-4.41

These savings for the pages tested converted into the following performance improvements. All tested on a simulated Moto G4 mobile on a 3G connection.

Homepage

The visual progress of the homepage has improved by approximately 110 ms as can be seen in the visual progress graph:

And we see the a reduction in bytes for all the expected assets:

Coronavirus page

The visual progress graph for this page starts off worse by 400ms, but quickly catches up and the viewport for Brotli completes rendering 200ms before gzip:

And we see the a reduction in bytes for all the expected assets:

Showing a 5.53% reduction.

Bank holidays

The visual progress of the bank holidays has improved by approximately 200ms as can be seen in the visual progress graph:

And we see the a reduction in bytes for all the expected assets:

Showing a 4.60% reduction.

Search page

The visual progress of the search page has improved by approximately 400ms initially, 200ms at the end. As can be seen in the visual progress graph:

And we see the a reduction in bytes for all the expected assets:

Showing a 5.56% reduction.

Summary

In all the pages tested across the frontend apps (17 in total) we see between 3-6% reduction in the number of bytes a browser is having to download over the network. This saving is being converted to improvements in the visual metrics a users is seeing in the browser.

Proposal

The proposal is to enable Brotli compression at the edge using the Fastly Brotli Compression LA. Browsers that support Brotli will receive these versions of the files. Any browsers that don't will receive the gzip version of the file. This can therefore be applied using the progressive enhancement methodology. In doing so we MUST update the configuration VCL so as to automate this process. During testing it was simply enabled via the Fastly user interface.

We SHOULD also purge the static cache once completed as it was found during testing that certain Fastly PoP's continued to serve the uncompressed version of the file since there was a mismatch between Accept-Encoding header from the browser and Content-Encoding version available on the CDN. Once the cache was purged this issue was rectified. However, this issue would rectify itself over time even without the manual purging, since Rails inserts an MD5 fingerprint into the filename of each file. Once a file is changed it will have a new URL, thus invalidating the old cache.

Enabling HTTP/3 on GOV.UK

2023-09-13T00:00:00Z

Note: This is an RFC I wrote to persuade the GOV.UK Senior Technology team to enable HTTP/3 on GOV.UK. It was originally posted on the GOV.UK RFCs GitHub repo. It can be viewed here. I'm posting it to my blog so as to own my own work and hopefully give it more visibility.

Summary

HTTP/3 is the latest version of the HTTP protocol that is currently in use in production by Facebook and Google, as well as many others. It offers web performance improvements over HTTP/2, specifically when connections with high packet loss are being used. Users on unstable mobile connections and rural areas should see the most improvement with this new protocol adoption.

Problem

According to the latest data (October 2020) by MLab, the UK has a median download speed of 20 MBit/s and 7.2 MBit/s upload speed. However, exporting this dashboard data to a spreadsheet paints a very different picture.

The data shows that there are parts of the UK that have connection speeds less than 0.3 MBit/s for both download, and upload. This is roughly equivalent to a 2G connection. It's worth noting that as these are median values, many users in these areas will be seeing much lower connection speeds than this in reality. These areas have a connection speed that is 98.5% slower than the median for the whole country.

Users in these areas still have a need to access government services and the information that GOV.UK offers, but under these conditions it won't be a very pleasant experience. These are the users that will benefit from a transition to HTTP/3.

Brief history of HTTP

Each iteration of the HTTP protocol has set out to improve the web performance of the previous version of the protocol.

HTTP/1.0

The first version of the HTTP protocol came with a number of web performance issues, a primary issue being that only a single asset could be downloaded on a Transmission Control Protocol (TCP) connection before the connection was then closed. There was no way for these connections to persist for multiple asset downloads. This resulted in a lot of wasted negotiation time to establish separate TCP connections for each asset.

HTTP/1.1

HTTP/1.1 fixed the persistent connections issue, allowing multiple assets to be downloaded over a single connection once established. Unfortunately this didn't fully solve the web performance issues seen with the HTTP protocol (stemming from the TCP layer).

Under HTTP/1.1 the browser must download 100% of an asset before it can move onto the next asset. Meaning it can only download a single asset at a time per connection, the other assets are blocked. This is called Head-of-Line Blocking (HOL Blocking). Browser vendors quickly tried to solve this issue by allowing a browser to open up to 6 TCP connections per domain, so 6 assets can be downloaded in parallel.

Unfortunately a number of issues still exist with this method:

Most sites have many more than 6 assets per page, so HOL blocking is still occurring in almost every page load under HTTP/1.1.
Each TCP connection is independent and have no knowledge of each other. All 6 are fighting for limited bandwidth which in turn can saturate the network.
Many websites resorted to "domain sharding" (hosting assets across multiple domains), to allow a browser to open up more that 6 TCP connections. This in turn exacerbated the problem mentioned in point 2.

Another solution needed to be found.

HTTP/2

HTTP/2 is a major iteration on the HTTP protocol. It started life as the SPDY protocol created by Google, which was eventually used as the foundation for HTTP/2. One major shift in this version is the ability to multiplex stream assets over a single TCP connection.

Each TCP packet now includes a set of "frames" that contains additional meta information about each stream. Each asset is downloaded over a separate stream and then reconstructed by the browser. This allows multiple assets to be downloaded at the same time over a single TCP connection, removing the HOL blocking at the application (HTTP) layer. An example of this setup can be seen in the diagram below:

Source: "Head-of-Line Blocking in QUIC and HTTP/3: The Details" by Robin Marx.

Unfortunately this didn't solve all the HOL blocking issues. HTTP/2 is built upon TCP: a protocol that offers the reliable transmission of packets. TCP's job is to make sure that the packets leaving one device arrive on the other device in exactly the same order, and with no missing packets. If packet loss occurs, TCP re-transmits the packets. Packets behind the missing packet cannot be processed until retransmission occurs, so all assets being transferred on that single TCP connection are held up.

This is where we are currently at with GOV.UK. The vast majority of users on strong, stable connections will have seen improved performance from enabling HTTP/2. Unfortunately, some users may have seen performance drop if they use a connection with high packet loss. This is due to the HOL blocking that now exists at the network layer. This is a primary issue HTTP/3 sets out to solve.

HTTP/3

To solve the HOL blocking issue at the network layer, HTTP/3 uses a completely new transport layer protocol called QUIC. QUIC initially stood for "Quick UDP Internet Connections", but the IETF now don't use it as an acronym. It is simply the name of the protocol. QUIC is built upon the User Datagram Protocol (UDP), so isn't bound by the restrictions of TCP. It has been designed so that the network layer now understands the the concept of independent streams, which was restricted to the application layer under HTTP/2:

In the illustration above, if packet 2 is lost (using HTTP/2), packets 3-8 are blocked until retransmission occurs. Using QUIC if packet 2 is lost only packets 1 and 3 are blocked since they exist in the same stream. The other streams can progress as normal since they are fully independent. QUIC doesn't require in-order packet transmission across the whole connection, but it does still retain ordering within each resource stream.

The effect of packet loss

Using WebPageTest we can roughly simulate the effect that packet loss has on performance for our users. For testing I picking a fairly content heavy page like the past Prime Ministers page and a low spec Moto G4 Android device. For the connection speed I've picked on one of the worse case scenarios from the MLab data. I'm emulating the connection the village of Colintraive has, which reportedly has a 450/230 Kbps connection speed. This is a connection speed equivalent to a 2G or slow 3G connection.

Visual filmstrip

Below we can see a comparison of the page load under different packet loss conditions using HTTP/2. The top is the baseline, with 0% packet loss, progressing all the way down to 3% packet loss:

As you can see, the visually complete metric sits at 9.5 seconds with no packet loss, but jumps to 13.5 seconds with 3% packet loss. This is a 42% increase.

Visual progress graph

The visual progress graph clearly shows the effect packet loss has on page load times. As the packet loss increases, the page load slows down:

HTTP/3 + QUIC is designed to help improve the page load speed for users experiencing these connection conditions in the real world.

Connection quality measurement

Is it possible to measure the connection quality that our users are experiencing when visiting GOV.UK. Fastly provide a range of Varnish Configuration Language (VCL) options that we can use to monitor the quality of user connections should we wish to capture this data:

client.socket.ploss - An estimate of the packet loss on the current connection. Provides a ratio from 0-1. 0 = no loss.
client.socket.tcpi_rtt - The round trip time (RTT) for the current TCP connection to the client. RTT is used for loss detection and congestion control across the connection.
client.socket.tcpi_rttvar - Measures the variance in RTT of the connection. The higher the variance value, the less stable the connection.
client.socket.tcpi_delivery_rate - Gives a recent value for the delivery bandwidth to the client in bytes per second.

Our Users

Since HTTP/3 + QUIC offers better performance for users on unstable connections, it is essentially fixing performance for those who were left behind during the transition to HTTP/2. These users are likely to be those who are in the most need of this performance increase.

Since COVID-19 many people have had to work from home. Many will be in rural areas where high-speed internet access either isn't available, or is expensive. In more populated area many users could still be on ageing cable / DSL connections with high contention ratios.

The aim of this transition to HTTP/3 is to allow an ever more diverse level of user connection quality to access GOV.UK, essentially maximising access for all users, no matter what connection they use. By focusing on the long-tail of our users with poor connections, we can effectively shorten it and in doing so make the internet "good enough" to use in more situations. This is all without effecting performance for users already on a good internet connection.

Browser support

The support for HTTP/3 + QUIC in browsers is visible on Caniuse.

The browsers support for HTTP/3 at the time of writing is:

Chrome - Enabled (see Chromium blog)
Edge (chromium version) - Enabled
Firefox - Enabled in the next version (88)
Safari 14 - Disabled (currently behind a feature flag)

Looking at the March 2021 analytics, approximately 55% of GOV.UK users are using a browser that can now take advantage of HTTP/3 + QUIC. This is close to the 66% that the CanIUse data estimates for HTTP/3 usage globally.

Usage on the web

Although the HTTP/3 and QUIC specifications are still in draft, they are both used in production on large sites with billions of daily page views:

YouTube

In the image below we see Chrome 89 (latest stable) connecting to YouTube and using h3-Q050. This is HTTP/3 with the latest version of Google QUIC (Q050).

Google Search

In the image below we see Chrome 89 (latest stable) connecting to Google Search and using h3-Q050. This is HTTP/3 with the latest version of Google QUIC (Q050).

Facebook

Facebook are using HTTP/3 and QUIC in production, but are using the IETF version of QUIC (h3-29). Below we see Firefox 88 connecting via HTTP/3:

Other advantages to HTTP/3 + QUIC

I've only covered a single advantage related to packet loss with web performance above, but HTTP/3 + QUIC comes with a number of other advantages:

Improvement in connection startup times via TLS v1.3 + 0-RTT
Better security via end-to-end encryption at the transport layer
Establish an ecosystem that is easier to evolve from our current TCP usage

Eventually our users will also be able to take advantage of Connection Migration, where QUIC end-points will be able to migrate connections to different IP addresses and network paths at will.

Connection migration is an optional part of the specification that isn't currently implemented anywhere. Once enabled it will allow a users device to seamlessly transition between multiple QUIC connections (e.g. Cellular to known Wi-fi). This is achieved by introducing the concept of a connection ID that is carried by QUIC packets, which can be used to identify a connection.

A user is likely to find this useful if they are live streaming data (web conferencing, large file upload / download, real-time navigation etc). A live demo of this technology in cation can be found here.

Since this is optional in the QUIC specifications, realistically its practical adoption may be a number of years away.

Summary

Our move to HTTP/3 + QUIC offers web performance gains to users on unstable, high packet loss connections. These users were left behind in the migration to HTTP/2 across the web platform. These web performance gains will come with no loss in performance to users already on a strong, stable connection. All our users will have an improvement in security, as well as improvements in connection startup times.

It has never been more important to fix issues for our long-tail users, since many will now be working from home due to the Coronavirus (COVID‑19). Users no longer working in an office environment may now be limited to unstable connections in rural areas, or tethering via limited mobile data plans, or using a connection in populated inner-city area with a high local contention ratios.

Any browsers that don't support HTTP/3 + QUIC will fallback to HTTP/2 + TCP. Therefore HTTP/3 can be enabled using the progressive enhancement methodology.

Proposal

Fastly have now announced HTTP/3 + QUIC is available to all customers as it is out of Limited Availability (LA) and the beta phases. Enabling HTTP/3 + QUIC for us involves switching it on via the Fasty UI. No modifications are needed to our application code, or servers. And since we have already invested work in optimising for HTTP/2, we are already well prepared for a migration to HTTP/3. So no modifications are needed to our frontend applications.

There is a CNAME config change required if we don't use a dedicated Fastly IP address once enabled in the UI. From the documentation:

Unless you use Fastly's dedicated IP addresses, then as a final step to enabling HTTP/3, you must ensure the DNS records of your domains are routing users to the correct HTTP/3 enabled Fastly addresses by using one of the following CNAMEs:

0-RTT enabled?	IPv4 and IPv6 dual stack routing	IPv4-only routing
Yes	`dualstack.n.sni.global.fastly.net`	`n.sni.global.fastly.net`
No	`dualstack.m.sni.global.fastly.net`	`m.sni.global.fastly.net`

I know GOV.UK currently has IPv6 enabled, so if a CNAME change is required I propose we use 0-RTT = No (dualstack.m.sni.global.fastly.net), since I belive Firefox is still the only browser that supports it. We can always enable this at a later date quite easlily once it becomes more promenant in browsers.

Note: According to Richard Towers, we currently don't use dedicated Fastly IP's so the CNAME change will be required, this shouldn't be a major issue because we have a couple of levels of CNAME in place (we don't need to talk to JISC). It's also worth noting that TTL on prod is 300 seconds (5 mins), which should be short enough for the change when made.

Date to be enabled

I've opened an additional RFC-147, that will allow us to distinguish users using HTTP/2 and HTTP/3 in SpeedCurve RUM. I believe we should impliment this RFC first then allow it to capture user data for 1-2 months, before we enable HTTP/3. This will allow us to quantify the before / after change in our RUM metrics, which would make for an interesting PR piece from a web performance and user focus point of view. So I'd propose to enable HTTP/3 some time in July / August 2022.

HTTP/3 enabled on GOV.UK!

I'm very pleased to say that HTTP/3 + QUIC from Fastly was enabled on GOV.UK on the 11th January 2024! See the tweet thread here for more information.

Migrating from Github Pages to Netlify & Cloudflare

2021-09-06T00:00:00Z

So in recent months I've had a fair bit going on in my life which has held me back in many ways, but also pushed a few things up the priority list too. One of the areas that has been bumped up the list is the current setup of this very blog. So I decided to clean up the years of "it still works, why would I change it", technical debt. Isn't it incredible how many years you can do this for!

Dreamhost

Just a very brief look back on the history of the blog and allow me to vent a little. Feel free to jump to the next section if this is of no interest to you.

I started using Dreamhost for my hosting & domain registrar back in April 2006, just over 15 years ago! At the time this was because they offered a lot in terms of functionality all rolled into one. You could easily purchase your domain and set up the hosting quickly. So this became my default option for friends and families websites. The initial version of the blog used WordPress (~v2.0). At the time this was a great way to learn how to work with PHP/MySQL, and WordPress felt like the only real quick ‘self-hosted' option available (I'm sure many will disagree with that statement).

Over the years my Dreamhost control panel just filled with random domains and projects that weren't maintained. So in the past 3 months I decided to clean this all up. Remove all the old unused domains, and migrate any friends' websites over to other hosting options, so I'd no longer need to pay for a service I really don't need. My plan was to try and get friends to open their own Dreamhost account and basically get the Dreamhost team to do an ‘internal transfer' between accounts. Simple right?!

So I contacted Dreamhost telling them my current situation and my plan. Their response was basically: "Sorry to hear of your situation...sure we can help you do that… for this one time admin fee of $99.00 USD...". Isn't capitalism great! Not that I'd expect it to be free, but after 15 years and the fact that I'm trying to get friends to sign up and pay for their services moving forwards, you'd have thought they would have the foresight to waive the fee.

I neither had the drive or the mental capacity to argue the point so left it at that and decided that Dreamhost isn't a company I want to be dealing with anymore, especially not with all the other options available on the market now! So time to transfer everything away, including them as a domain registrar, which is pretty much all it's been used for with this blog for many years anyway.

Github pages

This blog is built using the static site generator called Jekyll. It works okay, but can be a bit of a dependency hell when updating as it is all built on Ruby. The great thing about this setup is that it integrates really well with Github (GH) Pages. There's a whole guide on how to set up a new site really quickly here. So this is what I did for many years. I didn't even bother setting up a custom domain, the blog just existed on nooshu.github.io (more on this later). Once configured, just write your posts in Markdown, push your changes to the repository and GH pages will handle the rest.

The downside with this setup is:

It's fairly limited. You can only use specific Jekyll plugins supported by GH pages. So you can't take advantage of all plugins available. It's a paid for only option if using a private repository (which my blog is). I don't want to change this restriction.

In all honesty it's mainly the paid for option that I'm looking to remove. Not to be too blunt about it all, but should anything happen in the coming months, future payments wouldn't be made and the blog would soon be taken offline.

Netlify

This is where Netlify steps in. Netlify is a Git-based workflow and powerful serverless platform that can be used to build, deploy, and collaborate on web apps. It comes with a fantastic ‘Starter' plan for personal projects, hobby sites, or experiments. I can't even tell you how simple it is to set up a site if you're already using Jekyll. Literally point Netlify towards your site repository, a minute or so later you'll have yourself an exact copy of the site (although on a randomly assigned URL, but we will get to this part soon). If this hasn't convinced you to give it a try, it is also been supported and maintained by a fantastic bunch of amazing developers like:

Now technically you could pretty much stop there. Let Netlify handle the custom domain, DNS, Hosting, SSL, deployment etc as you don't actually need Cloudflare. But where's the fun in that?? So let's layer on Cloudflare anyway!

Cloudflare

Cloudflare are one of the big market leaders when it comes to Content Distribution Networks (CDN) on the web. But don't let their huge size put you off, they still have an excellent free plan that anyone can use to speed up their website. It's incredible the features they offer on their free plan:

There really is so much there for a standard website or a hobbyist looking to do some web performance experimentation.

Another bonus is that they can also act as a domain registrar either for new domains, or to transfer existing domains to them. So this is what I've done. My Dreamhost domains have now all been transferred to Cloudflare and are all in a single place.

Setting it all up

Enough of the babbling, let's dig into setting it all up!

Domains

First thing I did was to start the domain transfer process. Dead simple to do. Make sure the domain is unlocked, grab the auth code from the previous registrar then wait 5 days… It's annoying that you wait 5 days for this but Dreamhost doesn't give you an option to bypass the ‘cooling off' period. It would be great if there was an option to start the transfer process right away, but alas it is what it is. I did this via Cloudflare Registrar, but any will work in mostly the same way.

Netlify

As mentioned above, Netlify is so simple to get setup. Click the "New site from git" button, select your repo and configure your build settings.

Clicking the button will bring up the following options:

A few minutes later you'll have a new site on a random Netlify URL that you can easily access.

SSL Certificate + DNS

This is where it got slightly more complicated. By default you could leave Netlify to handle the DNS settings and SSL certificate, but since we want to front our site with Cloudflare we have to make a few changes.

SSL Certificate

So we aren't going to use Netlify to provision the SSL certificate, we are going to be using Cloudflare. This is important if you want to use the CDN functionality of Cloudflare. As it is the "front" of the website, we are going to be using Netlify as the origin server.

An SSL certificate provisioned by cloudflare will allow encryption of traffic between the origin (Netlify) and the CDN (Cloudflare). I believe this is also important when it comes to using the HTTP/3 + QUIC functionality that Cloudflare offers on their free plan. This is because TLS is very much embedded within the QUIC protocol, rather than a separate layer like it is in HTTP/2. So if Netlify were to handle the SSL certificate, it would limit our ability to use HTTP/3. Note: this is just an educated guess of mine. As I could only get HTTP/3 + QUIC to work when Cloudflare provisioned the SSL certificate.

So first you need to generate an origin certificate in the Cloudflare control panel for your domain:

I enabled Full (strict) mode here that requires the Cloudflare CA certificate.

Next you need to enter this certificate info into Netlify via the "Site settings → Domain Management → HTTPS". Click the ‘Add (or Update) custom certificate'.

Install your custom certificate by copy & pasting the corresponding fields into the textareas provided. You will also need to provide the ‘intermediate' certificate details. For this, go to the Cloudflare Origin configuration page and download the Cloudflare Origin RSA PEM file and open it in your favorite text editor. Copy & paste the contents into the last textarea.

Once completed your HTTPS settings in Netlify should look similar to this:

That's about it for SSL certificates. You've now installed the Cloudflare origin certificate on your origin server (Netlify) so they can talk to each other over an encrypted connection.

DNS

Okay, here's the part that I don't mind admitting took me bloody ages to figure out! When it comes to Domain Name System (DNS) I know enough to be dangerous, but not enough to be useful! So there was a fair amount of head scratching and trial and error to get this all setup. It doesn't help that these settings take a while to propagate across the internet. So you never know if they are still propagating, or if you just broke something! For ease, here are the settings I currently have in place in my Cloudflare DNS panel:

First we have an A record pointing to Netlify's load balancer IP address (75.2.60.5). More info on this is available on their "Configure external DNS for a custom domain" page.

Second we have a CNAME record pointing www to nooshu.com. I don't want the ‘www' in my URL, so it is just being pointed to the bare domain (https://nooshu.com).

Note that for both of these records the ‘Proxy Status' column is set to ‘Proxied' (notice the orange cloud). This is important as we want to make sure the hostname is running through Cloudflare as our CDN provider.

Under this panel I pointed the domain to the Cloudflare Nameservers (braelyn.ns.cloudflare.com and cartman.ns.cloudflare.com). Finally in the DNS page I enabled DNSSEC. This isn't required, but it adds a layer of extra security by protecting against forged DNS answers. It comes for free with the free plan, so may as well I guess! (Unless anyone knows otherwise?)

That's it for the Cloudflare DNS page! A little word or warning: if you go back to the Netlify ‘Domain Management' panel you will see some scary looking warning messages next to your custom domains.

These are fine to ignore. It's simply because the domain's DNS isn't being handled by Netlify, it's been handled by Cloudflare.

Netlify Forms

This is an optional extra. If your site requires a contact form you can't go wrong with Netlify. It comes with the ability to manage forms and submissions without any server-side code or JavaScript. It's all done via Netlify Forms. It is dead simple to set up and comes with built-in spam protection via Akismet and an optional honeypot field. All for free (up to a certain usage limit).

Email

Next up is email setup. Ideally I wanted name@domain.com to forward to my GMail account, but Cloudflare doesn't handle emails at all. Thankfully I found an excellent free service to do this called ImprovMX. Simply enter the alias you require and the email address you want to forward too, then follow the instructions given. You enter all these details into your Cloudflare DNS panel. An example of my setup is below:

That's all there is to that really. Quick and simple!

Redirects

This was a big concern for me at first. My blog has been on the default Github domain for a number of years (nooshu.github.com). So what would happen if I suddenly changed to nooshu.com? Would all the links built up over the years just break? Github pages is very limited as to what you can do technically. It's not like it uses Apache for example, where you can use something like mod_rewrite to create redirects.

Thankfully Github does come with some useful functionality to help in this situation. Navigate to ‘Settings → Pages → Custom domain' page. Enter the domain you wish to serve the site from into the input box. Behind the scenes this will create a CNAME file in the root of the repository that will redirect users to the new domain.

So in my case nooshu.github.com redirects to nooshu.com automatically. Assuming you keep the rest of your URL structure the same, you should be all good to go!

Web Mentions

I use Webmentions on my blog. I actually wrote about how I do it here. The ‘issue' with using web mentions via the Webmention.io service is that it is tied to a particular domain. But what happens once the domain is changed? Suddenly all the webmentions that were associated with nooshu.github.io are no longer pulled in since the blog has moved to nooshu.com. Thankfully after a brief chat with Kevin Marks on Twitter a simple solution was found.

In my JavaScript I'm querying for any webmentions that are associated with the nooshu.github.io domain. The webmention.io API allows you to query multiple domains at the same time. So I just modified my JavaScript (opens in new tab) to look for mentions from both domains. This allows for mentions from the past (nooshu.github.io) and also any mentions in the future (nooshu.com) to be collected and parsed. Simple!

Web Performance

This final section of the post was the really interesting part for me. How to get the best performance out of the new Netlify + Cloudflare setup. It's worth mentioning upfront that by default it really isn't optimal. Here's the WebPageTest result for this site before any changes were made. Here I'm testing on a real Moto G4 on a 3G Fast connection. So essentially a low spec device on a slow connection.

So the test result isn't actually too bad in terms of performance. A Largest Contentful Paint (LCP) and visually complete of around 2 seconds on a low spec device is a good place to start. But there are three scores that stand out:

Security: E
Cache static content: F
First Byte: B

All these scores are very much intertwined, but I will go through each of them individually the best I can..

Security

Since this is a small static blog with no admin area or any form of login, security isn't a major issue. But it would be nice to get an ‘A' in this anyway. Thankfully it really isn't that hard to do if you are using Cloudflare as a CDN. Security expert Scott Helme covers most of what to do in his blog post "The brand new Security Headers Cloudflare Worker". If you are looking for a guide on setting up a Cloudflare Worker, I've written a post here you can follow. The issue with the score is that the site is being served with some missing security headers:

Note: There's no need to add these missing headers using a Worker anymore as Cloudflare has just released some new functionality called "HTTP response headers with Transform Rules", see my update below on how to do this.

As you can see from the scan above by securityheaders.com, we are missing 5 security headers (in red). By following Scott's guide and a small tweak related to ‘Feature-Policy` header you can easily get it up to an A+. Here's the headers I have included in my Cloudflare Worker:

let securityHeaders = {
    "Content-Security-Policy" : "upgrade-insecure-requests",
    "Strict-Transport-Security" : "max-age=31536000; preload; includeSubDomains",
    "X-Xss-Protection" : "1; mode=block",
    "X-Frame-Options" : "DENY",
    "X-Content-Type-Options" : "nosniff",
    "Referrer-Policy" : "strict-origin-when-cross-origin",
    "Permissions-Policy": "fullscreen=(), geolocation=(), camera=()"
}

This code, combined with Scott's boilerplate code adds the headers to the HTML response from the CDN. Now you don't necessarily need to use a Worker to add all these headers. You could use functionality that Netlify has available where you can modify the headers coming from the origin using a _headers or netlify.toml file. But I didn't have much luck with this route. Certain headers were being stripped somewhere in the process so I eventually just reverted to using a Worker to do it. Simon Hearne has an excellent blog post all about this, where he managed to get it to work if you are interested. The above setup will also allow you to submit your domain to the HSTS Preload list too, ensuring that the domain (and all subdomains) are only accessible with a valid SSL certificate.

Once the worker is appending the missing headers to your HTML response the score jumps from an E to an A+! Star pupil indeed! All using a Cloudflare Worker that comes for free on the free plan. Win-win all round!

Using HTTP response headers with Transform Rules

As mentioned above Cloudflare have just relesed functionality that allows you to easily modify response headers without the need of a Worker. It's all done via the Cloudflare UI.

Just select your site then navigate to 'Rules' --> 'Transform Rules'. Then create a new set of response rules like below:

Note: You can't currently use the expression builder to target only the HTML response, but you can paste in the following expression if you only want to target a specific content-type:

http.request.method eq "GET" and any(http.response.headers["content-type"][*] contains "text/html")

Also worth nothing that this feature is on the list to be added to the expression builder. If you don't only target your HTML, your finely crafted Cache-Control asset headers will be decimated and all overwritten with public, max-age=60.

Then I've simply added 9 static rules to the response from Cloudflare:

The headers I added can be found below, the top 8 bump the security score from E to A+ on Snyk.io. The final header is a specific preload header for this blog and can be ignored.

/** Important security related headers**/
Cache-Control: public, max-age=60
Content-Security-Policy : upgrade-insecure-requests
Strict-Transport-Security : max-age=2592000
X-Xss-Protection : 1; mode=block
X-Frame-Options : DENY
X-Content-Type-Options : nosniff
Referrer-Policy : strict-origin-when-cross-origin
Permissions-Policy: fullscreen=(), geolocation=(), camera=()

/**Specific preload for my blog**/
Link: ; rel=preload; as=image; nopush, ; rel=preload; as=image; nopush

Once this is done you can deploy and strip all the headers from the worker that exist in the securityHeaders object in the Worker code. Done!

Many thanks to the Cloudflare team who quickly removed a restriction with the Strict-Transport-Security header in a couple of hours. Amazing work!

Cache static content

This is where I first spotted a real performance issue, and many thanks to Anthony Ricaud and Andy Davies for helping me solve this issue. Logging into the Cloudflare dashboard, on the ‘overview' page it gives you a whole set of useful graphs. The interesting one was the ‘Percent Cached' graph:

As you can see from the graph above, it's sitting around a 2% cache rate. For a static site this is truly shocking. There's literally nothing dynamic going on with any of the pages. Why was the cache hit percentage so terrible?

To help debug the issue I used a handy feature that Chrome has built-in, where you can expose a custom header in the browsers network tab. In the case of Cloudflare you want to be exposing the cf-cache-status header. There's an explanation of what this header means here. A quick TL;DR; is that we want as many of our assets to have a value of HIT, which means the asset is stored on the CDN and served to our user from their closest Point of Presence (PoP). As you can see from the image below, we were far from the case!

There's not a single HIT in the above image! No wonder the graph percentage was so low! So what's going on here? Well this is where we have a collision between the Netlify and Cloudflare headers. By default many of the assets served by Netlify come with a Cache-Control header value of public, max-age=0, must-revalidate.

Using Jake Archibald's "Caching best practices & max-age gotchas" and Harry Roberts handy "Cache-Control for Civilians" we can start to make sense of these somewhat confusing header values:

public - this is good, no issue here.
max-age=0 - This is bad, it forces the cache to revalidate (clears the cache).
must-revalidate - Here we are forcing the browser to go back to the server to revalidate and check for a new resource, Harry writes about it in detail here.

The important thing to realise is that even though Cloudflare is fronting the domain, it still looks at, and in many cases, just forwards these headers to the user. So Netlify is effectively asking the CDN not to store the asset and go back to the origin to revalidate the asset on every request. This isn't exactly what we want! This puts load on the origin and also adds to our users Time to First Byte (TTFB), since their nearest PoP isn't being used.

So you may be asking why exactly are Netlify serving up these headers that are very aggressive in terms of anti-caching? Well Netlify has a superb build process that integrates incredibly well with lots of different static build tools. Honestly, if you haven't tried it you really should! Often you can point Netlify at a Github repository and 2-3 mins later you'll have a fully hosted, working, static site you can share with friends and family. One of the huge advantages to this is that you can commit a change to Github and minutes later it is reflected on the live site. No more FTPing changes up to production! Or in other words, Continuous Delivery / Continuous Deployment (CD). For this to work you need to ensure that a developer (or user) has the latest assets from the server. There's no point deploying a change to a site if the browser continues to use an already cached version in a users browser!

Note: There are ways around this by using something called asset fingerprinting. What this does is generate a unique filename based on the contents of the file (usually a MD5 or SHA hash). So for example styles-2f3f4bb27f.css. Now whenever this file changes, even by a single byte, the filename will also change, thus creating a new URL and invalidating the cache. I've done this for the CSS and JavaScript on this blog which you can see if you view your DevTools network panel.

Now, Netlify has no idea if you are fingerprinting your assets, so it can't assume you are. Instead they make sure a user gets the latest assets by defaulting to the Cache-Control headers above. But remember you can override them if you wish using the header options mentioned earlier.

So what did I do to solve the issue? Well I created a simple netlify.toml file and placed it in the root of my project:

# netlify.toml

# remove `immutable` if you don't have file fingerprinting available
[[headers]]
  for = "*.css"
    [headers.values]
    Cache-Control = "public, max-age=31536000, immutable"
    Vary = "Accept-Encoding"

# remove `immutable` if you don't have file fingerprinting available
[[headers]]
  for = "*.js"
    [headers.values]
    Cache-Control = "public, max-age=31536000, immutable"
    Vary = "Accept-Encoding"

[[headers]]
  for = "*.ico"
    [headers.values]
    Cache-Control = "public, max-age=31536000"

[[headers]]
  for = "*.json"
    [headers.values]
    Cache-Control = "public, max-age=31536000"

[[headers]]
  for = "*.png"
    [headers.values]
    Cache-Control = "public, max-age=2419200, must-revalidate, stale-while-revalidate=86400"

[[headers]]
  for = "*.jpg"
    [headers.values]
    Cache-Control = "public, max-age=2419200, must-revalidate, stale-while-revalidate=86400"

[[headers]]
  for = "*.jpeg"
    [headers.values]
    Cache-Control = "public, max-age=2419200, must-revalidate, stale-while-revalidate=86400"

[[headers]]
  for = "*.svg"
    [headers.values]
    Cache-Control = "public, max-age=2419200, must-revalidate, stale-while-revalidate=86400"

[[headers]]
  for = "*.avif"
    [headers.values]
    Cache-Control = "public, max-age=2419200, must-revalidate, stale-while-revalidate=86400"

[[headers]]
  for = "*.webp"
  [headers.values]
  Cache-Control = "public, max-age=2419200, must-revalidate, stale-while-revalidate=86400"

There's nothing particularly clever about this file. I'm basically pulling out particular file extensions and changing their Cache-Control header values. I'm sure there are ways to simplify this file, if there are please do let me know!. Note: You may notice I've used immutable on my CSS and JS files. This is because I have fingerprinting in action on these files. I'd recommend removing immutable if you don't.

Let's see what this change looks like in the network panel on our temporary Netlify domain:

Well the Netlify origin headers are certainly looking a lot better! You can see the headers we've set in the netlify.toml file are reflected in the responses. Let's see now how this translates when we run it through Cloudflare:

Here I'm using Chrome and exposing the cf-cache-status header for debugging as mentioned above. As you can see it's looking a lot healthier! Here we now have 6 assets with a status of HIT and 2 with a status of DYNAMIC. So most of our page assets now reside in the CDN cache.

We're getting there, but what are those DYNAMIC cf-cache-status's all about? Well Cloudflare still thinks these assets are changing a lot, so they are never cached. The CDN will always go back to the origin for these assets, which is not what we want for a static site. So how do we fix this issue?

Page Rules

This is where the awesomeness of Cloudflare and their free plan delivers again. They have functionality called Page Rules available to us. Page Rules give you fine-grained control over how Cloudflare handles certain URLs (or URL patterns). Luckily for us, they give you 3 rules on the free plan. More than what we actually need, we'll only need 1!

With the page rules we want to tell Cloudflare to ‘Cache Everything'. Basically everything served from our origin is cacheable. We also want to tweak the Edge Cache Expire TTL. This is the easiest way to override any existing headers coming from our origin. We are going to do this for the whole domain, so I set nooshu.com/* for the URL pattern:

Let's now see what our CDN caching looks like with the above page rules set:

Well look at that! 8 out of 8 assets with a cf-cache-status of HIT! Awesome, we're looking a lot better!

Note: The minimum TTL you can see on the free plan is 2 hours. This means that if you make an update on your origin the CDN won't check for an updated version of the file for 2 hours, unless you purge the CDN cache. I'm personally fine with this limitation. If you decide to pay for the 'Pro' plan this can be reduced to as little as 30 seconds. So it really depends what functionality you need and what you are willing to pay for. You can always do a custom cache purge on a single URL if needed in an emergency, so that's an option too.

We're not quite finished yet. Notice how the HTML page has a must-revalidate value. This value is redundant since our max-age is now set to 1 year, so let's remove it. You could either do this via the netlify.toml or our Worker code. I personally prefer to do it at the Edge so have done it in the Worker. Simply add the following in the Worker securityHeaders object above:

"Cache-Control": "public, max-age=60",

I'd also recommend renaming the object to something new, since it no longer only contains security headers. Adding this line overrides the must-revalidate value sent out in the response from Netlify.

I'm not the only person to write about this problem, Vlad Iliescu has also written about it here, if you want a more detailed explanation.

Bonus: Link Preload Header`

This part is completely optional, and totally depends on your domain and use case. Using a Worker it is dead simple to add and test the Link Preload HTTP Header. This is a header that is sent along with the HTML file that tells the browser to immediately request certain assets you list in the header value. This can be useful for images and fonts for example. Assets that are listed in CSS files have to wait for the CSS to be downloaded and parsed by the browser, before a request can be made to the server to download them. By preloading the asset, it can be cached by the browser and used immediately when needed.

Be warned though, use the preload functionality sparingly, and be sure to test that it actually improves the performance of your website! Read all about the puzzle that is preloading assets in this blog post by Andy Davies. Here it is in action on this blog. This is the single line I added to the Worker:

"Link": "</images/image1.png>; rel=preload; as=image; nopush, </images/image2.svg>; rel=preload; as=image; nopush, </images/image3.svg>; rel=preload; as=image; nopush"

So what's the result of all this tweaking on our cache percentage in the Cloudflare control panel? Well it looks a lot healthier:

As you can see from the image above, we've gone from a truly terrible 1.98% cache, all the way up to 87.35% cached, and this is growing hour by hour! Not bad considering how little work we've actually had to do.

First Byte

The last point to cover is the First Byte score which was a respectable ‘B'. For it to be bumped up to an ‘A' we need the TTFB to be around 1 second (1004 ms). Thankfully we've already done all the work to reduce this TTFB time. Our request no longer goes all the way back to the origin (Netlify). Instead it is cached by the CDN and is delivered to our user from their closest geographic PoP. When it comes to latency the speed of light is the limit, so the closer the server is to the user, the better!

So let's take a look at the final score from WebPageTest with all the above tweaks applied:

That's a lot of green! Considering this is on a low spec device on a 3G Fast connection, more powerful devices on faster connections are going to get an even better experience. Here are some raw stats from comparing my two sets of test runs:

16.4% reduction in visual complete time
25% reduction in TTFB
22.3% reduction in load time
16.8% reduction in LCP

And a comparison filmstrip:

And no web performance dick measuring contest would be complete without a Lighthouse report including animated fireworks:

Summary

Well considering this was only supposed to be a "quick blog post", it's turned into a bit of a beast! There's a lot in the post, so let's recap what we've covered and what infusing Netlify & Cloudflare brings to the table for a static sites like mine:

HTTP/3 + QUIC
0-RTT
Cloudflare Workers
Preload Link HTTP Header
DNSSEC
Excellent cacheability of static assets
SEO consideration moving from Github pages to custom domain
Netlify's amazing Continuous Delivery pipeline
Netlify Forms integration
Easy email integration via ImprovMX
Free hosting on Netlify
Free CDN usage via Cloudflare
Page Rules for fine-grained cache control
Improved security
Improved web performance

It's staggering the amount of functionality and features that are available (for free!) on the web nowadays.

Hopefully you've found this post interesting and useful! As always, feedback and improvements are always welcome. Just let me know via Twitter.

Post changelog:

06/09/21: Initial blog post published.
06/09/21: Added missing aknowlegements for Anthony Ricaud and Andy Davies for pointing me in the right direction on how to solve the Cloudflare caching issue!
06/09/21: Thanks again to Anthony Ricaud for spotting a number of errors in the post. This is why you let folks proof-read posts first!
23/11/21: Updated the Worker code to remove the added headers and use the new 'HTTP response headers with Transform Rules' from Cloudflare.
09/12/21: Updated the post to include the correct expression allowing you to only target the HTML response with the Response Transform rules. Thanks to Richard Boulton for his help fixing this issue.

We've spotted something on your scan...

2021-05-12T00:00:00Z

So a word of warning, this isn't a blog post about web performance. I'm afraid those posts may need to be paused for the moment.

If you've already read the original post you can jump straight to the latest update here, or all updates here.

Note: I feel the need to point this out. This post is about a brain tumor and cancer. It's important to realise that the doctors considerations, and ultimately treatment, are unique to my situation. Anything mentioned in this post shouldn't be taken as gospel. If a reader finds themselves in a similar situation, please do take the advice of your own specialists. What works (or doesn't work) for me will 100% be different for other people.

Diagnosis

To be fully upfront about all this, on Wednesday 21st April 2021 I was diagnosed with a brain tumor. Completely out of the blue. In the morning I was busy working away still trying to get Brotli compression tested and enabled on GOV.UK, 7 hours later I was given the news that I had a large brain tumor on my right frontal lobe. Hell of a day, huh!

So you may be asking: "Why are you writing about this?". Well, there are 3 main reasons:

I want to get this info out of my head (hah!) while I can still remember it.
From the conversations I've had with people since this happened, people are fascinated with how it was discovered.
To offer help to others in a similar situation that I (and my family) are currently in.

I've seemingly learned a fair amount about myself and the topic in the past 3 weeks. If you'd like to know more feel free to read on, or if you'd prefer to stick to web performance I have plenty of blog posts here which are much more light hearted!

Note: We have now enabled Brotli on GOV.UK, thanks to Jake Archibald for the humorous reminder :)

What happened?

This is a question I've been asked a lot. From nurses, doctors, pharmacists, out of hours doctors, and family and friends. So I'll give you a rundown of what happened below:

Initial signs something "wasn't right"

Back in October 2020 I started to develop issues with my lower left back and left leg. Multiple consultations with the doctors were of the opinion it was sciatica. I was given strong painkillers, told to look at getting physio, and make adjustments to my working environment. This actually all made sense at the time. Working from home for 8 months, with less physical exercise, and generally working more hours than usual, all could have easily caused this issue. Unfortunately, no matter what I tried it didn't solve the issue. I tried: a standing desk, new office chair, new shoes, lots more walks, physio. Nothing seemed to be helping. It got so bad at one point I was crawling on my hands and knees at 3am to go to the bathroom because I was in so much pain. Not exactly what you expect to be doing at the age of 39!

After several months the back and leg pain seemed to fade, but it then turned into extreme heel pain in my left foot, which the doctor said was likely to be plantar fasciitis. Apparently runners tend to get this a lot, but I certainly wasn't running! So yes, a fun 6 months!

But things have developed further and over the past 4 - 6 weeks I have experienced 3 seemingly very minor seizures. Throughout each I was fully conscious. I will try to describe each of them as I remember them the best I can.

Seizure 1: Standing at my desk

I now have a desk that allows me to both sit and stand very easily. I've got into the habit of standing for meetings with people while on a video call, but when it comes to "real work", I always sit. This tends to give me a good balance of both sitting and standing throughout the day.

For the meeting in question I just so happened to be on mute and wasn't speaking at the time. I suddenly felt quite dizzy and had to lean on my desk a little. At first I thought I was having some sort of panic attack. I don't remember the topic of discussion for the meeting, but at the time it happened I was thinking "maybe this is making me anxious?". I do remember having a really strange out-of-body experience (OBE) while it was happening. It's hard to describe, but it's almost like coming to the realisation that that "you are fully in control of everything you are doing" and "you have an effect on your surroundings and the people you interact with". It feels more like a powerful emotion than anything physical. Or another way I've described it is imagine someone turned off the auto-pilot in your head. You are now having to consciously breathe and blink for yourself, as well as think about all the other things going on too.

Within 10 - 12 seconds this intense feeling had passed and I suddenly felt fine again. It was the first seizure I'd had, so I thought nothing of it, and simply moved on with the rest of the day.

Seizure 2: Driving in the car

This was around 2 weeks later. Now this one scares me now that I know more! Technically I wasn't moving at the time, I was parked at traffic lights waiting for the light to turn green so I could then pull across a carriageway into the local supermarket.

Suddenly, I had the same OBE I'd felt before, where I felt like someone had pulled "me" out of the back of my own head. In front of me the lights turned green and I managed to pull forwards and start moving a little, so I had some control of my feet. Unfortunately I didn't have the strength in my left arm to fully disengage the handbrake, so the warning alarm sounded in the car to tell me it was still engaged. I was fully conscious. My brain was telling my hand to push down and release the handbrake, but my hand simply couldn't. It was a very strange experience where your arm and hand seemingly has a mind of its own.

Because the alarm was beeping, my wife looked over and mentioned that the handbrake wasn't down, but she didn't notice anything else in me that looked different. I didn't try to speak to her, and I'm not even sure if I'd have been able too if I'd tried. Again, within 10 - 12 seconds this had all passed and I was fully able to move and control the car. We went to the shops and returned home with no further issues. Once again I put it down to a panic attack, but as it all happened so quickly I dismissed it and carried on with life. At this point I hadn't linked the 2 separate events together.

Seizure 3: Sitting at my desk

This is the one that happened on the 21st April, the day I was diagnosed. It was around 10:50am and I was sitting at my desk working on a document. Suddenly the OBE I'd felt before came back. Only this time I lost control of both lower arms. I was unable to raise them to the keyboard to continue typing. It then developed further and I felt like I'd lost control of the lower half of my face. Almost like someone was gently pulling down on both sides of my cheeks and I was unable to stop them.

I realised this was happening, as I was fully conscious. During this time I must have tried to move, as I'd started to lean to the right side of my chair. I was rotating slowly, unable to really control my movements. Again as with the other seizures, within 10 - 12 seconds this all passed and I could again control what I was doing. But this time it was different.

With this seizure I'd been sitting at my desk, nothing strenuous was happening at all. It was quiet and I was calm. Because of this I was able to notice that I simply couldn't control my hands and that my face had "drooped". This immediately alerted me into thinking that I was having a mini-stroke. Now, I've seen the FAST campaign adverts on TV, and realised a stroke can become very serious if not acted on quickly, so that really freaked me out!

NHS 111

I immediately opened the NHS 111 website (the UK non-emergency advice service), to fill in the details of what had just happened.

Right on the front page of this service it says:

Check it's not an emergency - Call 999 now if you have: signs of a stroke - face dropping on one side, can't hold both arms up, difficulty speaking seizure (fit) - someone is shaking or jerking because of a fit

At this point I was thinking, "okay.. I'm going to have to phone 999" (for the first time ever in my life). But before I did I thought I'd best discuss it with my wife downstairs. I walked downstairs and calmly told her about what had just happened and also the other brief episodes I'd had weeks before (as I hadn't mentioned them to her). We both agreed that 999 would be the best option.

It was a strange conversation with the 999 dispatcher. They were talking to me, "the patient", asking if I was fully conscious and able to lift my arms, move my face etc. These are all standard checks for a stroke. I was able to do everything they were asking me to do with ease. At this point I felt like a total fraud, and I was simply wasting their time. I further described what had happened and that it had now passed, but I still felt a little weak in my lower arms.

I was categorised as a priority 2 (I believe) and told an ambulance would be with us within the next 5 minutes. So I quickly started to pack a few things just incase I needed to leave with them for the hospital sharpish.

Ambulance

Within 5 minutes there was an ambulance outside the door. Two very nice paramedics came in the house and questioned me about how I was feeling and what had happened. They checked my blood pressure, checked my blood oxygenation level, asked me to focus on their fingers, push & pull with my arms against theirs etc. These are all standard checks for suspected stroke patients. It turns out they couldn't find any issues, which was embarrassing. It got me thinking: "had I just imagined this all happening?" and "am I just overacting and wasting their time?".

One of the paramedics decided to speak to the specialist unit at the hospital to ask for advice, as they were unsure what would be best (and so was I!). After a 5-10 minute discussion I had 2 options:

the paramedics would leave and I'd take this up with my doctor to see what was happening. This could take weeks.
come with them in the ambulance and go to hospital for additional checks.

As I was still thinking this could be early signs of a stroke, I (thankfully!) decided that going to the hospital was the best course of action. Even if just to rule out the possibility of a stroke! So that's what I did. I got in the back of the ambulance and 20 minutes later I arrived at the hospital and was admitted to a very busy ward.

Hospital

I'm afraid not much really happened at this point for an hour or so. I had chats with nurses and the doctors on duty, describing what had happened as I mentioned above. The doctor looking after me was a little perplexed by it all, as it was very unusual. I can't remember the exact term he used, but I believe he was thinking these symptoms sounded like some sort of dissociative disorder. This was because of the OBE I'd described to him.

Computerized Tomography (CT) scan

As part of the standard checks I was given a CT scan. Not only that, they injected me with a special dye called 'contrast material'. One of the side effects of this contrast is you feel like you've wet yourself about 20 seconds after they inject you with it. I honestly thought I'd had an accident when it happened. Apparently it's related to the kidneys flushing the dye through your system. Either way, it feels weird!

Due to the ward being extremely busy I needed to wait for the results to be examined. This probably took around 5 hours. The first four hours seemed to go quite quickly. I had my noise cancelling headphones and a good book I was enjoying, so I was entertained (although slightly uncomfortable in the high back chairs and a face mask for COVID).

We've spotted something

About 5 minutes before being shown into a private room, the doctor looking after me came up to me and said the sentence:

We've not forgotten about you, we've just spotted something on your scan

Strangely enough, I couldn't concentrate on my book or music anymore. That was a long 5 minute wait! Well, the 5 minutes passed and the doctor came back over and asked me to follow him into one of the rooms. "Fine", I thought, "how bad could it be?". But a few seconds later I could immediately tell something wasn't right about his body language and manner. He seemed to be struggling to find the right words. And then he came out with it:

I'm really sorry to tell you this, but we've found a brain tumor on your right frontal lobe

I have to admit the rest of the conversation is mostly lost due to the enormity of what I'd just been told. It felt like the ground dropped from beneath me. But I do remember him saying it "was large", and that "if you were to wish to have a brain tumor, this would be the type you would want to have". A strange statement now I think about it, but one that has been confirmed since.

After about 3 or 4 minutes of chatting about who knows what, I just wanted to get out of the room. I actually felt quite sorry for the doctor. He must have done this so many times with patients over the years. I can't imagine it ever gets any easier. I have a huge respect for anyone who has to do this on a daily basis, it must be totally crushing at times.

I quickly managed to find the words to thank him for letting me know and promptly left the room to try and work out how the hell I was going to tell my wife this over the phone. We'd been in constant communication all day via messenger, since due to the COVID restrictions she wasn't allowed to stay with me. I knew she would be worried sick at home. It was around 7pm and she'd be getting the kids ready for bed.

I finally mustered up the courage to phone her and tell her what had been said. Again, I don't really remember the conversation at all. But an hour or so later she'd managed to arrange childcare and she was on her way over to pick me up. She arrived and I think we were both still in shock. We didn't really say much, other than repeating exactly what I remembered from the doctors conversation. That evening I was sent away with a concoction of drugs to start taking immediately, and then told to return to the hospital tomorrow morning 9am for an Magnetic resonance imaging (MRI) scan.

So what is it?

See, now that's the million dollar (pound) question isn't it. Not all tumors are the same. Some are benign, some are fast growing, slow growing, large, small etc. The CT scan could only show something was there, but not in the detail needed for a more conclusive result of exactly what.

At this point I'm going to skip ahead. I had the MRI scan the next day. Then a few days later another 'wet yourself' inducing CT scan. The full body CT scan was clean, showing that there were no other tumors or areas of concern in my body. Great news!

Meet my tumor

Time to cut to the chase. This is my brain tumor. There are many like it, but this one is mine. I've nicknamed him Gary Glia. Gary because it is a seemingly unpopular name nowadays in the UK (can't imagine why...), so it seemed quite appropriate. And Glia because it is the type of cell in my brain that has gone tumorous.

Gary is an unpaying tenant that has taken up residence in my head. What a prick!

A few days later I had a meeting with the neurosurgeon where I was shown the above images. Now full respect to the neurosurgeon I spoke too, I like him. He was straight down the line and straight to the point. No sugar coating, no forewarning about the size of the tumor. After 30 minutes of chatting about all of the above, he simply pulled his phone out and showed me image number 1 you see above. You could say I wasn't exactly prepared for the image, a understatement if ever there was one. But in all honesty I don't think I ever would be. So it was a good tactic. Make it quick and painless, just like ripping off a band-aid!

So, what do we know about Gary?

He's big... look at the size of this unit. I'm amazed I can even function at all to be honest!
He's of the type called oligodendroglioma and currently believed to be a Grade 2 (slow growing) tumor. Wondering how you pronounce that? Isn't YouTube great!
He's very rare, with only 1,217 people diagnosed with this type in the US every year out of a population of 328 million!
He's a number of years old. The neurosurgeon said a couple of years, but I personally suspect he could be 7 or 8 years old due to other symptoms in my past I now think could be related.
He's a prick.

So I've basically won a really shit version of the lottery... but then again maybe not. It could be so, so much worse! I actually have options that I go into in the next section. There are people out there who are given this devastating news that include words like: 'Stage 4', 'aggressive', 'inoperable location' and are literally given months to live, and there's nothing that can be done. Thankfully, I'm not in that place!

The positives

I don't want to depress everyone, so let's focus on some of the positives I've since found out:

It's believed to be a slow growing tumor, that is currently classed as the less aggressive type (oligodendroglioma rather than anaplastic oligodendriogliomas).
It originated and is primarily located in my brain. I have no other areas in my body showing tumors.
It responds really well to chemotherapy. If I turn out to be 1p/19q Codeleted then chemotherapy is even better.
My symptoms (seizures) can be managed via drugs, which now look to have minimal side effects.

What happens next?

I've had a really great set of meetings with some very knowledgeable folks from Oxford University and the John Radcliffe hospital in Oxford. There are three options available to me:

1 - Do nothing and active monitoring

Basically manage the symptoms and look at how the tumor progresses over time. Go for regular checkups and see how long I hold out.

2 - Biopsy to determine the exact type

At the moment Grade 2 is suspected because the CT scan hasn't shown the usual telltale signs of Grade 3 (which is bright white 'spots' in the scans). Doing this would require brain surgery to remove a small portion of the tumor to be tested. I'd then need to recover from this, before a more specific treatment plan is created.

3 - Remove as much as possible via major surgery

This is the last option, and the one I've decided to go for. This is where the surgeon open's my head and will try to remove as much of the tumor as safely as possible. Once completed, additional surgery may be required to remove any tumor that is left over, or radiotherapy and (or) chemotherapy can be used to finish it off.

This surgery can sometimes take up to 9 hours, and it will most likely involve me being awake for it too (so not under general anaesthetic). I find that part both fascinating and terrifying all at the same time! The reason for this is because they want to be able to ask me questions while they are operating e.g. "move your left hand", "blink your eyes", "speak to us". That way, they can be sure they are minimising the damage to the good parts of my brain. It blows my mind (hah!) that it's possible to even do this, but I'm very glad they can!

But, as with all types of surgery (especially on the brain) there are some inherent risks:

10-15% chance that it could cause a stroke or paralysis
less than 5% chance of a seizure occurring during surgery. This can be very bad indeed, and will result in the operation being stopped immediately
small chance of temporary paralysis on my left hand side
small chance of infection due to the head being open for a number of hours

Decision

As mentioned above, looking at the size of Gary, and weighing up the risks, 3 seems like the only viable option to me that could lead to the best overall outcome. Considering I'm 39 and not in my 70's, compared to the alternative of doing nothing, it feels like a simple decision to make. So that's what I'm doing. I refuse to go gentle into that good night, so it's time to make plans to evict Gary Glia from my head.

It's also worth remembering: neurosurgeons are amazing people and they do this day in, day out, as their job! This is just a walk in the park for them...

Moving forwards

So what happens now? Well I'm now on the waiting list for the surgery, and I have additional monitoring and scans coming up. It's basically a case of waiting for a date and managing the symptoms. If it is that the seizures come back, or anything else gets worse, then things get reassessed by the specialists and the plan changes. I'm continuing to work as much as possible, and spending time with family. For me it's all about keeping my thoughts away from the dark places they could so very easily slip into, which I fully admit they did for the first week after diagnosis.

Support

So here's a section I've compiled for readers who may find themselves in the same position as I'm now in. Thankfully, brain tumors are rare, even more so if of the type oligodendroglioma and anaplastic oligodendroglioma, so there may not be many of you out there. But if there are, here are a few support links to help you out:

As you can see, there's lots of support out there. Macmillan and the Brain Tumour Charity have been fantastic over the past few weeks. The work they do for people needs to be talked about more. They really are unsung heros working behind the scenes to make people's lives better, in some of the most challenging times they will ever face.

Pro tip: Be very very wary of 'Googling' your type of tumour and reading random forums. There's a lot of scary stats and topics out there that may be totally fictional, or only apply for that specific individuals case! A 'random Google' is a dangerous path to take. Trust me I did it and quickly stopped! If you can, stick with approved sources and those recommended by some of the charities listed above. Or speak to the team looking after you for advice.

Practical steps

Now, I realise there are some really terrible topics to have to talk about in the section below, but unfortunately they need to be considered. I've actually found the practical side of dealing with all this really useful. I'm completely powerless in terms of a future date for the surgery and the results of said surgery. But what I can do is make preparations for eventualities that may happen. It may not seem like much, but with the practical side it feels like I'm actually doing something useful. I'm moving the ball ever so slightly forwards.

So here are a few topics you could (should) start to think about:

Admin tasks

Make sure you share passwords with your other half! I recommend setting up a family 1Password account. With a shared vault you can securely share documents, passwords, notes, all in the cloud. No need for unsecure notebooks with written passwords that could be lost or stolen!
Cancel your driving licence and remove yourself from the car insurance. In the UK you won't be allowed to drive if you have a brain tumor / seizures.
Make sure your partner is listed on household bills, and can talk to the companies.
Is your paperwork filed away in a single place and ordered logically?
- Does your partner have easy access to it all?
Do you have a safe? If so, do they know how to get into it? Share the combination / backup key with them.
Start to make a list of what you will need to (eventually) take to the hospital. Maybe even start to pack a bag straight away. It's not much, but at least it's there ready to go when needed. Involve your partner in this planning too.

Finances

Check if your employer offers any pension scheme / death in service benefits. Is your partner listed correctly on these?
Ensure your partner can speak / access bank accounts if required.
Share details with your partner about your life insurance, income protection, mortgage payment protection etc. Maybe even speak to an Independent Financial Adviser (IFAs) about all this to check it is all in order.
Are there any extra benefits you are now eligible for e.g. Personal Independence Payment (PIP)? The charities listed above can help you with advice on this.
Consider setting up a joint account with your partner, and have shared money "just in case".
Create a spreadsheet with information about your household utility providers e.g. who's your electricity provider. Include account numbers, contact numbers, cost per month, payment method etc
- On the same sheet you could include details about finances (Credit Cards, loans, higher purchases etc)
- Include details about pensions, banks, building societies you are with including account numbers and contact details.
- Remember to securely store this information (see the 1Password recommendation above)
If in the UK you will now most likely be eligible for free prescriptions on the NHS. You can apply for this through your local doctors surgery. So make sure you do this, as you'll likely going to need a fair few drugs over the coming months / years!

Documents

Make sure you have a last will and testament prepared. Make sure executor's are informed and listed correctly.
Ensure you have your Lasting Power of Attorney in place with your partner listed, as well as backups. You can 'do it yourself', or speak to an specialist advisor for this. I opted for an advisor, I don't have the mental capacity or time at the moment to do it manually.
Make a list of questions to ask nurses / doctors etc. I've shared the list I made here that you can copy if you are struggling.

Above all else remember, you can make plans and be prepared, but even the most well-made plans can go pear-shaped quickly. It simply isn't possible to pre-think and pre-plan everything for every eventuality. So there's only so much you can do. But much that is listed above could be considered "good practice" for any adult anyway, even if you aren't in the same situation I'm in.

Plans for this blog post

So what are the plans for this blog post? Well, I plan to keep this post updated with my progress for those who are interested. This will be the only blog post where I mention Gary the prick. Basically it will be a self-contained blog post about a topic I never expected to be living, let alone writing about! And as with my web performance blog posts, I'll try to keep them as 'living documents' that are constantly updated as things change. I will continue to use the 'Post changelog' below in exactly the same way.

Thanks

So I have a huge number of folks to thank who have been amazing over the past few weeks:

My wife, who has been so incredibly brave throughout this. Immediately looking for information and support that is out there (both emotional and practical). She amazes me every day.
My parents, sister, brother-in-law and immediate both sides of the family who have rallied around to help us in whatever way they can. At the moment COVID restrictions make this hard, but this should become easier in the UK very soon. I can't thank them enough.
Line managers from GDS, both past and present, who have helped ease my mind when it comes to the logistics of work in the future.
The Frontend community at GDS. I feel so lucky to be able to head up and work with such an amazing group of talented and dedicated individuals. They inspire me every day.
A certain ex-GDS colleague who shall remain nameless. They have been a fountain of knowledge in so many different areas that, in all honesty, I didn't even realise existed 3 weeks ago!
And last, but certainly not least, the wonderful paramedics, doctors, nurses, and all workers of the National Health Service. The quality of care I've received so far has been truly outstanding. It's all the more impressive considering the huge strain COVID is placing on the whole system at the moment. They really are heros!

And finally, thank you reader for taking the time to read the post. It wasn't at all easy to write, but I feel so much better for actually taking the time to do it. Hopefully some of you find it useful. If helps even just a single person get through such a monumental, life changing event such as this, then it was totally worth my time writing it.

Updates

Another seizure - Monday 10th May

On Monday night I had a really 'fuzzy' head. That's the best way I can describe it. It was almost like I'd taken some sort of sleeping tablet. I also had a 'prickly' head and scalp, quite sensitive to touch. I went to bed early but woke up at 2:30am with the same OBE I'd experienced before. I think I had another seizure in my sleep. I can't be 100% sure as I fell to sleep again right after it. I just remember waking up to the same feeling.

On speaking to the team of specialists they said it's most likely from lack of sleep, my body adjusting to the drugs and general stress with everything going on at the moment. They aren't too concerned and my medication will be staying at the same level for the moment. If it gets worse in the coming weeks they will look to up the amount of anti-seizure medication.

I just need to work out what the triggers are, and the warning signs of one occurring. It's a whole new world for me, so I don't recognise the signs yet. You can live 39 years in a body, and still have no idea what is going on with it...

Messages of support - Thursday 13th May

I've been absolutely blown away by the huge number of messages of support I've had since writing this post. I posted it on Twitter and on LinkedIn and have received literally hundreds of messages. I've tried to respond to as many as I can, but I may not have seen them all. So sorry if I haven't replied! Just to say thank you to everyone who has reached out. I really do appreciate the support. It's all been quite overwhelming, and will admit to welling up a few times from the sheer number of them. What a crazy couple of days.

Neurosurgery pre-assessment clinic - Saturday 15th May

On the 20th May I have another appointment at the hospital. This is the pre-assessment for the surgery. I'm not 100% sure what is involved, but I'm going there with a list of questions I'm hoping to get answered. On speaking to a senior member of the team during the week the timeline for surgery has been revised down from 3 months to 4-6 weeks (assuming other higher priority cases don't bump me down the list). This is good news, but also makes me a little anxious! I was mentally preparing myself for having 3 months to get used to the idea of awake open-head surgery. 4-6 weeks seems way too soon! But it is what it is. I'll get used to it, it's not like I really have much of an option...

Yet another seizure - Wednesday 19th May

A short update. I had a very short 'mini' seizure in the morning just after waking up, around 6:10am. I was feeling perfectly fine, then suddenly my face felt quite warm. I could feel something was about to happen. I had the feeling of losing control of my face again (drooping), then started rocking gently backwards and forwards in my seat as has happened before. I was fully aware and conscious during this time. Around 10 seconds later I was back in control. It was nowhere near as intense as some of the other seizures I've had, but it's still a seizure. So it's important to keep the specialists informed. I believe we may look into upping the dosage of the Keppra drug I am on in the coming days.

Pre-operative assessment appointment - Thursday 20th May

So today was an interesting day. I learnt a lot about the upcoming treatment and the weeks ahead. The more information I have at the better, as planning anything is virtually impossible at the moment! I had a 2.5 hour meeting with a friendly senior nurse on the team looking after me. She answered all my questions and gave me a rundown of the coming weeks, months, and years. I've tried to break what was said down into sections following a chronological order of future events. Ultimately the reason for the consultation was to prepare me for surgery in the future, which is a good thing. Although I'm still very nervous about the thought of a whole team poking around in my head, the information given to me today has really helped me understand it all.

Future seizures

I need to start logging any previous and future seizures, this is very important. Ultimately the plan is to achieve a sustained period where I don't have any seizures, while keeping me on the minimal amount of drugs possible (due to drug side effects). There's a fine balance to achieve this, and we aren't there yet. I seem to be getting one seizure a week at the moment. I'd say my drugs will be increased in the coming days. I've also been asked to try to speak the next time a seizure occurs. The team has questions like: "Am I able to talk at all?", "Does it cause slurred speech?", "Are the words I want to say totally different to what comes out of my mouth?". The team are interested to know how a seizure affects my speech, since this can give an insight into the parts of the brain being affected by the tumor. So that's on my to-do list for next time!

Neuropsychology assessment

I'm scheduled to have a neuropsychology assessment in the coming weeks. I didn't realise how important this actually was until today. It's going to be a long process, possibly a 3 hour+ meeting where members of the team will conduct many memory and cognitive tests on me. Ultimately, this testing will be used as to set a 'baseline' for various aspects of my current brain function. This baseline will then be used both during and after the surgery.

It was explained to me that during surgery they will be constantly asking me questions and asking me to move parts of my body. While this is happening they will be manipulating and testing neurons in my brain to see what effect it has on my cognitive ability. Only once the surgeon is 100% happy that said part of the brain is safe to remove, will it actually be removed. As you can imagine this is going to be a long process with a tumor the size of mine! The baseline will also be used to compare brain and memory function during the recovery period. Without this baseline it would be extremely difficult, if not impossible, to measure the effect the surgery has had on me.

Only after this baseline has been established will my surgery be scheduled. That's how important this assessment is!

Pre-surgery preparation

I've been given a set of items to use the night before surgery.

Antibacterial body wash
Antibacterial cream (for my nose to combat MRSA)
Antibacterial mouthwash

Basically I need to be as germ free as possible before having surgery. I will be using all three of these the day before and on the morning of the surgery. Body wash all over, leave for 1 minute then rinse. Rub the cream up my nose 3 times throughout the day and in the morning. Mouthwash 3 times the day before, and again in the morning of surgery. Doesn't sound pleasant, but if it means there's more chance of success I'm more than happy to do it!

The surgery

Now this is where it all starts to get very interesting, as the process was explained to me in detail so that I know what to expect on the day. I will likely be told on a random Tuesday in the future that I've been scheduled for surgery on the following Friday. So I'll only get 3 days notice! That's going to be an interesting week! I'll have a COVID test 2 days before, then on the Thursday I will go into the ward and prepare for surgery the following day.

I will have another MRI scan and a brain mapping scan. The brain mapping scan is used to map the coordinates and direction of travel which the surgeon and team will enter my skull. All this is planned to the smallest detail. Nothing like Dr. Nick off The Simpsons (thankfully!).

The surgery is likely to last 5 hours, 3 of those with me being awake. They will put me to sleep for the first hour to prepare me since it sounds a little unpleasant. I will be attached to a metal frame via a set of small screws that are drilled into my skull. I will then be woken up and won't be able to move my head. My head will already be open. Upon waking I will be surrounded by a team of people and won't be able to see or feel anything that is happening behind me. The team will continue to ask me questions and to do things while they operate and slowly remove the tumor (using the methodology mentioned earlier). Once they are happy, they will again put me to sleep for an hour, patch me back up and I will be sent to recovery. It all sounds so simple doesn't it!

Recovery

Assuming the surgery goes well I will be admitted to the recovery ward for around 24 hours. I'll have a tube coming from my head to allow fluid from the wound to drain safely away, as well as various other pipes attached to my body. The aim is to get me up and mobile (at least a little) within the first 24 hours. This is to reduce the chance of blood clots, as there's a raised chance these can happen for up to 3 weeks after surgery. If all goes well and I don't show signs of seizures, I will be sent home. How crazy is that, major brain surgery to home within 24 hours! If it is that there are complications, I'm unable to move etc, I will be kept in for longer. It's all a complete unknown at the moment.

Once Gary the prick has been extracted, he will be tested to see what exactly he is (or was), and this data will be used to inform my ongoing treatment plan. Genetic marker testing (e.g for the 1p/19q Codeleted gene) will also be conducted, and this data will also be used to plan the best treatment moving forwards. The 'default' treatment for this type of tumor is surgery, so it's likely I will have more head screwing fun in the future! Radiotherapy can be used once, and chemotherapy can be used multiple times to clear Gary's leftovers should it be needed. We shall only know more after the messy uninvited squatter has been forcibly removed.

Back to 'normal' and work

I'm likely to need a minimum of 6-8 weeks off work. In this time I will likely exhibit changes in mood, behavior, concentration, and personality. This is all perfectly normal due to the fact that my brain will be adjusting both physically and chemically to the tumor not being there anymore. If I require further surgery and/or treatment like chemotherapy/radiotherapy, I may need a little longer to recover. Again this won't be known until after surgery.

The key is to take this slowly. I already know this is going to be an incredibly frustrating time for me. I know myself. I'm not one for sitting around doing very little. I'm always on the go and doing something. But I'm going to have to live with it for a few months. Maybe I can start watching all of the box sets & movies I've always wanted to watch! Or start tackling the tonne of books I've never read over the years. I mean after all, it sounds better than the alternative!

A bumpy weekend - Saturday 22th - Sunday 23rd May

So it was a bit of a bumpy weekend in terms of how I've been feeling. I'm now on anti-seizure medication that I will be taking for the foreseeable future, maybe even forever depending on how surgery goes.

Saturday

Day went well until about 10:30am, suddenly I felt extremely tired, sat on the sofa and promptly fell to sleep. Woke up 10 mins later to find my 4 year old son had covered me in cushions and was laughing his little head off as I woke up. I was completely and utterly confused as to what had happened. The medication I'm on mentions drowsiness / tiredness, so I put it down to that. I didn't feel 100% all day, generally quite weak. I had the same feeling around 6:30pm where I could barely lift my arms and just wanted to sleep (this is not at all like me, as I generally hardly sleep!). This feeling passed after an hour or so, after a little food.

Sunday

Went to the park with my wife and children, no issues at all. Around 1pm I had a sudden feeling of weakness. Went to the living room, sat on the sofa, promptly fell to sleep. My 4 year old son is very opportunistic, again covered me in cushions, again in hysterics when I woke up 20 mins later. He's going to be a fun teenager...

Seizure in the kitchen - Monday 24th May

Due to everything that happened over the weekend, which we assumed were the side effects of the drugs I contacted the team of specialists looking after me early in the morning via email. I knew because of this they would be in contact with me later in the day.

Around 7:30am I was standing in the kitchen having just prepped the kids breakfast. I suddenly had what I can only describe as an emotionally driven feeling that something was about to happen. I could feel something building. I immediately recognised something wasn't right and sat down next to my wife at the kitchen table. She could see something wasn't right either, so stood up next to me and so as to shield me from the children. I suddenly couldn't really control my movements and started gently rocking backwards and forwards. I was conscious and remembered that I'd been asked to speak the next time this happened. I said out loud "Can you hear me?" to my wife. It sounded perfectly clear in my head, but to my wife it sounded slightly slurred. She could still make out what I was trying to say though. Apparently while this was happening my eyes were very wide, but that could really have just been panic on my part. It all passed so quickly as they always seem to do.

Going back to the emotion I felt. I'm not sure if this was something triggered by my subconscious, as in "bubbling up" to tell me "stuff is going on here, get ready for it", or if it was me thinking about a seizure and then this triggered it because I thought of it. I'm thinking the former to be honest.

On speaking to the specialist nurse she seemed to think that the episodes that occurred over the weekend weren't side effects of the drugs. These were actually a different type of seizure that were happening over a longer period of time. Well at least now I know! Because of this my anti-seizure medication has been increased, and will be increased again in a week. The aim is to have a long period of time between each, or even stop them completely before surgery. So we shall see over the next week if this increase actually works.

Obtained my full scan data - Tuesday 25th May

When in one of my follow-up meetings (20th May) with a senior nurse I happened to ask the question if I could have a copy of the original photos from the scan. All I had at the time was a screenshot from a video call that was shown to me off the neurosurgeons smartphone (what a crazy sentence, isn't modern technology amazing!). I was told to visit the hospitals radiology department, fill out a form at reception and it could all be arranged. On the form the interesting part for me was the following section:

The top section allows you to choose how they send you the data. For a patient it was either as an online link or as an encrypted CD. It shows you how old the form is, no mention of USB sticks, or even DVD media! I chose the option where an online link is sent to me via email.

It's also worth noting the sentence below this box:

The patient is entitled to a copy of all the Radiology imaging we retain at this trust. However if specific episodes are required, please specify below.

So depending on your local NHS trust, you should be able to do this yourself (should you be interested).

I received an automated email saying this could take up to 28 days, but 2 days later I had an email from a member of the team saying that my scans were scheduled for the overnight upload and I'd receive the link the following day. Upon receiving the link they sent me a one time password (OTP) via my mobile number that allowed me to login. The web performance of the system was shocking. Took around 20 seconds to load on a 50 Mbit connection and a modern browser! Thank you JavaScript...

Once I clicked on the download button I realised this wasn't just a set of high resolution images, as it was a zip file for 1.3 GB of data! Intriguing! It took me a number of attempts to actually download the data, and I had close friends try too. Their server connection was slow and unreliable, and it also doesn't seem to support HTTP range requests, so it wasn't possible to resume a download after already managing to grab 880 MB of it! Best thing to do I found was to wait until later in the evening. I managed to get a download speed of around 2.7 MB per second, so it only took around 15 minutes in the end.

The reason the zip file is so huge is because it is sent in a standardized format for use in specialist tools that can interpret the data. Here's me thinking it would be just a few images... nope! It's literally all the images from all scans completed so far! Hence the box on the form stating which scans you require.

I believe there are around 4,000-5,000 images in total from 1 CT and 1 MRI scan! Thankfully there are tools that you can use to view these images. And not just view them, they allow you to navigate through all the scans. Each image is essentially a "slice" of the brain. The tools allow you to move through all 3 axis in near real time, with each of the corresponding panels changing depending on the input from the others.

In the above image you can see my MRI scan. It's essentially an orthogonal view of my head, with each of the panels represents a different axis:

Left panel: Top of my head (Axial view)
Top right: Front to back of my head (Coronal view)
Bottom left: Side of my head (Sagittal view)

The solid blue line you see in the views is the current plane I have selected to see. As you move this plane, all three images change accordingly. Notice how the blue line on the axial view (left panel) matches that of the coronal view (top right) in terms of position. The resulting image in the sagittal view (bottom left) is the image that corresponds with this selection I have made.

It's so incredibly clever and absolutely fascinating (to me at least)! I spent the best part of 2-3 hours looking at my own brain scans. I wish I knew what it all meant, but I plan on quizzing the experts when I get the chance. I'd like to write a blog post all about it in the future. So, watch this space!

For those interested in doing the same the tools I used are:

Horos takes it a step further, as it allows you to view your scan in 3D, and even export an animation to show all your friends at the next soirée you attend! So yes, if you ever fancy looking at your own insides in detail, remember you can request your CT and MRI scan data!

Cloudy head seizures - Friday 28th May

So I think I've started to recognise the signs of having a seizure. But this isn't an obvious type of seizure that started this whole saga off 5 weeks ago. This looks to be a different type that I touched on in a previous update. In my 'Seizure spreadsheet log' I've written:

Weakness in arms, "cloudy" thinking. Feel it around my face, tingling in forehead, almost throbbing sensation. Sleepy. Hard to keep my eyes open. Muscles changing in my face (odd tightening up from the middle). Feeling coming on for a number of hours. Feels like it is building up to a seizure. Brain feels "heavy". Went to sleep for 40 mins, felt like something 'lifted', may have had another seizure in my sleep. Had the same out of body experience while sleeping.

It's hard to say, but I think I may have had another more obvious seizure while resting. I remember it happening so I did try to move my arms and they seemed fine. So who knows! Urghhh! But after my 'sleep' for 40 minutes or so I felt a whole lot better.

Also, I've noticed my left foot (heel) has started to hurt, which is one of my initial symptoms that I was blaming on sciatica. So I'm going to be mentioning that at my next catchup with the team.

Date for the memory test - Tuesday 1st June

So I've now been given a date for the baseline memory test. This is happening on the 29th June, a whole month away. Not ideal, and I'm a little torn with the date. In some ways I want it sooner to "get it out the way", but in other ways I want it later. At least when it's later it gives me more time to 'prepare'. I still feel I have so much general life administration left to complete before the 'big day'.

The symptoms with my left foot still persist but tend to vary day to day. Because of this I may need to be given different drugs and / or have additional scans to see what Gary is up to. I'd like to think he hasn't gotten even bigger in the last month, but it wouldn't surprise me!

So now I have the date. I'm guessing the operation date will be 1-2 weeks after this. So around mid-July... or in other words around the time of the F1 at Silverstone... Damn you Gary!

Back on the steroids - Thursday 3rd June

Unfortunately over the past 2-3 days the symptoms in my legs and feet have returned. I was hoping that the anti-seizure drugs were the ones keeping these at bay, but it turns out it must have been the steroids which I came off a number of weeks ago. The team have decided to put me back on a short dose so as to improve my mobility (which I'm very grateful for!). I'll be halving the dosage in a week, then likely coming off them again the week after. Since it takes a while for them to leave your system I'm hoping that takes me up very close to a surgery date, at which point the root cause of the issue can be ejected from one's skull!

Reduced seizures and steroid side effects - Monday 7th June

So the good news, the seizures seem to have reduced a lot over the past few days. I've had very little to write in my seizure log for the team. I like this direction of travel. Unfortunately as happened before the steroids have decimated my sleeping pattern. I'm currently writing this update at 00:40 and don't feel tired at all. And yesterday I went to bed / sleep at around 3am, then woke up at 6:50am. Apparently under 4 hours of sleep a night isn't the ideal amount you should be having! Oh well, at least I'm getting lots more stuff done!

Neurologist appointment - Thursday 10th June

Due to the seizures I've been having I was referred to the Neurologist who is part of the multidisciplinary team who are looking after me. It was a somewhat confusing meeting, but also very interesting that lasted around 40 minutes.

From previous conversations I've had with doctors, I was under the impression that the tumor was likely causing the issues I'd seen with walking in October (left back, leg, foot pain etc). But upon talking to the Neurologist and describing my symptoms, they seem to think that isn't the case and it's all purely coincidental. Confusing! I'd prefer it to be the tumor causing it to be honest. Having 1 major brain operation and solving 2 problems, sounds like a good deal! But alas, we shall see when Gary finally buggers off into the hospital incinerator. Only then shall we know for sure!

What was interesting for me was the conversation around classifying my seizures. I've been keeping a seizure log (yet another fun Google spreadsheet!). All fairly standard columns:

Date
Time
What was I doing
(Copious) Notes

Now I'd already started to notice this myself: not all my seizures are alike, and since taking the anti-seizure medication I'd started having different symptoms. The Neurologist confirmed this today. So I've now added another column:

Is it a seizure involving the ‘core features'

A convoluted column title I know, but I'll explain what It means. The ‘core features' are effectively the most obvious (and serious symptoms). For me I've identified them as:

The feeling of being ‘outside my own head'
The emotionally driven feeling of everything being bigger than my own self (existentialism)
Any loss of motion and control over my body
Loss of consciousness (not happened as of yet thankfully!)

Basically if I have a seizure that displays these core features, then it is logged as one. The other seizures I'm having are still logged in detail, but not listed as a ‘core feature' seizure.

So why is this important? Well, without this classification every little seizure suddenly contributes to what looks like an escalating number of seizures. If this is the case then getting the balance for medication is difficult. I want to be on enough to stop the ‘core feature' seizures, but not so much that I have other side effects. This classification allows me to pull out the really important ones and therefore allow the team looking after me to see if they indeed are escalating, or if they are being managed.

So there we go, top tip if you have a brain tumor. Keep a seizure log and look for patterns in what happens when they occur!

A 'core feature' seizure - Friday 11th June

After the call yesterday it was all going so well. Nothing I would class as a 'core feature' seizure for the best part of 2 weeks. Unfortunately, one happened this evening at about 10 past 9pm. Watching the TV sitting on the floor, my wife just paused the TV to get something from the kitchen then I felt it. Again like an emotional response. Something was about to happen. I could feel it coming on. I tried to turn around and 'shake myself' out of it and convince myself I could stop it from happening. But within 5 seconds I'd lost control of my upper body and was shaking gently back and forth, unable to stop it.

As I've been told to see if my speech was affected I made sure my wife could hear me this time. She had walked in the room and not noticed what was happening so played the TV. I just started saying "Pause it", "Pause it" "Pause it" over and over again, then eventually I said "I'm having a seizure". Now I could have stopped myself from speaking, but I wanted to make sure we knew what the effect of it was on my speech, so I continued talking. It turns out she could hear me fine, no slurred speech. Said I sounded quite normal. I guess that's a good thing? In total from the feeling to the end I'd guess it lasted about 30-40 seconds. Even 2 hours after it I still feel weak across the back of my shoulder blades and upper arms.

I'm hoping it was due to it being a long and busy week, maybe just tired. Or the fact that I'm now weaning myself off the steroids on the request of the Neurologist. I'm not sure! But to be honest, if I only get them once every 2 weeks, I'll take that at the moment!

A sleepy seizure - Thursday 17th June

All has been quite quiet on the seizure front for the past few days. I've been gradually feeling weaker as the steroids pass through my system and my body takes over producing whatever it needs to produce... (can you tell I'm an expert in this doctor stuff!)

I've been getting very tired, very quickly. One moment I'm fine, 5 mins later I can hardly keep my eyes open. This happened around 5pm, just after having dinner with the kids. So I needed to go sit down in the living room. Within a few minutes I'd fallen asleep. While sleeping I'm 99% sure I had a seizure. Unfortunately due to the fact that I was asleep I'm struggling to classify it, as I'm unsure if I lost control of my arms etc, since I wasn't moving.

The reason I believe I had one was because I remember coming too, and it felt like my brain split in two. Suddenly a pressure was released and my head felt ‘open'. I didn't even realise I had that pressure there; it was only once it had been released I noticed the difference. I woke up a few minutes later and felt so much better, so maybe that's how it works? The ‘pressure' builds in my head and I don't feel better until it is released. I'm not sure, I'm still getting used to the whole having seizures, and what triggers them.

An epiphany - Friday 18th June

It's hard to stop yourself thinking about everything that is going on. It's one of the reasons I enjoy continuing to work. I get to escape from thinking about the current situation and feel ‘normal' for a short period of time. Well, at least until I remember what is happening again…

So on Friday I had an epiphany about the whole situation. I came to the realisation that I'm dying. Now I know what you're thinking… "Damn, that got dark very quickly!" I came to the conclusion via a little thought experiment.

Imagine the wonders of modern medicine didn't exist. No drugs, no ability to operate and remove Gary. What would be happening to me in this situation? The reason I know about any of this at all is because the tumor has got to a sufficient size to cause the seizures, so it must have crossed some tipping point recently. The tumor would be getting bigger, the number of seizures I'd be having would be ever increasing. I believe the tumor would just be getting bigger and bigger, putting more and more pressure on my brain until it eventually caused so many issues that I passed away. I'm actually on a path in my life where this would be happening to me, but thankfully I have options to try to change this path.

I like to think of all this as water slowly filling a bucket. If the bucket overflows, bad things happen. At the moment the drugs have created a hole in the bucket to help stop it overflowing, but eventually the bucket will overflow, even with the hole. The only way to stop it from overflowing is to stop the water, and that basically means removing Gary. So thank goodness for modern medicine, else there'd be overflowing buckets all over the place!

Well that was an uplifting read wasn't it? Thanks for reading, have a great rest of the day! :)

A very good roast dinner - Sunday 20th June

Due to covid I'd not been to see my sisters new house. Not so new anymore since she moved in in February 2020! With restrictions gradually lifting and testing readily available we'd decided to go visit and have a Sunday dinner. The kids loved the adventure, investigating the house and going to visit a new park just 5 mins away.

We ate around midday. It had been a while since having roast dinner, especially since it was beef, so I may have eaten a little too much too quickly. About 5 minutes after finishing while just chatting at the table I suddenly felt incredibly tired. I simply couldn't keep my eyes open. My arms felt heavy and I found I couldn't concentrate on the flow of the conversation. There was no immediate seizure, but it felt like something was happening. Thankfully my sister had a spare room with a bed so I decided to go lie down to try to sleep it off.

I woke up around 1.5 hours later feeling much better. As I've mentioned a couple of times before, I may have had a seizure in my sleep. The ‘pressure' in my head seemed to lift it one point while sleeping, I remember it happening. It isn't really ‘pressure' as such, but it's hard to describe. Something ‘lifted' and I suddenly felt much better.

So there you have it. My sister now has quite a claim to fame. Her roast dinners are so good they gave her brother a seizure!

The baseline memory test - Tuesday 29th June

At the beginning of the month I was told a date of the 29th of June for my memory test, so the whole month we've been waiting for this day. I wouldn't say I'd been nervous. Really just anxious to get it ‘out the way'. I had no idea what was involved. All I knew was it was quite intensive and it would last a few hours. Thankfully you can't revise for a memory test, as it did feel like being back at school waiting for an exam. The testing involved a chat with the neuropsychologist for 1 hour and then some intensive testing that lasted 2 hours.

The first part was easy. I enjoyed chatting about the situation to test myself and see what I could remember. And also to verbalise my understanding of the situation. Now I'm pretty sure the doctor had read all my medical notes, but there were points in the chat where I felt she was feigning ignorance about certain details. I'm guessing this is a tactic used to try to get me to explain my thoughts and feelings so they gather a greater understanding of my current mood and attitude to the whole situation. Not that I mind at all, it's good to talk about all this. I often find I don't know how I'm feeling until I try to explain it to another person.

The actual testing was really interesting. I was given around 12-14 different tests over a 2 hour period. Some were more difficult than others. There were tests that involved pronouncing words, and also remembering words. The tests always started very easy, then gradually got harder. So you'd start off thinking how smart you were, then by the end think the complete opposite! But I guess that's the point when trying to look for a baseline result, each and every person doing the tests will find some more difficult than others.

I personally found one set of tests quite difficult. It involved thinking of as many words starting with a particular letter in 60 seconds. The only rules were they couldn't be a person's name or places. Any other word in the english language starting with that letter. There are that many surely it can't be that hard! Well apparently not. For some reason my brain was insistent on thinking of swear words starting with whatever letter I was given. So each and every time I'd think to myself, "No I can't say that word, it's rude…". Once you get into that little game with your brain ("say that, it would be funny!"), It's hard to stop! Thanks brain! I think I could only think of between 10-15 words in 60 seconds. Not many at all so I'd say I'm below average for that test!

The easiest tests I found were the ones involving patterns. Look at this sequence of colours and shapes then pick the next one in the sequence from a list. I think I did okay in those tests, but who knows! Maybe I'm below average for those too.

The testing was intentionally hard as this is to try to gauge where you're at and set that baseline. After surgery they can then repeat the test and look at the difference between the two sets of results. I guess since there were lots of different types of tests, if they see a drop in the results for one type of test, it suggests a change in a certain area of the brain. It's all very clever!

Now with this date out the way, it's just waiting on a date for surgery. All I know is that it is likely to be at some point in July. So July is going to be an interesting month!

Shielding - Sunday 4th July

Unfortunately there's been a number of positive covid cases at my children's school. Year 2 and year 5 have been isolating at home for 10 days. We've been testing every day, and have been for a drive-through PCR test. Thankfully they all came back negative!

With both children back at school, and covid literally so close (there are a number of schools in Oxfordshire having similar issues), my wife and I decided that it would be best if I were to shield. It's pretty crappy to have to avoid your own children, but having waited 2.5 months for a surgery date I really don't want to lose it because I catch covid. That would be bad!

So I'm now staying with my parents. It's the first time I've lived with them in 21 years! I'm still working and plan to do so, up until I get a surgery date. This will happen on a Tuesday at some point in July. Every Tuesday they triage the patient list and make plans for the week. So I'll receive a phone call saying surgery will be on Friday! You don't get much notice, only a couple of days. Although that's probably a good thing. I don't want to be dwelling on the fact that they are going to be removing part of my brain in a couple of days for too long!

When folks have asked me how I'm feeling about it all, It's hard to describe. In some ways I want the surgery to happen right away to at least "get it out the way". But in others I don't want it to happen at all, as it really is a watershed moment in my life. There's always going to be a ‘pre-surgery' and ‘post-surgery' me from beyond that point. Fingers crossed those two versions of me don't change much!

I'm siding on the fact that I actually want it done now. My family and I have been in limbo for a few months now, so it would be good to be able to move forwards and start to actually plan things! I've gained so much respect for people who are on transplant lists for years. Waiting for hearts, kidneys, livers etc. I've only been waiting 2.5 months and it has felt like an eternity! I can't even imagine living your life waiting years for surgery. Not being able to properly plan anything because "what if we get the call?".

My next update will likely be when I have an actual surgery date. Not too long I hope!

No news - Tuesday 6th July

So having waited around all day for a call from the hospital, it unfortunately never came, so no surgery this week. Annoying but at least it gives me a little more time to get a few more things done! Let's see what next Tuesday brings!

In other news, there's been a few more reports of covid in schools in Oxfordshire so I still feel sheilding was the right thing to do!

Still no news - Tuesday 13th July

Mid-way through July now and still no news about surgery. I was hoping to have it at the start of the month so I could at least start the recovery process and maybe have the latter end of the summer to do something "normal", (whatever that means in 2021!).

Being on the waiting list for hospital appointments is pretty stressful at the moment as covid cases are surging, I really feel for the folks working in the NHS at the moment. It must feel like a never-ending tsunami of stress and unpredictability.

Well at least I've been given a date of "July", that's a lot more than some people have! I've a sneaking feeling it will be right at the end. Fingers crossed the next few weeks go quickly. Although, saying that, these past few months have been the longest of my life!

Between Life and Death - Thursday 15th July

Earlier in the week I discovered there'd been a short series on Channel 4 here in the UK called "Brain Surgeons: Between Life and Death". Reading the synopsis it all sounds very interesting. I thought it would give me some insight into what to expect on the day. Unfortunately I wasn't mentally prepared to watch it. I made about 5 mins into the program and had to stop. It hadn't even got to any of the surgery yet and was just talking about the people and their families. I think that brought it all home how serious this all is, not just for me but for everyone around me. What finished me off was seeing the guy lie down on the operating table and realising in only a few weeks time I'm going to be doing exactly the same. It's not a pleasant position to be in.

I mean let's be honest here, the clue is in the title: "Between Life and Death". It probably isn't the best thing to be watching weeks before major brain surgery! Saying that, I do plan on watching it post-surgery. I'll have a very unique perspective of what it's like from ‘inside' the process. I can then compare notes from ‘outside' the process too.

Close but no brain surgery - Friday 23rd July

So this week has been quite a rollercoaster. For the past 3 weeks I've been mentally preparing for a date of the 30th July for surgery. Now, I realise none of this is set in stone, but as that's the only date you're given it tends to be the one you focus on. Unfortunately I had an update at the start of the week stating that the date was going to have to be pushed back. It sounds like I was actually quite close, but a few emergency cases came in that took priority. So I'm back to the 2 week holding pattern (which really sucks!).

Saying that, I was given some interesting information during the conversations I've had with the team. Another MRI scan has been requested as I'm coming up to the 3 month point since diagnosis. But this isn't your usual MRI scan. It's actually being pitched as the pre-surgery scan. The neurosurgeon (and team) would like to understand what Gary has been upto over the past 3 months. From what I understand, this will be a more detailed scan that can be used for surgery planning. I don't know if this means it will take longer, or if they just conduct more scans / more detailed analysis(?) I guess a question for the test operator when I meet them soon.

I was also given information that the operation is likely going to take longer than expected. The team usually conducts 2 operations on the Friday of each week. But the neurosurgeon is now considering only operating on me on the day, due to the size of Gary. I've yet to decide if this is a good or bad thing? Good in the fact that they can take their time to extract "the prick", but bad in the fact that it is needed (and I'll be awake for most of it!).

Fingers crossed the 6th August materialises. Folks have said to me that the waiting is the worst part. I've no way of knowing that at the moment, but I'm happy to agree. It is pretty bad! Let's see what next week brings I guess!

Pre-surgery MRI scan - Monday 26th July

Today I had my pre-surgery MRI scan. Now that suggests that I have a date for surgery, but alas no, not yet. I'm supposed to have an updated scan every 3 months to look at what Gary has been up to, so instead of having 2 scans in a short period of time, the standard scan has been upgraded to a pre-surgery scan. This means that the scan takes a little longer than usual (around 40 mins). They take more detailed scans and I assume maybe a few more types? This means there's no ‘blocker' for surgery now. When I get a slot, I'm not waiting on an MRI scan. This is good news!

The scan itself was fairly easy. There was a slight miscommunication so I was waiting around 30 mins longer than needed, but compared to how long I've been waiting for surgery it's nothing. At least it is done! 40 minutes stuck in a cold, confined tube that screams at you like a ZX Spectrum isn't the most relaxing experience. At 35 mins in they came in and injected the contrast into my arm via the cannula (I hate cannulas, so uncomfortable!). It was cold as it ran up my arm. A few seconds later I could swear I had a mini-seizure. I had the ‘emotional' response I'm getting used to, but since my head was tightly strapped to the MRI table and I was lying down in a confined tube, I'm not sure if it affected my movement. I'm going to raise this with the next doctor I speak to. Maybe it's a known side effect?

As another Tuesday rolls around, here's hoping I get any news on a date for surgery. The wait is really starting to get me down.

One confusing week! - Sunday 1st August

The last week has all been a little crazy. On Monday I had my 3 month MRI scan, which was also the pre-surgery MRI scan too. I've mostly been on annual leave from work as there's just so much going on it's hard to concentrate on anything else.

On Tuesday I was back to work for the day, but something unusual happened. I was in a meeting with my incredibly supportive line manager, but 30 mins into our 45 minute video call I had to stop quite abruptly. I completely lost my train of thought and felt like I was on the cusp of having a seizure. Thankfully I can now recognise the telltale signs of this happening. This is a worrying sign as work has been a haven for me over the past 3 months. I've been able to zone out of "real life" for a few hours and concentrate on work, completely oblivious to the fact that Gary even exists. For a few hours at least! It now seems that even this little haven is eroding.

On Wednesday I had a call from the hospital with the results of my Monday scan. It's a bit of a mixed bag. The good news is that the tumor hasn't grown in the past 3 months. The less good news is that there's still no date for surgery. The bad news is that some swelling can be seen around the tumor, also known as cerebral edema. I think this explains the symptoms I saw on Tuesday.

Thursday and Friday were a bit of a writeoff to be honest. All day I wasn't feeling 100%. Incredibly sleepy, dizzy and at times quite confused. It's strange, and it comes on so quickly. One minute you feel fine, next minute it all changes for no apparent reason. Thursday evening was particularly bad, where I could barely keep my eyes open or even really speak to my wife. I was completely disconnected from reality and didn't have the energy to even try to re-engage.

On Friday evening it got so bad that my wife decided to ask for advice from the 111 team (non-medical emergency number). We went through the triage process and eventually the very nice on-call GP was sent out to see me. She was satisfied I didn't need to go to hospital but did want to phone the specialists in the oncology and neurology departments. On speaking to the on-call neurology consultant who looked at my scans they recommended that I should be back on steroids to reduce the swelling in my brain.

Now here's the difficult part. The team I've been under the care of for the past 3 months have been very keen to take me off steroids when I've been on them, since they are quite strong and have some nasty side effects. Steroids take around 2 weeks to leave your system once you take them. So I've been stuck between a rock and a hard place. Take the steroids and I could potentially compromise a future surgery date. Don't take them and my symptoms could get worse.

Weighing up these options, it was 100% my decision NOT to take the steroids until I've been given the all-clear from the team looking after me at the hospital. There's no way I want to lose the opportunity of surgery because I took a drug that is now in my system for weeks. So due to the fact that I was having unpredictable symptoms from the swelling, I've moved back with my parents for a few days. At least they can easily keep an eye on me without having to keep two young children entertained.

So there you have it, quite an eventful week! I'm seriously hoping that the upcoming week is the one where I'm given a surgery date. I don't think I can cope with much more waiting around "treading water". Roll on Tuesday. Fingers crossed the phone call!

The brain fog has lifted! - Friday 6th August

Yet another crazy week with quite a few highs and lows. I've been using some of my annual leave to get a few things sorted before the surgery. If I'm honest, I haven't managed to do much for the past few weeks which is very frustrating. Being so tired and dizzy makes it hard to concentrate. I've been sleeping a lot and very rarely left the house.

Tuesday came and went with no real updates about a surgery date which was very disappointing. I did hear back from the team at the hospital about the steroids, so from Wednesday morning I started on them again (to reduce the swelling around the tumour). Wednesday and Thursday there was some improvement like they were beginning to make a difference. I had much more energy and I could finally walk properly again (as the issue with my foot has returned).

But it was today (Friday) where I've really felt better. The best I have felt in weeks to be honest. I realised while having a shower this morning that I was actually thinking and planning out my whole day ahead of time. Multiple streams of thoughts about everything I wanted to get done. I'd not had this ability for a few weeks. It has literally been quite ‘single threaded' (there's a web developer term for you). Basically I've only been able to focus on the single task at hand, nothing more. I must admit I didn't notice this change, it was only once I ‘reverted back' to feeling normal again I realised it had happened. So yes, I went into overdrive with work today because I could finally do things again! I worked until 7:30pm (naughty I know) getting tonnes of stuff sorted, and started making plans for the department's stance on legacy browsers (cough... IE11), which should come into effect in March to June in 2022. So planning almost 10 months ahead of time. I simply couldn't have done that even 3 days ago.

Next week is going to be interesting. I have a couple of meetings with the hospital to go over symptoms and then speak to the neurosurgeon again. I have lots more questions for the team. So let's see what next week brings! Maybe some clarity on a date, like I've had from the steroids over the past few days. That would be nice!

Another mixed few days - Sunday 8th August

Well it has been an odd weekend symptoms wise. Friday was a very good day. The best one I'd had in a while actually. I could finally think clearly again. Unfortunately I think I went a little bit overboard!

Saturday

I really paid the price on Saturday. As Friday had gone so well, we decided to head out to an open day event for a charity. We were under the impression it wouldn't be particularly busy, but unfortunately not. It seemed like everyone had turned up. This isn't a bad thing in terms of covid as it was all outdoors and we all kept our distance. It was just the sheer number of people, sounds, noise, interactions… the general busyness of it all that broke my brain.

I lasted about an hour then had to admit defeat and go back to the car to sleep with some quiet music on for 2 hours. I believe this was a mixture of the tumour, swelling, and having just started taking steroids. Apparently steroids can heighten your senses, making sounds louder, colours brighter etc. I think it was a combination of it all that my brain just needed to get out of the situation.

Thankfully after a couple of hours of rest and getting home I felt much better, and ended up doing virtually very little for the rest of the day.

Sunday

After a great Friday and poor Saturday we decided that I best not risk doing much, so I mainly pottered around the house looking for little bits to keep me occupied. I've certainly noticed that I'm much more comfortable and settled while at home. I'd assumed it was all a part of the past 18 months and the pandemic, but I'm beginning to think it is all interrelated. I think (unknowingly) to me this tumour has been developing and growing and it has been changing how my brain works and thinks. Although I don't feel anxious about leaving home, there's always something in the back of my head where I can only fully relax once I'm back.

Now, day to day you don't notice this, why would you after all? Such small changes. It's just like watching your children (or pets) grow up. Day to day you see no change, but looking back everything has changed. So when these subtle differences are compounded over months (and even years), then you see the huge change staring you in the face. The more I think about it, the more it makes sense. My brain is actively looking for a place with the least cognitive load. Familiar surroundings and people with complete predictability. This equates to a brain that doesn't need to do much (which is good when you have a brain tumour!).

Annoyingly on the flip side, the steroids seem to give me enough energy to put me on edge (almost twitchy), but then I don't have the energy or brain capacity to do anything particularly useful with it. Stuck between a rock and a hard place really. So the day was mainly resting and mentally preparing myself for another week of uncertainty.

The week that will never end! - Wednesday 11th August

Well, the title says it all really. This feels like it has been a very long week. I've had 2 important meetings at the start of the week, and after those it's hard to believe there are still more days left to go!

Monday

Since starting back on the steroids last week to reduce the swelling on my brain the team has wanted to keep a close eye on me. If there's one takeaway from the past 3 months it's that they don't want me on them! That's fine with me, I'm happy to listen to the experts!

So I had a general catchup with the nursing team going over my symptoms since I started back on them again. I explained the highs and lows of the weekend and it all sounds like classic steroid side effects. One day you feel on the top of the world, the next day you are back in the depths. When you couple it together with the tumour, swelling, and the fact that steroids obliterate my sleeping pattern (3-4 hours a night when on them), it all makes sense really. I'm on mild sleeping tablets to combat the issue, but they don't work too well.

The rest of the call was mainly to prepare me for the Tuesday call with the neurosurgeon. I'd already started making a list of questions, but off the back of the conversation I now had a few more. The more information you can get from the person who will actually be operating on you the better really!

Tuesday

I worked in the morning as the call wasn't until the afternoon. For some reason I was very nervous about the whole thing, so I was glad to have the distraction. I'm not quite sure why I was so nervous, but it's always in the back of my head: "maybe they made a mistake, and the 2nd scan will show a much worse prognosis". Unlikely I know, as I'm sure they would have mentioned it, but it's hard to stop thinking about it!

But overall the 45 minute conversation went really well! There's been no growth in the tumour size over the past 3 months which is amazing news. The tumour still looks to be of the type oligodendroglioma. The fact it hasn't grown point towards it still being grade II and not anaplastic oligodendriogliomas (grade III). We still won't know this for sure until 10 days after surgery, as that's how long it takes to get biopsy results back.

The swelling around the tumour isn't as bad as thought. Only slight swelling really. But this then points to the fact that the symptoms I've been getting (dizziness, ‘fuzzy head', loss of concentration, disconnection with the world etc) all relate to the tumour itself. I'm not sure which I prefer it to be, the tumour or the swelling!

I got to ask lots of great questions and get most of the answers I needed. One of the fundamental takeaways is that I "must not get covid", (direct quote from the neurosurgeon). I'm more than happy to keep to that! Having been double jabbed and not really going out, I'd hope the chance is pretty slim anyway.

The main discussion was around a date for surgery, and this came up almost right away. The team were very apologetic that it had taken so long, but the pressures on the whole system at the moment are large. Now as I seem to have a grade II tumour that is large (but not aggressive) this drops me down the priority list somewhat. There are some types of brain tumour like the glioblastoma that are really really aggressive and can't be cured. The neurosurgeon explained that in these cases sometimes you only have a matter of days / weeks where you can make any difference to a patient's life expectancy! So at the moment I'm now top of the list, barring any emergency cases coming in. So in theory I should hopefully be operated on in the next two weeks(!). Off the back of the meeting we are looking at a date of "August". If this is the case, it fits well with the children going back to school in September, as I can see the risk of getting covid in this period increasing once they are back. And what with my instructions "not to get covid", these two worlds don't fit together!

I asked about how big he thought my tumour was. I've already exported a 3D animation of Gary which you can see here, but I wasn't sure actually how big ‘he' is in terms of numbers. The neurosurgeon estimates that ‘he' is taking up around 10-15% of my brain. That's pretty impressive! So I only have the capacity to use 85-90% of my brain anyway. I also asked the question if this is the type of tumour I could have had since birth, and I've just grown up with it? The answer to that was a blunt "no", so that's the end of that conversation then!

I found out a more about the operation itself and recovery period:

Operation

It is likely to take 6-9 hours, with me being awake for approximately 6 of those hours. That sounds like a hell of a long time to me, but the neurosurgeon assures me it will go quickly. He said I'd be surprised how quickly it passes, so watch this space. I'll let you know after the event! A successful operation is where he manages to remove around 80% of the tumour, leaving the rest to be removed via other means (radiotherapy / chemotherapy / additional surgery).

Recovery

Now because the tumour is so large and deep into my brain, it comes with a few risks (as I've mentioned before in the original post). One of the main risks is the fact that the tumour is wrapped around my motor cortex, which is the part of the brain that controls limb movement. As Gary is sitting in my right frontal lobe, any issues will occur on my left hand side (since the brain is reversed in this respect).

The neurosurgeon explained that there is something like a ‘switch' that can effectively turn on and off if the brain is damaged (or changed). During the operation there's no warning that this will occur. One moment I could have full control over my left arm / hand, the next moment nothing. Quite a scary thought really! Thankfully in most cases this is only temporary and lasts 10-14 days. It's very rare that it is permanent. Over time, as the brain re-adjusts to being without the tumour, it relearns how to process limb movement and the ‘switch' is turned back on. If this were to happen, my seemingly ridiculously short 24-48 hour stay in hospital will turn into a couple of weeks in rehabilitation. So there's just another unknown to throw into the mix!

So that was mainly what I learned from the first two days of the week! Quite a lot to take in, and there are still 3 days left to go! I plan to take it a little easy to process it all and start mentally preparing myself for the next two weeks, where there's a lot that can happen very quickly indeed. I feel I'm about to reach a watershed moment in my life. There's forever going to be a pre-surgery me, and post-surgery me on the horizon.

Tune back in soon for more on the eviction of Gary ‘the prick' Glia!

Houston, we have a date! - Tuesday 17th August

So it's been over three months since I was first diagnosed, and I must admit I've no idea where the time has gone! In some ways it feels like it has gone quickly, but in others it feels like it has been the longest three months of my life! Living your life in weekly chunks isn't much fun. Waiting for every Tuesday to come around for a surgery date is quite stressful. The day comes and goes with a whole range of emotions. A mixture of anxiety, excitement, all the way through to disappointment and sadness, then rinse and repeat for another week.

Thankfully the waiting now seems to be coming to an end as I've finally been given a date for the operation: Friday 27th August, or G-Day as I'm calling it. G-Day stands for "Gary get the f*ck out of my head"-Day. The plan is for me to go into hospital the day before and stay overnight, with the operation happening on the Friday morning. It should last approximately 8-9 hours. Beyond that is a big unknown, so we'll have to see how it goes!

Unfortunately, there is always a chance that it could be cancelled at the last minute, or the hospital doesn't have a bed for me. If this happens there are multiple options. The nurse explained: "Plan A is the aim, but plans B-E also exist too". We will only know the plan I'm sticking to on Thursday. It will be a relief to get a hospital bed, as that's the final step before surgery!

In other news: I also found out that during recovery I'm going to need to wear deep vein thrombosis (DVT) stockings for 1 month(!) after surgery. Night and day, 24/7! Apparently the risk of blood clots post-brain tumor surgery is high, so to reduce the risk the stockings are required. Now as a person who doesn't even like to wear socks to bed, this sounds horrendous! But if it means I don't get blood clots, well I guess it's a thing I'm going to need to get used to. I think the next few months are going to be very difficult. A character building experience some might say...

Last day at work - Friday 20th August

Today has been a strange day. It was my final day "in the office", as I'm stopping work to prepare for my operation next week. I have to admit I'm feeling a little sad about it. Work has been a real safetynet over the past few months. When I needed a distraction from all this, and an "injection" of normality I could always fall back on work. The work isn't easy. In fact it is incredibly frustrating at times. But that's part of the distraction. Being able to make plans for the department in the coming years is time consuming and slow, but ultimately very rewarding. I've handed over as much as I can and tried to remove myself as "a blocker" from as many areas as I can, but it could be that certain areas stall until I get back to finally drive them forwards. Here's hoping for a quick recovery and staggered return to "normality", whatever the hell that is in 2020/2021!

Houston, we have a seizure! - Friday 20th August

Just when Friday wasn't strange and emotional enough, Gary steps in and makes things worse. 6:30pm rolls around and I'm making a cup of tea in the kitchen. I suddenly feel something "isn't right". I'm standing and realise I need to sit down quickly before it happens. I start to walk the few meters across the kitchen to the nearest chair but don't make it in time. My body goes into a focal (partial) seizure.

As I'm walking I say to my wife: "I need to sit down", but the words came out slurred (they sounded fine to me). Luckily I manage to lean on the kitchen worktop when it happens, so as not to fall over. I gently start to rock back and forwards, this is out of my control. At that point Claire told me to keep speaking, as it's good to know what effect the seizure is having on my speech. At that moment I actually struggled to speak. That isn't through lack of ability, it was through lack of "being bothered", would be the best way to describe it. There seemed like a real disconnect between myself and reality which I didn't have the energy or inclination to break down at the time.

The seizure seemed to pass quite quickly. Probably 20-30 seconds in total. I managed to sit down and take in what had just happened. Annoyingly I didn't feel any ‘release' of pressure in my head which I've had in the past sometimes. I actually didn't feel any difference at all.

So this was the first clear ‘core function' seizure I've had since the 11th June. There are two intriguing similarities between them: they both happened on a Friday after work, and both happened while coming off steroids. Now this may be a complete coincidence? I have no idea. Maybe I'll ask the team next week. Next Friday can't come soon enough!

Time slows down - Sunday 22nd August

I knew this would happen. As soon as I had a date everything would shift. Suddenly time slows down as I'm no longer living each day waiting for a phone call, I'm living each day with a future date on ‘x number of days' away. It certainly doesn't help that sleeping is difficult. Only sleeping 3-4 hours a night makes for very long days! I think this may be the longest week of my life. I need to find a distraction (that isn't work) to keep me occupied. Roll on next weekend and the Belgian Grand Prix. I will never be so happy as to be able to watch a Grand Prix knowing surgery is over!

Cancelled surgery #1 - Tuesday 24th August

So today was a bit of a roller coaster. I woke up very early (I'm still not sleeping too well), as I needed to complete my PCR test at 7am. This works out as 3 days before surgery on Friday. Once completed I went for a little walk to post it in a priority post box. I saw absolutely nobody at that time of the morning, which was handy for COVID. I just so happened to walk past my old secondary school too, which was a trip down memory lane. Not much looked to have changed apart from some of the dilapidated classrooms have been knocked down and replaced with sports courts. I remember them being bad in 1998, I hate to think what they would have looked like 23 years later!

Unfortunately the positive start to the day wasn't to last. A few hours later I received an email from the hospital stating that my operation on Friday had been cancelled due to a staffing issue with the team. I do wonder if this is all related to COVID, as it seems very strange that an operation planned 1.5 weeks in advance suddenly doesn't have an important member of the team available. This is disappointing, but in all honesty I'd much prefer they are 100% happy with the team and everyone is available, rather than a ‘settling' for a team just to get the operation ‘done'. So for the sake of another few weeks (after waiting 4 months), that's fine by me!

Note that I've called this update ‘Cancelled surgery #1'. I'm hoping that there won't be a #2!

The end of a long week - Friday 27th August

Today was supposed to be the day of my surgery. Unfortunately it was cancelled on Tuesday. In some ways I'm glad it was cancelled so early into the week, as that has given me time to get used to it all. If it had been cancelled the day before, that would have been even worse.

I've now been signed off work by my GP until my surgery date. I've managed to work for over 3 months, but the combination of the drugs and tumour are starting to take its toll. I can have dizzy spells and get tired very easily. One minute I feel fine, 5 minutes later I'll feel totally different, almost drugged / drunk (I don't drink). This all has had an impact on my concentration and work. This is especially true now that I'm off the steroids. They gave a little boost that enabled me to power through, but without them it's difficult.

So all in all, a very long week. I've started making plans for the coming weeks as I've been given a very loose date of the 10th September for surgery. It's not guaranteed, but it's the only date we have to work towards at the moment, so it will have to do. With the children going back to school on the 2nd September I can see this being an interesting few weeks ahead!

A promising end to the week - Saturday 4th September

It has been a very busy week, lots going on! In preparation for recovery we've had a third of the garage converted into an office. This frees up a room that can be used as a spare bedroom. This will be really useful if I need friends and family to stay and help during this period. Because of this, I've been gradually cleaning up and making sure we can find things. I can't explain how frustrating it is not to be able to find tools in the garage. A 5 minute job takes 20-30 minutes just because I can't find the tools! Saying this, it's all looking great, not 100% yet, but I'll be able to potter about during recovery and finish it all off.

My kids are now back at school so I'm self-isolating at my parents. The risk of me getting COVID has jumped since we can no longer control or monitor who the kids are interacting with. Since it takes around 48 hours for COVID to become detectable using a lateral flow test, I could easily catch it and not know about it for a few days. This would be bad in terms of the operation. It puts me at risk, as well as the whole 12-14 people team who will be involved in the operation.

I had some really positive news on Friday when the neurosurgeon who will be forcefully ejecting Gary, phoned me. He phoned for 3 reasons:

To apologise for the cancellation on the 27th. To check on how my seizures are at the moment. To see if I would be interested in taking part in a new feasibility study related to more detailed brain imaging.

In relation to point 1, although this was disappointing, I really appreciate the fact that he made the difficult decision to cancel surgery. It was made because a critical team member wasn't available. I'd much prefer to go into a long operation with the correct team, than a team pulled together just to ‘get the job done'. This makes me feel more comfortable about the surgery, both from the fact that I know the best team is involved, and also that the neurosurgeon has the team he wants around him. This must be a confidence boost for him too. Not that I expected (or even needed) an apology, it's nice to be given it. The NHS are doing a fantastic job considering the tidal wave of craziness they have been under for the past 18 months!

Point 2 is an interesting one. It's been a few weeks since I've had a ‘core feature' seizure, but that doesn't mean my brain hasn't tried! There are certain days where I feel quite ‘detached' from the world. If not ‘detached', then I can be quite fuzzy headed where concentration is hard. I'm no expert, but this seems to me as though my brain is trying to have a seizure, but the drugs I'm on are stopping it. I keep a detailed seizure log and have been keeping the hospital team constantly updated with how I'm feeling day to day.

There have been discussions this week about upping my drugs, or even putting me on a second drug to try to combat these episodes during the operation. If I were to have one during the operation it could be bad! Although now it sounds like the decision has been made not to change my medication. I'm personally not concerned either way. I'm happy to follow the team's advice. They are the experts afterall! According to the neurosurgeon it could be that the operation is adjusted to account for these ‘almost' seizures, this could mean I'll be awake for less time, which is safer if I do have a seizure. But it also means they won't be able to take out as much of the tumour because they won't know if they are inadvertently taking out ‘good' parts of my brain. Ultimately it's the neurosurgeon's decision, and he will be making it on the day. I just need to keep the team informed as to how this week goes. That's all I can do!

Point 3 is super exciting. I've been offered the chance to take part in a new feasibility study related to Fluorodopa PET Imaging in Glioma. I'll be administered intravenously with a mildly radioactive drug, called fluorodopa. The PET scanner then detects the radiation within the tumour. This imaging should give the team a lot more information about the internals of the tumour that a MRI & CT scan can't show. So for example, the more aggressive parts of the tumour will show up on the scan, allowing the neurosurgeon to focus on those areas and make sure they are removed as a priority. In this instance the more data the team has the better, so I'll be consenting to taking part in the study. It's great to think that these scans could help with diagnosis of tumours in other people in the future. I'm also keeping my fingers crossed that I will also have access to these more detailed scans when I request them all post-surgery.

All in all a very busy week! Next week could be G-day week! We shall see, I expect to hear more on Tuesday, so watch this space. Gary, your days could be well and truly numbered. 6 days and counting (hopefully!).

MRI & PET scan day - Wednesday 8th September

So it's G-day + 2, and all is still looking good for Friday! I had to sign a lot of forms today related to the feasibility study and the actual operation. It was mainly going over the risks associated with surgery and what to expect. I've now been put on a 2nd type of seizure medication to try and head off the last of the seizure symptoms I'm having (pun intended!). My first PCR test came back clear, and I had another PCR test today. I will most likely have another 2 this week. My poor nose!

The MRI scan went well today. It lasted around 40 minutes. Thankfully I didn't have a reaction when they injected the ‘contrast' this time which is great. It was loud but tolerable. The doctors and staff at the hospitals (I went to 2 today) were all fantastic. I couldn't wish for a better team. I feel very fortunate to live in Oxfordshire.

The PET test unfortunately wasn't as pleasant. It was nothing to do with the fluorodopa, it was the fact that CT scanners really aren't built for comfort. The scan lasted a whole hour, and honestly it felt like a lot longer as I had no reference point for the time! I was strapped to a machine with my head restricted and my feet were cold. The worst part was that there didn't feel like there was enough padding behind my head. After 10 minutes the back of my head started to hurt, and once you notice it, it's hard to stop focusing on it. Especially when there's nothing else to do! Try it some time, it takes a surprising amount of energy to lie perfectly still for 1 hour while staring at the ceiling. I'm very glad it is now done though. Apparently the scans came out really well and they got all the data they needed for Friday's operation!

I also found out a little about the surgery itself. I'll be lying on my left side for the operation with some sharp points tightly pushed against my skin to stop my head moving. I'll have a cannula in my right arm, as they want me to be able to move my left arm when I'm asked too. Since there's a risk of partial paralysis on my left side, they will be testing for this throughout the operation. It sounds like this could be quite uncomfortable on my neck which is a worry, but I'm hoping I'm drugged up enough to really know about it!

So anyway, this is likely to be my last update for a little while, as I should be heading into the hospital around midday tomorrow. Fingers crossed Friday all goes well and I'll be able to post updates soon. This should include the PET scan imaging as well as a before / after image of my brain, minus Gary!

No matter what happens, thanks everyone for all your support, kind messages & visits over the past 4 months. They have been very much appreciated. Here's hoping the rest of 2021 / 2022 improve!

Thrust into the post-surgery world

So apologies to readers waiting for an update, it's been quite a few months as I've been a little busy in recovery, a lot of things have happened so this is going to be a long update! I'll pick up from where I left off on what I can only describe as the longest week of my life!

Into hospital maybe? - Thursday 9th September

So Thursday was a very long day. I hadn't been sleeping well all week, probably due to a combination of drugs and the stress of waiting for the operation. The fact that I was still having seizure-like symptoms didn't help either. So I found myself awake at 3am thinking what can I do with my time as I didn't think I'd be able to sleep. So I decided to do something I'd been putting off for months, simply because it isn't pleasant and not what you expect to be doing at the age of 39 years old. So I planned my own funeral.

My sister is the executor of my will, so I wrote a fairly brief email for her, listing my basic wishes. For example:

Cremation or burial
Religious or non-religious
Music / Dress Code
Flowers or donations to a (brain cancer) charity
Budget plans for payment
Where to spread my ashes

I titled the subject line of the email "Unpleasant email" and used Gmail's scheduled send functionality to send it at a more sociable time. In hindsight I should have set the subject to ""Unpleasant email [Don't read in public]", as I think I caught her a little off guard. It turns out she read it while in the supermarket, which unfortunately made her cry down the bread aisle of her local Sainsbury's. I found this out in her reply back to me later in the day. The email chain is still difficult and upsetting for me to read even many months later.

So there you have it, learn a lesson from me: When you are sending an email about your funeral to your relatives make sure you consider when and where they might be when they read it! You can use the short checklist above as a starting point for your planning. Depressingly (for a 39 year old) I got much of this information from a checklist on the AgeUK website.

After that I still couldn't sleep so I wrote a draft "Final" blog post for this very blog, as I'd already arranged for it to be published by a good friend and colleague at work should the worst happen. Thankfully as you can probably tell this preparation hasn't been needed! On a more positive note I now have an idea of what I want my funeral to be like at the ripe old age of 39!

Let's move past this deeply depressing section because the Thursday hasn't even started yet! Due to the COVID risk from my children I'd been staying with my parents for a few weeks. So my parents and I would be making the 1.5 hour journey back to Oxfordshire in preparation for going into hospital later in the day (2pm).

Earlier in the month I'd had multiple phone calls with the team looking after me as to how the days before surgery would go. We had a "Plan A" all the way to a "Plan E". Plan A was: I'd phone the hospital at 2pm the day before and a bed would be available. Travel to the hospital that day and stay overnight before the operation first thing on Friday morning. Of course nothing ever goes to plan! I phone at 2pm and there's no bed available and nobody to really speak to about it.

At this point I was in the middle of feeling terrible due to my brain wanting to have a seizure but the drugs stopping it from happening. So I went to bed for a few hours to try and sleep it off. During those hours my wife and eventually my mother tried to get a straight answer from the hospital to see if I would have a bed that night. Around 7pm we got confirmation from the hospital to say that a bed wasn't available but the operation was still going ahead in the morning. So I'd be sleeping in my own bed the night before the operation and leaving early for the hospital in the morning. In hindsight this was actually a good thing as you will read later in the update.

I'd just like to point out that this isn't a dig at the excellent work all the NHS admin staff are doing. If there are only a limited number of beds available there's not much you can do! Especially during Covid, which was on the rise in Oxfordshire at the time.

Day of Days (G-day is finally here) - Friday 10th September (day)

G-day stands for G(ary get the f*ck out of my head)-day. And it's all a bit of a blur to be honest. I woke up early as again I couldn't sleep due to nerves. I was hoping to sleep a little in the car but wasn't feeling well at all. My brain felt ‘fuzzy', I couldn't concentrate and had very little energy. I couldn't have any breakfast due to the operation being in the morning so that didn't help either. We set off early so as to avoid any traffic. The last thing I needed was to get stressed because we were late!

When we arrived at my home I finally got to see my wife and children for the first time in a few weeks. Covid and cancer really don't mix well! My wife and I had already decided that she wouldn't be coming to the hospital with me. She knew she would get upset, which would likely upset me too, which isn't what you need just before a major operation. So she stayed at home and took the kids to school, to give the impression it was just another uneventful day! They knew that "Daddy was going into hospital to have the lump in his head removed", but being young didn't fully realise the seriousness of it all.

I needed to be at the hospital for 9am to be admitted to the ward for the operation prep. Once there nobody was at the reception to admit me so I ended up speaking to another patient waiting there. He was also being operated on today by a different team. He'd already had brain surgery 14 years earlier, the scar was clearly visible on his head where his hair hadn't grown back. We made small-talk for a while but as we were both nervous it felt quite forced.

The reception staff arrived and admitted me. I had a brief consultation with the consultant anesthetist as he wanted to check I hadn't eaten anything and wasn't allergic to any medicines that I knew of. All standard questions for pre-surgery. Interestingly this is the consultant anesthetist who wasn't available for my previous surgery date due to him being in Covid isolation. I guess the Neurosurgeon has a team that he likes to have around him, especially considering how big my tumour was and how long the operation would be! After the consultation I waited in reception for 5 minutes or so until one of the team led my mom and I to the bed I'd eventually be in after surgery.

The other patient who I'd been speaking to was in the bed opposite. Knowing that I wasn't on my own my mom left and went back to the car where my dad was waiting to take them home. It all happened so quickly we both said our ‘final' goodbyes and that was that. I was more worried about her getting lost in the hospital than the actual operation at that point.

Now I'd only been on my own for 1 or 2 minutes and the nurse came in to start pre-surgery prep. She took my blood pressure and oxygen levels etc. Then I had to put on what would quickly become the bane of my life for a month: Deep Vein Thrombosis (DVT) leggings. The reason for wearing these is that stats show that after brain surgery you are 30% more likely to develop blood clots which can be fatal if they travel to the heart or brain.

Once prepped I got to meet my Neurosurgeon for the first time in person, he introduced himself and checked I was doing okay. Usually I would have shook his hand but due to Covid I didn't think it was appropriate. You'd be surprised how much a little thing like that annoyed me. You're about to put your whole life and future into this person's hands and you can't even thank them with a handshake.

The Neurosurgeon left, and before I knew it I had a group of people around me leading me down to the operating theatre. I said goodbye and ‘good luck' to the other patient in the room and out of the ward we went. I have to be honest I don't remember the walk to the operating theatre at all, or what we discussed on the way there. My brain was just trying to take in the enormity of the situation ahead of me. After 6 months of mentally preparing for this moment it is finally here!

We entered the side room to the operating theatre and the team started to work on me. They told me to get up and sit on the trolly in the centre of the room. I distinctly remember thinking "This could be the last thing I ever do as a person", It took all my courage to get onto the trolley and I have to say it's the scariest thing I've ever had to do in my whole life. There were 3-4 people in the room prepping me, for the most part I had no idea what they were all doing or saying to me. But I do remember them marking the right side of my neck with an arrow pointing up to my head in black permanent marker and then them telling me they would be putting a cannula into my right hand, then once done they would put me to sleep and put something called and Arterial cannula directly into my artery in my right arm. They also put 2 other cannulas into the top of each of my feet. I'm very glad they did all this while I was asleep as I hate cannulas at the best of times! The last three sounded deeply unpleasant.

So that was that, cannula in the right hand, they said they were injecting me with anaesthetic and it would feel cold running up my arm and that I may have a strange taste in my mouth. I felt the anaesthetic reach the right side of my neck and I passed out.

During the time I was passed out the team prepared me for surgery. I was moved onto my left hand side and my head was placed in a metal contraption with 4 very sharp points sticking into my skin; these were to hold my head in place for the operation. The team cut the skin on the top of my head and peeled and folded it forward to expose my skull. At this point they then used some sort of medical saw to cut through my skull and expose my brain (and Gary) The incision started at the front of my skull just after my hairline and travelled backwards right down the centre before turning to the right and coming back to just before my right temple in a large horseshoe shape. They'd basically had to remove the right hand side of my skull due to the sheer size of Gary (the prick). Thank goodness for anaesthetic and anesthetists is what I say!

Once all this was done they reduced the anaesthetic and used other drugs to bring me round. I remember waking up lying on my left hand side, my head was already open and I couldn't move, there was a large transparent plastic cover in my peripheral vision to the right just above me (like the plastic covers you see on children's buggies to protect them from the wind and rain). There were 2 people in front of me and a television screen just behind them. One person was the consultant anesthetist just watching proceedings, as I guess he'd done most of his job and was just monitoring the situation. The other person was a lady. She was the Neurophysiologist looking after me, her job was to be the go-between between me as a patient and the Neurosurgeon. She'd ask me questions and update him on how I was reacting to those questions.

Then the last thing in front of me was the TV. It had live pictures of the inside of my head, now they weren't gory pictures with lots of blood and bone etc. They were the 3D scans of my head taken from my last MRI scan. The crazy thing I remember is they updated in real-time, I could see exactly where the Neurosurgeon was putting his tools. There was no pain while he was doing this but I could feel a distinct "pressure" inside my head that I'd never felt before, so I knew he was there.

I remember finding the real time 3D imaging fascinating. When they gave me the chance to ask them questions I remember asking them how it all worked. I was thinking it was some sort of haptic technology driven tool but it turns out it was a lot more clever than that. The team had placed 3-4 sensors on the outside of my skull which could detect the "marking tool" the surgeon was using inside my skull. By monitoring the input from these sensors they could get a computer to update the screen in real-time, even for a person who works in technology on a day-to-day basis this blows my mind!

So what exactly were they marking in my head? Well it turns out this is where the PET scan that I'd had on Wednesday comes in. The feasibility study in theory should be able to be used to identify the different cancer grades within the tumour, something that an MRI / CT scan on it's own can't show. You can see the results of my PET scan below:

Now this is an ongoing study and I'm one of around 30 patients in the trial. The final results of the study won't be known for many years. The above image is the result of the hour-long scan I had to endure. The radioactive isotope was absorbed by Gary, then depending on the grade, different parts emit radiation at different rates. This Is then picked up by the CT scanner resulting in the image above, hence showing Gary's internal structure. My assumption is that the red areas are the higher grade (e.g. grade 3), through to green / yellow being lower (e.g. grade 2). But this won't be known and confirmed until the study concludes in a few years.

The reason I'm telling you this is because when I woke up I heard the team were having a conversation where they were talking about colours and numbers. As it turns out the team were talking about these scans. They'd identified certain areas of Gary that they would like to biopsy for further analysis, before moving onto removing Gary completely. I think because Gary was so big there was a treasure trove of data for the study! I heard from the team post-surgery that they removed 8 areas of Gary for further analysis. They would have liked to have taken more but the Neurosurgeon put his foot down to say no more, as the procedure with only 8 added 1-1.5 hours more to the surgery time. One thing I remember finding amazing was how the whole team was working together. The Neurosurgeon was working inside my head but he obviously couldn't see the real-time scans with the areas he was supposed to be removing, so the team were speaking to him telling him to move left / right / up down etc. Only the language they used was more medical than that, so using more medical references from the brain. It was incredible to see and hear.

I remember asking a few more questions about what was going on that the Neurophysiologist couldn't answer and getting the response "that would be a question for the Neurosurgeon, but he's a little busy at the moment!". Then I think they put me to sleep again (probably to shut me up to be honest!). I've been told there were between 12 - 14 people in the operating theatre at the time but I only remember seeing 2 people and hearing the Neurosurgeon. The other folks must have been supporting the Neurosurgeon "above me".

The next thing I remember is coming to, my eyes were closed so I couldn't see anything but I could feel they were stapling my head back together. I felt no pain but I just remember the pressure of the staple gun reverberating through my body. I also remember thinking "that's a lot of staples!". If you've ever used a staple gun you will know that you have to put quite a lot of pressure on the handle before the staple is released. Well it sounded exactly like that, only into the top and side of my head around 95 times!

Annoyingly that's all I really remember from the surgery. I estimate maybe 45 mins to an hour of the 7 hour operation. I've actually been quite disappointed that I don't remember more of it and nothing more has come back to me over time. A couple of weeks after surgery I spoke to one of the Neurosurgeons who was also in the operation helping, about this. And she said that this was intentional. They constantly increase and decrease the drugs throughout the surgery as there are certain unpleasant things they don't want you to remember. So I'm thankful for that, but in another way I will always wonder what happened in those 6+ hours of my life I can't remember!

Recovery starts - Friday 10th September (evening & night)

The next thing I remember is waking up on the hospital ward with the absolute worst headache I've ever had, not really surprising when they've just removed approximately 15% of my brain matter! I could barely move and had to sleep upright due to the pressure lying flat would cause.

I'm so glad they put me to sleep before inserting the Arterial Cannula, as the bruise I had on my right arm was huge! It must have been quite a struggle to get it in there!

Now I have to be open and honest. This was the worst night's sleep I think I've ever had, hence why I was grateful I slept at home the night before. I'm not a fan of hospitals anyway, but this night will stay with me forever.

Due to the risk of blood clots I had my DVT leggings on, but as an extra "brucie bonus", I also had a pair of leggings that inflated & deflated every 10-30 seconds. The only way I can describe it is when you go to have your blood pressure checked and the strap on your arm inflates then deflates. Well imagine that on the total length of both calves. Inflate / deflate all night long every 10-30 seconds!

I also found I couldn't really move both because of the leggings and the pain it caused in my head. I found my legs were roasting hot but my top half was cold because I couldn't pull the blanket up over myself. What made things worse was I couldn't find the call button to ask for help. I must have spent 20-30 mins looking for it before I eventually found it (it may not have been that long but it felt like it!).

Just to make things worse the patient opposite me who I'd spoken to before surgery was being violently sick all night. I think he must have had a bad reaction to the anaesthetic. So yes, all in all a terrible night's sleep 0/10 would not recommend!

Let's get the hell out of here - Saturday 11th September

After that one night in hospital I was absolutely determined to not have to stay another. So once it was light the nurses brought around food that I couldn't eat. Everything tasted strange from the anaesthetic. I eventually managed to eat dessert, which was a rhubarb crumble with custard. The inflating leggings were taken off so I could finally move my legs. At that point I realised I had a much bigger scar than I expected so I wanted to go into the toilet to take a look. The toilet was only a few meters away but it felt like miles. I asked for a walking frame so I could get there myself but I had to be escorted for the first few times. Looking in the mirror was a weird experience. I couldn't see the scar at the moment as it was covered in a blue gause, but it was a hell of a lot bigger than I expected. I'm not sure why I thought it would be smaller looking at the size of Gary. How else were they going to extract the prick?!

At the same time as looking at my head for the first time, I managed to brush my teeth! But no matter how much I tried I couldn't get the weird taste of anaesthetic out of my mouth. Eventually after what seemed to take an age to do these small things I was escorted back to my bed, mostly under my own steam (with a walker).

Next came something that would be a thorn in my side for the next 3-4 weeks. Lying down or sitting up caused terrible pain in my head. This is because, unlike pretty much everyone but medical folks would know, when you get up / sit up / lie down, the intracranial pressure inside your skull changes. Usually you'd never notice this, but when someone has just removed a huge chunk of skull and stapled it back on, you suddenly feel it!

It seemed to take me ages to lie down on the bed but I eventually managed it and got comfortable. But I was determined not to have to stay another night in the hospital so I decided I needed to get moving. Over the next few hours I kept getting up and down again and used the walker to move around the ward. Once I was steady on my feet the nursing staff were happy to let me walk on my own (it was more of a shuffle really!).

At 12pm the nurse came over and said my wife (Claire) would be visiting in the next couple of hours, this perked me up a bit. It would be good to see a familiar face after what had felt like weeks since the previous morning! Claire was very surprised when the nurse mentioned that I was already up and walking about the ward.

2pm came and my wife brought in some M&S chocolates and she got to see my humongous scar. Thankfully I'd had no memory loss, but I was finding speech and concentration very difficult. I don't think we said much. I think I was mainly staring off into space or at the ceiling wishing the pain in my head would go away. The only painkillers I could have were paracetamol, none of the good stuff I'm afraid! Thankfully they took the edge off the pain. After an hour it was time for my wife to leave. She said she would be back to see me again tomorrow. But I had other plans.

Claire phoned my parents after leaving the hospital and pretty much said there is no way I'm going to be released from hospital tonight and it will most likely be Monday. So my parents stood down from coming to pick me up since they were a 1.5 hour car journey away. But during this time one of the senior doctors came to see me. He said that as long as I'm up and about moving we could try and get a CT scan sorted to check everything in my head was okay and try to get me discharged. I jumped on that in a flash!

I had to wait an hour or so for the CT scan then the porter came to wheel me across to the other side of the hospital. The CT scan took 10-15 mins or so then they took me back up to the ward. It was around 30-40 mins later that the same doctor came to see me and said everything looked great, so if I wanted to he'd be happy to discharge me. I obviously said yes, and he went off to pull together my discharge paperwork (which would take a couple of hours). I immediately tried to phone Claire but I had a slight weakness in my left hand, and I also found that the phone didn't make as much sense as it had done the day before. Things hidden behind icons were difficult to find. I found this very worrying considering how easy it was just a single day before. But I eventually managed to work out how to use the phone and rang her. At first she couldn't believe they were discharging me as I was still slurring my speech down the phone. Somehow I managed to get her to believe me with the help of one of the nurses who overheard via speakerphone. So it was then arranged that my parents would travel down to pick me up from the hospital and take me home. This eventually happened around 10pm. So from admission through surgery to discharge I'd been in hospital for 36 hours! That's pretty crazy when you think about it. 15% less brain and out in a day and a half! To give you an idea of how busy the hospital was at the time, I heard the nurses discussing the next patient who was going to take up my bed once I'd been discharged! The nurses really are unsung heroes, day in, day out working with sick patients, and it isn't easy work at all!

I don't remember any of the journey home, but I do remember that the car was really low and difficult to get into especially given all the staples in my head! I don't think my parents had a very good night's sleep that night. Mine wasn't perfect but it was 1000% better than the night before. I still had to sleep at an angle for another few days which was uncomfortable but atleast I wasn't in hospital!

How does the TV work? - Sunday 12th September

I woke up early as I was in pain and uncomfortable. The F1 Italian Sprint race had happened on Friday during my operation so I wanted to catch-up on what had happened. My Dad had recorded it on Sky+ so before they woke up I decided to go downstairs and try to watch it. Unfortunately, I couldn't work out how to use the Sky box. I was able to turn it on and find the recordings but when I played them it wasn't the Formula 1, it was the Formula 2. This totally confused me because the thumbnail preview was showing a Formula 1 car. I tried multiple times to get to the F1, but had no luck. Eventually I gave up and used my phone to watch the highlights on YouTube. This was the start of an ongoing theme over the next few days. Things that had seemed so simple and obvious only 2 days before I was suddenly struggling with. I had the same issue with the Netflix User Interface. I found it too busy and confusing to use. For a once technically proficient person I found this genuinely scary. Thoughts of "how exactly am I going to do my job" if I'm struggling with these simple things! Not fun, and quite a contrast to me only 2 days before. It really did give me some insight into the importance of making things simple and understandable for users that are less technically proficient. If only every web development team could have similar brain surgery. I think we'd have a much more user-friendly internet if so.

Is my head leaking? - Monday 13th September

Odd title I know, but I woke up on Monday and found spots of red transparent liquid on the white pillowcase I'd been sleeping on. Further to that I was certain I could feel "movement" underneath the blue gauze which covered the staples in my head. When I felt something drip onto the bridge of my nose, I immediately got in contact with the Neuro-Oncology team via email to let them know. Within 5 mins they'd got back to me via email and a phone call. They asked me to come back to the hospital as quickly as possible. Of course I was 1.5 hours away at my parents so it wasn't as simple as that. So I quickly packed a bag of clothes just in case I needed to stop overnight and we headed back along the motorway to the hospital.

Once we got there I waited with Claire who'd met us there for 5-10 mins and one of the Junior doctors took us into a side room and started to remove the staples from my gauze so they could get a better look at my head. She explained that if it was that Cerebrospinal Fluid (CSF) or "brain juice" (as I liked to call it) was leaking from the wound it also means that things can get into the wound / my skull. If this happens it can cause an infection, leading to a whole host of other issues that may even require more surgery. So it's best to get it checked out. The Junior doctor couldn't find any obvious leaks so she consulted one of her more senior colleagues. He came to see me and also couldn't find any obvious leaks. So he decided to do a very scientific test to make sure. He placed a white absorbent pad on my head (like a kitchen towel) and then had me cough and lie down on the bed. This increases the intracranial pressure inside the skull (I know because I could feel it!) and if there's a leak CSF would come out of the wound and be seen on the absorbent pad. Thankfully nothing was seen on the pad!

The senior doctor was certain the movement and liquid I'd felt / seen was the petroleum jelly (think Vaseline) used to keep the wound moist and clean. My warm head was heating the jelly, turning it to liquid and it felt like water (because I couldn't see it to verify it wasn't). He did leave me with a nice vision: Saying that if I do have a CSF leak it would literally be pouring down my face, and if this were to happen phone the emergency services immediately. I'm pretty sure I'd have done that anyway, but it's always good to know I guess!

With the panic over I headed back to my parents to continue my recovery, with hopefully less drama moving forwards!

General struggles - Tuesday 14th September

So I'm not going to give a day by day update on what happened as it's very boring and tedious. It felt like a long time for me because I was obviously living it, so to write about it would be even worse! But here's a summary of some of my general post-surgery struggles I had.

The DVT leggings were terrible to wear. I hate wearing socks to bed and now I had to wear these really tight leggings over my lower legs for a month! They were a pain to get off and it took help from both of my parents to get them back on. I guess it's a small price to pay compared to fatal blood clots!

As mentioned before, sitting up and lying down was painful. Basically going from 0 to 90 degrees and vice-versa caused a terrible pain in my head, so I tried to avoid moving, but you'd be surprised how often you make these movements going about your general day-to-day life! This was the same for coughing, laughing and sneezing, all these increase the intracranial pressure in the skull. Everytime i sneezed it felt like the top of my head was about to split open!

Related to the above issue, lying down and getting into bed was a chore! I remember having to get up in the night to go to the toilet, then getting back into the bedroom, and it literally took me 15-20 minutes to pluck up the courage to lie-down on my pillow to go back to sleep! I knew that it would only take 2 seconds to do it, but when I also knew it was going to hurt, my brain and body refused to move. I just ended up rocking backwards and forwards on the bed for minutes at a time, hoping the momentum would carry me forwards and I could then go back to sleep.

A strange one but due to the fact that I'd had open-head awake surgery, when they close your head it's impossible to remove the air pocket that is leftover. So for 6-8 weeks I had a huge air pocket at the top of my skull. And everytime I moved my head I could feel this air pocket move around my head. It wasn't uncomfortable, it was just weird. It was a similar sensation to someone lightly brushing the top of your head. There's nothing that the doctors can do about this, it just takes time for your body to naturally replace and recycle the CSF around the brain every 6 weeks or so. So the air was gradually reabsorbed back into my bloodstream. But this sensation lasted 6-8 weeks after surgery, maybe even more.

Related to the air in my head, I could feel my nose randomly "creaking". It's hard to describe but it felt like small air bubbles at the top of my nose releasing. I think it may have been related to the CSF fluid redistributing itself and the pressure inside my head stabilising itself over time. This lasted 2-3 months after surgery.

I had to keep the 95 metal staples in my head for 2 weeks and in this time I kept having phantom movements on the outside of my head. Almost like my body was realising that there was a foreign body (or 95 of them) attached to the outside of my skull. This stopped as soon as the staples were removed and the wound started healing.

And finally I think I may have been suffering from something called Abulia. This is described as "a state of diminished motivation". Now this is a self-diagnosis not from the doctors, but from reading up on the symptoms it certainly seems like what I was suffering with, at least in the first 3-4 weeks anyway.

All in all I think I've got off quite lightly considering the size of the scar and the size of Gary. The struggles above weren't pleasant but they were manageable. I'm very surprised how few headaches I've had. If I were to have guessed what the most likely result of major brain surgery would be it'd be headaches. But apparently not. At Least in my case anyway!

It's also worth noting that the only painkiller advice I was given post-surgery was to take 2 paracetamol! So remember that the next time you have a headache. Paracetamol are so good they even work for folks who've had 15% of their brain matter removed!

Biopsy results - Tuesday 21st September

As with most things happening in the NHS at the moment teams are extremely busy. So I wasn't expecting anything from the hospital on this day until I received an email at 9am asking if I was free for a call with my Neurosurgeon + team at 12:30pm. I replied back and accepted, I mean It's not like I had much else on! I wasn't quite sure what the call was about so really wasn't mentally prepared for it at all.

I jumped on the call with the team, my wife and parents were listening in too. My Neurosurgeon, never one to mince his words, immediately spoke and came out with the biopsy results of Gary, the now (mostly) evicted prick! He told me the team had managed to remove a whopping 95% of Gary, but the biopsy results had come back to say that some parts of him were Grade 3, rather than Grade 2. Grade 3 are called anaplastic oligodendrioglioma, these are malignant (cancerous) and can be fast growing. And because of this I'd need to start a 6.5 week course of radiotherapy within the next month, followed almost immediately by a 6 cycle course of PCV chemotherapy which will last approximately 7 months. As I'd experienced previously with my Neurosurgeon there was no warning about this news, no "are you sitting down, brace yourself". Just boom, you've got Grade 3 brain cancer! Well you may as well pull off the plaster quickly and get it over with I guess! In hindsight I was expecting it to be a multi-grade tumour even before they told me, I mean why would they have been wanting to biopsy multiple areas in the surgery if it had all been a single grade tumour?

After the initial shock, I managed to compose my thoughts and asked a few questions about the surgery itself. I asked if there'd been any nerve damage at all, as I had a slight weakness in my left hand. He looked slightly bemused by this and said no there'd been no nerve damage and the operation had gone very well. It turns out there'd been a slight miscommunication from what was said to Claire immediately after she got the phone call from the Neurosurgeon to say surgery had gone well and that I was in recovery.

I was led to believe that they stopped surgery because they had caused slight nerve damage. But this wasn't the case. It was actually that if they went any deeper it could have caused nerve damage, so they stopped. I later found out that the team were using a technology during the operation called Neurophysiology. This involved strategically inserting 5-8 metal pins into the left hand side of my body and monitoring the signals from these pins as they fired electrical impulses into my brain. It was basically a type of automated testing to see when they were getting close to nerves that actually did something, rather than just prodding Gary the useless prick. So once the safest amount of Gary had been removed, the team decided I was getting very tired and I needed to save my strength for recovery. So that's why the operation only lasted 7 hours!

In the above images you can see the before at the top then the after scan of my brain with Gary highlighted in red. As you can see he's mostly gone apart from some left over in the deepest part of my brain. This leftover is what the Radiotherapy and Chemotherapy are going to be targeting as well as any residual cancer cells around the edges of the surgery.

//: # (Image of my brain before (left) and after (right).)

Now a question I've been asked many times off the back of the quite frankly incredible image above is "what happens with the hole moving forwards?". Well this is a question I asked back when I was first diagnosed and they described the operation to me. The hole will now alway be there. Brain cells don't grow back. Over time it fills with CSF, but it will always be there. And a fascinating observation I've noticed since the operation is I can hear the cavity where Gary used to reside in my head. This is especially true when I'm in the shower and I run the water over my head. The right hand side of my head sounds hollow compared to the left! I like to compare it to looking for a stud wall behind some plasterboard. As you tap on the plaster the sound changes from hollow to dense. It actually took me a while to work this out and what was different when I was having a shower in the mornings. It's a strange feeling to know there's now just a big hole in my head!

After the appointment I didn't have long to wait as my initial appointment with the Oncology team to discuss the radiotherapy as it was set for the following day. No time to waste with Grade 3 brain cancer I guess!

Further treatment appointment - Wednesday 22st September

The whirlwind that my life had become continued with a meeting at a new hospital and a whole new team of experts. The meeting was to discuss the post-surgery treatment I was signing up for. This meeting was mainly about radiotherapy, with the chemotherapy discussion coming at a later date.

There were 2 oncologists and a number of nurses in the meeting as well as my wife. I'd made the decision that whenever something important was being discussed I should always have company as I knew for a fact I wouldn't remember it all, since it had only been 12 days since my surgery! There was no chance of me remembering much!

The meeting seemed to last ages and there was so much information being discussed I found it hard to concentrate on what was being said, so I'm very glad Claire was there with me to take notes and remember everything.

As well as introducing the timeline for the radiotherapy all the wonderful side-effects were discussed too. And they always seem to save the best (and rarest) to last, which includes the fact that radiotherapy in some cases can lead to other forms of cancer. What's better than having 1 type of brain cancer? Well 2 forms of brain cancer I guess!

Once all the difficult conversations were discussed, I signed on the dotted line to say I agreed with the treatment. I'd already decided this was the direction I wanted my treatment to go even before hearing all the side-effects so it wasn't a hard decision to make.

With a room full of doctors and nurses I decided to ask the question if it would be possible to take the staples out of my head. It was coming up to 2 weeks since surgery and this was close to the date they had said, so I may as well have it done while I was there anyway.

I felt quite sorry for the nurse when she saw the sheer number of staples in my head. I wouldn't even know where to start. The technique involves taking every other staple out first, just in case the wound hadn't healed. I guess they don't want your head to suddenly open up again, and every other staple allows for some support while assessing the wound. Removing the staples wasn't as uncomfortable as you'd expect. There were a few minor sharp pains with a few of the staples, but most of them were pain free. Because there were 95 of them it seemed to take ages!

Once the staples were out I was done for the day and I was travelling back home to be with my wife and children. I also managed to wash my hair / head for the first time in just under 2 weeks. It was the most amazing shower and I actually felt a little more human after it!

My kids were at school when I got back, and I needed to sleep for a few hours. This was the first time they'd seen me since before the surgery so it was going to be interesting to see how they would react to the huge scar on my head. My youngest son wasn't phased at all but as soon as my eldest saw me I could tell he wasn't comfortable with my new Frankenstein's monster look. We tried not to make a big deal of it but I ended up having to wear a hat around the house for a few weeks while my head healed to stop him freaking out.

Prognosis

So jumping back to the meeting with the team they went over the scans and some of how well the surgery had gone and then got down to the nerve-racking part, which is the future prognosis. They could only give me timescales for the "average" patient in my position. It was mentioned that this isn't a cure and it's likely to always be there.

The timescales they gave me are below:

Do nothing - the average patient can expect to live 7 years
Radiotherapy followed closely by chemotherapy - the average patient can expect to live 14 years

So here's hoping I'm the exception to the rule or not the "average" patient. Else I certainly won't need to worry about retirement plans (that's probably a blessing actually)! Given the retirement age for men is 68 in the UK, I'll fall some 15 years short. But a lot can happen in 14 years, you never know with further funding and research more options could be available by then!

Making my mask - Friday 1st October

Because Radiotherapy is a very targeted treatment, they need to make sure the treatment area is accurate for every session, and this involves me not moving my head at all for around 6-8 minutes while the treatment happens. This of course is impossible without aid, so they make a fitted mask that you wear throughout the treatment, that you can then take home should you wish to at the end!

The mask is made by heating up a rubber mesh to around 70 degrees centigrade in what looks like a pizza oven. While you are lying on the "couch" they quickly pull the heated mesh over your face / head and manipulate it a little to make sure it fits snugly. Then you have to wait 20 minutes underneath it while it cools, before it can then be taken off.

It's a very strange sensation when they first pull it over your face. It's like someone is pulling a plastic bag over your head and for the first second or so there's initial panic in that you think you can't breathe. But once you breathe through your nose it's actually quite pleasant and warm. You also need to close your eyes so as to not pull out your eyelashes when they remove it. A side effect of this means that during the treatment itself you have no idea what is happening around you.

It's also worth noting that when I say you lie on a "couch'' it's not like something you sit on to watch TV. It's actually a flat carbon fiber board that isn't built for comfort. It's built for accuracy.

Radiotherapy Starts - Tuesday 19th October

The radiotherapy happened over 6.5 weeks, with 33 sessions in total. I needed to go into the hospital everyday apart from weekends. The actual treatment only took 8-9 minutes once I was in the room. It was the travelling there and back (~45 mins each way), plus there was sometimes a wait for the machine to be ready, as there were sometimes delays with other patients. The worst day I had was one particular Friday where I left the house at 8am and used the NHS hospital transport. My treatment was delayed by 1.5 hours, so I missed my transport slot to go home. I eventually got back at 14:30 and was absolutely shattered.

I'd got into the routine of having the treatment, coming home, eating, then going to bed for 2-3 hours. Because of this I was so tired, there was no way I could do anything really for 6.5 weeks. The effect of the treatment was cumulative, so as the weeks went on I got more and more tired.

I'm not going to cover each day, but will give a week by week blow of the whole experience.

Week 1: 19th - 22nd October

This week was mainly getting used to the schedule of how it all worked, signing in everyday, waiting in a particular area, hoping that there wasn't a delay on my particular machine etc. It's incredible the cognitive load that the "unknown" puts on you. But once into the routine it was fine. The only real side effects I had in the first week was tiredness, and getting used to staying perfectly still when I had the mask on. It was also trying to get used to the machines and what exactly they were doing. As mentioned earlier, My eyes were shut for the whole process so I had no idea what exactly was happening around me for 8-9 minutes, I only had the sound of the machine to rely on. For my first couple of sessions it felt like the whole machine was rocking backwards and forwards because I had no point of reference. With my eyes shut and touching nothing but the "couch", as soon as your brain thinks you are moving it is very hard to stop it from thinking it! Trust me I tried multiple times! I'm just very glad I don't suffer from claustrophobia because if I did the whole 6.5 weeks would be a nightmare as that mask was very tight! Sometimes so tight it left an mesh imprint on my nose and forehead.

Week 2: 25th - 29th October

Week 2 was mainly travelling in and out via hospital transport so the days were long and it very much started to feel like "groundhog day". The funny thing about Hospital transport is the drivers are all volunteers, some were very chatty, others wouldn't say a word for the whole journey. I often found the drivers were chatty on the journey back, when I was tired after hours of being at the hospital. Small talk is difficult at the best of times, but 6 weeks after major surgery and radiotherapy it is a struggle! The main side effect from this week was I noticed I'd started to get a distinctive tan on my forehead from the radiation. It was like going on holiday without all the fun.

Week 3: 1st - 5th November

Week 3 was a tough week. The whole process was getting very repetitive and I was only just getting to the half-way point! This is also the point at which I noticed my hair was starting to fall out.

I was having a shower one morning and realised there was hair in my mouth. I quickly spat it out then a few seconds later realised there was hair in my mouth again. It was only when I washed my hair I noticed whole clumps of hair were coming out from the inside of my scar. I'd been expecting this as they'd warned me about it, but in the back of my mind I thought it may not happen to me. So it's still quite a shock when it does happen!

Week 4: 8th - 12th November

Groundhog day continues and I notice more of my hair is falling out, this time from the other side of my head. I guess because of the geometry of how they are targeting Gary, they are hitting it from multiple angles. So I ended up having hair at the back and sides of my head and very little in the middle, with a small patch of hair right at the front of my head. It looked ridiculous so I decided to shave my head completely. This completed my full on "cancer look", and it certainly shows off my scar a lot more!

The one thing I notice is how much colder my head actually is! You wouldn't have thought such little hair would make a difference, but it really does! Thankfully I have a few hats that I wore both inside and outside the house.

Week 5: 15th - 19th November

At this point I'm well and truly over the whole process. The journey in and the appointment takes up the whole morning, then I'm sleeping in the afternoon. So by the time I wake up it is already getting dark. So the day is gone and I've got nothing to show for it. So incredibly frustrating!

Week 6: 22nd - 26th November

Thankfully getting closer to the end of the whole process now, but I'm starting to feel incredibly drained. It got so bad at the end of the week that on Friday Claire had to phone the emergency number as I felt like I was about to have a seizure. This is something they had warned us about but I hadn't had any seizure-like symptoms since before surgery. After lying down in a dark room for an hour or so and taking some extra medication that the oncall oncologist had recommended I felt a little better. But it felt depressing, as it was almost like I'd regressed weeks rather than moved forwards.

Week 7: 29th November - 2nd December

The final week! In the weekly Tuesday consultation with the consultant / senior nurse, it was decided that I was going to be kept on stronger seizure medication in the evenings for the foreseeable future. Thankfully I haven't had any huge side effects on this drug so I didn't mind. As long as it keeps the seizures away!

Radiotherapy Ends - Thursday 2nd December

In my final session on the Thursday I took a whole bag full of chocolates in for the nurses and reception staff as they'd all been so helpful and friendly for the past 6 weeks. They seemed very happy with them. I guess they must get it quite a lot. Since it was my final session I asked if I could film my radiotherapy session as I was curious as to what exactly the machine was doing, having had no idea for the past 32 sessions. Thankfully they allowed it.

Once the machine passed over me for the final time I remember clenching both my fists and saying a little "yes!" to myself, 6.5 weeks and 33 sessions over! And the final time I'd need to put on the uncomfortable mask! You get the option at the end about if you want to keep the mask and take it home. I decided I wanted to keep it. I'm not sure why. Maybe I can pass it onto my kids and grandkids in the future? Maybe it will become a family heirloom that they can one day take onto the Antiques Roadshow! But for now it sits up in the loft so as not to scare the hell out of them!

We had a little celebration as a family, a mini post-radiotherapy party with party plates, hats and balloons too! No jelly and ice cream though. Probably a good idea as I'm trying to cut down on sugar!

What's next? - Friday 3rd December

After 6.5 weeks of going to the hospital everyday, Friday was unusual. I didn't need to be anywhere or subject my head to both an internal and external suntan! I think I spent most the day sleeping trying to recover from the cumulative effects of the radiotherapy then had a takeaway (Indian) in the evening. All in all the treatment had been long and repetitive, but the side effects I had were all manageable.

The one thing I would say about the whole experience is it's quite lonely. That's not to say I'm not surrounded by friends,family and colleagues (past and present) but at the end of the day it's me who is waiting around and then me lying in the machine(s) for the treatment and you can only ever do that alone. It certainly doesn't help at the moment that Covid restrictions don't allow anyone to be with you in the hospital. So if you are waiting 1.5 hours for a delayed machine. You are waiting alone.

But I'd like to say a huge thank you to all my friends / family and the wonderful NHS Transport service for getting me too and from each of my sessions over the 6.5 weeks.

So what next? Well I have approximately 6 weeks off over Christmas then should start 4-6 cycles of PCV chemotherapy some time mid-January. All being well this will last approximately 7 months. Well that's a lot of 2022 planned for then! I'm hoping that around March they will be able to do another MRI scan of my brain and tell me what difference the radiotherapy made to Gary's leftovers. Apparently it takes around 3 months for the brain to recover and repair after radiotherapy, and until it does they can't see on the scans if it has made a difference.

So watch this space for more updates in 2022, where the remaining 5% of Gary is hopefully evicted from my head!

Oncology Meeting - Wednesday 29th December

So since my last update at the start of December, not much has really happened. I've had 5-6 weeks off, had a good Christmas with family and tried not to think too much about 2022 (although I must admit it's almost impossible). As mentioned before, the plan moving forwards was radiotherapy, followed almost immediately by chemotherapy. At my initial follow-up meeting back in September very little was mentioned about the chemotherapy. This was intentional by the team, since getting my head around radiotherapy was enough for me at that point! So I've only had a few basic details about it all. This meeting was to tell me a lot more.

As with most of my meetings, it was a video call, it included two oncologists and a senior nurse. They went through a few of the details about the type of chemo I'm going to be having (it is called PCV), PCV stands for:

P – Procarbazine
C – Lomustine (CCNU)
V – Vincristine

These are the three types of drugs I'm going to be taking for the next few months. Vincristine is taken intravenously, the other two drugs are prescribed to me and taken in pill form at home.

I'm going to be taking these drugs in what are called cycles. Each cycle is 42 days long, and I will have up to 6 cycles (assuming my body allows for it). A cycle consists of:

Day 1 - Go to the hospital and sit while they give me Vincristine via a drip.
Day 1 to 10 - take the Lomustine and Procarbazine every day at the same time each day
Day 11 to 42 - This is a rest period for my body to recover from the chemotherapy drugs.
Repeat this process up to 6 times if my body allows for it.

Now, the difference between the radiotherapy and chemotherapy is that radio is very targeted (hence my ‘sunburnt' head!). But chemotherapy isn't targeted at all, it is given to me and can have an impact on all the cells in my body. If you take a look at the potential side effects this is the reason why there can be so many!

A big negative of chemotherapy is it can have quite a big impact on my immune system. It reduces the white blood cell count in my body. These cells are used to combat infections. So the less white blood cells you have, the more susceptible to other illness'. This is what I mean when I mention "if my body allows for it". At the start of each cycle, the hospital looks at my white blood cell count to determine if I am healthy and able to continue with the next cycle. If not then they may lower the drug dosages or if it's terrible, pause the chemotherapy completely. Each and every person's body reacts differently to the drugs, so I won't know until I'm into the first couple of cycles as to how my body reacts. So it's another journey into the unknown!

Six cycles at 42 days each, so I'm looking at around 7 months of chemotherapy! That's quite a long time to try and not get sick! Especially during a global pandemic! Speaking of which, I asked about Covid and If I do end up catching it the NHS are able to give me special drugs to help my body combat the virus.

So it's people in my position across the world who are in danger when a huge portion of a population refuse to take the vaccine. As if life isn't bad enough when you have cancer, only to find going out people around you could make you even sicker! So yes, for the next 7 months I'm generally going to be avoiding people I don't know, and asking friends and family I see to take a test before meeting with them. Not a great position to be in! I'll get off my soapbox now…

The only disappointing part of the meeting was they were unable to give me specific dates for my treatment or the hospital it would be at. This isn't the fault of the doctors, it's just a scheduling and administration bottleneck given the pressure the NHS is under at the moment. They said I'd be receiving a few phone calls in the coming days to confirm dates etc. Until then, we wait!

Houston, we have dates! - Thursday 30th December 2021

Well, thankfully I didn't need to wait long for the dates. I received a few phone calls the day after the meeting:

Monday 10th January - chemotherapy pre-admission
Tuesday 11th January - start of cycle 1

In the previous meeting, they also mentioned that I'd need to have another MRI scan to see what effect the radio has had on the remaining 5% of Gary, but they said this could take a while to schedule, and I may have started chemo before it actually happens.

Death to 2021 - January 1st 2022

As I said at the start of this very long blog post. Just when I thought 2020 was bad, 2021 comes along and proves me wrong! It has undoubtedly been one of the worst years of my life, but that being said I've achieved a lot (well up until August anyway!). I worked up until 2 weeks before surgery, so I managed to pack quite a bit into the first 8 months. My year in review:

January: Spoke at FOSDEM21
February: Spoke at London Web Performance
March: Cloudflare Worker experimentation blog posts
April: Diagnosed with a Grade 3 Brain tumour
May: Spoke at Web Directions Lazy Load
June: Lots of Hospital appointments
July: Missed the F1 at Silverstone due to the Covid risk before my surgery date.
August: Stopped work before surgery
September: Had awake open-head brain surgery
October: Recovery + Radiotherapy
November: Lots of radiotherapy
December: Turned 40 and got a Puppy

Not bad, all things considered! Here's hoping for a better 2022! Fingers crossed the final 5% of Gary gets evicted from my head!

MRI Scan scheduled - Thursday 6th January 2022

So much for it taking a while to schedule the MRI! I received a phone call asking me to come in to hospital the next day (Friday) for my MRI scan. Not much notice, but the good thing (I hope) about this date is they should have the scans back for my pre-admission meeting, so you never know they may actually be able to give me an update on how the rest of Gary the prick is looking!

MRI Scan - Friday 7th January

You'd have thought by now that the thought of lying still in a tube doesn't phase me, but it still does. There's nothing particularly scary about it. It's very loud and a little uncomfortable (and very boring). But other than the cannula for the contrast they inject, it's not that unpleasant. Even so, I still get a little nervous before it happens. I think it's because when you are in the machine alone with your thoughts, there's no way to ignore the fact that you have cancer. It isn't a bad dream that you are going to suddenly wake up from. It's actually that you are lying in this machine (again) for 25 minutes!

Anyway, the scan was uneventful. Just another hurdle to jump over in the whole process!

Chemotherapy pre-admission - Monday 10th January

The meeting was a bit of a disappointment. I was expecting there to be a bigger team to discuss what was going to be happening, but in the end it was "only a nurse" (her words, not mine!). I managed to get a few questions answered, but I have many more for the consultants / doctors.

Unfortunately, due to it being a specialist cancer centre, Claire was unable to be at the meeting with me. She had to sit in the car and listen in while I had her on speakerphone. This is due to the current Omicron covid variant restrictions. The team mentioned that if I'd started just 2 weeks earlier, it wouldn't have been an issue. But they've had quite a few patients partners walk into the centre who knowingly (or unknowingly) were covid positive, which obviously puts all patients at risk. So now they only allow patients in for meetings and treatment. And as a patient, you have to bring a negative lateral flow test from that day with you before you are allowed to be admitted into the centre.

On a more positive note, I've got all the dates for my cycles. So if all goes to plan, and I manage all 6, my ‘last day' will be on the 20th September. Although I have no idea how long chemotherapy drugs stay in your system, so they may linger for a little while after that??

Chemotherapy cycle 1 start - Tuesday 11th January

I was incredibly nervous about the chemotherapy, I'm not 100% sure why, as the doctors had mentioned that it would likely not be as bad as the radiotherapy in my instance. I think it's probably due to the longevity of the treatment. The surgery was scary but in all honesty if something went wrong I doubt I'd have known very much about it, I think it would have all just ended. Radiotherapy was localised long and repetitive, but once in the room it only took 5 minutes for the treatment, then you were done for the day. But with chemotherapy, I'd be taking the treatment "home with me". With a tonne of possible side effects, and a total of up to 6 cycles lasting up to 7 months, I'm actually going to feel the effects of this for a long time into 2022. There's pretty much no avoiding getting ill in some way once started.

Thankfully, the journey to the cancer centre is nowhere near as tedious as the hospital trips for radiotherapy, and I only need to go once every 42 days, the rest of the cycle is at home. Once I'd chosen a chair to sit in, the nurse came over and checked I was who I said I was, then explained what was going to happen (since it was my first cycle).

I was given the vincristine intravenously into my arm, then while that was happening I'd speak to the pharmacy team about the rest of my drugs. Unfortunately, the drip has to go into the back of the hand or wrist for this type of drug. This is because vincristine can cause damage to skin when outside of veins. The inside of my elbow (where'd I'd prefer it to be given) has a void behind the vein, so if there were a leak then the nurses wouldn't know about it! Not ideal, but I can see why it is done! So in the end we managed to come to a compromise where I was cannulated on the side of my wrist behind my thumb.

Once the cannula was in my wrist, the drip only took 5 minutes to give me all the medicine it needed to. In this time, I spoke to the pharmacy team with Claire on the speakerphone, so she could listen in (she was sitting in the car due to covid restrictions). It was only then that I realised the sheer number of pills I was going to have to take at the start of each cycle. A minimum of 69 tablets over the first 10 days at different times of day, on top of the seizure tablets I already take! I'm very glad I don't have problems taking tablets!

Once the drip had finished, that was it. I was sent home with a small green bag full of all the medication I'd need for the next 41 days. Before the process starts all over again!

Side effects It's been just under a week into my first cycle as I'm writing this, and I have a pretty good idea as to how my body is reacting to it all. The first two days were pretty awful, very tired and lethargic and just generally not feeling 100%.

But It was into the third day that I started to feel the main side effect I've noticed. Eating my breakfast on the third day, the milk tasted metallic and my tongue / mouth were extremely sore. I'd love to say that this has improved, but it has actually got worse over the past days. Nothing I eat or drink tastes the same, and it has put me totally off food. It feels like I have a mouth full of ulcers. Not pleasant! Apparently this is a side effect of the vincristine so should hopefully diminish over time.

The only thing that has alleviated the pain is taking regular dosages of painkillers (more pills!), Corsodyl mouthwash, and Difflam mouthwash.

If anyone is in the same position, here's a list of the foods I've had so far that still taste like anything palatable:

Rice Pudding
Custard
Baked Beans
French fries (no salt)
Steak pie
Boiled (or mashed) potatoes
Sausages
Gravy

I'll keep this list updated as I find more foods that taste okay, I hope there are more, else I'm in for a very boring 7 months of food!

On a positive note, I haven't experienced any nausea, which is what I thought may happen. But then again, I am taking 3 types of anti-sickness tablets! So maybe they are doing their job at the moment and I have that to look forward to later in the cycle? I hope not!

1p/19q Codeleted gene update

I mentioned 1p/19q Codeleted gene) before which will effect how effective the chemotherapy is against Gary, Thankfully it turns out that I do have this gene mutation, this is a good thing as it should mean that the chemotherapy should be more effective in my case! Yay for genetics!

Oncology Update - Wednesday 19th January

I was told this meeting was to discuss the results of my MRI scan, so I was quite nervous in the days leading up to it. But in the end it turned out to mainly be a catchup about how cycle 1 was going. I updated the team on the side effects I was experiencing and asked them a number of questions, one of these was how my brain was looking on the scan. Due to the fact that my brain is still healing from Decembers radiotherapy, they couldn't tell me too much. But they did say that the tumour was looking "stable", which I guess is good! They said that my next scan (date unknown) should tell us a lot more, as it will be past the 10-12 week cut-off for my brain healing from radiotherapy.

Chemotherapy cycle 1 end - Friday 21st January

So technically this isn't the end of the cycle, but it is the end of taking all the chemo drugs for cycle 1! 10 days and 70+ tablets later, It's been a busy week, and I've learnt a lot that I can take into the next 5 cycles.

So what have I learned in cycle 1?

Well, first thing is to make sure you are taking the correct anti-sickness pills! I'd mistakenly been taking the wrong ones for the first few days. When you are feeling like crap and half-asleep at 6 in the morning, this was an easy mistake to make! They both look the same in colour and shape and come with some wonderful side effects! So yes, I won't be making that mistake again!

Another thing is to start using both sets of mouthwash and dosing up on constant painkillers from day 1. The mistake I made was to wait for the side effects to become almost unbearable, before contacting the cancer triage number. Hitting it hard from day 1 will hopefully make the first 4-5 days (the worst ones), more manageable!

I've also learnt that only 1 in 10 people on PCV chemotherapy get the jaw / mouth side effects like I have! I mean, I wouldn't expect it any other way given the rarity of the type of tumour I have, where's the fun in that, huh, body? What a stupid bag of meat.

Lastly and probably most important is the types of foods I can and can't eat. After day 2 normal breakfast is out the window and replaced with food with an actual taste. By Day 5 my mouth starts to feel slightly better, at day 12 I was almost back to normal.

Now of course this is assuming the other 5 cycles are exactly the same as the first, but the Oncology team did say there would likely be a slight cumulative effect as the cycles progress, so prepare myself for it to get worse over time!

All in all, though, given the number of possible side effects, I feel I've got off lightly for my first cycle. I'm just hoping it stays that way for the following 5! But that's totally out of my control. Here's hoping my white blood count stays high enough to continue, and I don't catch any illness' over the next 8 months! This may be a big ask given I'm currently starting a phased return to work!

Ongoing work - Months of March / April 2022

Now, considering I've only been working a limited number of hours during my "phased return to work" period, I've felt It's been very productive. Obviously not 100%, but I set myself a daily set of tasks I'd like to complete, then anything that doesn't get done gets "bumped" onto the next day or later in the week. Obvious really when you think about it, but I'm really having to be that strict with myself, else I know I'll end up doing full days again and burning myself out! Not good for me (or work either!). I've been logging all my working hours and what I've been doing to see what pattern emerged over my "recovery periods". Interestingly, you can see a pattern. As time passes and the drugs work out of my system, I'm able to work slightly longer and achieve more each day. Some highlights at work for the past months include:

Recruitment: The community and I are very much recruiting FE developers. I've been involved in all stages of this (CV sift, phone interview and face-to-face interviews).
Joiners: We've had a number of new joiners in the community. So I've been involved in welcoming them to the department and making sure our guidance around the communities new starter "buddy system" is set out and well-documented.

But the absolute highlight for me has to be the GOV.UK team has managed to remove jQuery from all Frontend apps across GOV.UK.

Sorry to go into "work mode", but this basically means 32 KB of compressed JavaScript has been removed that our users no longer need to download when they visit all GOV.UK pages. Now I know what you may be thinking, 32 KB that's nothing! So that's true for a single user, but GOV.UK see's millions of users every month, so it multiplies to a huge data saving for all our users! I tweeted about the results we saw in our performance monitoring (RUM, Synthetic) and the response was quite crazy!

The performance data speaks for itself, really. I highlighted the fact that this has improved the web performance of GOV.UK for the users who really need it (our P95 users), These are users on limited data plans and low-spec devices. By improving performance for these users actually improves performance for everyone!

So a massive pat on the back to the GOV.UK team for getting this change out the door it's been a slog but if you don't remove tech-debt it will eventually drown you and stop you from moving forwards.

We also had some sad news in the community, Alex J a fantastic Lead Frontend Developer and great friend and colleague decided to leave the department to return home to Romania. I can't tell you how genuinely gutted I was when I heard the news he was leaving, his dedication and knowledge on GOV.UK and his leadership in the Frontend community will be surely missed! I wish Alex all the best and hope our paths cross again in the future!

Sorry for that digression into work. I'm also happy that I've proved that I'm still able to work sans-Gary (the prick!). So now back to the "fun part", my cancer treatment (what an odd statement!)

Oncology Update - Wednesday 6th April (morning)

As always, a busy start to the next cycle. Oncology review in the morning, chemotherapy treatment in the afternoon.

I was genuinely nervous about this meeting, as 2 weeks before I'd had yet another MRI scan. This was the first scan that would be able to "show anything". Since it takes 3 months for the brain to "repair" after radiotherapy. In the last MRI scan, nothing could be said for certain, as damaged brain matter looks the same as a brain tumour on an MRI scan (talk about ironic!).

But anyway, the meeting went really well. My bloods looked great, and the scan was described as "stable" or in other words what is left of Gary hadn't grown. The Oncologist at one point also said that the tumour was very difficult to see! I think that's the best news I could have had! It certainly makes up for the nights leading up to the meeting where I would wake up thinking about it!

The chemotherapy would be going ahead as planned on the same dosage of Vincristine I'd had before.

Chemotherapy cycle 3 start - Wednesday 6th April (afternoon)

Nothing really to report here, only we turned up an hour early for my appointment! I think my frazzled brain got the time wrong, thankfully the cancer centre were able to fit me in any way. I had the infusion, but then had to wait awhile for my other medication (it takes time to prepare). I think my punctuality caused that issue, not the nursing team!

But once completed, I set off on my merry way with a body full of vincristine and a bag of 90+ tablets.

Chemotherapy cycle 3 end and recovery - Saturday 16th April

As I've found with the previous cycles, it starts off slowly in the first few days, then ramps up around day 3 or 4. This is while the body adjusts to the concoction of drugs sent in to kill cells (mainly the bad ones, but there is friendly fire with chemotherapy as it isn't targetted like radiotherapy).

I was able to work a couple of hours on the Thursday, which got me thinking "oh, maybe this cycle won't be so bad". Boy was I wrong about that! I think I jinxed it, so I won't be making that mistake again in the future!

I won't go into a day-by-day account, but let's just say the cycle turned out to be worse than cycle 1. I've never felt so bad for so many days with no end in sight! The fact that I knew I needed to keep taking tablets every day that were only going to make me feel even worse, is a horrible feeling. It's a real struggle to make yourself take them at some points in the 10-days! Some wonderful side effects from this cycle (in no particular order) have been:

Nausea (no vomiting, thankfully)
Weakness (as in, can't get out of bed or even move)
Headaches (thank you paracetamol!)
Light-headed and dizziness (even when sitting on the sofa)
Spiked blood pressure
Lack of appetite
Random pains all over my body
Insomnia

And to top it all off, I now seem to have developed Tinnitus in both ears, this is a known side effect of the vincristine and can be temporary or permanent (only time will tell).

So, yes, if you are reading this and are new to chemotherapy it probably isn't a fun read, but as they say "no pain, no gain" (FYI: don't say this to a person going through chemo unless you want a stern look or a strongly-worded letter!).

So what have I learned in cycle 3?

Don't go into any new cycle expecting the same side effects from the previous cycles, you are setting yourself for a fall!
Only take the anti-sickness pills if you are actually feeling sick (see my comments below on this)
Make sure you are open and honest with the team managing your treatment, keep them updated regularly! They are there to help you!
Try to focus on the recovery period, it takes a while for the drugs to leave your system. It can take up to 6 weeks for some people (it's likely you won't feel "normal" again until this happens).

Just to jump back to point 2, there's conflicting information here. You are told to take them only when you feel sick, but also that it is easier to cure the "feeling of sickness" than the sickness itself! So yes, get in a time machine and see if you are likely to feel sick or be sick, then choose the correct path!

The reason why this point is so important is that each drug you take has its own unique set of side effects for your body. I found the anti-sickness pills were making me extremely weak and tired, so I stopped taking them midway through the cycle. Only then to find that the last few days of the cycle I'd suffer from terrible Insomnia, because I no longer had the weakness and tiredness from the anti-sickness pills! It's a minefield! I'll be mentioning all this at my Oncology review meeting at the start of Cycle 4 (May 18th 2022), maybe they can adjust my drugs in some way?

To end on a more positive note, I'm now over half-way through! 3 cycles down, 3 to go! Roll on October, and at some point back to "normality".

So as before I likely won't be updating the post again until after my next cycle ends (end of May), unless something drastic happens!

Diagnosis Anniversary - Thursday 21st April

So here's an anniversary that I never thought I'd be ‘celebrating' in a million years, but here we are. Celebrating is totally the wrong word, but it is a significant date, as 1-year ago today it felt like the whole ground fell from under me. My life, what was once a path paved and walked day by day, suddenly became a fog of unknown obstacles and fear. So many questions that my whole family and I had no idea about! I mean, why would you read into brain cancer if you didn't need to?!

It has been both the longest and shortest year of my life, when you live it day-to-day everything drags! Radiotherapy is relentless, a 6-week-long tidal wave that never seems to end. Avoiding everyone during chemotherapy is depressing because you are scared to even catch a common-cold! It's been a real rollercoaster of emotions!

Some particular points that stand out to me:

Scariest moments

Seeing the size of Gary for the first time over the video call with the surgeon!
Closely followed by heading (hah!) into the surgery for my awake craniotomy, Not knowing if I was going to wake up parallelised, partially-parallelised, or even wake up at all!

Low points

Obviously being diagnosed with brain cancer is up there at the top of the list, but also telling my wife, and parents and sister of the diagnosis.
The week between initial diagnosis and actually having more detailed information from the hospital (that was a long week!)
Seeing the look on my mom's face when we were initially shown the size of Gary.
Having those horrible conversations with my wife about what to do if the "worst should happen"
This list could go on for a while, so I'll spare you and stop at 4!

High points

Realising there were lots of options available, and I shouldn't see this as a "death sentence".
Seeing the outpouring of support both personally (family & friends), at work, and across the whole web community when I first announced my diagnosis.
Being told they'd been able to remove 95% of the tumour with little to no-damage to any of the good parts of my brain.
Finishing radiotherapy after a lot off back-and-forth to the hospital every week day for 6 weeks!
Receiving the UCS Lego Millennium Falcon from some very kind friends in the Web Performance community (build in progress at the moment!)
Realising I can still think (and work) in the post-Gary world ("full recovery" may take a while, but I can do it!)
My wife raising over £2,700 for The brain Tumour Charity for her entry into the 2022 London Marathon. Thanks to everyone who donated, you rock!
Seeing how we have come together as a family to cope with the scary and ever-changing world of the cancer diagnosis and treatment process.

To jump on the last point a more. I can't thank my close family and friends enough over what has been an incredibly tough year! I certainly couldn't have made it this far without the daily support of my wife, kids, parents, sister and her husband, and my mother-in-law.

Admittedly, my kids are the absolute perfect age to drive me to distraction, but that's exactly what is needed. Life would be so boring without their daily arguments and tantrums!

And last (but certainly not least!), a huge thank you to the help, support and dedication of the whole of the NHS workers I've interacted with over the past year (so so many individuals!). Without them and modern medicine, I hate to think where I would be now, 1-year into my diagnosis!

Treatment Tally Table

I thought I'd pull together a tally of all the treatment I've had along the way in my quest to evict Gary from my skull, which I will keep updated below.

As I've said before, thank you again to the NHS. All this would have cost a fortune in certain countries without universal healthcare (sorry American readers)!

Treatment	Number	Finished?
Radiotherapy	33	Yes
Chemotherapy Cycles	3	No
MRI Scans	5	No
CT Scans	3	No
PET Scans	1	Yes
Major Surgery	1	Yes?

Post changelog:

21/04/21: Tumor first diagnosed.
12/05/21: Initial post published.
15/05/21: Posted 3 new updates from the week. Next appointment on the 20th May.
19/05/21: Added information about another seizure.
21/05/21: Added information about the pre-operative assessment appointment.
25/05/21: Added update about the weekend and another seizure I've had.
29/05/21: Added information about obtaining my scan data and a different type of seizure.
03/06/21: Added update on date for memory test and overall symptoms.
07/06/21: Added steroid and reduced seizures update.
10/06/21: Added update from latest Neurologist meeting.
11/06/21: Update on first 'core feature' seizure I've had in 2 weeks.
19/06/21: Added an update about the sleepy seizure and an epiphany.
18/07/21: Added an updates about the month of July and "Between Life and Death".
23/07/21: Update on the latest timings and the surgery info I've been given.
26/07/21: Pre-surgery MRI scan update.
01/08/21: Scan results and swelling symptoms update.
06/08/21: Back on the 'roids and finally a productive day!
12/08/21: The highs and lows of steroids and finally some information on an operation date!
23/08/21: Added info about a surgery date and the fact I had another seizure.
27/08/21: Surgery cancelled, and an update on the very long week.
04/09/21: New surgery date is looking promising update, fingers crossed it will be next week!
08/09/21: Added update on the scans today and my final post for a little while.
21/12/21: Added a huge post-surgey update on the recover, radiotherapy and future plans.
17/01/22: Added an update about my chemo meetings and first cycle. Also added a tally table for all the treatment I have recieved so far.
17/01/22: Added an update to my 1p/19q Codeleted gene result, thanks to Miles Mandelson for reminding me about this result.
24/01/22: Added update on Cycle 1 and 2 of chemo and the oncology meetings
19/04/22: Added update on Cycle 3 recovery and return to work.
21/04/22: Added update on the 1 year diagnosis anniversary.

How to use WebPageTest's Graph Page Data view

2021-04-13T00:00:00Z

WebPageTest is a mysterious beast. You can use it almost every day for years and every so often a link catches your eye that you've never noticed before. It then leads you to a whole new set of functionality that you didn't know it had. It's happened so often I've actually written a whole section about the 'hidden gems' in my 'How to read a WebPageTest waterfall chart blog post. In this blog post I'm going to focus on one of those 'hidden gems': the 'Plot Full Results' link that sits under the performance results on the 'Summary' tab. Unsure what I'm talking about? See the image below:

Clicking on this link will bring you to a page with a number of options and graphs available to you. It may be a little confusing as to what you are seeing at first, but let's go through it together.

Context

First let's cover what this page is doing, as there isn't much on the page to really explain it. This page view exists to give you an overview of all the test runs that WebPageTest ran for the site you are testing. WebPageTest will pick the results from the median test run for either the 'SpeedIndex' or 'Load Time' metrics. By default it will use 'SpeedIndex' to choose the median run.

In the image above the data being shown in the table is from 'Run 2' of 3. The reason for multiple runs is because loading the same page under what seems like the same conditions, often returns wildly different results. This is the non-deterministic nature of the web. Variations in network traffic between the test agent and the website being tested (as well as many other factors) creates these variations in the end result. In order to reach a result that accurately represents the actual performance of a website at the given time, WebPageTest runs multiple test runs. It then picks the test run slap bang in the middle for the chosen median metric as the representative run.

This is the reason why you should run as many test runs as possible when using WebPageTest. On the public instance of WebPageTest you are limited to a maximum of 9 test runs. But if you have a private instance you can run as many test runs as you like. Just remember to always pick an odd number of runs, since you are wanting WebPageTest to return the median run. An even number of tests has no median test run, since dividing the total number of runs (minus 1 for the median) by 2 will return a non-integer number. e.g. (8 runs - 1) / 2 = 3.5. How do you pick the 3.5th test run?

But what about if you are interested in the results from the other runs in the test, not just the one WebPageTest has chosen? Well this is where the graph_page_data.php page helps. It gives you information about important metrics reported across all test runs that completed.

The graphs

So I'm going to describe this in reverse compared to the actual page layout, as I think it's easier to explain once you have an idea of what metrics are being plotted in the graphs. The complete list can be seen below:

Web Vitals - First Contentful Paint (FCP)
Web Vitals - Largest Contentful Paint (LCP)
Web Vitals - Cumulative Layout Shift (CLS)
Web Vitals - Total Blocking Time (TBT)
Load Time (onload)
Browser-reported Load Time (Navigation Timing onload)
DOM Content Loaded (Navigation Timing)
SpeedIndex
Time to First Byte (TTFB)
Base Page SSL Time
Time to Start Render
Time to Interactive
Time to Visually Complete
Last Visual Change
Time to Title
Fully Loaded
Estimated RTT to Server
Number of DOM Elements
Connections
Requests (Fully Loaded)
Requests (onload)
Bytes In (onload)
Bytes In (Fully Loaded)
Custom Metrics included below.

There are a total of 24 metrics tracked in total (at the time of writing).

Basic graph

Let's focus on a basic graph and examine it:

The graph is for the last visual change metric. Along the horizontal (x-axis) we see each of the 9 runs from our test. Up the vertical (y-axis) we see the time measured in milliseconds. The value from each test run for this metric is represented by a red dot on the graph. As you can see, there's quite a variance for this metric over the 9 runs. The lowest being run number 2 at 9,043 ms. The highest, run number 5 at 10,920 ms.

But each graph comes with an associated table that extracts important statistical information about the metric across all runs.

Basic table

Below we see the table associated with the graph seen above. Last visual change metric over 9 runs.

Let's go over each of the columns one by one:

Mean

This is the arithmetic (population) mean, or also commonly referred to as the average value of all the test runs for a selected metric. For this you add all the metric values together and divide by the total number of runs. It's worth mentioning that the mean can be easily distorted if you have extreme variations higher or lower than the expected value (i.e. a high standard deviation).

Median

I touched on the median value a little earlier in the post. The median (or p50) is the value of the metric that falls right in the middle of the set of results. Each set is the total number of test run by WebPageTest.

p25

p25 stands for the 25th percentile. In web performance terms this is basically saying: "25% of users will have a score for this metric that is better than this value". Users in this region are likely receiving a good overall experience.

p75

p75 stands for the 75th percentile. The same sentence as used with p25 works here: "75% of users will have a result for this metric that is better than this value". By improving the performance of a metric at p75, you will be improving it for all users sitting at, or beyond it in the distribution curve (e.g. the low performing 25% long-tail users).

p75 - p25

In statistics this is known as the interquartile range, or IQR. This is giving us an idea of the difference between the 75th percentile and the 25th percentile. If this value is small we know the values for the metric are all fairly close to the median.

In our table we can see that if the last visual change metric value increases by 807 ms, it slips from the 25th percentile to the 75th percentile. IQR is a good measure of variation because it doesn't take outliers into account. They are removed from the calculation since it is only focussing on the inner 50% of the data where most results will be clustered.

In terms of web performance we want to minimise this value. We want to pull p75 users up closer to p25 users so everyone receives a better experience.

StdDev

This stands for 'standard deviation', which is represented with the Greek letter σ (sigma) . It tells us how spread out our data set is from the mean. The standard deviation can be used to tell us if a result is statistically significant, or part of the expected variation. See the "68–95–99.7 rule" for more information on how standard deviation can be used to see if a result is expected, or an outlier.

A low standard deviation means the data is clustered close to the mean value. The data is much more predictable and the effect of variance is lower. We will see fewer outliers in the interquartile range. A high standard deviation tells us the data is dispersed over a wider range from the mean.

A lower standard deviation in a web performance context means the specific metric is more stable and predictable.

CV stands for the Coefficient of Variation. This is the measurement of relative variability. It is simply the standard deviation divided by the mean, which produces a ratio. It is presented in the table as a percentage which is an optional, but sometimes more understandable step.

CV is essentially a normalisation process. It is useful when you want to directly compare sets of results that have different measures. E.g. If test A has a CV of 8%, and test B a CV of 15%, you would say test B has more variation than test A, relative to its mean.

In summary, all of the above values are giving us some insights into the spread and variance of the data from the test runs for a specific metric. For web performance, high variation is bad as it shows instability and less predictability for a metric.

Repeat view row

The 'Repeat View' row isn't seen in the above image but there is an option to run and view the repeat view data. 'First view' data comes from a browser under 'cold cache' conditions. By that I mean all assets need to be downloaded, nothing is cached.

If you have 'Repeat View' selected there'd be a second row below with another set of results for when the browser loads with a 'warm cache' scenario. Comparing these two rows of data gives you some insight into the effect caching has on the metric being examined.

Configuration options

Now that we've gone over the basic graph and table for a specific metric, it's a good time to look at the configuration options available to us at the top of the page:

Let's examine these one by one.

View (First/Repeat)

Pretty self-explanatory really. Which results do you want to be plotted to the graphs? I'd say most of the time you always want the first view results plotted to see the metrics under 'cold cache' conditions. If you have run a repeat view, then you can enable this to see how the same metric changes under 'warm cache' conditions.

Median (Of plotted metric)

When selected, WebPageTest will draw a line on the graph through the run that has been selected as the median.

In the graph above for Speed Index we can see the median run that has been selected is from run number 3, and it has a value of approximately 7,750.

Median (Run with median metric SpeedIndex)

By default, the median run for the graphs is selected by examining the Load Time metric. The run that sits in the middle of all the runs is selected as the representative run for this test. But you can use another metric. Instead of looking at Load Time, we can tell WebPageTest to look at the SpeedIndex metric for the representative run.

In the graph above we can see the difference that changing the median metric has on each metric. Two lines are now shown, one for when Load Time is used, the other for when SpeedIndex is used:

Load Time: run 9 is selected which has a value of 12,558 ms
SpeedIndex: run 3 is selected which has a value of 11,706 ms

This functionality is useful as it gives you a quick way to examine what the difference would be to all the metrics if you change the metric used to select the median run.

Statistical Comparison Against

So this is where it starts to get really interesting. It isn't at all clear, but you can pass in multiple test ID's into this page view, as well as associated human readable labels to make the graph keys easier to understand. This is done using the following URL pattern:

https://www.webpagetest.org/graph_page_data.php?tests=[TEST-ID-1]-l:[TEST-LABEL-1],[TEST-ID-2]-l:[TEST-LABEL-2],[TEST-ID-3]...

But don't worry if that sounds too laborious I have a tool that I'll show you later to make it easier.

What this functionality allows you to do is pass in multiple test ID's and compare statistics against each other. So for example, say you are testing out a performance improvement and you are interested in seeing how this change has affected each metric at a statistical level. It allows you to do this.

In the dropdown you can select which test run you want to statistically compare all the other runs against. In the example below I'm testing loading a page with 0% packet loss against one with 3% packet loss.

Basic graph - Multiple test ID's

Now that we've added another test ID, the graphs have changed:

Each graph now has two sets of points in different colours. We can now compare each set of test runs against each other. On the graph for the SpeedIndex metric we can see each median line has also been drawn:

Zero % PL median - run 3 selected with a SpeedIndex of approximately 7,800 ms
Three % PL median - run 4 selected with a SpeedIndex of approximately 11,000 ms

Now considering these two sets of tests are comparing the same page load under different packet loss conditions, it makes sense that the 3% packet loss has a slower SpeedIndex. But we can find more data to back this up.

Advanced Table

Once the 'Statistical Comparison Against' dropdown has been populated, a new table appears above each metrics graph. This table gives you a set of statistics that compares multiple sets of data for each metric:

You will notice that the bottom row has a set of blank cells. This is because it is the variant we are comparing all others against. First let's set some context: We are comparing two populations of data, one with no packet loss, one with 3% packet loss. The question is:

Does this packet loss have an effect on the SpeedIndex metric, and if so, how likely is it that it just happened at random?

Let's go through what each of the columns mean and see if there's enough evidence to answer the question:

Variant

Pretty self explanatory, which set of data are we looking at in this row. By default the label you provide is used, if not then the test ID is used. Due to the length of a WPT test ID, it breaks the whole table UI and makes the table harder to read if you don't add your own human readable label (-l).

Count

The number of test runs that were conducted during each test.

Mean +/- 95% Conf. Int

This is a confusing cell when you first look at it, but it actually contains 2 numbers. The first is the mean for the metric, the second is the value above and below the mean that covers the 95% confidence interval (CI). This is known as a normal based confidence interval. The 95% is saying that: '95% of the data for the test will sit between this range'. The true value for the mean if we had a larger sample size (by having more test runs), has a very high probability of sitting within this range. The probability of observing a value outside of this range is less than 0.05 (or 5%).

Translating this directly to the table we see above:

0% packet loss has a mean SpeedIndex of 7,583 ms, and the 95% confidence interval ranges from 6,744 ms to 8,422 ms
3% packet loss has a mean SpeedIndex of 11,041 ms, and the 95% confidence interval ranges from 8,894 ms to 13,188 ms

Diff of mean from [selected test]

What's the difference between the means for each set of tests. In the table above we see that 'Three PL' has a mean value that is 3,457 ms higher than that of the 0% packet loss test.

p-value (2-tailed) P-value stands for percentage value. The p-value is the probability of obtaining test results at least as extreme as the results actually observed. 2-tailed means we consider values on both ends of the distribution curve (extremely high, or extremely low). The lower the p-value, the more meaningful the result because it is less likely to be caused by noise.

In the table above we see a probability of 0.006 (or 0.6%) is extremely low. So the likelihood of this result randomly occurring is very low.

Significant? Whether a result is considered significant depends on the p-value we set for significance before we start testing. In the case of WebPageTest, it uses the 95% confidence interval, this p-value is set at 0.05, or 5%. This corresponds to the 5% that sits "outside" our confidence interval. So there's a 5% chance the results beyond the 95% confidence interval occurred at random.

In the above table the data has a p-value of 0.006, which means the results are significant, as the p-value is below 0.05 we set for significance. This is why the value in the table is 'TRUE' rather than 'FALSE'.

Or to summarise all the above: the fact that the Speed Index metric for the 3% packet loss test has a much higher mean than the 0% packet loss data isn't a random occurrence. It's a significant result that we should pay attention too, and we can feel confident that it's real. It isn't a result that is likely to occur just from random variance.

WebPageTest Compare Tool

So I mentioned a tool to simplify URL generation. Well back in April 2020 I created a very basic tool for generating comparison URLs for WebPageTest which I blogged about here. The tool is available at https://wpt-compare.app/. Since writing this post you are reading now, I've added a new bit of functionality that will generate both the filmstrip URL (/video/compare.php) and the graph comparison page (/graph_page_data.php).

To use it simply enter the test URLs and then give each test a more readable label. Click 'Generate URLs' and follow the links provided.

Real-world example

So let's take a look at a real world example and see what the data from the /graph_page_data.php page shows us. Here's the test setup:

URL: https://www.lenovo.com/gb/en/
Comparing: Gzip vs Brotli compression
Browser: Chrome Desktop
Connection: 3G (1.6 Mbps/768 Kbps 300ms RTT)
Test runs: 201

The reason for choosing this site is because it has a high number of assets compressed using either Gzip or Brotli compression. I counted 40 in total. I used the list that Paul Calvano posted on Twitter here to choose an appropriate website. 201 runs is a lot, so this was run on my own private WebPageTest instance (where there's no limit on the number of runs you can complete). If you try the same, just be prepared to leave it running overnight!

Testing Brotli vs Gzip compression with WebPageTest is very easy. Just add the following script in the 'Script' tab when running the gzip test to force gzip compression:

addHeader accept-encoding: gzip,deflate
navigate %URL%

This adds a request header to all requests that overrides the browsers default. In doing so we are now essentially telling the server we only support gzip compression, so it doesn't serve us brotli compressed files.

Update: Removing DNS lookup noise

After posting this blog post to Twitter I had a really useful conversation with Tim Vereecke and Andy Davies. Tim raised an excellent point that depending on DNS performance between the runs, this could be adding noise to the results. After all, the DNS performance is out of our control and really has no part to play in the results we are trying to examine. So if we could remove it then we'll have a cleaner set of results. Thankfully this is entirely possible using WebPageTest scripting:

setDnsName www.lenovo.com www.lenovo.com
navigate %URL%

So what is this magic that is happening here? Thankfully Patrick Meenan was on hand to explain how this works:

setDnsName does lookups for the second param then stores them as overrides in the hosts file, effectively caching the DNS lookups locally.

But we can do better than this. That's when Andy suggested a more generic (and flexible) version of the script:

setDnsName %host% %host%
navigate %url%

This does the same as the first script, only you are no longer hard-coding the URL's. navigate %url% will take whatever URL you have added to the test URL input box and use it within the script.

I reran all 402 test runs to see what difference it made. And it really has made quite a difference to the results. So I've added a second version of the graphs to the metrics that were affected by this change.

A stable metric: SSL Time - with DNS lookup

First let's take a look at a graph from a stable metric.

First thing to notice about the graph above is how grouped together the results are. Both the green and red median lines are pretty much on top of each other. And this is confirmed with the values for the means and the CI's. Gzip's CI is plus or minus 1.14 ms. So over 201 runs, 95% of the data is going to sit between 330.4 ms and 332.7 ms. That's an incredibly small range. Brotli has slightly more variation, from 331.96 ms and 347.85 ms. With the p-value of 0.042, there's a 4.2% chance of one of the run values sitting outside the 95% confidence interval. This is a significant result, that we can feel confident is real.

Overall we can see that Brotli vs Gzip compression has no performance impact on SSL time. Which of course when you think about it, it can't. So I'm glad we now have graph data to back this up!

A stable metric: SSL Time - without DNS lookup

Since this test is examining only the SSL time in isolation, the DNS lookup time has no impact on this metric. The resulting graph was the same as above.

An unstable metric: TTFB - with DNS lookup

Let us now turn our attention to what looks like a much more unstable metric: the Time to First Byte results:

I'll be completely upfront about this graph, I'm very surprised by it. From my understanding, I didn't think compression would have an impact on TTFB at all, since I'm assuming all compressed asset versions were being cached by the CDN. But the graph shows a distinction between the two compression algorithms, with gzip having both a lower mean and median across 201 test runs. The p-value comes out as 0 (which may be an error due to rounding of significant figures, I'm unsure). Either way it is low compared to our 0.05 significance level set by the 95% confidence interval. So yes, this is a significant result that isn't just from random variation.

I have a theory about why this could be. Brotli compression at levels 10 & 11 is incredibly resource intensive. So if for some reason this is happening "on-the-fly" each time, rather than being compressed and then cached, this could be causing the difference. It's worth noting that Lenovo use Akamai's Resource Optimizer to do this compression, so I'd guess this isn't the case. But it's the only explanation I have at the moment. Any other explanations or ideas please do let me know, I'd love to find out what is going on.

Here's another point about the 'graph page data' view now worth mentioning: If it hadn't of been for the statistical analysis seen here, this would have been missed. Now ultimately it may turn out to be a false positive that means nothing at all, but at least it is now visible and can be investigated further.

An unstable metric: TTFB - without DNS lookup

Let's examine the difference DNS lookup time has on TTFB. Note: the y-axis scales are different so you can't directly visually compare.

We are looking at a mean value of 214 ms reduction (1,632-1,417) in the mean time for gzip, and a huge 441 ms reduction (1,754-1,313) in the mean TTFB for brotli. This has given us a significant result now saying that TTFB is improved by using brotli in this instance, with a 104 ms improvement in the mean over gzip.

Obvious winners: Bytes in (onload and fully loaded) - with DNS lookup

Well as we are examining compression it's probably a good idea to look at a metric where we're pretty sure we'll see improvements. The total number of bytes in:

In the above graph we see the number of bytes downloaded by the browser up to the onload event firing. As you can see, Brotli compression has saved 60 KB up to this point in the page load. The p-value is low, and the result is real since significant is set to 'TRUE'.

Moving onto the fully loaded graph:

Here we see the total number of bytes downloaded by the browser during the page load. The graph shows 2 distinct strips of green and red, clearly showing the two compression methods in action. The stats show that 44 KB was saved with Brotli compression being used. P-value is 0 meaning it is therefore significant, so is a reliable result.

In both graphs above, what surprises me is that these graphs don't just show a strip of green and red blobs across the graph. Some variance is actually shown. I have a couple of theories as to why this could be:

The CDN is under CPU load so a different version of files is served to ease the bottleneck (e.g. a gzip version is served instead of brotli)
Lenovo could have A/B testing in action on the homepage. Each bucket is slightly different meaning different assets are loaded

Again, it's good that this graph view has exposed this peculiarity. If this were happening on my website I'd like to get to the bottom of why it is happening. With this data it's now possible to investigate further.

Obvious winners: Bytes in (onload and fully loaded) - without DNS lookup

Since these graphs refer to the number of bytes downloaded by the browser, there aren't affected by the DNS lookup time. They are the same as seen in the graphs seen above.

The unexpected: FCP and Start Render - with DNS lookup

And finally let's look at a set of graphs that (for me) were quite unexpected, those related to First Contentful Paint (FCP) and start render:

In the above FCP graph we see that the mean and the median for gzip compression is actually lower than brotli, meaning the gzip page is rendering slightly quicker than brotli. Again the p-value is 0, so the results look to be trustworthy.

The same observation can be seen with the 'Start Render' which is actually totally expected given how start render is measured. So this just solidifies the FCP observation above. Brotli start render time is 330 ms slower than gzip at the mean. Again the p-value is 0, so the results look to be trustworthy.

So why could this be happening? Well it's worth remembering that web performance is a problem space with many dimensions. Fewer bytes down the wire doesn't always mean quicker. In this case it could be that brotli is slower to decompress on the device, so painting pixels to the screen is slower than gzip. Again this is another theory, but the data and analysis seem to suggest there's a significant result here that shouldn't be ignored.

The unexpected: FCP and Start Render - without DNS lookup

Let's re-examine the FCP and start render metrics with DNS lookup time removed. Note: the y-axis scales are different so you can't directly visually compare.

There's been a huge change in the mean where brotli comes out a huge 632 ms quicker than gzip for FCP at the mean. Again another significant result with an incredibly low p-value of 0.001!

Moving onto start render:

Where we really see the difference is in the mean values. In this analysis brotli is 512 ms quicker for start render than with gzip, compared to 331 ms slower from the first set of results! Again the p-value is 0 meaning this is a significant result. DNS lookup time really does have an impact on performance.

User interface updates

While writing this blog post I raised a couple of issues and including feature requests for this page view:

Compare metrics across multiple test IDs

Tim Kadlec kindly looked over these requests and added the ability to compare metrics like Mean, Median, StdDev etc across multiple test runs when more than 1 test ID is provided. So the new statistics tables look like below when passing in multiple test ID's:

In the example above we are comparing the FCP times as various percentages of packet loss are introduced to the connection.

Force graphs y-axis to start at zero

In some cases where comparing across test runs the graph would start at a y-axis value that wasn't 0. This can sometimes give the impression that the results are better than they actually are, because it amplifies the gains or losses seen. By forcing the y-axis back to 0 gives a more representative view of the data and the results. It is now possible to do this in the config options at the top of the page:

Many thanks to Tim for fixing the layout and adding these features.

Summary

In this blog post we've examined the seemingly invisible 'Plot Full Results' link that sits under the summary table on every WebPageTest result summary page. A link that you may have never even clicked on! We've examined what all the graphs and tables are and tried our hand at interpreting the results. On examining some real-world data we've pulled out a few unusual results that may have been hard to spot without this page view. We've also examined the difference that DNS performance can have on your test results. So consider removing this noise for your own testing in the future.

So next time you run a test using WebPageTest, why not click the 'Plot Full Results' link and see what it tells you. Thanks for reading.

Note: It's been far too many years than I care to admit since I studied statistics, so there's been a fair amount of 'on-the-job' learning happening while writing this post. So if you happen to spot anything that is totally wrong, or just poorly explained please do let me know and I'll correct it and give you credit in the 'post changelog' below.

Post changelog:

13/04/21: Initial post published. Big thanks to Tom Natt for checking and feeding back on parts of this post during the draft stage.
16/04/21: Added new graphs to show the difference DNS performance can have on the results. Thanks to Tim Vereecke for the tip, and to Andy Davies for the script optimisation.
22/05/21: Added information about the UI updates from Tim Kadlec.

Setting up Cloudflare Workers for web performance optimisation and testing

2021-03-14T00:00:00Z

I recently published a blog post about Cloudflare Worker recipes. It occurred to me after publishing that I'd offered no guidance at all to the reader as to how to get started using them. You need to do that before you can start using the recipes I'd listed. So I aim to rectify that in this follow up blog post where I go through the process step by step, including images.

The aim of this blog post is to get you all setup with a Worker, then write some code to allow us to rewrite the CSS rules on the BBC News homepage so that all the fonts render using 'Comic Sans'. A very silly objective but one that should teach you the basics.

The famous quote from Ian Malcome (Jeff Goldblum) in Jurassic Park feels very relevant right now:

Your scientists were so preoccupied with whether they could, they didn't stop to think if they should.

So let us proceed... If you already have a Cloudflare account you can skip step 1.

This is dead simple, visit this URL and create yourself an account.

Step 1a - Choose a subdomain

Your worker will require a unique subdomain. You will then create your own workers under this unique subdomain. For example in the instance below a Worker named "my-worker" would be deployed to: https://my-worker.nooshu-test-worker.workers.dev:

Step 1b - Choose a plan

Next up is time to choose your plan. As you are just starting out you could use the free plan for testing. But for only $5 per month you get a whole bunch of benefits. This post isn't sponsored by Cloudflare, I promise!

Step 1c - Verify your email

Final part of signing up, check your email to verify your new account!

I then had to complete a Google reCAPTCHA after clicking the email link to identify all the boats in the image before being redirected to the login page for the final step in verification:

Step 2 - Create your Worker

After login you will be presented with the screen below. Here we are given 2 routes. Either we create a new Worker using by clicking the big blue 'Create a Worker' button to the right of the page, or we can use the wrangler CLI tool used to wrangle your workers. I'll show both routes below. Note: you only need to use one route, so choose your preferred method.

Step 2a - Using the user interface (UI)

This is the quickest method: setup a new Worker is via the UI as it does it all for you. Once clicking on the 'Create a Worker' you will be presented with this screen. I've annotated it to highlight key areas:

The screen seen above gives you everything you need to develop your first Worker. The name of the worker has been automatically generated by the system, but it can be renamed as needed. The Worker script goes in the left panel. It comes pre-populated with a JavaScript hello world application. You can send a request through the Worker and preview the response in the right panel. A preview of the actual webpage can be seen in the 'Preview' tab. At the bottom right of the page you may notice a section that looks very familiar. It's a cut-down version of the Chrome DevTools panel. It allows you to view console messages, network requests through the worker and also a sources panel for any local sources (worker.js is seen by default). Saving and deploying the above hello world app makes it viewable at https://summer-bread-11b8.nooshu-test-worker.workers.dev (this link isn't live).

The above method works well, but it's a little restrictive. I'm sure ideally you are going to want to use your own code editor on your machine for most Worker development. This is where wrangler comes in.

Step 2b - Using `wrangler`

Wrangler is the open source CLI tool you can use to manage and deploy your Workers from your local machine. It comes with great documentation on how to set it up, but I'll cover it here too for completeness. First, I'd recommend creating a new directory for your Worker code.

Install

You can use npm to install wrangler by using the following command:

npm install -g @cloudflare/wrangler

Authenticate with your Cloudflare account

Running the following command will start the authentication process:

wrangler config

This will then ask you to enter an API Token, I go through this process below.

Create an API key process

Via the CLI Quick Start you will find a 'Get your API token' button, or you can go to it directly via this link:

This then brings you to a screen to start the creation process. Click the 'Create Token' button:

We then get the option to create a Token from a set of templates. Click the 'Use template' button on the 'Edit Cloudflare Workers' row:

The 'Edit Cloudflare Workers' template will show. Most of it can be left with the defaults. Just populate the dropdowns highlighted in red and click the 'Continue to summary' button:

Last step in the process is to review the summary page and click the 'Create Token' button:

The final page is the one that gives you the actual token you need for authorisation with Cloudflare. Make sure to copy this down as it is only displayed once for security reasons. If you don't you will need to repeat the whole process again!

When prompted in your terminal, you should paste in the API token you've just created above. wrangler will then run through the configuration process. Once complete you are now authenticated to make changes to your worker file locally.

Start a project from an existing template

The quickest way to get up and running is to use a pre built template. You can do this by running the following command:

wrangler generate my-project https://github.com/cloudflare/worker-template

This will create a my-project directory with the required worker files you need. index.js is where your Worker code exists. The wrangler.toml file is the configuration for this Worker. You should edit this file and add in your account_id. This can be found on the Workers overview page:

Once done, save the edited wrangler.toml file.

Preview the worker

Last step is to preview your new worker. You can do this by running the command below (make sure you are in the Worker directory we created with above):

wrangler preview

This will output a URL to the command line and automatically open your default browser with your new Worker visible:

Now admittedly this isn't the most exciting example, but we now have a solid base to work from. When it comes to editing and redeploying this code simply edit the index.js file with your changes then run:

wrangler publish

This command will push your changes up to the Worker that you can then view on the Worker subdomain you chose earlier. For this example it was https://my-project.nooshu-test-worker.workers.dev.

That now completes the setup of the Worker, so we can finally move onto the more interesting part: Making it actually do something useful!

Step 3 - Write some code

So my basis for the Worker code is the boilerplate code I listed in my 'Cloudflare Worker recipes for frontend performance testing', which you can find here (opens in new tab). We are going to take this code and use it as a template to modify the BBC News homepage.

What do you want to change?

So the next decision for you to make is what exactly do you want to change on the site you are testing? I've compiled some common web performance tasks in the recipes blog post, but it's completely up to you as to what you want to change. The question to ask yourself is: "What changes can I make to this page to improve performance?". Now the experiments themselves may actually make no difference at all, or even make performance worse! But at least you've found that out early in the process, allowing you to iterate quickly and try again.

So for this silly example I'm simply going to change the BBC News font from the standard font to:

font-family: "Comic Sans MS", "Comic Sans", cursive;

It's a completely pointless example but it illustrates how you can use a Worker to manipulate a page.

Working locally in the browser

So before diving into the Worker code you may wish to experiment locally first to work out exactly what changes you would like to make on the page, and persist them across page reloads. For this I'd recommend the incredibly useful Chrome feature called 'Local Overrides'. Local overrides allow you to make modifications to a page asset on your local machine, that will then persist across page reloads. So the browser will choose to use your local copy, rather than the network version from the website.

Working in the 'cloud'

Another option you have is to work within the Cloudflare Worker UI I described in Step 2a - Using the user interface (UI). The advantage with this method is you get instant feedback. You can tweak the code on the left hand side and instantly see the result reflected in the response and/or preview window. In the image below you can see that I have:

Made a request for the exact CSS file that I want to change https://m.files.bbci.co.uk/modules/bbc-morph-news-page-styles/2.4.1/enhanced.css
Set the x-host: www.bbc.co.uk header.
Set the Accept: text/css,*/*;q=0.1 header (since a direct request to the resource doesn't set it automatically).
Modified the body font-family from Helvetica,Arial,freesans,sans-serif to '"Comic Sans MS", "Comic Sans", cursive;'.

We can see the modified response body in the built in DevTools console window, so we know that the changes we're expecting are reflected in the resulting response. These changes once deployed to the Worker will modify all CSS files that pass through it., but there's a recipe where I restrict that to a single CSS file if needed.

One annoying downside to working in the UI is it's quite restrictive in terms of space for editing. So if you want to use your local code editor the complete code from the above image can be found in this gist (opens in new tab). If you prefer a version without the comments, see here (opens in new tab).

Working in your text editor

The last option, and the one I find most convenient is working from my standard code editor, since this fits in with my usual workflow. Once set up, it is dead simple to make changes. Simply edit the index.js file in the worker directory we created earlier with the changes you want, then publish the changes to Cloudflare using wrangler publish. These changes will be deployed instantly and also automatically reflected in the Worker UI we used earlier.

Step 4 - Web performance testing

So what's the point in making web performance changes if you can't measure the difference between results? This is where WebPageTest comes into play. Once configured we can easily test our website performance before and after the modifications and compare the results to see if we have improved performance (or made it worse!)

For this we are going to use some basic WebPageTest scripting:

overrideHost	www.bbc.co.uk	my-project.nooshu-test-worker.workers.dev
overrideHost	m.files.bbci.co.uk	my-project.nooshu-test-worker.workers.dev
navigate	https://www.bbc.co.uk/news

What the code is doing above is redirecting requests the WebPageTest browser is making to specific domains through the Worker so they can be modified as required. WebPageTest will automatically add a different x-host request header to the requests coming from each domain when it does this.

Simply edit the above script as needed and then run a test. If you are unsure how to run a test, I have a blog post here all about it.

Note about the `x-bypass-transform` header

So in many of the recipes written with WPT and CFW's there's sometimes a custom header (x-bypass-transform) you can add to bypass any page transformations. Using this allows for a more accurate web performance measurement since you are running the 'baseline' page (as in no changes), as well as your modifications through your Worker. This makes them more directly comparable. Unfortunately in the example detailed above CORS reared it's ugly head and broke all the font loading when any custom headers were added to the test using the addHeader WebPageTest script. So I stripped out the bypass code from the example gist for this particular test.

What I believe is happening is the custom header is being added to the font request like so: access-control-request-headers: x-custom-header-123. This is triggering a preflight request (:method: OPTIONS) to check to see if this header is allowed. The server responds back with access-control-allow-headers: * and then the font fails with a 403. This makes absolutely no sense to me at all. I assume the preflight is failing authorisation for reasons... then causing the 403. So yes, I still hate CORS! If anyone has an explanation as to why adding any header to a request ends up killing the web fonts responses, please do let me know!

Step 5 - Result

That's it, you're done. If all has gone to plan the results of your modifications should be visible within the WebPageTest filmstrip. The requests and responses have been routed through our Worker, with the response being modified on the fly before being received by the WebPageTest browser agent:

The results of my silly change can be seen below:

Before

After

The before image shows the standard page being run through our Worker with no changes made. In the after image I've swapped out the font for 'Comic Sans'. It's a simple visual example that has no performance effect on the page as all the fonts are still loaded anyway, but using the same methods listed above you can make many different changes to a webpage and test the difference it makes to the page performance. And as a bonus no production code was harmed during this experiment!

Summary

We've covered a lot in this post. Setting up a Cloudflare account all the way through to writing Worker code to modify a CSS file on the fly. Hopefully this post will have introduced you to the basics of using Cloudflare Workers for web performance testing. If you have any feedback, or recipes of your own please do let me know! P.S. this worker used in this blog post has now been deleted, so my-project.nooshu-test-worker.workers.dev will give you an error if you try it.

Post changelog:

14/03/21: Initial post published.

Cloudflare Worker recipes for frontend performance testing

2021-03-02T00:00:00Z

Table of contents

Cloudflare Workers (CFW) (or Workers) have very much become part of my workflow when it comes to examining and ultimately improving frontend web performance (hopefully!). I use them both for work and personal projects. This post I've mainly written for myself, as I've found I've been repeating certain recipes over and over again. So it feels like a good idea to have a place to log them for reference in the future. I'd love to get feedback on what could be improved, or any other examples you may have. If you do, please do Let me know!

Note: If you are unsure where to start with these recipes I've written a step by step guide on how to get started with Cloudflare Workers here.

What are CFW's?

Workers are a serverless offering from Cloudflare built upon their Content Delivery Network (CDN). They provide a lightweight JavaScript execution environment built on the v8 JavaScript engine. There are many uses for Workers. Many examples are listed on the Worker documentation page, including How to build a Slackbot, and How to build a QR code generator. But I'm going to focus on a particular area that I find them useful for: frontend web performance.

Why only frontend web performance?

Back in February 2012, Steve Souders published his now pretty famous blog post 'the Performance Golden Rule'. To quote from the post:

80-90% of the end-user response time is spent on the frontend. [So] Start there.

I highly recommend reading the post, but a quick TL;DR; is essentially: once the server has received a request, done all the database lookups, generated the HTML, and delivered the response to the browser, the rest of the page performance now depends on how it is handled on the frontend. So if you want to make large web performance gains, then focusing on the frontend is a way to do it. I'm sure there are backend optimisations you can explore with Workers, but I won't be focussing on them in this blog post (although I'd love to hear all about them).

Practical use case

To give you a practical example of where Workers come in useful I'll look at some of the work I do at Government Digital Service (GDS). The department has built and maintains GOV.UK. It is the website for the UK government. It's the best place to find policy, announcements, information about the government, and guidance for citizens. Since 2012 it has replaced 1,884 government websites with just one, to become the home of all central government's online content and services.

What I'm trying to tell you is that it's a big and complicated bit of software, both on the backend and frontend. The frontend that users see is made up of approximately 13 separate applications, and that's not including any of the publishing applications. So what would be seen as a "small change" for a small static site, can actually turn out to be a huge change for a large complex site of its size. This is an issue pretty much every organisation and developer will run up against at some point. I'm sure many readers have been in the position where they've said: "If only we could change this small piece of code, and examine the difference it makes", knowing full well that the change is potentially day's worth of work (if not more).

What we need is a way to experiment with changing production code, without actually making any changes to production code. It doesn't sound possible does it? But that's exactly what we can do with Workers. The Worker can sit between the production server and our browser and make changes to server responses on the fly. We can then observe the difference these response changes make to performance. Here's a tweet about an example I tried with GOV.UK, examining what difference the position of the JavaScript in the HTML source and the loading mechanism used (e.g. async / defer) has on performance. As I mention in the tweet thread, this set of changes would have taken much longer than the hour it took to create the experiments!

Boilerplate code

This code is all based off what Andy Davies has written in his excellent blog post all about the subject: 'Exploring Site Speed Optimisations With WebPageTest and Cloudflare Workers', which itself was based off the work Patrick Meenan mentions in his 'Getting the most out of WebPageTest' talk (slide 60). So with this post I really am kneeling on the shoulders of giants!

The heavily commented code that all these recipes are based off can be found in this gist (opens in new tab).

Code quality

At this point it's probably worth mentioning that the code used in the examples is pretty rough and ready "throwaway-code". It's simply there to transform certain server responses, rather than being production ready code. That being said, you should always try to login to the Cloudflare admin area and observe the CPU time that your worker is taking during execution. The free worker plan allows for up to 10ms CPU time per request. If you are going over these 10ms I'd imagine throttling will kick in. This is also a sure sign there are potential inefficiencies in the code which may lead to unrealistic final results (due to increased latency).

For example, I was recently looking to see what a really large HTML page can have on the CPU time in a Worker. I picked the NASA Astronomy Picture of the Day full Archive which returns 710 KB of uncompressed HTML. For certain DOM manipulation operations I managed to get it to spike to 35ms. And really it's not a complicated DOM at all, it's just large:

So it's worth checking the CPU graph if you notice latency issues during testing.

Recipes

Below you will find a set of recipes I've found useful when performance testing.

Checking all requests flow through the worker

The first thing that's worth checking is that all the expected responses for your test site are actually flowing through your worker. There are cases when this may not be the case. For example, if some of your requests use relative URLs, and others use absolute URLs. Your absolute URLs will likely be pointing to the original site. This could lead to less accurate results since your browser will be creating a TCP connection to both the worker URL and the original server. Thankfully this pretty simple to fix using the HTMLRewriter API and the 'Rewrite links' example.

// domain rewriter vars
const OLD_URL = "https://www.example.com/";
const NEW_URL = "/";
//...
if(acceptHeader && acceptHeader.indexOf('text/html') >= 0){
  // store this particular HTML response for modification
  let oldResponse = await fetch(url.toString(), request)
  // create a new response
  let newResponse = new HTMLRewriter()
      // rewrite the links from the following page elements
      .on("a", new AttributeRewriter("href"))
      .on("link", new AttributeRewriter("href"))
      .on("img", new AttributeRewriter("src"))
      .on("script", new AttributeRewriter("src"))
      .on("image", new AttributeRewriter("src"))
      .on("meta", new AttributeRewriter("content"))
      // transform the old response
      .transform(oldResponse)

  // return the modified page
  return newResponse
}
//...
// https://developers.cloudflare.com/workers/examples/rewrite-links
class AttributeRewriter {
    constructor(attributeName) {
        this.attributeName = attributeName
    }
    element(element) {
        const attribute = element.getAttribute(this.attributeName)
        if (attribute) {
            element.setAttribute(
                this.attributeName,
                attribute.replace(OLD_URL, NEW_URL),
            )
        }
    }
}

In the above example we are picking out selected elements in the page and making sure they all use relative URLs. Note: you should only do this for assets that actually exist on your sites server defined in the site variable, else you're just going to get a lot of 404's!

Code: Complete code for this example can be seen in this gist (opens in new tab).

Adding resource hints to the page `<head>`

In this example we're going to look at how we can modify our page <head> to add a few resource hints, and give the browser information about the page before it is fully parsed. There are many articles available to read about how these tags can help improve the performance of a website.

var resourceHints = `
<link rel="preload" href="/assets/font/font1.woff2" as="font" type="font/woff2" crossorigin="anonymous">
<link rel="dns-prefetch" href="https://fonts.gstatic.com/">
<link href="https://cdn.domain.com" rel="preconnect" crossorigin>
`
//...
if(acceptHeader && acceptHeader.indexOf('text/html') >= 0){
  // store this particular HTML response for modification
  let oldResponse = await fetch(url.toString(), request)
  // create a new response
  let newResponse = new HTMLRewriter()
    // add listed resource hints to the page head
    .on('head', new addResourceHints())
    // transform the old response
    .transform(oldResponse)

    // return the modified page
    return newResponse
}
//...
class addResourceHints {
    element(element) {
        // notice how we are prepending the hints, right after the opening head tag
        // can be changed to append if you want them right before the closing tag
        element.prepend(resourceHints, {html: true});
    }
}

Notice how we have different resource hints being added. In the example I've added a preload, dns-prefetch, and a preconnect. You can add as many or as few as you need. Also note how these tags are being added directly after the opening <head> tag. You can change this too if you so wish (e.g. using .append()).

Code: Complete code for this example can be seen in this gist (opens in new tab).

Adding resource hints using the link header

So we've added resource hints by modifying the HTML in the <head>, but it is also possible to use a response header instead. This is quite simple to do with a Worker, but there are a couple of caveats:

Be careful of absolute / relative URLs with your preloaded assets. Make sure the URL in the preload matches how the asset would usually load in the page. If they don't match you will end up preloading and loading the asset twice (e.g. a font is preloaded and then also loaded through the usual page load).
If using preload make sure to include the 'nopush' string. Cloudflare converts a link preload to a HTTP/2 server push by default. This may (or may not) be the functionality you actually want.

In the below example I've also allowed for both HTML changes as well as modification of the response headers.

//...
if(acceptHeader && acceptHeader.indexOf('text/html') >= 0){
  // store this particular request for modification
  let oldResponse = await fetch(url.toString(), request)
  // create a new response
  let newResponse = new HTMLRewriter()
  /**
    * Insert any HTML modifications here
    */
  // apply above modifications to the response
  .transform(oldResponse)

  // Make the headers mutable by re-constructing the already modified response.
  let response = new Response(newResponse.body, newResponse)
  // add our Link headers
  response.headers.set("Link", '</assets/fonts/font-file-1.woff2>;rel="preload";as="font";crossorigin; nopush, </assets/js/js-file-1.js>;rel="preload";as="script"; nopush');
  // return the modified page along with custom headers
  return response
}
//...

Code: Complete code for this example can be seen in this gist (opens in new tab).

Removing resource hints from the page `<head>`

I've given you an example of how to add resource hints to the <head> so I may as well give you an example of how to remove them too. Remember you can be very specific as to which ones you remove by making use of Attribute selectors:

//...
if(acceptHeader && acceptHeader.indexOf('text/html') >= 0){
    // store this particular request for modification
    let oldResponse = await fetch(url.toString(), request)
    // create a new response
    let newResponse = new HTMLRewriter()
        // blanket example of removing all resource hints
        .on("link[rel='preload']", new removeElement())
        .on("link[rel='prefetch']", new removeElement())
        .on("link[rel='dns-prefetch']", new removeElement())
        .on("link[rel='prerender']", new removeElement())
        .on("link[rel='preconnect']", new removeElement())
        // example were we only remove a selected preload hint for a font
        .on("link[rel='preload'][href*='our-woff2-font.woff2']", new removeElement())
        .transform(oldResponse)

    // return the modified page
    return newResponse
}
//...
class removeElement {
  element(element) {
    element.remove();
  }
}

Code: Complete code for this example can be seen in this gist (opens in new tab).

Removing resource hints headers

So above we added the Link header for our resource hints, but what about if our server is already serving them and you want to test what difference removing them makes? Well it's pretty much identical to the above code only we use the delete() method instead of set():

if(acceptHeader && acceptHeader.indexOf('text/html') >= 0){
    // store this particular request for modification
    let oldResponse = await fetch(url.toString(), request)
    // create a new response
    let newResponse = new HTMLRewriter()
    // remove all external scripts from the page
    /**
      * Make your HTML changes here
      */
    .transform(oldResponse)

    // Make the headers mutable by re-constructing the already modified response.
    let response = new Response(newResponse.body, newResponse)
    // delete our Link header(s)
    response.headers.delete("Link");
    // return the modified page along with custom headers
    return response
}

Code: Complete code for this example can be seen in this gist (opens in new tab).

Modifying CSS and JavaScript response bodies

The great thing about CFWs is you can easily make changes to responses for other types of responses too. For example, in your Worker you could look for CSS and JavaScript files and make changes to their response bodies. You could quite easily remove whole sections of JavaScript code, or add in brand new selectors to the CSS and observe the results in the page. In the simple example below we are:

replacing a common CSS string (the font-family property value)
adding a simple console.log() to the end of a JavaScript file

//...
if(acceptHeader && acceptHeader.indexOf('text/html') >= 0){
  // store this particular request for modification
  let oldResponse = await fetch(url.toString(), request)
  // create a new response
  let newResponse = new HTMLRewriter()
    /**
      * Make your HTML changes here
      */
    .transform(oldResponse)

    // return the modified page along with custom headers
    return newResponse
} else if(acceptHeader && acceptHeader.indexOf('text/css') >= 0){// Change CSS here
    // grab the CSS response
    const response = await fetch(url.toString(), request);
    // extract the body of the request
    let body = await response.text();
    // modify the CSS response body
    body = body.replace(/Arial, Helvetica Neue, Helvetica, sans-serif;/gi,'Georgia, Times, Times New Roman, serif;').replace(/Arial,Helvetica Neue,Helvetica,sans-serif;/gi,'Georgia, Times, Times New Roman, serif;')
    // return the modified response
    return new Response(body, {
        headers: response.headers
    });
} else if(acceptHeader && acceptHeader.indexOf('*/*') >= 0){// Change JavaScript here (uses the generic Accept directive)
  // being granular we only modify a single response for a specific JavaScript file
  if(url.toString().includes('our-specific-js-filename.js')){
    // grab the JS response
    const response = await fetch(url.toString(), request);
    // extract the JS body of the request
    var body = await response.text();
    // using template literals we add a console.log to the end
    body = `${body} console.log('String added last');`;
    // return the modified response
    return new Response(body, {
      headers: response.headers
    });
  }
}
//...

There's a caveat to the above code. The Accept header for scripts in all modern browsers is */*, so I've specifically targeted a single JavaScript file. You can easily do the same with the CSS using the same method.

Also, when you get to the point of var body = await response.text();, what you do next basically comes down to a JavaScript string manipulation exercise. You could easily add / remove / insert code into the files using the basic string manipulation methods.

Be sure to check to see if your <script> or <link> tags don't have an integrity attribute. If they do you have Subresource Integrity enabled on your JavaScript and CSS assets. If you modify these files on the fly like in the example above, the hash of the file will have changed and the browser will refuse to load it. If that's the case I'd remove the integrity attribute using the HTMLRewriter at the same time.

Code: Complete code for this example can be seen in this gist (opens in new tab).

Removing elements

This is an incredibly simple example, as the HTMLRewriter API does all the heavy lifting for us.

//...
if(acceptHeader && acceptHeader.indexOf('text/html') >= 0){
  // store this particular request for modification
  let oldResponse = await fetch(url.toString(), request)
  // create a new response
  let newResponse = new HTMLRewriter()
    // remove a specific script
    .on("script[src*='name-of-our-script.js']", new removeElement())
    // remove the third meta tag in the head
    .on("head > meta:nth-of-type(3)", new removeElement())
    // remove all div elements that start with 'prefix'
    .on("div[class^='prefix']", new removeElement())
    // remove all link elements that start with '/assets/' and end with '.css'
    .on("link[href^='/assets/'][href$='.css']", new removeElement())
    .transform(oldResponse)

    // return the modified page along with custom headers
    return newResponse
}
//...
class removeElement {
  element(element) {
    element.remove();
  }
}

You can be really creative with your CSS selectors if you need to be, after all the Worker is powered by Chrome's v8 engine, so it understands all the modern CSS selectors.

Code: Complete code for this example can be seen in this gist (opens in new tab).

Clearing and adding inline scripts

The problem with inline scripts is that most of the time they are just a simple <script> tag. There are no attributes to use to select them, and their position in the page may not always be consistent in the DOM. So a method I've found useful is to remove all inline scripts from a page then rebuild and add them back in where you need them. This gives you fine grain control over the HTML markup coming from the Worker:

//...
if(acceptHeader && acceptHeader.indexOf('text/html') >= 0){
  // store this particular request for modification
  let oldResponse = await fetch(url.toString(), request)
  // create a new response
  let newResponse = new HTMLRewriter()
    // remove inline scripts from the page
    .on("body > script:not([src])", new removeElement())
    // reinsert some inline JavaScript back into the page
    .on("body", new reinsertInlineScript())
    // transform the page
    .transform(oldResponse)

    // return the modified page along with custom headers
    return newResponse
}
//...
class removeElement {
  element(element) {
    element.remove();
  }
}

class reinsertInlineScript {
  element(element){
    let inlineScript = `document.body.className = ((document.body.className) ? document.body.className + ' js-enabled' : 'js-enabled');`;
    element.prepend(`<script>${inlineScript}</script>`, {html: true});
  }
}

Using this method you could completely rebuild where inline scripts sit in the page source, and experiment with what they contain (since they are blocking after all).

Code: Complete code for this example can be seen in this gist (opens in new tab).

Adding inline CSS

Using pretty much the same method as above you can add inline CSS into the <head> of your page. In doing so we are adding bytes to the browsers critical path, but it can be useful for experimentation. One particular use case could be you are wanted to explore is what difference adding critical CSS to a page does for rendering performance:

//...
if(acceptHeader && acceptHeader.indexOf('text/html') >= 0){
  // store this particular request for modification
  let oldResponse = await fetch(url.toString(), request)
  // create a new response
  let newResponse = new HTMLRewriter()
    // add the inline CSS to the head
    .on("head", new addNewCSS())
    // transform the page
    .transform(oldResponse)

    // return the modified page along with custom headers
    return newResponse
}
//...
class addNewCSS {
  element(element) {
    let newInlineCSS = `
      body {
        border: 10px solid red;
      }
    `;
    element.append(`<style>${newInlineCSS}</style>`, {html: true});
  }
}

Code: Complete code for this example can be seen in this gist (opens in new tab).

Adding attributes

There may be times where you want to add certain attributes to elements. A great use case for this in web performance is adding defer and async attributes to <script> tags, and seeing how this changes the loading performance of a page. In the example below we create classes to add these attributes to scripts. These can be adapted to add any attributes you might want to add to an element:

//...
if(acceptHeader && acceptHeader.indexOf('text/html') >= 0){
  // store this particular request for modification
  let oldResponse = await fetch(url.toString(), request)
  // create a new response
  let newResponse = new HTMLRewriter()
    // add defer to this script
    .on("script[src*='script-to-be-deferred.js']", new addDeferAttribute())
    // add async to this script
    .on("script[src*='script-to-be-asynced.js']", new addAsyncAttribute())
    // transform the page
    .transform(oldResponse)

    // return the modified page along with custom headers
    return newResponse
}
//...
class addDeferAttribute {
  element(element) {
    element.setAttribute('defer', 'defer');
  }
}

class addAsyncAttribute {
  element(element) {
    element.setAttribute('async', 'async');
  }
}

Code: Complete code for this example can be seen in this gist (opens in new tab).

Quickly adding scripts to the `<head>` and closing `<body>` tags

The position of scripts in the page source has become less of an issue since modern browsers have started to support the async and defer attributes, but there still may be cases where you want to position them very specifically in the source for experimentation. This is the method I used when I was investigating reducing the Cumulative Layout Shift (CLS) on GOV.UK in this tweet. I cleared all scripts from the page source and rebuilt it with the scripts repositioned as required:

//...
if(acceptHeader && acceptHeader.indexOf('text/html') >= 0){
  // store this particular request for modification
  let oldResponse = await fetch(url.toString(), request)
  // create a new response
  let newResponse = new HTMLRewriter()
    // remove all external scripts from the page
    .on("body > script[src]", new removeElement())
    // insert scripts back before the closing body tag
    .on("body", new reinsertBodyScripts())
    // insert scripts back before the closing head tag
    .on("head", new reinsertHeadScripts())
    .transform(oldResponse)

    // return the modified page along with custom headers
    return newResponse
}
//...
class reinsertBodyScripts {
  element(element) {
    var srcArray = [
      '/assets/js/body-script-1.js',
      '/assets/js/body-script-2.js'
    ]

    srcArray.forEach(function(val){
      element.append(`<script src="${val}"></script>`, {html: true});
    })
  }
}

class reinsertHeadScripts {
  element(element) {
    var srcArray = [
      '/assets/js/deferred-head-script-1.js',
      '/assets/js/deferred-head-script-2.js'
    ]

    srcArray.forEach(function(val){
      element.append(`<script src="${val}" defer></script>`, {html: true});
    })
  }
}

There are many combinations you may want to experiment with related to async, defer and standard blocking scripts. I can't cover them all here. But hopefully this example gives you an idea of how you'd do it.

Code: Complete code for this example can be seen in this gist (opens in new tab).

Testing performance

So, once you've made changes to your page you're going to want to check what difference they have made to the performance of the page. There are a couple of ways to do this:

In the browser

The browser is a great way to check to see if the page is being modified in the way you expect (e.g. view the page source), so I'd recommend setting up this method anyway. Using the ModHeader extension you can easily add request headers to your requests. In our examples above we'd set x-host to the site we want to change (e.g. x-host: www.gov.uk), and x-bypass-transform to either true or false depending on if we want to toggle the Worker changes on or off. You could then use Chrome Lighthouse and run a performance audit with the changes both on and off. Using a tool like Lighthouse CI Diff you could then compare the difference between the two sets of results.

Using WebPageTest

I use the browser method above for verification and debugging of the changes, but my preferred method of comparing results is using WebPageTest. It gives you incredibly detailed performance information that you can compare across multiple experiments. The test setup uses the overrideHost scripting functionality to reroute any requests to the original domain through the Worker for modification. An example script I use for testing with GOV.UK can be found below:

setCookie https://www.gov.uk/ cookies_policy={"essential":true,"settings":true,"usage":true,"campaigns":true}
setCookie https://www.gov.uk/ cookies_preferences_set=true
setCookie https://www.gov.uk/ global_bar_seen={"count":0,"version":8}
addHeader x-bypass-transform:false
overrideHost www.gov.uk govuk-worker.nooshu.workers.dev
navigate https://www.gov.uk/

Enter the above script into the 'Script' tab of WebPageTest, configure all your test settings then run a test. Once completed you can use the compare view functionality to examine the difference between the different sets of results.

Summary

In this post I've covered what Cloudflare Workers are, and how they can be used for various frontend performance tweaks and testing. Moving onto how you can easily check to see you are seeing the changes you expect, and then performance difference they make.

If you've watched Harry Roberts' talk 'From Milliseconds to Millions' where he talks about the importance of a pages <head> tag for page performance, it's now possible to easily try this. Using a Cloudflare Worker you have a way of testing changes quickly and easily against your production environment, without having to touch a single line of production code! Imagine being able to sell in performance work internally (or externally) with actual metrics you could see if production code is changed. Note: The results may not be 100% accurate compared to your production environment, but they are certainly a good way to give you an indication as to what is possible.

I personally think that's a pretty great tool to have in your web performance toolkit!

I'd love to hear any feedback you have on the blog post above, or if there are any other recipes you use, please do let me know!

Testing priority hints with WebPageTest

2021-02-20T00:00:00Z

Back in January 2019 the Chrome team published an Intent to Experiment: Priority Hints message to the blink developers Google group. This set out a plan to start an origin trial to allow developers to experiment with this new web platform feature and give feedback before eventual rollout across all origins.

Unfortunately Priority Hints are currently on hold as the Chrome team weren't able to to see statistically significant performance wins with the current design. So does that mean you can't play with this experimental web platform feature now? No, not at all! Read on to find out more how you can still experiment with it using both your browser, and more interestingly, WebPageTest with Cloudflare Workers!

What are priority hints?

Lets very briefly look at what priority hints are and their expected usage. So priority hints are a way for developers to tell a browser the relative priority of resources on a page. A developer can then increase or decrease the priority of an asset depending on how they judge it's priority in the page load process. So for example:

<!-- The main image is in the viewport,
so considered important by the browser, but we disagree -->
<img src="/images/hero-image.png" importance="low" alt="Main hero image, but considered unimportant">

<!-- we want the browser to know to fetch this script,
but don't prioritise over other important assets -->
<link href="/js/required-later.js" importance="low" rel="preload"  as="script">

<!-- script set to async so it won't block the thread while JS is fetched
but we consider this script to be important so bump its priority for download -->
<script src="/js/async-script.js" importance="high" async></script>

Now of course you need to be careful when changing priorities. If you bump one asset's priority level, you are lowering the priority of another. Or if you bump all assets to high, then really you have no prioritisation at all! There's a very important note to be made at this point. These are hints to the browser. They aren't instructions. Prioritisation in the browser is very complex and it will use heuristics to determine how best to load a page. These hints are a developer's way to give it more information for consideration. But ultimately it is up to the browser if it actually decides to use the hint for network prioritisation. It may simply choose to ignore them and continue as is.

But one of the interesting use cases for priority hints is not boosting an assets priority with the high value, it's actually lowering an assets priority using low. By doing so you are giving other assets on a page network priority. So say for example you had an image at the top of your page that you know will be in the main viewport. Browser heuristics state that this will automatically be a high priority asset. But what if that image doesn't actually add value to a user's journey. It's a 'nice to have' (eventually), but concentrate on the rest of the page before loading it. That's just a very simple example of where this functionality could be useful.

There's a really informative blog post: "Get Ready for Priority Hints" by Jeremy Wagner and Yoav Weiss which goes into priority hints in much more detail.

So how do I use them?

Okay so enough waffle already, how do you try them today? It's actually really simple, as they are sitting behind the 'Experimental Web Platform features' in Chrome flags (chrome://flags/#enable-experimental-web-platform-features).

Browser

In Chrome you can either enable the 'Experimental Web Platform features' flag manually, or use the command line interface to fire up a Chrome window with it already enabled. On OSX you simply enter this command into the terminal:

/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --enable-experimental-web-platform-features

WebPageTest

Doing this via WebPageTest is just as simple as above. Just make sure you are running a test using a Chromium browser and enter --enable-experimental-web-platform-features into the 'command line custom options' under the 'Chromium' tab (see below):

The Chromium browser running the test will now have priority hints enabled (assuming it's based on a build after M73).

Are priority hints enabled?

So how do you know if priority hints are enabled? Well I've created some incredibly simple demo pages that will allow you to test to see if they are. The pages contain a selection of images with their priorities modified using the importance attribute. You will quickly be able to spot if priority hints are enabled by looking at the network tab in DevTools and making sure the 'Priority' tab is visible.

In the image below, priority hints aren't enabled. All images get a high priority and the preloaded image gets a low priority:

Now lets see what happens when the experimental flag (and client hints) and enabled:

The above image looks very different. Each of the images has a priority set depending on the image name (mirrored with the importance attribute) and the preload has been forced to be high priority. Note that images with importance="auto" return the default priority determined by the browser heuristics. A very unscientific and simple demo, but it allows you to test to see if client hints have been enabled.

Using it in the wild

So you may be asking: "Why would I be adding the importance attribute to my website when no browsers actually support them!". And for that I completely agree, that's not a good idea. But that doesn't mean we can't test in "production environments". It's time to roll out our favorite new friend, Cloudflare Workers (CFW).

I've blogged before about how I've started to use CFW to experiment with web performance changes. And since priority hints are easily applied using the importance attribute they seem like the perfect candidate for the HTMLRewriter runtime API. But what's even better is we can easily combine a CFW with WebPageTest. By doing so, we can experiment with changing our production environment and measure the difference in performance the importance attributes would make to page loading.

Cloudflare Workers setup

With this setup I'm piggybacking off the code Andy Davies lists on his blog post.

Preloading Print CSS files

In this very simple example we take 'IDLE' priority CSS files (print stylesheets) and bump them up the priority order by preloading them. It's then possible to tweak the priority of the preload using the importance attribute. Note: This is a completely contrived example to show the attributes in action. I wouldn't recommend doing this in practice!

Default waterfall

Without preloading our print style sheets have an 'IDLE' priority and are seen at requests 11 and 12:

Preloaded CSS

Now let's use a Cloudflare Worker to inject a <link rel="preload"> right after the opening <head> tag:

In the example what is seen is very predictable. Our print stylesheets have been bumped up the network request order, now sitting at request 2 and 3. Also the priority for these CSS files is now set to HIGHEST.

Preloaded CSS with priority hints

Now let's enable priority hints in Chrome and modify the preloads to add the importance="low" attribute:

In the example above we still have the print stylesheets sitting at requests 2 & 3, but the important difference is the priority of this preload has been set to LOWEST. We have successfully given the browser a hint that this preload exists but isn't important. Now this is such a simple page and example that the change actually has no effect on the performance of the page (other than slowing it down slightly because we are preloading print CSS!). But it does show how easy it is to modify a page's HTML using a Cloudflare Worker and experiment with priority hints.

If you are interested in the code used to create this very simple example I've dropped it all into a gist here (opens in new tab).

Modifying a preloaded JavaScript bundle

In this example I look at the UK version of the Nike website. Running the site through WebPageTest and looking at the source I noticed that they are preloading two JavaScript files in order to boost its priority. The site is built using React, and as a whole has a lot of JavaScript, coming in at around 1.2 MB of compressed. So using a Cloudflare Worker and WebPageTest let's investigate what happens when we alter the priority of these 2 preloads.

Vendor.js preload

There's a file with vendor.js in the filename that comes in at 170 KB in size. The file looks to contain some helper libraries for classnames, Brightcove video, and a production version of React v16.12.0. Note: If any developers from Nike are reading this, you look to be including another version of React later in the page load (v16.13.0), so that's a pretty good optimisation to look at right there!

By default this vendor.js file is given a medium priority, with a weight of 220. Note: Very roughly speaking, the higher the weight value, the higher the priority. With a standard preload this file downloads at request number 5 in the waterfall, as seen below:

But as mentioned above with priority hints we have the ability to give the browser a hint that this preload should have a lower priority. So using our Cloudflare Worker to modify the HTML lets see what happens when we add importance="low" to this single preload. The resulting waterfall can be seen below:

Priority weight drops from 220 to 147 and from request 5 to 17.

Client.js preload

Next on the list is the preload containing client.js in the filename. This file is 83 KB of compressed JavaScript and looks to be the core code for the Nike React app. By default this file gets a priority of medium, weighted at 220 and is seen at request number 6 in the waterfall:

Again we use priority hints to add importance="low" and investigate what happens:

Priority weight drops from 220 to 147 and from request 6 to 16.

Both together

So we've dropped the priority of each preload individually, so let's see what happens when we do it for both at the same time:

With both preloads having importance="low" added, we see the following:

vendor.js weight drops from 220 to 147. Request number drops from 5 to 15.
client.js weight drops from 220 to 147. Request number drops from 6 to 18.

Performance

So enough of looking at random waterfalls. We can see that the priority hints are working as expected. But what really matters is the performance a user sees. We now have 4 tests we can compare using WebPageTest:

Baseline: No HTML modifications, priority hints enabled but not used. Cloudflare Worker used as a basic proxy.
Vendor.js file preload set to low: HTML modification to a single preload, priority hints enabled and used.
Client.js file preload set to low: HTML modification to a single preload, priority hints enabled and used.
Both preloads set to low: HTML modification to a both preloads, priority hints enabled and used.

WebPageTest outputs a very interesting filmstrip and set of graphs:

In the image above we see the baseline at the top, and both now low priority preloads at the bottom. The difference in performance is quite striking. Pixels are being rendered to the screen a whole 2 seconds faster. Not bad for a 3G Fast connection (on Chrome desktop)!

Next let's take a look at the timing information graph:

The most important difference for me is the render metrics. Comparing the baseline to both preloads being set to low:

Speed Index changes from 5,630 to 5,172, or an improvement of 8%.
First Contentful Paint and First Meaningful Paint change from 3,650 to 1,972, or an improvement of 46%.
Visually Complete from 10,900 to 6,800, or an improvement of 38%.
Total Blocking Time from 564 to 328, or an improvement of 39%.

Now it isn't all positive, we do see certain metrics get worse:

Time to Interactive changes from 6,686 to 8,392 or a regression of 26%.
Largest Contentful Paint changes from 6,737 to 6,949, or a regression of 3%.
Load Time (Fully Loaded) changes from 13,354 to 13,487, or a regression of 1%.

The visually progress graph really shows the difference in the way the page is rendered for each test:

The green line where both preloads are set to low clearly starts a lot sooner than the baseline (blue). The green line is also ahead of the rest all the way up to 99% complete. Also note the interesting JavaScript rehydration dip for all tests around the 9 second mark.

I find it quite incredible what dropping the priority of two JavaScript files (170 KB and 83 KB) can do for page performance. If you are prioritising perceived performance metrics this could be considered a win, although there are regressions in other metrics. But it's also worth noting that this page is only preloading 2 JavaScript files out of the 59 (yes, 59!) JavaScript it loads in total. So with some further investigation into priority hints it may be possible to improve these other metrics too. This is left as an exercise for the reader.

If you are interested in the worker code for this experiment it can be found in this gist (opens in new tab).

Looking for feedback

So although priority hints are currently on hold, the Chrome team are still looking for feedback for future iterations. There are a number of issues open in the github repo with feedback from the community and Addy Osmani has also stated the team are interested in how much control developers would like over resource loading. Is it:

More to do with "when" (e.g. <script load-after="first-paint">)
High-level influence (e.g. importance=low/high)
Granular (modulo H/2 priority gotchas) (e.g importance=3)

The team have seen priority hints come in most useful when there's demonstrable network contention between resources and you want to re-order requests for more optimal use of available bandwidth (e.g. using importance="low" to mark certain resources less important, so use bandwidth elsewhere).

Summary

In this post we've looked over what priority hints are, their current status, and how to enable and experiment with them on your local browser. Taking this one step further you can also test on live sites using a Cloudflare Worker and WebPageTest. It just shows what a powerful combination these two tools are when used together!

I'm personally a fan of priority hints and I'd love to see them being developed in the future. I like the idea that I have some influence over the priority of assets loading in a page. The web is such a varied medium with so many edge cases that it seems impossible (to me anyway) for browser heuristics to cover all of these issues automatically. So giving developers the means to steer a browser in a certain direction sounds like a win to me.

So if you've spotted an issue that you think could be solved using priority hints, why not try them out in production using a Cloudflare Worker and WebPageTest and see what results come back. I'm sure the Chrome team would be very interested in hearing about it!

Post changelog:

21/02/21: Initial post published. Thanks to Addy Osmani for the input around feedback and where the team have seen wins with priority hints.
22/02/21: Minor change related to what 'weight' really means in terms of prioritisation in Chrome. It's a massive oversimplification that I plan to address in the future with a new section. Thanks to Robin Marx for highlighting!

Using Puppeteer and Squoosh to fix the web performance of embedded tweets

2021-02-06T00:00:00Z

In January I was doing some investigation work into the web performance of some of the UK Governments blog posts using lighthouse-parade and I stumbled across a recurring theme: some of the pages that performed the worst were ones that included Twitter embeds. Investigating further lead me to write the following Tweet (very ironic huh!):

You are reading that correctly. As well as reducing page performance metrics and the Lighthouse score, a single embed adds an extra 2.7 MB of JavaScript to a page! That's a crazy amount of code for what is really just a simple bit of text and (maybe) a few images. I'm not the first to notice this issue either.

The problem

So the problem I have, and one I'm looking to optimise is that I have many tweets embedded across lots of WordPress blog posts that I'd like to quickly fix. I'd also like for others to be able to follow guidance I write to fix these issues too. So I'm looking for a fairly simple & semi-automated solution.

Now WordPress comes with a really useful feature (for content authors). Simply paste a tweet into a blog post then let the WordPress OEmbed feature take over. Jetpack will pull in the Tweet data as a <blockquote>, then pull in the widgets.js JavaScript and hydrate the tweet to make it more "twittery".

This is great for an author, it's very "set it and forget it", but it's pretty poor from a web performance point of view. As far as I can tell you can't control how this link is expanded and hydrated once it hits the page in WordPress (please do correct me on this if this assumption is wrong!). For example, we could simply omit the JavaScript widget.js from being added to the page. Therefore no 2.7 MB JavaScript download, so it's progress. The downside is that the tweet doesn't look like it does on Twitter (see the image below):

The above image is taken from Firefox when the Enhanced Tracking Protection setting is set to strict, as the browser will block all social media trackers by default. Personally I'm fine with how it looks. I can still read the content, and if I want to see the actual tweet I can click on the date and it will take me to Twitter. But others will probably disagree. So how do we keep the "look" of a tweet, minimise the web performance impact it has on the page and still allow it to be accessible to screenreaders?

The solution

So the solution I've picked is actually very simple. Grab a screenshot of the tweet, use the following HTML:

<figure id="custom_ID">
    <a href="https://twitter.com/...." rel="noreferrer noopener">
        <img src="/images/screenshot-of-tweet.jpg" alt="Tweet by [name] on [date/hour] (tweet content below)" width="123" height="321" />
    </a>
    <figcaption>"Text contained within the tweet goes here"</figcaption>
</figure>

The code above is simplified, but hopefully you get the idea. You'd probably want to indicate that it links to an external page, and you could provide different image formats too (e.g. WebP, AVIF). The alt text contains information about the person who tweeted, and the date it happened. This means that even multiple tweets from the same author on a page will be seen as unique to assistive technologies.

There's an interesting point you should consider before using this method. What if the author of the original tweet deletes it? So your screenshot now becomes a permanent record of the tweet they wanted removed. I think this is beyond the scope of this blog post, but it's something Terence Eden discusses in his How to preserve deleted Tweets in WordPress? blog post.

So quickly skipping over the morality question of preserving tweets, we run into the next issue. How do you screenshot lots of tweets and grab the tweet text without it taking hours? There are ways to screenshot tweets very easily: Tweet Cyborg is a service that allows you to input any tweet and it will convert it to an image to download. Or you can even do it directly in both Chrome and Firefox Devtools:

Both these methods work, but if you had to do this for 100's of tweets, it's going to take a while. And that isn't even taking into account capturing the text for the images alt attribute!

Puppeteer to the rescue!

This is where we can use the fantastic Puppeteer to help us out. So what is Puppeteer? Quoting from the documentation:

Puppeteer is a Node library which provides a high-level API to control Chrome or Chromium over the DevTools Protocol. Puppeteer runs headless by default, but can be configured to run full (non-headless) Chrome or Chromium.

So it allows you to programmatically open a browser, load a page, manipulate the page, capture data, then close the browser down and repeat. All without any input from us once started! And it just so happens to be able to capture screenshots too (since it uses the DevTools API after all, which we've already seen has the ability to capture node screenshots above).

Presenting `tweet-grabber`

I know, I'm terrible with names... but let's look beyond that. It's a Puppeteer script I've created that allows you to quickly capture screenshots of multiple tweets. Functionality includes:

Ability to capture a single tweet, multiple (comma separated) tweets, or pass in a file with a list of tweets to be captured.
The uncompressed PNG of the tweet will be captured as well as the text from the tweet (in an accompanying txt file).
Automatic compression of the tweets using the Squoosh CLI.

So for example, the tweet you can see at the top of this blog post was captured using tweet-grabber like so:

$ node index.js https://twitter.com/TheRealNooshu/status/1350578919389470721

The final compressed image as well as the tweet text is output to the compressed_tweets directory.

It's not a complicated script, all fairly straightforward. I've already been putting it to good use to run through 10-15 tweets at a time, all output in less than 40 seconds. Much quicker than manually capturing these tweets!

What about copy / pasting tweet text?

So Thomas Steiner raised an excellent point after I'd published this post, one which I hadn't considered. When embedding the tweet as an image you are preventing a user from easily copy and pasting the tweet text should they want to do this. A user could view the image alt text and / or visit the original tweet on Twitter. But both of these methods aren't as simple as copying directly from a page. He also came up with a suggestion about using Puppeteer to capture the CSS for a tweet, while still keeping the original HTML (and discarding the widget.js). This sounds like an interesting idea I'd like to investigate further. Although I still need to consider the fact I will be writing guidance and maybe training a few "non-technical" folks on how to do this within the limitations of WordPress.

So keep this limitation in mind when using this method. There are a number of alternative solutions listed below that you may want to consider too.

Update: A simple solution I've found to this issue is to use the <figure> and the <figcaption> elements. The text is shown below the images and can be seen by screen readers, search engines, and can be copy / pasted by users if required.

Other alternatives

There are others who have looked into this problem too and have different solutions. I'm linking to them here for completeness:

Jane Manchun Wong has worked on a solution using graphql fragments, dataloader, relay and nexus. Follow up tweet here with more info.
Luis Alvarez has built a Static Tweet Next.js Demo.
Johan Janssens has been working on a very work in progress Cloudflare Worker solution, code seen here.
Lucas Pardue uses a simple bash script to strip out the unwanted (and heavy) resources.
Terence Eden is currently working on a really interesting and clever solution to convert a tweet to an SVG. Thread with info can be found here. A writeup of this method can be found on his blog: "Minimum Viable Tweet to Semantic SVG"
Terence Eden investigated a similar image capture method back in 2016 using the Scheenshotlayer API and mogrify when Making a Twitter Collage.
Bruno Quaresma has created TweetPik, a tool to easily convert tweets to images SVG, PNG and JPG.
Kyle Mitofsky has created the eleventy-plugin-embed-tweet plugin that allows you to embed a tweet during the static build process.
Umar Hansa has created better-twitter-embed that he uses on his Dev Tips website.

So there are a few options around to use if you want to include tweets in your site without the web performance hit.

Improvements from team Twitter

I briefly discussed this issue with Addy Osmani who pointed me in the direction of a tweet from Charlie Croom, a web developer at Twitter. It sounds like the team are fully aware of this issue and it's on their roadmap to be fixed in the future! So with any luck these workarounds won't be needed in the future.

Summary

Embedded tweets are bad for performance, but there are ways to avoid the performance impact. There are downsides to the image capture solution I presented above, but I feel they are fairly minor compared to the 2.7 MB of JavaScript and 25 extra requests embedding a tweet on a page inflicts upon a user visiting a page. If it doesn't work for you there are other solutions available too, thanks to the wonderful open source community.

As always please let me know if you have any feedback and comments. I'm especially interested in how this script could be improved!

Post changelog:

06/02/21: Initial post published. Added a link to Terence Eden's "Making a Twitter Collage" blog post (Thanks Terence). Added link to TweetPik (thanks Alfredo Lopez for the link).
07/02/21: Added two other ways of embedding tweets (eleventy-plugin-embed-tweet & better-twitter-embed). Thanks to Alex Russell & Umar Hansa for pointing them out. Added feedback from Thomas Steiner about the copy & pasting limitation if using an image.
11/02/21: Updated the HTML sample in the page to make use of <figure> and the <figcaption>. This replicates the HTML output by WordPress when using a caption.
15/02/21: Added a link to Terence Eden blog post on his semantic SVG version of a tweet. Updated the HTML to improve overall accessibility, as the initial version failed the 4.1.2 Name, Role, Value WCAG Success criterion. Many thanks to Arnaud Delafosse for his feedback, and Šime Vidas for flagging it.

The web performance of internal systems is important, so optimise them too

2021-02-03T00:00:00Z

A few weeks ago I had a really interesting chat with a service team from the Department for International Trade (DIT), who were interested in improving the web performance of one of their services. These are the conversations I really enjoy in my current job role. I get to speak about a subject I really enjoy, (virtually) meet new people, teach them a new skill they may not know much about, and the cherry on the top is the potential to improve a users experience of the service. Unfortunately my heart sank a little when they informed me that the service was internal facing and not publicly accessible to the web. This meant my usual goto tools for evaluating web performance weren't available. These are:

But that got me thinking, how can teams test their internal services and products? In this blog post I'm going to investigate this.

It's never been more important for fast internal systems

As everyone knows (I'd hope!) the whole world has changed due to a global pandemic. With lockdowns happening across the globe this has meant people have had to work from home (if they can). This fact has caused a measurable effect on internet performance across the globe, generally resulting in higher traffic and slower download times. Because of this, the digital tools workers rely on to do their jobs need to be quick to load and become interactive to user input.

Performance issues that may not have been a problem before because your internal user was at the office on the local area network (LAN) where the system is located, suddenly are big issues. What compounds this issue even further is that many people may live in rural areas with slow, unstable connections. Or maybe live on streets in major cities with high contention ratio connections. Being as we are talking about internal systems, you may even need to throw a virtual private network (VPN) into the mix just to be able to connect to your office network to use them. And as I'm sure any VPN user will know, they can sometimes have quite a negative effect on internet speed.

So now's a good time to remember that staff members are users too. Or if you want to convert that phrase into Government speak: "Civil servants are users too". So make sure you optimise your internal systems too, as slow website performance causes stress. But what tools can you use to do this, given that these internal systems may not be publicly available? Read on to find out.

WebPageTest

First thing I should clear up: when testing an internal system you won't be able to test using the public instance of WebPageTest. But what you can do is setup your own private instance of WebPageTest. That way you can have it sitting within your own internal network.

Once up and running you get access to a huge amount of data for all of your internal systems. And of course there are guides out there on how to extract as much info as you can out of WebPageTest. Setting up your own instance isn't as complicated as it sounds. There are a few excellent guides on how to do this for yourself on AWS in just a few minutes:

What this also opens up for you is the ability to run WebPageTest periodically over time via automation. There are a few ways to do this:

So as you make changes to the system you can see if that are making a positive (or negative) effect on performance.

Lighthouse

The next goto tool is one you probably have on your computer already, and that is Lighthouse by Google. If you have a copy of Google Chrome on your computer, then using Lighthouse to audit an internal system is dead simple:

In DevTools simply look for the 'Lighthouse' tab and click 'generate report'.

A minute of so later after the audit has run it will return a result like those in the image above. One thing I'd highly recommend when doing this is setting up a new Chrome profile specifically for Lighthouse testing, as browser extensions can have a huge negative performance impact on the result depending on what they are doing to the page.

But there's a lot more to Lighthouse than the audit panel. Here are just a few options available to you:

You can run Lighthouse using the Command line interface (CLI)
You can run Lighthouse across all pages of a site very easily
Compare before and after performance results using Lighthouse CI Diff
Automatically run Lighthouse at regular intervals across multiple sites using a test runner
Add your own custom audits to monitor specific parts of a site
Share results publicly via a Github Gist and the Lighthouse Report Viewer.

Sitespeed.io

I'm bumping Sitespeed.io up the list as I really don't think it gets enough love. It truly is a fantastic set of tools to improve your site's performance. It's also quick and easy to setup via a simple docker or npm command. It's easy to run on your local machine for quick testing, or can also be configured for continuous testing using AWS or Digital Ocean.

With sitespeed.io you can continuously monitor as many pages as you need and this data can then be pushed into Graphite/Grafana to give you dashboards like this. Under the hood, Sitespeed.io is actually a collection of tools all packaged together. These are:

Coach - An automated tool that gives you advice on how to make your page faster.
Browsertime - It's at the heart of the Sitespeed tools as it interacts with the test browsers (e.g. Chrome, Firefox, iOS Safari). It collects performance metrics, images, videos and much more.
PageXray - Used for converting HTTP Archive format (HAR) files to JSON for easier reading and use. (There's more info on HAR files later in the post).
Throttle - CLI tool for accurately throttling your network connection when performance testing. Note: it throttles the whole computer's connection, so remember to stop it when you've finished!
Compare - An online tool so you can quickly and easily compare HAR files (e.g. before / after results).

Chrome DevTools

All modern browsers come with developer tools built in. We've come a long way since Firebug in Firefox! And these tools are becoming more and more powerful with every browser release. This is fantastic for developers, and ultimately end users too, since in theory websites should be less buggy right? (hah!).

The above image shows you the detailed information that an audit in the 'Performance' tab can give you about the runtime performance of your web page. But there's so much more to Chrome DevTools than just the performance tab:

The network tab gives you detailed information about all the requests (and responses back) in the browser.
You can gather detailed information about memory usage including learning how to fix them.
Look for unused CSS and JavaScript in a page using the 'Coverage' tab.
There's actually an incredibly comprehensive post all about Performance Analysis in DevTools that covers many other aspects.

And if you are looking for non-Google articles on the subject, you can read:

Firefox DevTools

There are other browsers out there that can also help when it comes to evaluating performance issues via their DevTools. As a Firefox user, I use its developer tools every day. It too comes with a really comprehensive set of tabs you can use to look for site issues:

You can use the tools to:

Identify and debug intensive JavaScript on the page.
Easily visualise the types of assets on a page both with a cold and a warm cache.
Inspect your site for Accessibility issues (Firefox has a great set of a11y tools).

Libraries for extending analytics data

This may not always be the case for internal tools, but if you are tracking their usage via some form of analytics (e.g. Google Analytics, Fathom, Matomo) you can extend the data captured to include more comprehensive web performance data.

There are a couple of libraries you can use to do this:

Perfume.js - Can be configured to capture a large set of RUM data for pushing into an analytics tool.
Analytics - Perfume.js plugin - Wrapper for Perfume.js to easily push web performance data into many different analytics tools.
web-vitals - accurately capture Core Web Vitals information, as well as all other Web Vitals.

Perfume.js really stands out in the amount of Real User Monitoring (RUM) data it captures. It is fully configurable to capture as much, or as little data as you need. There are a few tutorials on how to do this here:

JavaScript analysis

I'm not here to comment on the use of JavaScript in an internal system, although I do believe you should minimise it's usage if you wish to improve its web performance and overall resilience. But assuming you are using it, then you should make sure it is as optimised as much as possible.

Thankfully there are a whole range of tools you can use to do this:

bundle-wizard - CLI tool used to generate visualisations to see what a JavaScript bundle is made up of (see image above) so you can remove anything not needed.
Bundle Phobia - Find the cost of what adding an npm package to your bundle would do.
Webpack Bundle Analyzer - Visualize the internals of webpack output files with this tools interactive zoomable treemap.
source-map-explorer - Use source maps to analyse JavaScript code bloat (this tool also works with Sass and LESS for CSS analysis).

Specific framework performance analysis tools and articles are also available:

reactopt - A CLI React performance optimization tool that identifies potential unnecessary page re-rendering.
TracerBench - a controlled performance benchmarking tool for web applications. Providing clear, actionable and usable insights into performance deltas.
Measuring React app performance - DebugBear

CSS Analysis

As well as the 'Coverage' tab mentioned in Chrome DevTools earlier, there are also tools you can run via the CLI that will analyse your CSS, looking at its complexity as well as identifying unused selectors across a whole website:

analyze-css - CSS selectors complexity and performance analyzer run using the CLI.
uCSS - Crawl a whole site looking for unused CSS selectors that can be then removed.

Server benchmarking

Although the performance golden rule states that:

80-90% of the end-user response time is spent on the frontend.

It's still a good idea to make sure that the backend / server is optimised. After all Time to First Byte matters.

You also want to make sure that your server will be able to continue to function under load should that ever occur. There are a number of tools available to help you do both of these things:

httpstat - A small Python script to visualise the connection timing data returned from curl (see image above).
h2load - A benchmarking tool for HTTP/2 and HTTP/1.1 run from the CLI.
Hey - A tiny program for sending load to a server.
k6 - Build realistic load tests by writing JavaScript. Comes with an API and CLI.
Measuring Performance With Server Timing - Configure your server to send a header containing server timing information. Exposing backend server performance timings to the frontend so they can be easily observed in the browser.

Automation using Puppeteer

Puppeteer is a Node library which provides a high-level API to control Chrome or Chromium over the DevTools Protocol. Most things you can do manually in a browser can be replicated using Puppeteer, so how can this be leveraged to test web performance? Addy Osmani has blogged about using it for web performance testing and also shared the code on Github. These tests can be easily run via the CLI to test both internal and external websites.

Browser Extensions

There are a whole host of browser extensions you can use to evaluate web performance. When running these, I'd recommend using a separate profile with a minimal number of extensions enabled. This is because some browser extensions interact with the page and can slow down performance. Leading to skewed results. Extensions you can use are:

sloth - An extension that slows down the CPU and network in the browser, allowing you to easily mimic page performance on slower devices.
Perfmap - When used, the browser will create a heatmap of the resources loaded in the browser and their individual performance impact according to the Resource Timing API.
Web Vitals Chrome Extension - Chrome extension that reports the core web vitals metrics (LCP, CLS, FID) for any page visited. Note: this is soon to be built into Chrome DevTools, and can currently be seen in Canary.
perf-diagnostics.css - This isn't strictly a browser extension, it's a selection of CSS you can add to your page for debugging common performance issues. Simple and effective way to highlight images without a width / height attribute, among a number of others.

There are also extensions available to specifically help improve the performance when using certain JavaScript frameworks:

React Developer Tools - This extension add a 'Profiler' tab to DevTools, allowing you to drill down into the performance of your React application.
Angular Augury - A Chrome and Firefox extension that allows you to debug and profile Angular applications.
Ember Inspector - Debug and dig into the detail of a web application built with Ember.

Network Throttling

Network throttling, as the name suggests this is a way to slow your network connection down. It's important to realise that many users won't have a fast (and stable) broadband connection in a major city. Many users could be in rural areas with poor broadband and very little mobile signal. By throttling your own network connection you get an idea of the performance of a site for users under these specific network conditions.

Now you may be asking: "Why would I throttle my network when Chrome Devtools has this built in?". Well it's important to realise that not all network throttling methods are created equally. When throttling using Chrome DevTools it applies a delay at the browser-level when each response is received. Lighthouse runs the test at full speed, then simulates the connection speed afterwards, sacrificing accuracy for reporting speed. The following tools are much more accurate. They use OS-level network throttling which works at a much lower level. Note: these tools below throttle the connection for the whole machine, so make sure you minimise the number of applications running when testing, and disable throttling once done!

I use throttle all the time as it's so simple to use:

# Enable 3G Slow
throttle 3gslow

# Use a custom connection profile
throttle --up 1600 --down 780 --rtt 250

# Disable throttle
throttle --stop

HAR file analysis

I mentioned HTTP Archive format (HAR) files earlier in the post. These files are a way of capturing a web browsers network interaction with a website. The great thing about these files is you can capture them for any site you can get to via your browser (internal or external). Capturing them is very simple in both Firefox and Chrome developer tools:

Firefox

Chrome

Once captured there are a bunch of different tools you can use to view and analyse them:

HAR analyzer - In-browser HAR analysis tool from Google.
YSlow Command Line HAR analyser - Very old now, but YSlow will analyse the HAR file looking for performance improvements.
PageXray - Convert a HAR file to JSON so it is easier to read and interact with (used in Sitespeed.io).
Compare - An online tool so you can quickly and easily compare HAR files (e.g. before / after results).
HAR Viewer - Simple online HAR viewer to visualise the metrics within.
Charles Proxy - HTTP proxy / HTTP monitor that can accept HAR files as input for detailed analysis.
Fiddler - a debugging proxy server tool that can log, inspect, and alter traffic between a computer and a web server. It can also be used to analyse HAR files.
Har2CSV - A CLI tool which provides an easy way to convert HAR file HTTP log entries into a flat CSV format to make it easier to analyze.
Har2CSV & Har2TSV - My own version of the har2csv tool above but written with jq rather than node.

Web APIs

Going back to browser basics there are a few APIs you can use to measure the performance of a site if you want to do this yourself without using any abstraction libraries.

performance.now() - The now() method of the browser Performance interface returns a high resolution timestamp from when the method was called. Allowing you to measure the time between 2 calls very easily. Adding these before and after specific code will allow you to measure, and optimise this code.
Navigation Timing - This API allows developers to capture high resolution timing data related to the navigation of a document.
Resource Timing - This API allows developers to collect complete timing information about the resources loaded by a document.
Assessing Loading Performance in Real Life with Navigation and Resource Timing - Jeremy Wagner - A very comprehensive blog post all about how the two APIs listed above can be used to assess the loading performance of a web page.

Summary

Hopefully the resources listed above will give you the tools needed to improve your internal systems. Just because a service (or website) is only able to be accessed internally on a company network, doesn't mean that it can't be (and shouldn't be!) optimised for web performance. With so many people working remotely at the moment, don't assume that everyone has the luxury of a fast and stable connection. Because remember, you and your colleagues are users too!

As always, if you have any feedback about this post, or if I've missed anything please do let me know. Credit will be given in the changelog below.

Post changelog:

03/02/21: Initial post published.
03/03/21: Added 'Har2CSV' to HAR file analysis. Thanks to Paul Calvano for the link.
04/03/21: Added my version of 'Har2CSV' that uses jq.

The importance of `@font-face` source order when used with preload

2021-01-23T00:00:00Z

The other day I decided to run a quick WebPageTest (WPT) run over the latest version of the White House website launched for President Biden's term in office (it's amazing what I do for fun in lockdown huh!). The WPT run returned something curious:

The above waterfall was captured using Chrome on a real Moto G4 with a 3G Fast connection. That's a lot of font files being downloaded (red requests), 14 files in total.

So let's investigate what is happening here. First thing to notice is that there are 6 fonts that are loading right after the HTML has downloaded and parsed (request 2-7). This is a sure sign that these fonts are being preloaded. What usually happens is the DOM is constructed, then the CSS downloaded to create the CSSOM. These are both combined to form the render tree. At this point, fonts referenced in the @font-face rule will be discovered and requested by the browser (assuming they are needed to render the specified text on the page).

Note: This blog post isn't a criticism of the excellent work the team at the U.S. Digital Service does. It's just an interesting observation I've never encountered or considered before. Being a Civil Servant myself, I understand how hard everyone works to deliver user friendly services (especially under current conditions!). Keep being awesome!

Note 2: Turns out USDS didn't build this website!

Note 3: As of 29th Jan the font loading issue has now been fixed by the White House developer team and can be seen on the live site.

Font load priority

There's an important point to consider here. The order in which you list your font sources in the @font-face rule is very important. According to the CSS Fonts Module Level 3 specification:

It is required for the @font-face rule to be valid. Its value is a prioritized, comma-separated list of external references or locally-installed font face names. When a font is needed the user agent iterates over the set of references listed, using the first one it can successfully activate. Fonts containing invalid data or local font faces that are not found are ignored and the user agent loads the next font in the list.

Or in other words, the browser will start at the top of the list looking for a font format it supports. The first one it encounters is loaded, and if successful the sources listed later are ignored. Or in code it looks like this:

@font-face {
  font-family: bodytext;
  src: url(our-test-font.woff2) format("woff2"), /* WOFF2 tried first */
       url(our-test-font.woff) format("woff"), /* WOFF tried second */
       url(our-test-font.ttf) format("opentype"); /* TTF tried if others not supported. */
}

There's a large crossover between browsers that support WOFF and those that support WOFF2. Or to phrase it another way, if a browser supports WOFF2, it also supports WOFF. This is important, as if you are listing your WOFF fonts first in the @font-face rule, the browser is going to download and use these instead of the WOFF2 versions (which offer much better compression because they use Brotli). So for example if you do this:

@font-face {
  font-family: bodytext;
  src: url(our-test-font.woff) format("woff"), /* WOFF tried first */
       url(our-test-font.woff2) format("woff2"), /* If WOFF is successful, WOFF2 is ignored */
       url(our-test-font.ttf) format("opentype"); /* TTF tried if other not supported. */
}

Assuming the WOFF font doesn't 404, the WOFF2 font will never be requested by the browser!

The Waterfall

The above information can now be used to explain what is happening with the waterfall we see above.

Preload (requests 2-7)

Examining the code from the White House website we can see the following in the <head>:

<link rel="preload" href="...fonts/Decimal-Book.woff2" as="font" type="font/woff2" crossorigin="anonymous">
<link rel="preload" href="...fonts/Decimal-Semibold.woff2" as="font" type="font/woff2" crossorigin="anonymous">
<link rel="preload" href="...fonts/Decimal-Medium_Web.woff2" as="font" type="font/woff2" crossorigin="anonymous">
<link rel="preload" href="...fonts/MercuryTextG2-Roman-Pro_Web.woff2" as="font" type="font/woff2" crossorigin="anonymous">
<link rel="preload" href="...fonts/MercuryTextG2-Semibold-Pro_Web.woff2" as="font" type="font/woff2" crossorigin="anonymous">
<link rel="preload" href="...fonts/MercurySSm-Medium-Pro_Web.woff2" as="font" type="font/woff2" crossorigin="anonymous">

These preload links tell the browser to immediately make the requests for the WOFF2 versions of the fonts as soon as the browser parses the <head>, which it happily does as it trusts that these fonts will be used later in the pages lifecycle. Once these requests have been made and are sent over the network the browser is going to get a response back (assuming the fonts are available and don't 404). Even if the browser were to cancel the requests, it will still receive bytes back from the server for the fonts. They just wouldn't be used. There's an example waterfall of this happening here and a blog post all about it here.

CSS (request 9)

At request 9 we can see the CSS file being downloaded and parsed by the browser. Also notice how only 2 of the preloaded fonts have been fully downloaded. This is a critical part of the page load with limited bandwidth available. Under the hood the browser still needs fonts to be able to render text on screen. A preload will prime the browser cache, but it won't add a font to the FontFaceSet. For that you either need to use the CSS Font Loading API, or the traditional @font-face rule.

Within this CSS file on request 9 we see the @font-face rules (I've only referenced 1 as an example, but there are many):

@font-face {
    font-family: MercuryTextG2-Semibold-Pro_Web;
    src: url(.../fonts/MercuryTextG2-Semibold-Pro_Web.woff) format("woff"),
    	url(.../fonts/MercuryTextG2-Semibold-Pro_Web.woff2) format("woff2");
    font-weight: 400;
    font-style: normal;
    font-display: block
}

Notice the ordering of the src fonts in this @font-face rule. The WOFF version of the font is listed first, then WOFF2 listed second.

The result

The browser is now making 8 more font requests to WOFF font files (requests 10-17). To the browser it sees these as completely separate files. It doesn't know that the glyphs contained within them are actually identical to the ones from the preloaded fonts! The browser is now doubling up on the number of fonts requested because it has stopped at the WOFF font src in the @font-face rule. It has no knowledge about the WOFF2 version directly below because it has found the first one it supports, then stopped (as defined in the specifications).

This is probably easier to explain on a waterfall chart:

The above waterfall is from Chrome on a Moto G4 with a 3G Fast connection, but I have also managed to replicate the same result on iOS Safari, Microsoft Edge (Chromium), and Firefox (which now supports preload).

Cloudflare Workers

Here's where I get the chance to play with Cloudflare Workers again. For information on how this is done I recommend reading Andy Davies' excellent 'Exploring Site Speed Optimisations With WebPageTest and Cloudflare Workers' blog post, or you can watch him talk about it at London Web Performance here. If you'd like to see a fully commented sample of my worker code I've created a gist here (opens in new tab).

Modifying the `preloads`

First let's investigate what happens if we 'fix' the preload links and point them to the WOFF fonts referenced in the @font-face rules.

In the resulting waterfall we now see the WOFF fonts preloaded in requests 2-7. This in turn reduces the number of fonts requested by the @font-face rules after the CSS has been parsed.

Modifying the CSS

Now let's do the reverse, let's update the CSS @font-face rules and swap the order so the WOFF2 fonts are first. This will then match the WOFF2 fonts listed in the preload links in the head.

In this waterfall the preload is untouched, but I've simply made the WOFF2 fonts come first in the src order like so:

@font-face {
    font-family: MercuryTextG2-Semibold-Pro_Web;
    src: url(.../fonts/MercuryTextG2-Semibold-Pro_Web.woff2) format("woff2"),
    	url(.../fonts/MercuryTextG2-Semibold-Pro_Web.woff) format("woff");
    font-weight: 400;
    font-style: normal;
    font-display: block
}

It's amazing what such a small change can do to a waterfall.

Effect on performance

But it's not a cleaner looking waterfall that's important, it's the performance a user sees after the change. So the first thing to note is the impact on the number of bytes downloaded:

Version	Total Size (bytes)
Original	2,133,401
Fixed preload	1,904,496
Fixed `@font-face`	1,696,008

Reordering the @font-face rule alone has reduced the number of bytes downloaded by 440KB. That's a pretty decent saving by simply swapping a few 2's in the font filename and format().

But where you really see the difference is in the visual progress graph:

Version	Visually Complete (ms)
Original	8,668
Fixed preload	5,681
Fixed `@font-face`	5,311

We've shaved off almost 3.4 seconds from the visual complete time on a Moto G4 / 3G Fast connection, a 39% reduction. That's a huge improvement in performance for such a minor change. Your users will really notice that change. If you are interested in the full test results then here's the WebPageTest comparison view with all three tests.

Is this a problem in the real world?

So a good question to ask is now: is this a one-off occurrence, or does it happen quite frequently? Well, with December data from the HTTP Archive and the help of Barry Pollard we are able to answer this question:

	Mobile (abs)	Desktop (abs)	Mobile (%)	Desktop (%)
Download same WOFF / WOFF2 font (one fails)	155,541	132,044	2.17	2.19
Download same WOFF / WOFF2 font (both return 200)	69,805	59,387	0.97	0.99

We see 155,541 (2.17%) sites try to download both WOFF and WOFF2 fonts with the same filename. What most likely happens in these instances is the WOFF2 file fails, so the browser moves onto the WOFF fallback. This still isn't great for web performance since an extra request / response is made and adds time to the page load and blocks text rendering (I'm over-simplifying here I know). But what's worse is that we see 69,805 (0.97%) sites successfully double download both WOFF and WOFF2 versions of the exact same font! That's a lot of unnecessary bytes being downloaded which simply won't be used. If you are interested in the full dataset for this query you can view it here.

Spotting the `preload` issue

So how would you know if this is an issue for your site? Well the easiest thing to do is to look in your browser console. Both Firefox and Chrome will give you a set of warnings if you are preloading assets that aren't actually used in the page:

What if the first `src` fails

So let's now look at what happens with a waterfall when the first source in the @font-face fails. This isn't related to the White House website but the HTTP Archive data shows that it is happening for approximately 86,000 (1.2%) of sites in the HTTP Archive data. There's a caveat with this data. The only thing we know is that the browser tries to download both types of font files (WOFF & WOFF2). We also know that one of these fonts doesn't return a 200 status code (hence the fallback font then loads). But we don't have an exact figure on if one of the font requests was triggered by a preload, then falling back to @font-face. We also don't have the data on what source order the fonts are loaded in if using @font-face. So all in all, it's complicated! But what we do know is that the browser tried to load fonts of both types, and for some reason one of them failed (4xx status code).

What does this look like on a waterfall and what's the performance impact?

The waterfall & performance impact

Below you can see the annotated waterfall for a site that requests the WOFF2 font first but fails. This test was again run on a real Moto G4 / 3G Fast connection:

There's a fair amount going on in this waterfall so let's step through it:

On request 37 we see the TCP negotiation happen for the font domain. Once completed the font request goes out for the WOFF2 font. This whole process takes 2 seconds to complete. It takes ~865 ms to fail. That includes a full round trip to the server and back.
On request 41 it takes a full 1.7 seconds to fail. Notice how once failed the browser almost immediately requests the fallback WOFF font (request 46).
Notice how this fail / request pattern for all 4 fonts are aligned. Requests 37 & 43, 39 & 45, 40 & 44, and 41 & 46 are all font pairings (WOFF2 vs WOFF).

So in other words the browser must receive a font failure before it can request the fallback. This lack of primary font source is adding 2 seconds to the font load in this test case. What's worse in the test case above is that the browser ends up downloading the larger of the two font file types. Since WOFF files can be 20-30% larger than WOFF2 files. Ultimately the user gets a double whammy in terms of poor performance: added time to make a successful font request, and more data to download.

So how can you check for this? Well, again I'd look to the browser console. If you notice 404 errors for fonts, I'd put those errors pretty high up your prioritisation list to fix!

Summary

So in summary, when you are preloading fonts make sure that what is preloaded matches with the src defined in your @font-face rules. Remember the first src the browser finds wins. If these don't match then your users may be downloading two sets of exactly the same fonts. Make sure you check your browser console for preload warnings and font 404 errors. As both of these issues can seriously impact page performance and adding to the data requirements for the site.

Post changelog:

23/01/21: Initial post published. Thanks to Barry Pollard for the HTTP Archive data and Andy Davies for the Cloudflare Worker debugging.
24/01/21: Added more detail about the performance impact of fonts failing to load, thanks to Barry Pollard for the scenario. Added links for more information about Cloudflare workers. Added info about USDS having no involvement in the White House website. Added a link to the gist with a sample of the cloudflare worker code I used.
29/01/21: Added update to say the issue with fonts has now been fixed.

How to run a WebPageTest test

2020-12-31T00:00:00Z

Table of contents

In this post I'm going to examine all the individual settings from the homepage of WebPageTest (WPT). You could consider this blog post a prequel to my two other blog posts related to WebPageTest: 'How to read a WebPageTest Waterfall View chart' and 'How to read a WebPageTest Connection View chart'. Both blog posts give you information on how to interpret the results of the tests you run off the back of what I'm about to show you in this post.

Simple testing tab

The first section I'll cover goes against the WPT ordering of the option tabs. The first tab is 'Advanced Testing', but I'm going to introduce the 'Simple Testing' tab first just so we cover some basic concepts before diving into the finer details.

Most of this is probably pretty self-explanatory, but some of the details will help you understand how WPT works.

URL input field

Short and sweet, this is the URL you want WPT to examine. For the public instance of WPT this URL will need to be publicly accessible on the internet. If you need to test private sites that are on an internal network, or behind a basic HTTP Basic authentication there are options that I will cover later in the post.

Test Configuration

When you select the 'Simple Testing' tab almost all the custom configuration options are preselected for you. The dropdown gives you a set of options for you to choose from:

Mobile - Slow 3G (400 Kbps download, 400 Kbps upload, 400ms Latency)
Mobile - Regular 3G (1.6 Mbps download, 768 Kbps upload, 300ms Latency)
Mobile - Fast 3G (1.6 Mbps download, 768 Kbps upload, 150ms Latency)
Mobile - 4G (9 Mbps download, 9 Mbps upload, 170ms Latency)
Desktop (5 Mbps download, 5 Mbps upload, 28ms Latency)

The mobile devices available are a Motorola G4, and all test using the default Chrome browser. The desktop device is a virtual machine (VM) running Chrome. The Moto G4's are real devices all currently located in Pat Meenan's basement in Dulles, Virginia (not a joke, 100% true).

As for the different connection speeds available and how that works, a whole separate blog post could be written about this topic. Pat describes the setup in the forum here, and in the WPT documentation here. These resources are a few years old now so a more up to date TL;DR; is that most phones reverse-tether their networking through a Raspberry Pi where netem is used for traffic shaping. Quoting from Pat in the forum post below:

FWIW, we did a LOT of testing at Google with actual networks and dummynet. With dummynet we could simulate a given set of conditions just about exactly (with more consistency) but it's worth noting that it's exactly that - one set of conditions. In the case of the default 3G profile I have set up right now, it's basically modeled around the 60-70th percentile of measured connections globally so it doesn't match an exact market - some will be faster and some will be slower.
The main things that you can't duplicate with traffic shaping are:
Radio start-up delay - given a few seconds of idle, most phones will shift the radio into low power modes and it can take anywhere from a few dozen ms to several seconds before the radio comes back and is able to transmit data. It's more important for multi-page navigations where the radio can turn off between each step but it could also add completely dead time to the beginning of a page load (but that case is completely out of the control of the site owner). In really extreme cases it's possible for the radio to turn off while waiting for a server response - but if your TTFB is that long you have serious issues to look into anyway.
Network architecture - on traffic-shaped WiFi you will still be going out through your normal ISP's network (Verizon Fios in the case of Dulles). You won't go through any carrier proxies, deal with them routing the traffic back to a central egress point or coming from a Carrier's mobile IP block in case there is any server logic tied to it.
The agents themselves work perfectly fine on actual networks and the test results come back over the wired connection from the tethered host that is driving the agents so the data use is just for the actual browsing and isn't really why I chose to use WiFi. The main issue is the insane variability using an actual carrier network. The performance varies quite significantly from day to day and even within a day which makes it REALLY hard to compare results to each other.

Old WPT documentation from Pat can be found in this link here.

The traffic shaping models come from real world connection data. For example the 3G profile is based around the 60-70th percentile of measured connections globally in 2014. Latencies haven't changed much since then so they are still relevant today. It's worth noting that these are approximations. Some connections real users experience will be quicker, others will be slower.

The whole purpose of this dropdown is to give you an idea of what a user with the selected device and connection speed will experience when loading the URL you want to test.

Include Repeat View

When WPT first loads your URL, the browser has what is known as a 'cold cache'. All assets that are needed must be downloaded. By enabling this checkbox the test will run twice on the defined page (browser closing between test runs). This is a great way to test the caching strategy for the page being tested, as the second run will be using a 'warm cache'. So in theory if the page has an efficient caching strategy configured for the page assets, very little will need to be downloaded on the second run.

Run Lighthouse Audit

Again, quite self-explanatory. As well as running a WPT audit across the URL, a separate audit will be completed using Lighthouse. This separate audit will use '3G Fast' as the connection speed (as this is the default in Lighthouse tests), and any connection settings selected via the dropdown will be ignored. This Lighthouse report will be run on an emulated Moto G4 with 4x simulated CPU slowdown.

Running a test

Now we've gone over what everything in the simple testing tab is, it's time to run a test. Simply enter the URL to be tested, select your options and press the big yellow 'Start Test' button to the right.

The next screen you will see will look like below. Giving you details about the URL being tested, the test instance being used, number of test runs (more on this later), and the amount of time the test has been running.

Now the time it will take to complete the test depends on the test settings selected. So for example, a 'Slow 3G' setting is going to take much longer to load a page than '4G'. But don't worry, you don't have to keep the page open while the test runs. Tests run completely independently of your web browser. You can return to the test later via the same test URL (or the 'Test History' link in the main navigation).

Note: At this point I highly recommend signing up for a WPT account. If you plan on doing a lot of testing WPT will save all of your tests so you can refer to them again at a later date should you need to.

Once the testing is complete you will see a result that looks similar to below:

The returned test results are immutable and can't be modified once saved (or even deleted). If you scroll further down the page you'll be presented with a huge amount of interesting web performance data for you to dig through. The information on this page is beyond the scope of this article, but if you are interested in knowing more about the resulting waterfall charts, I have a blog post all about them here. For the similar looking but different connection view chart, I have a blog post here you can read.

Advanced Testing tab

So now we've covered some of the core concepts of how to run a simple WPT test, let's get stuck into the more advanced (and interesting) settings you can modify. The bulk of the rest of this article will be spent in this section.

Again there are some self-explanatory fields, but I will cover them here for completeness.

URL input field

This is identical to the 'Simple Testing' explanation given above.

Test Location

This is one of the major differences from the 'Simple Testing' setup. Under 'Advanced Testing' you are given the options of the type of device you wish to use for testing, and geographically where in the world the test is run from. So for example you can choose from a set of real (not simulated) Android or iOS devices (all located in Dulles, Virginia). You are also given the option to choose from many test locations dotted across the globe. There are way too many to list here, but you can find detailed information about each location on the Location Status page. This page is also a great way to spot the busy locations and avoid them should you want your test be added to a shorter queue.

Now you may be asking why test location is important. Well depending on your users location in relation to your origin server (assuming a CDN isn't used), this latency will have a big difference on the overall page performance. There's actually a whole chapter dedicated to this topic in High Performance Browser Networking by Ilya Grigorik. So which location should you choose? Well if you have any analytics data available for the site you want to test, it would be worth checking where the majority of your users are browsing from. If 90% of your users are coming from a particular country, then it would make sense to choose a test location in (or as close to) the country as possible. This will give you a more accurate representation of how most of your users experience your website.

Browser

The next setting to choose is the browser you want to test your site with. Now these will change depending on the device / location you choose in the 'Test Location' dropdown above it. This is another setting that can be driven by your user data if you know the most popular browser visiting your site. Or if you are more interested in how different browsers handle specific server / page setups, this is a great way to directly compare them using the huge amount of data returned from each of the tests. Most of the time you may only be interested in how your site performs on a Chromium based browser (since they are pretty popular!). If so, go right ahead and choose Chrome.

Beyond this point is where the number of configuration options ramps up rapidly. Each of the tabs beneath this dropdown has its own set of options to configure independently.

Test Settings tab

This tab contains most of the 'high-level' settings for the test. Some of these will look familiar to areas covered above in the 'Simple Testing' tab.

Connection

Pretty obvious really: what connection speed do you wish to replicate when the test is being run. I covered a fair amount of how this works in the 'Test Configuration' section above. But this time there are many more options available for you to choose from. A full list of options as well as the bandwidth and latency values available can be found here.

It's worth noting that you should always select some form of bandwidth shaping from the options. 'Native Connection (No traffic shaping)' is an option, but the results returned won't really be comparable from test run to test run over time. It's likely to be an incredible quick connection, so not representative of most of your users. Also, as the amount of traffic changes from hour to hour, it's likely that the true download / upload will differ depending on how busy the network is at that point time. By selecting some form of traffic shaping you can be sure that this variable is consistent across test runs.

Number of Tests to Run

When you enter a URL for WPT to test it doesn't only test the URL once, it will run the same test multiple times. This is because network latency and traffic can have a huge effect on how a website performs, and these variables are constantly changing. In order to have a more accurate (and representative) result, the test should be run multiple times. For example, should you run a single test run that has very little traffic across the network at that point in time, the resulting performance data will come back very positive. Whereas in reality the network isn't always like this. So by repeating the test WPT is sampling these variables at different points in time and looking for a happy median.

Note that between each standard test run no cache is shared (unless 'repeat view' is selected), so each time WPT visits the page it is starting from a 'cold cache'. On the public instance of WPT you can run a maximum of 9 test runs. If you set up your own private instance you can run as many runs as you choose. You should always select an odd number of test runs due to the fact that WPT will select the median result once the test has completed. An odd number of runs allows WPT to easily select the run that sits in the middle in terms of performance.

Repeat View

This was covered in the 'Include Repeat View' above. But it's worth mentioning that if you select 'repeat view', this will be doubling the number of tests run against that particular page. For example, say you have 9 runs set: each run tests with a first view (cold cache), then again with a repeat view (warm cache). So the page will be visited 18 times. The median run for each first view and repeat view is then highlighted in the 'Performance Results'.

What metric is used to select the median run? By default WPT will use the 'loadTime' metric, but you can also use the 'SpeedIndex' metric to perform this comparison too. Simply append ?medianMetric=SpeedIndex to your test result URL and you will see the 'Performance Results' summary change. In the examples below, the top image is using 'SpeedIndex', the bottom image is using LoadTime. Notice how the run selected for the 'repeat view' median changes from 3 to 2.

Capture Video

Very simple: do you want to capture a video of the page loading under the conditions you have selected?

Why is this useful? Well, a really important aspect of web performance is the perceived performance a user experiences. If a site loads completely in 8 seconds, but it goes from a blank screen to fully rendered over the space of a single frame, a users perceived performance isn't going to be as good as if the site loads more incrementally over the same time period. If you can get the first pixels rendered to the screen within the first couple of seconds or so, it gives a user something to look at. The user perceives that something is happening during this loading period, giving them the perception that the page is loading quicker.

The 'Capture video' setting allows you to examine the page load frame by frame and also exactly how the browser is rendering the page. This is incredibly useful when you come to comparing results. WPT has the ability to render multiple video's side by side from different tests, thus allowing you to demonstrate exactly what difference your changes to the site made (both good and bad!).

Label

The last input field for this tab is a free text area allowing you to give a particular test a user friendly label. Should you wish to compare tests before and after a particular set of changes, this is a good way to remind yourself at a later date. Note that this is an optional field and it can be added (or modified) once the test has completed. This is very useful when it comes to comparing tests via the 'Filmstrip view', as can be seen in this section here

Advanced tab

This tab gives us a whole bunch of exciting advanced settings to play with.

I'll go over each of them individually.

Stop Test at Document Complete

When a WPT run completes it will usually wait for the 'Document Complete' event from the browser to fire and all network activity over the network to go quiet for 2 seconds. At this point WPT considers the page to have completely loaded. But under certain conditions this may not always happen. For example, old versions of Internet Explorer didn't fire the onload event when it was still waiting for something to download. So the test would simply time out after 2 minutes waiting for this to happen. To avoid this you can check the 'Stop Test at Document Complete' checkbox. This will force the test to finish once the document is complete (onload event has fired). This point is represented by a blue vertical line on the WPT waterfall chart.

Disable JavaScript

Very easy option to explain. Simply disable JavaScript in the browser that is running the test. Not to get too preachy, but this is a great way to test what your users will see when the JavaScript for a site fails in some way (e.g. network error, blocked, malformed code etc). Remember the use of the Progressive Enhancement methodology is a great way to give users in this situation a simplified experience (not just a blank page).

Clear SSL Certificate Caches

This is a setting that is useful when the browser is being run on a Windows based instance. Windows by default doesn't clear certificates. When intermediate certificates already exist in the cache this can prompt a browser to hide certificate checks (Chrome with EV certs, and Internet Explorer (IE)). Around the end of 2012 WPT changed the default behaviour for clearing the OS certificate caches which caused SSL negotiation times to increase and skewed the results (since it was clearing all the certificates including root providers). In this case IE was having to validate root certificates too, which takes time. Since there's no way to clear only leaf certificates, the option to clear all SSL certificate caches was moved to the advanced options to be run on a per-test basis and disabled by default. This isn't an issue for linux instances because there is no certificate caching. I have a whole post about the impact of SSL certificate revocation on web performance should you be interested. Quoting what Pat said in the forum post below:

With the 2.10 release I changed the default behavior for clearing the OS certificate caches.
Somewhere around 6 months ago I started clearing the OS certificate caches which cased SSL negotiation times to go up significantly for the IE agents that were running on Windows Vista or later (IE on XP does not do CRL/OCSP checks).
There was quite a bit of push-back because it was clearing all of the certificates, including the certs from the root providers so IE was validating things like verisign's root cert which is something that will hardly ever happen to end users.
The real world lies somewhere in between but I don't have a way to just clear the leaf certificates and it was skewing site's performance enough that it was hiding other issues so I made it an option on a per-test basis and it is disabled by default (it's available in the advanced settings if you want to re-enable it for a given test).

Ignore SSL Certificate Errors

Sometimes a browser will show a SSL certificate error if the certificate chain is broken, or a certificate simply doesn't exist. Possibly caused by differences in the root certificate lists or other browser-specific policies. Another reason could be if you are running a private WPT instance to test a domain that doesn't have correct SSL certificates configured. In these instances you can check the 'Ignore SSL Certificate Errors' checkbox to bypass this issue. Note: This doesn't work in Firefox as Firefox has disabled any way to ignore these types of errors for security reasons. So it may only be a setting that works in IE and Chrome.

The quote from the forum post is below:

Firefox disabled any way to ignore cert errors as a security measure so it no longer works for Firefox (at this point it may be Chrome and IE-only). The legacy Windows agent used to force it by intercepting the API calls internal to Firefox but that is no longer possible with the cross-platform agent.
The only way to do it now for Firefox is to run a private instance of WPT and install the certs (or a custom root) on the test machines so they see the self-signed certs as valid.

Disable Compatibility View (IE Only)

There's a lot of history behind this setting. For web developers who worked on sites through this time (myself included) it will probably bring back frustrating memories. A TL;DR; is basically in order for newer versions of IE to not 'break the web', the browser implemented a feature that allowed it to mimic older versions (e.g. make IE8 render like IE7 did). This was needed because of certain IE specific coding practices that existed and were very popular. This setting forces IE to load in 'Strict mode' no matter what the DOCTYPE or http-equiv="X-UA-Compatible" meta tag says. (Thankfully) it is very unlikely you will need this setting on the modern web.

Capture network packet trace (tcpdump)

Although WPT can give you a lot of information about the order in which a page loaded as well as all the intricacies involved in this process, sometimes you need to dig a little deeper into the low level details. This is where capturing a network packet trace comes in extremely useful. To do this WPT uses a command-line tool called tcpdump. What this tool does is allow us to display TCP/IP and other packets being transmitted or received over the network for the instance that is running the test. If checked WPT will save what is known as a Packet CAPture (PCAP) file that can be downloaded on the test result page:

The great thing about packet level capture when done via a WPT instance is the isolation. The only network traffic that is being captured is solely related to the page being tested. For example, if you were to run a capture on your local machine you'd most likely have to filter so much data due to all the other processes running on the machine in the background.

So what can you do with this PCAP file once you have it? Well, you can use tcpdump -r [capture_filename] to read in the actual file to your local version of tcpdump and examine the internals using the CLI. Or if you are after a GUI interface then it's hard to beat Wireshark. If you are decoding HTTPS traffic you will also need the TLS Key Log file. Follow the instructions under the 'Load your data in Wireshark' header from this page on how to decode secure traffic. If you are looking to capture traffic from iOS devices then you are going to need to go through quite a few other steps that Andy Davies covers in his 'Capturing and Decrypting HTTPS Traffic From iOS Apps Using Frida' blog post.

A really quick way to open Wireshark and not worry about all the settings, grab the PCAP and the TLS keylog file and run the following command:

wireshark -r your-pcap-file.pcap -o tls:keylog_file:your-keylog-file.keylog

Word of warning, Wireshark is an incredibly powerful (and complex) bit of software with a steep learning curve associated with it. But once mastered, it unlocks a whole new world of data for you to examine.

Save response bodies

By default when a WPT test runs the response bodies of the requests are discarded once downloaded. This is fine for most cases, but what about if you wanted to store these bodies for future reference. Say you were making a change to a pages <head> tag in order to improve performance. Wouldn't it be great to be able to view that HTML in the future once you knew which version performed best. This is exactly what this option does. You could almost consider it 'view source' functionality for WPT tests.

It's worth nothing that WPT only does this with text based responses (HTML, CSS, JS etc). Images and other binary formats aren't stored in this process. There currently isn't an option to enable this functionality, even when using your own private instance, but Pat has mentioned that it wouldn't be too hard to add if there were more requests for it in the future.

Preserve original User Agent string

One issue running WPT against a live site is that it will make impressions on your analytics data. Since to an analytics tool a visit from a WPT browser looks like any other browser visiting the site. For that reason WPT appends a PTST string to the User-Agent (UA) string. A random example of this is given below:

user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36 PTST/201209.031210

Using the 'PTST' string you can easily filter out any data with it in the user-agent header. Now you may be asking why PTST? What does it stand for? Well, it actually stand for PageTeST, which was the name of the IE plugin version of WPT before there was a web interface in the version we see today.

User Agent String

This setting allows you to completely override the UA string should you wish to do so. For example if I enter 'my-custom-ua-string' into the input box, the header for every request will be:

User-Agent: my-custom-ua-string PTST/201223.161206

Now if you wanted a completely clean UA string without the PTST information you'd simply check the 'Preserve original User Agent string' checkbox as well. That will give you the following output:

User-Agent: my-custom-ua-string

This functionality isn't supported by all browsers. For example it doesn't work in IE11 and Edge. It does work in Chrome (including mobile device versions), Firefox, and Safari on iOS.

Append to UA String

This input box allows you to append a custom string to the end of the UA string in the file requests from the chosen test browser. This advanced functionality was added off the back of a request by Rick Viscomi that can be seen here. Additional functionality was added in this request that allows the test ID (and other details) to be appended to the UA string when you run a test via the API. This allows you to extract the test URL from your logs in the logging software you use. See this thread for more details on how to do this.

Capture Full Size Video

By default WPT will capture a video of the page load (assuming the 'Capture Video' checkbox is checked in the 'Test Settings' tab). But it won't capture the full size video. By default it captures a video which is scaled down by approximately 50%. In my simple test using Chrome:

Default: 408px x 360px
Full: 904px x 680px

The dimensions of the video captured depend on what device the test is running on. For example, running a test on a mobile device will produce a video with mobile phone dimensions.

This setting is extremely useful if you are looking to examine the finer visual details of how a page is loading, or if you are wanting to show the video in a presentation.

What this setting also unlocks is the full size images in the 'filmstrip view'. Selecting 'Huge' will give you a filmstrip full of images where the largest side is 600px. You can further increase their size to the full native width if you modify the thumbSize query parameter in the URL for the test.

Minimum test duration

As mentioned earlier in the 'Stop Test at Document Complete' explanation, by default WPT will stop a test once the 'Document Complete' event has fired and the network has gone quiet for 2 seconds. But there may be some instances where you want to force a test to run for a minimum number of seconds. A use case where I've used this setting is in testing the web performance of a 404 page. Without this setting WPT would see the 404 status code and immediately stop the test. Another example where you may use this setting is for long running XHR requests. If they take longer than 2 seconds the response will never be captured.

Custom headers

This textarea allows you to add custom headers to all requests made by the browser to the server. This can be useful when it comes to various debugging scenarios. You add each custom header on a newline:

For example these request headers could be used to prevent the test agent from being blocked by a server, or another way to exclude tests from your analytics data.

Inject Script

The inject script textarea is an incredibly useful tool when it comes to debugging. When the test runs the script will be injected into the page once the document exists. This will be as near to the opening <head> tag as it can be. But the exact position in the source usually depends on what browser is being used.

Some example use cases for this setting include:

Experimenting with resource hints to see how they effect web performance
Injecting custom performance marks (opens in new tab) for various page events
Setting cookies / local storage for the removal of page overlays and cookie banners
Performance testing Single Page Apps (SPA)

Quote from Andy Davies about cookie banner testing below:

I tend to work out what cookie or local storage entry the overlay is checking for and then add it via a WPT script, or inject JS

It's alway good practice to wrap your custom script in an Immediately Invoked Function Expression (IIFE), that way you know your variable & function names won't clash with what is already set on the page. Here's a very simple example of a script you could add to the textarea:

(function(){
  // this will trigger a font load
  var customFont1 = new FontFace('custom font name', 'url([FONT_URL_HERE])', {
    // other font properties here
  });

  // add the font to the document
  document.fonts.add(customFont1);

  // monitor the font load (optional)
  customFont1.loaded.then((fontFace) => {
    // log some info in the console
    console.info('Font status:', customFont1.status); // optional
    // log some info on the WPT waterfall chart
    performance.mark('wpt.customFont1Loaded'); // optional
  }, (fontFace) => {
    // if there's an error, tell us about it
    console.error('Font status:', customFont1.status); // optional
  });
})();

Here we are loading a new font into the page programmatically using the CSS Font Loading API. We are adding a performance mark to the WPT waterfall chart so we know the exact point at which the font is loaded. There are so many uses for the 'Inject Script' functionality, I'm sure there could be a whole blog post dedicated to it.

Important note: If you are looking to inject a script while using WPT scripting ('Script' tab, covered later), you will need to use a different method. Either exec or execAndWait. Documentation for these can be found here.

That brings us to the end of the 'Advanced' tab. Next on our list is the 'Chromium' tab.

Chromium tab

The 'Chromium' tab has a whole bunch of settings specific to the Chromium based browsers (Chrome, Edge, Opera, Brave, Samsung, Vivaldi).

Capture Lighthouse Report

This was covered above in the 'Simple Testing' section above under 'Run Lighthouse Audit'.

Emulate Mobile Browser

This setting isn't just emulating the viewport settings of the device selected in the dropdown. Behind the scenes there is a reference VM that tests the performance of all the real devices in the device lab. WPT uses this data to throttle the CPU to closely match the performance of the real devices. If you examine the device options in this dropdown you will see it closely matches the real devices available in the lab. The setup for this emulation can be seen in the code here.

It's worth noting that the results from using this setting are still estimates of the performance compared to a real device. A real device will have different RAM available, and the CPU architecture will be different. Remember that you will still need to throttle the network connetion via the 'Test Settings' tab (covered earlier).

Capture Dev Tools Timeline

If you are running the Chrome browser this setting allows you to download the full DevTools timeline from the browser running the test. This file includes network, layout, and JavaScript parsing / executing data. The file can be opened via your local version of Chrome DevTools, or you can open it in the online viewer using the '(view)' link:

This data gives you an incredible amount of detail about what is happening on the browsers main thread. If you are interested in finding out more about this functionality I've written about it in a bit more detail here.

Capture V8 Runtime Call Stats

I'm replicating the order listed in the 'Chromium' tab, but this setting makes a lot more sense if you read 'Capture Chrome Trace' and 'Trace Categories' first. This setting is effectively a shortcut to enable the 'Capture Chrome Trace' setting and only select the v8 and disabled-by-default-v8.runtime_stats categories. This setting is mainly used by the Chrome browser development team to easily collect data on where the v8 engine is spending its time. There are many advantages to collecting this data in a WPT instance: 'clean' data with only a single tab running and no extensions, easily repeatable results, bulk testing thousands of pages using the API, and collection of data from a set of real devices.

Capture Chrome Trace (about://tracing)

This setting is a reference to the chrome://tracing/ tab that you can access in Chrome. It gives you access to an incredible amount of underlying data not only from a page load perspective, but also the underlying performance of the Chrome browser itself at a system level. Once enabled you can access this data via the following links:

Note: A word of warning about enabling this option. If you are only interested in the web performance of a web page, DON'T enable this option. Some categories can add a lot of overhead to the tests, which in turn can result in the skewing of overall timings. This setting is primarily targeted at browser engineers, not web developers. For more information about this setting and the data visualisations available using it I have written about them more here.

Trace Categories

This setting is used in conjunction with the 'Capture Chrome Trace' setting above, and is only relevant when the checkbox is checked. The amount of data returned when capturing a Chrome trace is huge. Now you may only be interested in one particular type of data. This input box allows you to select only the data you are interested in (e.g. for blink data, delete all other categories listed). Again as mentioned above, you only really need to be concerned with this setting if you are a browser developer.

Capture Network Log

This setting is similar to the 'Capture V8 Runtime Call Stats' mentioned earlier. It is effectively a shortcut to enable tracing and select only the netlog category in the 'Trace Categories' input field. Enabling this also passes the command-line flag to store the non-trace version of the netlog too: chrome://net-internals generates a JSON file that can be used for debugging, and the trace-version is layered on top of this. Enabling this option makes WPT capture both netlog versions.

Enable Data Reduction

This setting is a browser option that users can choose to enable on their devices. For example, if a user is on a limited data plan they may prefer to view what may be considered a 'cutdown' version of a website. A site with fewer images, different image compression, slightly different (and more data efficient) page design.

In practical terms WPT will append the save-data: on header to all of the requests sent to the server. More info about this header can be found on the Save-Data MDN page, and I've written a blog post about its usage here. I also highly recommend this talk by Tim Vereecke from London Web Performance, November 2019 - Data-s(h)aver strategies. It has a whole heap of information about this header and how you could be using it.

Host Resolver Rules

This setting is basically just like a hosts file only with wildcard(*) support. It's also what WPT uses for Chrome when the setDns WPT script command is used. So for example entering MAP * 127.0.0.1 into the input field forces all hostnames to be mapped to 127.0.0.1 (you're not going to get much back if you use this exact command!).

An example use case for this setting is if you are testing your live website, but you wanted to map some requests back to your development server instead. Very useful for some areas of testing and debugging.

Command-line

Chrome comes with a whole bunch of options you can enable & disable via the command-line when you run it. Some change behaviour of features, others are for debugging or experimenting. You can enter these command-line options into this input box and they will be passed through to Chrome in the test.

A specific use case I've used this for is to disable HTTP/2 using the --disable-http2 flag. So if you want to compare the performance of HTTP/2 vs HTTP/1.1 for the same site using WPT, simply pass in this setting and it will force the use of the HTTP/1.1 protocol. You can then compare the results of HTTP/1.1 vs HTTP/2 using WPT.

Auth tab

This is a much simpler and self explanatory tab compared to the previous tabs. This tab allows WPT to access a website that is sitting behind HTTP Basic authentication (BA). Useful if you wish to test a site that isn't publicly accessible.

As the warning says in the tab, make sure you use a test account when using this functionality! You don't want to be sharing an account with higher privileges with anyone by them viewing the test results. Note that when in use the test won't appear in the 'Test History' tab in the main navigation because it is automatically set to private.

Script tab

The script tab allows you to automate testing via the use of the WPT scripting language.

The documentation for the script commands can be found here. Possible use cases for these scripts include:

Following a user journey across multiple pages
Acceptance of cookie banners and the setting off cookies
Login to a member area for performance testing
Test a sites performance with third-parties blocked
Performance testing Single Page Applications (SPAs)

An example of a very basic script looks like this:

setCookie https://www.gov.uk/ cookies_policy={"essential":true,"settings":true,"usage":true,"campaigns":true}
setCookie https://www.gov.uk/ cookies_preferences_set=true
setCookie https://www.gov.uk/ global_bar_seen={"count":999,"version":8}
navigate https://www.gov.uk/

This script sets three cookies that will hide the cookie banner on GOV.UK then navigate to the homepage. This example can be useful if you want a clearer picture of the Cumulative Layout Shift (CLS) seen on a page (since cookie banners can cause a high CLS value on page load).

Script includes sensitive data

There may be times where you need to include sensitive data in your test script (e.g. login cookie data, personal information etc). If you share these results with other people you may not want them to be able to see this data. In this case select this checkbox and the cookie headers and WPT script will be discarded in the results and no longer viewable (see images below):

Checkbox unchecked

Checkbox checked

Discard all HTTP headers

When enabled, all of the request headers will be removed from the reporting in the test results. So clicking on a request and navigating to the 'Request' tab, the corresponding panel will be empty.

Block tab

The block tab does "exactly what it says on the tin", it blocks requests and domains listed in the two separate textareas.

These textareas take a space separated list of strings. These blocks cause what is known as a 'fast fail'. The browser will try to grab the resource, fail, then move on.

Some use cases for blocking include:

Testing to see how fast a site would be without adverts
Testing to see how fast a site would be without an A/B testing library

For this you would run a standard test, then run a second test with said request(s) blocked, then compare the two versions.

Block Requests Containing

This textarea allows you to block individual requests by doing a substring (case-sensitive) match. An example of what you could paste in the textarea and what it blocks is seen below:

// block all requests containing 'spacer.gif', 'ckeditor', 'texedit', and 'www.example-domain.co.uk'
spacer.gif ckeditor texedit www.example-domain.co.uk

Note that the blocked requests don't appear in the resulting waterfall at all, but you can see they were blocked at the top of the test page in the test details:

This method works by using the Chrome extension API (when testing in Chrome). An extension is used that blocks requests much like a standard adblocker does. Because of this, there can be some time increases due to the overhead of the extension API. When it comes to blocking domains (like we have with www.example-domain.co.uk in the example above), you are better to use the 'Block Domains' setting detailed next.

Block Domains

Very similar to the 'Block Requests' functionality, only this time it accepts a space separated list of hostnames (full, not wildcards). An example of what you could paste in the textarea and what it blocks is seen below:

// block Google Fonts, Google Analytics and Google Tag Manager
fonts.googleapis.com www.google-analytics.com www.googletagmanager.com

The great thing about the block domains functionality is that it runs at the DNS level rather than via the extension API (in Chrome). So it has less overhead compared to matching URL substrings. But for that reason it only works with full URL strings.

Note: It is possible to use both blocking functions via WPT scripting using the block and the blockDomains commands. It's also possible to do the inverse with the blockDomainsExcept command where you block all URL's except the one listed. Andy Davies has a great article on 'Measuring the Impact of 3rd-Party Tags With WebPageTest' where he uses this technique to great effect.

SPOF tab

SPOF stands for Single Point Of Failure. The textarea accepts a list of hosts that you want to silently fail (one host per line).

Now you may be asking "What's the difference between 'Block' and 'SPOF'?". As mentioned before when you block a URL substring or domain it will 'fail fast'. The browser will know almost instantly and move on with the page load. SPOF on the other hand will redirect the request to blackhole.webpagetest.org and then silently fail. This is essentially leaving the browser hanging waiting for a response.

A great use case for this is to establish what happens when one of your third-party dependencies fails (e.g. an ad-network, a/b testing tool, JavaScript CDN). Do you really want your site to slow to a crawl because a third-parties infrastructure (outside of your control) fails? This can happen even to the largest tech companies in the world as Google recently showed. So it's a good idea to test if you have a hard dependency with a third-party now, rather than wait for the failure.

Custom tab

The custom tab allows you to define custom JavaScript that will run at the end of a test to collect custom metrics (that will be returned in the results).

Custom Metrics

This is where you paste in the name of the metric being captured along with the JavaScript to be executed at the end of the test (remember to return a value from the JavaScript!). An extremely simple example of this in action can be seen below:

[num-total-scripts]
return document.querySelectorAll('script').length;

This example, taken from Tim Kadlec's excellent 'WebPageTest Custom Metrics with Request Data' article simply counts the number of script tags on the page and returns them via a custom metric. The results look like this:

Custom metrics data is also added to the JSON data that you can query via the API. More complex examples can be seen in the WPT documentation related to Custom Metrics.

Hidden settings

WPT comes with a number of hidden settings that aren't displayed in the UI by default. To enable these settings you must pass in a query parameter.

Bulk Testing Tab

You can enable this setting by appending ?bulk=1 to the WPT URLNote: You will need an account to use this setting. This will give you a whole new tab under the 'Advanced Settings' called 'Bulk Testing':

In terms of usage it's pretty self explanatory. This setting adds the ability to test multiple URL's all using the same test settings via the UI. You can either paste in a list of URL's into the textarea, or upload a file with a list. Make sure each URL is on it's own line. For example, if you wanted to test multiple pages on the BBC News website you would paste this:

https://www.bbc.co.uk/news
https://www.bbc.co.uk/news/coronavirus
https://www.bbc.co.uk/news/politics
https://www.bbc.co.uk/news/technology
https://www.bbc.co.uk/news/stories

You can also add labels to individual test pages like so:

Homepage=https://www.bbc.co.uk/news
COVID=https://www.bbc.co.uk/news/coronavirus
//...

Once you've submitted the test you can bookmark the page and come back to it later. No need to keep the window open, the tests run independent of the browser window:

Once completed WPT gives you a handy table with specific metrics listed, as well as links to many sources of raw data you can examine further:

The page even gives you the ability to compare test URL's via the checkbox and compare button to the left hand side of the table. Doing so will direct you towards the compare view page which I talk about more in the 'Visual comparison tab' section below.

It's also worth noting that all of these tests are visible from the 'Test History' link in the main navigation. They will each be listed separately, and a link to the 'Bulk Test' summary table can also be seen on the list. Again, you can compare results from this view as well:

Hidden input fields

There are also quite a number of query parameters that won't display anything in the UI, but they will add a hidden input field to the page source and pull the query value from the query parameter. These aren't all limited for use on the homepage, many can be used on the results pages too (e.g. 'result', 'details', 'compare').

iq (`?iq=`)

With this you can specify the JPEG image compression level used for screenshots and video capture. Value range between 30-100.

pngss (`?pngss=`)

Set to ?pngss=1 to save a full resolution lossless PNG of the fully loaded page being tested.

discard (`?discard=`)

The discard parameter allows you to discard the first x number of tests, but it is highly likely that this isn't implemented in the new agents. The value provided is the number of test runs to be discarded.

timeout (`?timeout=`)

This is the maximum time a test is allowed to run in seconds. The default is 120 seconds.

appendua (`?appendua=`)

This parameter accepts a string that will be appended to the user agent string. See the 'Append to UA String' section for more details about this setting.

keepvideo (`?keepvideo=`)

This parameter if set to ?keepvideo=1 when you run a test, will instruct WPT to keep the raw video without any additional compression so the frames can be analysed further.

medianMetric (`?medianMetric=`)

This parameter it used to instruct WPT which metric will be used to decide the median run. By default loadTime is used, but you can also pass in SpeedIndex if you wish to use that. I discussed this setting in the 'Repeat View' section earlier.

affinity (`?affinity=`)

I'll refer to the API docs for this explanation:

Specify a string that will be used to hash the test to a specific test agent. The tester will be picked by index among the available testers. If the number of testers changes then the tests will be distributed to different machines but if the counts remain consistent then the same string will always run the tests on the same test machine. This can be useful for controlling variability when comparing a given URL over time or different parameters against each other (using the URL as the hash string).

tester (`?tester=`)

This allows you to specify a specific test machine to run your test on. Note that this value must match a 'Computer Name' from the list of test machines found here, else it will never run.

minimal (`?minimal=`)

In theory this should run a cut down version of the tests optimised for individual run speeds. But at the moment it doesn't actually do anything. So consider it a work in progress.

noopt (`?noopt=`)

Enable / disable the optimisation checks as seen under the 'Performance Review' section in the test results (e.g. keep-alive, GZip, CDN Usage etc). Enable this by setting ?noopt=1, this will speed up your tests.

debug (`?debug=`)

This setting keeps the debug output of the agent along with the test results. Mainly used for debugging test instances.

throttle_cpu (`?throttle_cpu=`)

I touched on the methodology behind CPU throttling for mobile emulation in the 'Emulate Mobile Browser' section. The throttle_cpu parameter allows you to throttle the CPU for Chromium based browsers (Chrome, Edge, Opera, Brave, Samsung, Vivaldi). This excludes the physical Android devices. The amount goes from 0 (no throttling) upwards. The highest value in the settings ini is 5.5 for MotoE and AndroidOne emulated devices. This signifies that these devices will have 5.5x slowdown.

browser_width (`?browser_width=`)

Set the width of the browser window you wish to use in the test. Set using display pixels. Usage: https://www.webpagetest.org/?browser_width=800

browser_height (`?browser_height=`)

Set the height of the browser window you wish to use in the test. Set using display pixels. Usage: https://www.webpagetest.org/?browser_height=600. Set both height and width by listing query parameters: https://www.webpagetest.org/?browser_width=800&browser_height=600.

width (`?width=`)

Here we set the width of the actual viewport using CSS pixels. Usage: https://www.webpagetest.org/?width=320.

height (`?height=`)

Here we set the height of the actual viewport using CSS pixels. Usage: https://www.webpagetest.org/?height=640.

thumbsize (`?thumbsize=`)

This setting is used on the filmstrip page (compare.php) and it controls the size of the largest side in the in the resulting thumbnails in the filmstrip. For example, if the thumbnails are landscape, this value will refer to the width of the image. For portrait it will refer to the height of the thumbnail. You can enter a custom value, or you can select the preset thumbnail sizes from under the filmstrip. I've written a little more about the filmstrip view here.

fps (`?fps=`)

How many frames per second are captured by the video for desktop devices. This is set to 10fps. Increasing this value adds overhead to the tests.

discard_timeline (`?discard_timeline=`)

Don't keep the DevTools traces and timeline data after the test has finished and they have been processed.

htmlbody (`?htmlbody=`)

Set this value to ?htmlbody=1 if you only want to save the response of the base HTML page. This is a slightly different setting than discussed in the 'Save response bodies' section above which saves all the test responses.

disable_video (`?disable_video=`)

This setting stops ffmpeg from running on the instance (?disable_video=1). Usually, even if video=0 and a video isn't being captured, ffmpeg still runs.

lighthouseThrottle (`?lighthouseThrottle=`)

Use Lighthouse's version of CPU throttling with the Lighthouse audit rather that WPT's own version.

warmup (`?warmup=`)

This used to allow for a throw-away test run before collecting data to warm up server caches. This functionality is no longer hooked up.

Visual comparison tab

Now that we've gone over a lot of the basic and advanced settings above, explaining this tab is much easier. This tab is essentially a set of shortcuts allowing you to quickly compare two sites with each other using a real device on a network connection speed of your choosing.

Once the test has been started simply sit back and wait while WPT completes 3 test runs across each URL.

Once the tests have completed on all URL's, you will be shown the standard compare view for each of the sites under the corresponding test conditions:

This view allows you to visually compare the loading speed and render process for each of the sites. If you scroll down the page you can also compare waterfall charts. You'll also be presented with 6 graphs with information ranging from 'Visual Progress' all the way through to 'Total Bytes' loaded by each page (and their corresponding filetype). I've written about the compare view in the links below if you wish to know more:

Traceroute tab

This may be one of the least well used of the main tabs available in WPT. The tab allows you to complete an Internet Control Message Protocol (ICMP) trace route to a specific host / IP address from a particular test location. The test can be run multiple times (default is 3), and then you are presented the results via the UI:

In the example above I've chosen a test instance in Dulles, VA and the host edition.cnn.com. The test has been run 3 times, and each test lists 4 hops along with the time to get to each hop, and their corresponding IP address. If there's no response back you will simply receive a blank page once the tests have completed.

Quote from Pat's forum post below:

You can now run traceroutes from the test agents through the web UI. It's still pretty experimental but you can get to it here: http://www.webpagetest.org/traceroute
Here is a sample traceroute to o.aolcdn.com from the Dulles test agent.
When there's no response you'll get a blank entry (may update this to display a * so it looks more like the command-line). The traceroutes are done with ICMP for now (eventually I'd like to implement it as a TCP port-80 traceroute but that's a lot harder to do on Windows than it should be).

This is essentially the same as the traceroute command that you can run in a terminal window. Only instead or running it from your local machine you are running it from the location of the test instance.

A use case for this functionality would be if you were interested in the traffic across the network between the instance and site server at any point in time. If you happen to be seeing unusually poor performance from an instance to the site server, running a traceroute between the two will allow you to evaluate and understand why this may be occurring.

Common Scenarios

In this section I'm going to list a few scenarios you may want to test and the settings you will use. I'll add more of these over time as I encounter them.

Testing HTTP/2 against HTTP/1.1

Testing a site using both protocols is fairly straightforward using WPT. That is assuming your server already has HTTP/2 enabled. The method for testing relies on the fact that browsers that don't support a specific protocol will simply 'fallback' to a version that they do. This is in order that browsers 'Don't break the web'. If a browser were to simply fail on encountering an unsupported protocol, progress on the web would be very difficult!

Step one is to test the site using HTTP/2. Since it is already enabled, this is really just a standard test run. So select your device and connection speeds then run the test. Remember to add a label to the test so you know which version you are looking at later:

Next on the list is to run the HTTP/1.1 version of the test. Keep all the same settings, you just need one minor adjustment in the 'Chromium' tab. In the 'Command-line' input enter --disable-http2. Remember to add a label marking it as HTTP/1.1. We then run the 2nd test:

This command-line flag has disabled HTTP/2 in the browser so in order to communicate with the server it is having to fallback to HTTP/1.1. Now that we have both tests we are able to compare the results in WPT and examine which performs better.

Using the command-line to disable features is a great way to test the difference between new and old features. The challenge often lies in finding the right command-line flag. This page should help you find these.

You could use the same method to test:

the difference between TLS versions using --ssl-version-max
how a website loads without webfonts using --disable-remote-fonts
the difference preconnect makes using --disable-preconnect to disable it (assuming the flag is still available).

Testing a site without third-party scripts blocked

In this scenario we use the WPT scripting language to block third-party scripts. I'll give a brief overview but you can read a much more detailed version of this technique by reading the blog post 'Measuring the Impact of 3rd-Party Tags With WebPageTest' by Andy Davies.

First we will need to run a standard test run. This can be done via the UI or with a script. Here I'm using the UI:

Next we need to run the same test but with a small tweak. We will use the 'Script' tab to write a very simple script that will navigate to 'https://www.thetimes.co.uk/' and block all other domains:

Once both tests are complete it is time to compare the results:

Now the method for blocking other domains is quite a drastic one, as it may be blocking domains that the site requires (e.g. a CDN or an assets domain). But as a quick test to see the impact these domains have of performance it works really well. The stats from the 2 tests say it all really:

Visually complete: 11.2s vs 3.7s (block vs no-block)
Fully Loaded: 9.6s vs 40.8s (block vs no-block)
Requests: 39 vs 177 (block vs no-block)
Bytes: 1.1 MB vs 2.4 MB

The test above gives some quick insight into the impact third-party domains can have on performance.

Useful tips & tricks

In this section I'll document any useful tips and tricks I run into related to running a WPT test.

How do I see all the WPT advanced settings?

Here's a little tip from Andy Davies and Simon Hearne. What about if you don't want to click through all the advanced tabs to find the setting you are looking for? Wouldn't it be great if you could see them all on the same tab. Well, now you can with this handy little snippet:

document.querySelectorAll('.ui-tabs-hide').forEach(el => el.style.display='block')

Simply paste this into your browser DevTools console and execute it. The resulting page will look like this.

Conclusion

We come to the end of another long blog post related to WebPageTest. In the post I've tried to document every setting I could find related to running a WebPageTest test, and also give you an explanation as to what they do and how they can be used. Please consider this a living document, so if you have spotted any errors or I've missed anything, please do let me know. Credit will be given in the post changelog. If you would like to learn a lot more about WebPageTest, I highly recommend the book: "Using WebPageTest" by Rick Viscomi, Andy Davies, Marcel Duran.

Finally, another huge thanks go to Pat Meenan who I've inundated with random questions while writing this post!

Post changelog:

31/12/20: Initial post published.
31/12/20: Added info about needing an account for bulk testing, and the handy Wireshark snippit for faster examination. Many thanks to Andy Davies for this info!

Fixing a legacy HTTP/1.1 server configuration with Cloudflare Workers a CDN

2020-11-01T00:00:00Z

So the configuration discussed in this post I've run into a couple of times recently while examining the performance of some websites you'd class as running on 'legacy' infrastructure. When a site is running on HTTP/1.1, you expect to see multiple TCP connections opened to download all the site assets. The HTTP/1.1 specification (RFC 2616) used to recommend:

A single-user client SHOULD NOT maintain more than 2 connections with any server or proxy.

But that RFC was later obsoleted by RFC 7230 which changed this to:

Previous revisions of HTTP gave a specific number of connections as a ceiling, but this was found to be impractical for many applications. As a result, this specification does not mandate a particular maximum number of connections but, instead, encourages clients to be conservative when opening multiple connections.

In terms of practical application by browser vendors they usually set the maximum number of connections to a single domain to 6. So if you see a waterfall that has a higher number than 6 connections to the same domain, that's probably worth investigating:

In the WebPageTest waterfall chart seen above we have a total of 10 page assets being downloaded. But as you will notice from the orange and purple lines, there are 9 TCP connections established to do this! So what's happening here?

The `connection` header

On examining this waterfall closer it was possible to verify the server was running HTTP/1.1 as expected. But I noticed there was also a problematic little response header included too: connection: close. We can refer back to RFC 7230 for an explanation as to what this header is doing:

The "close" connection option is defined for a sender to signal that this connection will be closed after completion of the response.

Put simply, for this page communicating using HTTP/1.1, the server is opening up 6 connections to download the page assets. The inclusion of connection: close response header means that these TCP connections can't be reused for other assets after one has downloaded, they are simply closed. Another brand new TCP connection must be opened for the other assets. This isn't good! One of the major improvements of HTTP/1.1 over HTTP/1.0 was the ability to allow TCP connections to persist across multiple asset downloads. So by including the connection: close response header you are basically bringing back HTTP/1.0 for your users.

TCP connections are expensive

Now you may be asking: "What's the big deal?. How much difference can opening and closing multiple connections be?". Well it actually has a big impact on a number of areas:

Opening and closing connections takes up CPU time and memory on the server, client, and many other bits of hardware in-between.
Congestion across a network is increased because there are more packets being transmitted purely to setup TCP connections.
Web performance is decreased because there's added latency when established a new connection to download other assets.
More efficient TCP congestion control features can't be used. Each new TCP connection suffers from TCP Slow Start).

By minimising the opening of new TCP connections you are improving both the web performance and battery life for your users. Focussing internally you will also be reducing load on your server and the overall network. Basically everyone wins!

Focus on web performance

Lets take some time and focus on the web performance aspect of this configuration. If only there were a way to modify the request / responses coming from our inefficient server config... Thankfully there is! Using a Cloudflare Worker we can do exactly that.

How does it work?

First and foremost lets get something out the way. It actually doesn't. In this instance the Cloudflare Worker isn't actually doing anything for us. And this is where I regret putting the technology used in the blog post URL! It makes things very hard to change if it needs to! First I think it's important to go over how it doesn't work, and give you an idea of where my mental model broke down:

How it doesn't work

In my head I was seeing a Cloudflare Worker as a simple single layered black box that does everything for you. E.g. when a users connection is established they are connecting directly to the worker, and the worker is connecting directly to the origin. This isn't how it works at all. The worker isn't like a load balancer (which does pass through), it's much more complex than that.

Lots of layers

First it's important to understand that a connection is going through multiple disconnected layers, not just the worker itself. So for example the layers involved are:

Browser communicates back and forth with the Edge nginx
Edge nginx communicates back and forth with the Worker
Worker communicates back and forth with the cache tier
Cache tier communicates back and forth with the origin fetch servers
Origin fetch servers communicates back and forth with the origin

Each tier manages multiple requests independently. Like a pub/sub queue. Actual connections are on both ends but completely independent. Because of all these layers, we can simply remove the worker completely as just using a CDN implements things like HTTP/2, keep-alive, and Brotli. CDNs can fix a lot of broken origin behaviors just by simply being in the path, which is exactly what happens in my use case mentioned in this blog post.

If there are any folks from Cloudflare reading this blog and you happen to have a diagram of all the above layers, that would be super helpful to see, so please do share it with me. I had a look around the web and most diagrams just seem to display a Worker as a single box full of hidden tricks. It would be great to look at all the independent layers involved to get more of an idea of how they work together.

The example worker

So even though the use case I mention doesn't require a worker (only a CDN), I'm going to leave this section in as an example of how a Cloudflare worker works. For those of you familiar with Service Workers, a Cloudflare Worker is basically the 'edge' version of this API, with some limitations due to the environment not being the browser. With a little JavaScript you can write a fully programmable network proxy, allowing us to control how network requests from our pages are handled.

The example code run in the worker below is actually fairly straightforward:

// this is the site we are proxying
const site = 'www.example.com';

// do stuff when the 'fetch' event occurs
addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request))
});

/**
 * Respond to the request
 * @param {Request} request
 */
async function handleRequest(request) {
  const url = new URL(request.url);

  // x-host header added by WebPageTest when using overrideHost
  const xhost = request.headers.get('x-host');

  // if x-host header not found, abort
  if(!xhost) {
    return new Response('x-host header missing', {status: 403});
  }

  // set our hostname to that listed in the xhost header when using overrideHost in WPT
  url.hostname = xhost;

  // store the original response
  let originalResponse = await fetch(url, request)

  // create a new response to be modified (response properties are immutable)
  let response = new Response(originalResponse.body, {
      headers: originalResponse.headers,
  })

  // simple example of how you would add a custom header to the response
  // response.headers.set("x-custom-header", "header-value")

  // return the modified response
  return response
}

Then in WebPageTest we add the following script:

overrideHost www.example.com example-modifier.nooshu.workers.dev
navigate www.example.com

This script above is saying: when you come across any asset requested from www.example.com, route it through our Cloudflare Worker (example-modifier.nooshu.workers.dev) so the request / response can be modified accordingly. If you are interested in what else you can do with Cloudflare Workers, I highly recommend reading the blog post 'Exploring Site Speed Optimisations With WebPageTest and Cloudflare Workers' by Andy Davies.

What difference does it make to web performance

Now that we have a CDN sitting between the browser and the origin, we have the following setups available that we can examine using WebPageTest:

Original config (HTTP/1.1 + connection: close header)
HTTP/2 transformed by the CDN (connection ignored or removed)
HTTP/1.1 transformed by the CDN (HTTP/1.1 + connection: keep-alive header)

NOTE: To connect to the CDN using HTTP/1.1, you can simply use the --disable-http2 command line option in Chromium, so it drops back to HTTP/1.1.

So let's see what difference it makes:

Page waterfall (HTTP/1.1 + keep-alive)

In the waterfall chart below we can immediately see the difference this makes verses the chart shown at the start of the post. Instead of 9 separate TCP connections to download 10 assets, we see 4. A reduction of 56%.

Page waterfall (HTTP/2)

While we are examining waterfalls, we may as well look at a H2 version for comparison. HTTP/2's ability to multiplex streams over a single connection has reduced the TCP connections down to 2. The other TCP connection is the anonymous connection needed to download the web fonts.

Timings comparison

But where it all really matters is in the page timings. Here we can see a comparison chart with all 3 configurations visualised:

From the graphs you can clearly see that the original config (HTTP/1.1 + connection: close) is much worse for performance in almost all metrics.

Visually progress comparison

From the visual progress graph below we get an idea of how a user with perceive the performance of the website. It's easy to see from the graph that HTTP/2 is the winner. By rendering something to the viewport almost half a second quicker than our modified version of HTTP/1.1, and 1.1 seconds quicker than the original config! It's also worth noting that with the original config the page is rendered in a single big bang. From 0% to 98% visually complete over what looks like a single frame. Where as H2 and H/1.1 + keep-alive build the page more incrementally. This is essentially giving a user something to look at on their device sooner. This is always a good idea as it shows that something is actually happening.

Testing

If you are wondering how you test to see if a server is configured with connection: close enabled, there are a couple of quick methods:

DevTools

Both Chrome and Firefox DevTools allow you to view response headers as separate columns in the 'Network' tab:

So if you see a row with 'HTTP/1.1' as the protocol, and 'close' under the connection column, then you know the server config isn't optimal.

Curl

Or if you are looking for a way to test this via the command line then you can always use curl. Simply update the server URL and run the following command:

curl -sIX GET https://www.example.com | grep -E 'Connection|HTTP/'

# Output:
# HTTP/1.1 200 OK
# Connection: close

It will extract the information for you and output it to the terminal window.

Conclusion

Now I admit this blog post hasn't actually 'fixed' the issue at the origin unless you plan on dropping a CDN in front of it (you should seriously consider doing this). But what it has done is give you actual performance data you can use to 'build a case' to get it fixed. If you report back to a business and / or management with graphs that show that by making a small change on the server both web performance and server load will improve. You could even use it to push for enabling HTTP/2 if a considerable improvement is seen, or even the use of a CDN.

It's amazing what a difference a single header can make when it comes to web performance. There are probably so many 'legacy' server configs running across the web with this exact issue: where just a small modification to the config will make an enormous difference. So if you have a server that you know is still running HTTP/1.1, why not do your users (and your server) a favour and check you don't have connection: close included in the response headers.

Post changelog:

01/11/20: Initial post published.
02/11/20: Update to mention that RFC 2616 was obsoleted by RFC 7230, which includes updated connection concurrency guidance. Thanks Barry Pollard.
02/11/20: Update to the post after a conversation with Pat Meenan. This version now calls out where my mental modal of Cloudflare Workers was incorrect and adds a bit more details on how it all works! Thanks (again) Pat!

Using jq with the WebPageTest API

2020-10-09T00:00:00Z

I've written quite a few blog posts on this blog about WebPageTest (WPT). The reason for this is it tends to be the first tool I (and I'm sure many) look to when they want to debug web performance issues with a website. Underneath its retro looking user interface (UI) is an incredibly powerful data collection tool for you to examine and analyse. Now what you may not realise is that all the data you can see in the UI, is also available via the WebPageTest API.

Update: All the jq scripts from this post can be found in this wpt-jq-data-scripts repo, along with a link to the example graphs in Google Sheets that can be found here.

Accessing the API

Accessing the API is very simple. You can either enter the URL https://www.webpagetest.org/jsonResult.php?test=[TESTID] in your browser, or follow the 'View JSON result' link on a tests summary page (see below):

Follow this link in Firefox and you will get a loverly inbuilt JSON viewer with a filter tool to quickly find what you are looking for. But if you happen to be a Chrome user you're out of luck, as you will simply get the raw JSON output on the screen. But there are JSON formatter extensions on the Chrome Web Store you can install should you want this functionality.

The JSON blob

So what comes out of the API in terms of data, and how much data are we talking about? Frankly it's a lot of data! The JSON blob contains detailed information about every single test run, and all of the individual file requests (including cookie and header information) for each run too. So you can imagine that as the number of runs increases (to give you a more stable median), the file size increases dramatically. For example, a single test run using Chrome for the homepage of this blog produces a JSON blob that is approximately 520KB in size (uncompressed / unminified). Taking it to the other extreme, with the GDS private instance of WebPageTest I've run a number of tests with 200+ test runs. The JSON blob from these tests is around 50 MB (uncompressed / unminified). So if you are looking for a great way to stress test your JSON viewer or parser, the WebPageTest API is an excellent way to do it!

Bringing in jq

So the output from the WebPageTest API can be pretty huge. Packed full of interesting data waiting to be analysed and explored. But how do you do that in a quick and easy way? That's where jq comes in. So what is jq? I think the jq author Stephen Dolan explains it best:

jq is like sed for JSON data - you can use it to slice and filter and map and transform structured data with the same ease that sed, awk, grep and friends let you play with text.

Basically jq is an incredibly powerful tool for manipulating JSON data. It can be used to extract basic data, or even fully restructure it if required. When I say it is a tool, that's actually doing it a huge disservice. jq is in fact a Turing Complete programming language. Proof of its turing completeness can be seen here, with a Brainfuck interpreter written in jq.

So how can we use jq with the WebPageTest API to simplify our workflow and extract that data we need? That's what the rest of this blog post will be talking about.

The basics

So first, the absolute basics. For readers who already have experience using jq, you can probably skip this part. But I'm always one for being inclusive and covering first principles in blog posts. Downloading / installing is fairly simple, and there are plenty of ports available with instructions on the download page. Once installed we can use curl to grab our JSON blob like so:

curl -s https://www.webpagetest.org/jsonResult.php?test=[WPT_TEST_ID] > test-data.json

Here we are saving our test data to a test-data.json file on the local machine. Once done, we can read it using jq:

jq '.' test-data.json

This command will print the whole contents of the JSON file to the screen. The '.' is known as the identity filter. It takes the input, and pipes it directly through to the output. This is an excellent way to pretty print a set of JSON data:

jq '.' test-data.json > pretty-test-data.json

The above method works, but it's very long-winded. Thankfully jq has the ability to either read a file, or it can take its input from stdin. So the above commands can be simplified to:

curl -s https://www.webpagetest.org/jsonResult.php?test=[WPT_TEST_ID] | jq '.' > pretty-test-data.json

Here we are passing the output from the (silenced with -s) curl command into jq. jq is then pretty printing it to a file on the computer.

The above commands are the basis for the rest of the commands listed in this post.

What data is in the API

So let's assume we don't have a JSON viewer to dig down into the data. So what can we do to help us understand what data the API contains? There are a couple of jq commands that can do that:

# query the API, pipe into `jq`
curl -s https://www.webpagetest.org/jsonResult.php?test=[WPT_TEST_ID] | jq -r '.data | path(..) | map(tostring) | join ("/")' > full-schema.txt

# example output
fvonly
successfulFVRuns
average
average/firstView
average/firstView/minify_total
average/firstView/responses_200
average/firstView/testStartOffset
average/firstView/bytesOut
average/firstView/gzip_savings
average/firstView/requestsFull
average/firstView/start_epoch
average/firstView/cpu.MarkLoad
average/firstView/connections
average/firstView/cpu.EventDispatch
...

Examining this filter:

.data - navigate down into the data object
path(..) - recursively decent into the large structure and extracting all the paths
map(tostring) - map the array into a new array and convert paths to strings
join ("/") - join each string together with a forward slash

We then output this result to a file. What this has now created is a high-level schema as to how the WebPageTest API is structured. If you are interested in the full output of this command you can find the Gist here (opens in new tab).

Top level structure

Examining the API closely a pattern emerges. At the top level you have some basic test run data and a set of objects with (much) more data:

average - contains data about the average for each metric across all runs
standardDeviation - contains data about the standard deviation of each metric across all runs
median - contains data about the median run (depending on median metric that has been selected)

Delving a level deeper we find each of these has the following:

firstView - data from the first view of the page
repeatView (if selected in the GUI) - data from the repeat view of the page (useful for examining caching between page navigation)

And within each of these we come to a huge amount of data that WebPageTest has gathered from the selected browser when loading the page. And this same data structure can be found within each of the runs objects at the top level:

runs - each of the individual run data is contained within here
runs/1 - data about run 1
runs/2 - data about run 2
runs/n - data about run n...

Inner data structure

We can use another jq filter to extract information about the inner data structure within the average, standardDeviation, median, and individual run numbers:

jq -r '.data.median.firstView | to_entries | def parse_entry: {"key": .key, "value": .value | type}; map(parse_entry) | sort_by(.key) | from_entries' > inner-structure.json

# example output
{
  "Colordepth": "number",
  "Dpi": "string",
  "Images": "string",
  "LastInteractive": "number",
  "LayoutShifts": "array",
  ...
}

Examining this filter:

.data.median.firstView - navigate down into the median firstView data
to_entries - convert an object into an array of objects with a key and a value
def parse_entry - define a jq function to rewrite the object into key then examine the type of value it is
map(parse_entry) - create a new array, running each object through this function
sort_by(.key) - sort result alphabetically by the key name
from_entries - convert the array back into an object

Then we output the final modified JSON to a file to be used where needed. The full output of this command can be seen in this Gist here (opens in new tab).

Problematic keys

Update: So Pat Meenan has come to the rescue (again) and quickly solved this issue. If you append normalizekeys=1 to a JSON response, the key names will now be normaized, stripping away any non-alphanumeric characters. For the API this means any ., -, and _ characters will be removed. I'm leaving in the section below for future reference.

It's worth mentioning at this point that the naming of some of the keys coming from the API can be problematic depending on how they are being used. There are many keys that contain a . which may not sound too bad, but if you are extracting the data using JavaScript, it causes issues. For example, to access values in a JavaScript object you may often use the dot notation:

var simple_example = {
	"connectivity": "Cable",
	"loadTime": 887,
	"firstImagePaint": 627
}

// Accessing values
simple_example.connectivity // "Cable"
simple_example.firstImagePaint // 627

But look what happens when one of our keys contains dots:

var simple_example = {
	"connectivity": "Cable",
	"loadTime": 887,
	"firstImagePaint": 627,
	"cpu.v8.compile": 21
}

// Accessing values
simple_example.cpu.v8.compile // "cpu is undefined"

The JavaScript is looking inside the simple_example object and looking for the cpu object, then for the v8 object, until it finds the compile key. Obviously this chain doesn't exist as cpu.v8.compile is a single key name, so the JavaScript fails.

There are also another set of keys that can cause issues too: some of the keys include - in the name. This is again an issue when using the dot notation to try to extract the value. The presence of - in the key is seen as a subtraction expression, e.g. data.median.firstView.score_keep-alive is seen as data.median.firstView.score_keep - alive, which of course fails. There is a fix for this issue in JavaScript. You can use the array syntax to look up the values:

var simple_example = {
	"score_keep-alive": 1,
	"cpu.v8.compile": 21
}

// Accessing values
simple_example['score_keep-alive'] // 1
simple_example['cpu.v8.compile'] // 21

But wouldn't it be great if we could get jq to clean the data for us, so we don't need to even worry about it? Thankfully we can with this filter:

jq 'walk( if type == "object" then with_entries( if .key | contains("-") then .key |= sub("-";"") else . end) else . end) | walk( if type == "object" then with_entries( if .key | contains(".") then .key |= gsub("\\.";"") else . end) else . end)'

It's a bit of a beast, but it is basically walking through the data examining the keys. If they contain a - or a . then it is replaced with nothing.

NOTE: This isn't the most efficient filter since it is walking through the data twice. If anyone has any pointers on how to reduce this down to a single walk() function call, please let me know.

Scenarios

Now that we've taken a closer look at the data available inside the API, let's look at a few jq commands we can run to extract the data we want.

Extracting first and third party domains

This isn't as hard as it sounds since the API returns a domains object with each domain visited in the test and some connection information about the domain too. Using jq we can easily extract a list of all the domains. Note the use of the --raw-output option (or -r), which stops the resulting output being formatted as JSON. In this example we don't want quoted strings, we want them as a simple list.

jq -r '.data.median.firstView.domains | to_entries[] | .key' > domains-list.txt

Or if you only wanted first-party domains (assuming 'nytimes.com' is the website being tested):

jq -r '.data.median.firstView.domains | to_entries[] | select(.key | contains("nytimes.com") ) | .key' > first-party-domains-list.txt

Or what about the inverse, if you wanted all third-party domains:

jq -r '.data.median.firstView.domains | to_entries[] | select(.key | contains("nytimes.com") | not ) | .key' > third-party-domains-list.txt

So where might this be useful? Well WebPageTest has the ability to block requests to certain domains, so you could quickly block any / all third-party domains by copy / pasting the result of this command into the GUI. If you are wanting to block all third-party requests there's an easier way using WebPageTest scripting (blockDomainsExcept) that Andy Davies covers in his 'Measuring the Impact of 3rd-Party Tags With WebPageTest' blog post. But this method gives you a way to gradually add / remove domains from the list and retest the impact of each if required.

Plotting connection setup metrics across runs

Here we'll look at plotting the values from a large number of test runs related to the connection setup time (similar to the graphs I created in this blog post). So we are looking for the following values from the API:

DNS start & end (runs.number.firstView.requests.0.dns_start, runs.number.firstView.requests.0.dns_end)
Connect start & end (runs.number.firstView.requests.0.connect_start, runs.number.firstView.requests.0.connect_end)
SSL start & end (runs.number.firstView.requests.0.ssl_start, runs.number.firstView.requests.0.ssl_end)

Note: I realise the API comes with dns_ms, connect_ms, ssl_ms, but in the tests I'm looking at there are a lot of 0 values for this key, so I will calculate it manually.

Our plan here is to extract only the data listed above and output into a comma-separated values (CSV) format. That way we can easily import the data into any graphing tool and graph the data. Here's the command for this (version 1):

jq -r '["DNS Start", "DNS End", "Connect Start", "Connect End", "SSL Start", "SSL End"],(.data.runs | to_entries[].value.firstView.requests[] | select(.final_base_page == true) | [.dns_start, .dns_end, .connect_start, .connect_end, .ssl_start, .ssl_end]) | @csv' > csv-output.csv

This is a long command, so let's break it down a little:

["DNS Start", "DNS End", "Connect Start", "Connect End", "SSL Start", "SSL End"] - these are the headers for each of the columns in the resulting CSV file
.data.runs | to_entries[].value.firstView.requests[] - drill down into the runs data and convert each run into a set of data objects with a key and a value (and remove them from the wrapping array). Then pull out the runs individual requests from the first view and unwrap from the array.
select(.final_base_page == true) - here we are filtering to find the request that has come from the HTML of the page (that will contain the connection information we need)
[.dns_start, .dns_end, .connect_start, .connect_end, .ssl_start, .ssl_end] - construct a new array only containing the data we will need in the CSV file
| @csv - pipe the final array wrapped data into the CSV formatter
> csv-output.csv - output to a file

This gives us a CSV file with 6 columns, and from those columns we can calculate the time each of the DNS, Connect, SSL stages took. But we can actually get jq to do that calculation for us:

jq -r '["DNS", "Connect", "SSL"],(.data.runs | to_entries[].value.firstView.requests[] | select(.final_base_page == true) | [(.dns_end - .dns_start), (.connect_end - .connect_start), (.ssl_end - .ssl_start)]) | @csv' > csv-output.csv

The above command is very similar to version 1, only now we have got jq to do some of the calculation leg work for us, as it can do the basic arithmetic when the new array of data is being constructed. So no need to do it in our spreadsheet / graphing tool now.

It's worth mentioning that we've just reduced a 46MB JSON file with 201 test runs down to a 2KB CSV file by discarding all the information we weren't interested in.

Importing the data into Google Sheets, it is now possible to create a graph that look like this:

Now I don't know about you, but the thought of having to type out those 289 characters to run that command doesn't sound like much fun. So let's look at how we can simplify the workflow.

`jq` command reuse

There's a really useful bit of functionality where jq can be used in a shebang line. Using this method it is possible to hide the jq code inside a separate file, then point your JSON at the file and out pops the data in the format we need! So if we take the complicated looking filter above, and create a connection-metrics.jq file (gist here (opens in new tab)):

#!/usr/bin/env jq -rMf

# Filter to extract DNS, Connect, SSL metrics from WebPageTest run JSON

# Headers for resulting CSV
["DNS", "Connect", "SSL"],
	# drill down into the runs data
	(
	.data.runs
	# convert the run data into an object and drill down into the run request data
	| to_entries[].value.firstView.requests[]
	# only select the request that corresponds to the HTML page (so it has DNS,Connect,SSL data)
	| select(.final_base_page == true)
	# build an array of the resulting data we want in the CSV
	| [(.dns_end - .dns_start), (.connect_end - .connect_start), (.ssl_end - .ssl_start)]
	)
	# pass to the CSV formatter
	| @csv

Notice how we can now easily comment the filter and split it over multiple lines for better readability. Save this to a file, make sure the file can be executed (chmod +x connection-metrics.jq), then point it at your WebPageTest JSON like so:

curl -s https://www.webpagetest.org/jsonResult.php?test=[WPT_TEST_ID] | ./connection-metrics.jq > result-output.csv

or for a JSON file on your local machine:

./connection-metrics.jq wpt-api-input.json > result-output.csv

Both of these commands produce a CSV output like below, that can be easily imported into any program that happens to support CSV's.

"DNS","Connect","SSL"
37,29,45
32,30,45
33,29,45
...

Use as a template

So the great thing about the above example is it can now be used as a template for extracting other values from each of the WebPageTest runs. The command has already drilled down to the run level data after all. So what about if we extract and plot some other metrics? For example lets look at:

Time to First Byte (TTFB)
First Contentful Paint (firstContentfulPaint)
Start Render (render)
DOM Complete (domComplete)
Fully Loaded (fullyLoaded)

We modify the above example script to find the new metrics (seen in this gist (opens in new tab)), then run it over our data set. This then gives us a graph that looks like this:

So over the 201 test runs it's clear to see that the TTFB was very stable, with DOM Complete and Fully loaded second in terms of stability. FCP and start render are almost identical and jump around a lot. But there's an issue: there's a huge spike in the results which is clearly an error, and it breaks the scale of the graph. Now of course we could look through all of the data manually and remove it, but lets get jq to do it for us. After all if we ever had a data set with lots of these occurrences, automating the task is going to save us time.

First we need to find a way to pinpoint the broken test run. The most stable metric is TTFB, which we have a fairly good idea about what this value should be (~2000ms), so we can use it to filter out any large differences from the data. To do this we simply add a | select(.TTFB < 2200) into the filter. With this we are only allowing run results through that have a TTFB of less than 2.2 seconds. Here's an updated gist (opens in new tab) with the minor code changes.

As you can see from the graph above, the vertical axis is a lot more applicable to our overall results, thus giving us a better idea of how the metrics fluctuate over time.

Examining CLS information

There's been a lot of talk recently about Cumulative Layout Shifts (CLS), since it is now considered part of Googles Core Web Vitals. In the future a poor CLS result could actually hurt your search engine page rank (at least with Google). Thankfully WebPageTest monitors a page for layout shifts and gives detailed information about when they happen and the score for each shift. We can use this information to plot the layout shifts for each run, and see how they vary between them:

jq -r '.data.runs | to_entries[].value.firstView | [.LayoutShifts[].score] | @csv' > cls-scores.csv

This is a relatively simple filter:

.data.runs | to_entries[].value.firstView - drill down into the runs data and convert each run into a set of data objects with a key and a value (and remove from the array). Then extract the runs first view data.
[.LayoutShifts[].score] - look at all the layout shift information, unwrap from the array and pull out each score, wrap in an array for the CSV formatting (CSV's require an array an an input)
@csv - pipe the final array wrapped data into the CSV formatter
> cls-scores.csv - output to a file

Plotting this data in Google Sheets gives you a graph that looks like this:

The site tested above is the old web performance community favorite: CNN.com. 21 runs using Chrome on a cable connection. Each column on the x-axis shows an individual run. The y-axis shows the cumulative CLS score for that run. Each of the colours is a separate 'shift' that occurred during page load. What strikes me about this graph is the difference between each of the runs, with the lowest cumulative score of around 0.04, and the highest just under 0.8! If Google is going to be factoring CLS into search rankings (presumably from user RUM data), I'd personally want this score to be a little more stable!

This graph could be used by a team as a prioritisation exercise. The blue column is large and prominent in each of the runs, so this would be worth investigating as a priority to see if this shift can be reduced.

More examples to follow...

As I come up with other examples of data to extract from the API using jq, I will post them here. If you have a handy jq snippet you are willing to share please do let me know, full credit will be given to you.

Tools to help manipulate JSON data with `jq`

So how do you manipulate megabytes of JSON data quickly? Because, I have to be honest there's a lot of trial and error with jq (at least for me). Oftentimes what I think will happen with a particular filter, doesn't. So the ability to tweak filters and see the result quickly is key to using jq. And as a VS Code user there are a number of extensions available to optimise jq workflows. The fact that WebPageTest outputs such large JSON blobs makes it quite a resource intensive use case for these extensions. So let's examine how well they work.

Visual Code jq playground

Visual Code jq playground is the most feature complete extension of the three that I've looked at. It has great documentation and examples, and also has the ability to show the results in an editor window or the 'Output' tab. It managed to return the manipulated results from a 50 MB JSON blob really quickly.

The major downside that I have found with this extension is that it only works on file types with a .jq extension. So when you have a huge JSON blob from WPT, this needs to be renamed to blob.jq. Once done you type your filter directly above the JSON (as seen in the image). You can then show the result in the console, or another editor window. I'm not a fan of both of these workflow requirements. If this extension could be filtered via a separate input box, and it worked on .json files, then this extension would be almost perfect. Find the Visual Code jq playground here.

vscode-jq

At first it took me a little while to figure out how to use this extension as the documentation is non-existent. You need to open the 'Command Pallet..' then type >jq, at which point you will be given an empty input box to enter your filter string. Press 'Enter' and the result will be displayed in the 'Output' panel at the bottom of the window (see image below).

I tested the plugin with very large JSON files, all the way up to 50 MB and it worked perfectly. It was also very quick to return a result considering the size of some of the files. There are a couple of downsides though: The 'output' tab at the bottom of the window is very limited in terms of vertical size. And as JSON files tend to spread over thousands of lines this can be quite frustrating. Second, there's no JSON syntax highlighting in the 'output' tab, so it can be quite hard to read the output. And finally, the history of previous filters isn't saved. Considering the trial and error aspect of jq filters, this is a real shame. Having to retype the whole filter again for a minor change means debugging is slower than it needs to be. All in all, I love the speed and JSON size capacity, it's just missing a few features from my workflow to be perfect. Find the vscode-jq extension here.

jq Live View

This extension looked really promising. Upon opening a JSON file you get a small 'jq' icon in the top right of the window which you click to bring up the filter input. Input your filter, press 'enter' and a new file window pops up with the result. It also remembers your history of filters which is a nice touch.

Unfortunately it comes with a major flaw when it comes to working with WebPageTest JSON blobs. I could only get it to work with files that were less than 1 MB in size. The error stdout maxBuffer length exceeded with large files raised its ugly head with files larger than 1 MB. To put that into context, passing this site's homepage through WebPageTest with 3 runs creates a 950 KB blob (which worked). Upping that to 5 runs creates a 1.4 MB blob (which failed). I'm going to investigate and see if there is a way to fix this issue, as I really like the extension and think it is almost perfect. Quick and easy to use, just not recommended for WPT blobs at the moment! Find the jq Live View extension here.

jq>play - A playground for jq 1.6

To be clear, this isn't an extension for VS Code, it is a separate website created by the author of jq that allows you to manipulate JSON in (almost) real time. It gives you a number of pointers below the main panels as to what manipulation options are available. But be warned: clicking on one of these will clear your current filter text!

It's a fantastic website for quickly seeing what a filter does to pasted JSON code. But it does come with a few limitations: The JSON file size limit is 10 MB, so no super massive JSON blobs allowed. There's also no input delay on the filter input text box, so while you are typing you will likely see compilation errors due to an incomplete filter. This can be very annoying, especially when filtering a large amount of data, as it takes a number of seconds to complete. A setting to toggle this functionality on and off and replace it with a submit button would solve this issue. So if you do decide to use this tool I highly recommend configuring your filter on a JSON blob with a low number of runs first, then transfer this over to the jq command on the command-line.

Conclusion

So in this blog post we've looked at the WebPageTest API, figured out how to reduce the amount of data to something more manageable using jq, and looked over a few examples as to how jq can be used to extract specific web performance data for further examination. Finally we looked at a few tools to help with the writing of jq filters when manipulating JSON data. I hope you found this post useful. It really has only scratched the surface of what is possible with jq.

If you have any go to filters of your own, or any feedback on the ones written here please do let me know. Thanks for reading.

Post changelog:

09/10/20: Initial post published.
09/10/20: Post updated with details about the new normalizekeys=1 parameter Pat Meenan has added to WebPageTest to strip out non-alphanumeric characters from the JSON response. Thanks Pat!
14/10/20: Added link to the 'wpt-jq-data-scripts' repo with all the scripts and more for people to use. Feedback and ideas welcome!

Why you should be testing your 404 pages web performance

2020-08-25T00:00:00Z

So some recent web performance monitoring work for GOV.UK piqued my interest in the web performance of 404 pages. In this post I'm going to go into some of my thoughts around this and the finer details of existing 404 pages on the web today.

Anatomy of a 404 page

So before I start, let's get the basics out of the way. What is a 404? Well I'd hazard a guess that the 404 status code is the most well known of all the HTTP status codes. If you were to ask a random person on the street what a 404 is, I'd bet many could guess as to what it is. Maybe something along the lines of "it's when the website is broken", or "the page is broken".

Let's take a look at what the Internet Engineering Task Force (IETF) RFC 7231 has to say about the 404 status code

The 404 (Not Found) status code indicates that the origin server did not find a current representation for the target resource or is not willing to disclose that one exists.

So saying the website or page is broken is close, but it isn't always about 'pages', as the rfc mentions the word 'resource'. The user will most likely see a 404 as a broken page. A browser on the other hand could be seeing 404's for a huge number of reasons, and in many cases a user won't even realise this. I can't stress this point enough: 404 responses aren't just caused by a user mistyping a URL, there's a whole lot more to it than that. I'll back this claim up with data later in the post.

What decides what is sent in a 404 response

When a browser requests a resource from the server, and the server doesn't have it available, what decides what is sent back to the browser? Well multiple representations of a resource can sit on the same URL. The process for selecting which resource is the best representation to be used and sent back to the browser is called content negotiation. The MDN documentation explains it well. The key part is:

The server uses [HTTP headers] as hints and an internal algorithm chooses the best content to serve to the client. The algorithm is server-specific and not defined in the standard.

So the browser gives the server as much information as it can about the type of content it supports and what it is expecting (as a hint), then it is the decision of the particular server's algorithm exactly what is in the response. And this is where the selection happens. It then depends on how the server is configured as to if a custom error page is served (e.g. site branding, more natural languages, site CSS and JS), or if the servers default '404 Not Found page' is used.

Collecting and refining the data

In terms of 404 status data, thankfully there's an awesome project that has been running for a number of years called the HTTP Archive, which tracks how the web is built. Every month, millions of URLs are 'crawled' and a huge amount of data is captured and stored that can then be used by the web community to learn about the state of the web today, and how it is changing over time. This data just so happens to include error data too! So we can take a look at the current state of 404 pages across the web in great detail.

Using a fairly straightforward SQL query we can pull out the response body size of all the 404 pages in the HTTPArchive as of June 2020:

#standardSQL
select status, respBodySize, reqBodySize, url
from httparchive.summary_requests.2020_06_01_mobile
where status = 404
order by respBodySize desc

Big thank you for Andy Davies for the query and pulling the data from the archive. Just to give you an idea of how much data comes back: it gives you a CSV file containing 3,382,962 rows, and is around 290MB uncompressed! Given the maximum file size for a CSV in Google Sheets is 20MB, and there is a cell limit of 5 million cells, we're going to need to reduce the amount of data by a lot at some point in this post!

Full dataset breakdown

Before I refine the dataset to make it more manageable, let's gather some high-level information about where all these 404 errors are coming from. First thing to do is to de-duplicate the data, making sure any repeated URLs are removed. For many sites you're going to get the same 404 repeated across multiple pages. Doing this removes 443,080 rows from the CSV, so not a bad start! You can find a Google Sheet with my basic analysis of the full dataset if you'd prefer to view it there.

First let's see the file types that are missing which are causing the 404 errors:

Filetype	Number of occurrences	% of total
*.ico	639,344	21.75
*.jpg	618,335	21.03
*.png	526,360	17.90
*.js	298,193	10.14
*.css	148,884	5.06
*.gif	144,576	4.92
*.woff2	54,758	1.86
*.woff	53,202	1.81

Unsurprisingly image files are the main cause of 404 errors. But the top filetype is *.ico. Any guesses as to why this is? Winner gets the prize of one lovely favicon.ico to download.

What about if we look for common files in these 404 errors:

Filename	Number of occurrences	% of total
favicon.ico	632,384	21.51
ajax-loader.gif	19,855	0.68
favicon.png	18,247	0.62
manifest.json	11,411	0.39
site.webmanifest	4,783	0.16
sw.js	1,859	0.06
spacer.gif	1,375	0.05
Favicon.ico	295	0.01

So I think we've confirmed the answer as to why the *.ico filetype has such a large number of occurrences. That's 21.51% of the unique URLs listed in the full dataset are caused by missing favicons!

Finally let's look for common URL patterns that are causing 404 errors. These are just a few that I pulled out that I found quite interesting:

URL Pattern	Number of occurrences	% of total
"/wp-content/uploads/"	375,123	12.76
"/wp-content/themes/"	136,303	4.64
"apis.google.com"	71,264	2.42
"/wp-content/plugins/"	65,967	2.24
"cdn.shopify.com"	42,871	1.46
"fontawesome-webfont"	33,916	1.15
"/undefined"	22,482	0.76
"/null"	9,540	0.32
".gov.uk"	971	0.03
".nhs.uk"	551	0.02
"/Array"	470	0.02
"[object%20Object]"	292	0.01
"/NaN"	153	0.01

The top two URL patterns are related to WordPress, which to be honest isn't very surprising, considering according to the 2019 Web Almanac the most popular CMS (by far) is WordPress.

Refined dataset breakdown

So the full dataset with 3,382,962 rows is far too big to manipulate and analyse (for me using Google Sheets), so moving forwards I decided to reduce the original dataset by:

removing any URLs with a response body size of less that 100KB
again de-duplicated the remaining urls to make sure they were unique

This reduced 3,382,962 rows down to a much more manageable 11,621 rows. Why 100KB? Good question. It just seemed like a fairly round number which gave a suitably large number of results remaining. I personnel think 100KB for a 404 page is still quite large (but much more on this later). If you are interested in the resulting raw data, you can find the Google Sheet here.

NOTE: For readers curious about how to access HTTPArchive data, I highly recommend reading 'Getting Started Accessing the HTTP Archive with BigQuery' written by Paul Calvano, and 'Using BigQuery without breaking the bank' by Tim Kadlec.

So for the top 11,621 URLs (in terms of response size) that were 404 errors, what file types are missing that are causing the errors:

Filetype	Number of occurrences	% of total
*.png	2,201	18.94
*.jpg	2,120	18.24
*.js	1,962	16.88
*.ico	1,836	15.80
*.css	533	4.59
*.gif	392	3.37
*.json	348	2.99
*.woff	285	2.45

Icon files are no longer top of the list in the refined dataset, but images and scripts are.

What about if we again look for patterns in common files causing these 404 errors:

Filename	Number of occurrences	% of total
favicon.ico	1,757	15.12
manifest.json	259	2.23
favicon.png	230	1.98
site.webmanifest	112	0.96
ajax-loader.gif	94	0.81
serviceworker.js	12	0.10
sw.js	8	0.07
Favicon.ico	7	0.06
manifest.webmanifest	7	0.06
spacer.gif	0	0.00

So even in our refined dataset the favicon.ico file creates real problems!

Finally what about common URL patterns causing the 404 errors:

URL Pattern	Number of occurrences	% of total
"/wp-content/uploads/"	2,284	19.65
"/wp-content/themes/"	1,294	11.14
"/wp-content/plugins/"	1,116	9.60
"/undefined"	255	2.19
"fontawesome-webfont"	166	1.43
"/null"	33	0.28
"/Array"	23	0.20
".gov.uk"	2	0.02
"apis.google.com"	0	0.00
"cdn.shopify.com"	0	0.00
".nhs.uk"	0	0.00
"[object%20Object]"	0	0.00
"/NaN"	0	0.00

Again, the top entries are all related to the WordPress CMS. A Google Sheet with my basic analysis of the refined dataset can be found here.

Response sizes

A really important aspect of these 404 responses I've only briefly touched on so far (other than the 100KB limit) is the actual response body size that is being sent to a clients browser. When a user visits a site's 404 page, either directly or unseen via the browser requesting a resource that doesn't exist on the server, the server is going to respond with a 404 status code and the associated 404 page. Looking at the refined dataset gives you some pretty staggering examples of 404 response bodies that are quite frankly huge. There are 118 examples of 404 pages that are 1MB+. 17 examples where the response body size is 2MB+. And the award for the largest 404 response body size is a whopping 6.7MB!

Huge 404 page waterfall

So what does a WebPageTest waterfall chart look like for one of these sites that has a huge 404 page response? I've picked the 2nd largest on the list for the moment as it is a very basic direct example. I will examine the first place website later in the post, as it comes with quite a few quirks! To force WebPageTest to run a test on a 404 page I've had to follow the instructions I have written here, as if you don't WebPageTest will simply abort the test (it is giving a 404 error response after all, so it's to be expected).

In this particular instance the HTML makes up 85.4% of the total page weight, or 7.5MB on its own! This looks to be mainly inline CSS within the page. After forcing the test to run for 20 seconds the page still hadn't fully loaded. My guess is that there's some broken server code or a plug-in that has gone off the rails and is injecting all this CSS multiple times and increasing the page weight by megabytes! The fact that I forced the test to run for so long, and it had only just finished loading suggests to me that the size logged in the HTTP Archive data is actually smaller than it should be!

That is one huge 404 page! And given that this is a page that ultimately a user doesn't actually want to be on, that's a lot of wasted time, data, and device resources!

Examining the breakdowns

Now that we have the broken down what is causing the 404 error data, what conclusions can be drawn from it?

Favicons

21.5% of the unique URLs in the complete data set came from a missing favicon.ico. Refining the data down to the top 11,621 URLs with the largest response bodies reveals that 1757 (or 15.1%) URLs on that list show the same issue. It just so happens that the largest 404 page in the whole data set is being triggered because of a missing favicon.ico on the server. By default a browser automatically looks for the favicon.ico in the websites root directory (if a <link rel="icon"> isn't present). If no icon is found, a server will respond with a 404 page. In the case of the largest 404 page a 6.7MB response from the server is sent to the client. This is on top of any data actually needed by the browser to render the page! All the time this is happening the user has no idea. It's not like they see a broken image. The page is just incredibly slow to load. And to top it all off this happens for every page on the site (as the browser is still looking for that elusive favicon.ico it wants).

You can see this happening via a WebPageTest waterfall on a cable connection using Firefox. And yes, you are reading that correct, 17.3 seconds to load the 404 response, amounting to 44 seconds for the full page load!

The impact of missing favicons on web performance has been written about before, and under certain conditions can cause you hours of debugging. So do yourself and your users a favour, as a bare minimum stick the most basic favicon.ico at your server root.

Wordpress requests

The "/wp-content/uploads/" URLs are understandable in many cases. A content author uploads an image, maybe modifies the post or the image is renamed / deleted from the server, thus creating the 404 error.

It's the "/wp-content/themes/" and "/wp-content/plugins/" URLs that really worry me. WordPress has an excellent upgrade mechanism for both plugins and themes: simply click a button and it does it all for you. Great! But what you are essentially doing is then trusting that the developers of every plugin and theme you use on your site aren't now injecting a broken resource link into your pages! If they do, you aren't likely to notice it unless you check the browser console, it breaks functionality in some way, or you happen to be monitoring your server logs for 404 responses. But when that broken resource is injected into every page, and a 404 status code and page is sent, that's when it could effect web performance. And as you can see from both sets of data this isn't a rare occurrence, it is happening a lot for WordPress sites!

In the above example we see a page trying to load a broken WordPress plugin called WaveSurfer-WP, only the JavaScript file is missing causing the browser to download an additional 288KB!

JavaScript

Both datasets show a high percentage of broken JavaScript requests that are generating 404 errors. But it's actually worse than that. At least if a whole JavaScript file fails to load you are likely to notice it. But what about if you are using JavaScript to generate URLs for a particular component on your site. It then downloads assets from your server in a very Ajax fashion, but the JavaScript doing this fails in some way. This isn't hypothetical, we can actually see it happening in the HTTP Archive data in these URL patterns:

"/undefined"
"/null"
"/Array"
"/NaN"
"[object%20Object]"

These are all URLs being generated using JavaScript which are failing for some reason. And in doing so they are making a request to the server for a resource that doesn't exist. At which point the server responds with a 404 status and the corresponding page page. So unless you have a rigorous set of tests to catch these edge cases in your JavaScript, these 404's could be occurring and you'd have very little knowledge of them happening. The user certainly isn't likely to check their browser console, so unless it breaks the site completely, they may not even know it is happening.

In the above example we see some broken JavaScript requesting a resource from '/undefined', and the browser receives a 294KB response back.

Standard files

Outside of the usual favicon.ico file, a whole range of files appear in many of the 404's logged, and they are all manually inserted by developers:

"favicon.png"
"ajax-loader.gif"
"manifest.json"
"site.webmanifest"
"manifest.webmanifest"
"sw.js"
"serviceworker.js"

Proving mistakes do happen. Even though a developer has added the <link rel="manifest" href="/manifest.webmanifest"> to the <head>, the actual manifest file has been forgotten about. Or maybe it was added but then deleted in a future release by mistake. Who knows. Again in both instances causing a 404 response on all pages it sits on (maybe all of them!).

In the above example the favicon.png file doesn't exist on the server, so it returns a 285KB 404 page. But it also sends the page with a 200 response code, with a 404 response body. This is known as a 'soft 404', and can be the result of incorrectly configured server software.

Presentational assets

I was actually quite surprised at how many CSS and font files (WOFF2, WOFF, TTF) were causing 404 errors in the datasets. Of all the assets that are likely to display presentational issues (and therefore likely be noticeable to both developers and users alike), these would be it. But it could be that these CSS and font files are only being used for particular pages or site components, so maybe hidden away in depths of the site. Or if you have a good CSS font stack configured, the use of a web font may not always be noticeable (which begs the question why load it in the first place). In any event, these filetypes appear a huge number of times across the full dataset, so it's quite a common occurrence.

In the example below a missing CSS file triggers a 270KB 404 page response:

Ultimately with each of the five examples of potential causes of 404's listed above, there's a chance that an unoptimised 404 response will cost the user dearly, both in terms of web performance and also device resource and data usage.

What about testing?

So here's the thing about synthetic monitoring and automated performance testing of 404 pages: it's actually almost impossible to do with many services at the moment (if you know of any, please let me know). The reason being is that many synthetic performance testing tools are built upon either Google Lighthouse or WebPageTest. As an example, let's see what happens with these tools if we try to audit a 404 page:

Lighthouse

Running a Lighthouse audit on a 404 page causes the tool to abort the process with no data being captured about the page. This is fully understandable really, it is an error page after all! But it's a shame there isn't an option to override this implementation detail and force the collection of data if a 404 status code is encountered.

WebPageTest

This is the default setup for WebPageTest, and the waterfall that is generated if you point it at a 404 page. Now there is a way to force WebPageTest to capture data from a page when it encounters a 404 status code. It's a bit of a hack but it does work.

Browser DevTools

Your best bet for examining the asset sizes for your 404 pages at the moment is to resort to your browser developer tools. Take a look in the network tab and see what is being loaded, and approximately how long it is taking. You could even try throttling your network connection to observe just how long it takes for the 404 page to be loaded and rendered.

404 Caching

Using the browser developer tools it is possible examine and optimise the caching strategy used on your 404 page. There's two types of caching to consider when it comes to 404 pages: CDN cache and local browser cache.

CDN cache

The thing about hitting a 404 page is that there's a high possibility that this request is going to be sent straight to your origin server. But if you place a CDN in front of your origin, you have the chance to cache the 404 page response on the CDN. This is a good idea since without it, hitting a 404 page could be used as a DoS attack vector, by forcing the origin server to respond to every 404 request rather than letting the CDN respond with a cached version.

Check the documentation for your specific CDN to see how they handle caching of 404 pages by default, as each CDN has their own strategy for caching 4xx status codes received from an origin server.

Local browser cache

Should a browser cache the HTML page and assets when it encounters a 404 page? Caching and cache correctness related to 4xx responses is actually specifically mentioned in rfc 2616 - Hypertext Transfer Protocol -- HTTP/1.1:

A correct cache MUST respond to a request with the most up-to-date response held by the cache that is appropriate to the request which meets [one of] the following conditions: It is an appropriate 304 (Not Modified), 305 (Proxy Redirect), or error (4xx or 5xx) response message.

So there's nothing technically stopping you from setting Cache-Control headers for your 404 page and associated assets and requesting that a browser cache them.

Design considerations and caching

Now you may be asking: why on earth would I want to cache 404 page assets? Most users will likely visit a 404 page and then immediately hit the browser back button.

That will be true in many cases, but think about it this way: the assets have already been downloaded so why not request to cache them. Then if you can offer a user an easy way to get to the rest of your site (e.g. homepage link, search functionality, popular articles call to action), the 404 page could actually be used to warm the cache for the rest of the site visit. This initial failure could be converted into a web performance win, should a user decide to explore the rest of the site.

Users encountering your 404 page should be factored into your user journey planning, because at some point they are going to see it. So take the time to consider how users can recover from their mistake (or the mistake of another person e.g. mistyping a URL in a blog post).

Possible solutions

So there are a couple of possible solutions to help fix issues with 404 pages on your site:

Favicon and monitoring

First is make sure you have a favicon.ico file at the web root of your server. It doesn't even need to be of anything, it could simply be transparent. Just as long as it is present and the server can serve it if it is requested. Next step is setup monitoring so you can see where 404's are occurring on your site so they can be fixed. This can be done using a variety of methods, from Web log analysis to the Google Search Console.

Correct server setup

But you could take it further. Ideally you only want to be serving HTML to the browser when it expects a HTML response. And for all other responses serve a small error payload. How this is setup depends on the web server or CDN being used. Listed below are example configs for NGINX and Apache, where as a default we are setting the custom error page, then for specific locations we are serving the small 404 response (assuming the files don't exist).

# NGINX
location ~* \.(ico|jpg|jpeg|png|gif|svg|js|css|swf|eot|ttf|otf|woff|woff2)$ {
    error_page 404 /small_404_response.txt;
}
error_page  404 /custom_404.html;

# ApacheConf
<Location ~ "(\.ico|\.jpg|\.jpeg|\.png|\.gif|\.svg|\.js|\.css|\.swf|\.eot|\.ttf|\.otf|\.woff|\.woff2)$">
	ErrorDocument 404 "404 Not Found"
</Location>
ErrorDocument 404 /custom_404_page.html

You can complete the same task using various different methods in Apache and NGINX, the above is just one example.

You can actually see this in action by looking at how Facebook has their servers setup.

Here we aren't matching against a particular location (e.g. https://www.facebook.com/test.). Because of this a full HTML response is being served:

But if we request a location that matches a particular file extension (e.g. https://www.facebook.com/test.jpg) the 404 response is much smaller:

A 72 byte response saying 'Not Found' then a link back to Facebook.

Conclusion

Assuming that an optimised 404 page is only required because users will mistype a URL in their browser is short-sighted. As the HTTP Archive data has shown, there are many other reasons why a user may encounter a 404 response (even if they have no idea they actually are!). The web performance impact of a users browser loading an unoptimised 404 page can be huge, and it can have a real impact on their experience of your whole site. All it takes is a forgotten file or misplaced ; in some JavaScript, and your users could be encountering it.

Automated web performance monitoring of 404 pages is very difficult at the moment, as most tools don't capture data when encountering a 404 status code. So it's likely you will need to check your 404 page manually using a browsers developer tools. I'd love to see this change in the future and see tools implement some form of '404 mode' for testing 404 pages. It's really important that users aren't penalised for a mistake (either theirs or that of another person).

So why not check the state of your sites 404 page right now? Check your server logs so spot where 404's are happening across your site. Because I guarantee at some point a user is going to encounter one, even if they don't realise it.

June 2024 update

As of October 2023 this functionality was available in Lighthouse by using the ignoreStatusCode option. And as of the 5th June this is also available in PageSpeed Insights. See a sample 404 report here. For more information on this change see issue 10493 in the Google Chrome Lighthouse GitHub repository. Many thanks to Paul Irish and the whole of the Google Lighthouse team for adding this feature!

Post changelog:

25/08/20: Initial post published.
26/08/20: Minor adjustments / clarifications, thanks Barry Pollard. This is why you always get someone else to proofread before publishing!
27/08/20: Added information about 'soft 404s' and the 'What decides what is sent in a 404 response' section. Horizontal scrolling of wide tables is now possible across the whole site (Thanks to Ed Horsford for reporting)
28/08/20: Added a section about possible solutions related to 404 page monitoring, and serving a smaller response depending on the location the client requests. Thanks Barry Pollard and Paul Calvano for the feedback.
07/07/24: Added an update from Paul Irish about support for 404's in Lighthouse and PageSpeed Insights. Thanks Paul!

A 'Vary' unusual waterfall

2020-08-12T00:00:00Z

So as I've mentioned a few times before on this blog, we (GDS) use SpeedCurve for our synthetic web performance monitoring. We've been using it for a while, but in a very ad-hoc fashion. Recently I finally got round to setting up a whole set of monitoring on our services to let us know when things change (both positive and negative). So back in April an alert appeared for one of our services: GOV.UK Notify. Seemingly overnight without warning the font size in Chrome jumped from 157KB, to 331KB. That's a whopping 110% increase!

Checking in with the team, nothing had changed with the code that could cause this for quite some time. Delving into the details a little further we could see that the number of font requests had jumped from 5 before, to 8 after. Very unusual considering no code had changed to trigger it.

The waterfall

One of the great features about SpeedCurve is the ability to jump directly into the underlying test data, since it is all powered by WebPageTest, it's a tool I'm a little more familiar with. But the waterfall I was presented with was one I'd never seen before:

On first sight the browser is requesting both WOFF2 and WOFF fonts at the same time. Looking a little closer requests 6 and 7 are very strange. Both are requests by the browser for the WOFF2 fonts. On examination you can see the server attempting to send the browser (Chrome) the font files. This is indicated by the dark red bands (known as chunks). But the strange thing is the independent, unconnected bands. It looked as though the server sends some data, stops for 10-20ms, then tries again. Eventually the sending of data fails, the fonts are not received and the browser moves on.

The fallback WOFF fonts

The intriguing part for me is looking at where the 'fallback' WOFF fonts are requested by the browser:

Notice how just as the browser receives the first WOFF2 font data it immediately initiates the request for the equivalent WOFF fonts. My interpretation of this is that as soon as the browser sees the WOFF2 data, it knows there is an issue and that it can't use the data, so it immediately requests the fallback. This happens for both WOFF2 fonts on the page. Assuming my interpretation is correct, once the WOFF2 files have been requested they will be using up bandwidth, and if the browser can't use the data they will be wasting that bandwidth. By the time the browser cancels the font requests (HTTP/2 only), the fonts may have already arrived (as seen in the waterfall above).

Historic data

Thankfully we'd had SpeedCurve setup on a few Notify URLs for approximately 6 months, so we had a large amount of data to look back on. In examining historic data about the font size, an interesting pattern emerged:

This font size 'change' had happened at least 25 times in the past 6 months, then mysteriously reset itself. It's also possible to see in the data a few 'half jumps', where only one of the WOFF2 font files had the issue, and the other downloaded fine. I concluded from this that it wasn't anything the team were doing, this was something else much more random.

But what really triggered the investigation was that after the 21st April, the font size became permanently stuck (as can be seen to the right of the graph).

More investigation

Chatting to the team further they mentioned something quite intriguing. While developing the site they occasionally ran into seemingly random console errors. They'd pop-up for a single load, then immediately disappear after a refresh, meaning they had no chance to debug the issue or consistently reproduce it to allow them to fix it:

As you can see above, it's a Cross-Origin Resource Sharing (CORS) error related to the Access-Control-Allow-Origin header on both WOFF2 fonts. This gave us the first indication as to what the issue was.

So what changed?

Another great point about SpeedCurve, since it is all built upon WebPageTest, under all the useful graphs you can delve into an incredible amount of information about the page load process. Given that it's possible to look at the point before the issue happened, then the point after it happened, it's possible to compare test results and look for clues. Since it looked to be CORS related the obvious place to look was the response headers for the WOFF2 files. This is what a successful response looked like, and here's the failed response, both for the same file. Just a few initial observations when comparing the headers:

Both responses came from the same CDN point of presence (PoP): LHR3-C1.
content-length and the etag values are both exactly the same, so the binary file itself hasn't changed (byte-for-byte identical since a strong ETag validator)
The date, last-modified, and age headers have changed.
The failed response is missing all access-control-* headers.

The access-control-* headers are CORS headers. So CORS rears its ugly head again. The date, last-modified, and age headers indicate that although the file itself may not have changed (in terms of bytes), the surrounding metadata has. And given the last-modified value was now Tue, 21 Apr 2020 14:32:51 GMT, the date our issue appeared, I'd guess they are related!

Caching on a CDN

So GOV.UK Notify hosts all of their static assets on the Amazon CloudFront content delivery network (CDN). In order to minimise latency for users, CDN's will cache requests on their own servers. This takes the load off a website's origin server, and allows a CDN to serve assets to a user from a PoP that is geographically closest to them. This is because lower latency generally equals better web performance. But there's a popular saying in Computer Science:

There are only two hard things in Computer Science: cache invalidation and naming things. -- Phil Karlton

Caching is a difficult problem to solve. Getting something into cache is easy, but making sure that it is still an accurate representation of the original at all times is difficult! So what if that's what has happened here. The version stored in the CDN cache is in some way broken and it is being served to the client, and the client is rejecting it. This actually happens a lot, sometimes by accident, other times used maliciously as a security exploit. This is known as web cache poisoning. The theory being that by injecting compromised code into a CDN's cache, it could then be served up to all users. If you are interested in finding out a lot more about this topic I highly recommend reading 'Practical Web Cache Poisoning' by James Kettle.

Just to be clear, I don't think what happened is malicious in any way, shape, or form. It just highlights that this could be the issue, and gives it a name for further investigation. A quick Google for 'CloudFront no Access-Control headers' very quickly brings you to this incredibly detailed post on Serverfault all about it. Reading through that post quickly leads you to a possible solution: the vary header.

The `vary` header

So let's briefly chat about the vary header. CDN's use what is called a cache key to check to see if the request being made for a file and the file that already exists in the CDN cache is the file the browser is expecting. For that reason you want your cache key for a particular resource to be unique. That way you can avoid cache collision, where an outdated version of a file is served to a user. Default cache key settings differ depending on the CDN being used, but they can often be customised and configured by a developer. For example, the default cache key could be the complete Uniform Resource Identifier (URI) for a particular request. So say we have a file already cached on a CDN, https://nooshu.github.io/images/fake-image.jpg and we make a request for it using the following URI reference:

Request for https://nooshu.github.io/images/fake-image.jpg = match
Request for http://nooshu.github.io/images/fake-image.jpg = no match (HTTP, not HTTPS)
Request for /images/fake-image.jpg = no match (Missing scheme and authority)
Request for https://nooshu.github.io/images/fake-image.jpg?cachebuser=123 = no match (URI queries matter)

What the vary header allows you to do is add specific client request headers to also be 'keyed' by the cache. So for example, if you wanted to make sure a CDN is always serving up the correct version of a file depending on its compression type, you would specify vary: Accept-Encoding in the response. The CDN would then examine a future clients Accept-Encoding header to determine which file to serve from its cache. If the client requests uncompressed, it serves an uncompressed version from cache. If the client requests a gzipped version, it gets a gzipped version from cache. If a version being requested doesn't exist in the CDN cache, it will go back to the origin server for a copy (then save it for the next client who requests it).

`vary` origin

So what is the vary: origin response header doing? It is saying to the client that "when I serve a file to you, I'm taking into account the origin header from which the request is being made from." So for example, if two requests are being made for the same file but the origin header is different for each request, multiple versions of the same file will now exist in the CDN cache. Should another client come along and make the request from the same origin, they get the version of the file associated with that origin only.

Font's (for reasons we won't go into) need the access-control-allow-origin header to be set when loaded as part of a page. This can either be a wildcard, or the request origin reflected back. origin headers are only sent with CORS and POST requests, so the CORS request to the WOFF2 fonts that sit on the GOV.UK Notify assets domain already contains an origin header. So what would happen if we didn't set up the CDN to consider this origin header in the request when caching font files on the CDN? Well the CDN could be serving a version of the font files (including headers) to a client that is missing vital information it needs to use them e.g. missing CORS headers (access-control-*).

So what exactly happened with the CDN cache?

Well that's the million-dollar (pound) question isn't it. And unfortunately why exactly this was happening and what triggering it is still a complete mystery to me. But the theory I'm going with is this:

something triggered the CDN cache to be cleared, meaning the WOFF2 requests no longer existed in the cache
A request was made by someone for the WOFF2 fonts that primed the cache with a non-CORS version of the font response e.g. by someone directly navigating to the file (a navigation request is non-CORS)
Because the the Origin isn't being considered as a cache key (Vary: Origin), this non-CORS version is then served back to the Chrome client without the correct access-control-* headers
No CORS headers set on the font response is going to cause issues, leading to the WOFF2 font not being used by Chrome
The browser immediately requests the fallback WOFF fonts which are then used (assuming they are cached correctly)
SpeedCurve sees the additional font requests (even though the fonts weren't actually used) and it updates its graphs accordingly

Cache persistence

Another strange thing I noticed was that this issue was easily reproducible on the public instance of WebPageTest. All you needed to do was select the 'London, UK - EC2 - Chrome' instance. But when I selected any other instance it didn't show the issue. I found this interesting because SpeedCurve run their own WebPageTest instances on Amazon EC2. How is it that 2 separate WebPageTest instances on EC2 are displaying the same issue, but other instances are fine. On examining the response headers it is easy to see why:

SpeedCurve instance (failed response): x-amz-cf-pop: LHR3-C1
Public EC2 instance (failed response): x-amz-cf-pop: LHR3-C1
Public non-EC2 instance (successful request): x-amz-cf-pop: LHR61-C2

The two EC2 instances (both in London) were receiving responses from the same CloudFront edge node (LHR3-C1), whereas the other public non-EC2 instance was receiving the request from another edge node (LHR61-C2). And this is what was happening every single day when the test was run. The SpeedCurve instance was hitting the same edge node which had the poisoned cache, hence the reason why the graph was constantly showing the issue. Same Chrome browser, same EC2 instance every day with the same poisoned cache (with long-lived cache headers max-age=315360000, immutable) is going to show the same issue (unless the CDN cache is cleared).

Re-examining the waterfall

Now that we have a bit more of an idea about what is going on, lets re-examine the original waterfall to pinpoint what is happening:

First the WOFF2 font files are requested by the browser after a new TCP connection is established
A very thin slither of data is received (most likely the headers)
The browser immediately cancels the request as there are no required CORS headers. This explains the 'empty space' between the data.
The browser immediately requests the next fonts in the @font-face declaration (requests 8 & 9)
The browser still receives chunks of WOFF2 data because this data was already in transit over the network or sitting in one of many buffers along the way.

It never ceases to amaze me the amount of information you can get from a waterfall chart once you know what to look for!

Solution

Well after all the discussion above the actual solution is very simple (in theory). Add vary: Origin to the response for the web fonts and purge CloudFront cache on every node. Unfortunately it's not always that simple. Looking at the CloudFront documentation on how it handles vary headers, they have specific rules around what is automatically removed. But it is still possible. Once merged and deployed the issue was fixed:

Now admittedly this isn't the most exciting graph ever, but it does show that it immediately fixed the issue, bringing the number of font requests and the total font size down to the correct value after being stuck there for a number of months. And from all the discussion above, this should now have solved the issue for good. No more random CORS errors for Chrome users and potential wasted bytes on fonts that couldn't actually be used.

Learnings

So there are a couple of learnings here:

Make sure you setup monitoring and alerting for your website across a number of different browser configurations. Without SpeedCurve graphs to inspect and investigate, we would still be trying to reproduce the random CORS error in the browser.
When making a CORS request for fonts, make sure the vary: Origin header is set.

On point 2, it is worth mentioning that this issue is such a common occurrence it is specifically mentioned in the Fetch specification.

Conclusion

Well that was a lot of investigation work for such a small change, but it just goes to show how difficult it is to debug and fix unknown unknowns. Without fully understanding the detailed ins and outs of caching on a CDN, and how CORS requests for fonts are handled within Chrome, how would a developer ever know that simply adding the vary: Origin header will solve this random error message that occasionally pops up in the browser console.

Post changelog:

12/08/20: Initial post published. Thank you to the GOV.UK Notify team developers (Chris, Tom, and David) for their time investigating this issue with me and rolling out the fix. Big thank you to Gaël Métais for spotting the original issue, and Barry Pollard and Andy Davies for answering lots of my questions!
13/08/20: Fix a mistake I made around the ACAO header not accepting a wildcard for fonts. Thanks to Eric Lawrence for flagging!

Measuring and examining TLS 1.3, IPv4, and IPv6 performance

2020-07-30T00:00:00Z

Earlier in the year we enabled Transport Layer Security (TLS) 1.3 on the Fastly point of presence (POPs) for GOV.UK. Fastly have been gradually rolling out a whole set of improvements to the cache nodes in their POPs. This includes a new h2o TLS architecture, which was required for enabling TLS 1.3. I believe with this upgrade it also paves the way for enabling HTTP/3 & QUIC in the future, which should improve performance on connections that suffer from high packet loss (e.g. unstable, rural connections). A win-win all round! So assuming you are using a modern browser (and aren't stuck behind a proxy that forces a TLS downgrade), you should see something like this when you next visit GOV.UK:

TLS 1.3 is the latest version of the Transport Layer Security cryptographic protocol, and it offers a number of improvements over previous versions, including:

improved security by removing insecure or less secure ciphers (as well as insecure features)
improved performance due to a reduction in the number of round trips (RTT) to establish a secure connection

Improved security and web performance, sign me up! So when TLS 1.3 was enabled on the UK POPs for GOV.UK on 15th April, I was a little disappointed to not see anything reflected in our synthetic web performance tests in SpeedCurve, as I'd expected to see a change in the Time to First Byte (TTFB) metric. I thought it may be related to TLS session resumption, but WebPageTest gives details about this in a pages HTML 'Raw Details' tab, and they are all "tls_resumed": "False". But these results are only from synthetic testing on a limited set of devices and connections, so that's probably why. I'd guess if we were capturing Real User Monitoring (RUM) data over the same time period, we would have most likely have spotted it. So after reading Chendo's blog post TLS 1.3 performance compared to TLS 1.2, it got me thinking: how could I replicate these tests for GOV.UK and verify (purely for my own sake) that TLS 1.3 does in fact offer a web performance improvement for our users. And does IPv4 vs IPv6 have any effect on the results (IPv6 is enabled for GOV.UK). The rest of this blog post goes into how I did this, what was involved and what were the results.

Ways of measuring connection time

The first step in this process is being able to collect data about the connection timings. There are a few ways of measuring the time taken for TCP connections to be established. Here are a few I looked into:

Time Command

The time command is what Chendo used in the blog post I referred to earlier. The command isn't specific to timing network activity. In fact it can be used to time any command used on the command line. It's incredibly simple to use, just prepend time to the command you wish to time:

time npm -version
# output: 0.16s user 0.04s system 109% cpu 0.186 total

The above is the output from Zsh: 0.16s CPU time was spent in user mode. 0.04s spent in kernel mode. The task got 109% of the CPU time, and took a total of 0.186 seconds to run. There's a whole heap of customisations possible which can be seen on the commands man-page.

cURL

curl is used in command lines or scripts to transfer data. It's open source and pretty much used everywhere, from cars to mobile phones. The number of options available on its man-page is huge. If you need to transfer data with URLs, then cURL is the tool for you. Thankfully we will only need a very small number of settings: those related to connection timing. Joseph Scott wrote an excellent blog post back in 2011 about how to extract detailed connection timing information using cURL, which can then be used to find things like the TTFB or TLS negotiation time. Joseph has even created a very simple shortcut called curlt that outputs the timing information and response headers for a particular connection.

HTTPStat

httpstat is essentially a visualisation plugin built upon curl. It takes the curl connection timing statistics and visualises them in an easy to digest format. Very simple to install and also outputs all the response headers too, so a very handy tool for basic debugging.

One of the advantages of using cURL on its own is that you have fine-grained control of the output from the command. So for example, if you want to write a large number of results to a file as JSON or a CSV, then this is entirely possible. Given this option, this is the route I decided to take when measuring TLS 1.3 performance.

Testing performance

So there were a few specific points I wanted to cover in testing:

what's the impact of the distance from a user to the server when it comes to latency and TLS negotiation time
semi-automate a way to collect a 'large' amount of data related to connection metrics
could I spot any difference in connection metrics when using IPv6 or IPv4

The last point is only really because my ISP has IPv6 available (and is enabled), as does GOV.UK. So I was just curious if anything could be seen in the resulting data when using IPv6 vs IPv4. Does IPv6 offer web performance advantages? There are a number of posts out there that compare performance. Some seem to suggest there is, others not.

Collecting results

As mentioned before I'm using cURL to capture detailed timing information. I captured a set of data which could then be plotted on a graph for each of the connection scenarios. So lets go through how this is done step by step.

Capture a single request / response

Using the method Joseph mentions in his blog post, it is very simple to capture detailed timing information from a URL. Once the curl-format.txt file has been created (opens in new tab), run the following command:

curl -w "@curl-format.txt" -o /dev/null -s https://www.gov.uk

The output gives the response headers, as well as the detailed timing information that looks like this:

{
	"time_namelookup":      0.045055,
	"time_connect":         0.058824,
	"time_appconnect":      0.107569,
	"time_pretransfer":     0.109468,
	"time_redirect":        0.000000,
	"time_starttransfer":   0.127577,
	"time_total":           0.130754
}

What do each of these values mean? They are detailed on the cURL man-page, but I've also pulled together a quick visualisation to help you understand it too:

It's important to remember that many of these metrics are relative to the start of the connection, not the end of the previous activity. This is solved with a little simple data manipulation in the spreadsheets (see results for more info on this).

Capture multiple request/responses over time

Now that we can capture a single response from a URL, what about if we wanted to capture timing information over a period of time? This is where a simple bit of shell script comes in handy. Note that the curl_format has been moved into the shell script for ease.

#!/bin/bash
# create `timing-data.sh` and make it executable (chmod +x timing-data.sh)

curl_format='{
	"time_dns":      	%{time_namelookup},
	"time_connect":         %{time_connect},
	"time_appconnect":      %{time_appconnect},
	"time_pretransfer":     %{time_pretransfer},
	"time_ttfb":   		%{time_starttransfer},
	"time_total":   		%{time_total}
}'

# used as headers for the CSV file
echo '["DNS","Connect","SSL","Wait","TTFB","Total Time"]'

total=521

for ((n=0;n<total;n++))
do
	curl -w "$curl_format" -k --compressed -s -o /dev/null "$@"
	sleep 0.3 #space the timings out slightly
done

The above script is used like so:

timing-data.sh https://www.gov.uk --tls-max 1.2

The script goes off and captures a large number of timing results (in this case 521). I've omitted time_redirect from the above script as I know this will always be 0 in my case, but you could leave it in if you are using on a URL with redirects. The above test took approximately 10 minutes to run. Here I'm capping the TLS version to 1.2 (--tls-max 1.2), as this allows us to compare different versions. Without this cap, cURL would use the highest version supported by the server, which is 1.3 for www.gov.uk. So the script above works, but it isn't capturing the results to a file, or converting it to a format that can be easily imported into a spreadsheet and plotted. It comes out as one huge string of timing info in a JSON object. Time to bring in jq.

jq is a Swiss army knife for manipulating JSON data on the command line. You can easily transform JSON data into any structure you need:

timing-data.sh https://www.gov.uk --tls-max 1.2 | jq -r '[.[]] | @csv'

Here I'm taking the JSON output from the cURL response and piping it into jq. I'm then constructing an array made up of each element in the input (the timing values). Then lastly I'm piping the array into the @csv formatter to output the values as a CSV. Once converted to a CSV, the data can be easily imported into a spreadsheet (e.g. Google Sheets) for further manipulation and graphing.

Lastly, lets make sure we write all this manipulated data to a file so we can actually use it. A final bit of shell script can do this for us:

# final script
timing-data.sh https://www.gov.uk --tls-max 1.2 | jq -r '[.[]] | @csv' > output-test-results.csv

Here we redirect the output to a file. The file is created automatically if it doesn't exist.

Capture multiple locations

The script is now capturing a large amount of connection timing data from a URL, but we have no control over the server cURL is looking at to get this data from. As mentioned above, I'd like to see how a servers geographic location (CDN POP) effects TLS and connection negotiation speed. Thankfully cURL comes with a useful --resolve option that allows us to do this:

# final script
timing-data.sh https://www.gov.uk --resolve www.gov.uk:443:[another_edge_node_ip] --tls-max 1.2 | jq -r '[.[]] | @csv' > output-test-results.csv

Here we are asking cURL to resolve any secure requests for www.gov.uk to a different IP address. So instead of allowing the CDN to choose the closest geographic location for our request (most likely London for me), we can tell it to request it from elsewhere. For example Sydney, or Vancouver. Why those locations? Well they are 10,585 and 4,663 miles away respectively. Therefore the latency from these locations will be much higher e.g. higher RTT. So it will be much easier to see the difference between TLS 1.2 and 1.3 (and indeed IPv4 and IPv6 if any can be seen) due to the greater distance to travel.

Of course this is all very contrived, as the very purpose of using a CDN is to minimise the latency between a user and the server, thus giving better web performance. To do this CDN's use technologies like Anycast, Unicast, Broadcast, and Multicast. The details of which I won't go into here (because I don't fully understand them myself!) But this --resolve method is a great way to test the TLS negotiation improvements in 1.3.

So now we have a method of capturing a large amount of connection timing data from an origin, and we can store it in a format that is easily imported into a graphing tool. For graphing I'm using Google Sheets, as it has lots of options available and is easily sharable.

Testing

In terms of testing I chose three of Fastly's POPs. UK (London), Canada (Vancouver - YVR), Australia (Sydney - SYD). London to my home is obviously very close (short distance), my home to Vancouver is 4,600 miles (medium distance), and my home to Sydney is 10,500 miles (long distance). For each one I would request the GOV.UK homepage, but force the use of a particular POP. I also forced the use of either IPv4 or IPv6 with the --resolve option. Both the IPv4 and the IPv6 tests were run concurrently for a particular TLS version, so as to keep the network conditions consistent. Each cURL command was run 521 times, with a 0.3 second delay between each run (so as to elongate the testing period). Each test took approximately 10 - 15 minutes to complete, with no other significant activity happening over my home network during the runs (so as to minimise last-mile interference).

NOTE: For those wanting to know how IPv4 or IPv6 is chosen in practical terms, there's a whole RFC called 'Happy Eyeballs Version 2' dedicated to it. TL;DR: A DNS server provides an A record (IPv4) and an AAAA record (IPv6) for the particular URL you requesting. These are sent at almost the same time. Of the two DNS queries, the one that completes first is chosen to be the protocol that is used for the connection. But it's not quite that simple though, as IPv6 is given two advantages:

the IPv6 AAAA record is sent first before the A record
if the IPv4 A record is received by the client first, there will be a recommended wait of 50ms to allow the AAAA record to complete

This is essentially trying to migrate internet traffic towards IPv6 (assuming it is available on the clients connection and at the server).

Results

Before I get into the results let me provide you with links so you can interpret them for yourself should you wish to:

Both the data and larger versions of the graphs can be found in other sheets found on those links. I've also compiled the timing difference between TLS versions and protocol versions at the different percentiles, both for the TLS negotiation and the total time (right of the graphs).

TLS 1.2 vs TLS 1.3

This is an easy one to call. TLS 1.3 beats 1.2 in all scenarios, by a fairly substantial set of percentages too. Usually between 20-50% improvement at the median TLS negotiation time for both IPv4 and IPv6. It is very easy to see this by looking at the graphs. The purple 'SSL handshake' is always a lot thinner with TLS 1.3, meaning it took less time to negotiate the secure connection. This in turn leads onto a significant total time reduction in nearly all results. Usually a reduction between 10-30%. Not a bad saving considering all you need to do is "enable it"!

Now this isn't ground breaking news, as it is completely expected. This fact is touted as one of the advantages of enabling TLS 1.3. But I'm very glad to be able to verify that this is the case for GOV.UK. So users with a browser that supports TLS 1.3 should see a significant improvement in TTFB under many circumstances.

But wait, there's more! TLS 1.3 also supports something called 0-RTT, or Zero Round Trip Time Resumption. This basically allows a client that has visited a website previously, to pick up where they left off. No need for a new TLS negotiation, one was established before, so use that. When in use I'd expect the purple in the graphs above to disappear. That means all the time needed to establish a secure connection and start sending data will be removed. That's a pretty impressive web performance boost (when it eventually becomes better supported).

IPv4 vs IPv6

TLS 1.3 is quicker than 1.2, that is plain to see. But IPv4 vs IPv6 isn't as clear cut. As mentioned, both IPv4 and IPv6 tests were run at the same time so as the traffic at the time was consistent.

Comparing TLS 1.2 over IPv4 and IPv6 (negotiation speed, total time):

UK to UK, IPv6 was marginally slower (5%, 3% at median)
UK to Vancouver, IPv6 was marginally slower (5%, 5% at median)
UK to Sydney, IPv6 was marginally quicker (7%, 4% at median)

Comparing TLS 1.3 over IPv4 and IPv6 (negotiation speed, total time):

UK to UK, IPv6 was significantly quicker (20%, 4% at median)
UK to Vancouver, IPv6 was marginally slower (4%, 5% at median)
UK to Sydney, IPv6 was marginally quicker (4%, 2% at median)

So what's going on here? Well, in all honesty the sample size of.. just me, isn't going to answer this question. It's important to understand that with IPv4 and IPv6, you essentially have two independent connections to the internet. So you have two different paths from client to server. Although the tests were run at the same time, the path that each was running over are separate. Meaning that each path:

may have different bandwidth available
may have more congestion than the other
may have more packet dropouts than the other

These are just a few considerations. Avery Pennarun explains it very well in his excellent 'IPv4, IPv6, and a sudden change in attitude' blog post. So it was an interesting experiment (for me), but I wouldn't rely on the accuracy of the IPv4 vs IPv6 results. They were accurate for my connection at the point in time I measured them, but in no way are they conclusive. It's also worth remembering that IPv6 adoption is still considerably low considering it became a IETF Draft Standard back in December 1998. There are also huge swathes of the world that simply haven't adopted it yet. I guess now that globally we have run out of IPv4 addresses, there should be some pressure to move adoption forwards.

Conclusion

So in this blog post I've covered a range of topics, from how to collect data using curl and converting this into a format that can be plotted (CSV), all the way to IPv4 vs IPv6. Results show that enabling TLS 1.3 is a good idea. It offers more security and better performance for your users. It's also worth noting that TLS 1.3 will be a requirement to use the QUIC transport layer network protocol in the future. This will pave the way to HTTP/3. And once 0-RTT becomes more prevalent, for repeat website visits the purple on the graphs displayed above will disappear completely. Even faster connections for all (at least for those that use a browser that supports it anyway).

Update: Using Google Analytics server connection time

After publishing this article I got chatting to Rockey Nebhwani who suggested looking at Google Analytics (GA) data to see if I could spot the change. I always tend to forget about the performance timing data that GA captures. It has some limitations (sample size being unfixable and its use of averages), but if you have no other data sources then it's better than nothing! Turns out it's very easy to spot where the change occurred when looking over 5 months of data:

Admittedly it's not dropped a huge amount, only 10ms or so. But considering this is the average time of heavily sampled data (14,302,686 page views sent sample data over 5 months), it shows that users are seeing an improvement. For those interested in how you find this data in GA: Behavior > Site Speed > Overview > Avg. Server Connection Time.

Post changelog:

30/07/20: Initial post published. Thanks go out to Joseph Scott, Robin Marx, Chendo, Lucas Pardue, Andy Davies, Barry Pollard, and Radu Micu for pointing me to a tonne of resources and answering lots of questions for this post.
30/07/20: Updated TLS 1.3 advantages to mention insecure ciphers have been removed in 1.3, rather than new more secure ones added (Thanks Jamie H).
05/08/20: Added graph of GA data (Avg. Server Connection Time) that shows the when TLSv1.3 was enabled (Thanks Rockey Nebhwani).

Bringing HTTP/2 to GOV.UK - Resources

2020-07-14T00:00:00Z

So at the start of February 2020 we permanently enabled HTTP/2 on GOV.UK (that feels like a long time ago now!). I spoke about this at London Web Performance, read a whole lot about HTTP/2 (and surrounding topics), and produced some work of my own in the process. Without all this, it would have been very difficult to justify the permanent change, as it wasn't as simple as just "turning it on".

So in an effort to help others in the same position, and to fully document the process, I've pulled together a post with as much information as I can remember about the work. It was inspired by Oliver Byford and his excellent 'Inclusive forms: Anatomy of a (fictional) GOV.UK service' blog post.

London Web Performance

My LdnWebPerf presentation and associated slides:

Slides: Bringing HTTP/2 to GOV.UK - Matt Hobbs, March 2020

Internal research

Initial Research: Bringing HTTP/2 to GOV.UK - Matt Hobbs, October 2018
Initial Data: Performance testing HTTP/2 data - Matt Hobbs, October 2018
Data comparison Sheet: HTTP/2 results before / after - Matt Hobbs, October 2018
Initial Analysis: Was switching to HTTP/2 a good thing? - Matt Hobbs, October 2018
Initial research: summary sheets - Matt Hobbs, October 2018

GOV.UK Fixed!

Bringing HTTP/2 to GOV.UK - First technical draft - Matt Hobbs, February 2020
GOV.UK RFC-115: Enabling HTTP/2 on GOV.UK - Matt Hobbs, January 2020
Blog: Speeding up GOV.UK with HTTP/2 - Matt Hobbs, July 2020
Case study: How GDS improved GOV.UK's frontend performance with HTTP/2 - Matt Hobbs, July 2020

Subresource Integrity (SRI)

Blog post: HTTP/2 and Subresource Integrity don't always get on - Matt Hobbs, December 2019
GOV.UK RFC-114: Changing SRI to allow for HTTP/2 to be enabled on GOV.UK - Matt Hobbs, December 2019
Specification: Subresource Integrity - W3C Recommendation, June 2016

HTTP/2 prioritisation

HTTP/2 connection coalescing

Blog post: HTTP/2 connection coalescing - Daniel Stenberg, August 2016
Docs: Connection management in HTTP/1.x: Domain sharding - MDN web docs

Docs: Cross-Origin Resource Sharing (CORS) - MDN web docs
Video: Demystifying CORS: It's Simpler Than You Think - Bill Parrott, November 2017
Tool: Will it CORS?

Learning materials

Video: Revisiting HTTP/2 - LDNWebPerf - Hooman Beheshti, May 2018
Blog post: Improving Perceived Performance With the Link Rel=preconnect HTTP Header - Andy Davies, March 2019
Book: HTTP/2 in Action - Barry Pollard, March 2019
Blog post: HTTP/2 is easy, just turn it on... - Neil Craig, November 2017
Blog: HTTP-in-depth - Journey to HTTP/2 - Kamran Ahmed, August 2016
Web Almanac: Part IV Chapter 20 - HTTP/2 - Barry Pollard, November 2019

Post changelog:

13/07/20: Initial post published.

Example synthetic web performance monitoring dashboards using SpeedCurve

2020-06-17T00:00:00Z

Government Digital Service (GDS) have been rolling out synthetic web performance monitoring across all GOV.UK applications (all 13 of them), and the other services run by by GDS. Using SpeedCurve we can now set a whole bunch of performance budgets and alerts that let us know via email and Slack, when a particular threshold has been crossed in the metrics we are monitoring. In rolling this out to all teams, I found myself creating dashboards with a consistent set of graphs. The reason for this method was to:

try and extract as much useful information from the synthetic tests as possible
'standardise' the graphs across teams so I could document and talk about what each meant easily
compare and contrast performance between each of the applications / services

During this process I created a document for all teams to read with an explanation of the graphs and metrics involved. Instead of keeping it locked away in a Google Document, I've decided to adapt it into a blog post. By doing so I'm hoping it will:

help others understand the power of the dashboard and graphs you can create using SpeedCurve
help others by giving example dashboards and graphs they can copy and adapt
validate my thoughts and understanding of the graphs and metrics with other people in the web performance community
gather feedback from others to find out if there's anything I'm missing

Browser & devices we are testing on

Synthetic web performance testing gives you fine grain control of the conditions for which the tests are run. Certain browsers and settings have been selected to test against, primarily based off our Google Analytics (GA) data.

Browsers I've currently selected to test with are:

Chrome Desktop (Cable connection)
Internet Explorer 11 (Cable connection)
Firefox (Cable connection)
Samsung Galaxy S4 (3G connection)
iPhone 5 (2G connection)

Chrome

Chrome is the 2nd most popular browser used on GOV.UK (after Safari on iOS). It makes up ~30% of our users browser usage. Users tend to use both the desktop and mobile version. It is an obvious choice for inclusion in testing.

Internet Explorer 11

This is the last Internet Explorer (IE) version we officially support according to the Service Manual. Looking at the last 3 months GA data from GOV.UK, I can see it makes up 99.2% of IE usage. Again, an obvious choice for inclusion in testing.

Firefox

Firefox is currently only used by 1.48% of GOV.UK users over the past 3 months (me included). This does make it a candidate for not being included. I'm personally keeping it in as it is the only non-WebKit based browser on our list, and it is always good to diversify testing, as browser engines differ.

Samsung Galaxy S4

Simulated specifications for this mobile are:

Connection: Mobile 3G
Viewport: 360x640
CPU: 3 x slowdown

Many of the tests that I chose to run are using this simulated Samsung S4 mobile phone. The results returned are set at the 95th percentile (p95). The reasons for this are:

The S4 is a low specification 'legacy' device, with limited CPU and RAM.
It is a device that is currently used by many users of GOV.UK (~45,000 users in the past 3 months, or 0.04%)
A 3G connection is fairly standard and expected in the UK for many users.
The 95th percentile gives essentially the worst case scenario (i.e. 95% of users are going to have a better experience than this). By setting the bar so low, any performance improvements we can make at this level will likely improve performance for every device setup.

2G Mobile (effective iPhone 5/SE)

This is a custom browser profile that I have created in the SpeedCurve settings. Specifications for this simulated mobile are:

Browser: Chrome
Dimensions: 320x568
Pixel ratio: 2
Connection: Mobile 2G
CPU: 4 x slowdown
User Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B137 Safari/601.1

This phone is currently used by ~87,000 GOV.UK users over the past 3 months (or 0.08%).

Example dashboard 1 - A GOV.UK application

Now that we understand the test devices, let's look at the whole 'standard' dashboard used for each GOV.UK application:

Each application runs a particular set of pages on GOV.UK, so it's good to get a general overview of how well the application is delivering pages to a user. So let's delve into what each graph is telling us.

Web Vitals - iPhone 5 - 2G - 95th Percentile

Web Vitals is a marketing name for a set of metrics that are soon to become very important in the Search Engine Optimisation (SEO) world. Google published a blog post recently stating that in 2021 its search ranking algorithm will start to use the scores from these metrics to influence a page's rank. The metrics they have chosen to use are user-centric and ensure that a page must:

Load quickly (Largest Contentful Paint)
Be interactive within a reasonable time (JS Total Blocking Time for Synthetic testing, First Input delay for RUM)
minimise jarring interstitials that affect visual stability (Cumulative Layout Shift)

These metrics have been chosen so as to help start ranking pages based on their overall usability for a user. Pages that score poorly in these areas will be penalised.

Note: An iPhone 5 on a 2G connection has been used in this instance, as when using the Samsung Galaxy S4 device, it doesn't seem to support Cumulative Layout Shift. So this selection gives an idea of the result of these metrics on a low-specification device, on a very slow connection.

JS Total Blocking Time (TBT)

According to the Core Web Vitals, First Input Delay (FID) is the metric that should be used for indicating if a page is interactive for a user. Unfortunately, this metric is only available via Real User Monitoring (RUM) testing, and not available in synthetic testing. An alternative synthetic metric is available to measure page interactivity. It is called Total Blocking Time.

Total Blocking Time measures some of the activity on the browsers main JavaScript thread. Too much activity on the main thread can cause a page to not respond to user input, causing confusion and frustration for users. Any script that takes over 50ms to execute is known as a 'long task'.

So the 'total' in TBT is a little misleading. It in fact doesn't measure the total as you would expect (initial navigation to fully interactive): In reality TBT:

only tracks the long tasks between First Contentful Paint (FCP) and Time to Interactive (TTI)
only tracks time over the initial 50ms (the threshold for a 'long task')

TBT is the sum of the scripts after these two criteria has been applied. So TBT is great for understanding your Lighthouse score, but there are other ways to track scripts with long tasks on your page. There's a whole post here from Mark Zeman which talks all about it if you want to know more.

Cumulative Layout Shift (CLS)

Cumulative Layout Shift measures the visual stability of a page. The visual stability of a page is affected by late displaying elements, causing the whole page to 'shift'. This is especially problematic on mobile devices, and sometimes causes users to click on buttons and adverts they didn't intend to.

A brief explanation as to how it works: when rendered elements move during page load, they are labeled as 'unstable'. A score is calculated by taking the distance that the unstable elements have moved in the viewport (this is the distance fraction), and multiplying it by the surface area of the viewport that's affected by unstable elements moving (this is the impact fraction).

The cumulative layout shift score is the sum of the above calculations for all unstable elements on a page that move around during the page load. Diagram credit goes to the Calibre app team.

Largest Contentful Paint (LCP)

This metric measures the loading performance of a page. It does this by reporting on the render time of the largest content element visible within the viewport. A LCP of 2.5 seconds or less is considered good, 2.5 - 4 seconds needs improvement, 4+ seconds equates to a poor user experience.

Lighthouse Scores

Google Lighthouse is a tool built into every version of Chrome DevTools. It is a set of best practice audits that can be run against any website via the DevTools 'Audit' tab. The results give you a score out of 100 for the following areas:

Performance
Accessibility
Best Practice
SEO (omitted from our tests)
Progressive Web App (omitted from our tests)

We aren't tracking SEO and PWA scores since they are a lower priority for our services.

Lighthouse Performance Score

The lighthouse performance score gives you an idea of how well your service is performing. Scoring is based on a range of metrics that are weighted according to priority. The focus of this audit is on user-centric metrics, so it optimises for performance a user will actually care about and respond too. Performance scoring details can be found here for both v5 and v6 of Lighthouse.

Lighthouse Accessibility

The accessibility score gives developers a way to identify accessibility issues on a particular page. These issues are often 'low-hanging fruit', that may be relatively simple to fix. A full list of the accessibility checks the tool runs can be found here.

Note: It's important to realise that an accessibility score of 100 doesn't mean a service is 100% accessible. It just means it has passed those specific tests. A service should still be professionally audited and tested with users.

Lighthouse Best Practice

The best practice score looks for issues related to infrastructure and coding best practices in the page source. It is marked on a number of separate criteria. A full list of the criteria it examines and the relative weights applied can be found here.

Core Metrics - Galaxy S4 - 3G - 95th Percentile

This graph is used to track a ‘core' set of web performance metrics that are considered to be very important in how a user experiences a website. Not all the metrics are 'user-centric', but they do give you a good sense of how well a page is performing and where bottlenecks are likely to be located.

Backend (TTFB) Time

Backend time (also known as time to first byte), is the time it takes for a request packet to leave the browser, for the server to process the request and a response to come back to the client. DNS lookup, connection, and TLS negotiation time are also included in this time. In many cases this should be one of your more stable metrics. Things like database lookups, routing, and template compilation can have an effect on this metric. There's a whole blog post here by Harry Roberts if you'd like to find out more about it.

Start Render

Start Render is the time between the initial page navigation and when the first pixel content is rendered to the display. This metric is very good for understanding the point at which a user sees anything on screen. It is calculated by analising of the page loading, so is only available in synthetic testing.

Fully Loaded Time

Fully loaded is defined as the time from the start of page navigation, up to the point where no network activity has occurred for a total of 2 seconds after document complete (onload event fired). This metric is useful for understanding the amount of data being downloaded after the onload event has fired (e.g. via a Single Page App, or third-party JavaScript on a page).

Visually Complete

The visually complete metric focuses on the viewport that the page is being rendered on. It is calculated by analysing a video of the page load, then comparing each frame to the pages final state at the end of the test. This essentially plots page 'completeness' over time. It will give you an indication of how quickly pixels are being shown to the user, and at what point the viewport reaches 100% rendered. Note: there may be other areas of the page still loading outside of the viewport (‘below the fold').

Speed Index

Speed index takes the visual progress of the visible page load (in the viewport), and calculates an overall score to help understand how quickly content was painted to the viewport. This score is calculated using the same method as mentioned above in Visually complete. The 'completeness' of the page is plotted over time by doing a frame-by-frame analysis of each frame against the final frame. By plotting the completeness over time and calculating the (bounded) area above the graph, it gives information as to what portion of the page is left to be rendered at any point in time. A lower Speed Index score is better.

Asset sizes

The last three graphs are fairly self-explanatory. They give an indication of how the size of each type of static asset on the page has changed over time. It is a good idea to monitor these metrics across releases to see what difference new changes have made to the download size. The aim is to minimise the size of all these assets to improve frontend performance.

Example dashboard 2 - An important GOV.UK page

Here we've setup a slightly different dashboard to get more information back from the test device to see if there are any CPU bottlenecks on the page (and how much CPU time is being used):

Web Vitals - iPhone 5 - 2G - 95th Percentile

Same as listed above.

Milestone Timings - Samsung S4 - 3G - 95th Percentile

The graph shows important milestone metrics in the lifecycle of a page load. The aim is to reduce these metrics as much as possible to help improve user experience. It's worth noting that many of these aren't considered to be user-centric metrics, but they are worth monitoring to give you a more complete picture of a pages performance.

Backend (TTFB) time

See above under core metrics.

DOM Content Loaded time (DCL)

DOM Content Loaded Time is the point at which the DOMContentLoaded event fires. This is indicates when the Document Object Model (DOM) has been loaded and parsed. The event does not wait for other assets like stylesheets and images to load before firing. This metric is often used by JavaScript libraries to identify when they can start manipulating the DOM.

Start Render

See above under core metrics.

Time To Interactive (TTI)

Time to interactive is a complex metric, as it doesn't measure what most people think it measures which is the time it takes for a page to become interactive. This is very hard to actually measure via synthetic performance testing. What this metric should be called is Time to consistently interactive, as the page will likely be semi-interactive before this point. The metric indicates the time at which a site is visually rendered and consistently interactive for a user. There's a whole blog post here all about it for those who want to know much more.

First Contentful Paint (FCP)

First Contentful Paint measures how long it takes for the browser to render the first piece of DOM content to the page after user navigation. DOM content is defined as either:

Text and images
Non-white <canvas> elements
SVG's
Note: anything inside an <iframe> isn't included

FCP is a user-centric metric. It answers a user's question "Is it happening?". 0-2 seconds is considered good, 2-4 seconds is moderate, over 4 seconds in considered to be slow.

CPU Metrics - Galaxy S4 - 3G - 95th Percentile

The graph shows important CPU metrics related to the browsers page loading process. Any increase in these metrics means that the browser is having to work harder to paint pixels to the screen. High CPU usage can result in pages that are slow to respond to users, and have a negative impact on a devices battery life. The aim is to minimise all of these metrics.

CPU JS Scripting To Start Render

This is the amount of CPU time the JavaScript used to get to the start render point. Start render time is defined as the time between initial page navigation, to the first pixel being displayed on screen.

CPU JS Scripting To Page Load

This is the amount of CPU time the JavaScript used to get to page load. Page load is when the onload event fires, and is the point where the page and all its dependent assets have finished loading.

CPU JS Scripting

This is the total CPU time used when parsing and executing the JavaScript on the page.

CPU Layout

This is the CPU time taken to layout the page. Layout computes the exact position and size of each object on a page.

CPU Loading

This is the time taken to parse the HTML and the CSS, and create the render tree (DOM + CSSOM).

CPU Painting

This is the CPU time taken to paint pixels to the page. The paint process takes the final render tree and renders the pixels to the screen.

First CPU Idle

This is how long it takes a page to become minimally interactive. It is minimally interactive when most (but not necessarily all) UI elements are interactive, and the page responds to the user in a reasonable amount of time.

Lighthouse Scores

See above under Example dashboard 1.

Core Metrics - Galaxy S4 - 3G - 95th Percentile

See above under Example dashboard 1.

Image Size

So we are already tracking the asset sizes (CSS, JS, fonts) at an application level as seen in the Example dashboard 1 above. But because we are now focussing on an important individual page, we are also tracking the size of images on this page. This is a great idea if these images are uploaded through a content management system, as a lot of the time they may not be fully optimised.

Conclusion

Hopefully I've given you an idea of what graphs you could add to a dashboard, and what each of the corresponding metrics mean. Synthetic tests give you so much information about a page, so it's important to pick out the metrics that will help you identify and rectify web performance issues. As mentioned at the start of the post, if you have any feedback on the graphs I've chosen, or if I'm missing anything please let me know via Twitter. I'd love to see the graphs & dashboards you are creating in your testing tools to maximise the amount of useful data you are extracting from synthetic tests.

Post changelog:

21/06/20: Initial post published.
22/06/20: Updated Total Blocking Time explanation as it was incorrect. Thanks Mark Zeman.

WebPageTest comparison URL generator

2020-04-22T00:00:00Z

As any reader of this blog will have noticed, in recent years I've taken a great interest in web performance both for my current job role and also as a hobby. And anyone interested in webperf will quickly realise that one of the best tools you can use to benchmark the performance of your website is WebPageTest. It's a tool that will give you a huge amount of information about how your website is performing, including useful data and charts that you can use to identify areas for improvement. Simply enter your URL, choose from a few settings and click 'Start Test'. I've written a lot more information on interpreting the results from these tests in my blog posts 'How to read a WebPageTest Waterfall View chart' and 'How to read a WebPageTest Connection View chart' for those interested in knowing more. But one of the most powerful features of WebPageTest is its ability to compare the results from multiple tests.

Measuring regressions and improvements

How do you know if you are improving a websites performance (or making it worse) if you can't easily compare results before and after you have made a change? This is where WebPageTest comes into its own. First: make sure you are logged in to your account (so your tests are saved), then run a number of tests. Next navigate to the 'Test history' page via the main menu. If you've run a number of tests you should see something similar to below.

On this page the area we are interested in is to the left of the 'Date/Time' column (highlighted in red below). This section can be easy to miss since the form controls are quite small, but this is how you compare test results.

Simply select all the tests you would like to compare (you can select more than 2 if you wish), then click the 'Compare' button. WebPageTest will go away and the load a whole new view with a number of graphs and tools to help you gauge the performance difference between the tests that you have been selected (see below).

In the image above you can see a visual representation of how each of the pages loaded under the test conditions you chose. I won't go into the details here, but if you are interested in finding out more information about this view, I've written a little in this section from one of my previous blog posts.

How does this work

So under the hood, all the WebPageTest compare functionality is doing is creating a really long URL with a whole bunch of query parameters included. For example, above we selected 3 tests to compare, so the resulting URL contains three 42 character long test ID's within the tests parameter. This parameter driven functionality comes with two great advantages:

Results for the specific comparison view can be bookmarked and very easily shared with anyone
By manipulating the URL, you can change how the tests are rendered and compared on screen

Now point 2 sounds very useful, but it also comes with a caveat: WebPageTest URL's can be huge, making editing them manually quite a chore! When you consider each test ID is 42 characters long, and you include all of the advanced settings you can choose from, editing a URL in the browsers address bar probably isn't a feasible option. Also, mistakes are easy to make. Misplace the comma between test ID's or forget to URL encode your custom label names and you've got a broken page view on your hands.

WebPageTest comparison URL generator

The scenario I mention above I've personally struggled with too many times to count! So I recently decided to make my life a little easier and create a tool to do this URL manipulation for me. I've also published it online so others can use it if they so wish. It's available at https://wpt-compare.app/.

It really is quite a simple little application. All it is doing is building the long URL for you. But it comes with a few features that you may find useful:

Ability to add as many test URLs as you need
Drag / drop functionality to manipulate the order in which the tests appear
Quickly select which median metric is used in the comparison
Accepts multiple URL formats to extract the WebPageTest test ID's
Compatible with SpeedCurve test URLs and custom instances

I've also included all the 'Advanced customization options' as seen in the modal that pops up after clicking 'Advanced customization options...' under the 'Create Video' button in the comparison view (see image above). These advanced settings include:

Label: customise the label used to identify a specific test
Specific run: Select a specific test run to use in the comparison view
Repeat view: Compare the repeat view only, not the first view
Specific step: If you are using scripting to navigate through a page journey, each page is considered a 'step'. So choose a specific step to compare. (Thanks Andy)
Specific end time: Stop the comparison view at a specific number of seconds (e.g. 2.3 seconds)
Background and text colour: I'm not sure if anyone uses this feature to customise the text and background colour of the page, but it's there if needed

SpeedCurve URLs

As mentioned in the feature list above, SpeedCurve URLs are accepted by the tool too. You may (or may not) know that SpeedCurve runs WebPageTest instances under the hood to generate their synthetic test results that power the graphs. So if you happen to find a graph where there has been a large change between tests, you can either drill down into the results within SpeedCurve itself (click 'View Test' - see below), or you can look at them directly on the SpeedCurve WebPageTest instance. It's worth noting that the traditional 'Test history' functionality (mentioned earlier) is disabled on the SpeedCurve WebPageTest instances, probably due to the load on the servers it would create to list thousands of automated tests.

Now as we use SpeedCurve at GDS to monitor our services frontend performance, I find myself repeating this specific task a lot! So I added some helper functionality into the tool to allow me to quickly create the SpeedCurve specific comparison URLs. If you are a SpeedCurve user, why not give it a try.

Graph page data URL's

As of the 13th April 2021 I added the functionality to allow you to generate URL's for the 'Graph Page'. With this view you can generate a set of graphs and compare different sets of test runs under specific conditions. I've written a whole post about it that you can see here.

Feedback and feature requests

So there you have it: a simple tool to help you generate those huge WebPageTest comparison URLs for all your blogging and screenshotting needs! Hopefully you find it useful. If you do have any feedback, bugs, feature requests etc, please do reach out to me on Twitter.

Post changelog:

22/04/20: Initial post published.
13/04/21: Added information about using the tool for generating the graph page data view.

Improving perceived performance with the CSS `font-display` property

2020-02-23T00:00:00Z

Typography on the web has come a long way since the days of Scalable Inman Flash Replacement (sIFR) and later cufón. It was tough times for typographers and frontend developers on the web back then. I used to dread seeing a PSD file with some exotic font used in the design, as I knew hours of cross-browser adjustments lay ahead. Thankfully, on the modern web we have the @font-face CSS at-rule which allows us to specify a custom font when displaying our text. It hasn't always been plain sailing, but today in 2020 many of the issues have been ironed out and browser support for WOFF2 fonts is looking really good. So what's the catch? Web Performance.

As the proverb says: "With great power comes great responsibility". Give people the ability to use custom fonts easily on a website and surly that won't be abused? Unfortunately it does, and it has an impact on web performance, and ultimately the usability of a website.

Fonts and Web Performance

If you use a non-system font on a website, the browser is going to have to fetch that font file from the server. Not a problem you may think, but if a site uses multiple custom fonts and font weights, or if you are having to support multiple languages, sometimes the size of the font files can make up a large percentage of the total page download size.

So if the font is downloading, what does the browser do with the content? Well, typically what browsers used to do was render an invisible placeholder font, then replace it with the actual font once it had downloaded. This method is known as the Flash of Invisible Text (FOIT). That's not ideal, but at least a usr can see something right? But what happens if a user is on a slow connection and the font takes 10's of seconds to download? Or what if the font never loads at all? The user is left looking at a website that looks like this:

In the example above a user is having to wait 5 seconds before any text is rendered to the viewport. That's a long time to be waiting considering Google research found that 53% of mobile website visitors will leave a website if it doesn't load within three seconds. So what can we do to improve this perceived performance issue? font-display to the rescue!

Font display property

The font-display property is defined in the CSS Fonts Module Level 4 specification. What it does is allow a developer to choose what happens during the font rendering phase of a page load. Just to be 100% clear, it has no effect on the actual loading of the font itself over the network, just what happens to typography on the page during this loading phase. Font display has five possible values, let's discuss these briefly:

Block

This is the default for many browsers. This setting will immediately draw the invisible placeholder text (known as the block period), invisibility period persists indefinitely until the web font is loaded. The browser has an infinite period to swap the invisible font for the web font once it has loaded.

Fallback

According to the specification:

[Fallback] gives the font face an extremely small block period (100ms or less is recommended in most cases) and a short swap period (3s is recommended in most cases).

So, on page load invisible text could be shown for up to 100ms (known as the block period). Beyond 100ms, the next font in the CSS font stack is displayed (e.g. Arial, Helvetica Neue, Helvetica, sans-serif). The browser is then given 3 seconds to swap the fallback font for the web font. Beyond 3 seconds the fallback font will be used for the rest of the page life and the web font will never be displayed (even if it successfully downloads).

The 3 second cutoff is a superb setting if you know you have users on a slow connection. Imagine a page loading on a slow device. The fallback font is rendered, so a user starts to read the content. They may be 5 - 10 seconds into reading, then suddenly the web font (that they know nothing about) suddenly loads. If the font metrics of the web font and fallback font are different, this can results in a large page jump which is quite disruptive.

From my observations (as seen later in the article), the practical application of the specification is true to its word. Invisible placeholder text is displayed for 100ms, then the fallback font is painted.

Swap

The specification says:

[Swap] gives the font face an extremely small block period (100ms or less is recommended in most cases) and an infinite swap period.

Or to put it another way, on page load the browser could show the invisible text and then wait for up to 100ms before displaying the fallback font. Beyond this point the browser has as much time as needed to swap the fallback font for the web font once loaded.

But the practical implementation in browsers is different. For swap there is no block period. The fallback font is displayed as soon as the page is painted. So even though the specification makes fallback and swap sounds very similar on page load, they are actually different in terms of perceived performance.

Optional

The optional value is the one I find most interesting, yet it is the least used according to the 2019 Web Almanac. The specification is the longest of the five values, but in simple terms: it gives the browser the option to abort the font download, or load it at the lowest priority. This setting could be very useful if you are on a slow cellular network. Should the browser decide that the connection is too slow for the font to load, it will abort the request completely and simply display the fallback font for the rest of the page's lifetime.

I'm intrigued as to why this option isn't used more widely, given its obvious benefit to both the usability of a website and its improved data footprint for slow connections. But if I were to hazard a guess, I'd say it's because it is very strict in terms of repainting. If the fallback font is displayed, the web font will never be shown, even if loaded quickly. So this results in users on a fast device & connection where the fallback has been painted, the web font has loaded but it isn't rendered. Only if the user navigates through to another page, is the web font then rendered. So I'd imagine a number of designers being quite annoyed if their chosen web font isn't visible on initial page load!

Auto

A very simple one to understand. auto will use whatever setting the user-agent already has defined. For many browser this will be block.

Browser support

Browser support for font-display is actually looking great, with around 82% of users globally using a browser that supports it. A couple of the surprising outliers for me are:

Samsung Internet

Samsung Internet (SI) is a browser that has been gaining some market share on Android devices. It currently doesn't support font-display. I suspect this is because SI v10.x is based on Chrome 71 (which didn't support font-display). There are rumours on the web that v11.x will be based on Chrome 75, which does support font-display. So you never know, support may be just around the corner.

Edge (Chromium)

Final edit of this section. It turns out that the data in caniuse was incorrect. Edge does support font-display in both @font-face and via the API, as this test proves. I opened an issue and it was fixed in under 10 hours. The power of open source strikes again! If you check out the FontFace API: display now, you will see the correct values. I'm leaving this section here to remind readers you can't always trust your sources! Thanks to Zach Leatherman for helping clear up this mystery, and vinyldarkscratch, Fyrd for the fixes.

Testing the perceived performance

So how can we test the effect these different font-display settings have on a websites perceived page performance? One way would be to simply update your @font-face rule, deploy it, then run it though WebPageTest. That works but there is another way that may simplify this workflow.

And that's by leveraging WebPageTest's 'Inject Script' functionality (under the 'Advanced' tab), and using the CSS Font Loading API.

Here we use a very similar method to what Andy Davies demonstrates in his Improving Perceived Performance With the Link Rel=preconnect HTTP Header blog post.

You can find some simplified code below which can be used to do this, or on a Gist found here (opens in new tab):

(function(){
  // this will trigger a font load
  var customFont1 = new FontFace('custom font name', 'url([FONT_URL_HERE])', {
    display: 'block', // display setting to test here
    weight: '700' // font-weight
    // other font properties here
  });

  // IMPORTANT: add the font to the document
  document.fonts.add(customFont1);

  // monitor the font load (optional)
  customFont1.loaded.then((fontFace) => {
    // log some info in the console
    console.info('Font status:', customFont1.status); // optional
    // log some info on the WPT waterfall chart
    performance.mark('wpt.customFont1Loaded'); // optional
  }, (fontFace) => {
    // if there's an error, tell us about it
    console.error('Font status:', customFont1.status); // optional
  });

  // repeat above for multiple fonts

})();

Modify the above code and paste it into the 'Inject Script' textarea. The above code will be injected into the test page before the test executes and will load and define a new font (using the same font-family name) with your updated settings.

The key to get this solution to work is to ensure the manually added font is added after the CSS @font-face fonts are registered. As the priority of fonts with the same font-family name is: the last one added wins. This mirrors CSS, e.g. if you have two selectors with the same specificity, the one which comes last 'overwrites' the first.

Lets look at hypothetical test timeline:

A WebPageTest run starts, the browser negotiates the server connection and requests the HTML.
HTML is downloaded and parsed, other page assets like CSS and JS are requested.
At this point WPT injects the JavaScript and it executes. Our new font is defined and loaded using the Font Loading API.
The CSS is downloaded and parsed, it also contains an @font-face rule with the same basic info (font name, font URL etc).
The FontFace object and @font-face are CSS-connected. The parsing of the CSS adds the font to the bottom of the list (it is last), so it is prioritised over our manually added font.
Page loads as normal, no changes are observed from the injected script.

Point 3 is the part of this process that is unpredictable. There are no timing guarantees that the injected script will run after CSS download / parsing. It may be different depending on the browser and it runs as soon as the document exists. So if the above code doesn't work for you in your particular site setup, what else can you try?

Solutions

So there are two solutions that I have found to rectify this issue:

Block the font CSS

This one is less than ideal, but it works. If you happen to have your @font-face rules defined in their own separate CSS file, then you can simply use WebPageTest's 'block' functionality to block the font CSS:

What happens in this situation is our injection script creates all the relevant FontFace objects, and they don't get deprioritised by the @font-face rules in the CSS. There are a couple of major issues with this method:

What happens if you have lots of fonts? You need to replicate them all in the injection script.
What about if your font CSS is concatenated along with your other CSS? You essentially need to block your whole pages CSS.

So yes, this method could work, but it's far from perfect.

Detect the CSS load

Disclaimer: The following method does work, but web page load is complex and nondeterministic. So what works for one run (as seen in the tests below), suddenly will stop working in other tests using the exact same code. The CSS blocking method is much more predictable.

A better method I have found is to hook into the fact that you can detect when a CSS file has loaded with a little bit of JavaScript. It is possible to modify the injection script to do this. Here's a Gist (opens in new tab) with the modified code:

(function(){
  // create our custom link tag for the stylesheet
  var url = "https://www.example.com/static/app.css"; // IMPORTANT: this is the CSS file that contains your @font-face rules
  var head = document.getElementsByTagName('head')[0];
  var link = document.createElement('link');
  link.type = "text/css";
  link.rel = "stylesheet"
  link.href = url;

  // append the stylesheet to the head
  head.appendChild(link);

  // wait for the CSS file to load before modifying the font setup
  link.onload = function () {
    // define our font face and modify the properties (will trigger a load)
    var customFont1 = new FontFace('nta', 'url([FONT_URL_HERE])', {
      display: 'swap', // display setting to test here
      weight: '700' // font-weight
      // other font properties here
    });

    // IMPORTANT: add the modified font to the FontFaceSet
    document.fonts.add(customFont1);

    // monitor the font load (optional)
    customFont1.loaded.then((fontFace) => {
      // log some info in the console
      console.info('Font status:', customFont1.status); // optional
      // log some info on the WPT waterfall chart
      performance.mark('wpt.customFont1Loaded'); // optional
    }, (fontFace) => {
      // if there's an error, tell us about it
      console.error('Font status:', customFont1.status); // optional
    });

    // repeat above for multiple fonts
  }
})();

With this script we are hooking into the load event of the CSS that contains our @font-face rules (this is important). So here's what is happening in the browser:

A WebPageTest run starts, the browser negotiates the server connection and requests the HTML.
HTML is downloaded and parsed, other page assets like CSS and JS are requested.
At this point WPT injects the JavaScript and it may execute. If it does we create a copy of the <link> element that is loading our @font-face CSS and wait for it to load.
CSS loads and is parsed. Browser creates FontFace objects for the @font-face rules in the CSS (CSS-connected).
CSS load event fires and our custom FontFace objects are added to the FontFaceSet last, so is therefore prioritised over the CSS-connected font settings (e.g. the font-display property).
Page loads with our modified font settings.

And there you have it. A quick way to test out different font-display settings on a live website and observe the results in WebPageTest.

Debugging a WebPageTest run

This is a really useful tip from Andy Davies that I've slightly modified for these examples. If you want to see what the console output is for a browser in a test. Paste the following code into the 'Inject Script' textarea:

// hijack the console.log method, allowing you to log all console massages
window._consoleMessages = [];
console.log = function(m){_consoleMessages.push(m)};

// specifically used for this font example to 'copy' the `FontFace` object
function flatten(obj) {
    var result = {};
    for(var key in obj) {
        result[key] = obj[key];
    }
    return result;
}

Then the following into the 'Custom Metrics' box under the 'Custom':

[consoleMessages]
for(var font of document.fonts){
  // push results into the hijacked console
  // could easily return a simple array though if required
  console.log(flatten(font));
}

return JSON.stringify(_consoleMessages, null, 2);

This will give you an output that looks similar to this:

Very useful if you are looking for specific information about a browser in a test run.

Let's see it in action

I've created a super simple set of pages to demonstrate this technique. There's also a little information on what is printed to the console in browsers that support the API. If you are familiar with DevTools, simply open them up and you will see the information in the console. I've created 4 test pages:

Standard page with a single web font loaded via the the CSS @font-face rule. This is our control (font-display: auto) link.
Standard page + the same web font added a second time using the Font Loading API, with font-display: swap set link.
Standard page + the same web font added a second time using the Font Loading API, with font-display: fallback set link.
Standard page + the same web font added a second time using the Font Loading API, with font-display: optional set link.

Pages 2, 3 and 4 are a proof-of-concept. They each include the above script after the fonts.css file. So on these pages the font will be manually added to the FontFaceSet along with a new font-display property. Will it show any changes in WebPageTest? NOTE: The web font is only applied to the header (h1) on the pages, so that's what you should be concentrating on in the following tests.

You can take a closer look at the comparison of the 4 pages here, but here are the results as I see them:

Optional: The header rendered at the start with invisible placeholder text. ~100ms later fallback text is rendered. The web font never renders.
Fallback: The header rendered at the start with invisible placeholder text. ~100ms later fallback text is rendered. The font loads within the 3 second cut-off, so is swapped out.
Swap: The header immediately loads with the fallback font visible (e.g. the 0s block period in action). The fallback is swapped for the web font once loaded.
Auto: As expected, this is set to block. Invisible placeholder text is rendered, no text is shown until the web font is loaded.

All of the above results are the expected outcome for each font-display value, and the fact that we see a difference shows that a font added programmatically via the CSS Font Loading API, can change the page. So let's take it one step further. Let's get WebPageTest to inject the script for us.

In these tests we use the control page in all instances, then get WebPageTest to inject the script with each of the different font-display values. A full comparison of the results from these tests can be seen here.

It looks remarkably similar to the image above doesn't it? In fact, on closer inspection we see exactly the same results as listed above for our manually injected script pages. So we can actually use WebPageTest to change a pages font-display property (and really any property that isn't read only). I'm sure others may be able to come up with other uses for this technique too. Give it a try for yourself: check out this Gist (opens in new tab) with a link to the test page and the script for you to play with.

Conclusion

And there you have it, a way to change the font-display settings of a page when using WebPageTest. No need to manually update the code to see what effect the property has on a pages perceived performance. I'd love to hear your ideas and / or improvements that can be made to this method, so please let me know via Twitter.

Post changelog:

23/02/20: Initial post published.
25/02/20: Posted an update about font-display support on Edge (Chromium). Thanks Zach Leatherman!
26/02/20: Posted another update around font-display support in Edge. It doesn't support display in the Font Loading API.
27/02/20: Final update for font-display in Edge. It is fully supported in both @font-face and the API. Caniuse data was incorrect!
28/02/20: Added information about debugging a test run via the 'Inject Script' and 'Custom Metrics' functionality.

The impact of SSL certificate revocation on web performance

2020-01-26T00:00:00Z

Introduction

The process of establishing a connection to a website is a complicated process. First the browser must turn the domain name into an IP address (DNS lookup), then once found it must negotiate a connection with the server via Transmission Control Protocol (TCP). Lastly if the site is being served over HTTPS (which around 83% of web requests are), the browser needs to negotiate an encrypted connection (via Transport Layer Security (TLS)) with the server. The last TLS stage is where digital certificates are involved, and this process is what the rest of this blog post will focus on.

If you'd like more information on the transport stage of the connection process, check out the High Performance Browser Networking chapters here and here.

Types of certificate validation levels

There are three validation levels that are used with digital certificates, each with their own criteria to be met:

Domain Validation

To obtain a Domain Validation (DV) certificate you must prove you have control over one of the following criteria related to a domain: the whois records, DNS records file, email or web hosting account.

Organisation Validation

An Organisation Validation (OV) class certificate will be issued if you can meet two criteria related to a domain: one of the rights to administer a domain (as listed above with DV certificates), and an organisations existence as a legal entity.

Extended Validation

The Extended Validation (EV) class of verification is the most comprehensive of the levels. To obtain one you must prove you have control over the right to administer a domain (as with DV and OV certificates), an organisations existence as a legal entity (as with OV certificates), and there will also be verification checks by a verifying agent from the CA.

An important point to note about all three certificates above, in terms of technology they are exactly the same. They are all X.509 public key certificates. The only difference is what is contained within the certificates. As you might expect, EV certificates will contain more information and have different properties set to distinguish them from DV/OV certificates. For example, EV certs have a different subject which identifies a legal entity rather than a domain.

Establishing trust

So how does your browser ensure that the site being visited is who they say they are?

Certificate Authority

A Certificate Authority (CA) is the trusted organisation that will issue a certificate once you (or your company) have met the criteria listed above. They are the trusted third party between the browser (user) and the server (website). When communicating with a server we want to make sure that they are who they say they are. And this is achieved by completing one of the validation processes mentioned earlier.

The Chain of Trust

There is something called a Chain of Trust which is relied upon when establishing the encrypted tunnel. And it works a little like this. The CA holds what is known as the root certificate and key, which is used to sign (verify) the intermediate certificate. The intermediate certificate and its key is then used to sign the sites (leaf) certificate.

The browser trusts the CA (root) which has signed the intermediate certificate. So the browser will also trust the intermediate certificate. The same process happens with the site certificate. It has been signed by the intermediate (via the root) so the browser now trusts the site certificate. With a chain of trust linking site certificate to the root, the browser can now trust the site.

If at any point in the process a site certificate has been modified in any way, it wouldn't validate against the layer directly above it (intermediate then root). The browser would be able to see that the Chain of Trust has been broken and will report back to the user that the encrypted tunnel has been compromised (an unsecured network).

A SSL error can also happen if the webserver isn't correctly setup, and doesn't provide correct intermediate certificates in the chain to lead back to the root. In a report from 2016, missing intermediates account for >10% of all certificate validation errors in Chrome. Thankfully there is an extension in SSL certificates called Authority Information Access (AIA) that provides information about the issuer, and will include a link to the intermediate certificates if they are missing. Your users therefore shouldn't see an ugly NET::ERR_CERT_AUTHORITY_INVALID error message if AIA is used. NOTE: If the browser does AIA, it will slow down the SSL negotiation as the browser needs to fetch the intermediate certificate from the CA before the SSL negotiation can complete. So remember, check that your server is serving your intermediate certificates correctly.

For a little more clarification, there are a few more details on each of the certificate types below:

Root certificate - usually physically distributed with the operating system and is used to issue intermediate certificates.
Intermediate certificate(s) - CA's use the intermediate certificates to insulate themselves from having to use the root certificate to sign leaf certificates. A compromised root certificate is bad, so issuing off an intermediate certificate is much safer. It's next to impossible to revoke a root and replace it, and it would take years for the new one to filter through the ecosystem.
Leaf certificate - This is the sites certificate which is issued from the intermediate certificate.

NOTE: It is possible to have multiple intermediate certificates, although not recommended as it adds extra steps to the checking process.

TLS Handshake

When using HTTPS (as is increasingly common for websites and is a recommended best practice), the connection must be setup and secured cryptographically. The method of encryption and which keys to use are negotiated during the TLS handshake, which must happen before a HTTPS connection can be established. Before any application data can be sent over the connection, the HTTPS connection must be established.

It's important to remember that a TLS handshake doesn't come for free. Time and resources are used to establish the secure connection. It can typically take 2-RTT to establish this encrypted tunnel (depending on the server setup), but if the certificate size is large it could be more. If the server supports TLSv1.3 then it is possible to establish the connection in 1-RTT, or even 0-RTT in some circumstances.

For example, let's assume our hypothetical server takes 2-RTT to establish this encrypted tunnel, and our user is on a mobile with a 3G connection. A 3G connection will likely have a minimum RTT of around 300 ms. So it will take a minimum of 600 ms to establish the encrypted tunnel, and that's not even taking into account the DNS lookup, connection negotiation and any network congestion. This is why Transmission Control Protocol (TCP) connections are even more expensive in the new world of HTTPS. Thankfully QUIC, a new internet transport protocol which uses User Datagram Protocol (UDP) has been created which should hopefully solve this issue.

Broken trust

There are times where the trust between browser and server can be broken. It could be that a certificate has been compromised, and when this happens there needs to be a way to let the browser know about it. If the certificate has been compromised, you don't want a person with malicious intentions to be able to mimic a trusted website. That would be incredibly damaging to users, the website, and the Certificate Authority involved. To allow this to happen, ways to revoke certificates have been invented:

Certificate Revocation List

A Certificate Revocation List (CRL) is exactly what the name suggests. It is a large list containing the serial numbers of revoked certificates. Each and every CA updates this list regularly, and the list is shared with browsers. Now as you can imagine, the web has a lot of certificates, and therefore there's going to be a lot of revoked certificates. So this list will continue to grow and grow. Thankfully expired certificates aren't revoked as there's no need too, and their serials are often trimmed from the CRL's.

To use the list, a browser must download it in its entirety, and loop through each serial number to check and see if the specific certificate they are examining has been revoked. This takes both time and resources, which can slow down the TLS handshake. And what happens if the browser can't update the CRL? Does it just assume that the certificate it has been sent hasn't been revoked, then proceed as normal hoping there isn't an issue with it? (this is known as a 'soft-fail'). Given that the vast majority of certificates are NOT revoked, a 'soft-fail' is really the only viable option. Unfortunately a 'soft-fail' strategy has a critical security flaw, as described here. How a 'soft-fail' is handled depends entirely on the browser being used.

Online Certificate Status Protocol

So if CRL doesn't work particularly well, what else can we use? This is the reason the Online Certificate Status Protocol (OCSP) was created. Essentially, it allows a browser to send information about the certificate it is verifying to an OCSP responder, and the responder will send a message back saying either: Good, Revoked or Unknown. The payload is small and looping through a huge list of revoked certificate serials isn't required. Sounds better doesn't it?

So with the creation of OCSP has the problem of certificate revocation been solved? I'm afraid not. OCSP also comes with a number of issues:

Checking requires a TCP connection to the OCSP responder - This network overhead takes time to establish and adds latency to the SSL negotiation.
OCSP slows down the TLS handshake - The SSL negotiation cannot fully complete until a response is received from the OCSP responder. (there is a fallback which I mentioned earlier: 'soft-fail')
An OCSP responder can sometimes fail - What happens if the browser doesn't get a response? It can either blindly trust the certificate is still valid ('soft-fail'), or terminate the connection ('hard-fail').
Privacy concerns - By contacting the responder, the browser is leaking what website a user is accessing to the CA. Sounds like a potential method to track a person across the web that they don't know about or opt-out of.

Proprietary systems

Due to the issues with both CRL and OCSP, browser vendors and Operating System (OS) manufacturers have come up with their own proprietary systems for handling certificate revocations:

Chrome (Chromium) uses CRLSets
Firefox uses OneCRL
Safari is handled at the OS level
Edge(Chromium) uses CRLSets
Edge(EdgeHTML) handles it at the OS level
Internet Explorer handles it at the OS level
Mobile browsers... could be OS level, I'm not 100% sure.

This is my current understanding from lots of searching. If anyone has any information on how mobile browser revocation works I'd love to know. I've struggled to find any definitive answers.

How browsers handle OCSP

Next, let's examine what browsers do with OCSP when the different types of certificates are encountered, as this is an important consideration when it comes to web performance. For this investigation I've used a mixture of WebPageTest for the visualisation of the waterfall, tcpdump and Wireshark for the packet level analysis, and plain old Google detective work.

The spreadsheet I've compiled can be found here. There's some missing information where I either couldn't test, or couldn't find any solid information about what happens. If you spot any errors or can fill in the blanks please do let me know.

I have to admit, finding this information isn't easy. Browsers progress so fast, documentation can quickly become out of date, and analysing the network for OCSP requests isn't as simple as I first thought it would be. But assuming that the spreadsheet is mostly correct, you will see that a number of browsers actually use OCSP to check the revocation status of SSL certificates.

Web Performance and OCSP

So as OCSP checks are made in quite a number of browsers (and more specifically always made with EV certificates in Chrome), what's the issue with OCSP and web performance? Let's go into a few details:

TCP connections

A OCSP revocation check to a responder requires opening one or more TCP connections. Now admittedly a couple of TCP connections (over HTTP, not HTTPS) to another domain isn't a big deal (especially considering the number of JavaScript requests on sites nowadays). But it's the position they occur at in the waterfall that is important. They are right at the start of the waterfall. If there are network connectivity issues with the responder, or you are on a flakey mobile connection, this TCP connection is going to damage the performance. And considering no actual HTTP data can be transferred until the connection is established, your user could end up staring at a completely blank page.

In the example waterfall below you can see two TCP connections to two separate domains. Request 1 is for the intermediate certificates OCSP check, and request 2 is the OCSP check for the leaf certificate.

Don't be fooled into thinking you only require a single TCP connection to an OCSP responders domain either, as that isn't always the case. In the example below (taken from the WebPageTest connection view), the browser opens 3 connections to the responder:

And I'm not quite sure what is happening with this connection view below. We have 1 successful attempt to the responder at request 1, and what looks to be 5 failed requests. Then later in the waterfall we see 5 successful connections to the same responder. It's quite an unusual waterfall chart and I'd like to think there's something unusual with either the server setup or test itself, and this is a rare occurrence.

Roundtrip Times

When it comes to web performance and loading websites quickly, you may think it is the bandwidth of the connection that is the most important factor. But that isn't the case. It's actually the roundtrip time (RTT). The longer it takes a packet to travel to the server and the response coming back again, the slower the whole experience will be for a user. Let's see what happens if the RTT is large:

Here's an example setup to demonstrate:

A WebPageTest run on Apple.com (which uses an EV certificate)
Google Chrome used as the browser so an OCSP check occurs
3G connection with 300ms of latency added to the last mile
Our user is located in Vietnam

Now you may be asking "Why Vietnam?". Well, it's important to remember that the location of your users has a huge effect on the RTT. If your user is located in a country outside of America, but the OCSP responder is only located in America, that's a fair bit of latency being added to the RTT. So what does the start of a waterfall look like in this scenario:

Notice how the OCSP request isn't made right away because it needs to get the certificate from the negotiation to know what to check! Before this point the DNS lookup and initial TCP connection need to complete. Both of these stages are also impacted heavily by a large RTT. Once the SSL negotiation starts it takes 1.5 seconds on its own(!), and 67% (1022 ms) of this time is spent waiting for the OCSP response.

It's hard to know if 1.5 seconds is a long time for a SSL negotiation if we have nothing to compare it with. Thankfully we can use Mozilla Telemetry data to see how long a typical SSL negotiation takes across a large number of requests (from real browser data):

Firefox beta 72 (all connections types)
740 billion measurements
Median time for SSL completion: 177 ms
95th Percentile: 1090 ms

Although the browsers are different in the comparison (Chrome vs Firefox), the numbers give you an idea of what the usual SSL negotiation time is for many requests. The median, and even the 95th percentile are nowhere near the 1.5 seconds we see above. What's also interesting is we know from the browser spreadsheet I compiled earlier in the post, Firefox will OCSP check all certificates types. So even accounting for that in the large data set, the results listed aren't anywhere close to the 1.5 seconds we see in the demonstration.

OCSP is synchronous

Looking closely at the waterfall where and OCSP request is made, you can see that these responses are synchronous and blocking.

As can be seen in the waterfall above, the OCSP check at request number two must wait for request number 1 to complete. And the whole SSL negotiation has to wait for both checks to complete before it can fully negotiate the encrypted tunnel. Having synchronous and blocking requests at the start of a waterfall doesn't sound too great for web performance!

Responder reliability

As you can probably guess, the reliability of the OCSP responder in this whole revocation process is very important. After all, if the response is slow (or it times out) this is going to have a impact on the connection negotiation, the Time to First Byte (TTFB), and pretty much the whole waterfall after it. Again Mozilla Telemetry can give us an indication of what Desktop Firefox 71 beta users are experiencing. These results suggest that over 7% of OCSP checks are timing out today.

And this is what a waterfall looks like if the OCSP check(s) take a long time:

This waterfall is from a WebPageTest user who'd recently added EV certificates to their website. They'd then noticed that this change had caused a delay of up to 22 seconds with agents located in China, Russia, and India. They also note it occurs on a number or websites that require OCSP checks, not just their own. You can read the whole forum post here if you are interested. Upon reading it, it was one of the reasons why I decided to write this very blog post.

There's a very interesting section related to this in the EV SSL Certificate Guidelines 1.7.1. Section 13 states that:

The requirements in Section 4.9 of the Baseline Requirements apply equally to EV Certificates. However, CAs MUST ensure that CRLs for an EV Certificate chain can be downloaded in no more than three (3) seconds over an analog telephone line under normal network conditions.

Strangely, there's next to no mention of OCSP in these guidelines. So I wonder what the minimum response requirements would be for OCSP responders?

So a question you may be asking is what happens if the browser fails to receive an OCSP response? Maybe the connection times out, or the responder is offline, or the request is being blocked (corporate firewall etc)? Since the whole point of the OCSP check is to see if a certificate is still valid and hasn't been revoked, you may assume that the SSL negotiation will fail and the connection won't complete. That isn't the case though. This is where a browser will do what is called a 'soft-fail', which involves the browser trying to check the certificates revocation status and failing to get a response. Instead of killing the connection the browser proceeds with the connection regardless. In doing so the browser is assuming that the certificate hasn't been compromised. Sounds a little risky doesn't it? There's a whole post by Adam Langley from 2014 all about this particular issue that's well worth reading if you want to know more.

Third party domains can require OCSP checks too

It isn't just the origin domain where these checks can happen. Any third party domain you load from may require the browser to perform an OCSP check on their domain too:

In the waterfall above we can see that the browser (Firefox) is making a number of OCSP requests for multiple domains, not just the origin. So it's always a good idea to load these scripts in an asynchronous fashion, that way they aren't blocking. You could even use progressive enhancement to allow whatever content to be seen before the JavaScript has loading. Once loaded, this JavaScript then enhances the user experience.

EV certificates

Lets's focus on EV certificates, as I feel there are some unique things to consider when it comes to web performance.

EV Certificates and OCSP

I'd like to bring your attention to the results from Chrome in the browser spreadsheet, as you will see EV certs are always OCSP checked when using it, but other certificate types aren't. Now considering Chrome is the most popular browser on the planet (with around 56% usage according to W3Counter), that's a lot of users having to do an OCSP check on sites that use EV certificates. And that's not even including all the other browsers that require it too. Browsers like:

Firefox
Edge (Chromium & EdgeHTML)
Firefox (Android)
Samsung Internet
IE7-11

We already discussed the performance impact of OCSP revocation checking. So the fact that by choosing a certain type of certificate will force the most popular browser on the planet to encounter these performance issues, we should examine why you'd choose to use one.

Why use an EV certificate?

So what are the advantages of choosing to use an EV certificate over say, a DV certificate?

One of the driving forces behind their adoption could probably be marketing related. Browsers used to show the name of the website right next to a little green lock in the URL bar:

My guess would be that because you have your site name next to the little green lock, the site looks 'more legitimate', 'more trusted' and 'more secure'... maybe? But it isn't that simple. The name next to the lock may not match the branding of the website, due websites being owned by parent/subsidiary companies or other legal entities. And if that were the case then it just looks strange. To a user it may even look like the site is broken. They have no idea who the parent company is of the site they are visiting (or probably even care). So why does the little green lock reference a company they've never heard of? A user reaction could be: "Is this website trying to steal my bank details?".

The green lock and name in the certificate have been visible in the past, but what about in 2020: Do modern browsers still show the name next to the green lock for EV certificates?

Chrome: Has removed the name from the URL bar from version 77, but it's shown when clicked on. The padlock is no longer green.
Safari: Removed the name from the URL bar in June 2018, but it's shown when clicked on. The padlock is still green.
Firefox: Moved the name from the URL bar in version 70, but it's shown when clicked on. The padlock is no longer green.

There's a whole thread from Scott Helme on Twitter that talks all about these changes. In my opinion the marketing angle of EV certificates was weak before, but now it's non existent. Scott also has a blog post here discussing many other aspects of the perceived 'positives' of EV certificates, which makes for an interesting read.

How popular are EV Certificates?

So how many sites actually use EV certificates? Mozilla Telemetry data gives us some insightful stats about the number of EV certificates that are encountered in their data set (based on real browser usage):

Firefox desktop, beta 72 (all connection types)
19.2 billion measurements
434M (2.24%) invalid certificates
18.7B (96.51%) DV certificates
242M (1.25%) EV certificates

As you can see, EV certificates are pretty rare on the web. This is exactly what Scott Helme reported back in 2017 some 2.5 years ago. So I'd guess that number may be even lower now? A look at some more of Scott's stats from February 2019 sees the same downwards trend of EV certificate usage. And according to SSL Labs SSL Pulse, EV certs are only used on 8.3% of the top 150,000 websites from Alexa's list of the most popular sites in the world.

If you are interested a whole host of WebPageTest research on EV certificates and OCSP stapling, I highly recommend reading Aaron Peters superb blog post 'EV Certificates Make The Web Slow and Unreliable'.

How to improve performance?

So OCSP doesn't sound great for performance, but what can we do about it? Thankfully there is something called 'OCSP stapling' that can alleviate some of the performance pain (but not all of it!).

OCSP stapling

The problem with OCSP requests is every browser of every user must make these requests (assuming the browser does the checks). That sounds like a very wasteful strategy. So what if we got the websites server to periodically check on the OCSP status of its own certificate (which is cryptographically signed so that its validity can be verified). It could then send the response from this check during the TLS handshake to the browser, at the same time as the certificate. In doing so the users browser gets the revocation information it needs during the TLS handshake, so there's no need for it to make a separate request to the OCSP responder.

What I've just described is exactly what OCSP stapling does. You enable it on the server and it will do what I described above. Resulting in better privacy for the user, better performance, and less load on the OCSP responder. Excellent stuff! So what's the catch?

Unfortunately OCSP stapling doesn't solve all the problems. When a server staples a response it will only do it to one 'level' deep. So you can staple the leaf certificate check (the one for the site), but you can't staple the full chain. This is a known issue and there's an RFC6961 from June 2013: 'Multiple Certificate Status Request Extension' to add this ability to TLS, but this has now been obsoleted by RFC8446: 'The Transport Layer Security (TLS) Protocol Version 1.3', so maybe this will be possible with TLS 1.3? I'm interested to find out if this is possible in 1.3, so if you know please get in touch.

So what does OCSP stapling look like on a WebPageTest waterfall? NOTE: The site the tests were run on uses an EV certificate.

Stapling off

Two OCSP checks can be seen below. One for the intermediate certificate at request 1 (which contacts the root OCSP responder), the other check is for the sites certificate at request 2 (which contacts the intermediates OCSP responder):

Stapling on

With stapling enabled, one of the checks is removed. The OCSP check made to the intermediate responder for the site certificate has now been 'stapled', and is sent to the browser along with the site certificate during the TLS handshake:

OSCP stapling and EV certificates in Chrome

So what does OCSP stapling mean for EV certificates? Unfortunately it doesn't completely solve the performance issue. Even with OSCP stapling enabled, Chrome is going to run an OSCP check on the intermediate certificate. So you're going to have a OCSP performance hit with EV certificates no matter what:

As they say, the proof is in the WebPageTest waterfall chart. Above I have run a test on Apple.com, it has an EV certificate and the server has OCSP stapling enabled (SSLLabs test). The test is run using Chrome and the OCSP check can still be seen at request number 1. OCSP stapling has stapled the leaf certificate check and sent it along with the leaf certificate during the TLS handshake, but the check on the intermediate certificate to the root OCSP responder is still visible. It is worth noting that commonly used intermediate certificates will be cached between sites, so the impact of this may not be as large as it sounds. But caching of intermediate certificates comes with some privacy concerns in Firefox (I've not seen any similar reports for Chrome).

The future of certificate revocation

From the outside looking in (as I was, and still am, a complete 'n00b' to the subject until recently) it comes across to me as incredibly complicated and all a bit of a mess. CRL doesn't scale well, and native OCSP has been all but abandoned by Chrome (apart from for EV certs). OCSP stapling solves some problems with OCSP, but also comes with its own set of problems. So what else is on the horizon?

Well there's an interesting new technology called CRLite that Mozilla have been testing in the Firefox Nightly builds since December 2019.

How does CRLite differ from other CRL revocation technology?

CRLite promises a lot, but how is it different to other CRL revocation technology available in browsers? I'll try and summarise below:

CRL

As mentioned before, CRL is a list of digital certificates that have been revoked by the issuing CA. The browser downloads a copy of the list which it can then use to check to see if the certificate it has been provided is on the list. A big problem with CRL is the list of serials is large and can be quite a considerable file size over the network. And to make matters worse, if a client doesn't have a fresh copy of the list, it has to fetch one during the initial connection to the site! This slows down the whole SSL negotiation process.

OneCRL

OneCRL is what Firefox currently uses for CRL revocation checking. It's been using it since version 37, released back in March 2015. In practice it's a centralised list of revoked certificates that is periodically checked and updated. This list is then pushed out to the browser. To do this, Firefox extends an existing mechanism called blocklisting, which is already used to protect users from errant add-ons, plugins and buggy graphics drivers.

OneCRL comes with a major benefit: speed. If a certificate is covered by OneCRL, a live OCSP request isn't required across the network, so the revocation check can all be done locally. No additional latency is incurred. This is very important for EV certs where a positive OCSP response is required. An important point to note is OneCRL only covers CA intermediate certificates so as to limit the size of the blocklist.

CRLSets

CRLSets is the primary method Chrome (and Chromium based browsers) currently use for quickly revoking certificates in emergency situations. A secondary function is to revoke non-emergency certificates. These secondary revocations are found by crawling CRL's published by CA's. The method of how the sets are delivered to the browser sound very similar to OneCRL in Firefox. The list is compiled daily and then the delta is pushed to the browser via an auto-update mechanism. Adam Langley wrote a very interesting post about CRLSets back in 2014, which gives some insight into how they work and some of their pitfalls.

CRLite

There's a detailed post about how CRLite works here, but as I understand it, the whole revocation list resides inside a users browser and it is updated four times per day. This sounds very much like OneCRL, only with a major difference:

CRLite has low bandwidth costs: it can represent all certificates with an initial download of 10 MB (less than 1 byte per revocation) followed by daily updates of 580 KB on average.

That's an impressive amount of compression! Where OneCRL could only cover intermediate certificates, CRLite should be able to cover them all. If it works, it promises to allow revocation checks when required that replace a network round-trip, with a local lookup for a larger set of revoked certificates. So no web performance overhead like what is seen when establishing connections to an OCSP responder. Sounds promising! I'm interested in seeing how it develops and if it can be standardised and other browser vendors are willing to adopt it too.

Summary

So we come to the end of a meandering blog post on different types of SSL certificates and the performance impact of OCSP revocation checks. As we've seen, these checks don't come for free. They can have a significant impact on web performance depending on what browser a user is using, and the location they are browsing from. And if a site uses an EV certificate, it is forcing the most popular browser on the planet (Chrome) to open a revocation check, which will have an impact on performance for those users who visit your site using it. It makes you wonder if the 'perceived' advantages of using an EV certificate are really worth it? Personally, I don't see any advantages, only disadvantages. And there are a number of blog posts that agree with this stance.

So when you are next considering what type of certificate to purchase for your website, it's worth weighing up the pro's and con's of each, and how your choice could effect your sites performance.

Post changelog:

26/01/20: Initial blog post published. Thanks Barry Pollard for the (lots of) technical review and feedback.
27/01/20: Small tweaks to some sections and added information about the Authority Information Access (AIA) SSL extension. Thanks Barry Pollard.
30/01/20: Added link to Aaron Peters blog post 'EV Certificates Make The Web Slow and Unreliable'.

Full Atom archive feed for my blog by Matt Hobbs (Nooshu.com)

How a Simple CSP Tweak Turned into an AI Project

Introduction

Problem

Features

Audit

Build

Improve

Deploy

Result

trusted-types (TT)

upgrade-insecure-requests

Summary

FractalAI: Generating Infinity in the Browser

Why I Built FractalAI

The Constraint: 100% AI-Generated Code

What FractalAI Actually Does

Why the Browser Is Surprisingly Good at Rendering Fractals

AI as a Design Partner, Not Just a Tool

The Architecture (Without Writing Code Myself)

Machine Learning Inside the Browser

What Worked Surprisingly Well

Planning

Crafted Prompts

Rules

Writing Tests

Refactoring

Proof of concepts

Making unfamiliar things make sense

Building momentum

Where AI Fell Short (and Needed Steering)

The definition of insanity

Over-engineering simple problems

AI output

Flexbox (modern)

Grid (modern)

Confidently wrong outputs

Inconsistent outputs between prompts

Prompt 1

Prompt 2

Performance blind spots

Forgetting earlier constraints

Elementary mistakes

Performance Challenges in the Browser

What Building This Changed About My View of AI

The Future of AI-Generated Software

Summary

FractalAI project

Using Cloudflare Workers and reCAPTCHA v3 for a Static Site Contact Form

Introduction

Previous solutions I’ve used

Netlify Forms

Cloudflare Pages Functions + MailChannels

Formspree.io

Formspark.io (previously Submit Form)

Botpoison

Resend

Overview of the setup

Google reCAPTCHA v3

Behavioural analysis of a user's session

Browser and device fingerprinting

Network and reputation signals

Action Context

Scoring using Machine learning

Technical setup

The contact form (HTML and client-side)

Honeypot and reCAPTCHA to block spam

Client and server validation to ensure safe input

Accessible error handling and focus management

UX updates to keep the user informed during submission

Secure server side processing

Simplified CSP

The Cloudflare Worker (Pages Function)

Using Cloudflare Turnstile

Why change?

Summary

Precompressed HTML at the Edge: Eleventy Meets Cloudflare Workers

Introduction

Caching

Compression

Why we need a Cloudflare Function instead of just the `_headers` file for HTML Brotli?

1. `_headers` can only change headers, not the file itself

The Cloudflare `_headers` file