Test website performance with Puppeteer

How tests written only in JavaScript helped to discover that the service worker bypasses the problem of the speed of light.

Published: 2017-12-22

Measuring website performance has never been so important as today. Just look at tools like Lighthouse, WebPagetest, PageSpeed Insights, or simply performance tab in the browser. In this article, I'm going to take advantage of Puppeteer to automatically test the performance of a website.

All the code from this article you can find here.

  1. Tested application
  2. Navigation Timing API
  3. Chrome DevTools Performance Timeline - First Meaningful Paint
  4. Custom page metrics
  5. Extract data from network trace
  6. Emulate slow network and throttle CPU
  7. Control browser cache and service worker on repeat visits
  8. Results

Tested application

For tests, I picked an application Vue Hacker News 2.0 - one of HNPWA implementations. I choose this app because it uses good performance practices also it is easy to clone the repo and run it locally. All the examples are run locally, but if you don't want to you can use live demo of it https://vue-hn.now.sh. Simply replace in my examples http://localhost:8080 with https://vue-hn.now.sh. But if you use live demo you will not be able to measure custom page metrics because it requires inserting console.timeStamp() in a source code.

I've forked Vue Hacker News 2.0 with modification enabling service worker on localhost and console.timeStamp() in code. Clone the repo and follow build steps.

At the beginning, we are going to measure Navigation Timing. The output should be the same as running window.performance.timing in the browser console.

index.js
const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('http://localhost:8080');

  const performanceTiming = JSON.parse(
    await page.evaluate(() => JSON.stringify(window.performance.timing))
  );
  console.log(performanceTiming);

  await browser.close();
})();

Code above covers all "hello world" needs. puppeteer.launch() creates new browser instance in headless mode, next browser.newPage() can be identified with creating new tab. page.goto('http://localhost:8080') will wait until event load occurs or in bad situation 30s pass.

The entire measurement comes down to sending window.performance.timing in page context by page.evaluate() and decode results with JSON.parse().

After all page measurements, we are simply close browser browser.close() and it will also remove all cache/service workers since we didn't pass any userDataDir parameter to puppeteer.launch().

After running node index.js you will see raw Navigation Timing data like this:

{
  navigationStart: 1513433544980,
  unloadEventStart: 0,
  unloadEventEnd: 0,
  redirectStart: 1513433544980,
  redirectEnd: 1513433545292,
  fetchStart: 1513433545292,
  domainLookupStart: 1513433545292,
  domainLookupEnd: 1513433545292,
  connectStart: 1513433545292,
  connectEnd: 1513433545292,
  secureConnectionStart: 0,
  requestStart: 1513433545019,
  responseStart: 1513433545289,
  responseEnd: 1513433545292,
  domLoading: 1513433545296,
  domInteractive: 1513433545339,
  domContentLoadedEventStart: 1513433545540,
  domContentLoadedEventEnd: 1513433545540,
  domComplete: 1513433545602,
  loadEventStart: 1513433545602,
  loadEventEnd: 1513433545602,
}

This results don't tell you anything? For me too. As you can see those points are represented in some arbitrary point in time. We should calculate difference of each point and navigationStart time. Not all points are interesting for us, we can filtered out some not relevant ones. Additionally, it's time for some refactoring.

index.js
const puppeteer = require('puppeteer');
const testPage = require('./testPage');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  console.log(await testPage(page));
  await browser.close();
})();
testPage.js
const { extractDataFromPerformanceTiming } = require('./helpers');

async function testPage(page) {
  await page.goto('http://localhost:8080');

  const performanceTiming = JSON.parse(
    await page.evaluate(() => JSON.stringify(window.performance.timing))
  );

  return extractDataFromPerformanceTiming(
    performanceTiming,
    'responseEnd',
    'domInteractive',
    'domContentLoadedEventEnd',
    'loadEventEnd'
  );
}

module.exports = testPage;
helpers.js
const extractDataFromPerformanceTiming = (timing, ...dataNames) => {
  const navigationStart = timing.navigationStart;

  const extractedData = {};
  dataNames.forEach(name => {
    extractedData[name] = timing[name] - navigationStart;
  });

  return extractedData;
};

module.exports = {
  extractDataFromPerformanceTiming,
};

index.js contains browser initiation specific code, testPage.js is focused only on running tests, and helpers.js have functions specific for parsing and transformation results.

Now results are well parsed and represented in milliseconds:

{ // all results are in [ms]
  responseEnd: 23,
  domInteractive: 44,
  domContentLoadedEventEnd: 196,
  loadEventEnd: 241
}

Chrome DevTools Performance Timeline - First Meaningful Paint (code)

This chapter will use Chrome Performance Metrics. But haven't we already used some performance timing in the previous chapter? Yes, you can be confused as I was at the beginning. window.performance.timing is browser agnostic measure standard maintained by W3C and all browser should have the same API. On the other hand, Performance Metrics in this chapter is based on metrics specific to Chrome browser (like in performance tab), and they contain not only timings but also some other metrics like:

[
  { name: 'Timestamp', value: 35037.202627 },
  { name: 'AudioHandlers', value: 0 },
  { name: 'Documents', value: 3 },
  { name: 'Frames', value: 2 },
  { name: 'JSEventListeners', value: 63 },
  { name: 'LayoutObjects', value: 435 },
  { name: 'MediaKeySessions', value: 0 },
  { name: 'MediaKeys', value: 0 },
  { name: 'Nodes', value: 506 },
  { name: 'Resources', value: 11 },
  { name: 'ScriptPromises', value: 0 },
  { name: 'PausableObjects', value: 39 },
  { name: 'V8PerContextDatas', value: 1 },
  { name: 'WorkerGlobalScopes', value: 1 },
  { name: 'UACSSResources', value: 0 },
  { name: 'LayoutCount', value: 2 },
  { name: 'RecalcStyleCount', value: 5 },
  { name: 'LayoutDuration', value: 0.0860430000029737 },
  { name: 'RecalcStyleDuration', value: 0.00374899999587797 },
  { name: 'ScriptDuration', value: 0.0770069999925909 },
  { name: 'TaskDuration', value: 0.297364000020025 },
  { name: 'JSHeapUsedSize', value: 6295344 },
  { name: 'JSHeapTotalSize', value: 10891264 },
  { name: 'FirstMeaningfulPaint', value: 35036.03356 },
  { name: 'DomContentLoaded', value: 35036.122972 },
  { name: 'NavigationStart', value: 35035.833805 },
]

Now it's a time to explain that Puppeteer is a high-level API over the Chrome DevTools Protocol (CDP). Puppeteer really helps with common test tasks (like clicking on element and filling inputs etc.) but what you cannot find in Puppeteer API you can achieve with raw CDP. To use CDP we have to register it by const client = await page.target().createCDPSession(), now we can send command with await client.send(). In order to use functions from CDP we have to enable some domains by await client.send('Performance.enable') and after that we can get metrics from raw CDP like this await client.send('Performance.getMetrics').

testPage.js
const {
  getTimeFromPerformanceMetrics,
  extractDataFromPerformanceMetrics,
} = require('./helpers');

async function testPage(page) {
  const client = await page.target().createCDPSession();
  await client.send('Performance.enable');

  await page.goto('http://localhost:8080');

  await page.waitFor(1000);
  const performanceMetrics = await client.send('Performance.getMetrics');

  return extractDataFromPerformanceMetrics(
    performanceMetrics,
    'FirstMeaningfulPaint'
  );
}

module.exports = testPage;
helpers.js
const getTimeFromPerformanceMetrics = (metrics, name) =>
  metrics.metrics.find(x => x.name === name).value * 1000;

const extractDataFromPerformanceMetrics = (metrics, ...dataNames) => {
  const navigationStart = getTimeFromPerformanceMetrics(
    metrics,
    'NavigationStart'
  );

  const extractedData = {};
  dataNames.forEach(name => {
    extractedData[name] =
      getTimeFromPerformanceMetrics(metrics, name) - navigationStart;
  });

  return extractedData;
};

module.exports = {
  getTimeFromPerformanceMetrics,
  extractDataFromPerformanceMetrics,
};

This code is similar to this from the previous chapter, but remember window.performance.timing is executed in webpage context and Performance.getMetrics is executed on browser level (Chrome specific). That's why navigationStart time is different for both metrics.

If you spot the smelly part of code await page.waitFor(1000); in testPage.js you are right. But why I have to delay measurement of FirstMeaningfulPaint? You can be more confused if I told you that FirstMeaningfulPaint time is smaller than load event time in this example (and await page.goto('http://localhost:8080') will wait until load event). This is due to the fact that FirstMeaningfulPaint is not arbitrary point in time, this measurement is based on some heuristics and it is calculated after all page has rendered.

There is no event where FirstMeaningfulPaint is ready, so we cannot precisely detect when this metric is done. But we can make a workaround for checking each small amount of time if this metric is ready:

testPage.js
async function testPage(page) {
  // ...

  // await page.waitFor(1000);
  // const performanceMetrics = await client.send('Performance.getMetrics');

  let firstMeaningfulPaint = 0;
  let performanceMetrics;
  while (firstMeaningfulPaint === 0) {
    await page.waitFor(300);
    performanceMetrics = await client.send('Performance.getMetrics');
    firstMeaningfulPaint = getTimeFromPerformanceMetrics(
      performanceMetrics,
      'FirstMeaningfulPaint'
    );
  }

  // ...
}

Now when the code is free of race conditions, I can show example results:

{ // result is in [ms]
  FirstMeaningfulPaint: 175
}

Custom page metrics (code)

Previous chapters cover metrics which can be used for all websites - they are generic. Now we will try to measure some app-specific metrics. I choose for this example when pagination navigation buttons "< prev" "more >" are controlled by javascript. Why is this point in time important? Because this app uses SSR with hydration markup on the client side. This approach leads to "Uncanny Valley", FirstMeaningfulPaint can be identified with a start of this "Uncanny Valley", and our custom metric listLinksSpa will represent time when "Uncanny Valley" ends.

We must spot in the application where to put console.timeStamp('listLinksSpa'). I deduced that mounted() method in ItemList.vue will be a good place:

src/views/ItemList.vue
beforeMount() { /* ... */ },

mounted() {
  console.timeStamp('listLinksSpa');
},

beforeDestroy() { /* ... */ },

Now we have to register listener page.on('metrics', callback) before page.goto(). When our application calls console.timeStamp('listLinksSpa') callback in page.on('metrics', callback) will be invoked, and then we can extract the time of this metric.

testPage.js
const { getTimeFromPerformanceMetrics } = require('./helpers');

async function testPage(page) {
  const client = await page.target().createCDPSession();
  await client.send('Performance.enable');

  let listLinksSpa;
  page.on('metrics', ({ title, metrics }) => {
    if (title === 'listLinksSpa') {
      listLinksSpa = metrics.Timestamp * 1000;
    }
  });

  await page.goto('http://localhost:8080');

  const performanceMetrics = await client.send('Performance.getMetrics');
  const navigationStart = getTimeFromPerformanceMetrics(
    performanceMetrics,
    'NavigationStart'
  );
  await page.waitFor(1000);

  return {
    listLinksSpa: listLinksSpa - navigationStart,
  };
}

module.exports = testPage;

The code above should work. But the code quality with this await page.waitFor(1000) is far away from acceptable - the vulnerability to race condition is too obvious here. I made above example just to show a simple example. The code below solves this problem by wrapping page.on('metrics', callback) in a promise and make a use of async/await goodness.

testPage.js
const { getTimeFromPerformanceMetrics, getCustomMetric } = require('./helpers');

async function testPage(page) {
  const client = await page.target().createCDPSession();
  await client.send('Performance.enable');

  const listLinksSpa = getCustomMetric(page, 'listLinksSpa');

  await page.goto('http://localhost:8080');

  const performanceMetrics = await client.send('Performance.getMetrics');
  const navigationStart = getTimeFromPerformanceMetrics(
    performanceMetrics,
    'NavigationStart'
  );

  return {
    listLinksSpa: (await listLinksSpa) - navigationStart,
  };
}

module.exports = testPage;
helpers.js
const getTimeFromPerformanceMetrics = (metrics, name) =>
  metrics.metrics.find(x => x.name === name).value * 1000;

const getCustomMetric = (page, name) =>
  new Promise(resolve =>
    page.on('metrics', ({ title, metrics }) => {
      if (title === name) {
        resolve(metrics.Timestamp * 1000);
      }
    })
  );

module.exports = {
  getTimeFromPerformanceMetrics,
  getCustomMetric,
};

Now results without a problems should be like this:

{ // result is in [ms]
  listLinksSpa: 230
}

Extract data from network trace (code)

When you run performance in Chrome tab you can save data as JSON file. Exactly the same will be doing from the Puppeteer level. Just start recording trace with page.tracing.start({ path: './trace.json' }) before page.goto(), and when you feel that everything you need is recorded stop it with page.tracing.stop().

In the code below I will only show extraction of start and end network request time for CSS file.

testPage.js
const {
  getTimeFromPerformanceMetrics,
  extractDataFromTracing,
} = require('./helpers');

async function testPage(page) {
  const client = await page.target().createCDPSession();
  await client.send('Performance.enable');
  await page.tracing.start({ path: './trace.json' });

  await page.goto('http://localhost:8080');

  await page.tracing.stop();
  const cssTracing = await extractDataFromTracing(
    './trace.json',
    'common.3a2d55439989ceade22e.css'
  );

  const performanceMetrics = await client.send('Performance.getMetrics');
  const navigationStart = getTimeFromPerformanceMetrics(
    performanceMetrics,
    'NavigationStart'
  );

  return {
    cssStart: cssTracing.start - navigationStart,
    cssEnd: cssTracing.end - navigationStart,
  };
}

module.exports = testPage;

trace.json is really a mine of information, for this simple app it weights 683 KB. For typical websites, it can weight a few MB. Data inside this file are pretty raw and you should be prepared for really deep digging inside.

Code in getTimeFromPerformanceMetrics() searches for requested file in ResourceSendRequest tracing types. And when it finds it it will get it start time and resourceId. resourceId is used for finding the record when this resource ends.

helpers.js
const fs = require('fs');

const getTimeFromPerformanceMetrics = (metrics, name) =>
  metrics.metrics.find(x => x.name === name).value * 1000;

const extractDataFromTracing = (path, name) =>
  new Promise(resolve => {
    fs.readFile(path, (err, data) => {
      const tracing = JSON.parse(data);

      const resourceTracings = tracing.traceEvents.filter(
        x =>
          x.cat === 'devtools.timeline' &&
          typeof x.args.data !== 'undefined' &&
          typeof x.args.data.url !== 'undefined' &&
          x.args.data.url.endsWith(name)
      );
      const resourceTracingSendRequest = resourceTracings.find(
        x => x.name === 'ResourceSendRequest'
      );
      const resourceId = resourceTracingSendRequest.args.data.requestId;
      const resourceTracingEnd = tracing.traceEvents.filter(
        x =>
          x.cat === 'devtools.timeline' &&
          typeof x.args.data !== 'undefined' &&
          typeof x.args.data.requestId !== 'undefined' &&
          x.args.data.requestId === resourceId
      );
      const resourceTracingStartTime = resourceTracingSendRequest.ts / 1000;
      const resourceTracingEndTime =
        resourceTracingEnd.find(x => x.name === 'ResourceFinish').ts / 1000;

      fs.unlink(path, () => {
        resolve({
          start: resourceTracingStartTime,
          end: resourceTracingEndTime,
        });
      });
    });
  });

module.exports = {
  getTimeFromPerformanceMetrics,
  extractDataFromTracing,
};

Tracing for CSS file are shown below. Remember, this is only one of a thousand records in the tracing.

{ // all results are in [ms]
  cssStart: 27,
  cssEnd: 40
}

Emulate slow network and throttle CPU (code)

All above results get ridiculously fast times. That's because I run all the tests with a high-end device on localhost. Real users will have a weaker network connection and their computing power will be less powerful. We can easily emulate this conditions with Network.emulateNetworkConditions and Emulation.setCPUThrottlingRate. Also, setting fixed network condition helps with reproducibility of tests. It is such a miss that CPU throttling is only a relative slowdown of your machine CPU (across different machines you will get different results).

index.js
const puppeteer = require('puppeteer');
const testPage = require('./testPage');

(async () => {
  let browser = await puppeteer.launch();
  let page = await browser.newPage();
  const client = await page.target().createCDPSession();
  await client.send('Network.enable');
  await client.send('Network.emulateNetworkConditions', {
    offline: false,
    latency: 200, // ms
    downloadThroughput: 780 * 1024 / 8, // 780 kb/s
    uploadThroughput: 330 * 1024 / 8, // 330 kb/s
  });
  await client.send('Emulation.setCPUThrottlingRate', { rate: 4 });
  console.log(await testPage(page, client));
  await browser.close();
})();

Now you can see aggregated results from previous chapters with more realistic times.

{ // all results are in [ms]
  cssStart: 542,
  cssEnd: 789,
  listLinksSpa: 2322,
  FirstMeaningfulPaint: 1061,
  responseEnd: 517,
  domInteractive: 812,
  domContentLoadedEventEnd: 2111,
  loadEventEnd: 2336
}

Control browser cache and service worker on repeat visits (code)

We didn't test the impact of service worker on performance since our test was always with clean browser instance. To measure how a webpage will render with cache or service worker we must run our test second time in the same browser instance. After that, when we call browser.close() all the cached data and service worker will be wiped out since we didn't specify any userDataDir in puppeteer.launch().

If we want to test for example repeat visit only with cache without service worker we have to unregister service worker from the previous entrance by sending ServiceWorker.unregister between first end second enter. The same goes for Network.clearBrowserCache if we want to test only service worker in isolation.

index.js
const puppeteer = require('puppeteer');
const testPage = require('./testPage');

(async () => {
  let browser = await puppeteer.launch();
  let page = await browser.newPage();
  let client = await page.target().createCDPSession();
  console.log(await testPage(page, client)); // first enter
  console.log(await testPage(page, client)); // second enter with cache and sw
  await browser.close();

  browser = await puppeteer.launch();
  page = await browser.newPage();
  client = await page.target().createCDPSession();
  await testPage(page, client); // only for creating fresh instance
  await client.send('ServiceWorker.enable');
  await client.send('ServiceWorker.unregister', {
    scopeURL: 'http://localhost:8080/',
  });
  console.log(await testPage(page, client)); // second enter only with cache
  await browser.close();

  browser = await puppeteer.launch();
  page = await browser.newPage();
  client = await page.target().createCDPSession();
  await testPage(page, client); // only for creating fresh instance
  await client.send('Network.enable');
  await client.send('Network.clearBrowserCache');
  console.log(await testPage(page, client)); // second enter only with sw
  await browser.close();
})();

Results

You can have various reasons to analyze this data. It may be useful to examine the impact of the new functionality on performance change, observing some performance degradation in continuous integration or simply to show some fancy plots like I will do.

For each plot I run test 100 times which makes 600 pages entrances, it took about 10 - 20 min. per test suite.

Cable network (code)

For the typical user, the most important metrics are FirstMeaningfulPaint and listLinksSpa. Those metrics reflects how he perceives website speed. domInteractive has nothing to do with the time when a website is interactive for a user, this metric is represented in this example by a custom listLinksSpa.

responseEnd is a great metric to show the influence of network bandwidth and latency on page speed. Second enter only with cache is served usually with status 304 and it can be served not faster than double latency time - that's why responseEnd from cache occurs around 60 - 70 ms. On the other hand, responseEnd, served from service worker omit network layer, and it is not influenced by a latency. Only device processing power (CPU) effects on this metric when it is served from service worker.

There is no statistically significant difference between only service worker (sw) and service worker with cache, this is because all the network request in this app are covered by a service worker. If they were some for example pictures not handled by a service worker and only by traditional cache, we would see benefits of combining together service worker and cache.

Configuring the service worker requires some work, but the benefits are not so obvious if you look only at results from a good network and device performance. Everything changes if internet performance decreases.

Slow 3G on a good device (code)

Times of metrics handled by service worker is the same as in the previous plot. Requests served only from cache waste a lot of time due to double latency delay. That's why big latency is the most problematic factor in mobile network, not a small bandwidth.

Looking overall at this times it's clear why service worker is promoted to mobile devices. You can improve programmatically website speed (with service worker), you can improve network bandwidth, but you cannot improve speed of light.

Slow 3G on a poor device (code)

Slowing down CPU 6 times the most affected service worker's results. Comparing loadEventEnd from this plot and the previous one:

  • 1.4x slower for first enter
  • 2.6x slower for only cache
  • 5.8x slower for service worker

It looks like service workes is not optimized like traditional cache - it is still a new technology.


I hope I've helped you understand how to get results with the help of the Puppeteer, regardless of what you want to research. I found this tool really easy to adopt.

Just type npm install puppeteer.