Puppeteer is a node library which provides a high-level API to control Chrome over the DevTools Protocol. When combined with the power of the web technologies, it can be used for automating image processing and batch file generation, creating automated visual testing with device emulation, tracking page loading performance, enforcing performance and code coverage budgets on CI, crawling a SPA, capturing a timeline trace of your site to help diagnose performance issues and more! We'll explore those capabilities of Puppeteer API with combination of DevTools protocol and cloud functions (FaaS) with a showcase of real life use cases demonstrated by live-examples. Finally, we’ll go through the existing puppeteer based SaaS solutions such as Checkly and Browserless.

1. possibly
Puppeteer can
automate that!
with the help of minions

2. Önder Ceylan
@onderceylan
Sharing knowledge on #javascript, #typescript, #angular, #ionic and #pwa
JS Squad Lead @LINKIT
Speaker, Organiser @ITNEXT
Speaker, Organiser @GDG NL

3. Chrome

4. Chrome
canarychromiumchrome

6. Headless Chrome

7. Headless Chrome
chrome —-headless —-remote-debugging-port=9222

8. Puppeteer

9. Puppeteer
npm i puppeteer

10. “Most things that you can do
manually in the browser can be
done using Puppeteer! ”

11. Examples to get you started
• Generate screenshots and PDFs of pages
• Crawl a SPA and generate pre-rendered
content
• Automate form submission, UI testing,
keyboard input, etc
• Capture a timeline trace of your site to
help diagnose performance issues

12. puppeteer.launch().then(async browser => {
const page = await browser.newPage();
await page.goto('https://www.linkit.nl');
await page.screenshot({path: 'screenshot.png'});
await browser.close();
});

13. pptr.dev

14. try-puppeteer.appspot.com

15. github.com/transitive-bullshit/awesome-puppeteer

16. Puppeteer Recorder Extension

17. DEMOS

18. WhatsApp messages every day!

19. const browser = await puppeteer.launch(({
userDataDir: ".tmp",
}));
const page = await browser.newPage();
await page.goto('https://web.whatsapp.com/', {waitUntil: 'networkidle2'});
await page.waitForSelector('#side input[type=text]');
await page.type('#side input[type=text]', groupName);
await page.waitForSelector(`#pane-side span[title=“${groupName}"]`,
{visible: true});
await page.click(`span[title="${groupName}"`);
await page.waitForSelector('footer .copyable-text', {visible: true});
await page.type('footer .copyable-text', getTodaysMessage());
await page.keyboard.press('Enter');
await browser.close();
WhatsApp messages every day!

20. Emulating color scheme NEW
in v2

21. await page.goto('https://googlechromelabs.github.io/dark-mode-toggle/demo/index.html');
await page.emulateMediaFeatures([{ name: 'prefers-color-scheme', value: 'light' }]);
await page.screenshot({ path: 'light.jpg', type: 'jpeg', omitBackground: true });
await page.emulateMediaFeatures([{ name: 'prefers-color-scheme', value: 'dark' }]);
await page.screenshot({ path: 'dark.jpg', type: 'jpeg', omitBackground: true });
Emulating color scheme NEW
in v2

22. Side by side page load

23. const devices = require('puppeteer/DeviceDescriptors');
const nexus5X = devices['Nexus 5X'];
const browser = await puppeteer.launch({
headless: false,
args: [
`--window-size=${DEFAULT_VIEWPORT.width},${DEFAULT_VIEWPORT.height}`,
CENTER_WINDOWS_ON_SCREEN ? `--window-position=${x},${y}` : `--window-position=${dx},0`,
],
});
const page = await browser.newPage();
await page.emulate(nexus5X);
const session = await page.target().createCDPSession();
// Emulate "Slow 3G" according to WebPageTest
await session.send('Network.emulateNetworkConditions', {
offline: false,
latency: 400,
downloadThroughput: Math.floor(400 * 1024 / 8), // 400 Kbps
uploadThroughput: Math.floor(400 * 1024 / 8) // 400 Kbps
});
await session.send('Emulation.setCPUThrottlingRate', {rate: 4});
Side by side page load

24. Accessibility test

25. await page.goto(‘https://www.linkit.nl');
await page.addScriptTag(
{ url: 'https://cdnjs.cloudflare.com/ajax/libs/axe-core/3.3.2/axe.min.js' }
);
const results = await page.evaluate(() => axe.run(document));
expect(results.violations.length).toBe(0);
Accessibility test

26. Code coverage test

27. const pti = require('puppeteer-to-istanbul');
const page = await browser.newPage();
await page.coverage.startJSCoverage();
await page.goto('https://www.linkit.nl/');
const jsCoverage = await page.coverage.stopJSCoverage();
pti.write(jsCoverage);
await page.close();
Code coverage test

28. Chrome DevTools

29. throttle network
track memory usage
emulate devices
run audits

30. github.com/ChromeDevTools/devtools-frontend

31. Protocol Monitor
chrome://flags/#enable-devtools-experiments

33. chrome —-remote-debugging-port=9222
Chrome DevTools
Protocol

36. 127.0.0.1:9222/json/version

37. chromedevtools.github.io/devtools-protocol

39. Using DevTools
Protocol with PPTR

40. puppeteer.launch().then(async browser => {
const page = await browser.newPage();
await page.goto('https://www.linkit.nl');
const session = await page.target().createCDPSession();
session.on('Animation.animationCreated', () => {
console.log('Animation created!');
});
await session.send('Animation.enable');
await session.send('Animation.setPlaybackRate', {
playbackRate: 10,
});
await browser.close();
});

41. Puppeteer Core
npm i puppeteer-core

42. Puppeteer Core
browser = await puppeteer.connect({
browserWSEndpoint: 'ws://127.0.0.1:9222/.../...'
});
// or
browser = await puppeteer.launch({
executablePath: '/Applications/Google Chrome.app
/Contents/MacOS/Google Chrome'
});

43. Chrome Launcher
npm i chrome-launcher

44. const chromeLauncher = require('chrome-launcher');
// Launches a debugging instance of Chrome
function launchChrome(headless=true) {
return chromeLauncher.launch({
// port: 9222,
chromeFlags: [
'--window-size=412,732',
'--disable-gpu',
headless ? '--headless' : ''
]
});
}
launchChrome().then(chrome => {
console.log(`Chrome debuggable on port: ${chrome.port}`);
// chrome.kill();
});

45. const chromeLauncher = require('chrome-launcher');
// Launches a debugging instance of Chrome
function launchChrome(headless=true) {
return chromeLauncher.launch({
// port: 9222,
chromeFlags: [
'--window-size=412,732',
'--disable-gpu',
headless ? '--headless' : ''
]
});
}
launchChrome().then(chrome => {
console.log(`Chrome debuggable on port: ${chrome.port}`);
// chrome.kill();
});
chrome instance

46. Puppeteer Light
npm i puppeteer-core
npm i chrome-launcher

47. const chromeLauncher = require('chrome-launcher');
const puppeteer = require('puppeteer-core');
const util = require('util');
(async() => {
const URL = 'https://www.linkit.nl';
// Chrome executable launch arguments
const opts = {
chromeFlags: ['--headless'],
logLevel: 'info',
output: 'json'
};
// Launching chrome using chrome-launcher
const chrome = await chromeLauncher.launch(opts);
opts.port = chrome.port;
// Connecting to WS using puppeteer.connect()
const resp = await util.promisify(request)(`http://localhost:${opts.port}/json/version`);
const chromeVersion = JSON.parse(resp.body);
const browser = await puppeteer.connect({
browserWSEndpoint: chromeVersion.webSocketDebuggerUrl
});
// Cleanup of resources
await browser.disconnect();
await chrome.kill();
})();

48. const chromeLauncher = require('chrome-launcher');
const puppeteer = require(‘puppeteer-core');
const util = require('util');
(async() => {
const URL = 'https://www.linkit.nl';
// Chrome executable launch arguments
const opts = {
chromeFlags: ['--headless'],
logLevel: 'info',
output: 'json'
};
// Launching chrome using chrome-launcher
const chrome = await chromeLauncher.launch(opts);
opts.port = chrome.port;
// Connecting to WS using puppeteer.connect()
const resp = await util.promisify(request)(`http://localhost:${opts.port}/json/version`);
const chromeVersion = JSON.parse(resp.body);
const browser = await puppeteer.connect({
browserWSEndpoint: chromeVersion.webSocketDebuggerUrl
});
// Cleanup of resources
await browser.disconnect();
await chrome.kill();
})();

49. const chromeLauncher = require('chrome-launcher');
const puppeteer = require(‘puppeteer-core');
const util = require('util');
(async() => {
const URL = 'https://www.linkit.nl';
// Chrome executable launch arguments
const opts = {
chromeFlags: ['--headless'],
logLevel: 'info',
output: 'json'
};
// Launching chrome using chrome-launcher
const chrome = await chromeLauncher.launch(opts);
opts.port = chrome.port;
// Connecting to WS using puppeteer.connect()
const resp = await util.promisify(request)(`http://localhost:${opts.port}/json/version`);
const chromeVersion = JSON.parse(resp.body);
const browser = await puppeteer.connect({
browserWSEndpoint: chromeVersion.webSocketDebuggerUrl
});
// Cleanup of resources
await browser.disconnect();
await chrome.kill();
})();

50. const chromeLauncher = require('chrome-launcher');
const puppeteer = require(‘puppeteer-core');
const util = require('util');
(async() => {
const URL = 'https://www.linkit.nl';
// Chrome executable launch arguments
const opts = {
chromeFlags: ['--headless'],
logLevel: 'info',
output: 'json'
};
// Launching chrome using chrome-launcher
const chrome = await chromeLauncher.launch(opts);
opts.port = chrome.port;
// Connecting to WS using puppeteer.connect()
const resp = await util.promisify(request)(`http://localhost:${opts.port}/json/version`);
const chromeVersion = JSON.parse(resp.body);
const browser = await puppeteer.connect({
browserWSEndpoint: chromeVersion.webSocketDebuggerUrl
});
// Cleanup of resources
await browser.disconnect();
await chrome.kill();
})();

51. const chromeLauncher = require('chrome-launcher');
const puppeteer = require(‘puppeteer-core');
const util = require('util');
(async() => {
const URL = 'https://www.linkit.nl';
// Chrome executable launch arguments
const opts = {
chromeFlags: ['--headless'],
logLevel: 'info',
output: 'json'
};
// Launching chrome using chrome-launcher
const chrome = await chromeLauncher.launch(opts);
opts.port = chrome.port;
// Connecting to WS using puppeteer.connect()
const resp = await util.promisify(request)(`http://localhost:${opts.port}/json/version`);
const chromeVersion = JSON.parse(resp.body);
const browser = await puppeteer.connect({
browserWSEndpoint: chromeVersion.webSocketDebuggerUrl
});
// Cleanup of resources
await browser.disconnect();
await chrome.kill();
})();

52. DEMOS
even more

53. github.com/onderceylan/pwa-asset-generator

54. PptrGram

56. await Promise.all(filters.map(async (filter) => {
const page = await browser.newPage();
await page.setContent(`
<style>/* Page styles here */</style>
<figure class="${filter}">
<img src="${getImageBase64Url('./sample.jpg')}">
<span>@onderceylan</span>
</figure>
`, { waitUntil: 'networkidle2' });
await page.addStyleTag({
url: 'https://cdnjs.cloudflare.com/ajax/libs/cssgram/0.1.10/cssgram.min.css'
});
// Get original image dimensions
const { width, height } = await page.evaluate(() => {
return (({naturalWidth: width, naturalHeight: height}) =>
({width, height}))(document.querySelector('img'));
});
await page.setViewport({ width, height });
await page.screenshot({
path: `pptgram/pptrgram-${filter}.jpeg`,
type: 'jpeg',
quality: 70,
fullPage: true,
});
});
PptrGram

57. PptrFlow

58. PptrFlow

59. PptrFlow

60. await page.goto(`http://localhost:8002/tensorflow.html`);
const result = await page.evaluate(() => {
const img = document.getElementById('img');
// Load the model
return cocoSsd.load().then(model => model.detect(img))
});
console.log(result);
PptrFlow

61. Monitoring

62. Visual Regression Testing

63. const takeScreenshot = async (page, title) => {
if (!fs.existsSync('./.screenshots')) {
fs.mkdirSync('./.screenshots');
}
const filePath = `./.screenshots/${title}.png`;
if (fs.existsSync(filePath)) {
const newFilePath = `./.screenshots/${title}-new.png`;
await page.screenshot({
path: newFilePath,
fullPage: true
});
const result = await new Promise(resolve =>
looksSame(filePath, newFilePath, (err, equal) => resolve(equal)));
fs.unlinkSync(newFilePath);
return result;
} else {
await page.screenshot({
path: filePath,
fullPage: true
});
return true;
}
};
await page.goto('https://www.linkit.nl');
expect(await takeScreenshot(page, 'main-page.1')).toBeTruthy();
Visual Regression Testing

64. DOM Snapshot Testing

65. const page = await browser.newPage();
await page.goto('https://www.linkit.nl/');
expect(await page.content()).toMatchSnapshot();
DOM Snapshot Testing

66. Budgets on perf metrics

67. Budgets on perf metrics
{ name: 'Timestamp', value: 66567.150449 },
{ name: 'AudioHandlers', value: 0 },
{ name: 'Documents', value: 8 },
{ name: 'Frames', value: 3 },
{ name: 'JSEventListeners', value: 34 },
{ name: 'LayoutObjects', value: 455 },
{ name: 'MediaKeySessions', value: 0 },
{ name: 'MediaKeys', value: 0 },
{ name: 'Nodes', value: 970 },
{ name: 'Resources', value: 74 },
{ name: 'ContextLifecycleStateObservers',
value: 0 },
{ name: 'V8PerContextDatas', value: 4 },
{ name: 'WorkerGlobalScopes', value: 0 },
{ name: 'UACSSResources', value: 0 },
{ name: 'RTCPeerConnections', value: 0 },
{ name: 'ResourceFetchers', value: 8 },
{ name: 'AdSubframes', value: 0 }
{ name: 'DetachedScriptStates', value: 2 },
{ name: 'LayoutCount', value: 13 },
{ name: 'RecalcStyleCount', value: 22 },
{ name: 'LayoutDuration', value: 0.067929 },
{ name: 'RecalcStyleDuration', value: 0.029508 },
{ name: 'ScriptDuration', value: 0.122922 },
{ name: 'V8CompileDuration', value: 0.003031 },
{ name: 'TaskDuration', value: 0.336774 },
{ name: 'TaskOtherDuration', value: 0.116415 },
{ name: 'ThreadTime', value: 0.275266 },
{ name: 'JSHeapUsedSize', value: 7816504 },
{ name: 'JSHeapTotalSize', value: 11096064 },
{ name: 'FirstMeaningfulPaint', value: 66565.452541 },
{ name: 'DomContentLoaded', value: 66565.386449 },
{ name: 'NavigationStart', value: 66564.624457 }
Performance.getMetrics

68. Budgets on perf metrics
[
{
"path": "/",
"perfMetrics": [
{
"metric": "JSEventListeners",
"budget": 100
},
{
"metric": "Nodes",
"budget": 2000
},
{
"metric": "JSHeapUsedSize",
"budget": 20000000
}
]
}
]
[
{
"path": "/vacatures",
"perfMetrics": [
{
"metric": "Resources",
"budget": 80
}
]
}
]
budget.json

69. const { getBudgetMetricsOfPage, getMatchedPageMetrics,
getBudgetMetricByPageMetricName } = require(‘./helpers');
const assertMetricsOnUrl = async (siteUrl) => {
const page = await this.browser.newPage();
const protocol = await page.target().createCDPSession();
await protocol.send('Performance.enable');
await page.goto(siteUrl, { waitUntil: 'networkidle0' });
const budgetMetrics = await getBudgetMetricsOfPage(page);
const { metrics: pageMetrics } = await protocol.send('Performance.getMetrics');
const matchedMetrics = getMatchedPageMetrics(pageMetrics, budgetMetrics);
matchedMetrics.forEach(pageMetric => {
expect(pageMetric.value).toBeLessThan(
getBudgetMetricByPageMetricName(budgetMetrics, pageMetric)
);
});
await page.close();
};
Budgets on perf metrics

70. beforeAll(async () => {
this.browser = await puppeteer.launch();
});
afterAll(async () => {
await this.browser.close();
});
test('asserts budget performance metrics on the main page', async() => {
await assertMetricsOnUrl('https://www.linkit.nl/');
});
test('asserts budget performance metrics on vacatures page', async() => {
await assertMetricsOnUrl('https://www.linkit.nl/vacatures');
});
Budgets on perf metrics

71. FPS Monitoring

72. const protocol = await page.target().createCDPSession();
await protocol.send('Overlay.setShowFPSCounter', { show: true });
await page.goto('https://www.linkit.nl');
// Do graphical regressions here by interacting with the page
await protocol.send('Input.synthesizeScrollGesture', {
x: 100,
y: 100,
yDistance: -400,
repeatCount: 3
});
await page.screenshot({
path: 'fps.jpeg',
type: 'jpeg',
clip: {
x:0,
y:0,
width: 370,
height: 370
}
});
FPS Monitoring

73. Memory leak by Heap

74. const protocol = await page.target().createCDPSession();
await protocol.send('HeapProfiler.enable');
await protocol.send('HeapProfiler.collectGarbage');
const startMetrics = await page.metrics();
// Do memory regressions here by interacting with the page
await protocol.send('Input.synthesizeScrollGesture', {
x: 100,
y: 100,
yDistance: -400,
repeatCount: 3
});
await protocol.send('HeapProfiler.collectGarbage');
const endMetrics = await page.metrics();
expect(endMetrics.JSHeapUsedSize < startMetrics.JSHeapUsedSize * 1.1)
.toBeTruthy();
Memory leak by Heap

75. Monitor SSL certificate expiration

76. const page = await browser.newPage();
page.on('response', (resp) => {
const url = resp.url();
if (url === siteUrl) {
const secDetails = resp.securityDetails();
const now = Math.floor((new Date()).getTime() / 1000);
console.log((Math.floor((secDetails.validTo() - now) / 86400)),
'days to expire');
}
});
await page.goto(siteUrl, { waitUntil: 'networkidle0' });
Monitor SSL certificate expiration

77. Puppeteer on Cloud

78. const launchChrome = require('@serverless-chrome/lambda');
const request = require('superagent');
module.exports.getChrome = async () => {
const chrome = await launchChrome();
const response = await request
.get(`${chrome.url}/json/version`)
.set('Content-Type', 'application/json');
const endpoint = response.body.webSocketDebuggerUrl;
return {
endpoint,
instance: chrome,
};
};
PPTR on AWS Lambda via Serverless Framework

79. const puppeteer = require('puppeteer');
const { getChrome } = require('./chrome-script');
module.exports.hello = async (event) => {
const { url } = event.queryStringParameters;
const chrome = await getChrome();
const browser = await puppeteer.connect({
browserWSEndpoint: chrome.endpoint,
});
const page = await browser.newPage();
await page.goto(url, { waitUntil: 'networkidle0' });
const content = await page.evaluate(() => document.body.innerHTML);
return {
statusCode: 200,
body: JSON.stringify({
content,
}),
};
};
PPTR on AWS Lambda via Serverless Framework

80. PaaS
Puppeteer as a Service

81. browserless.io

82. checklyhq.com

83. Main Takeaways
• Chrome can be instrumented with a WS connection over
Chrome DevTools Protocol — CDP
• You might not need to download chromium revision every time
—puppeteer-core + chrome-launcher
• Headless chrome can be executed on servers— cloud, and CI
pipelines
• You can automate anything you do on DevTools, by using raw
protocol of CDP on puppeteer—CDPSession

84. bit.ly/pptr-demos

85. Thank you!
bit.ly/pptr-talk@onderceylan