Encontrar o limite que uma aplicação suporta sem causar outages é uma tarefa trabalhosa e muitas vezes cara. No entanto, é uma boa prática descobrir esse limite no ambiente de teste, por isso investimos tempo e dinheiro para descobrir o volume de requests que uma aplicação suporta antes que isso apareça em produção. Existem algumas ferramentas consolidadas no mercado que facilitam a nossa vida, por exemplo: Apache JMeter, BlazeMeter, Gatling e k6. Testamos muitas delas e decidimos construir nossa própria tool de load testing utilizando Playwright e AWS Batch. Confira como criamos usuários virtuais, cada um no seu browser, executando tarefas end-to-end utilizando Playwright e AWS Batch.

1. Por que Criamos uma Ferramenta
de Load Testing utilizando
Playwright e AWS Batch?

2. @anderparra
/anderparra
Anderson Parra
Senior Software Engineer

3. What’s SeatGeek?
With our combination of technological
prowess, user-first attitude and dashing good
looks, we, SeatGeek, are simplifying and
modernizing the ticketing industry.
By simultaneously catering to both the
consumer and enterprise markets, we’re
powering a new, open entertainment industry
where fans have effortless access to
experiences, and teams, venues and shows
have seamless access to their audiences. We
think it’s time that everyone can expect more
from ticketing.

4. SeatGeek
Vision Statement
Become the largest and
most loved ticket
marketplace by offering
consumers a meaningfully
better ticketing
experience.

6. The in-house queueing system
Virtual
Waiting Room

7. Virtual Waiting Room Operation Modes

8. Transitions from Blockade to Throttle Strate.
The Waiting Room Main States
All traffic to the protected zone is
blocked, showing a waiting room
page to end-users.

9. Transitions from Blockade to Throttle State.
The Waiting Room Main States
Allows a configurable
number of end-users
per minute into the
protected zone. After
initial seeding, users are
allowed in First-In, First-
Out (FIFO).

10. Let’s see an example:
I would like to buy
tickets for an
event with high
demand…

11. Blockade - Waiting
Room Mode

12. Throttle - Queue
Mode

13. Virtual Waiting Room Stack Overview

15. A browser-based load testing tool
Frenzy

16. import ws from 'k6/ws'
import { check } from 'k6'
export default function () {
const visitorToken = '???'
const url = `wss://stag.vroom.com/?vt=${visitorToken}`
const res = ws.connect(url, {}, (socket) => {
socket.on('open', () => console.log('connected'))
socket.on('message', (m) => {
/* check the message m */
socket.close()
)}
socket.on('close', () => console.log('disconnected'))
socket.on('error', (e) => {
if (e.error() !== 'websocket: close sent') {
console.log('Unexpected error: ', e.error())
}
})
})
check(res, { 'status is 101': (r) => r && r.status === 101 })
}
https://k6.io

17. case class ProtectedResource(path: String) {
val execute: ChainBuilder = ???
}
object VroomQueue {
val execute: ChainBuilder =
exec(ws("Connecting").connect("/?vt=${visitorToken}"))
}
class QueueSimulation extends Simulation {
val httpProtocol = http
.baseUrl("https://stag.seatgeek.com")
.doNotTrackHeader("1")
.wsBaseUrl("wss://stag.vroom.com")
val scenarioBuilder = scenario("Accessing PZ").exec(
ProtectedResource(path).execute,
doIf(session => session.contains("visitorToken")) {
VroomQueue.execute
})
val execution = scenarioBuilder
.inject(constantUsersPerSec(concurrency) during holdFor)
.protocols(httpProtocol)
setUp(execution)
}
https://gatling.io

19. So why did we
decide to
create our
tool?

20. Frenzy Tech Stack
Platform Tools Environment Observability Notification

21. How is it
connected?

22. Orchestrator
Protected Zone
seatgeek.com/event-1

23. JOB 0
JOB 1
JOB 2
Orchestrator
Protected Zone
seatgeek.com/event-1

24. JOB 0
JOB 1
JOB 2
browsers
browsers
browsers
Orchestrator
Protected Zone
seatgeek.com/event-1

25. JOB 0
JOB 1
JOB 2
browsers
browsers
browsers
Orchestrator
Protected Zone
seatgeek.com/event-1

26. JOB 0
JOB 1
JOB 2
browsers
browsers
browsers
Orchestrator
Protected Zone
seatgeek.com/event-1
AWS Batch

27. JOB 0
JOB 1
JOB 2
simulator
Orchestrator
Protected Zone
seatgeek.com/event-1
AWS Batch
Gitlab CI
simulator
simulator
browser
sessions
browser
sessions
browser
sessions

28. JOB 0
JOB 1
JOB 2
browsers
browsers
browsers
Orchestrator
Protected Zone
seatgeek.com/event-1
AWS Batch
Gitlab CI

29. JOB 0
JOB 1
JOB 2
browsers
browsers
browsers
Orchestrator
Protected Zone
seatgeek.com/event-1
AWS Batch
Gitlab CI

30. JOB 0
JOB 1
JOB 2
browsers
browsers
browsers
Orchestrator
Protected Zone
seatgeek.com/event-1
AWS Batch
Gitlab CI
6.000 AWS Batch jobs, if
each job runs 15 browsers
instances, then it results in
90.000 virtual users

31. JOB 0
JOB 1
JOB 2
browsers
browsers
browsers
Orchestrator
Protected Zone
seatgeek.com/event-1
AWS Batch
Gitlab CI
Scheduled Executions
► Monday (large)
► Tuesday (small)
► Thursday (small)

32. func (o *Orchestrator) OrchestrateFrenzy(ctx context.Context, conf FrenzyConfig) {
logger.Info("Transitioning zone to blockade")
_ := o.vroom.TransitionProtectedZone(ctx, conf.zoneULID, vroom.BlockadeState);
job, _ := o.batch.SubmitJob(ctx, &batch.SubmitJobInput{...})
// start watching for running jobs
watchRunningJobsCh := NewWatcher(o.batch).WatchRunningJobs(ctx, *job.JobId)
_ := o.awaitSpinup(ctx, conf, watchRunningJobsCh);
logger.Info("Transitioning zone to throttle")
o.vroom.TransitionProtectedZone(ctx, conf.zoneULID, vroom.ThrottleState);
// start watching for status jobs
watchStatusJobCh := NewWatcher(o.batch).Watch(ctx, *job.JobId)
_ := o.awaitBatchDone(ctx, conf, watchStatusJobCh);
}
func (o *Orchestrator) awaitSpinup(ctx context.Context,
conf FrenzyConfig, watchResultCh <-chan WatchResult) error {
for {
select {
case result := <-watchResultCh:
// await for AWS Batch jobs being in running mode
case <-spinupTimeout.C:
return fmt.Errorf("timed out waiting for virtual users to spin up")
}
}
}
func (o *Orchestrator) awaitBatchDone(ctx context.Context,
conf FrenzyConfig, watchResultCh <-chan WatchResult) error {
for {
select {
case result := <-watchResultCh:
// checks result.JobDetail.Status
case <-doneTimeout.C:
return fmt.Errorf("timed out waiting for job to finish after 1 hour")
}
}
}
Orchestrator

33. async function simulateUser(i, browser) {
let page;
const context = await browser.newContext();
page = await context.newPage();
await step(`(worker=${i}) WHEN I visit the page`, async () => {
const resp = await page.goto('stag.seatgeek.com/event-1');
expect(resp.status()).to.equal(200, "received non-200");
})
await step(`(worker=${i}) THEN I eventually see the target page`, async () => {
await page.waitForFunction(() => {
// validate if the content page matches to expected target
return true | false;
}, null, { timeout: 90e3 * 60 });
})
}
async function main() {
let browser = await playwright['chromium'].launch({
headless: true
});
let results = await Promise.all(
Array.from({length: numUsers})
.map((_, i) => simulateUser(i, browser)));
}
https://playwright.dev

34. Why AWS Batch instead of AWS Lambda?
The Lambda max timeout is 900 seconds
(15 minutes).
Lambda functions are short lived
With no max timeout restrictions
we can simulate long queues in
order to find bottlenecks.
AWS Batch Jobs are long lived

35. Summary

36. Prepare your apps for huge traffic.
Scaling is always a good thing for a web app, but it can also be a pain point. How can
you be sure you can handle more traffic than you currently have? And if you can’t
handle it, where is problem coming from?
Regardless the tool you are going to use, emulate heavy
traffic and analyse the outcome reports enable you to
find potential bugs and bottle necks.

37. The End
Thank You!
Thanks to everyone for your patience and time.
Hope everyone has a better understanding of
the loading test importance.
Questions?
If you want to know more about any of this,
please reach me out:
Anderson Parra
@anderparra
/anderparra