This document discusses using the Jesque framework and Redis for distributed job processing. It provides an example of configuring jobs in Jesque that run performance tests using Grails services. Workers pull jobs from the Redis queue in a distributed manner across multiple JVM processes and nodes. Results are saved back to the database. Dashboards can display performance test results.

1. Jesque In Action
gperf framework

2. Why?
Easy Job Configuration
Low Latency
Redis is Fast
Cache
NoSQL
Queue

3. And...
Phase 2: Redis & Jesque

4. And...
Phase 2: Redis & Jesque

5. Creating Jobs
perf {
runners {
largeNumberPerformanceRunner {
description = 'Large Number Performance Test'
maxWorkers = 20
workerClass = com.perf.runners.math.LargeNumberPerformanceService
}
}
}
class LargeNumberPerformanceService extends AbstractPerformanceService {
Result performTest() {
Long result = 1
def executionTime = benchmark {
100000.times {
result += it
}
}
new SimpleResult(testName: 'Long Number Performance Service', executionTime: executionTime)
}
}

6. The Jesque Job
class PerformanceRunnerJob {
GrailsApplication grailsApplication
RedisService redisService
ResultsService resultsService
def executorService
def perform(jobName, workers) {
println "jesque queueing up job ${jobName} with ${workers} threads"
Class clazz = grailsApplication.config?.perf?.runners[jobName]?.workerClass
if(!clazz) {
log.error "Can not start a performance worker without a workerClass defined in the config attribute"
}
PerformanceService service = (PerformanceService) grailsApplication.mainContext.getBean(clazz)
Integer.parseInt(workers).times {
runAsync {
println "running ${jobName} on thread :: ${Thread.currentThread().id}"
while(redisService.get(jobName) == PerformanceConstants.RUNNING) {
saveResults(jobName, service.performTest())
}
}
}
}
private void saveResults(String jobName, Result result) {
log.debug result
resultsService.saveResults(jobName, result)
}
}

7. Performance Distributed
Grails Web
Grails Grails Grails Grails
Jesque Jesque Jesque Jesque
Redis

8. Performance Distributed
Grails Web
Grails Grails Grails Grails
Jesque Jesque Jesque Jesque
Redis

9. Sequence Diagram

10. Dashboards!

11. Dashboards!

12. Demo
Twitter: @ctoestreich
Blog: http://www.christianoestreich.com
GPerf: http://bit.ly/zBHd6b
Github: https://www.github.com/ctoestreich