As release velocity increases, teams are finding innovative ways to detect and resolve performance issues earlier in the development cycle. This session will explore how to integrate performance testing into a development lifecycle and why the practice is gaining popularity. Learn how to create a balanced test strategy that matches test type, coverage, environment, and defect target. Scale to deliver high-performing applications. Through continuous integration platforms, performance testing tools, and AppDynamics, see how automated performance testing can reduce time-to-market while increasing overall quality. Key takeaways: o How to get started with performance test automation o How to detect and resolve performance issues earlier in the development lifecycle with AppDynamics o How to maximize the quality and value of your performance test strategy Brad Stoner Senior Sales Engineer, AppDynamics Resources: KonaKart - http://www.konakart.com/ Adminer - https://www.adminer.org/ Jenkins - https://jenkins.io/ NeoLoad - http://www.neotys.com/neoload/overview AppDynamics - https://www.appdynamics.com/free-trial/ GitHub - https://github.com/ Docker Hub - https://hub.docker.com/

1. Automate performance testing with
AppDynamics using continuous
integration platforms
Brad Stoner
Senior Sales Engineer | AppDynamics

2. APPDYNAMICS CONFIDENTIAL AND PROPRIETARY 2
Notice
The information and materials included in this presentation (collectively, the
“Materials”) are the proprietary information of AppDynamics, Inc. (“AppDynamics” or
the “Company”). No part of the Materials may be reproduced, distributed,
communicated or displayed in any form or by any means, or used to make any
derivative work, without prior written permission from AppDynamics.
The Materials may contain product roadmap information of AppDynamics.
AppDynamics reserves the right to change any product roadmap information at any
time, for any reason and without notice. This information is intended to outline
AppDynamics' general product direction, it is not a guarantee of future product
features, and it should not be relied on in making a purchasing decision. The
development, release, and timing of any features or functionality described for
AppDynamics' products remains at AppDynamics' sole discretion. AppDynamics
reserves the right to change any planned features at any time before making them
generally available as well as never making them generally available.
All third-party trademarks, including names, logos and brands, referenced by
AppDynamics in this presentation are property of their respective owners. All
references to third-party trademarks are for identification purposes only and shall be
considered nominative fair use under trademark law. © 2016 AppDynamics, Inc. All
rights reserved.

3. Key takeaways
• How to get started with performance test automation
• How to detect and resolve performance issues earlier in the
development lifecycle with AppDynamics
• How to maximize the quality and value of your performance test
strategy
AppDynamics Confidential and Proprietary 3

4. My background
• 7 years @ H&R Block Load and Performance Team
– 5 person team
– 100k + user concurrency
– Tax peak 2nd week after go-live
– 70 applications annually
– Diverse technology stack – including 3rd party
• 2 years @ Neotys – Senior Performance Engineer
• Currently Sales Engineer @ AppDynamics
AppDynamics Confidential and Proprietary 4

5. What is performance testing?
AppDynamics Confidential and Proprietary 5

6. Why bother?
AppDynamics Confidential and Proprietary 6
Google - Using page speed in site ranking
Facebook - Launches 'lite' mobile app
Amazon - 100ms delay -> $6.79M sales
decrease
Recent airline industry outages

7. Legacy performance testing
AppDynamics Confidential and Proprietary 7

8. Increasing velocity
…performance testing
isn't historically fast
AppDynamics Confidential and Proprietary 8

9. Keeping up with Agile / DevOps
AppDynamics Confidential and Proprietary 9

10. Performance defects
AppDynamics Confidential and Proprietary 10

11. Pull back the layers
AppDynamics Confidential and Proprietary 11
Prod / Perf
Pre-Prod /
Staging
Dev / QA

12. Dev / QA
• Short test cycle
• Low resources / cost
• Rapid feedback
• Component testing
AppDynamics Confidential and Proprietary 12

13. Staging / Pre-Prod
• Increased resources
• Long test durations
• Additional resources
• Multiple test
elements
• Integrated systems
• Build validation
AppDynamics Confidential and Proprietary 13

14. Prod / Perf
• High complexity
• Multiple integrated
systems
• Environmental variance
• Resource intensive
• High cost
AppDynamics Confidential and Proprietary 14

15. Resources and speed
Resources
Speed
Prod / Perf
Staging / Pre-
Prod
DEV/QA

16. Mobile web/app example
Dev testing - APIs
Staging testing – Capacity w/ UI
and API
Build automation
Baseline Pre-Prod / Staging -
platform
Prod / Perf testing (inside
firewall) – stability /
scalability
Prod / Perf testing
(outside firewall) –
network / load
balancing
QA testing – API flows
Optimize app chatter
and network
resources
Mobile app released
Mobile app built
Mobile site releasedMobile site built
APIs released/ BE functionality
Front End
Optimization

17. What if legacy test principles were applied?
Staging testing –
Capacity w/ UI and API
Baseline Pre-Prod /
Staging - platform
Prod / Perf testing (inside firewall) –
stability / scalability
Prod / Perf testing (outside
firewall) – network / load
balancing
Front End
Optimization
Optimize app
chatter and
network
resources
Mobile app released
Mobile app built
Mobile site releasedMobile site built
APIs released/ BE functionality

18. Demo

19. AppDynamics Key Components
AppDynamics Usage
Compare Releases Compare critical metrics for different tests (FE and BE)
Database Monitoring Measure and visualize performance deltas between tests
Server Monitoring Measure and visualize performance deltas between tests
Jenkins Plugin Enables automated pass/fail of tests
Troubleshoot Slow
Response Times
Quickly find root cause of performance issues and test failure
Service Endpoints Track shared services performance over time (APIs)
Custom Match Rules Name Business Transactions to match load test scripts
Custom Dashboards Management and technical level tracking of application
performance
AppDynamics Confidential and Proprietary 19

20. Questions and contact
• Email: brad.stoner@appdynamics.com
• Twitter: @sandbreak80
AppDynamics Confidential and Proprietary 20

21. Please give us your feedback—Session T5808
• Complete the online survey you'll receive via
email later today or via text at:
Text this number: 878787
Text this word: APPSPHERE
• Every time you submit a session survey, your
name will be entered in a random drawing.
We're giving away Amazon Echos
to 5 lucky winners!
• Thank you for your input
APPDYNAMICS CONFIDENTIAL AND PROPRIETARY 21
Win!

22. Thank you

23. Screenshots of live demo
for offline viewing

24. Jenkins for CI/CD
AppDynamics Confidential and Proprietary 24

25. Git and AppDynamics integration
AppDynamics Confidential and Proprietary 25

26. Jenkins job details
AppDynamics Confidential and Proprietary 26

27. AppDynamics Jenkins plugin
AppDynamics Confidential and Proprietary 27

28. Performance job execution
AppDynamics Confidential and Proprietary 28

29. NeoLoad performance report
AppDynamics Confidential and Proprietary 29

30. KonaKart test application
AppDynamics Confidential and Proprietary 30

31. Custom events in AppDynamics
AppDynamics Confidential and Proprietary 31

32. Identify slow queries
AppDynamics Confidential and Proprietary 32

33. Add missing index
AppDynamics Confidential and Proprietary 33

34. Compare release performance
AppDynamics Confidential and Proprietary 34

35. Reference material

36. Technology stack
AppDynamics Confidential and Proprietary 36
AppDynamics Confidential and Proprietary 36

37. Demo resources
• AppDynamics controller - https://www.appdynamics.com/free-trial/
• Docker Compose YML: https://github.com/sandbreak80/myapp
• NeoLoad project for KonaKart (localhost):
https://github.com/sandbreak80/konakart
– Jenkins user will need access to modify these files
• Docker Hub for mysql access via adminer:
https://hub.docker.com/r/sandbreak80/adminer/
• Docker Hub for konakart:
https://hub.docker.com/r/sandbreak80/docker_konakart/
AppDynamics Confidential and Proprietary 37

38. Demo software install
• Setup instance for docker
– Install Docker: curl -sSL https://get.docker.com/ | sh
– Avoid sudo for docker: sudo usermod -aG docker ubuntu
– Install docker compose:
• sudo apt-get -y install python-pip
• sudo pip install docker-compose
• Jenkins 2 install (if needed):
– wget -q -O - https://pkg.jenkins.io/debian/jenkins-ci.org.key | sudo apt-key add -
– sudo sh -c 'echo deb http://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'
– sudo apt-get update
– sudo apt-get install jenkins
• NeoLoad install
– wget https://d24mnm5myvorwj.cloudfront.net/documents/download/neoload/v5.2/neoload_5_2_2_linux_x64.sh
– sudo chmod +x neoload_5_2_2_linux_x64.sh
– mkdir /home/ubuntu/neoload
– sudo sh neoload_5_2_2_linux_x64.sh -q -dir /home/ubuntu/neoload -Vsys.installationTypeId=Controller -
Vsys.component.Common$Boolean=true -Vsys.component.Controller$Boolean=true
AppDynamics Confidential and Proprietary 38

39. Start demo environment
• Clone docker-compose file
– mkdir myapp
– cd myapp
– git clone https://github.com/sandbreak80/myapp.git
• Pull, build, and start docker containers
– sudo docker-compose start
AppDynamics Confidential and Proprietary 39

40. Configure AppDynamics
• Connect to the KonaKart instance
– sudo docker exec -i -t konakart /bin/bash
• Install VIM: apt-get install vim
• Edit the tomcat environment file
– vi /usr/local/konakart/bin/setenv.sh
– export CATALINA_OPTS="$CATALINA_OPTS -
javaagent:/home/appdynamics/java_agent/javaagent.jar
– -Dappdynamics.controller.hostName=your_controller_name_or_ip
– -Dappdynamics.controller.port=8090
– -Dappdynamics.agent.applicationName=KonaKart
– -Dappdynamics.agent.tierName=Kona_Server
– -Dappdynamics.agent.nodeName=Node1
– -Dappdynamics.agent.accountName=customer1
– -Dappdynamics.agent.accountAccessKey=your_access_key"
AppDynamics Confidential and Proprietary 40

41. AppDynamics authentication
• BASIC authentication is used in AppDynamics to create custom events and time ranges in the
next two slides
– As a reference, the AppDynamics REST API is documented here -
https://docs.appdynamics.com/display/PRO42/Using+the+Controller+APIs.
– The first call authenticates with the AppDynamics controller, and stores the authentication cookie into
/tmp/session.dat on the Jenkins Server. Note that we’re using a basic authentication header instead of
directly exposing our username and password.
– There are a number of utilities you can leverage to base64 encode a string, like
https://www.base64encode.org/.
– The username / password format to encode is: <username>@<account name>:<password> and the @
symbol needs to be URI encoded.
– By default, in single tenant controllers, the <account name> will always be customer1. So, to get our
base64 authentication header we can encode - admin%40customer1:appdynamics – to get our
authentication header
– YWRtaW4lNDBjdXN0b21lcjE6YXBwZHluYW1pY3M=
– ***Note***
– We need to URI encode our username and password string, which is why we use ‘%40’ instead of the
‘@’ character.
AppDynamics Confidential and Proprietary 41

42. Create custom event in AppDynamics
• Load test start (linux shell step):
– curl --verbose -s -c /tmp/session.dat --header "Authorization: Basic your_base_64_token" -X
GET http://your_appdynamics_controller:8090/controller/auth?action=login
– PARAMS="events?eventtype=CUSTOM&customeventtype=LOAD_TEST&summary=LOAD_
TEST_START-${JOB_NAME}-${BUILD_ID}&severity=INFO&tier=Kona_Server”
– curl -s -b /tmp/session.dat --data "${PARAMS}"
http://your_appdynamics_controller:8090/controller/rest/applications/KonaKart/events
• Load test stop (linux shell step):
– curl --verbose -s -c /tmp/session.dat --header "Authorization: Basic your_base_64_token " -X
GET http://your_appdynamics_controller:8090/controller/auth?action=login
– PARAMS="events?eventtype=CUSTOM&customeventtype=LOAD_TEST&summary=LOAD_
TEST_STOP-${JOB_NAME}-${BUILD_ID}&severity=INFO&tier=Kona_Server”
– curl -s -b /tmp/session.dat --data "${PARAMS}"
http://your_appdynamics_controller:8090/controller/rest/applications/KonaKart/events
AppDynamics Confidential and Proprietary 42

43. Create custom time range in AppDynamics
• Linux shell step in Jenkins for creating a custom time range based on job start and end:
#!/bin/sh
END=$(date +%s)
echo "$END"
END_MS=$(($END * 1000))
echo "$END_MS"
echo "$BUILD_TIMESTAMP"
jobstart=$BUILD_TIMESTAMP
START=$(date -d "${jobstart}" +%s)
echo "$START"
START_MS=$(($START * 1000))
echo "$START_MS"
curl --verbose -s -c /tmp/session.dat --header "Authorization: Basic YWRtaW4lNDBjdXN0b21lcjE6YXBwZHluYW1pY3M=" -X GET http://devopslabappsphere:8090/controller/auth?action=login
PARAMS="{"name":"LOAD_TEST-${JOB_NAME}-
${BUILD_ID}","description":"","shared":true,"timeRange":{"type":"BETWEEN_TIMES","durationInMinutes":0,"startTime":${START_MS},"endTime":${END_MS}}}"
curl --verbose -s -b /tmp/session.dat --header "Content-Type: application/json;charset=utf-8" --header "Authorization: Basic YnJhZCU0MGN1c3RvbWVyMTozQ29tMzgxMg==" --header "Accept-
Encoding: gzip, deflate" --header "Accept: application/json, text/plain" --data "${PARAMS}" http://devopslabappsphere:8090/controller/restui/user/createCustomRange
AppDynamics Confidential and Proprietary 43

44. NeoLoad with GIT
• GitHub project - https://github.com/sandbreak80/konakart
– mkdir neoload_projects/konakart
– cd neoload_projects/konakart
– git clone https://github.com/sandbreak80/konakart.git
• Pull project in Jenkins job (linux shell step):
– cd /home/ubuntu/neoload_projects/konakart
– git fetch –all
– git reset --hard origin/master
• Push test results (linux shell step):
– cd /home/ubuntu/neoload_projects/konakart
– git add *git commit -a -m "jenkins-${JOB_NAME}-${BUILD_NUMBER}”
– git push https://username:your_github_token@github.com/username/konakart.git --all
• Generate GitHub token:
– https://help.github.com/articles/creating-an-access-token-for-command-line-use/
AppDynamics Confidential and Proprietary 44

45. Jenkins plugins used
• AppDynamics - https://wiki.jenkins-
ci.org/display/JENKINS/AppDynamics+Plugin/
• NeoLoad - https://wiki.jenkins-
ci.org/display/JENKINS/NeoLoad+Plugin
• Timestamper - https://wiki.jenkins-
ci.org/display/JENKINS/Timestamper
– Manage Jenkins -> Configure ->
– Date pattern for the BUILD_TIMESTAMP
– EEE d MMM yyyy HH:mm:ss
AppDynamics Confidential and Proprietary 45

46. Adminer access
• http://your_server_name:8081/
• host: konakart
• user : monitor
• password: appd123
• database: konakart
AppDynamics Confidential and Proprietary 46

47. KonaKart access
• http://your_server_name:8780/konakart/
AppDynamics Confidential and Proprietary 47