Performance Test Bulls Eye by Mr. Haries Efrika

1. DEVOPS INDONESIA
Jakarta, 04 April 2019
Performance Test Bulls Eye
DevOps Community in Indonesia

2. Performance Test Bull’sEye

3. is a
in Indonesia

4. is a leading marketplace
in Indonesia

5. is a leading marketplace
in Indonesia

6. is a technology company
in Indonesia
that creates opportunities
and helps everyone achievemore

7. Our Mission is to
Democratize Commerce
Through Technology

8. 2 minutes to startabusiness
Over5 million merchants
70% first-timeentrepreneurs
Reach93% of districts inIndonesia
100 million products listedwith
samepriceandservices
100,000 payment collectionpoints
Affordableinvestmentsandworking
capitalsolution
Over30 digital productsthat
savetimeforconsumers
Our Impact Today

9. Marketplace &
Digital Goods
Fintech&
Payment
Logistic &Fulfillment
New Retail
Our Businesses
FreeC2Cbusinessplatform for
merchantsandbuyers. Merchant
tools thatempower merchantsto
domore.Also hostOfficialStores
for brands.30digitalproducts
thatsimplifylives.
Digitalwallet,affordable
investments,merchant
loans,virtualcreditcard,
protectionproducts,
data-drivencreditscoringfor
lending,investmentsand other
financialservices.
Shippingmadeeasywithour
integratedlogistics system.
Customerscanchoosetheir
preferredpackagearrivaltime
whichwill beaccommodated by
ourlogistic providers.
Mitra Tokopediaenables small
merchanttosell digitalgoods
andkeep theirofflinestore,and
allow storesto buyproducts
and stocks inwholesale.

10. Performance Test
Bull’s Eye

11. Whyduringrealeventtheservice is still down?
The Ultimate Question
Already had performance test; result RPS was high.

12. 1. Preparationto acceptspecial eventtraffic,muchbiggerthanusual
2. Measurehowstrongourserviceis
3. Earlyidentifyissues foundduringheavyload
Back to basics: what are the purpose of load test?
Expectations after doing proper loadtest
1. Know whereweare,howstrong weare,howweakweare
2. Know howbigweshall scale verticallyandhorizontally
3. Know whichpartof thecode has performancedebt
4. Know whichpoints to armourservices so theywon’tgetdown during
event
5. Know if ourservices will autorecoveror notafter heavyload passed

13. Before we continue, do not
forget the most important
aspect to setup!
Right! The Metrics
Load Test isnot about execution
…itis aboutpreparation!

14. Q: Where to put the Metrics ?
A: Everywhere that fits

15. 1. MaxRequests?
2. MaxRPS?
3. AverageRequests?
4. AverageRPS?
5. MaxLatency?
6. AverageLatency?
7. PercentileLatency?
8. Success/Error Rates?
9. Anythingelse? Dotheyallmatter?
1. All services/APIs/Pages exposedtooutside/internet
2. All dynamicresources thatareproducedbytheserver
3. Exclude:static resources should stayin CDNorsimilar/S3
What to load test?
Load Test Result must
be seen in multi
dimensional way!
Load Test Output torecord?

16. Time
Metrics
Users
RP
S
SuccessRate
Latency
This might be our breaking point
100%successrate
Success Rate
dropsbelow80%
Success raterecovers

17. What is successrate?
What is breakingpoint?
1. OurMain Score for loadtest
2. Point in RPS graphwheresuccess rateis less thanourpre-definedacceptable/ standard
rate,i.e.80%
3. Belowtheacceptablesuccess ratetheRPS informationis nolongeruseful,because theyare
failing;Andwedon’ttakeprideonfailures,dowe?
Percentageof Requestscomparedto previouspoint,whicharenotfailed.
● I.e. currentpointhas 9000requestscount(since beginningoftest)
● Previous point(i.e.fewseconds ago)has 8500requestscount
● At currentpoint,numberof totalrequestfailures is120
● At previouspoint,numberof totalrequestfailures was110
● Deltarequest=9000-8500=500 Deltafailures =120-110=10
● Hencecanbesaid that,from500requests,wehave10failures,which definesa
98%success rate

18. Generic rule to load testcase:
Needto cyclethroughall theparametersacceptedbyanAPI/ page,withrandom,dynamic values,actual
valuesandcombinationof themall.
How to load test a login page/API?
Randomlytryto login manydifferent(actual)users to thesystemandseehowit handles. More unique
users provided,is better.
How to load test a search page/API?
Randomlytryto usebothvalidkeywordandinvalidkeywordswhencallingtheAPI. We needto have
hugelist of keyword(s)database.
Hence, can be derived: How to load test a transactional flow?
● Randomlyloginauser
● Randomlysearchfor ashop
● Fromashop,randomlygetaproductto addto cart,thenaddtocart
● Randomlyuseaddress(if possible) tocheckout
● Makepayment

19. Load TestTips
● Use singleindependentconnectionperrequest.Donotre-useconnection/socket.
● Definethetimeoutcriteriaandstick withit as ourqualitymeasure.
● Ourloadtestqualitydependsonthedynamicdataweuseduringthetest.Please investgood
amountof timeto preparetheloadtestdataanddon’ttakeitlightly.
● Needto produceenoughloads onthetargetserver.If CPU orMemoryusageis still low,thenwe
arenotdoingit goodenough.
● Prepareprofilerbeforehand.If oursystemdoes nothaveon-demand/liveprofiler, thenwehave
to implementthis. Profiling shall bedonewhenloadtestreachesits peaktraffic,so thatwe
knowwhatfunctiontoimprove.
● Donotgetsatisfied withshort timeloadtest.Prolong thetestwithhightrafficto see if oursystem
canhandlethis in prolongedperiod.
● Runseveralrelatedload tests togetherto simulaterealfunnel.(Wemightbe surprisedat
whatissue canbecaused)

20. Service PerformanceTips
● SeparatetheDBusecases properly.Utilize in-memoryDBlikeRedis,useelastic- searchand
no-SQLas alternativeof conventionalACID DBaccordingto usecase.
● Use webservercachingfunctionalityfor static/ notso dynamicpages.
● Investin goodsystemarchitecturethatcanimmediatelyautorestarttheserviceif down.
● Investin goodServerLoad Balancerthatcanbeexpandedif needed.
● If bottleneckis inmicroserviceCPU, justdohorizontalscaling will solvethis.Butnot if this
happensin databaseCPU or I/O. Taketimeto reviewandreinvestin robust databasearchitecture.
Know whento separateDB,whento dosharding,whento use commercialcloudDBservice.
● In theendwecan’tmakeeverybodyhappy:thereforethesystemhas to apply incomingservice
limiters,andrejectimmediately(withfriendlyusermessage)when systemis alreadyfull.

21. Now, It’s Your Turn!
tokopedia.com/about