1. Best Practices for Rigorous Evaluation
of RFID Software Components
RFID-SysTech 2010, 06/15/2010
Matthieu-P. Schapranow

2. Outline
2
■ Introduction
■ Use Case: The European Pharmaceutical Supply Chain
■ Generating Realistic Test Data
■ Selected Metrics to Consider
■ Benchmarking EPC Network Components
■ Summary
Best Practices for Rigorous Evaluation of RFID Software Components, RFID-SysTech 2010

3. Introduction
3
■ Conducting rigorous evaluation of software artifacts is hard
■ This is especially true for RFID software components
□ high throughput and
□ large data volume
■ We want to share our insights into
evaluating RFID components
□ EPCIS servers and
□ EPC Discovery Services
Best Practices for Rigorous Evaluation of RFID Software Components, RFID-SysTech 2010

4. Outline
4
■ Introduction
■ Use Case: The European Pharmaceutical Supply Chain
■ Generating Realistic Test Data
■ Selected Metrics to Consider
■ Benchmarking EPC Network Components
■ Summary
Best Practices for Rigorous Evaluation of RFID Software Components, RFID-SysTech 2010

5. Use Case: The European
Pharmaceutical Supply Chain
5
Best Practices for Rigorous Evaluation of RFID Software Components, RFID-SysTech 2010

6. Use Case: The European
Pharmaceutical Supply Chain
6
Best Practices for Rigorous Evaluation of RFID Software Components, RFID-SysTech 2010

7. Use Case: The European
Pharmaceutical Supply Chain
7
Best Practices for Rigorous Evaluation of RFID Software Components, RFID-SysTech 2010

8. Outline
8
■ Introduction
■ Use Case: The European Pharmaceutical Supply Chain
■ Generating Realistic Test Data
■ Selected Metrics to Consider
■ Benchmarking EPC Network Components
■ Summary
Best Practices for Rigorous Evaluation of RFID Software Components, RFID-SysTech 2010

9. Generating Realistic Test Data
9
■ Two possibilities
□ Data Generators
□ Simulation
■ Our conclusion: only simulation can reflect the complex
dependencies of RFID-aided supply chains
■ We implemented a special-purpose simulator
■ Four scenarios of the pharmaceutical supply chain
Interference No interference
Scale 1:2000 Scenario 1 Scenario 3
Scale 1:1000 Scenario 2 Scenario 4
■ Download at epic.hpi.uni-potsdam.de/Home/RFIDSySTech2010
Best Practices for Rigorous Evaluation of RFID Software Components, RFID-SysTech 2010

10. Outline
10
■ Introduction
■ Use Case: The European Pharmaceutical Supply Chain
■ Generating Realistic Test Data
■ Selected Metrics to Consider
■ Benchmarking EPC Network Components
■ Summary
Best Practices for Rigorous Evaluation of RFID Software Components, RFID-SysTech 2010

11. Selected Metrics to Consider
11
■ Application Performance Index (Apdex)
□ satisfied: 0≤t<x
□ tolerating: x ≤ t < 4x
□ frustrating: t ≥ 4x
□ RS = number of satisfied response time samples
□ RT = number of tolerating response time samples
□ N = number of all samples in the report group
Best Practices for Rigorous Evaluation of RFID Software Components, RFID-SysTech 2010

12. Selected Metrics to Consider
12
■ Throughput
□ A = Amount of completed tasks
□ T = Observation time
Best Practices for Rigorous Evaluation of RFID Software Components, RFID-SysTech 2010

13. Outline
13
■ Introduction
■ Use Case: The European Pharmaceutical Supply Chain
■ Generating Realistic Test Data
■ Selected Metrics to Consider
■ Benchmarking EPC Network Components
■ Summary
Best Practices for Rigorous Evaluation of RFID Software Components, RFID-SysTech 2010

14. Benchmarking EPC Network Components
14
■ System in test
□ EPCIS server and/ or
□ EPC Discovery Service
■ Apache JMeter as a test tool
□ Apache Project
□ Widely accepted performance test tool for web applications
□ Property files for variables, e.g. server URLs
□ Very flexible through BeanShell scripting
Best Practices for Rigorous Evaluation of RFID Software Components, RFID-SysTech 2010

15. Single System in Test
15
Best Practices for Rigorous Evaluation of RFID Software Components, RFID-SysTech 2010

16. Multiple Systems in Test
16
■ Define system URLs in JMeter “user.properties”,
■ Store EPCIS server addresses in JMeter variables
Best Practices for Rigorous Evaluation of RFID Software Components, RFID-SysTech 2010

17. Multiple Systems in Test
17
Best Practices for Rigorous Evaluation of RFID Software Components, RFID-SysTech 2010

18. Multiple Systems in Test
18
■ and send events to respective EPCIS servers.
■ Files and Scripts for download at epic.hpi.uni-potsdam.de/Home/
RFIDSySTech2010
Best Practices for Rigorous Evaluation of RFID Software Components, RFID-SysTech 2010

19. Outline
19
■ Introduction
■ Use Case: The European Pharmaceutical Supply Chain
■ Generating Realistic Test Data
■ Selected Metrics to Consider
■ Benchmarking EPC Network Components
■ Summary
Best Practices for Rigorous Evaluation of RFID Software Components, RFID-SysTech 2010

20. Summary
20
■ Evaluating software systems is important
■ How to create realistic data  simulation
■ What to measure  see “metrics to consider” (Apdex, throughput)
■ How to put load on the software system  Apache JMeter
■ How to test multiple systems at once  JMeter & BeanShell
Best Practices for Rigorous Evaluation of RFID Software Components, RFID-SysTech 2010

21. 21
Thank you for
your attention!
Questions?
Best Practices for Rigorous Evaluation of RFID Software Components, RFID-SysTech 2010