Icst2012 zaman

•Download as PPTX, PDF•

0 likes•110 views

SAIL_QU

What is this study about?
Users
10 Requests
per user
Software System
2
1,000 Requests
10 Requests
Bad Response Time
Requests

Scenario-Centric View
Users
Software System
3
1,000 Requests
10 Requests
Bad Response Time
10 Requests
per user
1% bad request
instance

User-Centric View
Users
Software System
4
1,000 Requests
10 Requests
Bad Response Time
10 Requests
per user
0% bad
request
instance
50% bad
request
instance
1% bad request
instance
User’s
Perspective
System’s
Perspective

Data used in this study
• 3 systems
• 13 most used scenarios
Factor
Enterprise
System 1
Enterprise
System 2
Dell DVD
store
Functionality Telecommunications E-commerce
Vendor’s
Business
Model
Commercial Open-source
Size Ultra Large Large Small
Complexity Complex Complex Simple
5

Performance Load Test
Load Generator(s)
Requests
Requests
Requests
Software System
Under Test
6

Performance Load Test
Runs for hours or days
Data
Performance AnalysisLoad Generator(s)
Requests
Requests
Requests
Software System
Under Test
7

Current Practice of Performance
Analysis
8
Data Values
Density
Version 1
Data Values
Density
Version 2

Load Generator(s)
Requests
Requests
Requests
Software System
Under Test
Problem with Current Practice
Data aggregated for all users!!
Improved
Degraded
9

Scenario-Centric View
Load Generator(s)
Requests
Requests
Requests
Software System
Under Test
PASS
10

User-Centric View
Load Generator(s)
Requests
Requests
Requests
Software System
Under Test
User Group 1
U.G. # 2
U.G. # 3
U.G. # 4
FAILED
11

% of bad instances
Median
Median
±
St.Deviation
ResponseTime
Request Instance #
12

% of bad instances
Median
Median
±
St.Deviation
Bad instances = Out of “Median ± Standard Deviation”
= 6/20 = 30%
ResponseTime
Request Instance #
13

Study
14
Users
Requests
Software System
User’s
Perspective
System’s
PerspectiveVS
Data
Performance
Analysis

Our Study Dimensions
Overall Trend Consistency
vs
15

Overall Performance
Old New
% of bad
instances
16.48 0.21
Scenario Centric View User Centric View
16

Our Study Dimensions
Overall Trend Consistency
vs
10 out of 13 use-cases
showed a different
view
17

Performance Trend Over Time
Scenario Centric View User Centric View
Old
New
0 15 30 45 60
30354045
ResponseTime
Running Time
0 20 40 60 80 100 120 140
406080100120
ResponseTime
Instance # for a user
Old
New
18

Performance Trend Over Time
0 2 4 6 8 10 12 14
600060506100
ResponseTime(mean)
Instance # for a user
0 100 200 300 400 500 600
585059005950600060506100
ResponseTime
Running Time
System’s
perspective
User’s
perspective
19

Our Study Dimensions
Overall Trend Consistency
vs
10 out of 13 use-cases
showed a different
view
8 out of 13 use-cases
showed a different
view
20

Performance Consistency
Scenario Centric View
0 500 1500 2500 3500
200400600800
ResponseTime
Running Time
Old
New
User Centric View
0 5 10 15 20 25 30
-100005001500
ResponseTime
Instance # for user
Old
New
NewOld
21

Our Study Dimensions
Overall Trend Consistency
vs
10 out of 13 use-cases
showed a different
view
8 out of 13 use-cases
showed a different
view
All 13 use-cases
showed a different
view
22

Trend
8 out of 13 use-cases
showed a different
view
Our Study Dimensions
Overall
10 out of 13 use-cases
showed a different
view
Consistency
vs
All 13 use-cases
showed a different
view
23
VS
Consistency vs Overall Performance
InconsistentConsistent BadGood

0 10 20 30 40 50 60 70
0100002000030000
Variance
% of bad instances
Old
New
Consistency vs Overall Performance
Overall Performance experience
BadGood
1 2
34
C
I
consistency
24

Viewers also liked

Qsic2008 jiang

Icsm2010 kamei

Msr2010 ibrahim

Wcre2009 bettenburg

Compsac2010 malik

Icpc2010 bettenburg

Icse2011 build maintenance

Icsm2009 alam

Msr2012 adams

Issre2010 malik

Ase2010 shang

Icsm2009 adams ph_d

Msr2011 zaman

Icsm2012 selective codeintegration

SAIL_QU

Fse2011 bettenburg presentation

Icse2012 shang

Wcre2010 shihab

Wcre2009 alam

Icsm2009 shihab

Icse2010 adams presentation

SAIL_QU

Viewers also liked (20)

Qsic2008 jiang

Icsm2010 kamei

Msr2010 ibrahim

Wcre2009 bettenburg

Compsac2010 malik

Icpc2010 bettenburg

Icse2011 build maintenance

Icsm2009 alam

Msr2012 adams

Issre2010 malik

Ase2010 shang

Icsm2009 adams ph_d

Msr2011 zaman

Icsm2012 selective codeintegration

Fse2011 bettenburg presentation

Icse2012 shang

Wcre2010 shihab

Wcre2009 alam

Icsm2009 shihab

Icse2010 adams presentation

Similar to Icst2012 zaman

Using Control Charts for Detecting and Understanding Performance Regressions ...

SAIL_QU

load_testing.ppt.pptx

PerformanceTesting1

Video: https://vimeo.com/131387407 The micro task method is an efficient and objective way to measure performance benefits of control room interface solutions. The method supports benchmarking of innovative interfaces against conventional control room instrumentation. The micro task tablet app is a mobile companion for Human Factors and Human Reliability professionals who can develop targeted, efficient user testing scenarios. Micro tasks allow you to easily identify interface elements that cause performance problems. Micro task data lends itself to big data analystics since data from successive studies can be easily aggregated. Future developments include a micro task database as well as extending the method into the areas of teamwork and training. For more information, please contact michael.hildebrandt@ife.no

Micro Task Interface Evaluation Method

mhilde

Performance Testing

Selin Gungor

Performance Testing using LoadRunner

Kumar Gupta

Tablet tools and micro tasks

mhilde

B2 2006 sizing_benchmarking (1)

Steve Feldman

B2 2006 sizing_benchmarking

Steve Feldman

Final Project presentation (on App devlopment)

S.M. Fazla Rabbi

Developing Web-scale Machine Learning at LinkedIn - From Soup to Nuts

Kun Liu

Need for Speed: How to Performance Test the right way by Annie Bhaumik

QA or the Highway

Introduction to System, Simulation and Model

Md. Hasan Imam Bijoy

MeasureWorks - Why your customers don't like to wait!

MeasureWorks

I described these two studies during the Research in Practice: Studies UXers should know about workshop. I expected them to be drive-bys ... as in, "Yah, yah, .. have heard that ... let's move on." I was surprised to find that the group -- a sharp, engaged and thoughtful group-- didn't know these studies. Instead of a few minutes description, we discussed and debated how these studies might influence UX practice for almost an hour. Based on that, I got nudged (Culprit = @susandra Susan Dray) to presenting these two @ the UXPA unconference. There are many other studies studies that all UXPros should be familiar with ...

2 Studies UX types should know about (Straub UXPA unconference13)

Kath Straub

[Webinar] Getting Started with Performance Testing

Maharshi Shah

MeasureWorks - Velocity Europe - Real World Rum

MeasureWorks

Data quality isn’t always the sexiest topic, but it’s critical and one that buyers and suppliers often neglect to have. The ramifications of ignoring it can cost millions of dollars. Some of the industry’s largest buyers and suppliers have found a simple solution though and it’s one that is available to everyone else too. Come here about how the issue of data quality concerns haven’t gone away, and what others are doing to make sure they and their insights are protected.

Data Quality Doesn’t Just Happen: And Here’s What Some of the Industry’s Most...

InsightInnovation

What does it take to be a performance tester?

SQALab

Modeling & simulation in projects

anki009

Simulation Presentation

guest92c571a7

Similar to Icst2012 zaman (20)

Using Control Charts for Detecting and Understanding Performance Regressions ...

load_testing.ppt.pptx

Micro Task Interface Evaluation Method

Performance Testing

Performance Testing using LoadRunner

Tablet tools and micro tasks

B2 2006 sizing_benchmarking (1)

B2 2006 sizing_benchmarking

Final Project presentation (on App devlopment)

Developing Web-scale Machine Learning at LinkedIn - From Soup to Nuts

Need for Speed: How to Performance Test the right way by Annie Bhaumik

Introduction to System, Simulation and Model

MeasureWorks - Why your customers don't like to wait!

2 Studies UX types should know about (Straub UXPA unconference13)

[Webinar] Getting Started with Performance Testing

MeasureWorks - Velocity Europe - Real World Rum

Data Quality Doesn’t Just Happen: And Here’s What Some of the Industry’s Most...

What does it take to be a performance tester?

Modeling & simulation in projects

Simulation Presentation

More from SAIL_QU

In-app advertisements have become a major revenue for app developers in the mobile app economy. Ad libraries play an integral part in this ecosystem as app developers integrate these libraries into their apps to display ads. However, little is known about how app developers integrate these libraries with their apps and how these libraries have evolved over time. In this thesis, we study the ad library integration practices and the evolution of such libraries. To understand the integration practices of ad libraries, we manually study apps and derive a set of rules to automatically identify four strategies for integrating multiple ad libraries. We observe that integrating multiple ad libraries commonly occurs in apps with a large number of downloads and ones in categories with a high percentage of apps that display ads. We also observe that app developers prefer to manage their own integrations instead of using off the shelf features of ad libraries for integrating multiple ad libraries. To study the evolution of ad libraries, we conduct a longitudinal study of the 8 most popular ad libraries. In particular, we look at their evolution in terms of size, the main drivers for releasing a new ad library version, and their architecture. We observe that ad libraries are continuously evolving with a median release interval of 34 days. Some ad libraries have grown exponentially in size (e.g., Facebook Audience Network ad library), while other libraries have worked to reduce their size. To study the main drivers for releasing an ad library version, we manually study the release notes of the eight studied ad libraries. We observe that ad library developers continuously update their ad libraries to support a wider range of Android versions (i.e., to ensure that more devices can use the libraries without errors). Finally, we derive a reference architecture for ad libraries and study how the studied ad libraries diverged from this architecture during our study period. Our findings can assist ad library developers to understand the challenges for developing ad libraries and the desired features of these libraries.

Studying the Integration Practices and the Evolution of Ad Libraries in the G...

SAIL_QU

Studying the Dialogue Between Users and Developers of Free Apps in the Google...

SAIL_QU

Improving the testing efficiency of selenium-based load tests

SAIL_QU

Studying User-Developer Interactions Through the Distribution and Reviewing M...

SAIL_QU

Studying online distribution platforms for games through the mining of data f...

SAIL_QU

Understanding the Factors for Fast Answers in Technical Q&A Websites: An Empi...

SAIL_QU

Investigating the Challenges in Selenium Usage and Improving the Testing Effi...

SAIL_QU

Mining Development Knowledge to Understand and Support Software Logging Pract...

SAIL_QU

Which Log Level Should Developers Choose For a New Logging Statement?

SAIL_QU

Towards Just-in-Time Suggestions for Log Changes

SAIL_QU

The Impact of Task Granularity on Co-evolution Analyses

SAIL_QU

A Framework for Evaluating the Results of the SZZ Approach for Identifying Bu...

SAIL_QU

How are Discussions Associated with Bug Reworking? An Empirical Study on Open...

SAIL_QU

A Study of the Relation of Mobile Device Attributes with the User-Perceived Q...

SAIL_QU

A Large-Scale Study of the Impact of Feature Selection Techniques on Defect C...

SAIL_QU

Studying the Dialogue Between Users and Developers of Free Apps in the Google...

SAIL_QU

What Do Programmers Know about Software Energy Consumption?

SAIL_QU

Threshold for Size and Complexity Metrics: A Case Study from the Perspective ...

SAIL_QU

Revisiting the Experimental Design Choices for Approaches for the Automated R...

SAIL_QU

Measuring Program Comprehension: A Large-Scale Field Study with Professionals

SAIL_QU

More from SAIL_QU (20)