Applied Observational Study

1. Applied Observational Study
School of Information Technology and Engineering
Addis Ababa University
Research methodology for cyber security
1

2. Outline
Introduction
• Applied and observational research
• Applied study types
Applied observational study
• Applied exploratory studies
• Applied descriptive studies
Applied observation method selection
Data collection and analysis
Applied Exploratory Study: Stress test
Applied Descriptive Study: Case study
Reporting your results
2

3. CHAPTER OBJECTIVES
Explain the differences between applied and observational
studies
Discuss how to design applied studies
Walk through example using applied study methods
Introduce the topic of operational bounds testing
Provide a template for presenting results
3

4. Introduction
 What is applied research?
 Applied research includes designing, implementing, and testing
systems.
 It is a major aspect of cyber security.
 What is the key difference between applied study and observational
study?
4

5. Observational Study
Observational study is the entire cyber system without injection or
introduction of a change or variable from the observer.
Applied Study
Applied study introduces a specific change or subject that is to be evaluated.
General Definition
5

6. Observational Vs Applied
 The subject of fundamental observational study is the entire cyber system
without injection or introduction of a change or variable from the
observer.
 On the other hand, applied study introduces a specific change or subject
that is to be evaluated.
6

7. What is the key difference?
 Applied observational studies are likely the most common type of
research conducted in the field of cyber security.
 The key difference between applied study and observational study is
the differing scope.
 Applied study observes a specific subject for performance, function,
security, etc.
 Fundamental observational study observes the entire system
without presumption of behavior.
7

8. Applied observational study
 An applied study observes a new solution to understand how it performs
under different conditions. Often this is a new defensive feature or system
change.
 Most of the people assume that applied observational study is just like
fundamental observational study, but not.
 applied observational studies are likely the most common type of research
conducted in the field of cyber security.
Like computer science, and allied fields, researchers in cyber security
are often focused on presenting their technology, solution, algorithm,
or process to the public.
8

9. Applied study
 An applied study observes a new solution to understand how it performs
under different conditions.
 Often this is a new defensive feature or system change.
Furthermore, this is accompanied by an assumption or prediction.
 The researcher has an expectation or unwritten assumption of how the
subject should behave.
 A designer of an applied study seeks to understand the effect of some
change or effect under observation;
 this often comes with an assumption of performance or behavior.
9

10. Applied Observational study Types
 there are two categories one can define applied observational study,
The applied version of an observational study includes:
 Applied exploratory and
 Descriptive studies.
Note:-The research done using this Chapter can help inform and improve future
foundational and applied research and development
10

11. Applied exploratory research
 It is the process of observing and studying how an engineered system
behaves in different situations.
 This kind of study can introduce a specific change or subject that is to
be evaluated.
 Studies include sensitivity analysis and operational bounds testing such
as load, performance, and stress testing.
We will use an example of a new anomaly-based intrusion detection system to
demonstrate the concepts of applied exploratory studies.
11

12. Applied exploratory research…
Operational Bounds Testing: The objective of this type of applied
observational study is to explore the boundary conditions, limits, and
extremes of an observed cyber system.
Stress testing: Evaluates how to what extent a system can perform at
extremes.
Performance analysis: Evaluates how well system behavior conforms
to expectations.
Load testing: Evaluates the system or processes as maximum
expected load
12

13. Applied exploratory research…
One of the example is sensitivity analysis
 Sensitivity Analysis: The objective of sensitivity analysis is to study and
understand the scope, variability, and limitations of the system based on
changes to the inputs.
• It is the study of how precisely the outputs of a system are
correlated to the inputs of the system.
• Or mathematically, how the uncertainty in the outputs can be
related to the inputs.
classical stress testing of performance, for example read/ write speed, communication
latency, or cryptographic performance or password response times are another
examples of applied exploratory study
13

14. Applied exploratory research …
 The inclusion of any sort of controls or dependent and independent
variables would make this an experiment, quasi experiment, or applied
experimental.
 Does it solve the problem better than before? and Is it cheaper, faster, ‘better?’
These sorts of questions can be addressed with an applied descriptive study.
 This sort of research should also be sure to describe any adverse, negative, or
unintended consequences.
14

15. Applied descriptive research
 It is more focused on a specific subject under test.
 Often focus more on an individual subject or more specialized target
subject. Examples of this type of applied study include case studies,
elicitation studies, and case reports.
 Applied descriptive studies observes how application of knowledge,
process, or a system work in a real setting.
15

16. Applied Observation Method Selection
 Operational bounds testing and applied descriptive studies have different
objectives.
 Operational bounds testing are techniques that are good for exploring if
you have developed or selected the right solution based on your believed
requirements.
• The area of operational bounds testing is fairly self-explanatory.
• The objective of this type of applied observational study is to explore
the Applied Study boundary conditions, limits, and extremes of an
observed cyber system.
16

17. Operational bounds testing…
For example,
• how accurate is a system or process?
• How long does it take to conduct a task?
• What type of performance is possible under different conditions?
• This is often related to resource utilization?
 Applied Descriptive studies on the other hand are good for
documenting how you operationally integrated a new solution into a
real-world environment and what you learned.
17
With applied research, we will explain how affective knowledge is applied to solve a problem and explore
measuring the performance of some system or event. This is the key focus of applied studies.
Note:-

18. Data Collection and Analysis
Applied Exploratory Studies
 Data from operational bounds testing will be either collected or generated
around specific test conditions.
 If it is for stress testing then a large amount of data will be generated or if t
is load testing you may collect data from a real environment.
 For example in sensitivity analysis, the purpose is to evaluate extreme
conditions. For those types of study, the goal is pass or fail criteria and
minimum thresholds for different performance variables.
18

19. Data Collection and Analysis Continued…
 For General sensitivity analysis, using graphical methods is helpful.
 In addition to visualizing results, the Receiver operator characteristic
(ROC) curve is a sensitivity analysis technique well suited to large
categories of cyber security solutions.
 Data collected from descriptive studies will largely fall into qualitative
categories. This includes interviews, surveys, stream of thought journals,
and so on.
19

20. Data Collection and Analysis Continued…
Applied Descriptive Studies
 Issues with bias and sampling can inadvertently influence and even ruin
the results of an applied observational study.
 Applied observational research will still use the same statistical
techniques to make sense of the data collected.
 Approaches such as regression testing and statistical tests such as the
T-test will be applied.
20

21. APPLIED EXPLORATORY STUDY: STRESS TEST
Scenario: Let’s posit that you are a part of a larger research team working on Internet of
Things (IOT) and mobile applications, for first responders and emergency response. The
team has come up with a new communication application that enables peer-to-peer
communication, without hierarchical infrastructure (which would often be down in times
of disaster). The team wanted to make sure that the communication is as secure as
possible. The problem is that the cryptographic tools used could be power hungry, which
might unreasonably drain the battery and burden the end user.
21

22. Applied Exploratory Study: Stress Test…
We will divide the study design into three categories, the system, the
behavior, and the testing methodology. System, Behavior and testing
Methodology.
 System: The first part of the study that needed to be defined is the system under test itself.
 Behavior: Next, we will need to define the behavior to be studied. Since the software in
question secures communication, and because radio frequency (RF) transmission is
typically the greatest consumption of energy, we will scope the focus to evaluate at the
extreme level of communication.
22

23. Testing Methodology
• Finally, the last piece of the study is the testing methodology itself.
• We have several host-based testing tools that evaluate system battery consumption
of mobile devices, but the problem is that those tools themselves run on the device.
• There is concern that the collecting of battery consumption telemetry will
inadvertently affect the results using a host-based tool.
• Initial tests were run with this approach, but for this study direct evaluation of
battery status will be used (via wired hardware taps).
• Initially all 10 devices were going to be used in the test, but because stress testing
might inadvertently damage the hardware, we will pare this back to 3 baseline and
3 study devices
23

24. Testing Methodology…
 The specification of these batteries state 500 cycles before 80% of their
original capacity.
 Open literature seems to indicate numbers ranging from 400 to 1000 cycles.
 we first will establish baseline behavior on all six batteries.
 At each test we will measure: the capacity (Ah), state of charge (to get our %
battery), depth of discharge (%), open-circuit voltage (V), and time elapsed.
(Note: baseline tests are not needed for stress testing, but the researcher
wanted to ensure that no bad lemons (hardware failures) affect the results of
the stress testing).
24

25. Testing Methodology...
• Now for our performance and baseline runs we will additionally measure
the number of bytes transmitted.
• We intend to run both the baseline systems (without the secure
communication application) and the test systems through 30 cycles.
• Again this is a double stress test, evaluating the devices without the new
secure communication software and with it installed.
• This is two simultaneous stress tests.
• We will then collect the data to determine if that is sufficient for analysis,
before performing more tests.
25

26. Testing Methodology…
• The first 3 phones were programed to perform full bandwidth transmission
until the battery hit 75%, 50%, and 25%, then the capacity (Ah), state of
charge (to get our % battery), depth of discharge (%), open-circuit voltage
(V), time, and number of bytes transmitted are recorded.
• Similarly, the test devices were evaluated using the exact same protocol.
• 30 cycles later we collected sufficient information to conduct our statistical
analysis.
• We used a T-test to compare the two datasets.
• After conducting another 30 runs, we got consistent results. Essentially, we
are blasting an extreme amount of traffic to both systems to ensure that we
push the performance to limit.
26

27. The final test was to determine the impact on the entire lifecycle of the
device to determine if it held out for large numbers.
The test was again conducted, but this time for 100 battery cycles.
This was done four times to determine if more than 400 charge and
discharge cycles would have an considerable effect on the consumption
and drain of the ad-hoc communication software.
This sort of testing is used to evaluate the speed, consumption, utilization,
and general performance of systems.
Testing Methodology…
27

28. APPLIED DESCRIPTIVE STUDY: CASE STUDY
• case study. Imagine we have the same research team that has
developed a new secure communication tool for First Responder
mobile devices.
• Stress testing has been conducted to understand how the system
performs in laboratory conditions, and extreme use cases.
• But now we want to study how real users will use the phone.
• Our plan is to conduct two phases of a study.
• We will initially conduct “An applied study on the effectiveness of a
new secure ad-hoc communications protocol for first responders.”
28

29. APPLIED DESCRIPTIVE STUDY: CASE STUDY…
• First, we will provide the devices to first responders and ask them to use it
for their regular work phone.
• We will work with each subject to ensure that their typical applications and
functionality is available.
We will then conduct interviews every month to determine how the system
performs for four months.
(1) perceived decrease in performance or functionality
(2) any increased functionality (when traditional service was unavailable but secure
neighborhood area networking was available) and
(3) any user feedback to help improve the system
29

30. APPLIED DESCRIPTIVE STUDY: CASE STUDY…
• Second, the subjects are briefed on the context and importance of their
reporting.
• Third, the questions were designed to limit the subjectivity as much as
possible. Specific quantitative terms, ordinal values, and short answer
questions were designed to eliminate ambiguity and help address
subjectiveness in the answerer.
• And finally, the duration is sufficiently long enough to eliminate any
short-term preferential biases. Efforts like this can often go for 6 to 12
months with check in points.
30

31. APPLIED DESCRIPTIVE STUDY: CASE STUDY…
• We first solicit volunteer participants from the fire and police departments.
After collecting feedback, we were able to identify 22 police officers and 20 fire
and emergency service members who would like to participate.
• First month is begun, the staff are asked to use their phones as they would with
their normal work phone.
• After one month, questionnaires with preapproved questions were distributed
to the 42 participants.
• The final interview was in person again. The users were asked the same battery
of questions addressing goals 1 and 2, but they were also given the chance to
have open-ended responses.
31

32. APPLIED DESCRIPTIVE STUDY: CASE STUDY…
• Phase two would be an intense case study in mock disaster conditions,
coinciding with a regional exercise to test the secure communications
in disaster-like environments.
• The approach taken makes this a case study, the fact that a specific tool
or method was added to the environment makes it applied research.
32

33. REPORTING YOUR RESULTS
• The final important step of an applied observational study is reporting
your results.
• This is, however, another area where the process for applied
observational research might differ from foundational observational
research.
• Specifically, contract research is conducted on the request of the sponsor
organization, unlike grant-funded research, which is conducted in the
public interest.
• Not to complicate things, any government might use a contract to fund
research in the public interest, but it is important to realize the
difference.
• Any research should have a broad understanding of how their research
is being paid for and what expectations are placed upon the researchers.
33

34. REPORTING YOUR RESULTS…
The final important step of an applied observational study is
reporting your results.
 Title,
 Abstract,
 Introduction,
 Methods,
 Results and discussion,
 Future Work,
 Conclusion,
 Acknowledgments and
 References
34

35. Thank you!!!
35