The document discusses challenges related to software operation knowledge (SOK) integration. It describes how SOK data can be collected from various sources and used to improve software processes. However, challenges exist around visualizing and analyzing large amounts of technical and usage data, aligning business and technical metrics, handling big and real-time data, and addressing errors at different levels of software. The document advocates for continuous refinement of SOK integration objectives and requirements to optimize results.

1. Devnology: Software Operation Knowledge in Practice
Dr. Slinger Jansen
Utrecht University

2. Introductie UU en Software~VOC

3. 09-08-13
open dagen 3
Who am I?: Slinger Jansen
§ Scientist
§ Entrepreneur
§ Advisor
§ Dad
§ Driver of old cars

9. Software Intensive Systems Developers suffer from
§ Increasing quality requirements
§ Decreasing release times
§ Data overflow of feedback data

10. We have an unprecedented opportunity
to run A/B tests with online users and
innovate more quickly based on actual
user response. Microsoft needs to shift
the culture from planning the exact
features to planning a set of possible
features, and letting customers guide us.
- Ray Ozzie

11. Solution: Generic Data Acquisition and Data Mining
§ Generic data acquisition through instrumentation
§ Data gathering without altering the source code (aspect-oriented
development)
§ Microsoft found out through their crash reporting program that 50%
of crashes are caused by 2% of their bugs

14. State of the practice
§ Release sub-process
- Release at times that are convenient to customer: 57%
- Use of formal release planning: 45%
- All stakeholders can access the planning: 85%
- Use of formal release scenario: 56%
- Use of versioned configuration management: 54%
- Explicit management of in-house build tools: 70%
- Use of COTS: 38%
§ Delivery sub-process
- Informing the customer through a website: 84%
- Other channels not used often
- Delivering updates through the web: 85%
- Respondents that use CD & DVD: 40%
- Manual product pull: 92%
- Automatic product push: 12%

15. State of the practice
§ Deployment sub-process
- Use of an update tool: 30%
- Tool is able to update at runtime: 56%
- First time installation failure: 8%
- Possibility of de-installing previous release: 89%
- Undo an update: 59%
- Check of customer configuration: 73%
- Data backup: 59%
§ Usage sub-process
- Use of license agreement: 73%
- Knowledge about the customer platform: 73%
- Use of error reports: 35%
- Use of usage reports: 40%

16. Vision: Continuous Customer Configuration Updating
§ Continuously release software after every change to a release
repository
§ Continuously (be able to) release to customers
§ Continuously (be able to) deploy an update (run-time?)
§ Continuously change your software
§ (You should at least strive for continuous release!)

18. Possibly part of a Software Supply Network
Software Developer
Cust. 1
Cust. m
...
Software Developer
Cust 1
Cust n
...
Operational
Environment 1
Operational
Environment n
Operational
Environment m
...
...
Slinger Jansen and Wilfried Rijsemus: Balancing Total Cost of Ownership and Cost of Maintenance Within a Software
Supply Network , proceedings of the IEEE International Conference on Software Maintenance (ICSM2006,
Industrial track), Philadelphia, PA, USA, September, 2006.

19. Or even an ecosystem!
Do you “own” one?

21. SOK Propagation in Software Ecosystems
§ Relationship management in software ecosystems
– Keystones use various participation incentives to bind niche players to their
ecosystem(s)
– SOK resulting from software compatibility monitoring is input for
relationship management
– Other incentives are used to strenghten relationship
• Developer keys > certification programs > private SDK access

22. SOK Propagation in Software Ecosystems
§ Software quality management
– Vendors use SOK to manage various types of software quality
– Particularly useful in:
• Determining root cause of in-the-field software failures that do not occur at the
software vendor site
• Isolation of Heisenbugs
– A computer bug that disappears or alters its characteristics when an attempt is made to
study it
– EFfective SOK utilization ! quality management success

23. Continuous feedback mechanisms
§ Email
§ SMS
§ Ambient Orbs
§ Windows taskbar monitor
§ Sound
What does your office use?

25. Pragmatic Process Improvement through SOK
§ Introduction
– Many software vendors that acquire SOK, … just acquire it.
• Limited data mining, limited ‘integration’ in software processes
• Unclear who is responsible
– No structural integration with processes and tools in-place
• No protocols or workflows to involve acquired data
• SOK is not utilized to support decisions and steering
– Release management
– Requirements prioritization
– etc.
§ What do you collect, use and report on?
§ What kind of actions are connected to these data?

37. SOK Lifecycle
37
Who acquires data without using it?
And yes, support calls count!

39. Einde Theoretisch Deel

40. SOK Integration Template Method
§ The method is designed to guide vendors in:
– Analysis of integration environment and target processes
– Identification of relevant and valuable operation information
– Integration of selected information in, and adaption of, target processes
– Presentation of integrated operation information
§ We assume that, before using the method:
– Software operation information
– One or more target processes
§ are accessible and available for integration

44. SOK Integration Template Method
§ What do you want to look at immediately when you enter your office
tomorrow morning?
§ OK, but how should we use the method?
– For Stabiplan, we built Errata Garden
• Crash report mining and -analysis tool
• Allows access to in-the-field operation information based on acquired crash reports

45. SOK Integration Template Method
§ OK, but how should we use the method?
– Stabiplan had questions like
• How do we make people use this information?
• How do we prevent productivity to be descreased by EG monitoring?
• How frequently should Errata Garden reports be monitored?
• Who is responsible?
§ How can information provided by Errata Garden be effectively
integrated with our product software processes?

46. SOK Integration Template Method
§ OK, but how should we use the method?
– Template method was instantiated with three processes
• Software maintenance
• Software product management
• Customer support
– Action research
• Researchers present at the software vendor site
• Observing phenomena while adhering to CAR principles
– Researcher-Client Agreement, Learning through Reflection, etc.
Davison, R.M., Martinsons, M.G., Kock, N.: Principles of canonical action research. Information Systems Journal 14, 65–86
(January 2004)

48. Software
maintenance
Software product
management
Customer support
Developers not used to
involve reports during
maintenance
PMs rely on customer
support and salesmen
for release planning
Supporters rely on
customer explanation
for softw. operation
insight
Accelerated software
maintenance
Directed software
product (release)
management
Increased customer
intimacy
Independent requests
from developers
Evaluation of EG tool Short talks
Error report meta
information (status)
Aggregated error report
properties, operation
trends
Operation information
enabling particular
identification of
customers
Activity and deliverable
descriptions

49. Software
maintenance
Software product
management
Customer support
Maintainers were
selected by us and
appointed by CEO
PMs were selected by
us and appointed by
CEO
Customer supporters
were selected by us
and appointed by
CEO
Analysis of
maintenance process:
at least once a week
Analysis of SPM
process: at least
every Scrum sprint
Analysis of customer
support process:
possibly every call
EG list view with error
report status
EG aggregated error
report view
EG customer-spectific
error report views
Creating and
introducing error
report labeling
Creating and
introducing error
report aggregation;
organizing meetings
Extending error report
format and EG
analysis functionality
Activity and deliverable
descriptions

50. Software
maintenance
Software product
management
Customer support
EG extension
EG extension,
harvest meeting
introduction for
frequent analysis of
error reports
EG extension
Evaluative talks Evaluative talks Evaluative talks
Activity and deliverable
descriptions

51. SOK Integration Template Method
§ Main result: Increased software quality
– Report submissions increased about linearly from February, 2009
until June, 2010
– Trend was broken in September, 2010, while a major new version was released

52. SOK Integration Template Method
§ Based on observations and interviews, lessons learned are identified
– What should a vendor do to effectively involve such information in its
product software processes?
– What should it absolutely not do? What are considered pitfalls?
– Observing integration in retrospect, what are your lessons learned?

53. SOK Integration Template Method: Lessons Learned
§ 1. Integration Processes Should be Lean
– Handle additional (administrative) tasks pragmatically:
additional administration may negate the time gained by integration of
operation information

54. SOK Integration Template Method: Lessons Learned
§ 2. Integration Responsibilities and Results Should Be Evangelized
– Shared responsibility is no responsibility
– Increase awareness and acceptance of new way of working
• Both among employees as at management level!
Who is your evangelist?

55. SOK Integration Template Method: Lessons Learned
§ 3. SOK Integration Opens Up Black Boxes
– Yes, informed development, software failure fix prioritization, etc.
– But: improvement areas that were unnoticed before, are highlighted as
such after SOK integration
• Increased developer awareness (software quality, end-user situation)
• More customer-central, pro-active development mentality
– `Build what the customer will use, before the customer asked for it'
• Clearer and quicker interdepartemental communication
• No more HiPPOs!

56. SOK Integration Template Method: Lessons Learned
§ 4. Continuous Refinement of SOK Integration Objectives and
Requirements Leads to Optimization of Integration Results
– Integration results depend on integration objectives and requirements
– Both should be evaluated and refined continuously to correspond to your
product strategy, as well as the needs of your customers

63. Challenges
§ Visualization
§ Alignment business vs technical metrics
§ Big data (or just real time data) & data mining
§ Errors at different levels

64. Data Mining and Visualization

65. Business Metrics vs…
65
What’s your company’s mission statement?

66. Technical metrics
66
Performance
Metrics
Memory
Nonpaged System Memory Size
Paged Memory Size
Paged System Memory Size
Peak Paged Memory Size
Peak Virtual Memory Size
Peak Working Set
Private Memory Size
Virtual Memory Size
Working Set
Processor
Privileged Processor Time
Total Processor Time
User Processor Time
Network
Interface
Bytes Received/sec
Bytes Sent/sec
Packets Outbound Errors
Packets Received Errors
Total Bytes Received
Total Bytes Sent
Disk
% Disk Time
% Free Space
Avg. Disk Queue Length
Avg. Disk sec/Transfer
Disk Reads/sec
Disk Transfers/sec
Disk Writes/sec
Free Megabytes
Web Tier
Page Type Count
Query String Count
Total Request Duration
Authenticate Request Duration
Authorize Request Duration
Resolve Request Cache Duration
Acquire Request State Duration
Request Handler Execute Duration
Release Request State Duration
Update Request Cache Duration
Module Type Count
Method Type Count
Usage Metrics
User Details
Customer ID
IP/Hostname
Application
Details
Web site vs web application
Hosting
User Actions
Actions Performed
Page/function/method access
Timeframe
Entry/Exit point
Session ID
Errors

67. Tata Steel produceert, bewerkt
en distribueert hoogwaardig
staal voor producten die het
leven vergemakkelijken.

68. Belastingdienst: Geld innen voor de BV NL

69. Aviva Solutions: In eerste plaats biedt Aviva Solutions u
goede ICT-oplossingen door het bouwen van applicaties. Maar
buiten zorgen voor continuïteit en vernieuwen van uw
applicaties en systemen, helpt Aviva Solutions u graag bij
andere ICT-vragen. Als dé Microsoft Platform Specialist
zijn we immers expert op het gebied van: Integratie, Business
Intelligence, Custom Solutions & Ontwikkelstraten,
Information Worker en Internet & Commerce.

70. AFAS Software automatiseert bedrijven en organisaties uit
alle branches met complete en moderne (online)
bedrijfssoftware.

71. So… How to get started?
§ Assess most important KPIs
§ Collect them (tons of tools available)
§ Analyse them (tons of tools available)
§ Report them (…)
§ ACT!

73. ?