Jogging While Driving, and Other Software Engineering Research Problems (invi...David Rosenblum
invited talk presented for the Distinguished Lecturer Series of the Department of Computer Science at the University of Illinois at Chicago, 10 April 2014
invited talk presented for the Distinguished Speaker Series of the Institute for Software Research (ISR) at the University of California, Irvine, 5 April 2013
Jogging While Driving, and Other Software Engineering Research Problems (invi...David Rosenblum
invited talk presented for the Distinguished Lecturer Series of the Department of Computer Science at the University of Illinois at Chicago, 10 April 2014
invited talk presented for the Distinguished Speaker Series of the Institute for Software Research (ISR) at the University of California, Irvine, 5 April 2013
Continuous Automated Testing - Cast conference workshop august 2014Noah Sussman
CAST 2014 New York: The Art and Science of Testing
The Association for Software Testing www.associationforsoftwaretesting.org
COURSE DESCRIPTION
Automated tools provide test professionals with the capability to make relevant observations even in the fastest-paced environments. Automated testing is also a powerful tool for improving communication between software engineers. This is important because good communication is a prerequisite for growing a great software engineering organization.
This workshop will explore the continuous testing of software systems. Special focus will be given to the situation where the engineering team is deploying code to production so frequently that it is not possible to perform deep regression testing before each release.
People who participate in this course will learn pragmatic automated testing strategies like:
* Data analysis on the command line with find, grep and wc.
* Network analysis with Chrome Inspector, Charles and netcat.
* Using code churn to predict hotspots where bugs may occur.
* Putting stack traces in context with automated SCM blame emails.
* Using statsd to instrument a whole application.
* Testing in production.
* Monitoring-as-testing.
Technical level: participants should have some familiarity with the command line and with editing code using a text editor or IDE. Familiarity with Git, SVN or another version control system is helpful but not required. Likewise some knowledge of Web servers is helpful but not required. It is desirable for participants to bring laptops.
BIO
From 2010 to 2012 Noah was a Test Architect at Etsy. He helped build Etsy's continuous integration system, and has helped countless other engineers develop successful automated testing strategies.These days Noah is an independent consultant in New York. He is passionate about helping engineers understand and use automated tools as they work to scale their applications more effectively.
Professor Steve Roberts; The Bayesian Crowd: scalable information combinati...Ian Morgan
Professor Steve Roberts, Machine learning research group and Oxford-Man Institute + Alan Turing Institute. Steve gave this talk on the 24th January at the London Bayes Nets meetup.
What does it take to have high impact in software engineering research? Andreas Zeller, a "high impact" SE researcher, shares his personal story and perspective.
Renan Ranelli, especialista em desenvolvimento de software na locaweb, Nessa palestra apresenta os princípios de Random Testing.
Assista em: https://www.eventials.com/locaweb/random-testing/
Random Testing é uma tecnica de teste de software bem menos usual, mas que resolve várias limitações das abordagens clássicas. Em especial, é bastante eficiente para encontrar comportamentos indefinidos, problemas de validação e segurança. Em especial, a palestra é bastante focada em "Property Based Testing" e apresenta quais são os aspectos fundamentais para aplicar as técnicas e as limitações destas.
Computational Reproducibility vs. Transparency: Is It FAIR Enough?Bertram Ludäscher
Keynote at CLIR Workshop (Webinar): Torward Open, Reproducible, and Reusable Research. February 10, 2021. https://reusableresearch.com/
ABSTRACT. The “reproducibility crisis” has resulted in much interest in methods and tools to improve computational reproducibility. FAIR data principles (data should be findable, accessible, interoperable, and reusable) are also being adapted and evolved to apply to other artifacts, notably computational analyses (scientific workflows, Jupyter notebooks, etc.). The current focus on computational reproducibility of scripts and other computational workflows sometimes overshadows a somewhat neglected and arguably more important issue: transparency of data analysis, including data wrangling and cleaning. In this talk I will ask the question: What information is gained by conducting a reproducibility experiment? This leads to a simple model (PRIMAD) that aims to answer this question by sorting out different scenarios. Finally, I will present some features of Whole-Tale, a computational platform for reproducible and transparent computational experiments.
Secure Because Math: A Deep-Dive on Machine Learning-Based Monitoring (#Secur...Alex Pinto
We could all have predicted this with our magical Big Data analytics platforms, but it seems that Machine Learning is the new hotness in Information Security. A great number of startups with ‘cy’ and ‘threat’ in their names that claim that their product will defend or detect more effectively than their neighbour's product "because math". And it should be easy to fool people without a PhD or two that math just works.
Indeed, math is powerful and large scale machine learning is an important cornerstone of much of the systems that we use today. However, not all algorithms and techniques are born equal. Machine Learning is a most powerful tool box, but not every tool can be applied to every problem and that’s where the pitfalls lie.
This presentation will describe the different techniques available for data analysis and machine learning for information security, and discuss their strengths and caveats. The Ghost of Marketing Past will also show how similar the unfulfilled promises of deterministic and exploratory analysis were, and how to avoid making the same mistakes again.
Finally, the presentation will describe the techniques and feature sets that were developed by the presenter on the past year as a part of his ongoing research project on the subject, in particular present some interesting results obtained since the last presentation on DefCon 21, and some ideas that could improve the application of machine learning for use in information security, especially in its use as a helper for security analysts in incident detection and response.
At my talk "On Impact in Software Engineering Research", I present a number of lessons from (and for!) high-impact research:
* Work on a real problem
* Assume as little as possible
* Keep things simple
* Have a sound model
* Keep on learning
* Keep on moving
* Build prototypes
Video at https://youtu.be/md4Fp3Pro0o
Andreas Zeller is faculty at the CISPA Helmholtz Center for Information Security and professor for Software Engineering at Saarland University, both in Saarbrücken, Germany. His research on automated debugging, mining software archives, specification mining, and security testing has won several awards for its impact in academia and industry. Zeller is an ACM Fellow, an IFIP Fellow, an ERC Advanced Grant Awardee, and holds an ACM SIGSOFT Outstanding Research Award.
IaaS Cloud Benchmarking: Approaches, Challenges, and ExperienceAlexandru Iosup
IaaS Cloud Benchmarking: Approaches, Challenges, and Experience
Impact Award lecture at MTAGS/ACM SC'12. URL: http://datasys.cs.iit.edu/events/MTAGS12/biggest-impact-award.html
Andreas Zeller, On Impact in Software Engineering Research
Abstract: After 25 years in research, I am happy to have had some impact in Software Engineering Research - some of it on purpose, some of it accidentally. In this talk, I summarize six important lessons I have learned: Work on a real problem, assume as little as possible, keep things simple, have a sound model, keep on learning, and keep on moving.
Bio: Andreas Zeller is a full professor for Software Engineering at Saarland University in Saarbrücken, Germany, since 2001. His research concerns the analysis of large software systems and their development process. In 2010, Zeller was inducted as Fellow of the ACM for his contributions to automated debugging and mining software archives, for which he also was awarded 10-year impact awards from ACM SIGSOFT and ICSE. In 2011, he received an ERC Advanced Grant, Europe's highest and most prestigious individual research grant, for work on specification mining and test case generation. In 2013, Zeller co-founded Testfabrik AG, a start-up on automatic testing of Web applications, where he chairs the supervisory board. In 2018, he received the Outstanding Research Award from ACM SiGSOFT.
How do you make your research impactful? How do you get towards your book, your tool, your algorithm? Andreas Zeller shares lessons learned in his career.
MINIMIZING LOCALIZATION ERROR AND ENSURE SECURITY OF DVHOP APPROACHijceronline
In case of wireless sensor network there exist problem of determining the nodes which are symmetrical to each other. The nodes which are symmetrical and are at lesser distance are selected for data transfer. This identification of the distance between nodes is known as localization. In the proposed paper work on DVHOP is done. The DVHOP is the distance vector routing based protocol which is used to indicate whether there exist a path from source to destination or not. The malicious node can also be present which can take over the actual node causing problems in the transfer process. The most common attack which results from this will be DDOS attack. This will result in the duplication of the information and will cause traffic jamming. DVHOP with random key is used to handle DDOS attack.
Science Gateways – Leveraging Modeling and Simulations in HPC Infrastructure...Sandra Gesing
A tutorial on science gateways at the cHiPSet Training School – New Trends in Modeling and Simulation in HPC Systems, 21-23 September 2016 in Bucharest, Romania.
Known Unknowns: Testing in the Presence of Uncertainty (talk at ACM SIGSOFT F...David Rosenblum
talk presented in the Visions & Challenges Track of the ACM SIGSOFT 22nd International Symposium on the Foundations of Software Engineering (FSE 2014), Hong Kong, 20 November 2014; the paper won 2nd Prize in the track
SIGSOFT Impact Award: Reflections and Prospects (invited talk at SIGSOFT FSE ...David Rosenblum
Invited talk with Alexander L. Wolf upon receiving the first ACM SIGSOFT Impact Paper Award, at the 16th ACM SIGSOFT International Symposium on the Foundations of Software Engineering (ACM SIGSOFT FSE), 13 November 2008.
More Related Content
Similar to Probability and Uncertainty in Software Engineering (keynote talk at NASAC 2013)
Continuous Automated Testing - Cast conference workshop august 2014Noah Sussman
CAST 2014 New York: The Art and Science of Testing
The Association for Software Testing www.associationforsoftwaretesting.org
COURSE DESCRIPTION
Automated tools provide test professionals with the capability to make relevant observations even in the fastest-paced environments. Automated testing is also a powerful tool for improving communication between software engineers. This is important because good communication is a prerequisite for growing a great software engineering organization.
This workshop will explore the continuous testing of software systems. Special focus will be given to the situation where the engineering team is deploying code to production so frequently that it is not possible to perform deep regression testing before each release.
People who participate in this course will learn pragmatic automated testing strategies like:
* Data analysis on the command line with find, grep and wc.
* Network analysis with Chrome Inspector, Charles and netcat.
* Using code churn to predict hotspots where bugs may occur.
* Putting stack traces in context with automated SCM blame emails.
* Using statsd to instrument a whole application.
* Testing in production.
* Monitoring-as-testing.
Technical level: participants should have some familiarity with the command line and with editing code using a text editor or IDE. Familiarity with Git, SVN or another version control system is helpful but not required. Likewise some knowledge of Web servers is helpful but not required. It is desirable for participants to bring laptops.
BIO
From 2010 to 2012 Noah was a Test Architect at Etsy. He helped build Etsy's continuous integration system, and has helped countless other engineers develop successful automated testing strategies.These days Noah is an independent consultant in New York. He is passionate about helping engineers understand and use automated tools as they work to scale their applications more effectively.
Professor Steve Roberts; The Bayesian Crowd: scalable information combinati...Ian Morgan
Professor Steve Roberts, Machine learning research group and Oxford-Man Institute + Alan Turing Institute. Steve gave this talk on the 24th January at the London Bayes Nets meetup.
What does it take to have high impact in software engineering research? Andreas Zeller, a "high impact" SE researcher, shares his personal story and perspective.
Renan Ranelli, especialista em desenvolvimento de software na locaweb, Nessa palestra apresenta os princípios de Random Testing.
Assista em: https://www.eventials.com/locaweb/random-testing/
Random Testing é uma tecnica de teste de software bem menos usual, mas que resolve várias limitações das abordagens clássicas. Em especial, é bastante eficiente para encontrar comportamentos indefinidos, problemas de validação e segurança. Em especial, a palestra é bastante focada em "Property Based Testing" e apresenta quais são os aspectos fundamentais para aplicar as técnicas e as limitações destas.
Computational Reproducibility vs. Transparency: Is It FAIR Enough?Bertram Ludäscher
Keynote at CLIR Workshop (Webinar): Torward Open, Reproducible, and Reusable Research. February 10, 2021. https://reusableresearch.com/
ABSTRACT. The “reproducibility crisis” has resulted in much interest in methods and tools to improve computational reproducibility. FAIR data principles (data should be findable, accessible, interoperable, and reusable) are also being adapted and evolved to apply to other artifacts, notably computational analyses (scientific workflows, Jupyter notebooks, etc.). The current focus on computational reproducibility of scripts and other computational workflows sometimes overshadows a somewhat neglected and arguably more important issue: transparency of data analysis, including data wrangling and cleaning. In this talk I will ask the question: What information is gained by conducting a reproducibility experiment? This leads to a simple model (PRIMAD) that aims to answer this question by sorting out different scenarios. Finally, I will present some features of Whole-Tale, a computational platform for reproducible and transparent computational experiments.
Secure Because Math: A Deep-Dive on Machine Learning-Based Monitoring (#Secur...Alex Pinto
We could all have predicted this with our magical Big Data analytics platforms, but it seems that Machine Learning is the new hotness in Information Security. A great number of startups with ‘cy’ and ‘threat’ in their names that claim that their product will defend or detect more effectively than their neighbour's product "because math". And it should be easy to fool people without a PhD or two that math just works.
Indeed, math is powerful and large scale machine learning is an important cornerstone of much of the systems that we use today. However, not all algorithms and techniques are born equal. Machine Learning is a most powerful tool box, but not every tool can be applied to every problem and that’s where the pitfalls lie.
This presentation will describe the different techniques available for data analysis and machine learning for information security, and discuss their strengths and caveats. The Ghost of Marketing Past will also show how similar the unfulfilled promises of deterministic and exploratory analysis were, and how to avoid making the same mistakes again.
Finally, the presentation will describe the techniques and feature sets that were developed by the presenter on the past year as a part of his ongoing research project on the subject, in particular present some interesting results obtained since the last presentation on DefCon 21, and some ideas that could improve the application of machine learning for use in information security, especially in its use as a helper for security analysts in incident detection and response.
At my talk "On Impact in Software Engineering Research", I present a number of lessons from (and for!) high-impact research:
* Work on a real problem
* Assume as little as possible
* Keep things simple
* Have a sound model
* Keep on learning
* Keep on moving
* Build prototypes
Video at https://youtu.be/md4Fp3Pro0o
Andreas Zeller is faculty at the CISPA Helmholtz Center for Information Security and professor for Software Engineering at Saarland University, both in Saarbrücken, Germany. His research on automated debugging, mining software archives, specification mining, and security testing has won several awards for its impact in academia and industry. Zeller is an ACM Fellow, an IFIP Fellow, an ERC Advanced Grant Awardee, and holds an ACM SIGSOFT Outstanding Research Award.
IaaS Cloud Benchmarking: Approaches, Challenges, and ExperienceAlexandru Iosup
IaaS Cloud Benchmarking: Approaches, Challenges, and Experience
Impact Award lecture at MTAGS/ACM SC'12. URL: http://datasys.cs.iit.edu/events/MTAGS12/biggest-impact-award.html
Andreas Zeller, On Impact in Software Engineering Research
Abstract: After 25 years in research, I am happy to have had some impact in Software Engineering Research - some of it on purpose, some of it accidentally. In this talk, I summarize six important lessons I have learned: Work on a real problem, assume as little as possible, keep things simple, have a sound model, keep on learning, and keep on moving.
Bio: Andreas Zeller is a full professor for Software Engineering at Saarland University in Saarbrücken, Germany, since 2001. His research concerns the analysis of large software systems and their development process. In 2010, Zeller was inducted as Fellow of the ACM for his contributions to automated debugging and mining software archives, for which he also was awarded 10-year impact awards from ACM SIGSOFT and ICSE. In 2011, he received an ERC Advanced Grant, Europe's highest and most prestigious individual research grant, for work on specification mining and test case generation. In 2013, Zeller co-founded Testfabrik AG, a start-up on automatic testing of Web applications, where he chairs the supervisory board. In 2018, he received the Outstanding Research Award from ACM SiGSOFT.
How do you make your research impactful? How do you get towards your book, your tool, your algorithm? Andreas Zeller shares lessons learned in his career.
MINIMIZING LOCALIZATION ERROR AND ENSURE SECURITY OF DVHOP APPROACHijceronline
In case of wireless sensor network there exist problem of determining the nodes which are symmetrical to each other. The nodes which are symmetrical and are at lesser distance are selected for data transfer. This identification of the distance between nodes is known as localization. In the proposed paper work on DVHOP is done. The DVHOP is the distance vector routing based protocol which is used to indicate whether there exist a path from source to destination or not. The malicious node can also be present which can take over the actual node causing problems in the transfer process. The most common attack which results from this will be DDOS attack. This will result in the duplication of the information and will cause traffic jamming. DVHOP with random key is used to handle DDOS attack.
Science Gateways – Leveraging Modeling and Simulations in HPC Infrastructure...Sandra Gesing
A tutorial on science gateways at the cHiPSet Training School – New Trends in Modeling and Simulation in HPC Systems, 21-23 September 2016 in Bucharest, Romania.
Known Unknowns: Testing in the Presence of Uncertainty (talk at ACM SIGSOFT F...David Rosenblum
talk presented in the Visions & Challenges Track of the ACM SIGSOFT 22nd International Symposium on the Foundations of Software Engineering (FSE 2014), Hong Kong, 20 November 2014; the paper won 2nd Prize in the track
SIGSOFT Impact Award: Reflections and Prospects (invited talk at SIGSOFT FSE ...David Rosenblum
Invited talk with Alexander L. Wolf upon receiving the first ACM SIGSOFT Impact Paper Award, at the 16th ACM SIGSOFT International Symposium on the Foundations of Software Engineering (ACM SIGSOFT FSE), 13 November 2008.
Assertions a Decade Later (invited talk at ICSE 2002)David Rosenblum
Invited talk upon receiving the 2002 ICSE Most Influential Paper Award for ICSE 1992, at the 24th International Conference on Software Engineering (ICSE 2002), 22 May 2002.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Probability and Uncertainty in Software Engineering (keynote talk at NASAC 2013)
1. NASAC 2013,Tianjin, 9 November 2013
Probability and Uncertainty
in Software Engineering
David S. Rosenblum!
Dean, School of Computing!
National University of Singapore
2. NASAC 2013,Tianjin, 9 November 2013
Software Engineering
at NUS
Hugh
Anderson
Chin
Wei Ngan
Dong
Jin Song
Aquinas
Hobor
Joxan!
Jaffar
Stan
Jarzabek
Khoo
Siau Cheng
Damith
Rajapakse
David!
Rosenblum
Abhik
Roychoudhury
Bimlesh
Wadhwa
Yap
Hock Chuan,
Roland
3. NASAC 2013,Tianjin, 9 November 2013
Certainty in
Software Engineering
Engineering of software is centered around
simplistic,“yes/no” characterizations of artifacts
4. NASAC 2013,Tianjin, 9 November 2013
Certainty in
Software Engineering
Engineering of software is centered around
simplistic,“yes/no” characterizations of artifacts
Program is correct/incorrect
Program execution finished/crashed
Compilation completed/aborted
Test suite succeeded/failed
Specification is satisfied/violated
5. NASAC 2013,Tianjin, 9 November 2013
Example!
Model Checking
! ¬p → ◊q( )∧"( )
Model
Checker
✓
✕
State Machine!
Model
Temporal
Property
Results
Counterexample!
Trace
System
Requirements
6. NASAC 2013,Tianjin, 9 November 2013
Example!
Model Checking
! ¬p → ◊q( )∧"( )
Model
Checker
✕
State Machine!
Model
Temporal
Property
Results
Counterexample!
Trace
System
Requirements
7. NASAC 2013,Tianjin, 9 November 2013
Uncertainty in
Software Engineering
✓Nondeterminism
✓Randomized Algorithms
✓“Good Enough Software”
✓Test Coverage Metrics
8. NASAC 2013,Tianjin, 9 November 2013
Uncertainty in
Software Engineering
✓Nondeterminism
✓Randomized Algorithms
✓“Good Enough Software”
✓Test Coverage Metrics
Probabilistic Modeling and Analysis
9. NASAC 2013,Tianjin, 9 November 2013
Probabilistic
Model Checking
! ¬p → ◊q( )∧"( )
Model
Checker
✓
✕
State Machine!
Model
Temporal
Property
Results
Counterexample!
Trace
System
Requirements
P≥0.95 [ ]
0.4
0.6
Probabilistic
Probabilistic
10. NASAC 2013,Tianjin, 9 November 2013
Probabilistic
Model Checking
! ¬p → ◊q( )∧"( )
Model
Checker
✓
✕
State Machine!
Model
Temporal
Property
Results
Counterexample!
Trace
System
Requirements
P=? [ ]
0.4
0.6
Quantitative Results
0.9732Probabilistic
Probabilistic
11. NASAC 2013,Tianjin, 9 November 2013
Example
Die Tossing Simulated by Coin Flipping
Knuth-Yao algorithm,
from the PRISM group
(Kwiatkowska et al.)
0
3
2
1
6
4
5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
12. NASAC 2013,Tianjin, 9 November 2013
Example
Die Tossing Simulated by Coin Flipping
Knuth-Yao algorithm,
from the PRISM group
(Kwiatkowska et al.)
The behavior is governed by a!
theoretical probability distribution
0
3
2
1
6
4
5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
13. NASAC 2013,Tianjin, 9 November 2013
Probabilistic
Model Checking
! ¬p → ◊q( )∧"( )
Model
Checker
✓
State Machine!
Model
Temporal
Property
Results
Counterexample!
Trace
System
Requirements
P≥0.95 [ ]
0.4
0.6
Quantitative Results
0.9732Probabilistic
Probabilistic
14. NASAC 2013,Tianjin, 9 November 2013
Probabilistic
Model Checking
! ¬p → ◊q( )∧"( )
Model
Checker
✕
State Machine!
Model
Temporal
Property
Results
Counterexample!
Trace
System
Requirements
P≥0.95 [ ]
Quantitative Results
Probabilistic
Probabilistic
0.41
0.59
0.6211
15. NASAC 2013,Tianjin, 9 November 2013
Example!
Zeroconf Protocol
s1s0 s2 s3
q
1
1
{ok} {error}
{start} s4
s5
s6
s7
s8
1
1-q
1-p
1-p
1-p
1-p
p p p
p
1
from the PRISM group
(Kwiatkowska et al.)
16. NASAC 2013,Tianjin, 9 November 2013
Example!
Zeroconf Protocol
s1s0 s2 s3
q
1
1
{ok} {error}
{start} s4
s5
s6
s7
s8
1
1-q
1-p
1-p
1-p
1-p
p p p
p
1
The behavior is governed by an!
empirically estimated probability distribution
from the PRISM group
(Kwiatkowska et al.)
packet-loss rate
17. NASAC 2013,Tianjin, 9 November 2013
Perturbed Probabilistic Systems!
(Current Research)
• Starting Points!
✓Discrete-Time Markov Chains (DTMCs)!
✓… with one or more probability parameters!
✓… verified against reachability properties:
S? ∪ S!
Guoxin Su and David S. Rosenblum,
“Asymptotic Bounds for QuantitativeVerification of Perturbed Probabilistic Systems”,
Proc. ICFEM 2013
18. NASAC 2013,Tianjin, 9 November 2013
Parametric
Markov Chains
• A distribution parameter in a DTMC is represented as a
vector x of parameters xi!
• The norm of total variance represents the amount of
perturbation:!
!
• The parameter is allowed a “sufficiently small”
perturbation with respect to ideal reference values r:!
!
• Can generalize to multiple parameters
v = vi∑
x − r ≤ Δ
19. NASAC 2013,Tianjin, 9 November 2013
Perturbation Bounds
• Perturbation Function!
!
where A is the transition probability sub-matrix for S?
and b is the vector of one-step probabilities from S? to S!
!
• Condition Numbers!
!
ρ x( )= ι? i A x
i
i b x( )− Ai
i b( )( )i=0
∞
∑
κ = lim
δ→0
sup
ρ(x − r)
δ
: x − r ≤ δ,δ > 0
⎧
⎨
⎩
⎫
⎬
⎭
21. NASAC 2013,Tianjin, 9 November 2013
Additional Aspects
• Models
✓Markov Decision Processes (MDPs)!
✓Continuous-Time Markov Chains (CMTCs)
• Verification
✓LTL Model Checking!
using Deterministic Rabin Automata!
✓PCTL Model Checking!
with singular perturbations due to nested P[ ] operators!
✓Reward Properties!
✓Alternative Norms and Bounds!
Kullback-Leibler Divergence, Quadratic Bounds
22. NASAC 2013,Tianjin, 9 November 2013
Other Forms of
Uncertainty
“There are known knowns; there are things we know
we know. We also know there are known unknowns;
that is to say, we know there are some things we do
not know. But there are also unknown unknowns –
the ones we don’t know we don’t know.”!
!
— Donald Rumsfeld
23. NASAC 2013,Tianjin, 9 November 2013
Uncertainty in Testing!
(New Research)
1982: Weyuker: Non-Testable Programs!
- Impossible/too costly to efficiently check results!
- Example: mathematical software!
2010: Garlan: Intrinsic Uncertainty!
- Systems embody intrinsic uncertainty/imprecision!
- Cannot easily distinguish bugs from “features”!
- Example: ubiquitous computing
25. NASAC 2013,Tianjin, 9 November 2013
Example!
Google Latitude
When is an
incorrect location!
a bug, and when
is it a “feature”?
~ 500m
~ 50m
~ 2m
26. NASAC 2013,Tianjin, 9 November 2013
Example!
Google Latitude
When is an
incorrect location!
a bug, and when
is it a “feature”?
And how do!
you know?
~ 500m
~ 50m
~ 2m
28. NASAC 2013,Tianjin, 9 November 2013
Example!
Affective Computing
When is an!
incorrect!
classification a bug,!
and when is it a!
“feature”?
29. NASAC 2013,Tianjin, 9 November 2013
Example!
Affective Computing
When is an!
incorrect!
classification a bug,!
and when is it a!
“feature”?
And how do!
you know?
30. NASAC 2013,Tianjin, 9 November 2013
Sources of
Uncertainty
✓Output: results, characteristics of results!
✓Sensors: redundancy, reliability, resolution!
✓Context: sensing, inferring, fusing!
✓Machine learning: imprecision, user training
31. NASAC 2013,Tianjin, 9 November 2013
Sources of
Uncertainty
✓Output: results, characteristics of results!
✓Sensors: redundancy, reliability, resolution!
✓Context: sensing, inferring, fusing!
✓Machine learning: imprecision, user training
These create significant challenges for
software engineering research and practice!
32. NASAC 2013,Tianjin, 9 November 2013
Conclusion
✓Software engineering (certainly) suffers
from excessive certainty!
✓A probabilistic mindset offers greater insight!
✓But significant challenges remain for
probabilistic verification!
✓And other forms of uncertainty are equally
challenging to address
33. NASAC 2013,Tianjin, 9 November 2013
Probability and Uncertainty
in Software Engineering
David S. Rosenblum!
Dean, School of Computing!
National University of Singapore
ThankYou!