Successfully reported this slideshow.
Your SlideShare is downloading. ×

Python metaprogramming in linear time language for automated runtime verification with graph neural networks

Ad

Python metaprogramming
in linear time language
for automated runtime verification
with graph neural networks
Diploma thesi...

Ad

In other words…
Lovpy is a runtime logic verification library for Python.
2
Lovpy
Logic verification becomes mainstream!
L...

Ad

What is logic verification?
 Runtime verification approach at its heart.
► E.g. a deadlock can be detected only at runtim...

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Loading in …3
×

Check these out next

1 of 28 Ad
1 of 28 Ad

Python metaprogramming in linear time language for automated runtime verification with graph neural networks

Download to read offline

The term runtime logic verification defines a field that ranges from software verification for compliance with a set of specifications to assuring the adoption of good coding practices. Under this scope, we created lovpy, a novel metaprogramming library for python, that introduces to its ecosystem the capabilities of runtime logic verification. Definition of expected behavior is performed using the intuitive specifications language Gherkin, while using the library requires no code modifications. For its implementation we utilized a broad set of tools, ranging from the domains of graph theory, formal languages theory and temporal logic to deep learning, with specific focus on graph neural networks. We also, provided the mathematical foundation for a new type of graph, designed for representing temporal specifications. Based on it, we defined a set of mathematically proved logic algorithms. Then, we used these structures for implementing a novel theorem proving system, located at the heart of lovpy and ensuring the absolute validity of reported violations. We evaluated five different proving architectures, consisting from heuristics and simple neural models, to deep graph neural networks. For the training of neural systems, we implemented a mechanism for generating synthetic theorems, utilizing a series of mathematical properties. Finally, we used lovpy for detecting bugs in two popular open-source libraries, Django and Keras.

The term runtime logic verification defines a field that ranges from software verification for compliance with a set of specifications to assuring the adoption of good coding practices. Under this scope, we created lovpy, a novel metaprogramming library for python, that introduces to its ecosystem the capabilities of runtime logic verification. Definition of expected behavior is performed using the intuitive specifications language Gherkin, while using the library requires no code modifications. For its implementation we utilized a broad set of tools, ranging from the domains of graph theory, formal languages theory and temporal logic to deep learning, with specific focus on graph neural networks. We also, provided the mathematical foundation for a new type of graph, designed for representing temporal specifications. Based on it, we defined a set of mathematically proved logic algorithms. Then, we used these structures for implementing a novel theorem proving system, located at the heart of lovpy and ensuring the absolute validity of reported violations. We evaluated five different proving architectures, consisting from heuristics and simple neural models, to deep graph neural networks. For the training of neural systems, we implemented a mechanism for generating synthetic theorems, utilizing a series of mathematical properties. Finally, we used lovpy for detecting bugs in two popular open-source libraries, Django and Keras.

Advertisement
Advertisement

More Related Content

More from ISSEL

Advertisement

Python metaprogramming in linear time language for automated runtime verification with graph neural networks

  1. 1. Python metaprogramming in linear time language for automated runtime verification with graph neural networks Diploma thesis by: Dimitrios Karageorgiou (SRN: 8420) soulrain@outlook.com Supervisors: • Andreas Symeonidis (Associate Professor) • Emmanouil Krasanakis (PhD Candidate) Faculty of Engineering School of Electrical And Computers Engineering Department of Electronics and Computer Engineering Intelligent Systems and Software Engineering Labgroup Friday, 12th November 2021
  2. 2. In other words… Lovpy is a runtime logic verification library for Python. 2 Lovpy Logic verification becomes mainstream! Lovpy by Dimitrios Karageorgiou
  3. 3. What is logic verification?  Runtime verification approach at its heart. ► E.g. a deadlock can be detected only at runtime.  Extends verification techniques outside of strict software verification domain.  E.g. enforcing best practices to the users of a library is also a logic problem. 3 Instrumentation Python System Monitor Specifications Verdicts Events Specification Violated or Specification Holds e.g. Lovpy by Dimitrios Karageorgiou
  4. 4. Not another runtime verification library...  Only complex and application specific runtime verification libraries existed for Python.  Design goals:  Minimal user effort to enable verification, without required code modifications.  Specifications in an easy-to-learn and intuitive language.  Never report a violation that does not exist (0% false-negatives).  Report violations before they happen (prevent side-effects).  Report the last provably correct line of code (all specifications hold). 4 Lovpy by Dimitrios Karageorgiou
  5. 5. No code modifications required!  Enable verification by just executing the library: python –m lovpy <script.py> 5 Lovpy by Dimitrios Karageorgiou ► Lovpy AOT Preprocessor handles the rest: Python Code Lovpy AOT Preprocessor Augmented Python Code Python Runtime Python Objects Augmented Objects Lovpy Execution
  6. 6. Augmented Python Objects  Each Python object is augmented to hold its execution state: 6 Current System State Augmented Object 1 Object 1 State 1 Augmented Object 2 Object 2 State 2 Augmented Object N Object N State N  Objects can live anywhere (multiple threads, processes etc.)  Parallelization of original system is retained.  Verification on per-object basis. Lovpy by Dimitrios Karageorgiou
  7. 7. Specifications in Gherkin 7 Gherkin is:  Simple  Easy-to-learn  Intuitive 1. SCENARIO: 2. WHEN call acquire 3. THEN SHOULD NOT locked 4. AND locked 5. 6. SCENARIO: 7. GIVEN locked 8. WHEN call release 9. THEN NOT locked Lovpy by Dimitrios Karageorgiou
  8. 8. Everything is mathematically proved!  Monitor utilizes an Automated Theorem Prover: 8 Monitor Current System State Theorems Automated Theorem Prover System State Builder Specifications Parser Properties to Prove Execution Events Specifications Verdicts Lovpy by Dimitrios Karageorgiou
  9. 9. Automated Theorem Proving 9 Lovpy by Dimitrios Karageorgiou
  10. 10. Everything is a Temporal Graph  Execution States, Theorems and Properties are converted to Temporal Graphs.  Temporal Graph: o A kind of Abstract Syntax Graph. o Nodes are either logical operators or predicates. o Edges contain timestamps. o Each timestamp is the most recent moment the subgraph holds. o Timestamps can be relative or absolute. o Definition of mathematically proved logic algorithms (logic graph removal/addition, graph modus ponens, etc.) 10 Lovpy by Dimitrios Karageorgiou
  11. 11. Everything is a Temporal Graph  Execution States, Theorems and Properties are converted to Temporal Graphs. 11  …  lock = threading.Lock()  …  lock.acquire()  …  lock.release()  …  lock.acquire()  … Lovpy by Dimitrios Karageorgiou
  12. 12. Everything is a Temporal Graph  Execution States, Theorems and Properties are converted to Temporal Graphs. 12 1. SCENARIO: 2. GIVEN locked 3. WHEN call release 4. THEN NOT locked Lovpy by Dimitrios Karageorgiou
  13. 13. Everything is a Temporal Graph  Execution States, Theorems and Properties are converted to Timed Graphs. 13 1. SCENARIO: 2. WHEN call acquire 3. THEN SHOULD NOT locked 4. AND locked  Part of conclusion that refers to the same time moment with assumption, is always proved and becomes a theorem. Lovpy by Dimitrios Karageorgiou
  14. 14. Proving process initialization 14 Lovpy by Dimitrios Karageorgiou Property to Prove Execution State Graph
  15. 15. Theorem application #1 15 Lovpy by Dimitrios Karageorgiou Applied Theorem #1 Execution State Graph #1
  16. 16. Theorem application #2 16 Lovpy by Dimitrios Karageorgiou Applied Theorem #2 Execution State Graph #2
  17. 17. Property proved! 17 Lovpy by Dimitrios Karageorgiou Proved property Final Execution State Graph
  18. 18. Violation detected! 18 Lovpy by Dimitrios Karageorgiou ► Last correct line reported too!
  19. 19. Improve theorem proving capability  Deterministic next theorem selection:  Next theorem to apply is the one whose assumption uses the oldest predicates. х Problem: Oldest theorem is not always the appropriate one to apply.  Solution: Deep Learning and Graph Neural Networks ► Next theorem to apply is selected using a deep neural model.  Use Graph Neural Networks to embed graph.  Train model using synthetic theorems generated by:  Lovpy Synthetic Theorems Generator 19 Lovpy by Dimitrios Karageorgiou
  20. 20. Deep Neural Architecture Overview 20 Lovpy by Dimitrios Karageorgiou Concatenation Current State Theorem Instance Goal Property Theorem N Theorem 2 Theorem 1 Score N Score 2 Score 1 Current State Graph Encoder Theorem Instance Graph Encoder Goal Property Graph Encoder Current State Graph Theorem Graph Goal Graph
  21. 21. Neural Graph Encoder 21 Lovpy by Dimitrios Karageorgiou
  22. 22. Evaluation 22  Evaluated five different architectures on proving 2.5k synthetic theorems. Correct Proofs Heuristic 70.7% MLP 59.25 MLP + Heuristic 70.7% GNN 61.6% GNN + Heuristic 73.8% Lovpy by Dimitrios Karageorgiou ► Detected common bugs of 5 different domains, in 20 erroneous python programs.  Detected two bugs in popular open-source projects:  Django Web Framework  Keras
  23. 23. Future possibilities ► Public Lovpy’s Repository:  Community written specifications for specific domains (e.g. best practices for using Tensorflow)  Community trained neural models for theorem selection.  Specifications mining for eliminating the need for hand-written specifications.  Natural language support in Gherkin rules. 23 Lovpy by Dimitrios Karageorgiou
  24. 24. Questions??? 24 Thanks for watching! Lovpy by Dimitrios Karageorgiou
  25. 25. Slides Graveyard 25 Lovpy by Dimitrios Karageorgiou
  26. 26. Lovpy is available for everyone! ► Lovpy is available at PyPI: ► Also available as an open-source project on Github: 26 Lovpy by Dimitrios Karageorgiou python –m pip install lovpy https://github.com/dkarageo/lovpy
  27. 27. Detecting code violations 27  Detected two bugs in popular open-source projects:  Django Web Framework  Keras Violations Detected Threads Data Neural Math Common Total Heuristic 3 1 2 2 6 14 MLP 1 0 1 2 3 7 MLP + Heuristic 3 1 2 2 6 14 GNN 4 1 1 1 3 12 GNN + Heuristic 5 2 3 2 6 18 ► Detected common bugs of 5 different domains, in 20 erroneous python programs. Lovpy by Dimitrios Karageorgiou
  28. 28. Synthetic sample example 28 ► Generated by Lovpy Synthetic Theorems Generator. Lovpy by Dimitrios Karageorgiou

×