This document summarizes a presentation on dynamic data race detection in concurrent Java programs. It discusses what data races are and why they are dangerous. It then covers different approaches to automatic race detection, including static and dynamic methods. The presentation focuses on explaining the happens-before race detection algorithm using vector clocks. It also describes the implementation of the presenters' own dynamic race detector, including how it works, problems they solved, and examples of races it detected in real applications.
Lara Technologies are providing Software Training Division, Java/J2ee, Android, Web Services, Logical Coding, Basics Of C Language, Soft Skills, Aptitude, Etc.
To learn about the basic concepts of Object Oriented Programming and Inheritance plus different Inheritance Models and interview questions will be covered.
Automated Discovery of Deserialization Gadget ChainsPriyanka Aash
Although vulnerabilities stemming from the deserialization of untrusted data have been understood for many years, unsafe deserialization continues to be a vulnerability class that isn't going away. Attention on Java deserialization vulnerabilities skyrocketed in 2015 when Frohoff and Lawrence published an RCE gadget chain in the Apache Commons library and as recently as last year's Black Hat, Muñoz and Miroshis presented a survey of dangerous JSON deserialization libraries. While much research and automated detection technology has so far focused on the discovery of vulnerable entry points (i.e. code that deserializes untrusted data), finding a "gadget chain" to actually make the vulnerability exploitable has thus far been a largely manual exercise. In this talk, I present a new technique for the automated discovery of deserialization gadget chains in Java, allowing defensive teams to quickly identify the significance of a deserialization vulnerability and allowing penetration testers to quickly develop working exploits. At the conclusion, I will also be releasing a FOSS toolkit which utilizes this methodology and has been used to successfully develop many deserialization exploits in both internal applications and open source projects.
Java Serialization is often considered a dark art of Java programmers. This session will lift the veil and show what serialization is and isn't, how you can use it for profit and evil. After this session no NotSerializableException will be unconquerable.
Lara Technologies are providing Software Training Division, Java/J2ee, Android, Web Services, Logical Coding, Basics Of C Language, Soft Skills, Aptitude, Etc.
To learn about the basic concepts of Object Oriented Programming and Inheritance plus different Inheritance Models and interview questions will be covered.
Automated Discovery of Deserialization Gadget ChainsPriyanka Aash
Although vulnerabilities stemming from the deserialization of untrusted data have been understood for many years, unsafe deserialization continues to be a vulnerability class that isn't going away. Attention on Java deserialization vulnerabilities skyrocketed in 2015 when Frohoff and Lawrence published an RCE gadget chain in the Apache Commons library and as recently as last year's Black Hat, Muñoz and Miroshis presented a survey of dangerous JSON deserialization libraries. While much research and automated detection technology has so far focused on the discovery of vulnerable entry points (i.e. code that deserializes untrusted data), finding a "gadget chain" to actually make the vulnerability exploitable has thus far been a largely manual exercise. In this talk, I present a new technique for the automated discovery of deserialization gadget chains in Java, allowing defensive teams to quickly identify the significance of a deserialization vulnerability and allowing penetration testers to quickly develop working exploits. At the conclusion, I will also be releasing a FOSS toolkit which utilizes this methodology and has been used to successfully develop many deserialization exploits in both internal applications and open source projects.
Java Serialization is often considered a dark art of Java programmers. This session will lift the veil and show what serialization is and isn't, how you can use it for profit and evil. After this session no NotSerializableException will be unconquerable.
This is my attempt to compose a brief and cursory introduction to concepts in Java programming language. I call it Java Quicky.
I plan to extend and enhance it over time.
This presentation describes key concepts in Java. I call it The Java Quicky.
This is part of a series of presentations to cover the Java programming language and its new offerings and versions in depth.
Static analysis: Around Java in 60 minutesAndrey Karpov
Theory
Code quality (bugs, vulnerabilities)
Methodologies of code protection against defects
Code Review
Static analysis and everything related to it
Tools
Existing tools of static analysis
SonarQube
PVS-Studio for Java what is it?
Several detected examples of code with defects
More about static analysis
Conclusions
Java/Scala Lab 2016. Григорий Кравцов: Реализация и тестирование DAO слоя с н...GeeksLab Odessa
16.4.16 Java/Scala Lab
Upcoming events: goo.gl/I2gJ4H
В докладе будет расммотрена идеология использования неявных классов в Scala, рассмотрен пример построения DAO слоя с поддержкой нативной нотации MongoShell, показана лекость организации тестирования при таком подходе на примере DAO слоя взаимодействия с MongoDB через ReactiveMongo драйвер. В заключение будут представлены положительные и отрицательные стороны предложенного подхода
A Runtime Monitoring Framework for Event Streams with Non-Primitive ArgumentsSylvain Hallé
A runtime monitor is a tool that takes as input a model of some system, and observes in real time that the sequence of events produced by a run of that system follows the specification. While existing monitoring solutions generally use finite-state machines and temporal logic as their model language, the specification is ultimately tangled with hand-written, implementation-specific details which severely limit their range of application. We present a runtime monitoring platform that clearly separates the extraction of events in the running program from the specification and monitoring process. This separation allows one to cleanly monitor first-order properties involving arbitrarily complex native program objects, while still incurring reasonable overhead.
The great attractiveness of purely functional languages is their ability to depart from sequential order of computation. Theoretically, it enables two important features of the compiler:
1) The ability to reorder computation flow, making the program implicitly parallelisable. Modern imperative language compilers, even using careful synchronization of concurrent code, still generate huge chunks of sequential instructions that need to be executed on a single processor core; a purely functional language compilers can dispatch very small chunks to many (hundreds and thousands) of cores, carefully eliminating as many execution path dependencies as possible.
2) As the compiler formalizes different types of side effects, it can detect a whole new class of program errors at compile time, including resource acquisition and releasing problems, concurrent access to shared resources, many types of deadlocks etc. It is not yet a full-fledged program verification, but it is a big step in that direction.
Scala is a semi-imperative language with strong support for functional programming and rich type system. One can isolate the purely functional core of the language which can be put on the firm mathematical foundation of dependent type theories. We argue that it is possible to treat Scala code as it's written by now as an implicit do-notation which can be then reduced to a purely functional core by means of recently introduced Scala macros. The formalism of arrows and applicative contexts can bring Scala to a full glory of an implicitly parallelisable programming language, while still keeping its syntax mostly unchanged.
This is my attempt to compose a brief and cursory introduction to concepts in Java programming language. I call it Java Quicky.
I plan to extend and enhance it over time.
This presentation describes key concepts in Java. I call it The Java Quicky.
This is part of a series of presentations to cover the Java programming language and its new offerings and versions in depth.
Static analysis: Around Java in 60 minutesAndrey Karpov
Theory
Code quality (bugs, vulnerabilities)
Methodologies of code protection against defects
Code Review
Static analysis and everything related to it
Tools
Existing tools of static analysis
SonarQube
PVS-Studio for Java what is it?
Several detected examples of code with defects
More about static analysis
Conclusions
Java/Scala Lab 2016. Григорий Кравцов: Реализация и тестирование DAO слоя с н...GeeksLab Odessa
16.4.16 Java/Scala Lab
Upcoming events: goo.gl/I2gJ4H
В докладе будет расммотрена идеология использования неявных классов в Scala, рассмотрен пример построения DAO слоя с поддержкой нативной нотации MongoShell, показана лекость организации тестирования при таком подходе на примере DAO слоя взаимодействия с MongoDB через ReactiveMongo драйвер. В заключение будут представлены положительные и отрицательные стороны предложенного подхода
A Runtime Monitoring Framework for Event Streams with Non-Primitive ArgumentsSylvain Hallé
A runtime monitor is a tool that takes as input a model of some system, and observes in real time that the sequence of events produced by a run of that system follows the specification. While existing monitoring solutions generally use finite-state machines and temporal logic as their model language, the specification is ultimately tangled with hand-written, implementation-specific details which severely limit their range of application. We present a runtime monitoring platform that clearly separates the extraction of events in the running program from the specification and monitoring process. This separation allows one to cleanly monitor first-order properties involving arbitrarily complex native program objects, while still incurring reasonable overhead.
The great attractiveness of purely functional languages is their ability to depart from sequential order of computation. Theoretically, it enables two important features of the compiler:
1) The ability to reorder computation flow, making the program implicitly parallelisable. Modern imperative language compilers, even using careful synchronization of concurrent code, still generate huge chunks of sequential instructions that need to be executed on a single processor core; a purely functional language compilers can dispatch very small chunks to many (hundreds and thousands) of cores, carefully eliminating as many execution path dependencies as possible.
2) As the compiler formalizes different types of side effects, it can detect a whole new class of program errors at compile time, including resource acquisition and releasing problems, concurrent access to shared resources, many types of deadlocks etc. It is not yet a full-fledged program verification, but it is a big step in that direction.
Scala is a semi-imperative language with strong support for functional programming and rich type system. One can isolate the purely functional core of the language which can be put on the firm mathematical foundation of dependent type theories. We argue that it is possible to treat Scala code as it's written by now as an implicit do-notation which can be then reduced to a purely functional core by means of recently introduced Scala macros. The formalism of arrows and applicative contexts can bring Scala to a full glory of an implicitly parallelisable programming language, while still keeping its syntax mostly unchanged.
Object Oriented Code RE with HexraysCodeXplorerAlex Matrosov
In recent time we see a large spike of complex threats with elaborate object-oriented architecture among which the most notorious examples are: Stuxnet, Flamer, Duqu.
The approaches to analysis of such malware are rather distinct compared to the malware developed using procedural programming languages.
This presentation will take an in-depth look at challenges related to reversing object-oriented code with respect to modern malware and demonstrate approaches and tools employed for reversing object-oriented code.
How to write clean & testable code without losing your mindAndreas Czakaj
If you create software that is to be developed continuously over several years you'll need a sustainable approach to code quality.
In our early days of AEM development, however, we used to struggle with code that is rigid, hard to test and full of LOG.debug calls.
In this talk I will share some development best practices we have found that really work in actual AEM based software, e.g. to achieve 100% code coverage and provide high confidence in the code base.
Spoiler alert: no new libraries, frameworks or tools are required - once you know the ideas, plain old TDD and the S.O.L.I.D. principles of Clean Code will do the trick.
by Andreas Czakaj, mensemedia Gesellschaft für Neue Medien mbH
Presented at the adaptTo() 2017 conference in Berlin (https://adapt.to/2017/en/schedule/how-to-write-clean---testable-code-without-losing-your-mind.html).
Presentation video can be found on YouTube (https://www.youtube.com/watch?v=JbJw5oN_zL4)
NET Systems Programming Learned the Hard Way.pptxpetabridge
What is a thread quantum and why is it different on Windows Desktop and Windows Server? What's the difference between a blocking call and a blocking flow? Why did our remoting benchmarks suddenly drop when we moved to .NET 6? When should I try to write lock-free code? What does the `volatile` keyword mean?
Welcome to the types of questions my team and I are asked, or ask ourselves, on a regular basis - we're the makers of Akka.NET, a high performance distributed actor system library and these are the sorts of low-level questions we need to answer in order to build great experiences for our own users.
In this talk we're going to learn about .NET systems programming, the low level components we hope we can take for granted, but sometimes can't. In particular:
- The `ThreadPool` and how work queues operate in practice;
- Synchronization mechanisms - including `lock`-less ones;
- Memory management, `Span<T>`, and garbage collection;
- `await`, `Task`, and the synchronization contexts; and
- Crossing user-code and system boundaries in areas such as sockets.
This talk will help .NET developers understand why their code works the way it does and what to do in scenarios that demand high performance.
Automated Discovery of Deserialization Gadget ChainsPriyanka Aash
Although vulnerabilities stemming from the deserialization of untrusted data have been understood for many years, unsafe deserialization continues to be a vulnerability class that isn't going away. Attention on Java deserialization vulnerabilities skyrocketed in 2015 when Frohoff and Lawrence published an RCE gadget chain in the Apache Commons library and as recently as last year's Black Hat, Muñoz and Miroshis presented a survey of dangerous JSON deserialization libraries. While much research and automated detection technology has so far focused on the discovery of vulnerable entry points (i.e. code that deserializes untrusted data), finding a "gadget chain" to actually make the vulnerability exploitable has thus far been a largely manual exercise. In this talk, I present a new technique for the automated discovery of deserialization gadget chains in Java, allowing defensive teams to quickly identify the significance of a deserialization vulnerability and allowing penetration testers to quickly develop working exploits. At the conclusion we will also be releasing a FOSS toolkit which utilizes this methodology and has been used to successfully develop many deserialization exploits in both internal applications and open source projects.
SNMP Project: SNMP-based Network Anomaly Detection Using ClusteringLaili Aidi
This document contains implementation report of a system that is able to monitor the network using SNMP and identify the specific possible attacks (DoS and port scan) using a cluster analysis. In the first task, the program discovers the topology of the network. After successful discovery phase, it will be able to monitor the link utilization (network link-states) for a specified period of time, and then detect the anomaly, using k-means clustering scheme [1]. These anomalies will be analyzed to recognize the attack. Moreover, this program also provides an advance feature, which is defined as optional task, as it executes online monitoring and detects the attacks using Davies-Bouldin Index as quality scoring measurement [2].
--
Please contact trough lailiaidi at gmail.com for download request
1. 8th Central and Eastern European
Software Engineering Conference
in Russia - CEE-SECR 2012
November 1 - 2, Moscow
Dynamic data race detection in
concurrent Java programs
Vitaly Trifanov, Dmitry Tsitelov
Devexperts LLC
2. Agenda
What are data races and why they are dangerous
Automatic races detection
approaches, pros & cons
Happens-before race detection algorithm
Vector clocks
Our dynamic race detector
implementation
solved problems
3. Data Race Example
public class Account {
private int amount = 0;
public void deposit(int x) {amount += x;}
public int getAmount() {return amount;}
}
public class TestRace {
public static void main (String[] args) {
final Account a = new Account();
Thread t1 = depositAccountInNewThread(a, 5);
Thread t2 = depositAccountInNewThread(a, 6);
t1.join();
t2.join();
System.out.println(account.getAmount()); //may print 5, 6, 11.
}
}
6. Data Races
Data race occurs when many threads access the same
shared data concurrently; at least one writes
Usually it’s a bug
7. Data Races Are Dangerous
Hard to detect if occurred
no immediate effects
program continues to work
damage global data structures
Hard to find manually
Not reproducible - depends on threads timing
Dev & QA platforms are not so multicore
8. Automatic Race Detection
20+ years of research
Static
analyze program code offline
data races prevention (extend type system, annotations)
Dynamic: analyze real program executions
On-the-fly
Post-mortem
10. Static Approach
Pros
Doesn’t require program execution
Analyzes all code
Doesn’t depend on program input, environment, etc.
Cons
Unsolvable in common case
Has to reduce depth of analysis
A lot of existing tools for Java
FindBugs, jChord, etc
11. Dynamic Approach
Pros
Complete information about program flow
Lower level of false alarms
Cons
Very large overhead
No existing stable dynamic detectors for Java
12. Static vs Dynamic: What To Do?
Use both approaches
Static (FindBugs/Sonar, jChord, …)
Eliminate provable synchronization inconsistencies
on the early stage
Dynamic
Try existing tools, but they are unstable
IBM MSDK, Thread Sanitizer for Java
That’s why we’ve developed our own!
13. Data Race Detector Concept
Application uses libraries and frameworks via API
At least JRE
API is well documented
“Class XXX is thread-safe”
“Class YYY is not thread-safe”
“XXX.get() is synchronized with preceding call of XXX.set()”
Describe behavior of API and exclude library from
analysis
15. What Operations to Intercept?
Synchronization operations
thread start/join/interrupt
synchronized
volatile read/write
java.util.concurrent
Accesses to shared data
fields
objects
16. How It Works
Instrumented
Application
DRD agent app classes
classes
interceptor
Race
Config detection
module
19. JLS: Happens-Before & Data Races
X happens-before Y, when
X, Y - in same thread, X before Y in program order
X is synchronized-with Y
Transitivity: exists Z: hb(X, Z) && hb(Z, Y)
Data race: 2 conflicting accesses, not ordered by
happens-before relation
32. Code Instrumentation
Check everything => huge overhead
Race detection scope
Accesses to our fields
Foreign calls (treat them as read or write)
Sync scope
Detect sync events in our code
Describe contracts of excluded classes
Treat these contracts as synchronization events
33. Race Detection
private class Storage {
private Map<Integer, Item> items = new HashMap<Integer, Item> ();
public void store(Item item) {
items.put(item.getId(), item);
}
public void saveToDisk() { On each access of “items” field we check
On each access of “items” field we check
for (Item item : items.values()) {
//serialize and save race on this field
race on this field
saveItem(item);
//...
}
}
On each call of “items” method we check
On each call of “items” method we check
public Item getItem(int id) { race on this object
race on this object
return items.get(id);
}
public void reload() {
items = deserealizeFromFile(); Each field of class Item is protected the
Each field of class Item is protected the
} same way as field “items” of class Storage
same way as field “items” of class Storage
}
36. Composite Keys
AtomicLongFieldUpdater.CAS(Object o, long offset, long v)
param 0 + param 1
Volatile field “abc” of object o
object + field name
AtomicInteger.set() & AtomicInteger.get()
object
ConcurrentMap.put(key, value) & ConcurrentMap.get(key)
object + param 0
37. Solved Problems
Composite keys for contracts and volatiles
Generate them on-the-fly
Avoid unnecessary keys creation
ThreadLocal<MutableKeyXXX> for each CompositeKeyXXX
Loading of classes, generated on-the-fly
Instrument ClassLoader.loadClass()
38. Solved Problems
Don’t break serialization
compute serialVersiodUid before instrumentation
Caching components of dead clocks
when thread dies, its time frames doesn’t grow anymore
cache frames of dead threads to avoid memory leaks
local last-known generation & global generation
39. DRD in Real Life: QD
✔ 6 races found
QD
QD + DRD
40. DRD in Real Life: MARS UI
MARS
✔ 5 races found
MARS + DRD
41. DRD Race Report Example
WRITE_READ data race between current thread Thread-12(id = 33) and thread
Thread-11(id = 32)
Race target : field my/app/DataServiceImpl.stopped
Thread 32 accessed it in my/app/DataServiceImpl.access$400(line : 29)
--------------------------------Stack trace for racing thread (id = 32) is not available.------------
-------------------------------------Current thread's stack trace (id = 33) : ---------------------------
at my.app.DataServiceImpl.stop(DataServiceImpl.java:155)
at my.app.DataManager.close(DataManager.java:201)
...
Hard to find at development/review stage Control of program execution flow is frequently delegated to frameworks Hard to detect manually during testing
ThreadLocal<MutableInteger> : increment my counter, push it ’ s value to global AtiomicInteger each N-th hit. When thread dies, increment global generation. Each clock stores last known generation. When it becomes less than global, check who have died and cache corresponding frames.
Module testing – check module having contracts of others Lightweight mode : instrument little, most important parts of code Support for synchronization on array cells