Long thought to be relegated to the domain of fast, multithreaded desktop applications, race conditions have made their way into web applications. These bugs are often difficult to test for, and are becoming increasingly prevalent due to faster and faster clients, while server-side languages like Node.js and PHP are struggling to keep up. Race conditions are no longer just bugs- when they are found in critical components of web applications, they become a serious security vulnerability. If the proper checks and defensive measures are not in place, databases get confused, “one-time-use” becomes a relative term, and “limited” becomes “unlimited”. This talk will detail specific examples where malicious users could cause damage or profit from a race-condition flaw in a web application. A custom open-source tool will also be introduced to help security researchers and developers easily check for this class of vulnerability in web applications.

1. Racing The Web

2. Security Consultant at Security Compass
Professor of Application Security at
Georgian College
Software developer
Former sysadmin
etc.
Aaron Hnatiw
twitter: @insp3ctre

3. Why am I here?

4. Race conditions

5. OWASP Definition
A race condition is a flaw that produces an unexpected
result when the timing of actions impact other actions.
An example may be seen on a multithreaded application
where actions are being performed on the same data.
Race conditions, by their very nature, are difficult to test
for.
https://www.owasp.org/index.php/Testing_for_Race_Conditions_%28OWASP-AT-010%29

6. CWE-362: Concurrent Execution
using Shared Resource with Improper
Synchronization ('Race Condition’)
https://cwe.mitre.org/data/definitions/362.html

7. Basically, when you bet on
the wrong horse.
(or even “bet” at all)

8. Assumption

9. Race Condition

10. When does this become a
security issue?

11. “One time use” coupon
codes

12. Bug bounty payouts
https://cobalt.io/cobalt/cobalt/reports/587

13. Balance transfer between
accounts

14. What does a race condition
look like?

16. Python

17. Go

18. Surprise…

19. PHP

20. Testing for race conditions in
web applications

21. Usual whitebox method
1. Identify all shared data
2. Identify where that shared data is accessed across
systems
3. Find where that data access is not synchronized
4. Make a TON of requests

22. There’s a better way…

23. Introducing: Race-The-Web
(RTW)
A tool that automates race condition discovery
Simple configuration (TOML)
Open-source
Written in Go

24. Demo time

25. Try it out yourself!
RaceTheWeb.io

26. Comparison vs Burp Intruder

27. Speed
• Intruder setup:
• 10 threads (default is 5)
• 3 retries after network failure (2000 ms pause before
retry)- default
• Redirect: always
• 1000 requests (withdraw $1 x 1000)

28. Speed (cont’d)
• RTW setup:
• Verbose logging
• Follow redirects
• 1000 requests (withdraw $1 x 1000)

29. Speed (cont’d)
• Results
• Intruder: ~3:10
• RTW: ~2:00

30. Speed (cont’d)
• Intruder optimizations:
• 999 threads (maximum possible)
• No retries on network failure
• Result: ~2:15
• RTW is still faster

31. Key difference:
Built-in response comparison in
RTW

32. // TODO
• Add “proxy” option to config file
• Clean up codebase
• Write Burp plugin

34. Be careful of DoS.
Most bug bounties have restrictions such as:
https://hackerone.com/airbnb

35. Useful tip: check for CRUD
functionality.
(especially with remote resources)

36. Let’s talk about defence

37. Most effective: Locks
Use with shared resources

38. Lock #1:
Application-level

39. Python
• threading.Lock
• acquire()
• release()
• Others:
• threading.RLock
• threading.Condition
• threading.Semaphore
• queue: handles locking automatically for resources in the queue.
More: https://docs.python.org/2/library/threading.html

40. Python- fix

41. Go
• sync.mutex
• Lock()
• Unlock()
• sync.RWMutex
• Rlock()
• RWLock()
• Unlock()
https://golang.org/pkg/sync/

42. Go
Channels:
“Do not communicate by sharing memory; instead,
share memory by communicating.”

43. Go- fix

44. PHP
• You could compile PHP with “--enable-sysvsem"...
• Not supported everywhere
• Not useful in a distributed environment
• May not be possible in a shared hosting
environment
• You’re pretty much stuck with implementing this at
the database or file level (more on that later)
• If you know of any other way, please let me know!

45. Lock #2:
Database-level

46. Database MUST be ACID-
compliant

47. ACID-Compliant Databases
• Atomicity: All or nothing. A transaction either succeeds or
rolls back.
• Consistency: On the completion of a transaction, the
database is structurally sound. Otherwise, it reverts back to
the previous sound state.
• Isolation: Transactions do not interfere with each other.
• This point is KEY.
• Durability: The results of applying a transaction are
permanent, even in the presence of failures.

48. Isolation
• Highest level: serializable.
• Transactions essentially occur serially (one after another), rather
than concurrently.
• Next level: repeatable read.
• Close, but still allows race conditions.
• Be prepared to retry transactions often, because in most cases, these
isolation levels can result in a large number of transaction failures.
• Obvious downside- using higher levels of isolation can slow down
your application.

49. Solution- MySQL
• From the documentation: "MySQL Server (version
3.23-max and all versions 4.0 and above) supports
transactions with the InnoDB transactional storage
engine. InnoDB provides full ACID compliance."
• Use SERIALIZABLE isolation level
• Default is REPEATABLE-READ
• Can be set globally, for a session, or for individual
transactions
More info: https://dev.mysql.com/doc/refman/5.5/en/innodb-transaction-isolation-
levels.html

50. MySQL (cont’d)
• System variable:
SET GLOBAL @@GLOBAL.tx_isolation=`SERIALIZABLE`;
• Command-line option at mysqld startup:
--transaction-isolation=SERIALIZABLE
• Option file:
[mysqld]
transaction-isolation = SERIALIZABLE
• Command-line, BEFORE starting a transaction:
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;
More Info: https://dev.mysql.com/doc/refman/5.5/en/set-transaction.html

51. Solution- PostgreSQL
• Use the SERIALIZABLE transaction isolation level
• Default is READ COMMITTED: all queries see a snapshot of
committed data at the time of the query.
• Command-line:
• START TRANSACTION;
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;
• START TRANSACTION ISOLATION LEVEL SERIALIZABLE;
• BEGIN TRANSACTION ISOLATION LEVEL SERIALIZABLE;
More info: https://www.postgresql.org/docs/9.3/static/sql-set-transaction.html

52. PostgreSQL (cont’d)
• Configuration file: postgresql.conf
default_transaction_isolation = ‘serializable’
• Command-line option with postgres command:
postgres -c
default_transaction_isolation=‘serializable’ ...
• Environment variable:
env PGOPTIONS=“-c
default_transaction_isolation=‘serializable’” psql
• Mid-session SQL:
SET SESSION default_transaction_isolation = ‘serializable’;

53. Realistic compromise

54. For most use cases:
• Optimize your queries
• Use a single query whenever possible (e.g. UPDATE xTable SET yValue = yValue+1
WHERE id = ‘zID’)
• Inserts instead of updates
• Use unique indexes
• Use an ORM for “optimistic locking”
• Most ORMs do their own optimizations and locking to prevent race conditions, as
opposed to relying on the database’s strict “pessimistic locking”
• Use the READ COMMITTED isolation level
• Not as strict as Serializable, but provides more speed, less locking errors, and more
consistency than REPEATABLE READ

55. Solution- MongoDB
• No serialization
• From 2.2 on, uses database-level read and write locks, depending on operation (https://
docs.mongodb.com/manual/faq/concurrency/#which-operations-lock-the-database)
• Single document writes are atomic (but not isolated) by default
• $isolated operator
• Acquires an exclusive lock to all documents being written to (only applies when writing to multiple
documents).
• Does not work on sharded clusters.
More info: https://docs.mongodb.com/v3.2/core/write-operations-atomicity/

56. Lock #3:
File-level

57. Native Methods
• Windows
• LockFile function of the Windows API: https://msdn.microsoft.com/en-us/library/
aa365202.aspx
• Unix
• flock()/lockf(): essentially the same function
• fcntl(): http://pubs.opengroup.org/onlinepubs/9699919799/functions/fcntl.html
• Some typical “gotchas”: http://0pointer.de/blog/projects/locking.html
• Lock file: create a temporary file (e.g. ~myfile.lck), which exists while a file needs to
be locked. Check for the lock file before accessing its associated file.
• Probably the best way to do this at the file-level.

58. Don’t overdo it though!
Avoid locking hell.
Don’t share resources unless you have to.

59. Best Practices

60. Ensure your database can
keep up
• Often the slowest point in the application logic chain
• Database speed should keep at pace with the speed
of users making requests to your web application
• Best bet- host on the same network
• Not the same server though- tiered architecture is
best
• This provides defence-in-depth; not a panacea

61. Fetch data only right as you
need it
Again, defence-in-depth. This is by no means a
complete solution on its own.

62. CSRF Tokens
• You can’t automate a bunch of requests if they require
a unique token every time
• More of a client-side solution, does not necessarily
address the root cause of a race condition
• Do this even for non-sensitive actions
• Attacker’s perspective- found a CSRF vuln? Try
leveraging that into a race condition as well!

63. Further Reading
• https://www.josipfranjkovic.com/blog/race-conditions-on-web
• http://sakurity.com/blog/2015/05/21/starbucks.html
• https://defuse.ca/race-conditions-in-web-applications.htm
• Web Application Hacker’s Handbook, 2nd Ed.; chapter 11, "Example 12:
Racing Against the Login" (page 426)
• http://www.hakim.ws/BHUSA08/speakers/
Stender_Vidergar_Concurrency_Attacks/
BH_US_08_Stender_Vidergar_Concurrency_Attacks_in_Web_Applications
_Presentation.pdf
• https://www.owasp.org/index.php/Reviewing_Code_for_Race_Conditions

64. You’ve got another tool in
your tool belt

65. Now go and race the web!
http://RaceTheWeb.io