Locking and Race Conditions in Web Applications

8,771 views

Published on

Mutexes, locks, transactions -- they all may seem more relevant in compiled languages, lower level drivers or in databases; however, race conditions can be of equal dilemma in modern web applications. Something as simple as a user double clicking a submit form can yield unexpected results. These problems are difficult to replicate and to test, so they often go undetected. They can occur with or without significant traffic. Finally, with NoSQL alternatives growing in popularity for storing data and as caching layers, we need new alternatives to database transactions and locking.

In this session, I will present situations which are vulnerable to race conditions, along with solutions. I'll also talk about locking approaches that are reliable, efficient and scalable.

Published in: Technology

Locking and Race Conditions in Web Applications

  1. 1. Locking and Race Conditions in Web Applications<br />By Andrew Kandels<br />
  2. 2. What is a Race Condition?<br />When the completion of one operation depends on the completion of another, which in turn depends on the first. <br />The Olive Garden Example:<br />The waiter brings one breadstick for each person and then one additional. <br />Each person eats one bread stick, leaving one breadstick in the basket. No one takes the last breadstick in fear of being labeled a scrooge. <br />
  3. 3. Recipes for Race Conditions<br />Evil Forms<br />Multiple reads and writes as one “Unit of Work”<br />Operations over multiple systems<br />Relational models on distributed systems<br />Ajax and round robin load balancing web applications<br />High traffic or load<br />Other Problems:<br /><ul><li>Hard to test for
  4. 4. Can be difficult to replicate
  5. 5. Error logs tend to be misleading
  6. 6. Often run undetected</li></li></ul><li>Evil Forms<br />Prevent double-clicking submit<br />Tokenize forms (passive/active)<br />Tokenize with Ajax frequency polling<br />Tokenize as a user action item<br />Auto-cancel non-update submits<br />In place editing<br />Detect changes (before/after values)<br />
  7. 7. Multiple Operations in one “Unit of Work”<br />In a multi-threaded environment such as a web server, this is a candidate for a race condition:<br />Thread #1 fetch() = empty<br />Thread #2 fetch() = empty<br />Thread #1 create()<br />Thread #2 create() // Exception: duplicate key<br />
  8. 8. The Solution<br />Use a database transaction.<br />Use a global lock or semaphore.<br />Use the UNIQUE / PRIMARY keys in your application logic:<br />Best for frequent writes<br />(email table) <br />Best for frequent reads (setting lookup table)<br />
  9. 9. Operations over Multiple Systems<br />User record is created with login credentials (LDAP, RDMS)<br />Account record is created and linked to the user record (RDMS)<br />Billing record is created and linked to the account (Payment Processor)<br />Program settings and new user flags are set (RDMS)<br />Session and Caches are created (Memcached)<br />Signup analytics are logged (Data Warehouse, ODS)<br />Welcome email is sent (Gearman, Email Processor)<br />Partial Failures:<br /><ul><li>Inconsistent data
  10. 10. Orphaned rows
  11. 11. Faulty reporting
  12. 12. Unhappy customers</li></li></ul><li>ACID Programming<br />Atomic, Consistent, Isolated and Durable.<br /><ul><li>Small functions that perform an atomic data operation
  13. 13. Predictable successes and failures
  14. 14. Failures should leave the data source unaltered
  15. 15. Can be nested hierarchically:</li></ul>Sign Up<br />Billing System<br />Create Login<br />Data Warehouse<br />
  16. 16. Example Code<br />
  17. 17. Multi-Version Concurrency Control (MVCC)<br />MVCC relies on snapshots to retain multiple versions of data in order to provide consistency. Available in most traditional databases (MySQL, MS-SQL, Oracle, PostgreSQL, Firebird, etc.) and elsewhere (svn, git, reiserfs).<br />Pros:<br /><ul><li>Transactions
  18. 18. Reads are never blocked
  19. 19. Atomic operations
  20. 20. Point-in-time data ensures consistency in models</li></ul>Cons:<br /><ul><li>Can be expensive retaining multiple copies of data
  21. 21. Deadlocks
  22. 22. Reduced performance</li></li></ul><li>ANSI Isolation Levels<br />Degrees of consistency that deal with snapshot “phenomena” and lock collisions by balancing concurrency with throughput.<br />READ UNCOMMITTEDAllows “dirty reads” in which one transaction can access data from uncommitted changes in a second transaction.<br />READ COMMITTED<br /> Committed changes in one transaction appear in another, which can cause two identical queries to return different results.<br />REPEATABLE READ (InnoDB default)<br /> Within a transaction, two identical reads will always return the same result.<br />SERIALIZABLE<br /> Transactions running reads lock updates from other transactions.<br />Note: In MySQL, some levels do not allow statement based replication.<br />
  23. 23. Deadlocks – Another Race Condition<br />A deadlock is an (often unpredictable) situation where two or more competing operations endlessly wait for their counterpart to finish.<br />Common Causes:<br /><ul><li>Transactions with too many operations
  24. 24. Insufficient isolation level
  25. 25. Lack of indexes
  26. 26. Locking queries:INSERT INTO table1 SELECT *FROM table 2;</li></ul>Note: SHOW INNODB STATUS is a great tool for debugging deadlocks.<br />
  27. 27. Non-Database Locks<br />Locking in distributed systems is slow and is often not fully implemented in non-RDMS. Multi-system operations that require ACID need alternatives.<br />Alternative Locking Methods:<br /><ul><li>Filesystem lock
  28. 28. Database Semaphore
  29. 29. Memcached
  30. 30. MongoDB
  31. 31. APC
  32. 32. Redis</li></li></ul><li>Filesystem Lock<br />flock() is used by PHP’s native session handler and is more reliable than fileexists() because it’s a single atomic operation.<br />
  33. 33. Database Named Semaphore<br />A semaphore is different than a transaction because the lock is independent of any single data source or operation.<br />
  34. 34. Memcached Named Lock<br />Memcached offers an additional advantage by natively providing a timeout.<br />
  35. 35. JavaScript andAjax<br />Heavily interactive web applications often rely on large amounts of javascript and ajax which can wreak havoc on sessions as well as cause race conditions.<br />Suggestions:<br /><ul><li>Lock ajax requests to the same web node
  36. 36. Consider an ajax only web server
  37. 37. Read-only sessions by default
  38. 38. Variable isolation / separation
  39. 39. Increment caches with routine flushing
  40. 40. Use Comet as opposed to frequent timed checks</li></li></ul><li>Counters and Trackers<br />Most web applications rely on counters and trackers for analytics and security. For example: tracking page views, login attempts, date of last actions, etc.<br />Suggestions:<br /><ul><li>Horizontal partitioning
  41. 41. Memcached/MongoDB increment()
  42. 42. JavaScript to an analytics node (or GA)
  43. 43. Scheduled access log processing (e.g.: ETL)
  44. 44. MySQL blackhole engine with a replicated slave</li></li></ul><li>The End<br />Web:<br />Mail:<br />Twitter:<br />http://andrewkandels.com<br />mailto:akandels@gmail.com<br />@andrewkandels<br />

×