<Insert Picture Here>

Locking and Concurrency Control
Morgan Tocker, MySQL Community Manager
Safe Harbor Statement
The	
  following	
  is	
  intended	
  to	
  outline	
  our	
  general	
  product	
  direction.	
  
I...
4 Years of MySQL Innovation
MySQL Cluster 7.3

MySQL Migration Wizard

MySQL Workbench 6.0

MySQL 5.6

MySQL 5.5

Windows ...
Agenda
1.Table Locks, Row Locks and MVCC
2.Live Demo
3.Deadlocks and Lock Waits Discussion
4.ACID Discussion
5.Summary
Table	
  Locks,	
  Row	
  Locks	
  and	
  MVCC

Copyright	
  ©	
  2012	
  Oracle	
  and/or	
  its	
  affiliates.	
  All	
 ...
Customer Accounting System
customer
customerid

name

1

Acorn Enterprises

2

First National Squirrel Bank

3

Squirrels ...
Start Backup
customer
customerid

name

1

Acorn Enterprises

2

First National Squirrel Bank

3

Squirrels 5000 Inc

invo...
Continues…
customer
customerid

name

1

Acorn Enterprises

2

First National Squirrel Bank

3

Squirrels 5000 Inc

invoic...
Concurrent Access System - New
Customer Added
customer
customerid

name

1

Acorn Enterprises

2

First National Squirrel ...
Backup Continues
customer
customerid

name

1

Acorn Enterprises

2

First National Squirrel Bank

3

Squirrels 5000 Inc

...
Backup Continues…
customer
customerid

name

1

Acorn Enterprises

2

First National Squirrel Bank

3

Squirrels 5000 Inc
...
Backup Complete
customer
customerid

name

1

Acorn Enterprises

2

First National Squirrel Bank

3

Squirrels 5000 Inc

4...
What was really backed up
customer
customerid

name

1

Acorn Enterprises

2

First National Squirrel Bank

3

Squirrels 5...
Next Operation
customer
customerid

1

Acorn Enterprises

2

First National Squirrel Bank

3

Squirrels 5000 Inc

4

X

na...
Solution
• Operation needs to represent a single point in time
• When does not matter.
• This race condition can exist in ...
MyISAM Implementation
customer
customerid

name

1

Acorn Enterprises

2

First National Squirrel Bank

3

Squirrels 5000 ...
MyISAM Implementation
customer
customerid

name

1

Acorn Enterprises

2

First National Squirrel Bank

3

Squirrels 5000 ...
MyISAM Implementation
customer
customerid

name

1

Acorn Enterprises

2

First National Squirrel Bank

3

Squirrels 5000 ...
MyISAM
•
•
•
•

Readers may co-exist.
Writers are forbidden.
Locks a table by default.
Programs like mysqldump responsibly...
InnoDB Implementation
customer
Trx ID

Rollback Ptr

customerid

name

123

##

1

123

##

2

Acorn
Enterprises
First Nat...
InnoDB
• Hidden meta data attached to each row:
• Row version
• Rollback pointer to older versions if needed.
• Every tran...
InnoDB Implementation
Backup process is TRX ID #394.
Remembers it is not supposed to see
anything newer than itself.
New c...
MVCC Algorithm
• Stands for “Multi-version concurrency control” - http://
en.wikipedia.org/wiki/
Multiversion_concurrency_...
MVCC (cont.)
• Multiple versions of the same row may exist.
• What you will see depends on the context of your
transaction...
Live	
  Demo

Copyright	
  ©	
  2012	
  Oracle	
  and/or	
  its	
  affiliates.	
  All	
  rights	
  reserved.
Deadlocks	
  &	
  Lock	
  Waits

Copyright	
  ©	
  2012	
  Oracle	
  and/or	
  its	
  affiliates.	
  All	
  rights	
  rese...
What is a deadlock?
• A situation when two sessions are trying to acquire
each-other’s respective locks.
• Neither can pro...
What is a lock wait?
• One step short of a deadlock.
• You keep trying to do something, but it’s not in the
budget this qu...
Deadlock Detection
• InnoDB as a storage engine will detect deadlocks.
• Algorithm is Graph detection http://en.wikipedia....
Preventing Deadlocks
• Deadlock is a concurrency issue.
• Same solutions as when you have colleagues
arguing!
Solution #1
• Don’t make them work-together.
• Reduce concurrency / run known conflicting workers
in serial execution.

X
Solution #2
• Don’t make them work on big projects together.
• aka lower the stakes!
• Locks are held for the duration of ...
Preventing Deadlocks (cont.)
• Also, we should accept that some deadlocks are a
fact of life. Continuing co-worker analogy...
Preventing Deadlocks (cont.)
• You actually want databases to be optimistic.
• That is set as few locks as possible, knowi...
Handing Deadlocks
• Database only owns responsibility of alerting
application
• It may not reapply transaction, since stat...
Lock Wait Timeout
• Will proceed immediately if it can acquire the lock.
• Since most transactions are less than 1 second,...
“Requires a replay of the transaction”
•
•
•
•

The problems probably won’t happen often.
May be easier to kick back to th...
ACID

Copyright	
  ©	
  2012	
  Oracle	
  and/or	
  its	
  affiliates.	
  All	
  rights	
  reserved.
The “ACID” Contract
• Atomicity - All or none.
• Consistency - Generic term to say all constraints etc
must be honoured (p...
Atomicity
• Most people know this one:
• Transfer $20 from Account A --> Account B.
• Multiple failure states...
• The mon...
Consistency
• Not too much more to add on this one.
• InnoDB has foreign key constraints
• MySQL does not feature CHECK co...
Isolation
• Actually a very expensive feature for a database to
implement (may require lots of locking, stale data
kept ar...
Isolation Levels
• READ-UNCOMMITTED - No isolation
• READ-COMMITTED - Can see other session’s
committed rows.
• REPEATABLE...
Durable
• This one’s actually tunable as well.
• InnoDB will not lie by default.
• It means on COMMIT everything should su...
Summary

Copyright	
  ©	
  2012	
  Oracle	
  and/or	
  its	
  affiliates.	
  All	
  rights	
  reserved.
Application Responsibilities
• Wrap every logical unit of work in START
TRANSACTION and COMMIT.
• Expect failure because f...
<Insert Picture Here>
Mutexes, Latches & Semaphores
• Common names given to database “internal locking”.
• Today we are not going to differentia...
Example #1
• InnoDB buffer pool is full
• Free space needs to be created in order to be able to
load your desired page int...
Example #2
• When a page becomes too full, it may need to split
data into other pages.
• It is possible that there could b...
Internal Locking (cont.)
• When mutexes have high contention other tasks are
not runable.
• Which means that CPUs can appe...
Fixes in newer versions
• Responsibility to fix is on MySQL Engineering Team.
• High Priority for Us.
• Some DBAs will hav...
Time to Upgrade?
• Kernel mutex split (5.6)
• Lock open mutex split (5.6)
• Buffer pool mutex contention reduced w/8 buffe...
Locking and Concurrency Control
Upcoming SlideShare
Loading in...5
×

Locking and Concurrency Control

1,205

Published on

1 Comment
2 Likes
Statistics
Notes
  • A introductory presentation but lack of in-depth explanation.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
1,205
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
51
Comments
1
Likes
2
Embeds 0
No embeds

No notes for slide

Locking and Concurrency Control

  1. 1. <Insert Picture Here> Locking and Concurrency Control Morgan Tocker, MySQL Community Manager
  2. 2. Safe Harbor Statement The  following  is  intended  to  outline  our  general  product  direction.   It  is  intended  for  information  purposes  only,  and  may  not  be   incorporated  into  any  contract.  It  is  not  a  commitment  to  deliver   any  material,  code,  or  functionality,  and  should  not  be  relied  upon   in  making  purchasing  decisions.   
 The  development,  release,  and  timing  of  any  features  or   functionality  described  for  Oracle’s  products  remains  at  the  sole   discretion  of  Oracle.
  3. 3. 4 Years of MySQL Innovation MySQL Cluster 7.3 MySQL Migration Wizard MySQL Workbench 6.0 MySQL 5.6 MySQL 5.5 Windows installer & Tools MySQL 5.7 M y S Q L Cluster MySQL MySQL Enterprise Monitor 2.3 & e r M a n a g 3.0 Applier for Hadoop MySQL Enterprise Backup Security MySQL Utilities MySQL Workbench 5.2 & 6.0 Scalability MySQL Cluster 7.2 HA MySQL Enterprise MySQL Cluster 7.1 O r a c l e C e r t i f i c a t i o n s Audit
  4. 4. Agenda 1.Table Locks, Row Locks and MVCC 2.Live Demo 3.Deadlocks and Lock Waits Discussion 4.ACID Discussion 5.Summary
  5. 5. Table  Locks,  Row  Locks  and  MVCC Copyright  ©  2012  Oracle  and/or  its  affiliates.  All  rights  reserved.
  6. 6. Customer Accounting System customer customerid name 1 Acorn Enterprises 2 First National Squirrel Bank 3 Squirrels 5000 Inc invoice_item itemid customerid 1 1 # Acorn Enterprises 2 1 # Acorn Enterprises 3 3 # Squirrels 5000 Inc ... Amount 1 2 1 hrs Professional Services Acorn Parts $10 2 invoiceid Description 1 invoice invoiceid 3 2 $50 4 3 5 hrs Professional Services Delivery Fee 5 3 Replacement Acorn $20 $12.50 $17.50
  7. 7. Start Backup customer customerid name 1 Acorn Enterprises 2 First National Squirrel Bank 3 Squirrels 5000 Inc invoice_item itemid customerid 1 1 # Acorn Enterprises 2 1 # Acorn Enterprises 3 3 # Squirrels 5000 Inc ... Amount 1 2 1 hrs Professional Services Acorn Parts $10 2 invoiceid Description 1 invoice invoiceid 3 2 $50 4 3 5 hrs Professional Services Delivery Fee 5 3 Replacement Acorn $20 $12.50 $17.50
  8. 8. Continues… customer customerid name 1 Acorn Enterprises 2 First National Squirrel Bank 3 Squirrels 5000 Inc invoice_item itemid customerid 1 1 # Acorn Enterprises 2 1 # Acorn Enterprises 3 3 # Squirrels 5000 Inc ... Amount 1 2 1 hrs Professional Services Acorn Parts $10 2 invoiceid Description 1 invoice invoiceid 3 2 $50 4 3 5 hrs Professional Services Delivery Fee 5 3 Replacement Acorn $20 $12.50 $17.50
  9. 9. Concurrent Access System - New Customer Added customer customerid name 1 Acorn Enterprises 2 First National Squirrel Bank 3 Squirrels 5000 Inc 4 Savvy Squirrel Global invoice_item itemid customerid 1 1 # Acorn Enterprises 2 1 # Acorn Enterprises 3 3 # Squirrels 5000 Inc 4 4 # Savvy Squirrels ... Amount 1 2 1 hrs Professional Services Acorn Parts $10 2 invoiceid Description 1 invoice invoiceid 3 2 $50 4 3 5 hrs Professional Services Delivery Fee 5 3 $20 6 4 Replacement Acorn Premium Acorn $12.50 $17.50 $999
  10. 10. Backup Continues customer customerid name 1 Acorn Enterprises 2 First National Squirrel Bank 3 Squirrels 5000 Inc 4 Savvy Squirrel Global invoice_item itemid customerid 1 1 # Acorn Enterprises 2 1 # Acorn Enterprises 3 3 # Squirrels 5000 Inc 4 4 # Savvy Squirrels ... Amount 1 2 1 hrs Professional Services Acorn Parts $10 2 invoiceid Description 1 invoice invoiceid 3 2 $50 4 3 5 hrs Professional Services Delivery Fee 5 3 $20 6 4 Replacement Acorn Premium Acorn $12.50 $17.50 $999
  11. 11. Backup Continues… customer customerid name 1 Acorn Enterprises 2 First National Squirrel Bank 3 Squirrels 5000 Inc 4 Savvy Squirrel Global invoice_item itemid customerid 1 1 # Acorn Enterprises 2 1 # Acorn Enterprises 3 3 # Squirrels 5000 Inc 4 4 # Savvy Squirrels ... Amount 1 2 1 hrs Professional Services Acorn Parts $10 2 invoiceid Description 1 invoice invoiceid 3 2 $50 4 3 5 hrs Professional Services Delivery Fee 5 3 $20 6 4 Replacement Acorn Premium Acorn $12.50 $17.50 $999
  12. 12. Backup Complete customer customerid name 1 Acorn Enterprises 2 First National Squirrel Bank 3 Squirrels 5000 Inc 4 Savvy Squirrel Global invoice_item itemid customerid 1 1 # Acorn Enterprises 2 1 # Acorn Enterprises 3 3 # Squirrels 5000 Inc 4 4 # Savvy Squirrels ... Amount 1 2 1 hrs Professional Services Acorn Parts $10 2 invoiceid Description 1 invoice invoiceid 3 2 $50 4 3 5 hrs Professional Services Delivery Fee 5 3 $20 6 4 Replacement Acorn Premium Acorn $12.50 $17.50 $999
  13. 13. What was really backed up customer customerid name 1 Acorn Enterprises 2 First National Squirrel Bank 3 Squirrels 5000 Inc invoice_item itemid customerid 1 1 # Acorn Enterprises 2 1 # Acorn Enterprises 3 3 # Squirrels 5000 Inc 4 4 # Savvy Squirrels ... Amount 1 2 1 hrs Professional Services Acorn Parts $10 2 invoiceid Description 1 invoice invoiceid 3 2 $50 4 3 5 hrs Professional Services Delivery Fee 5 3 $20 6 4 Replacement Acorn Premium Acorn $12.50 $17.50 $999
  14. 14. Next Operation customer customerid 1 Acorn Enterprises 2 First National Squirrel Bank 3 Squirrels 5000 Inc 4 X name Acorn Intolerant Squirrels Inc itemid invoice invoiceid customerid 1 1 # Acorn Enterprises 2 1 # Acorn Enterprises 3 3 # Squirrels 5000 Inc 4 4 # Savvy Squirrels invoice_item ... X invoiceid Description Amount 1 1 $10 2 2 1 hrs Professional Services Acorn Parts 3 2 $50 4 3 5 hrs Professional Services Delivery Fee 5 3 $20 6 4 Replacement Acorn Premium Acorn $12.50 $17.50 $999
  15. 15. Solution • Operation needs to represent a single point in time • When does not matter. • This race condition can exist in all database operations - not just backups.
  16. 16. MyISAM Implementation customer customerid name 1 Acorn Enterprises 2 First National Squirrel Bank 3 Squirrels 5000 Inc 4 Savvy Squirrel Global invoice_item itemid customerid 1 1 # Acorn Enterprises 2 1 # Acorn Enterprises 3 3 # Squirrels 5000 Inc 4 4 # Savvy Squirrels ... Amount 1 2 1 hrs Professional Services Acorn Parts $10 2 invoiceid Description 1 invoice invoiceid 3 2 $50 4 3 5 hrs Professional Services Delivery Fee 5 3 $20 6 4 Replacement Acorn Premium Acorn $12.50 $17.50 $999
  17. 17. MyISAM Implementation customer customerid name 1 Acorn Enterprises 2 First National Squirrel Bank 3 Squirrels 5000 Inc 4 Savvy Squirrel Global invoice_item itemid customerid 1 1 # Acorn Enterprises 2 1 # Acorn Enterprises 3 3 # Squirrels 5000 Inc 4 4 # Savvy Squirrels ... Amount 1 2 1 hrs Professional Services Acorn Parts $10 2 invoiceid Description 1 invoice invoiceid 3 2 $50 4 3 5 hrs Professional Services Delivery Fee 5 3 $20 6 4 Replacement Acorn Premium Acorn $12.50 $17.50 $999
  18. 18. MyISAM Implementation customer customerid name 1 Acorn Enterprises 2 First National Squirrel Bank 3 Squirrels 5000 Inc 4 Savvy Squirrel Global invoice_item itemid customerid 1 1 # Acorn Enterprises 2 1 # Acorn Enterprises 3 3 # Squirrels 5000 Inc 4 4 # Savvy Squirrels ... Amount 1 2 1 hrs Professional Services Acorn Parts $10 2 invoiceid Description 1 invoice invoiceid 3 2 $50 4 3 5 hrs Professional Services Delivery Fee 5 3 $20 6 4 Replacement Acorn Premium Acorn $12.50 $17.50 $999
  19. 19. MyISAM • • • • Readers may co-exist. Writers are forbidden. Locks a table by default. Programs like mysqldump responsibly lock all tables they need access to before starting.
  20. 20. InnoDB Implementation customer Trx ID Rollback Ptr customerid name 123 ## 1 123 ## 2 Acorn Enterprises First National 123 ## 3 401 ## 4 Squirrel Bank Squirrels 5000 Inc Savvy Squirrel invoice_item Rollback Ptr Trx ID Rollback Ptr invoiceid customerid 123 ## 1 123 ## 2 1 # Acorn Enterprises 1 # Acorn 123 ## 3 401 ## 4 Enterprises 3 # Squirrels 5000 Inc 4 # Savvy ... itemid invoiceid Description Amount 123 ## 1 1 2 2 123 ## 3 2 123 ## 4 3 123 ## 5 3 401 ## 6 4 1 hrs Profess Acorn Parts 5 hrs Profess Deliver y Fee Replac ement Premiu m $10 123 ## invoice Trx ID $12.50 $50 $17.50 $20 $999
  21. 21. InnoDB • Hidden meta data attached to each row: • Row version • Rollback pointer to older versions if needed. • Every transaction gets allocated a monotonic Transaction ID.
  22. 22. InnoDB Implementation Backup process is TRX ID #394. Remembers it is not supposed to see anything newer than itself. New changes are permitted to continue. customer Trx ID Rollback Ptr customerid name 123 ## 1 123 ## 2 Acorn Enterprises First National 123 ## 3 401 ## 4 Squirrel Bank Squirrels 5000 Inc Savvy Squirrel invoice_item Rollback Ptr Trx ID Rollback Ptr invoiceid customerid 123 ## 1 123 ## 2 1 # Acorn Enterprises 1 # Acorn 123 ## 3 401 ## 4 Enterprises 3 # Squirrels 5000 Inc 4 # Savvy ... itemid invoiceid Description Amount 123 ## 1 1 2 2 123 ## 3 2 123 ## 4 3 123 ## 5 3 401 ## 6 4 1 hrs Profess Acorn Parts 5 hrs Profess Deliver y Fee Replac ement Premiu m $10 123 ## invoice Trx ID $12.50 $50 $17.50 $20 $999
  23. 23. MVCC Algorithm • Stands for “Multi-version concurrency control” - http:// en.wikipedia.org/wiki/ Multiversion_concurrency_control • Readers don’t block writers! • And actually writers don’t block readers, because they can read older versions.
  24. 24. MVCC (cont.) • Multiple versions of the same row may exist. • What you will see depends on the context of your transaction. Even the number of rows in a table depends on your context.
  25. 25. Live  Demo Copyright  ©  2012  Oracle  and/or  its  affiliates.  All  rights  reserved.
  26. 26. Deadlocks  &  Lock  Waits Copyright  ©  2012  Oracle  and/or  its  affiliates.  All  rights  reserved.
  27. 27. What is a deadlock? • A situation when two sessions are trying to acquire each-other’s respective locks. • Neither can proceed until the other backs down. • It’s no different to when two colleagues get in an irrational argument and neither will back down :)
  28. 28. What is a lock wait? • One step short of a deadlock. • You keep trying to do something, but it’s not in the budget this quarter? • Not quite sure of the analogy here, but you are free to retry. • But.. a fair amount of the time it still won’t work
  29. 29. Deadlock Detection • InnoDB as a storage engine will detect deadlocks. • Algorithm is Graph detection http://en.wikipedia.org/ wiki/Cycle_detection - happens virtually instantly. • Kills the transaction that modified the least rows. • All changes are rolled back - application developer must now handle/retry.
  30. 30. Preventing Deadlocks • Deadlock is a concurrency issue. • Same solutions as when you have colleagues arguing!
  31. 31. Solution #1 • Don’t make them work-together. • Reduce concurrency / run known conflicting workers in serial execution. X
  32. 32. Solution #2 • Don’t make them work on big projects together. • aka lower the stakes! • Locks are held for the duration of a transaction: • Put people who don’t work together only on small projects as needed (smaller transactions). • Reduce locking required for transactions (more efficient queries, better indexing).
  33. 33. Preventing Deadlocks (cont.) • Also, we should accept that some deadlocks are a fact of life. Continuing co-worker analogy: • Essential for business growth that people work together. • Have to be prepared to handle disputes. ! • MyISAM as a storage engine does not actually need deadlock detection (table locking). • Where does this leave concurrency?
  34. 34. Preventing Deadlocks (cont.) • You actually want databases to be optimistic. • That is set as few locks as possible, knowing that sometimes oopses will happen. • Alternative is more locks and less concurrency.
  35. 35. Handing Deadlocks • Database only owns responsibility of alerting application • It may not reapply transaction, since state has changed. • Application owns responsibility of handling errors. • Yet maybe 5% of applications do this :(
  36. 36. Lock Wait Timeout • Will proceed immediately if it can acquire the lock. • Since most transactions are less than 1 second, at 50 seconds seems unlikely retrying will work. • May need to voluntarily rollback and try again or give a user error.
  37. 37. “Requires a replay of the transaction” • • • • The problems probably won’t happen often. May be easier to kick back to the user: “try again - no changes have been applied”. Exception may be asynchronous workers? • Usually you’ve made the promise of work without opportunity to back-out.
  38. 38. ACID Copyright  ©  2012  Oracle  and/or  its  affiliates.  All  rights  reserved.
  39. 39. The “ACID” Contract • Atomicity - All or none. • Consistency - Generic term to say all constraints etc must be honoured (primary key, unique key). • Isolation - Changes in one transaction to not interfere with another transaction. • Durability - When you said that the atomic transaction was committed, you will honour it. Even if the server were to crash.
  40. 40. Atomicity • Most people know this one: • Transfer $20 from Account A --> Account B. • Multiple failure states... • The money may be subtracted from A, but has not reached B yet. • Wrap your code between START TRANSACTION and COMMIT and you get this solved for free.
  41. 41. Consistency • Not too much more to add on this one. • InnoDB has foreign key constraints • MySQL does not feature CHECK constraints where another database may allow more domain specific data validation here as well.
  42. 42. Isolation • Actually a very expensive feature for a database to implement (may require lots of locking, stale data kept around etc.) • Many databases make this somewhat tunable. • There are four levels of isolation describing what potential unsafe conditions are possible.
  43. 43. Isolation Levels • READ-UNCOMMITTED - No isolation • READ-COMMITTED - Can see other session’s committed rows. • REPEATABLE-READ - Won’t see other session’s committed rows once your session starts (default). • SERIALIZABLE - Reads actually lock. Can’t modify any data another session has changed.
  44. 44. Durable • This one’s actually tunable as well. • InnoDB will not lie by default. • It means on COMMIT everything should survive a power loss not be stuck in a buffer in memory.
  45. 45. Summary Copyright  ©  2012  Oracle  and/or  its  affiliates.  All  rights  reserved.
  46. 46. Application Responsibilities • Wrap every logical unit of work in START TRANSACTION and COMMIT. • Expect failure because failure is a database allowing concurrency. • LOCK WAIT TIMEOUT = Retry “may” be possible. • DEADLOCK = Everything must be retried.
  47. 47. <Insert Picture Here>
  48. 48. Mutexes, Latches & Semaphores • Common names given to database “internal locking”. • Today we are not going to differentiate between them. • Not unique to databases, required to provide “thread safety” in multi-threaded programs.
  49. 49. Example #1 • InnoDB buffer pool is full • Free space needs to be created in order to be able to load your desired page into memory. • In between freeing and loading, someone else fills desired slot.
  50. 50. Example #2 • When a page becomes too full, it may need to split data into other pages. • It is possible that there could be a thread deadlock if two pages left and right of each-other reach this condition at once. • Can’t decide who should proceed first.
  51. 51. Internal Locking (cont.) • When mutexes have high contention other tasks are not runable. • Which means that CPUs can appear idle, but nothing is happening.
  52. 52. Fixes in newer versions • Responsibility to fix is on MySQL Engineering Team. • High Priority for Us. • Some DBAs will have peripheral knowledge - know situations when it’s time to upgrade.
  53. 53. Time to Upgrade? • Kernel mutex split (5.6) • Lock open mutex split (5.6) • Buffer pool mutex contention reduced w/8 buffer pool instances (5.6) • Multiple rollback segments (5.5) • Log-sys mutex split (5.5) • Flush-list mutex split (5.5) More mentioned at: http://www.tocker.ca/2013/11/27/what-is-a-mutex-anyway.html
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×