Taming Cassandra@ odnoklassniki.ruby Oleg AnastasyevLead Platform Dev
5M users online250 000 pages/s, 50 ms      >100 Gbit/s ( >2x growth last year)> 4000 servers in 4 DCs       99% java
Data Storages we use• MS SQL 2005• Berkley DB based solution• Voldemort + Tarantool• Cassandra   forked since 0.6 at 2010
cassandra @ odnoklassniki• Social graph persistent store• Like AKA “Класс!” button servers• Photo Marks storage• ... and n...
Intro to cassandra     (oversimplified)
Cassandra : ring                  00              Gossip:          ★ Cluster membership192                              64...
Cassandra : partitioningPartitioner.toStringToken(key) : Token                                              00RP: Token=md...
Cassandra : replicationReplicationStrategy:                       00Token => InetAddress• Simple•RackAware                ...
Cassandra : writes                                                         Tunable        Client                          ...
Cassandra : reads     Client                                                00                                           D...
Cassandra: data model Keyspace (“Database”)                 1..* ColumnFamilyStore (“Table”) << dynamic >>                ...
Cassandra: local write(s)                                  name       value      ts           Write (Key, Column)         ...
Cassandra: local read(s)                                                name    value     timesta                         ...
SSTable anatomy SSTable-5-Filter.db         SSTable-5-Index.db            SSTable-5-Data.db  Bloom Filter                 ...
Conflict ResolutionSSTable “AccStatements-3456”              MemtableRowKey = “Oleg_Anastasyev”                RowKey = “Ol...
Conflict ResolutionSSTable “AccStatements-3456”                MemtabeRowKey = “Oleg_Anastasyev”                  RowKey = ...
Missed Update problemClient 1                                    Client 21. Read AccountStatement for    Key=”Oleg”       ...
Intro SummaryPros:                                 Cons:• No SPoF                             • No ACID, No rollbacks• Foc...
Tired of RTFM ?
Ok. Here is the problem
Ok. Here is the problem
Ok. Here is the problem
Ok. Here is the problem• PhotoMarks tablePhotoId:long   UserId:long    OwnerId:long   Mark:float       timestamp9999999999...
And the trouble is– 32 SQL clusters are serving it  (not counting stanby’s)– ... and they are close to their capacity in C...
Simple solutions ?• Add more SQL nodes – There are already 32 of them, add more 32 => 64 – Expensive (hardware + software)...
Cassandra !• Leveraging Pros  – Cheap NOT EXISTS ( bloom filter powered )  – CFs are enough to model this  – Data are store...
Data modelMarksByPhotoMarksByOwner (FreshFirst order)MarksUserIndex
Data modelMarksByPhotoKey                      Column           Column ValuephotoId:String           userId:byte[8]   owne...
#1: Column Bloom Filter• How ?– Stores (Key, Column name) pairs in SSTable *-Filter.db• Pros– No disk access on NOT EXISTS...
Profit!• New system– 8 cassandra nodes ( instead of 32 SQLs )– RF =2– Every row stored 2.5 times (in MarksByPhoto, ByOwner ...
#2: Split em !• How ?– Pre-partition every CF to 256 smaller CFs, based on key low byte  e.g. MarksByPhoto_00, _01, _FF– N...
#3: Uniform Replication          00                Thrift calls                                                   R1 192  ...
#3: Uniform Replication RF=3                      00                        Thrift calls                             R1   ...
What happens ?• On node failure  – Additional load on live node (THRIFT CPU, Hint storage)  – 300+ clients want to establi...
Own replication• How it works ? See source code at github/odnoklassniki Look for RackAwareOdklEvenStrategy• What it does ?...
That’s iton photo marks         Oleg Anastasyev         oa@odnoklassniki.ru         odnoklassniki.ru/oa         github.com...
Some problems left             application servers           PhotoMarksDAOImpl       THRIFT                   +hector     ...
Meet odnoklassniki-like                 application servers               PhotoMarksDAOImpl                 application se...
That’s it !Thank you !         Oleg Anastasyev         oa@odnoklassniki.ru         odnoklassniki.ru/oa         github.com/...
#3: Parallel readClient                                              00                                 Slow Data   slow r...
#3: Parallel read                                                          Slow ? FGC ?                                   ...
#3: Parallel readParallelReadResolver:1. asks all replicas for data                                                  002. ...
Cassandra : replicationReplicationStrategy:                       00Token => InetAddress                             22• S...
Data modelMarksByPhotoMarksByOwner (FreshFirst order)MarksUserIndex
Data modelMarksByOwner (FreshFirst order)Key               Column name                        Column ValueownerId:String  ...
Data modelMarksUserIndexKey              Column            Column ValueuserId:String    photoId:byte[8]   “” : byte[0]    ...
Other extensions•   Node local Maintenance Manager•   Commit log archive•   Data snapshot backup•   Local CF storages
Upcoming SlideShare
Loading in...5
×

Taming Cassandra

1,006

Published on

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,006
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
29
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • \n
  • 150 tomcats\n
  • 150 tomcats\n
  • &amp;#x412; &amp;#x440;&amp;#x430;&amp;#x431;&amp;#x43E;&amp;#x442;&amp;#x435; SQL &amp;#x441;&amp;#x435;&amp;#x440;&amp;#x432;&amp;#x435;&amp;#x440;&amp;#x43E;&amp;#x432;, &amp;#x445;&amp;#x440;&amp;#x430;&amp;#x43D;&amp;#x44F;&amp;#x449;&amp;#x438;&amp;#x445; &amp;#x43C;&amp;#x435;&amp;#x442;&amp;#x430;&amp;#x434;&amp;#x430;&amp;#x43D;&amp;#x43D;&amp;#x44B;&amp;#x435; &amp;#x43F;&amp;#x43E; &amp;#x444;&amp;#x43E;&amp;#x442;&amp;#x43E;&amp;#x433;&amp;#x440;&amp;#x430;&amp;#x444;&amp;#x438;&amp;#x44F;&amp;#x43C; &amp;#x438; &amp;#x441;&amp;#x432;&amp;#x44F;&amp;#x437;&amp;#x430;&amp;#x43D;&amp;#x43D;&amp;#x44B;&amp;#x435; &amp;#x441;&amp;#x443;&amp;#x449;&amp;#x43D;&amp;#x43E;&amp;#x441;&amp;#x442;&amp;#x438;(&amp;#x43E;&amp;#x446;&amp;#x435;&amp;#x43D;&amp;#x43A;&amp;#x438;, &amp;#x440;&amp;#x435;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x44F; &amp;#x43C;&amp;#x43E;&amp;#x434;&amp;#x435;&amp;#x440;&amp;#x430;&amp;#x442;&amp;#x43E;&amp;#x440;&amp;#x43E;&amp;#x432;), &amp;#x43D;&amp;#x430;&amp;#x431;&amp;#x43B;&amp;#x44E;&amp;#x434;&amp;#x430;&amp;#x44E;&amp;#x442;&amp;#x441;&amp;#x44F; &amp;#x43F;&amp;#x440;&amp;#x43E;&amp;#x431;&amp;#x43B;&amp;#x435;&amp;#x43C;&amp;#x44B;: &amp;#x43F;&amp;#x43E;&amp;#x432;&amp;#x44B;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x43D;&amp;#x430;&amp;#x44F; &amp;#x43D;&amp;#x430;&amp;#x433;&amp;#x440;&amp;#x443;&amp;#x437;&amp;#x43A;&amp;#x430;, &amp;#x43F;&amp;#x43E;&amp;#x432;&amp;#x44B;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x43D;&amp;#x430;&amp;#x44F; &amp;#x434;&amp;#x438;&amp;#x441;&amp;#x43A;&amp;#x43E;&amp;#x432;&amp;#x430;&amp;#x44F; &amp;#x43E;&amp;#x447;&amp;#x435;&amp;#x440;&amp;#x435;&amp;#x434;&amp;#x44C;, &amp;#x443;&amp;#x445;&amp;#x43E;&amp;#x436;&amp;#x434;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x435; &amp;#x43F;&amp;#x440;&amp;#x43E;&amp;#x446;&amp;#x435;&amp;#x441;&amp;#x441;&amp;#x430; &amp;#x432; &amp;#x43E;&amp;#x431;&amp;#x43B;&amp;#x430;&amp;#x441;&amp;#x442;&amp;#x44C; &amp;#x43F;&amp;#x43E;&amp;#x434;&amp;#x43A;&amp;#x430;&amp;#x447;&amp;#x43A;&amp;#x438; (swap), &amp;#x43F;&amp;#x430;&amp;#x434;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x44F;(&amp;#x432; &amp;#x442;&amp;#x43E;&amp;#x43C; &amp;#x447;&amp;#x438;&amp;#x441;&amp;#x43B;&amp;#x435; &amp;#x43C;&amp;#x43D;&amp;#x43E;&amp;#x433;&amp;#x43E;&amp;#x43A;&amp;#x440;&amp;#x430;&amp;#x442;&amp;#x43D;&amp;#x44B;&amp;#x435;), &quot;&amp;#x441;&amp;#x438;&amp;#x43D;&amp;#x438;&amp;#x439; &amp;#x44D;&amp;#x43A;&amp;#x440;&amp;#x430;&amp;#x43D; &amp;#x441;&amp;#x43C;&amp;#x435;&amp;#x440;&amp;#x442;&amp;#x438;&quot;.\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • &amp;#x412; &amp;#x440;&amp;#x430;&amp;#x431;&amp;#x43E;&amp;#x442;&amp;#x435; SQL &amp;#x441;&amp;#x435;&amp;#x440;&amp;#x432;&amp;#x435;&amp;#x440;&amp;#x43E;&amp;#x432;, &amp;#x445;&amp;#x440;&amp;#x430;&amp;#x43D;&amp;#x44F;&amp;#x449;&amp;#x438;&amp;#x445; &amp;#x43C;&amp;#x435;&amp;#x442;&amp;#x430;&amp;#x434;&amp;#x430;&amp;#x43D;&amp;#x43D;&amp;#x44B;&amp;#x435; &amp;#x43F;&amp;#x43E; &amp;#x444;&amp;#x43E;&amp;#x442;&amp;#x43E;&amp;#x433;&amp;#x440;&amp;#x430;&amp;#x444;&amp;#x438;&amp;#x44F;&amp;#x43C; &amp;#x438; &amp;#x441;&amp;#x432;&amp;#x44F;&amp;#x437;&amp;#x430;&amp;#x43D;&amp;#x43D;&amp;#x44B;&amp;#x435; &amp;#x441;&amp;#x443;&amp;#x449;&amp;#x43D;&amp;#x43E;&amp;#x441;&amp;#x442;&amp;#x438;(&amp;#x43E;&amp;#x446;&amp;#x435;&amp;#x43D;&amp;#x43A;&amp;#x438;, &amp;#x440;&amp;#x435;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x44F; &amp;#x43C;&amp;#x43E;&amp;#x434;&amp;#x435;&amp;#x440;&amp;#x430;&amp;#x442;&amp;#x43E;&amp;#x440;&amp;#x43E;&amp;#x432;), &amp;#x43D;&amp;#x430;&amp;#x431;&amp;#x43B;&amp;#x44E;&amp;#x434;&amp;#x430;&amp;#x44E;&amp;#x442;&amp;#x441;&amp;#x44F; &amp;#x43F;&amp;#x440;&amp;#x43E;&amp;#x431;&amp;#x43B;&amp;#x435;&amp;#x43C;&amp;#x44B;: &amp;#x43F;&amp;#x43E;&amp;#x432;&amp;#x44B;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x43D;&amp;#x430;&amp;#x44F; &amp;#x43D;&amp;#x430;&amp;#x433;&amp;#x440;&amp;#x443;&amp;#x437;&amp;#x43A;&amp;#x430;, &amp;#x43F;&amp;#x43E;&amp;#x432;&amp;#x44B;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x43D;&amp;#x430;&amp;#x44F; &amp;#x434;&amp;#x438;&amp;#x441;&amp;#x43A;&amp;#x43E;&amp;#x432;&amp;#x430;&amp;#x44F; &amp;#x43E;&amp;#x447;&amp;#x435;&amp;#x440;&amp;#x435;&amp;#x434;&amp;#x44C;, &amp;#x443;&amp;#x445;&amp;#x43E;&amp;#x436;&amp;#x434;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x435; &amp;#x43F;&amp;#x440;&amp;#x43E;&amp;#x446;&amp;#x435;&amp;#x441;&amp;#x441;&amp;#x430; &amp;#x432; &amp;#x43E;&amp;#x431;&amp;#x43B;&amp;#x430;&amp;#x441;&amp;#x442;&amp;#x44C; &amp;#x43F;&amp;#x43E;&amp;#x434;&amp;#x43A;&amp;#x430;&amp;#x447;&amp;#x43A;&amp;#x438; (swap), &amp;#x43F;&amp;#x430;&amp;#x434;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x44F;(&amp;#x432; &amp;#x442;&amp;#x43E;&amp;#x43C; &amp;#x447;&amp;#x438;&amp;#x441;&amp;#x43B;&amp;#x435; &amp;#x43C;&amp;#x43D;&amp;#x43E;&amp;#x433;&amp;#x43E;&amp;#x43A;&amp;#x440;&amp;#x430;&amp;#x442;&amp;#x43D;&amp;#x44B;&amp;#x435;), &quot;&amp;#x441;&amp;#x438;&amp;#x43D;&amp;#x438;&amp;#x439; &amp;#x44D;&amp;#x43A;&amp;#x440;&amp;#x430;&amp;#x43D; &amp;#x441;&amp;#x43C;&amp;#x435;&amp;#x440;&amp;#x442;&amp;#x438;&quot;.\n
  • &amp;#x412; &amp;#x440;&amp;#x430;&amp;#x431;&amp;#x43E;&amp;#x442;&amp;#x435; SQL &amp;#x441;&amp;#x435;&amp;#x440;&amp;#x432;&amp;#x435;&amp;#x440;&amp;#x43E;&amp;#x432;, &amp;#x445;&amp;#x440;&amp;#x430;&amp;#x43D;&amp;#x44F;&amp;#x449;&amp;#x438;&amp;#x445; &amp;#x43C;&amp;#x435;&amp;#x442;&amp;#x430;&amp;#x434;&amp;#x430;&amp;#x43D;&amp;#x43D;&amp;#x44B;&amp;#x435; &amp;#x43F;&amp;#x43E; &amp;#x444;&amp;#x43E;&amp;#x442;&amp;#x43E;&amp;#x433;&amp;#x440;&amp;#x430;&amp;#x444;&amp;#x438;&amp;#x44F;&amp;#x43C; &amp;#x438; &amp;#x441;&amp;#x432;&amp;#x44F;&amp;#x437;&amp;#x430;&amp;#x43D;&amp;#x43D;&amp;#x44B;&amp;#x435; &amp;#x441;&amp;#x443;&amp;#x449;&amp;#x43D;&amp;#x43E;&amp;#x441;&amp;#x442;&amp;#x438;(&amp;#x43E;&amp;#x446;&amp;#x435;&amp;#x43D;&amp;#x43A;&amp;#x438;, &amp;#x440;&amp;#x435;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x44F; &amp;#x43C;&amp;#x43E;&amp;#x434;&amp;#x435;&amp;#x440;&amp;#x430;&amp;#x442;&amp;#x43E;&amp;#x440;&amp;#x43E;&amp;#x432;), &amp;#x43D;&amp;#x430;&amp;#x431;&amp;#x43B;&amp;#x44E;&amp;#x434;&amp;#x430;&amp;#x44E;&amp;#x442;&amp;#x441;&amp;#x44F; &amp;#x43F;&amp;#x440;&amp;#x43E;&amp;#x431;&amp;#x43B;&amp;#x435;&amp;#x43C;&amp;#x44B;: &amp;#x43F;&amp;#x43E;&amp;#x432;&amp;#x44B;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x43D;&amp;#x430;&amp;#x44F; &amp;#x43D;&amp;#x430;&amp;#x433;&amp;#x440;&amp;#x443;&amp;#x437;&amp;#x43A;&amp;#x430;, &amp;#x43F;&amp;#x43E;&amp;#x432;&amp;#x44B;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x43D;&amp;#x430;&amp;#x44F; &amp;#x434;&amp;#x438;&amp;#x441;&amp;#x43A;&amp;#x43E;&amp;#x432;&amp;#x430;&amp;#x44F; &amp;#x43E;&amp;#x447;&amp;#x435;&amp;#x440;&amp;#x435;&amp;#x434;&amp;#x44C;, &amp;#x443;&amp;#x445;&amp;#x43E;&amp;#x436;&amp;#x434;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x435; &amp;#x43F;&amp;#x440;&amp;#x43E;&amp;#x446;&amp;#x435;&amp;#x441;&amp;#x441;&amp;#x430; &amp;#x432; &amp;#x43E;&amp;#x431;&amp;#x43B;&amp;#x430;&amp;#x441;&amp;#x442;&amp;#x44C; &amp;#x43F;&amp;#x43E;&amp;#x434;&amp;#x43A;&amp;#x430;&amp;#x447;&amp;#x43A;&amp;#x438; (swap), &amp;#x43F;&amp;#x430;&amp;#x434;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x44F;(&amp;#x432; &amp;#x442;&amp;#x43E;&amp;#x43C; &amp;#x447;&amp;#x438;&amp;#x441;&amp;#x43B;&amp;#x435; &amp;#x43C;&amp;#x43D;&amp;#x43E;&amp;#x433;&amp;#x43E;&amp;#x43A;&amp;#x440;&amp;#x430;&amp;#x442;&amp;#x43D;&amp;#x44B;&amp;#x435;), &quot;&amp;#x441;&amp;#x438;&amp;#x43D;&amp;#x438;&amp;#x439; &amp;#x44D;&amp;#x43A;&amp;#x440;&amp;#x430;&amp;#x43D; &amp;#x441;&amp;#x43C;&amp;#x435;&amp;#x440;&amp;#x442;&amp;#x438;&quot;.\n
  • &amp;#x412; &amp;#x440;&amp;#x430;&amp;#x431;&amp;#x43E;&amp;#x442;&amp;#x435; SQL &amp;#x441;&amp;#x435;&amp;#x440;&amp;#x432;&amp;#x435;&amp;#x440;&amp;#x43E;&amp;#x432;, &amp;#x445;&amp;#x440;&amp;#x430;&amp;#x43D;&amp;#x44F;&amp;#x449;&amp;#x438;&amp;#x445; &amp;#x43C;&amp;#x435;&amp;#x442;&amp;#x430;&amp;#x434;&amp;#x430;&amp;#x43D;&amp;#x43D;&amp;#x44B;&amp;#x435; &amp;#x43F;&amp;#x43E; &amp;#x444;&amp;#x43E;&amp;#x442;&amp;#x43E;&amp;#x433;&amp;#x440;&amp;#x430;&amp;#x444;&amp;#x438;&amp;#x44F;&amp;#x43C; &amp;#x438; &amp;#x441;&amp;#x432;&amp;#x44F;&amp;#x437;&amp;#x430;&amp;#x43D;&amp;#x43D;&amp;#x44B;&amp;#x435; &amp;#x441;&amp;#x443;&amp;#x449;&amp;#x43D;&amp;#x43E;&amp;#x441;&amp;#x442;&amp;#x438;(&amp;#x43E;&amp;#x446;&amp;#x435;&amp;#x43D;&amp;#x43A;&amp;#x438;, &amp;#x440;&amp;#x435;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x44F; &amp;#x43C;&amp;#x43E;&amp;#x434;&amp;#x435;&amp;#x440;&amp;#x430;&amp;#x442;&amp;#x43E;&amp;#x440;&amp;#x43E;&amp;#x432;), &amp;#x43D;&amp;#x430;&amp;#x431;&amp;#x43B;&amp;#x44E;&amp;#x434;&amp;#x430;&amp;#x44E;&amp;#x442;&amp;#x441;&amp;#x44F; &amp;#x43F;&amp;#x440;&amp;#x43E;&amp;#x431;&amp;#x43B;&amp;#x435;&amp;#x43C;&amp;#x44B;: &amp;#x43F;&amp;#x43E;&amp;#x432;&amp;#x44B;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x43D;&amp;#x430;&amp;#x44F; &amp;#x43D;&amp;#x430;&amp;#x433;&amp;#x440;&amp;#x443;&amp;#x437;&amp;#x43A;&amp;#x430;, &amp;#x43F;&amp;#x43E;&amp;#x432;&amp;#x44B;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x43D;&amp;#x430;&amp;#x44F; &amp;#x434;&amp;#x438;&amp;#x441;&amp;#x43A;&amp;#x43E;&amp;#x432;&amp;#x430;&amp;#x44F; &amp;#x43E;&amp;#x447;&amp;#x435;&amp;#x440;&amp;#x435;&amp;#x434;&amp;#x44C;, &amp;#x443;&amp;#x445;&amp;#x43E;&amp;#x436;&amp;#x434;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x435; &amp;#x43F;&amp;#x440;&amp;#x43E;&amp;#x446;&amp;#x435;&amp;#x441;&amp;#x441;&amp;#x430; &amp;#x432; &amp;#x43E;&amp;#x431;&amp;#x43B;&amp;#x430;&amp;#x441;&amp;#x442;&amp;#x44C; &amp;#x43F;&amp;#x43E;&amp;#x434;&amp;#x43A;&amp;#x430;&amp;#x447;&amp;#x43A;&amp;#x438; (swap), &amp;#x43F;&amp;#x430;&amp;#x434;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x44F;(&amp;#x432; &amp;#x442;&amp;#x43E;&amp;#x43C; &amp;#x447;&amp;#x438;&amp;#x441;&amp;#x43B;&amp;#x435; &amp;#x43C;&amp;#x43D;&amp;#x43E;&amp;#x433;&amp;#x43E;&amp;#x43A;&amp;#x440;&amp;#x430;&amp;#x442;&amp;#x43D;&amp;#x44B;&amp;#x435;), &quot;&amp;#x441;&amp;#x438;&amp;#x43D;&amp;#x438;&amp;#x439; &amp;#x44D;&amp;#x43A;&amp;#x440;&amp;#x430;&amp;#x43D; &amp;#x441;&amp;#x43C;&amp;#x435;&amp;#x440;&amp;#x442;&amp;#x438;&quot;.\n
  • &amp;#x412; &amp;#x440;&amp;#x430;&amp;#x431;&amp;#x43E;&amp;#x442;&amp;#x435; SQL &amp;#x441;&amp;#x435;&amp;#x440;&amp;#x432;&amp;#x435;&amp;#x440;&amp;#x43E;&amp;#x432;, &amp;#x445;&amp;#x440;&amp;#x430;&amp;#x43D;&amp;#x44F;&amp;#x449;&amp;#x438;&amp;#x445; &amp;#x43C;&amp;#x435;&amp;#x442;&amp;#x430;&amp;#x434;&amp;#x430;&amp;#x43D;&amp;#x43D;&amp;#x44B;&amp;#x435; &amp;#x43F;&amp;#x43E; &amp;#x444;&amp;#x43E;&amp;#x442;&amp;#x43E;&amp;#x433;&amp;#x440;&amp;#x430;&amp;#x444;&amp;#x438;&amp;#x44F;&amp;#x43C; &amp;#x438; &amp;#x441;&amp;#x432;&amp;#x44F;&amp;#x437;&amp;#x430;&amp;#x43D;&amp;#x43D;&amp;#x44B;&amp;#x435; &amp;#x441;&amp;#x443;&amp;#x449;&amp;#x43D;&amp;#x43E;&amp;#x441;&amp;#x442;&amp;#x438;(&amp;#x43E;&amp;#x446;&amp;#x435;&amp;#x43D;&amp;#x43A;&amp;#x438;, &amp;#x440;&amp;#x435;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x44F; &amp;#x43C;&amp;#x43E;&amp;#x434;&amp;#x435;&amp;#x440;&amp;#x430;&amp;#x442;&amp;#x43E;&amp;#x440;&amp;#x43E;&amp;#x432;), &amp;#x43D;&amp;#x430;&amp;#x431;&amp;#x43B;&amp;#x44E;&amp;#x434;&amp;#x430;&amp;#x44E;&amp;#x442;&amp;#x441;&amp;#x44F; &amp;#x43F;&amp;#x440;&amp;#x43E;&amp;#x431;&amp;#x43B;&amp;#x435;&amp;#x43C;&amp;#x44B;: &amp;#x43F;&amp;#x43E;&amp;#x432;&amp;#x44B;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x43D;&amp;#x430;&amp;#x44F; &amp;#x43D;&amp;#x430;&amp;#x433;&amp;#x440;&amp;#x443;&amp;#x437;&amp;#x43A;&amp;#x430;, &amp;#x43F;&amp;#x43E;&amp;#x432;&amp;#x44B;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x43D;&amp;#x430;&amp;#x44F; &amp;#x434;&amp;#x438;&amp;#x441;&amp;#x43A;&amp;#x43E;&amp;#x432;&amp;#x430;&amp;#x44F; &amp;#x43E;&amp;#x447;&amp;#x435;&amp;#x440;&amp;#x435;&amp;#x434;&amp;#x44C;, &amp;#x443;&amp;#x445;&amp;#x43E;&amp;#x436;&amp;#x434;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x435; &amp;#x43F;&amp;#x440;&amp;#x43E;&amp;#x446;&amp;#x435;&amp;#x441;&amp;#x441;&amp;#x430; &amp;#x432; &amp;#x43E;&amp;#x431;&amp;#x43B;&amp;#x430;&amp;#x441;&amp;#x442;&amp;#x44C; &amp;#x43F;&amp;#x43E;&amp;#x434;&amp;#x43A;&amp;#x430;&amp;#x447;&amp;#x43A;&amp;#x438; (swap), &amp;#x43F;&amp;#x430;&amp;#x434;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x44F;(&amp;#x432; &amp;#x442;&amp;#x43E;&amp;#x43C; &amp;#x447;&amp;#x438;&amp;#x441;&amp;#x43B;&amp;#x435; &amp;#x43C;&amp;#x43D;&amp;#x43E;&amp;#x433;&amp;#x43E;&amp;#x43A;&amp;#x440;&amp;#x430;&amp;#x442;&amp;#x43D;&amp;#x44B;&amp;#x435;), &quot;&amp;#x441;&amp;#x438;&amp;#x43D;&amp;#x438;&amp;#x439; &amp;#x44D;&amp;#x43A;&amp;#x440;&amp;#x430;&amp;#x43D; &amp;#x441;&amp;#x43C;&amp;#x435;&amp;#x440;&amp;#x442;&amp;#x438;&quot;.\n
  • &amp;#x412; &amp;#x440;&amp;#x430;&amp;#x431;&amp;#x43E;&amp;#x442;&amp;#x435; SQL &amp;#x441;&amp;#x435;&amp;#x440;&amp;#x432;&amp;#x435;&amp;#x440;&amp;#x43E;&amp;#x432;, &amp;#x445;&amp;#x440;&amp;#x430;&amp;#x43D;&amp;#x44F;&amp;#x449;&amp;#x438;&amp;#x445; &amp;#x43C;&amp;#x435;&amp;#x442;&amp;#x430;&amp;#x434;&amp;#x430;&amp;#x43D;&amp;#x43D;&amp;#x44B;&amp;#x435; &amp;#x43F;&amp;#x43E; &amp;#x444;&amp;#x43E;&amp;#x442;&amp;#x43E;&amp;#x433;&amp;#x440;&amp;#x430;&amp;#x444;&amp;#x438;&amp;#x44F;&amp;#x43C; &amp;#x438; &amp;#x441;&amp;#x432;&amp;#x44F;&amp;#x437;&amp;#x430;&amp;#x43D;&amp;#x43D;&amp;#x44B;&amp;#x435; &amp;#x441;&amp;#x443;&amp;#x449;&amp;#x43D;&amp;#x43E;&amp;#x441;&amp;#x442;&amp;#x438;(&amp;#x43E;&amp;#x446;&amp;#x435;&amp;#x43D;&amp;#x43A;&amp;#x438;, &amp;#x440;&amp;#x435;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x44F; &amp;#x43C;&amp;#x43E;&amp;#x434;&amp;#x435;&amp;#x440;&amp;#x430;&amp;#x442;&amp;#x43E;&amp;#x440;&amp;#x43E;&amp;#x432;), &amp;#x43D;&amp;#x430;&amp;#x431;&amp;#x43B;&amp;#x44E;&amp;#x434;&amp;#x430;&amp;#x44E;&amp;#x442;&amp;#x441;&amp;#x44F; &amp;#x43F;&amp;#x440;&amp;#x43E;&amp;#x431;&amp;#x43B;&amp;#x435;&amp;#x43C;&amp;#x44B;: &amp;#x43F;&amp;#x43E;&amp;#x432;&amp;#x44B;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x43D;&amp;#x430;&amp;#x44F; &amp;#x43D;&amp;#x430;&amp;#x433;&amp;#x440;&amp;#x443;&amp;#x437;&amp;#x43A;&amp;#x430;, &amp;#x43F;&amp;#x43E;&amp;#x432;&amp;#x44B;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x43D;&amp;#x430;&amp;#x44F; &amp;#x434;&amp;#x438;&amp;#x441;&amp;#x43A;&amp;#x43E;&amp;#x432;&amp;#x430;&amp;#x44F; &amp;#x43E;&amp;#x447;&amp;#x435;&amp;#x440;&amp;#x435;&amp;#x434;&amp;#x44C;, &amp;#x443;&amp;#x445;&amp;#x43E;&amp;#x436;&amp;#x434;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x435; &amp;#x43F;&amp;#x440;&amp;#x43E;&amp;#x446;&amp;#x435;&amp;#x441;&amp;#x441;&amp;#x430; &amp;#x432; &amp;#x43E;&amp;#x431;&amp;#x43B;&amp;#x430;&amp;#x441;&amp;#x442;&amp;#x44C; &amp;#x43F;&amp;#x43E;&amp;#x434;&amp;#x43A;&amp;#x430;&amp;#x447;&amp;#x43A;&amp;#x438; (swap), &amp;#x43F;&amp;#x430;&amp;#x434;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x44F;(&amp;#x432; &amp;#x442;&amp;#x43E;&amp;#x43C; &amp;#x447;&amp;#x438;&amp;#x441;&amp;#x43B;&amp;#x435; &amp;#x43C;&amp;#x43D;&amp;#x43E;&amp;#x433;&amp;#x43E;&amp;#x43A;&amp;#x440;&amp;#x430;&amp;#x442;&amp;#x43D;&amp;#x44B;&amp;#x435;), &quot;&amp;#x441;&amp;#x438;&amp;#x43D;&amp;#x438;&amp;#x439; &amp;#x44D;&amp;#x43A;&amp;#x440;&amp;#x430;&amp;#x43D; &amp;#x441;&amp;#x43C;&amp;#x435;&amp;#x440;&amp;#x442;&amp;#x438;&quot;.\n
  • &amp;#x412; &amp;#x440;&amp;#x430;&amp;#x431;&amp;#x43E;&amp;#x442;&amp;#x435; SQL &amp;#x441;&amp;#x435;&amp;#x440;&amp;#x432;&amp;#x435;&amp;#x440;&amp;#x43E;&amp;#x432;, &amp;#x445;&amp;#x440;&amp;#x430;&amp;#x43D;&amp;#x44F;&amp;#x449;&amp;#x438;&amp;#x445; &amp;#x43C;&amp;#x435;&amp;#x442;&amp;#x430;&amp;#x434;&amp;#x430;&amp;#x43D;&amp;#x43D;&amp;#x44B;&amp;#x435; &amp;#x43F;&amp;#x43E; &amp;#x444;&amp;#x43E;&amp;#x442;&amp;#x43E;&amp;#x433;&amp;#x440;&amp;#x430;&amp;#x444;&amp;#x438;&amp;#x44F;&amp;#x43C; &amp;#x438; &amp;#x441;&amp;#x432;&amp;#x44F;&amp;#x437;&amp;#x430;&amp;#x43D;&amp;#x43D;&amp;#x44B;&amp;#x435; &amp;#x441;&amp;#x443;&amp;#x449;&amp;#x43D;&amp;#x43E;&amp;#x441;&amp;#x442;&amp;#x438;(&amp;#x43E;&amp;#x446;&amp;#x435;&amp;#x43D;&amp;#x43A;&amp;#x438;, &amp;#x440;&amp;#x435;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x44F; &amp;#x43C;&amp;#x43E;&amp;#x434;&amp;#x435;&amp;#x440;&amp;#x430;&amp;#x442;&amp;#x43E;&amp;#x440;&amp;#x43E;&amp;#x432;), &amp;#x43D;&amp;#x430;&amp;#x431;&amp;#x43B;&amp;#x44E;&amp;#x434;&amp;#x430;&amp;#x44E;&amp;#x442;&amp;#x441;&amp;#x44F; &amp;#x43F;&amp;#x440;&amp;#x43E;&amp;#x431;&amp;#x43B;&amp;#x435;&amp;#x43C;&amp;#x44B;: &amp;#x43F;&amp;#x43E;&amp;#x432;&amp;#x44B;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x43D;&amp;#x430;&amp;#x44F; &amp;#x43D;&amp;#x430;&amp;#x433;&amp;#x440;&amp;#x443;&amp;#x437;&amp;#x43A;&amp;#x430;, &amp;#x43F;&amp;#x43E;&amp;#x432;&amp;#x44B;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x43D;&amp;#x430;&amp;#x44F; &amp;#x434;&amp;#x438;&amp;#x441;&amp;#x43A;&amp;#x43E;&amp;#x432;&amp;#x430;&amp;#x44F; &amp;#x43E;&amp;#x447;&amp;#x435;&amp;#x440;&amp;#x435;&amp;#x434;&amp;#x44C;, &amp;#x443;&amp;#x445;&amp;#x43E;&amp;#x436;&amp;#x434;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x435; &amp;#x43F;&amp;#x440;&amp;#x43E;&amp;#x446;&amp;#x435;&amp;#x441;&amp;#x441;&amp;#x430; &amp;#x432; &amp;#x43E;&amp;#x431;&amp;#x43B;&amp;#x430;&amp;#x441;&amp;#x442;&amp;#x44C; &amp;#x43F;&amp;#x43E;&amp;#x434;&amp;#x43A;&amp;#x430;&amp;#x447;&amp;#x43A;&amp;#x438; (swap), &amp;#x43F;&amp;#x430;&amp;#x434;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x44F;(&amp;#x432; &amp;#x442;&amp;#x43E;&amp;#x43C; &amp;#x447;&amp;#x438;&amp;#x441;&amp;#x43B;&amp;#x435; &amp;#x43C;&amp;#x43D;&amp;#x43E;&amp;#x433;&amp;#x43E;&amp;#x43A;&amp;#x440;&amp;#x430;&amp;#x442;&amp;#x43D;&amp;#x44B;&amp;#x435;), &quot;&amp;#x441;&amp;#x438;&amp;#x43D;&amp;#x438;&amp;#x439; &amp;#x44D;&amp;#x43A;&amp;#x440;&amp;#x430;&amp;#x43D; &amp;#x441;&amp;#x43C;&amp;#x435;&amp;#x440;&amp;#x442;&amp;#x438;&quot;.\n
  • &amp;#x412; &amp;#x440;&amp;#x430;&amp;#x431;&amp;#x43E;&amp;#x442;&amp;#x435; SQL &amp;#x441;&amp;#x435;&amp;#x440;&amp;#x432;&amp;#x435;&amp;#x440;&amp;#x43E;&amp;#x432;, &amp;#x445;&amp;#x440;&amp;#x430;&amp;#x43D;&amp;#x44F;&amp;#x449;&amp;#x438;&amp;#x445; &amp;#x43C;&amp;#x435;&amp;#x442;&amp;#x430;&amp;#x434;&amp;#x430;&amp;#x43D;&amp;#x43D;&amp;#x44B;&amp;#x435; &amp;#x43F;&amp;#x43E; &amp;#x444;&amp;#x43E;&amp;#x442;&amp;#x43E;&amp;#x433;&amp;#x440;&amp;#x430;&amp;#x444;&amp;#x438;&amp;#x44F;&amp;#x43C; &amp;#x438; &amp;#x441;&amp;#x432;&amp;#x44F;&amp;#x437;&amp;#x430;&amp;#x43D;&amp;#x43D;&amp;#x44B;&amp;#x435; &amp;#x441;&amp;#x443;&amp;#x449;&amp;#x43D;&amp;#x43E;&amp;#x441;&amp;#x442;&amp;#x438;(&amp;#x43E;&amp;#x446;&amp;#x435;&amp;#x43D;&amp;#x43A;&amp;#x438;, &amp;#x440;&amp;#x435;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x44F; &amp;#x43C;&amp;#x43E;&amp;#x434;&amp;#x435;&amp;#x440;&amp;#x430;&amp;#x442;&amp;#x43E;&amp;#x440;&amp;#x43E;&amp;#x432;), &amp;#x43D;&amp;#x430;&amp;#x431;&amp;#x43B;&amp;#x44E;&amp;#x434;&amp;#x430;&amp;#x44E;&amp;#x442;&amp;#x441;&amp;#x44F; &amp;#x43F;&amp;#x440;&amp;#x43E;&amp;#x431;&amp;#x43B;&amp;#x435;&amp;#x43C;&amp;#x44B;: &amp;#x43F;&amp;#x43E;&amp;#x432;&amp;#x44B;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x43D;&amp;#x430;&amp;#x44F; &amp;#x43D;&amp;#x430;&amp;#x433;&amp;#x440;&amp;#x443;&amp;#x437;&amp;#x43A;&amp;#x430;, &amp;#x43F;&amp;#x43E;&amp;#x432;&amp;#x44B;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x43D;&amp;#x430;&amp;#x44F; &amp;#x434;&amp;#x438;&amp;#x441;&amp;#x43A;&amp;#x43E;&amp;#x432;&amp;#x430;&amp;#x44F; &amp;#x43E;&amp;#x447;&amp;#x435;&amp;#x440;&amp;#x435;&amp;#x434;&amp;#x44C;, &amp;#x443;&amp;#x445;&amp;#x43E;&amp;#x436;&amp;#x434;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x435; &amp;#x43F;&amp;#x440;&amp;#x43E;&amp;#x446;&amp;#x435;&amp;#x441;&amp;#x441;&amp;#x430; &amp;#x432; &amp;#x43E;&amp;#x431;&amp;#x43B;&amp;#x430;&amp;#x441;&amp;#x442;&amp;#x44C; &amp;#x43F;&amp;#x43E;&amp;#x434;&amp;#x43A;&amp;#x430;&amp;#x447;&amp;#x43A;&amp;#x438; (swap), &amp;#x43F;&amp;#x430;&amp;#x434;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x44F;(&amp;#x432; &amp;#x442;&amp;#x43E;&amp;#x43C; &amp;#x447;&amp;#x438;&amp;#x441;&amp;#x43B;&amp;#x435; &amp;#x43C;&amp;#x43D;&amp;#x43E;&amp;#x433;&amp;#x43E;&amp;#x43A;&amp;#x440;&amp;#x430;&amp;#x442;&amp;#x43D;&amp;#x44B;&amp;#x435;), &quot;&amp;#x441;&amp;#x438;&amp;#x43D;&amp;#x438;&amp;#x439; &amp;#x44D;&amp;#x43A;&amp;#x440;&amp;#x430;&amp;#x43D; &amp;#x441;&amp;#x43C;&amp;#x435;&amp;#x440;&amp;#x442;&amp;#x438;&quot;.\n
  • &amp;#x412; &amp;#x440;&amp;#x430;&amp;#x431;&amp;#x43E;&amp;#x442;&amp;#x435; SQL &amp;#x441;&amp;#x435;&amp;#x440;&amp;#x432;&amp;#x435;&amp;#x440;&amp;#x43E;&amp;#x432;, &amp;#x445;&amp;#x440;&amp;#x430;&amp;#x43D;&amp;#x44F;&amp;#x449;&amp;#x438;&amp;#x445; &amp;#x43C;&amp;#x435;&amp;#x442;&amp;#x430;&amp;#x434;&amp;#x430;&amp;#x43D;&amp;#x43D;&amp;#x44B;&amp;#x435; &amp;#x43F;&amp;#x43E; &amp;#x444;&amp;#x43E;&amp;#x442;&amp;#x43E;&amp;#x433;&amp;#x440;&amp;#x430;&amp;#x444;&amp;#x438;&amp;#x44F;&amp;#x43C; &amp;#x438; &amp;#x441;&amp;#x432;&amp;#x44F;&amp;#x437;&amp;#x430;&amp;#x43D;&amp;#x43D;&amp;#x44B;&amp;#x435; &amp;#x441;&amp;#x443;&amp;#x449;&amp;#x43D;&amp;#x43E;&amp;#x441;&amp;#x442;&amp;#x438;(&amp;#x43E;&amp;#x446;&amp;#x435;&amp;#x43D;&amp;#x43A;&amp;#x438;, &amp;#x440;&amp;#x435;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x44F; &amp;#x43C;&amp;#x43E;&amp;#x434;&amp;#x435;&amp;#x440;&amp;#x430;&amp;#x442;&amp;#x43E;&amp;#x440;&amp;#x43E;&amp;#x432;), &amp;#x43D;&amp;#x430;&amp;#x431;&amp;#x43B;&amp;#x44E;&amp;#x434;&amp;#x430;&amp;#x44E;&amp;#x442;&amp;#x441;&amp;#x44F; &amp;#x43F;&amp;#x440;&amp;#x43E;&amp;#x431;&amp;#x43B;&amp;#x435;&amp;#x43C;&amp;#x44B;: &amp;#x43F;&amp;#x43E;&amp;#x432;&amp;#x44B;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x43D;&amp;#x430;&amp;#x44F; &amp;#x43D;&amp;#x430;&amp;#x433;&amp;#x440;&amp;#x443;&amp;#x437;&amp;#x43A;&amp;#x430;, &amp;#x43F;&amp;#x43E;&amp;#x432;&amp;#x44B;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x43D;&amp;#x430;&amp;#x44F; &amp;#x434;&amp;#x438;&amp;#x441;&amp;#x43A;&amp;#x43E;&amp;#x432;&amp;#x430;&amp;#x44F; &amp;#x43E;&amp;#x447;&amp;#x435;&amp;#x440;&amp;#x435;&amp;#x434;&amp;#x44C;, &amp;#x443;&amp;#x445;&amp;#x43E;&amp;#x436;&amp;#x434;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x435; &amp;#x43F;&amp;#x440;&amp;#x43E;&amp;#x446;&amp;#x435;&amp;#x441;&amp;#x441;&amp;#x430; &amp;#x432; &amp;#x43E;&amp;#x431;&amp;#x43B;&amp;#x430;&amp;#x441;&amp;#x442;&amp;#x44C; &amp;#x43F;&amp;#x43E;&amp;#x434;&amp;#x43A;&amp;#x430;&amp;#x447;&amp;#x43A;&amp;#x438; (swap), &amp;#x43F;&amp;#x430;&amp;#x434;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x44F;(&amp;#x432; &amp;#x442;&amp;#x43E;&amp;#x43C; &amp;#x447;&amp;#x438;&amp;#x441;&amp;#x43B;&amp;#x435; &amp;#x43C;&amp;#x43D;&amp;#x43E;&amp;#x433;&amp;#x43E;&amp;#x43A;&amp;#x440;&amp;#x430;&amp;#x442;&amp;#x43D;&amp;#x44B;&amp;#x435;), &quot;&amp;#x441;&amp;#x438;&amp;#x43D;&amp;#x438;&amp;#x439; &amp;#x44D;&amp;#x43A;&amp;#x440;&amp;#x430;&amp;#x43D; &amp;#x441;&amp;#x43C;&amp;#x435;&amp;#x440;&amp;#x442;&amp;#x438;&quot;.\n
  • &amp;#x412; &amp;#x440;&amp;#x430;&amp;#x431;&amp;#x43E;&amp;#x442;&amp;#x435; SQL &amp;#x441;&amp;#x435;&amp;#x440;&amp;#x432;&amp;#x435;&amp;#x440;&amp;#x43E;&amp;#x432;, &amp;#x445;&amp;#x440;&amp;#x430;&amp;#x43D;&amp;#x44F;&amp;#x449;&amp;#x438;&amp;#x445; &amp;#x43C;&amp;#x435;&amp;#x442;&amp;#x430;&amp;#x434;&amp;#x430;&amp;#x43D;&amp;#x43D;&amp;#x44B;&amp;#x435; &amp;#x43F;&amp;#x43E; &amp;#x444;&amp;#x43E;&amp;#x442;&amp;#x43E;&amp;#x433;&amp;#x440;&amp;#x430;&amp;#x444;&amp;#x438;&amp;#x44F;&amp;#x43C; &amp;#x438; &amp;#x441;&amp;#x432;&amp;#x44F;&amp;#x437;&amp;#x430;&amp;#x43D;&amp;#x43D;&amp;#x44B;&amp;#x435; &amp;#x441;&amp;#x443;&amp;#x449;&amp;#x43D;&amp;#x43E;&amp;#x441;&amp;#x442;&amp;#x438;(&amp;#x43E;&amp;#x446;&amp;#x435;&amp;#x43D;&amp;#x43A;&amp;#x438;, &amp;#x440;&amp;#x435;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x44F; &amp;#x43C;&amp;#x43E;&amp;#x434;&amp;#x435;&amp;#x440;&amp;#x430;&amp;#x442;&amp;#x43E;&amp;#x440;&amp;#x43E;&amp;#x432;), &amp;#x43D;&amp;#x430;&amp;#x431;&amp;#x43B;&amp;#x44E;&amp;#x434;&amp;#x430;&amp;#x44E;&amp;#x442;&amp;#x441;&amp;#x44F; &amp;#x43F;&amp;#x440;&amp;#x43E;&amp;#x431;&amp;#x43B;&amp;#x435;&amp;#x43C;&amp;#x44B;: &amp;#x43F;&amp;#x43E;&amp;#x432;&amp;#x44B;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x43D;&amp;#x430;&amp;#x44F; &amp;#x43D;&amp;#x430;&amp;#x433;&amp;#x440;&amp;#x443;&amp;#x437;&amp;#x43A;&amp;#x430;, &amp;#x43F;&amp;#x43E;&amp;#x432;&amp;#x44B;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x43D;&amp;#x430;&amp;#x44F; &amp;#x434;&amp;#x438;&amp;#x441;&amp;#x43A;&amp;#x43E;&amp;#x432;&amp;#x430;&amp;#x44F; &amp;#x43E;&amp;#x447;&amp;#x435;&amp;#x440;&amp;#x435;&amp;#x434;&amp;#x44C;, &amp;#x443;&amp;#x445;&amp;#x43E;&amp;#x436;&amp;#x434;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x435; &amp;#x43F;&amp;#x440;&amp;#x43E;&amp;#x446;&amp;#x435;&amp;#x441;&amp;#x441;&amp;#x430; &amp;#x432; &amp;#x43E;&amp;#x431;&amp;#x43B;&amp;#x430;&amp;#x441;&amp;#x442;&amp;#x44C; &amp;#x43F;&amp;#x43E;&amp;#x434;&amp;#x43A;&amp;#x430;&amp;#x447;&amp;#x43A;&amp;#x438; (swap), &amp;#x43F;&amp;#x430;&amp;#x434;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x44F;(&amp;#x432; &amp;#x442;&amp;#x43E;&amp;#x43C; &amp;#x447;&amp;#x438;&amp;#x441;&amp;#x43B;&amp;#x435; &amp;#x43C;&amp;#x43D;&amp;#x43E;&amp;#x433;&amp;#x43E;&amp;#x43A;&amp;#x440;&amp;#x430;&amp;#x442;&amp;#x43D;&amp;#x44B;&amp;#x435;), &quot;&amp;#x441;&amp;#x438;&amp;#x43D;&amp;#x438;&amp;#x439; &amp;#x44D;&amp;#x43A;&amp;#x440;&amp;#x430;&amp;#x43D; &amp;#x441;&amp;#x43C;&amp;#x435;&amp;#x440;&amp;#x442;&amp;#x438;&quot;.\n
  • \n
  • \n
  • &amp;#x412; &amp;#x440;&amp;#x430;&amp;#x431;&amp;#x43E;&amp;#x442;&amp;#x435; SQL &amp;#x441;&amp;#x435;&amp;#x440;&amp;#x432;&amp;#x435;&amp;#x440;&amp;#x43E;&amp;#x432;, &amp;#x445;&amp;#x440;&amp;#x430;&amp;#x43D;&amp;#x44F;&amp;#x449;&amp;#x438;&amp;#x445; &amp;#x43C;&amp;#x435;&amp;#x442;&amp;#x430;&amp;#x434;&amp;#x430;&amp;#x43D;&amp;#x43D;&amp;#x44B;&amp;#x435; &amp;#x43F;&amp;#x43E; &amp;#x444;&amp;#x43E;&amp;#x442;&amp;#x43E;&amp;#x433;&amp;#x440;&amp;#x430;&amp;#x444;&amp;#x438;&amp;#x44F;&amp;#x43C; &amp;#x438; &amp;#x441;&amp;#x432;&amp;#x44F;&amp;#x437;&amp;#x430;&amp;#x43D;&amp;#x43D;&amp;#x44B;&amp;#x435; &amp;#x441;&amp;#x443;&amp;#x449;&amp;#x43D;&amp;#x43E;&amp;#x441;&amp;#x442;&amp;#x438;(&amp;#x43E;&amp;#x446;&amp;#x435;&amp;#x43D;&amp;#x43A;&amp;#x438;, &amp;#x440;&amp;#x435;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x44F; &amp;#x43C;&amp;#x43E;&amp;#x434;&amp;#x435;&amp;#x440;&amp;#x430;&amp;#x442;&amp;#x43E;&amp;#x440;&amp;#x43E;&amp;#x432;), &amp;#x43D;&amp;#x430;&amp;#x431;&amp;#x43B;&amp;#x44E;&amp;#x434;&amp;#x430;&amp;#x44E;&amp;#x442;&amp;#x441;&amp;#x44F; &amp;#x43F;&amp;#x440;&amp;#x43E;&amp;#x431;&amp;#x43B;&amp;#x435;&amp;#x43C;&amp;#x44B;: &amp;#x43F;&amp;#x43E;&amp;#x432;&amp;#x44B;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x43D;&amp;#x430;&amp;#x44F; &amp;#x43D;&amp;#x430;&amp;#x433;&amp;#x440;&amp;#x443;&amp;#x437;&amp;#x43A;&amp;#x430;, &amp;#x43F;&amp;#x43E;&amp;#x432;&amp;#x44B;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x43D;&amp;#x430;&amp;#x44F; &amp;#x434;&amp;#x438;&amp;#x441;&amp;#x43A;&amp;#x43E;&amp;#x432;&amp;#x430;&amp;#x44F; &amp;#x43E;&amp;#x447;&amp;#x435;&amp;#x440;&amp;#x435;&amp;#x434;&amp;#x44C;, &amp;#x443;&amp;#x445;&amp;#x43E;&amp;#x436;&amp;#x434;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x435; &amp;#x43F;&amp;#x440;&amp;#x43E;&amp;#x446;&amp;#x435;&amp;#x441;&amp;#x441;&amp;#x430; &amp;#x432; &amp;#x43E;&amp;#x431;&amp;#x43B;&amp;#x430;&amp;#x441;&amp;#x442;&amp;#x44C; &amp;#x43F;&amp;#x43E;&amp;#x434;&amp;#x43A;&amp;#x430;&amp;#x447;&amp;#x43A;&amp;#x438; (swap), &amp;#x43F;&amp;#x430;&amp;#x434;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x44F;(&amp;#x432; &amp;#x442;&amp;#x43E;&amp;#x43C; &amp;#x447;&amp;#x438;&amp;#x441;&amp;#x43B;&amp;#x435; &amp;#x43C;&amp;#x43D;&amp;#x43E;&amp;#x433;&amp;#x43E;&amp;#x43A;&amp;#x440;&amp;#x430;&amp;#x442;&amp;#x43D;&amp;#x44B;&amp;#x435;), &quot;&amp;#x441;&amp;#x438;&amp;#x43D;&amp;#x438;&amp;#x439; &amp;#x44D;&amp;#x43A;&amp;#x440;&amp;#x430;&amp;#x43D; &amp;#x441;&amp;#x43C;&amp;#x435;&amp;#x440;&amp;#x442;&amp;#x438;&quot;.\n
  • &amp;#x412; &amp;#x440;&amp;#x430;&amp;#x431;&amp;#x43E;&amp;#x442;&amp;#x435; SQL &amp;#x441;&amp;#x435;&amp;#x440;&amp;#x432;&amp;#x435;&amp;#x440;&amp;#x43E;&amp;#x432;, &amp;#x445;&amp;#x440;&amp;#x430;&amp;#x43D;&amp;#x44F;&amp;#x449;&amp;#x438;&amp;#x445; &amp;#x43C;&amp;#x435;&amp;#x442;&amp;#x430;&amp;#x434;&amp;#x430;&amp;#x43D;&amp;#x43D;&amp;#x44B;&amp;#x435; &amp;#x43F;&amp;#x43E; &amp;#x444;&amp;#x43E;&amp;#x442;&amp;#x43E;&amp;#x433;&amp;#x440;&amp;#x430;&amp;#x444;&amp;#x438;&amp;#x44F;&amp;#x43C; &amp;#x438; &amp;#x441;&amp;#x432;&amp;#x44F;&amp;#x437;&amp;#x430;&amp;#x43D;&amp;#x43D;&amp;#x44B;&amp;#x435; &amp;#x441;&amp;#x443;&amp;#x449;&amp;#x43D;&amp;#x43E;&amp;#x441;&amp;#x442;&amp;#x438;(&amp;#x43E;&amp;#x446;&amp;#x435;&amp;#x43D;&amp;#x43A;&amp;#x438;, &amp;#x440;&amp;#x435;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x44F; &amp;#x43C;&amp;#x43E;&amp;#x434;&amp;#x435;&amp;#x440;&amp;#x430;&amp;#x442;&amp;#x43E;&amp;#x440;&amp;#x43E;&amp;#x432;), &amp;#x43D;&amp;#x430;&amp;#x431;&amp;#x43B;&amp;#x44E;&amp;#x434;&amp;#x430;&amp;#x44E;&amp;#x442;&amp;#x441;&amp;#x44F; &amp;#x43F;&amp;#x440;&amp;#x43E;&amp;#x431;&amp;#x43B;&amp;#x435;&amp;#x43C;&amp;#x44B;: &amp;#x43F;&amp;#x43E;&amp;#x432;&amp;#x44B;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x43D;&amp;#x430;&amp;#x44F; &amp;#x43D;&amp;#x430;&amp;#x433;&amp;#x440;&amp;#x443;&amp;#x437;&amp;#x43A;&amp;#x430;, &amp;#x43F;&amp;#x43E;&amp;#x432;&amp;#x44B;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x43D;&amp;#x430;&amp;#x44F; &amp;#x434;&amp;#x438;&amp;#x441;&amp;#x43A;&amp;#x43E;&amp;#x432;&amp;#x430;&amp;#x44F; &amp;#x43E;&amp;#x447;&amp;#x435;&amp;#x440;&amp;#x435;&amp;#x434;&amp;#x44C;, &amp;#x443;&amp;#x445;&amp;#x43E;&amp;#x436;&amp;#x434;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x435; &amp;#x43F;&amp;#x440;&amp;#x43E;&amp;#x446;&amp;#x435;&amp;#x441;&amp;#x441;&amp;#x430; &amp;#x432; &amp;#x43E;&amp;#x431;&amp;#x43B;&amp;#x430;&amp;#x441;&amp;#x442;&amp;#x44C; &amp;#x43F;&amp;#x43E;&amp;#x434;&amp;#x43A;&amp;#x430;&amp;#x447;&amp;#x43A;&amp;#x438; (swap), &amp;#x43F;&amp;#x430;&amp;#x434;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x44F;(&amp;#x432; &amp;#x442;&amp;#x43E;&amp;#x43C; &amp;#x447;&amp;#x438;&amp;#x441;&amp;#x43B;&amp;#x435; &amp;#x43C;&amp;#x43D;&amp;#x43E;&amp;#x433;&amp;#x43E;&amp;#x43A;&amp;#x440;&amp;#x430;&amp;#x442;&amp;#x43D;&amp;#x44B;&amp;#x435;), &quot;&amp;#x441;&amp;#x438;&amp;#x43D;&amp;#x438;&amp;#x439; &amp;#x44D;&amp;#x43A;&amp;#x440;&amp;#x430;&amp;#x43D; &amp;#x441;&amp;#x43C;&amp;#x435;&amp;#x440;&amp;#x442;&amp;#x438;&quot;.\n
  • \n
  • &amp;#x412; &amp;#x440;&amp;#x430;&amp;#x431;&amp;#x43E;&amp;#x442;&amp;#x435; SQL &amp;#x441;&amp;#x435;&amp;#x440;&amp;#x432;&amp;#x435;&amp;#x440;&amp;#x43E;&amp;#x432;, &amp;#x445;&amp;#x440;&amp;#x430;&amp;#x43D;&amp;#x44F;&amp;#x449;&amp;#x438;&amp;#x445; &amp;#x43C;&amp;#x435;&amp;#x442;&amp;#x430;&amp;#x434;&amp;#x430;&amp;#x43D;&amp;#x43D;&amp;#x44B;&amp;#x435; &amp;#x43F;&amp;#x43E; &amp;#x444;&amp;#x43E;&amp;#x442;&amp;#x43E;&amp;#x433;&amp;#x440;&amp;#x430;&amp;#x444;&amp;#x438;&amp;#x44F;&amp;#x43C; &amp;#x438; &amp;#x441;&amp;#x432;&amp;#x44F;&amp;#x437;&amp;#x430;&amp;#x43D;&amp;#x43D;&amp;#x44B;&amp;#x435; &amp;#x441;&amp;#x443;&amp;#x449;&amp;#x43D;&amp;#x43E;&amp;#x441;&amp;#x442;&amp;#x438;(&amp;#x43E;&amp;#x446;&amp;#x435;&amp;#x43D;&amp;#x43A;&amp;#x438;, &amp;#x440;&amp;#x435;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x44F; &amp;#x43C;&amp;#x43E;&amp;#x434;&amp;#x435;&amp;#x440;&amp;#x430;&amp;#x442;&amp;#x43E;&amp;#x440;&amp;#x43E;&amp;#x432;), &amp;#x43D;&amp;#x430;&amp;#x431;&amp;#x43B;&amp;#x44E;&amp;#x434;&amp;#x430;&amp;#x44E;&amp;#x442;&amp;#x441;&amp;#x44F; &amp;#x43F;&amp;#x440;&amp;#x43E;&amp;#x431;&amp;#x43B;&amp;#x435;&amp;#x43C;&amp;#x44B;: &amp;#x43F;&amp;#x43E;&amp;#x432;&amp;#x44B;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x43D;&amp;#x430;&amp;#x44F; &amp;#x43D;&amp;#x430;&amp;#x433;&amp;#x440;&amp;#x443;&amp;#x437;&amp;#x43A;&amp;#x430;, &amp;#x43F;&amp;#x43E;&amp;#x432;&amp;#x44B;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x43D;&amp;#x430;&amp;#x44F; &amp;#x434;&amp;#x438;&amp;#x441;&amp;#x43A;&amp;#x43E;&amp;#x432;&amp;#x430;&amp;#x44F; &amp;#x43E;&amp;#x447;&amp;#x435;&amp;#x440;&amp;#x435;&amp;#x434;&amp;#x44C;, &amp;#x443;&amp;#x445;&amp;#x43E;&amp;#x436;&amp;#x434;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x435; &amp;#x43F;&amp;#x440;&amp;#x43E;&amp;#x446;&amp;#x435;&amp;#x441;&amp;#x441;&amp;#x430; &amp;#x432; &amp;#x43E;&amp;#x431;&amp;#x43B;&amp;#x430;&amp;#x441;&amp;#x442;&amp;#x44C; &amp;#x43F;&amp;#x43E;&amp;#x434;&amp;#x43A;&amp;#x430;&amp;#x447;&amp;#x43A;&amp;#x438; (swap), &amp;#x43F;&amp;#x430;&amp;#x434;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x44F;(&amp;#x432; &amp;#x442;&amp;#x43E;&amp;#x43C; &amp;#x447;&amp;#x438;&amp;#x441;&amp;#x43B;&amp;#x435; &amp;#x43C;&amp;#x43D;&amp;#x43E;&amp;#x433;&amp;#x43E;&amp;#x43A;&amp;#x440;&amp;#x430;&amp;#x442;&amp;#x43D;&amp;#x44B;&amp;#x435;), &quot;&amp;#x441;&amp;#x438;&amp;#x43D;&amp;#x438;&amp;#x439; &amp;#x44D;&amp;#x43A;&amp;#x440;&amp;#x430;&amp;#x43D; &amp;#x441;&amp;#x43C;&amp;#x435;&amp;#x440;&amp;#x442;&amp;#x438;&quot;.\n
  • &amp;#x412; &amp;#x440;&amp;#x430;&amp;#x431;&amp;#x43E;&amp;#x442;&amp;#x435; SQL &amp;#x441;&amp;#x435;&amp;#x440;&amp;#x432;&amp;#x435;&amp;#x440;&amp;#x43E;&amp;#x432;, &amp;#x445;&amp;#x440;&amp;#x430;&amp;#x43D;&amp;#x44F;&amp;#x449;&amp;#x438;&amp;#x445; &amp;#x43C;&amp;#x435;&amp;#x442;&amp;#x430;&amp;#x434;&amp;#x430;&amp;#x43D;&amp;#x43D;&amp;#x44B;&amp;#x435; &amp;#x43F;&amp;#x43E; &amp;#x444;&amp;#x43E;&amp;#x442;&amp;#x43E;&amp;#x433;&amp;#x440;&amp;#x430;&amp;#x444;&amp;#x438;&amp;#x44F;&amp;#x43C; &amp;#x438; &amp;#x441;&amp;#x432;&amp;#x44F;&amp;#x437;&amp;#x430;&amp;#x43D;&amp;#x43D;&amp;#x44B;&amp;#x435; &amp;#x441;&amp;#x443;&amp;#x449;&amp;#x43D;&amp;#x43E;&amp;#x441;&amp;#x442;&amp;#x438;(&amp;#x43E;&amp;#x446;&amp;#x435;&amp;#x43D;&amp;#x43A;&amp;#x438;, &amp;#x440;&amp;#x435;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x44F; &amp;#x43C;&amp;#x43E;&amp;#x434;&amp;#x435;&amp;#x440;&amp;#x430;&amp;#x442;&amp;#x43E;&amp;#x440;&amp;#x43E;&amp;#x432;), &amp;#x43D;&amp;#x430;&amp;#x431;&amp;#x43B;&amp;#x44E;&amp;#x434;&amp;#x430;&amp;#x44E;&amp;#x442;&amp;#x441;&amp;#x44F; &amp;#x43F;&amp;#x440;&amp;#x43E;&amp;#x431;&amp;#x43B;&amp;#x435;&amp;#x43C;&amp;#x44B;: &amp;#x43F;&amp;#x43E;&amp;#x432;&amp;#x44B;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x43D;&amp;#x430;&amp;#x44F; &amp;#x43D;&amp;#x430;&amp;#x433;&amp;#x440;&amp;#x443;&amp;#x437;&amp;#x43A;&amp;#x430;, &amp;#x43F;&amp;#x43E;&amp;#x432;&amp;#x44B;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x43D;&amp;#x430;&amp;#x44F; &amp;#x434;&amp;#x438;&amp;#x441;&amp;#x43A;&amp;#x43E;&amp;#x432;&amp;#x430;&amp;#x44F; &amp;#x43E;&amp;#x447;&amp;#x435;&amp;#x440;&amp;#x435;&amp;#x434;&amp;#x44C;, &amp;#x443;&amp;#x445;&amp;#x43E;&amp;#x436;&amp;#x434;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x435; &amp;#x43F;&amp;#x440;&amp;#x43E;&amp;#x446;&amp;#x435;&amp;#x441;&amp;#x441;&amp;#x430; &amp;#x432; &amp;#x43E;&amp;#x431;&amp;#x43B;&amp;#x430;&amp;#x441;&amp;#x442;&amp;#x44C; &amp;#x43F;&amp;#x43E;&amp;#x434;&amp;#x43A;&amp;#x430;&amp;#x447;&amp;#x43A;&amp;#x438; (swap), &amp;#x43F;&amp;#x430;&amp;#x434;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x44F;(&amp;#x432; &amp;#x442;&amp;#x43E;&amp;#x43C; &amp;#x447;&amp;#x438;&amp;#x441;&amp;#x43B;&amp;#x435; &amp;#x43C;&amp;#x43D;&amp;#x43E;&amp;#x433;&amp;#x43E;&amp;#x43A;&amp;#x440;&amp;#x430;&amp;#x442;&amp;#x43D;&amp;#x44B;&amp;#x435;), &quot;&amp;#x441;&amp;#x438;&amp;#x43D;&amp;#x438;&amp;#x439; &amp;#x44D;&amp;#x43A;&amp;#x440;&amp;#x430;&amp;#x43D; &amp;#x441;&amp;#x43C;&amp;#x435;&amp;#x440;&amp;#x442;&amp;#x438;&quot;.\n
  • \n
  • \n
  • \n
  • \n
  • &amp;#x412; &amp;#x440;&amp;#x430;&amp;#x431;&amp;#x43E;&amp;#x442;&amp;#x435; SQL &amp;#x441;&amp;#x435;&amp;#x440;&amp;#x432;&amp;#x435;&amp;#x440;&amp;#x43E;&amp;#x432;, &amp;#x445;&amp;#x440;&amp;#x430;&amp;#x43D;&amp;#x44F;&amp;#x449;&amp;#x438;&amp;#x445; &amp;#x43C;&amp;#x435;&amp;#x442;&amp;#x430;&amp;#x434;&amp;#x430;&amp;#x43D;&amp;#x43D;&amp;#x44B;&amp;#x435; &amp;#x43F;&amp;#x43E; &amp;#x444;&amp;#x43E;&amp;#x442;&amp;#x43E;&amp;#x433;&amp;#x440;&amp;#x430;&amp;#x444;&amp;#x438;&amp;#x44F;&amp;#x43C; &amp;#x438; &amp;#x441;&amp;#x432;&amp;#x44F;&amp;#x437;&amp;#x430;&amp;#x43D;&amp;#x43D;&amp;#x44B;&amp;#x435; &amp;#x441;&amp;#x443;&amp;#x449;&amp;#x43D;&amp;#x43E;&amp;#x441;&amp;#x442;&amp;#x438;(&amp;#x43E;&amp;#x446;&amp;#x435;&amp;#x43D;&amp;#x43A;&amp;#x438;, &amp;#x440;&amp;#x435;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x44F; &amp;#x43C;&amp;#x43E;&amp;#x434;&amp;#x435;&amp;#x440;&amp;#x430;&amp;#x442;&amp;#x43E;&amp;#x440;&amp;#x43E;&amp;#x432;), &amp;#x43D;&amp;#x430;&amp;#x431;&amp;#x43B;&amp;#x44E;&amp;#x434;&amp;#x430;&amp;#x44E;&amp;#x442;&amp;#x441;&amp;#x44F; &amp;#x43F;&amp;#x440;&amp;#x43E;&amp;#x431;&amp;#x43B;&amp;#x435;&amp;#x43C;&amp;#x44B;: &amp;#x43F;&amp;#x43E;&amp;#x432;&amp;#x44B;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x43D;&amp;#x430;&amp;#x44F; &amp;#x43D;&amp;#x430;&amp;#x433;&amp;#x440;&amp;#x443;&amp;#x437;&amp;#x43A;&amp;#x430;, &amp;#x43F;&amp;#x43E;&amp;#x432;&amp;#x44B;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x43D;&amp;#x430;&amp;#x44F; &amp;#x434;&amp;#x438;&amp;#x441;&amp;#x43A;&amp;#x43E;&amp;#x432;&amp;#x430;&amp;#x44F; &amp;#x43E;&amp;#x447;&amp;#x435;&amp;#x440;&amp;#x435;&amp;#x434;&amp;#x44C;, &amp;#x443;&amp;#x445;&amp;#x43E;&amp;#x436;&amp;#x434;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x435; &amp;#x43F;&amp;#x440;&amp;#x43E;&amp;#x446;&amp;#x435;&amp;#x441;&amp;#x441;&amp;#x430; &amp;#x432; &amp;#x43E;&amp;#x431;&amp;#x43B;&amp;#x430;&amp;#x441;&amp;#x442;&amp;#x44C; &amp;#x43F;&amp;#x43E;&amp;#x434;&amp;#x43A;&amp;#x430;&amp;#x447;&amp;#x43A;&amp;#x438; (swap), &amp;#x43F;&amp;#x430;&amp;#x434;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x44F;(&amp;#x432; &amp;#x442;&amp;#x43E;&amp;#x43C; &amp;#x447;&amp;#x438;&amp;#x441;&amp;#x43B;&amp;#x435; &amp;#x43C;&amp;#x43D;&amp;#x43E;&amp;#x433;&amp;#x43E;&amp;#x43A;&amp;#x440;&amp;#x430;&amp;#x442;&amp;#x43D;&amp;#x44B;&amp;#x435;), &quot;&amp;#x441;&amp;#x438;&amp;#x43D;&amp;#x438;&amp;#x439; &amp;#x44D;&amp;#x43A;&amp;#x440;&amp;#x430;&amp;#x43D; &amp;#x441;&amp;#x43C;&amp;#x435;&amp;#x440;&amp;#x442;&amp;#x438;&quot;.\n
  • &amp;#x412; &amp;#x440;&amp;#x430;&amp;#x431;&amp;#x43E;&amp;#x442;&amp;#x435; SQL &amp;#x441;&amp;#x435;&amp;#x440;&amp;#x432;&amp;#x435;&amp;#x440;&amp;#x43E;&amp;#x432;, &amp;#x445;&amp;#x440;&amp;#x430;&amp;#x43D;&amp;#x44F;&amp;#x449;&amp;#x438;&amp;#x445; &amp;#x43C;&amp;#x435;&amp;#x442;&amp;#x430;&amp;#x434;&amp;#x430;&amp;#x43D;&amp;#x43D;&amp;#x44B;&amp;#x435; &amp;#x43F;&amp;#x43E; &amp;#x444;&amp;#x43E;&amp;#x442;&amp;#x43E;&amp;#x433;&amp;#x440;&amp;#x430;&amp;#x444;&amp;#x438;&amp;#x44F;&amp;#x43C; &amp;#x438; &amp;#x441;&amp;#x432;&amp;#x44F;&amp;#x437;&amp;#x430;&amp;#x43D;&amp;#x43D;&amp;#x44B;&amp;#x435; &amp;#x441;&amp;#x443;&amp;#x449;&amp;#x43D;&amp;#x43E;&amp;#x441;&amp;#x442;&amp;#x438;(&amp;#x43E;&amp;#x446;&amp;#x435;&amp;#x43D;&amp;#x43A;&amp;#x438;, &amp;#x440;&amp;#x435;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x44F; &amp;#x43C;&amp;#x43E;&amp;#x434;&amp;#x435;&amp;#x440;&amp;#x430;&amp;#x442;&amp;#x43E;&amp;#x440;&amp;#x43E;&amp;#x432;), &amp;#x43D;&amp;#x430;&amp;#x431;&amp;#x43B;&amp;#x44E;&amp;#x434;&amp;#x430;&amp;#x44E;&amp;#x442;&amp;#x441;&amp;#x44F; &amp;#x43F;&amp;#x440;&amp;#x43E;&amp;#x431;&amp;#x43B;&amp;#x435;&amp;#x43C;&amp;#x44B;: &amp;#x43F;&amp;#x43E;&amp;#x432;&amp;#x44B;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x43D;&amp;#x430;&amp;#x44F; &amp;#x43D;&amp;#x430;&amp;#x433;&amp;#x440;&amp;#x443;&amp;#x437;&amp;#x43A;&amp;#x430;, &amp;#x43F;&amp;#x43E;&amp;#x432;&amp;#x44B;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x43D;&amp;#x430;&amp;#x44F; &amp;#x434;&amp;#x438;&amp;#x441;&amp;#x43A;&amp;#x43E;&amp;#x432;&amp;#x430;&amp;#x44F; &amp;#x43E;&amp;#x447;&amp;#x435;&amp;#x440;&amp;#x435;&amp;#x434;&amp;#x44C;, &amp;#x443;&amp;#x445;&amp;#x43E;&amp;#x436;&amp;#x434;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x435; &amp;#x43F;&amp;#x440;&amp;#x43E;&amp;#x446;&amp;#x435;&amp;#x441;&amp;#x441;&amp;#x430; &amp;#x432; &amp;#x43E;&amp;#x431;&amp;#x43B;&amp;#x430;&amp;#x441;&amp;#x442;&amp;#x44C; &amp;#x43F;&amp;#x43E;&amp;#x434;&amp;#x43A;&amp;#x430;&amp;#x447;&amp;#x43A;&amp;#x438; (swap), &amp;#x43F;&amp;#x430;&amp;#x434;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x44F;(&amp;#x432; &amp;#x442;&amp;#x43E;&amp;#x43C; &amp;#x447;&amp;#x438;&amp;#x441;&amp;#x43B;&amp;#x435; &amp;#x43C;&amp;#x43D;&amp;#x43E;&amp;#x433;&amp;#x43E;&amp;#x43A;&amp;#x440;&amp;#x430;&amp;#x442;&amp;#x43D;&amp;#x44B;&amp;#x435;), &quot;&amp;#x441;&amp;#x438;&amp;#x43D;&amp;#x438;&amp;#x439; &amp;#x44D;&amp;#x43A;&amp;#x440;&amp;#x430;&amp;#x43D; &amp;#x441;&amp;#x43C;&amp;#x435;&amp;#x440;&amp;#x442;&amp;#x438;&quot;.\n
  • &amp;#x412; &amp;#x440;&amp;#x430;&amp;#x431;&amp;#x43E;&amp;#x442;&amp;#x435; SQL &amp;#x441;&amp;#x435;&amp;#x440;&amp;#x432;&amp;#x435;&amp;#x440;&amp;#x43E;&amp;#x432;, &amp;#x445;&amp;#x440;&amp;#x430;&amp;#x43D;&amp;#x44F;&amp;#x449;&amp;#x438;&amp;#x445; &amp;#x43C;&amp;#x435;&amp;#x442;&amp;#x430;&amp;#x434;&amp;#x430;&amp;#x43D;&amp;#x43D;&amp;#x44B;&amp;#x435; &amp;#x43F;&amp;#x43E; &amp;#x444;&amp;#x43E;&amp;#x442;&amp;#x43E;&amp;#x433;&amp;#x440;&amp;#x430;&amp;#x444;&amp;#x438;&amp;#x44F;&amp;#x43C; &amp;#x438; &amp;#x441;&amp;#x432;&amp;#x44F;&amp;#x437;&amp;#x430;&amp;#x43D;&amp;#x43D;&amp;#x44B;&amp;#x435; &amp;#x441;&amp;#x443;&amp;#x449;&amp;#x43D;&amp;#x43E;&amp;#x441;&amp;#x442;&amp;#x438;(&amp;#x43E;&amp;#x446;&amp;#x435;&amp;#x43D;&amp;#x43A;&amp;#x438;, &amp;#x440;&amp;#x435;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x44F; &amp;#x43C;&amp;#x43E;&amp;#x434;&amp;#x435;&amp;#x440;&amp;#x430;&amp;#x442;&amp;#x43E;&amp;#x440;&amp;#x43E;&amp;#x432;), &amp;#x43D;&amp;#x430;&amp;#x431;&amp;#x43B;&amp;#x44E;&amp;#x434;&amp;#x430;&amp;#x44E;&amp;#x442;&amp;#x441;&amp;#x44F; &amp;#x43F;&amp;#x440;&amp;#x43E;&amp;#x431;&amp;#x43B;&amp;#x435;&amp;#x43C;&amp;#x44B;: &amp;#x43F;&amp;#x43E;&amp;#x432;&amp;#x44B;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x43D;&amp;#x430;&amp;#x44F; &amp;#x43D;&amp;#x430;&amp;#x433;&amp;#x440;&amp;#x443;&amp;#x437;&amp;#x43A;&amp;#x430;, &amp;#x43F;&amp;#x43E;&amp;#x432;&amp;#x44B;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x43D;&amp;#x430;&amp;#x44F; &amp;#x434;&amp;#x438;&amp;#x441;&amp;#x43A;&amp;#x43E;&amp;#x432;&amp;#x430;&amp;#x44F; &amp;#x43E;&amp;#x447;&amp;#x435;&amp;#x440;&amp;#x435;&amp;#x434;&amp;#x44C;, &amp;#x443;&amp;#x445;&amp;#x43E;&amp;#x436;&amp;#x434;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x435; &amp;#x43F;&amp;#x440;&amp;#x43E;&amp;#x446;&amp;#x435;&amp;#x441;&amp;#x441;&amp;#x430; &amp;#x432; &amp;#x43E;&amp;#x431;&amp;#x43B;&amp;#x430;&amp;#x441;&amp;#x442;&amp;#x44C; &amp;#x43F;&amp;#x43E;&amp;#x434;&amp;#x43A;&amp;#x430;&amp;#x447;&amp;#x43A;&amp;#x438; (swap), &amp;#x43F;&amp;#x430;&amp;#x434;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x44F;(&amp;#x432; &amp;#x442;&amp;#x43E;&amp;#x43C; &amp;#x447;&amp;#x438;&amp;#x441;&amp;#x43B;&amp;#x435; &amp;#x43C;&amp;#x43D;&amp;#x43E;&amp;#x433;&amp;#x43E;&amp;#x43A;&amp;#x440;&amp;#x430;&amp;#x442;&amp;#x43D;&amp;#x44B;&amp;#x435;), &quot;&amp;#x441;&amp;#x438;&amp;#x43D;&amp;#x438;&amp;#x439; &amp;#x44D;&amp;#x43A;&amp;#x440;&amp;#x430;&amp;#x43D; &amp;#x441;&amp;#x43C;&amp;#x435;&amp;#x440;&amp;#x442;&amp;#x438;&quot;.\n
  • &amp;#x412; &amp;#x440;&amp;#x430;&amp;#x431;&amp;#x43E;&amp;#x442;&amp;#x435; SQL &amp;#x441;&amp;#x435;&amp;#x440;&amp;#x432;&amp;#x435;&amp;#x440;&amp;#x43E;&amp;#x432;, &amp;#x445;&amp;#x440;&amp;#x430;&amp;#x43D;&amp;#x44F;&amp;#x449;&amp;#x438;&amp;#x445; &amp;#x43C;&amp;#x435;&amp;#x442;&amp;#x430;&amp;#x434;&amp;#x430;&amp;#x43D;&amp;#x43D;&amp;#x44B;&amp;#x435; &amp;#x43F;&amp;#x43E; &amp;#x444;&amp;#x43E;&amp;#x442;&amp;#x43E;&amp;#x433;&amp;#x440;&amp;#x430;&amp;#x444;&amp;#x438;&amp;#x44F;&amp;#x43C; &amp;#x438; &amp;#x441;&amp;#x432;&amp;#x44F;&amp;#x437;&amp;#x430;&amp;#x43D;&amp;#x43D;&amp;#x44B;&amp;#x435; &amp;#x441;&amp;#x443;&amp;#x449;&amp;#x43D;&amp;#x43E;&amp;#x441;&amp;#x442;&amp;#x438;(&amp;#x43E;&amp;#x446;&amp;#x435;&amp;#x43D;&amp;#x43A;&amp;#x438;, &amp;#x440;&amp;#x435;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x44F; &amp;#x43C;&amp;#x43E;&amp;#x434;&amp;#x435;&amp;#x440;&amp;#x430;&amp;#x442;&amp;#x43E;&amp;#x440;&amp;#x43E;&amp;#x432;), &amp;#x43D;&amp;#x430;&amp;#x431;&amp;#x43B;&amp;#x44E;&amp;#x434;&amp;#x430;&amp;#x44E;&amp;#x442;&amp;#x441;&amp;#x44F; &amp;#x43F;&amp;#x440;&amp;#x43E;&amp;#x431;&amp;#x43B;&amp;#x435;&amp;#x43C;&amp;#x44B;: &amp;#x43F;&amp;#x43E;&amp;#x432;&amp;#x44B;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x43D;&amp;#x430;&amp;#x44F; &amp;#x43D;&amp;#x430;&amp;#x433;&amp;#x440;&amp;#x443;&amp;#x437;&amp;#x43A;&amp;#x430;, &amp;#x43F;&amp;#x43E;&amp;#x432;&amp;#x44B;&amp;#x448;&amp;#x435;&amp;#x43D;&amp;#x43D;&amp;#x430;&amp;#x44F; &amp;#x434;&amp;#x438;&amp;#x441;&amp;#x43A;&amp;#x43E;&amp;#x432;&amp;#x430;&amp;#x44F; &amp;#x43E;&amp;#x447;&amp;#x435;&amp;#x440;&amp;#x435;&amp;#x434;&amp;#x44C;, &amp;#x443;&amp;#x445;&amp;#x43E;&amp;#x436;&amp;#x434;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x435; &amp;#x43F;&amp;#x440;&amp;#x43E;&amp;#x446;&amp;#x435;&amp;#x441;&amp;#x441;&amp;#x430; &amp;#x432; &amp;#x43E;&amp;#x431;&amp;#x43B;&amp;#x430;&amp;#x441;&amp;#x442;&amp;#x44C; &amp;#x43F;&amp;#x43E;&amp;#x434;&amp;#x43A;&amp;#x430;&amp;#x447;&amp;#x43A;&amp;#x438; (swap), &amp;#x43F;&amp;#x430;&amp;#x434;&amp;#x435;&amp;#x43D;&amp;#x438;&amp;#x44F;(&amp;#x432; &amp;#x442;&amp;#x43E;&amp;#x43C; &amp;#x447;&amp;#x438;&amp;#x441;&amp;#x43B;&amp;#x435; &amp;#x43C;&amp;#x43D;&amp;#x43E;&amp;#x433;&amp;#x43E;&amp;#x43A;&amp;#x440;&amp;#x430;&amp;#x442;&amp;#x43D;&amp;#x44B;&amp;#x435;), &quot;&amp;#x441;&amp;#x438;&amp;#x43D;&amp;#x438;&amp;#x439; &amp;#x44D;&amp;#x43A;&amp;#x440;&amp;#x430;&amp;#x43D; &amp;#x441;&amp;#x43C;&amp;#x435;&amp;#x440;&amp;#x442;&amp;#x438;&quot;.\n
  • Transcript of "Taming Cassandra"

    1. 1. Taming Cassandra@ odnoklassniki.ruby Oleg AnastasyevLead Platform Dev
    2. 2. 5M users online250 000 pages/s, 50 ms >100 Gbit/s ( >2x growth last year)> 4000 servers in 4 DCs 99% java
    3. 3. Data Storages we use• MS SQL 2005• Berkley DB based solution• Voldemort + Tarantool• Cassandra forked since 0.6 at 2010
    4. 4. cassandra @ odnoklassniki• Social graph persistent store• Like AKA “Класс!” button servers• Photo Marks storage• ... and number of others in development
    5. 5. Intro to cassandra (oversimplified)
    6. 6. Cassandra : ring 00 Gossip: ★ Cluster membership192 64 ★ Failure detection 128
    7. 7. Cassandra : partitioningPartitioner.toStringToken(key) : Token 00RP: Token=md5(key)OPP:Token=toString(key) Data Range 192-00 Data Range 00-64 192 64 Data Range 128-192 Data Range 64-128 128
    8. 8. Cassandra : replicationReplicationStrategy: 00Token => InetAddress• Simple•RackAware R1 192 64 R3 128 R2
    9. 9. Cassandra : writes Tunable Client 00 consistency level: CL = ZERO CL = ONE THRIFT CL = QUORUM CL = ALL Row Mutationerror if# writes < CL 192 64 Hint Storage 128
    10. 10. Cassandra : reads Client 00 Data resolved result Hash 192 64 Wrong HashTunable consistency:CL = ONECL = QUORUM BackgroundCL = ALL Read Repair 128
    11. 11. Cassandra: data model Keyspace (“Database”) 1..* ColumnFamilyStore (“Table”) << dynamic >> 0..* ColumnColumnFamily 0..*(“Row”) name : byte[] << dynamic >> value : byte[]key : String timestamp : long
    12. 12. Cassandra: local write(s) name value ts Write (Key, Column) Commit Log Memtable Flusher Thread flushes SSTable 1 SSTable 2 SSTable 3 SSTable 4 Merge-Sorted Compaction Thread SSTable 5Writes are always sequential!
    13. 13. Cassandra: local read(s) name value timesta (part of) data mp resolve Memtable SSTable 1 SSTable 4 SSTable 5• get(Key, columnNames[])• slice(Key, from, count, direction)• key_range(from,count)
    14. 14. SSTable anatomy SSTable-5-Filter.db SSTable-5-Index.db SSTable-5-Data.db Bloom Filter Row Columns Key => Offset Per Row Bloom Filter“Row May Exist” Per Row Index False Positives So: - ZERO reads, if row is missing AND you’re lucky - 1 seek and read of index, if row is missing AND you’re not - 2 logical seeks per read (for small rows) - 3 logical seeks per read (for large ones)
    15. 15. Conflict ResolutionSSTable “AccStatements-3456” MemtableRowKey = “Oleg_Anastasyev” RowKey = “Oleg_Anastasyev”ColumnFamily = “AccountStatements” ColumnFamily = “AccountStatements”Column=”LV05HABA95142357516”Value= $1,000,000 vs Column=”LV05HABA95142357516” Value= $10 Which data is correct ?
    16. 16. Conflict ResolutionSSTable “AccStatements-3456” MemtabeRowKey = “Oleg_Anastasyev” RowKey = “Oleg_Anastasyev”ColumnFamily = “AccountStatements” ColumnFamily = “AccountStatements”Column=”LV05HABA95142357516”Value= $1,000,000 vs Column=”LV05HABA95142357516” Value= $10Timestamp = 1.1.2012 00:00:00 Timestamp=2.2.2011 00:00:00 The one with latest timestamp. period.
    17. 17. Missed Update problemClient 1 Client 21. Read AccountStatement for Key=”Oleg” 1. Read AccountStatement for(got $10, TS=12:00:00) Key=”Oleg” (got $10, TS=12:00:00)2. Deposit $1,000,0003. Save Key=”Oleg”, 2. Withdraw $1 Value=$1,000,010 TS=12:00:01.00 3. Save Key=”Oleg”, Value=$9 TS=12:00:01.005 Distributed counters (since 0.7) ! No generic solution for Read-Modify-Update !
    18. 18. Intro SummaryPros: Cons:• No SPoF • No ACID, No rollbacks• Focus on Availability • No conflict detection• High and Stable Write performance • No locks• On the fly cluster expansion • NoSQL => Think ahead for queries• Very efficient missing row read• No locks• No backups necessary• Instant Multi DC
    19. 19. Tired of RTFM ?
    20. 20. Ok. Here is the problem
    21. 21. Ok. Here is the problem
    22. 22. Ok. Here is the problem
    23. 23. Ok. Here is the problem• PhotoMarks tablePhotoId:long UserId:long OwnerId:long Mark:float timestamp9999999999 1111 2222 1.0f 11:00 – 2 billion shows a day (~ 50 000 on peak second) – 100 mi new marks on 10 M new photos a day ( ~ 2500 a second ) – 1.5 Tb of data + 800 Gb indexes and growing + 3 Gb per day• Query patterns – EXISTS ( PhotoId=?, UserId=?) - for 98% of calls answer is “NOT EXISTS” – SUM(), AVG() on PhotoId=? -- what are totals on photo ? – OwnerId=? ORDER BY Timestamp DESC -- who marked my photos ? – COUNT(*) OwnerId=? AND Timestamp>? -- how many new marks are ? – UserId=? --for cleanup
    24. 24. And the trouble is– 32 SQL clusters are serving it (not counting stanby’s)– ... and they are close to their capacity in CPU, disk array iops
    25. 25. Simple solutions ?• Add more SQL nodes – There are already 32 of them, add more 32 => 64 – Expensive (hardware + software) – Extension is offline and a lot of manual work – Repeat in half a year ( => 128 => 256 )• Memory cache – High NOT EXISTS queries render LRU cache useless – So you have to cache 100% of rows – 1.5Tb of RAM is not cheap – and you need 2-3x more for fault tolerance and queries
    26. 26. Cassandra !• Leveraging Pros – Cheap NOT EXISTS ( bloom filter powered ) – CFs are enough to model this – Data are stored on disk together (most queries are 2 disk reads) – Cold dataset is stored on disk – On the fly expansion of cluster – Strong Availability• Not hitting Cons – No ACID requirements – Eventual Consistency is ok – Marks are rarely changed and never changed concurrently – Have time for major compaction
    27. 27. Data modelMarksByPhotoMarksByOwner (FreshFirst order)MarksUserIndex
    28. 28. Data modelMarksByPhotoKey Column Column ValuephotoId:String userId:byte[8] ownerUserId+mark+time : byte[20] – SUM(), AVG() on PhotoId=? – EXISTS ( PhotoId=?, UserId=?) 98% calls => “NOT EXISTS”- We want no disk reads here, Cap!- But cassandra need to read columns...
    29. 29. #1: Column Bloom Filter• How ?– Stores (Key, Column name) pairs in SSTable *-Filter.db• Pros– No disk access on NOT EXISTS– ... i.e. 98% of operations are memory only– Larger bloom filter => less false positives• Cons– Bloom filters became large - 100s of MBytes– .. lead to GC Promotion Failures (originally it was implemented as single long[])– fixed it (CASSANDRA-2466)
    30. 30. Profit!• New system– 8 cassandra nodes ( instead of 32 SQLs )– RF =2– Every row stored 2.5 times (in MarksByPhoto, ByOwner and UserIndex CFs)– ~ 10 TB of data– which is > 1TB per every single node• :-( still– A lot of data per node => no use for SSD– Major compaction yields 400 - 500 Gb SSTable files– .. which are unmanageable– .. lead to full disk cache invalidation as soon as compaction finishes– .. cannot be split across disks (RAID 10 is SLOWER)– .. but we need more spindles
    31. 31. #2: Split em !• How ?– Pre-partition every CF to 256 smaller CFs, based on key low byte e.g. MarksByPhoto_00, _01, _FF– Now every node has 32*RF(2)=64 smaller CFs of size 5-10 Gb each– Made memtable flush every 2Mb– Made 10 separate data volumes + RAID 10 for commit log device• Pros– Even load distribution across disks ( and fixed it to be plain RR )– Cheap spinning disks– No massive disk cache invalidation on major compaction– 2Mb flushes do not stress disks– More parallelization abilities (e.g. we reduced startup time from 30 min -> 3 )
    32. 32. #3: Uniform Replication 00 Thrift calls R1 192 64 R2 86 107 R3 128
    33. 33. #3: Uniform Replication RF=3 00 Thrift calls R1 192 64 R2 86Even worse for RF=2 107 R3 128
    34. 34. What happens ?• On node failure – Additional load on live node (THRIFT CPU, Hint storage) – 300+ clients want to establish 20-30 connections each in 1 second (and thrift is slow on this) – ... Clients decide secondary replica is dead as well – ... No availability on part of keys ;-(• Inconvenient – You need to watch max pool sizes are 2x as large as necessary – Ensure each node capacity is > 2x
    35. 35. Own replication• How it works ? See source code at github/odnoklassniki Look for RackAwareOdklEvenStrategy• What it does ? – All Keys of every single node are replicated across ALL nodes from other DCs (and not only on its ring neighbors)• Profit! – Every single node failure adds a tiny fraction of load to others
    36. 36. That’s iton photo marks Oleg Anastasyev oa@odnoklassniki.ru odnoklassniki.ru/oa github.com/odnoklassniki/apache-cassandra cassandra.apache.org
    37. 37. Some problems left application servers PhotoMarksDAOImpl THRIFT +hector application servers cassandra cluster application servers• THRIFT is slow, esp on connect even newest one from cassandra 1.1 is slower on communication and ser/deser than our BLOS; employs funny IDL (see slideshare.net/m0nstermind/java-13078132)• DTO <->THRIFT <-> CS translation• Multiple roundtrips per transaction
    38. 38. Meet odnoklassniki-like application servers PhotoMarksDAOImpl application servers LikeS BLOS +hector application servers cassandra cluster application servers• All BL is co-located on CS JVM => much less net roundtrips; less partial failure probability; DTO <-> CS direct translation• App specialized caches => faster, less RAM, store DTO, own policy• CF Data listeners => custom replicas merge logic• Row processors => (custom) bulk data processing• Direct Access to in memory bloom filters => fast friends like it too
    39. 39. That’s it !Thank you ! Oleg Anastasyev oa@odnoklassniki.ru odnoklassniki.ru/oa github.com/odnoklassniki/apache-cassandra cassandra.apache.org
    40. 40. #3: Parallel readClient 00 Slow Data slow result, timeouts Hash 192 64 Wrong Hash Background Read Repair 128
    41. 41. #3: Parallel read Slow ? FGC ? Overloaded ?Client 00 Slow Data slow result, timeouts Hash 192 64 Wrong Hash Background Read Repair 128
    42. 42. #3: Parallel readParallelReadResolver:1. asks all replicas for data 002. waits for # of responds CL requires3. resolves and returns to client4. waits for the rest in consistency thread Slow Data Data 192 64Pros:• Unnoticed single node problems Wrong Data• Response time stabilityCons:• More traffic Background (negligible for small data) Read Repair 128
    43. 43. Cassandra : replicationReplicationStrategy: 00Token => InetAddress 22• Simple• RackAware 43 R1 192 64 86 R2 107 128 R3
    44. 44. Data modelMarksByPhotoMarksByOwner (FreshFirst order)MarksUserIndex
    45. 45. Data modelMarksByOwner (FreshFirst order)Key Column name Column ValueownerId:String time+photoId+userId : byte[24] mark : byte[4] time+photoId+userId : byte[24] mark : byte[4] ... ... – OwnerId=? ORDER BY Timestamp DESC -- who marked my photos ? – COUNT(*) OwnerId=? AND Timestamp>? -- how many new marks are ?
    46. 46. Data modelMarksUserIndexKey Column Column ValueuserId:String photoId:byte[8] “” : byte[0] photoId:byte[8] “” : byte[0] ... ... – UserId=? --for cleanup
    47. 47. Other extensions• Node local Maintenance Manager• Commit log archive• Data snapshot backup• Local CF storages
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×