Your SlideShare is downloading. ×
0
Unpack mechanism of the msgpack-c
Unpack mechanism of the msgpack-c
Unpack mechanism of the msgpack-c
Unpack mechanism of the msgpack-c
Unpack mechanism of the msgpack-c
Unpack mechanism of the msgpack-c
Unpack mechanism of the msgpack-c
Unpack mechanism of the msgpack-c
Unpack mechanism of the msgpack-c
Unpack mechanism of the msgpack-c
Unpack mechanism of the msgpack-c
Unpack mechanism of the msgpack-c
Unpack mechanism of the msgpack-c
Unpack mechanism of the msgpack-c
Unpack mechanism of the msgpack-c
Unpack mechanism of the msgpack-c
Unpack mechanism of the msgpack-c
Unpack mechanism of the msgpack-c
Unpack mechanism of the msgpack-c
Unpack mechanism of the msgpack-c
Unpack mechanism of the msgpack-c
Unpack mechanism of the msgpack-c
Unpack mechanism of the msgpack-c
Unpack mechanism of the msgpack-c
Unpack mechanism of the msgpack-c
Unpack mechanism of the msgpack-c
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Unpack mechanism of the msgpack-c

638

Published on

The unpack strategy of the unpacker class in msgpack-c is complecated. This slide helps to understand the mechanism.

The unpack strategy of the unpacker class in msgpack-c is complecated. This slide helps to understand the mechanism.

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
638
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
1
Comments
0
Likes
3
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. The mechanism of msgpack::unpacker msgpack-c committer Takatoshi Kondo 17/16/2014 Copyright OGIS-RI Co., Ltd.
  • 2. Overview Scenario 1 2 length The length of the “length” field is fixed size length msgpack msgpack msgpack msgpack Scenario 2 Receiving a fixed length msgpack packets msgpack packets with no length information (e.g. streaming) msgpack msgpack msgpack msgpack 7/16/2014 Copyright OGIS-RI Co., Ltd.
  • 3. Overview Client msgpack::unpacker • Client receives msgpack data asynchronously. • Data is separated as packets. • The packet format is as follows: length payload(msgpack format ) 2. receive “length” 3.1. allocate length bytes memory 3.reserve_buffer(length) unpacker may allocate more than “length” bytes for optimization. 1.create 4.buf = buffer() msgpack::unpacked 7. unp = create 8.bRet = next(unp) 8.1. set unpacked contents 9.obj = get() 10.z = zone() 3 Scenario 1 7/16/2014 Copyright OGIS-RI Co., Ltd. 6.buffer_consumed(received) 5. reveived = receive payload into buf
  • 4. Overview Client msgpack::unpacker 3.reserve_buffer(length) 1.create 4.buf = buffer() msgpack::unpacked 7. unp = create 8.bRet = next(unp) 9.obj = get() 10.z = zone() do application processes using obj and z 47/16/2014 Copyright OGIS-RI Co., Ltd. • Client receives msgpack data asynchronously. • Data is separated as packets. • The packet format is as follows: Scenario 1 length payload(msgpack format ) 2. receive “length” 3.1. allocate length bytes memory 8.1. set unpacked contents 6.buffer_consumed(received) 5. reveived = receive payload into buf
  • 5. Overview Client msgpack::unpacker 3.reserve_buffer(length) 1.create 4.buf = buffer() msgpack::unpacked 7. unp = create 8.bRet = next(unp) 9.obj = get() 10.z = zone() loop[bRet==true] next() returns true while an object is unpacked. 57/16/2014 Copyright OGIS-RI Co., Ltd. • Client receives msgpack data asynchronously. • Data is separated as packets. • The packet format is as follows: Scenario 1 length payload(msgpack format ) 2. receive “length” 3.1. allocate length bytes memory 8.1. set unpacked contents 6.buffer_consumed(received) 5. reveived = receive payload into buf
  • 6. loop[bRet==true] Overview Client msgpack::unpacker 3.reserve_buffer(length) 1.create 4.buf = buffer() 6.buffer_consumed(received) msgpack::unpacked 7. unp = create 8.bRet = next(unp) 9.obj = get() 10.z = zone() loop infinite loop for data receiving 67/16/2014 Copyright OGIS-RI Co., Ltd. • Client receives msgpack data asynchronously. • Data is separated as packets. • The packet format is as follows: Scenario 1 length payload(msgpack format ) 2. receive “length” 3.1. allocate length bytes memory 5. reveived = receive payload into buf 8.1. set unpacked contents
  • 7. loop [total received < length ] loop[bRet==true] Overview Client msgpack::unpacker 3.reserve_buffer(length) 1.create 4.buf = buffer() msgpack::unpacked 7. unp = create 8.bRet = next(unp) 9.obj = get() 10.z = zone() When payoads are received intermittently, this loop is effective. If msgpack object is incomplete and additional receive is required, the return false. The position of buf is advanced to received size. 77/16/2014 Copyright OGIS-RI Co., Ltd. • Client receives msgpack data asynchronously. • Data is separated as packets. • The packet format is as follows: Scenario 1 length payload(msgpack format ) 6.buffer_consumed(received) 2. receive “length” 3.1. allocate length bytes memory 5. reveived = receive payload into buf 8.1. set unpacked contents
  • 8. loop loop[bRet==true] Overview Client msgpack::unpacker 3.reserve_buffer(length) 1.create 4.buf = buffer() msgpack::unpacked 7. unp = create 8.bRet = next(unp) 9.obj = get() 10.z = zone() 8 The combination of each loop. 7/16/2014 Copyright OGIS-RI Co., Ltd. • Client receives msgpack data asynchronously. • Data is separated as packets. • The packet format is as follows: Scenario 1 length payload(msgpack format ) 6.buffer_consumed(received) 2. receive “length” 3.1. allocate length bytes memory 5. reveived = receive payload into buf 8.1. set unpacked contents loop [total received < length ]
  • 9. payload (msgpack format variable length data) loop loop [cap > threashold] loop[bRet==true] Overview Client msgpack::unpacker msgpack packets with no length information (e.g. streaming) 2.1. allocate at least N bytes memory 2.reserve_buffer(specific value N) 1.create 3.buf = buffer() 5.R=copy at most “cap” size data into buf 6.buffer_consumed(R) msgpack::unpacked 7. unp = create 8.bRet = next(unp) 9.obj = get() 10.z = zone() R is an actual received size e.g. N=1024 4.cap = buffer_capacity() cap >= N 9 Scenario 2 while cap is enough, continue receiving 7/16/2014 Copyright OGIS-RI Co., Ltd. 8.1. set unpacked contents
  • 10. Internal Structure memory management of unpacker unpacker char* m_buffer size_t m_used size_t m_free size_t m_off size_t m_parsed size_t m_initial_buffer_size zone m_z context m_ctx char* buffer() size_t buffer_capacity() void buffer_consumed(size) bool next() ... m_buffer+ m_used m_free m_used += size m_free += size 10 context unpack_stack unsigned int m_top unpack_stack m_stack[] ... undsigned int m_container_type object m_obj size_t m_count ... 7/16/2014 Copyright OGIS-RI Co., Ltd. https://github.com/msgpack/msgpack-c/tree/poc/0.6 The names of data members are based on the following branch:
  • 11. Internal Structure m_parsed = 0 m_buffer COUNTER_SIZE The start bytes of m_buffer is used as a reference counter constructor(init_size=default_init_size) If init_size is less than COUNTER_SIZE, then init_size=COUNTER_SIZE. 1 m_used = COUTNER_SIZE m_free = init_size - m_used m_off = COUNTER_SIZE construct m_buffer m_buffer+m_used m_m_initial_buffer_size = init_size 117/16/2014 Copyright OGIS-RI Co., Ltd. unpacker char* m_buffer size_t m_used size_t m_free size_t m_off size_t m_parsed size_t m_initial_buffer_size zone m_z context m_ctx char* buffer() size_t buffer_capacity() void buffer_consumed(size) bool next() ... m_buffer+ m_used m_free m_used += size m_free += size memory management of unpacker
  • 12. received data1 Internal Structure m_parsed = 0 Write received data into the memory that is returned by buffer() 1 m_used = COUTNER_SIZE m_free = init_size - m_used m_off = COUNTER_SIZE Writing received data m_buffer m_buffer+m_used m_initial_buffer_size = init_size received data1 m_parsed = 0 m_used += size; m_free -= size; 1 m_free m_off = COUNTER_SIZE buffer_consumed(size) m_buffer m_initial_buffer_size = init_size m_buffer+m_used msgpack(array, size=3) bool uint32 string received data1 num number of array elements 127/16/2014 Copyright OGIS-RI Co., Ltd. memory management of unpacker
  • 13. uint32 boolnum num Internal Structure received data1 m_parsed = 0 Parse from m_buffer+m_off 1 m_free next() m_buffer m_buffer + m_usedm_buffer+m_off received data1 Parse “num”, then store object into the top level of m_ctx. Advance m_off, increase m_parsed. 1 m_buffer m_buffer + m_off In order to parse the elements of array, decrement an internal level of the m_ctx. Parse “bool” and set object on m_ctx. Advance m_off, increase m_parsed. msgpack(array, size=3) bool uint32 string received data1 num boolnum received data11 m_buffer m_buffer + m_off There are not enough data to parse“uint32”, then return false. m_off and m_parsed are not updated. initialize the top level count of the m_ctx with num decrement the top level count of the m_ctx num m_parsed num m_parsed bool 13 “num” field means that this is an array object that has three elements. array size=numm_stack m_top m_stack m_top Array structure processes recursively. In order to implement these processes as loop, unpacker uses a stack structure. 7/16/2014 Copyright OGIS-RI Co., Ltd. memory management of unpacker
  • 14. uint32boolnum uint32boolnum received data2 Internal Structure received data1 Do nothing because m_free >= size. 1 m_free reserve_buffer(size) m_buffer m_buffer + m_usedm_buffer + m_off m_used += size; m_free -= size; buffer_consumed(size) received data2received data11 m_free m_buffer m_buffer + m_used msgpack(array, size=3) bool uint32 string received data1 num received data2 size m_buffer+ m_off num m_parsed bool 14 m_stack m_top 7/16/2014 Copyright OGIS-RI Co., Ltd. memory management of unpacker Writing received data Write received data into the memory that is returned by buffer()
  • 15. uint32uint32boolnum m_parsed uint32boolnum m_buffer + m_off m_parsed string Internal Structure received data2received data11 m_free m_buffer m_buffer + m_usedm_buffer + m_off next() received data2received data11 m_free m_buffer m_buffer + m_used msgpack(array, size=3) bool uint32 string received data1 num received data2 size num m_parsed bool 15 m_stack m_top 7/16/2014 Copyright OGIS-RI Co., Ltd. memory management of unpacker Parse from m_buffer+m_off Parse “uint32”, then store object into m_ctx. Advance m_off, increase m_parsed. There are not enough data to parse“string”, then return false. m_off and m_parsed are not updated.
  • 16. 1 string not parsed data stringuint32boolnum Internal Structure m_free < size then allocate a new buffer. reserve_buffer(size) received data2received data1 m_free m_buffer m_buffer+m_used msgpack(array, size=3) bool uint32 string received data1 num received data2 size m_buffer+m_off received data3 next_size = m_initial_buffer_size m_initial_buffer_size not_parsed not_parsed = m_used - m_off The data that is not parsed yet in the previous buffer. 1 COUNTER_SIZE While next_size is less than size + not_parsed + COUNTER_SIZE, next_size *= 2 m_buffer+m_usedm_buffer+m_off m_free Update m_off, m_used, m_free as follows: m_off = COUNTER_SIZE m_used = not_parsed + COUNTER_SIZE m_free = next_size - m_used Decrement the previous buffer’s reference count and if it is zero, then free it. uint32bool num m_parsed m_parsed is not updated 16 Copy the not parsed data to newly allocated buffer. m_stack m_top 7/16/2014 Copyright OGIS-RI Co., Ltd. memory management of unpacker
  • 17. 10 string not parsed data stringuint32boolnum Internal Structure m_free < size then allocate a new buffer. reserve_buffer(size) received data2received data1 m_free m_buffer m_buffer+m_used msgpack(array, size=3) bool uint32 string received data1 num received data2 size m_buffer+m_off received data3 next_size = m_initial_buffer_size m_initial_buffer_size not_parsed not_parsed = m_used - m_off The data that is not parsed yet in the previous buffer. 1 COUNTER_SIZE While next_size is less than size + not_parsed + COUNTER_SIZE, next_size *= 2 m_buffer+m_usedm_buffer+m_off m_free Update m_off, m_used, m_free as follows: m_off = COUNTER_SIZE m_used = not_parsed + COUNTER_SIZE m_free = next_size - m_used Decrement the previous buffer’s reference count and if it is zero, then free it. uint32bool num m_parsed m_parsed is not updated 17 Copy the not parsed data to newly allocated buffer. m_stack m_top 7/16/2014 Copyright OGIS-RI Co., Ltd. memory management of unpacker
  • 18. string not parsed data Internal Structure m_free < size then allocate a new buffer. reserve_buffer(size) msgpack(array, size=3) bool uint32 string received data1 num received data2 size received data3 next_size = m_initial_buffer_size not_parsed = m_used - m_off The data that is not parsed yet in the previous buffer. 1 COUNTER_SIZE While next_size is less than size + not_parsed + COUNTER_SIZE, next_size *= 2 m_buffer+m_usedm_buffer+m_off m_free Update m_off, m_used, m_free as follows: m_off = COUNTER_SIZE m_used = not_parsed + COUNTER_SIZE m_free = next_size - m_used Decrement the previous buffer’s reference count and if it is zero, then free it. uint32bool num m_parsed m_parsed is not updated 18 Copy the not parsed data to newly allocated buffer. m_stack m_top 7/16/2014 Copyright OGIS-RI Co., Ltd. memory management of unpacker
  • 19. 1 received data3 string not parsed data Internal Structure msgpack(array, size=3) bool uint32 string received data1 num received data2 size received data3 1 COUNTER_SIZE m_buffer+m_usedm_buffer+m_off m_free m_used += size; m_free -= size; buffer_consumed(size) received data3 string not parsed data1 COUNTER_SIZE m_buffer+m_usedm_buffer+m_off m_free next() uint32bool num m_parsed received data3 string not parsed data COUNTER_SIZE m_buffer+m_used m_buffer+m_off m_free Decrement the top level count of m_ctx string Because of the top level count of m_ctx is zero, msgpack object has fully constructed. As the result of that next() returns true. 19 m_stack m_top array size=num 7/16/2014 Copyright OGIS-RI Co., Ltd. memory management of unpacker Writing received data Parse from m_buffer+m_off Write received data into the memory that is returned by buffer() Parse “string”, then store object into m_ctx. Advance m_off, increase m_parsed.
  • 20. 12 received data3 string not parsed data Internal Structure msgpack(array, size=3) bool uint32 string received data1 num received data2 size received data3 1 COUNTER_SIZE m_buffer+m_usedm_buffer+m_off m_free m_used += size; m_free -= size; buffer_consumed(size) received data3 string not parsed data1 COUNTER_SIZE m_buffer+m_usedm_buffer+m_off m_free next() uint32bool num m_parsed received data3 string not parsed data COUNTER_SIZE m_buffer+m_used m_buffer+m_off m_free Decrement the top level count of m_ctx string Because of the top level count of m_ctx is zero, msgpack object has fully constructed. As the result of that next() returns true. When the msgpack object type is string or bin, increment the reference counter 20 m_stack m_top array size=num 7/16/2014 Copyright OGIS-RI Co., Ltd. memory management of unpacker Writing received data Parse from m_buffer+m_off Write received data into the memory that is returned by buffer() Parse “string”, then store object into m_ctx. Advance m_off, increase m_parsed.
  • 21. 12 received data3 string not parsed data Internal Structure msgpack(array, size=3) bool uint32 string received data1 num received data2 size received data3 1 COUNTER_SIZE m_buffer+m_usedm_buffer+m_off m_free m_used += size; m_free -= size; buffer_consumed(size) received data3 string not parsed data1 COUNTER_SIZE m_buffer+m_usedm_buffer+m_off m_free next() uint32bool num m_parsed received data3 string not parsed data COUNTER_SIZE m_buffer+m_used m_buffer+m_off m_free Decrement the top level count of m_ctx string Because of the top level count of m_ctx is zero, msgpack object has fully constructed. As the result of that next() returns true. When the msgpack object type is string or bin, increment the reference counter Instead of copy, msgpack object has a reference of string or bin in the buffer. 21 m_stack m_top array size=num 7/16/2014 Copyright OGIS-RI Co., Ltd. memory management of unpacker Writing received data Parse from m_buffer+m_off Write received data into the memory that is returned by buffer() Parse “string”, then store object into m_ctx. Advance m_off, increase m_parsed.
  • 22. zone, unpacker, context, object 22 client unpacker create and request unpack 7/16/2014 Copyright OGIS-RI Co., Ltd.
  • 23. zone, unpacker, context, object 23 client unpacker zone context buffer create createcreate 7/16/2014 Copyright OGIS-RI Co., Ltd. create and request unpack
  • 24. zone, unpacker, context, object 24 client unpacker zone context object buffer create createcreate reference object(bin,string) create place the object on the zone place the object on the zonemanaged by a reference count 7/16/2014 Copyright OGIS-RI Co., Ltd. create and request unpack
  • 25. zone, unpacker, context, object 25 client unpacker zone context object buffer create createcreate reference object(bin,string) create acquire the ownership from the unpacker 7/16/2014 Copyright OGIS-RI Co., Ltd. create and request unpack
  • 26. zone, unpacker, context, object 26 client zone object buffer reference object(raw,string) 7/16/2014 Copyright OGIS-RI Co., Ltd. acquire the ownership from the unpacker

×