RMod
1
Mariano Martinez Peck
marianopeck@gmail.com
http://marianopeck.wordpress.com/
Monday, August 22, 2011
THANKS a lot!!!
2
SummerTalk 2011
Student: Martin Dias
Mentor: Mariano Martínez Peck
Monday, August 22, 2011
Object references
3
Monday, August 22, 2011
Object graph
4
Monday, August 22, 2011
5
An objet graph serializer.
Monday, August 22, 2011
Serialize
6
Input: an object graph
Output: stream of bytes
Monday, August 22, 2011
Materialize(deserialize)
7
Output: an object graph
Input: stream of bytes
Monday, August 22, 2011
Once serialized...
8
Database File Memory Socket
Stream of bytes
Monday, August 22, 2011
Fuel’s main goals
Provide fast object serialization and materialization.
Be flexible and easy to customize.
Have a good OO ...
Key features
Fast serialization and materialization.
Class reshape support.
Serialization of any kind of object.
Cycles su...
Key Characteristics
Pickle format.
Objects grouped in clusters.
Analysis phase before writing.
Stack over recursion.
Two p...
Pickle format
12
Invest more time in serialization so that objects can then be
materialized much faster.
Monday, August 22...
Grouping objects in
clusters
13
“Similar” objects (they share writing/loading information) are
grouped together in cluster...
14
Monday, August 22, 2011
14
Each jar has
a specific
type of
element
Monday, August 22, 2011
14
Each jar has
a specific
type of
element
Jars are in
order
Monday, August 22, 2011
14
Label:
- What’s inside?
- How much?Each jar has
a specific
type of
element
Jars are in
order
Monday, August 22, 2011
14
Label:
- What’s inside?
- How much?
Different sizes
and different
amounts of
elements
Each jar has
a specific
type of
el...
14
Label:
- What’s inside?
- How much?
Different sizes
and different
amounts of
elements
Each jar has
a specific
type of
el...
14
Label:
- What’s inside?
- How much?
Different sizes
and different
amounts of
elements
Each jar has
a specific
type of
el...
14
Label:
- What’s inside?
- How much?
Different sizes
and different
amounts of
elements
Jars are in
order
s/jar/cluster
E...
14
Label:
- What’s inside?
- How much?
Different sizes
and different
amounts of
elements
Jars are in
order
s/jar/cluster
E...
14
Label:
- What’s inside?
- How much?
Different sizes
and different
amounts of
elements
s/jar/cluster
Each cluster
has a ...
14
Label:
- What’s inside?
- How much?
Different sizes
and different
amounts of
elements
s/jar/cluster
Each cluster
has a ...
14
Different sizes
and different
amounts of
elements
s/jar/cluster
Each cluster
has a specific
type of object
Clusters are
...
14
Different sizes
and different
amounts of
elements
s/jar/cluster
Each cluster
has a specific
type of object
Clusters are
...
14
s/jar/cluster
Each cluster
has a specific
type of object
Clusters are
in order
Label:
- Cluster ID
- Amount of
objects
D...
Pickle format basic
15
Stream
Monday, August 22, 2011
Pickle format basic
15
Stream
Monday, August 22, 2011
Pickle format basic
15
Stream
Monday, August 22, 2011
Pickle format basic
15
Stream
Monday, August 22, 2011
16
Why the pickle format is so fast in materialization?
Standard serializers Fuel pickle format
Monday, August 22, 2011
16
Why the pickle format is so fast in materialization?
Standard serializers Fuel pickle format
Recursive
materialization ...
Pickle advantages
Batch/Bulk/Iterative materialization.
Efficient since types are stored and fetch only
once.
Fast because ...
There is no silver
bullet...
18
Fast serialization
(without pickle)
Slow serialization
(with pickle)
Monday, August 22, 20...
Fuel requires
Traversing the object graph.
Mapping each object to a specific cluster.
19
This is done in a phase before
ser...
Analysis phase
20
cluster1 aPoint
cluster2 x y
...
...
... ...
key (a cluster) value (a set)
1) Traverse
(#fuelAccept: aVi...
Serialization
21
key (a cluster) value (a set)
serialize: anObject on: aStream
materializeFrom: aStream
ID
Cluster
A clust...
Stack over recursion
22
To traverse the object graph, Fuel uses a custom
stack implementation rather than a recursion.
Mon...
Basic steps
Serialization
1. Analyze.
2. Serialize header.
3. Serialize instances.
4. Serialize references.
5. Serialize r...
Fuel for software(so far)
Moose export utility.
SandstoneDB persistence.
Pier kernel persistence.
Newspeak language.
Marea...
Future Work
Continue efforts on performance optimization.
Create a tool for loading class and trait
packages.
Support user...
Future work 2
Enable to deploy serialization and
materialization behavior independently.
Support object replacement for se...
Serialization of primitive objects
27
Memory based stream
Monday, August 22, 2011
Serialization of primitive objects
27
Memory based stream
StOMP
SRP
Fuel
Monday, August 22, 2011
Serialization of primitive objects
27
Memory based stream File based stream
StOMP
SRP
Fuel
Monday, August 22, 2011
Serialization of primitive objects
27
Memory based stream File based stream
StOMP
SRP
Fuel
Fuel
SRP
StOMP
Monday, August 2...
28
Materialization
of primitive objects
Monday, August 22, 2011
28
Fuel
ImageSegment
StOMP
Materialization
of primitive objects
Monday, August 22, 2011
Non primitive objects
29
Monday, August 22, 2011
Non primitive objects
29
StOMP
SRP
Fuel
Monday, August 22, 2011
Non primitive objects
29
StOMP
SRP
Fuel
Monday, August 22, 2011
Non primitive objects
29
Fuel
ImageSegment
StOMP
StOMP
SRP
Fuel
Monday, August 22, 2011
Links
Website: http://rmod.lille.inria.fr/web/pier/
software/Fuel
Issue tracker: http://code.google.com/p/fuel
Source repo...
Excellent performance without special support
from VM and good OO design.
Conclusion
for us
31
Monday, August 22, 2011
Fuel is a vehicle. It is infrastructure. You can
build cool stuff on top of it.
Conclusion
for YOU
32
Monday, August 22, 2...
Thanks!
RMod
Mariano Martinez Peck
marianopeck@gmail.com
http://marianopeck.wordpress.com/
Monday, August 22, 2011
34
Concrete example
Monday, August 22, 2011
35
Monday, August 22, 2011
Analysis phase
3620
cluster1 aRectangle
cluster2 anOrigin aCorner
cluster3 10
key (a cluster) value (a IdentitySet)
20 30 ...
Serialization
instances
phase
cluster1 aRectangle
cluster2 anOrigin aCorner
cluster3 10 20 30 40
Monday, August 22, 2011
Serialization
instances
phase
cluster1 aRectangle
cluster2 anOrigin aCorner
cluster3 10 20 30 40
Monday, August 22, 2011
Serialization
instances
phase
cluster1 aRectangle
cluster2 anOrigin aCorner
cluster3 10 20 30 40
1 Rectangle 2 origi corne...
Serialization
instances
phase
cluster1 aRectangle
cluster2 anOrigin aCorner
cluster3 10 20 30 40
1 Rectangle 2 origi corne...
Serialization
instances
phase
cluster1 aRectangle
cluster2 anOrigin aCorner
cluster3 10 20 30 40
1 Rectangle 2 origi corne...
Serialization
instances
phase
cluster1 aRectangle
cluster2 anOrigin aCorner
cluster3 10 20 30 40
1 Rectangle 2 origi corne...
cluster1 aRectangle
cluster2 anOrigin aCorner
cluster3 10 20 30 40
aRectangle anOrigiaCorner20 40 10 30
instancesIndex
(Id...
cluster1 aRectangle
cluster2 anOrigin aCorner
cluster3 10 20 30 40
aRectangle anOrigiaCorner20 40 10 30
instancesIndex
(Id...
cluster1 aRectangle
cluster2 anOrigin aCorner
cluster3 10 20 30 40
aRectangle anOrigiaCorner20 40 10 30
instancesIndex
(Id...
cluster1 aRectangle
cluster2 anOrigin aCorner
cluster3 10 20 30 40
aRectangle anOrigiaCorner20 40 10 30
instancesIndex
(Id...
2 3
4 5 6 7
final stream
1 Rectangle 2 origi corne 1
1 Point 2 x y 2
54 10 20 30 40
{instances
{references
Monday, August ...
2 3
4 5 6 7
final stream
1 Rectangle 2 origi corne 1
1 Point 2 x y 2
54 10 20 30 40
{instances
{references
1{trailer
7 3{h...
Materialization
instances
{
2 3
4 5 6 7
1 Rectangle 2 origin corner 1
1 Point 2 x y 2
5 4 1 2 3 4
{references
1
{trailer
7...
41
Monday, August 22, 2011
Upcoming SlideShare
Loading in …5
×

Clustered Serialization with Fuel

393 views

Published on

Presentation of the research paper entitled "Clustered Serialization with Fuel" in ESUG International Workshop on Smalltalk Technologies (IWST 2011), Edinburgh, Scotland.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
393
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
4
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Clustered Serialization with Fuel

  1. 1. RMod 1 Mariano Martinez Peck marianopeck@gmail.com http://marianopeck.wordpress.com/ Monday, August 22, 2011
  2. 2. THANKS a lot!!! 2 SummerTalk 2011 Student: Martin Dias Mentor: Mariano Martínez Peck Monday, August 22, 2011
  3. 3. Object references 3 Monday, August 22, 2011
  4. 4. Object graph 4 Monday, August 22, 2011
  5. 5. 5 An objet graph serializer. Monday, August 22, 2011
  6. 6. Serialize 6 Input: an object graph Output: stream of bytes Monday, August 22, 2011
  7. 7. Materialize(deserialize) 7 Output: an object graph Input: stream of bytes Monday, August 22, 2011
  8. 8. Once serialized... 8 Database File Memory Socket Stream of bytes Monday, August 22, 2011
  9. 9. Fuel’s main goals Provide fast object serialization and materialization. Be flexible and easy to customize. Have a good OO design, well tested and benchmarked. No need of special support from the VM. Be a general purpose serializer. Allow tools to be built on top of Fuel. 9 Monday, August 22, 2011
  10. 10. Key features Fast serialization and materialization. Class reshape support. Serialization of any kind of object. Cycles support. Global objects references. Buffered writing. Support for some “hook methods”. 10 Monday, August 22, 2011
  11. 11. Key Characteristics Pickle format. Objects grouped in clusters. Analysis phase before writing. Stack over recursion. Two phases for writing instances and references. Iterative graph recreation. 11 Monday, August 22, 2011
  12. 12. Pickle format 12 Invest more time in serialization so that objects can then be materialized much faster. Monday, August 22, 2011
  13. 13. Grouping objects in clusters 13 “Similar” objects (they share writing/loading information) are grouped together in clusters. The most common case, yet not the only one, takes place when a class is a cluster for its instances. Monday, August 22, 2011
  14. 14. 14 Monday, August 22, 2011
  15. 15. 14 Each jar has a specific type of element Monday, August 22, 2011
  16. 16. 14 Each jar has a specific type of element Jars are in order Monday, August 22, 2011
  17. 17. 14 Label: - What’s inside? - How much?Each jar has a specific type of element Jars are in order Monday, August 22, 2011
  18. 18. 14 Label: - What’s inside? - How much? Different sizes and different amounts of elements Each jar has a specific type of element Jars are in order Monday, August 22, 2011
  19. 19. 14 Label: - What’s inside? - How much? Different sizes and different amounts of elements Each jar has a specific type of element Jars are in order s/jar/cluster Monday, August 22, 2011
  20. 20. 14 Label: - What’s inside? - How much? Different sizes and different amounts of elements Each jar has a specific type of element Jars are in order s/jar/cluster Monday, August 22, 2011
  21. 21. 14 Label: - What’s inside? - How much? Different sizes and different amounts of elements Jars are in order s/jar/cluster Each cluster has a specific type of object Monday, August 22, 2011
  22. 22. 14 Label: - What’s inside? - How much? Different sizes and different amounts of elements Jars are in order s/jar/cluster Each cluster has a specific type of object Monday, August 22, 2011
  23. 23. 14 Label: - What’s inside? - How much? Different sizes and different amounts of elements s/jar/cluster Each cluster has a specific type of object Clusters are in order Monday, August 22, 2011
  24. 24. 14 Label: - What’s inside? - How much? Different sizes and different amounts of elements s/jar/cluster Each cluster has a specific type of object Clusters are in order Monday, August 22, 2011
  25. 25. 14 Different sizes and different amounts of elements s/jar/cluster Each cluster has a specific type of object Clusters are in order Label: - Cluster ID - Amount of objects Monday, August 22, 2011
  26. 26. 14 Different sizes and different amounts of elements s/jar/cluster Each cluster has a specific type of object Clusters are in order Label: - Cluster ID - Amount of objects Monday, August 22, 2011
  27. 27. 14 s/jar/cluster Each cluster has a specific type of object Clusters are in order Label: - Cluster ID - Amount of objects Different sizes and different amounts of objects Monday, August 22, 2011
  28. 28. Pickle format basic 15 Stream Monday, August 22, 2011
  29. 29. Pickle format basic 15 Stream Monday, August 22, 2011
  30. 30. Pickle format basic 15 Stream Monday, August 22, 2011
  31. 31. Pickle format basic 15 Stream Monday, August 22, 2011
  32. 32. 16 Why the pickle format is so fast in materialization? Standard serializers Fuel pickle format Monday, August 22, 2011
  33. 33. 16 Why the pickle format is so fast in materialization? Standard serializers Fuel pickle format Recursive materialization Iterative materialization Monday, August 22, 2011
  34. 34. Pickle advantages Batch/Bulk/Iterative materialization. Efficient since types are stored and fetch only once. Fast because at materialization we know the size of everything. The generated stream is smaller. More next.... 17 Monday, August 22, 2011
  35. 35. There is no silver bullet... 18 Fast serialization (without pickle) Slow serialization (with pickle) Monday, August 22, 2011
  36. 36. Fuel requires Traversing the object graph. Mapping each object to a specific cluster. 19 This is done in a phase before serialization called “Analysis”. Monday, August 22, 2011
  37. 37. Analysis phase 20 cluster1 aPoint cluster2 x y ... ... ... ... key (a cluster) value (a set) 1) Traverse (#fuelAccept: aVisitor) 2) Fill dictionary (#fuelSerializer) Monday, August 22, 2011
  38. 38. Serialization 21 key (a cluster) value (a set) serialize: anObject on: aStream materializeFrom: aStream ID Cluster A cluster defines how its objects are serialized and materialized. cluster1 aPoint cluster2 x y ... ... ... ... Monday, August 22, 2011
  39. 39. Stack over recursion 22 To traverse the object graph, Fuel uses a custom stack implementation rather than a recursion. Monday, August 22, 2011
  40. 40. Basic steps Serialization 1. Analyze. 2. Serialize header. 3. Serialize instances. 4. Serialize references. 5. Serialize root. Materialization 1. Materialize header. 2. Materialize instances. 3. Materialize references. 4. Materialize root. 23 Monday, August 22, 2011
  41. 41. Fuel for software(so far) Moose export utility. SandstoneDB persistence. Pier kernel persistence. Newspeak language. Marea (my own research project!). 24 Monday, August 22, 2011
  42. 42. Future Work Continue efforts on performance optimization. Create a tool for loading class and trait packages. Support user-defined Singletons. Fast statistics/brief info extraction of a stored graph. Partial loading of a stored graph. 25 Monday, August 22, 2011
  43. 43. Future work 2 Enable to deploy serialization and materialization behavior independently. Support object replacement for serialization and materialization. Allow cycle detections to be disabled. Partial loading. 26 Monday, August 22, 2011
  44. 44. Serialization of primitive objects 27 Memory based stream Monday, August 22, 2011
  45. 45. Serialization of primitive objects 27 Memory based stream StOMP SRP Fuel Monday, August 22, 2011
  46. 46. Serialization of primitive objects 27 Memory based stream File based stream StOMP SRP Fuel Monday, August 22, 2011
  47. 47. Serialization of primitive objects 27 Memory based stream File based stream StOMP SRP Fuel Fuel SRP StOMP Monday, August 22, 2011
  48. 48. 28 Materialization of primitive objects Monday, August 22, 2011
  49. 49. 28 Fuel ImageSegment StOMP Materialization of primitive objects Monday, August 22, 2011
  50. 50. Non primitive objects 29 Monday, August 22, 2011
  51. 51. Non primitive objects 29 StOMP SRP Fuel Monday, August 22, 2011
  52. 52. Non primitive objects 29 StOMP SRP Fuel Monday, August 22, 2011
  53. 53. Non primitive objects 29 Fuel ImageSegment StOMP StOMP SRP Fuel Monday, August 22, 2011
  54. 54. Links Website: http://rmod.lille.inria.fr/web/pier/ software/Fuel Issue tracker: http://code.google.com/p/fuel Source repository: http://www.squeaksource.com/ Fuel Continuous integration server: https://pharo- ic.lille.inria.fr/hudson/job/Fuel/ 30 Monday, August 22, 2011
  55. 55. Excellent performance without special support from VM and good OO design. Conclusion for us 31 Monday, August 22, 2011
  56. 56. Fuel is a vehicle. It is infrastructure. You can build cool stuff on top of it. Conclusion for YOU 32 Monday, August 22, 2011
  57. 57. Thanks! RMod Mariano Martinez Peck marianopeck@gmail.com http://marianopeck.wordpress.com/ Monday, August 22, 2011
  58. 58. 34 Concrete example Monday, August 22, 2011
  59. 59. 35 Monday, August 22, 2011
  60. 60. Analysis phase 3620 cluster1 aRectangle cluster2 anOrigin aCorner cluster3 10 key (a cluster) value (a IdentitySet) 20 30 40 Analysis phase Monday, August 22, 2011
  61. 61. Serialization instances phase cluster1 aRectangle cluster2 anOrigin aCorner cluster3 10 20 30 40 Monday, August 22, 2011
  62. 62. Serialization instances phase cluster1 aRectangle cluster2 anOrigin aCorner cluster3 10 20 30 40 Monday, August 22, 2011
  63. 63. Serialization instances phase cluster1 aRectangle cluster2 anOrigin aCorner cluster3 10 20 30 40 1 Rectangle 2 origi corne 1 instance variable names instance count instance variable count class name cluster ID Monday, August 22, 2011
  64. 64. Serialization instances phase cluster1 aRectangle cluster2 anOrigin aCorner cluster3 10 20 30 40 1 Rectangle 2 origi corne 1 instance variable names instance count instance variable count class name cluster ID Monday, August 22, 2011
  65. 65. Serialization instances phase cluster1 aRectangle cluster2 anOrigin aCorner cluster3 10 20 30 40 1 Rectangle 2 origi corne 1 1 Point 2 x y 2 instance variable names instance count instance variable count class name cluster ID Monday, August 22, 2011
  66. 66. Serialization instances phase cluster1 aRectangle cluster2 anOrigin aCorner cluster3 10 20 30 40 1 Rectangle 2 origi corne 1 1 Point 2 x y 2 54 10 20 30 40 instance variable names instance count instance variable count class name cluster ID Monday, August 22, 2011
  67. 67. cluster1 aRectangle cluster2 anOrigin aCorner cluster3 10 20 30 40 aRectangle anOrigiaCorner20 40 10 30 instancesIndex (IdentityDictionary) 1 5 7 3 2 4 6 54 10 20 30 40 Serialization references phase instances references Monday, August 22, 2011
  68. 68. cluster1 aRectangle cluster2 anOrigin aCorner cluster3 10 20 30 40 aRectangle anOrigiaCorner20 40 10 30 instancesIndex (IdentityDictionary) 1 5 7 3 2 4 6 54 10 20 30 40 2 3 Serialization references phase instances references Monday, August 22, 2011
  69. 69. cluster1 aRectangle cluster2 anOrigin aCorner cluster3 10 20 30 40 aRectangle anOrigiaCorner20 40 10 30 instancesIndex (IdentityDictionary) 1 5 7 3 2 4 6 54 10 20 30 40 2 3 4 5 6 7 Serialization references phase instances references Monday, August 22, 2011
  70. 70. cluster1 aRectangle cluster2 anOrigin aCorner cluster3 10 20 30 40 aRectangle anOrigiaCorner20 40 10 30 instancesIndex (IdentityDictionary) 1 5 7 3 2 4 6 54 10 20 30 40 2 3 4 5 6 7 Serialization references phase instances references Monday, August 22, 2011
  71. 71. 2 3 4 5 6 7 final stream 1 Rectangle 2 origi corne 1 1 Point 2 x y 2 54 10 20 30 40 {instances {references Monday, August 22, 2011
  72. 72. 2 3 4 5 6 7 final stream 1 Rectangle 2 origi corne 1 1 Point 2 x y 2 54 10 20 30 40 {instances {references 1{trailer 7 3{header clustersCount objectsCounts root Monday, August 22, 2011
  73. 73. Materialization instances { 2 3 4 5 6 7 1 Rectangle 2 origin corner 1 1 Point 2 x y 2 5 4 1 2 3 4 {references 1 {trailer 7 3{header clustersCount objectsCounts root Monday, August 22, 2011
  74. 74. 41 Monday, August 22, 2011

×