Mariano Martinez Peck                                marianopeck@gmail.com                          http://marianopeck.wor...
THANKS a lot!!!                             SummerTalk 2011                          Student: Martin Dias                 ...
Object references                                  3Monday, August 22, 2011
Object graph                               4Monday, August 22, 2011
An objet graph serializer.                                      5Monday, August 22, 2011
Serialize              Input: an object graph                                           Output: stream of bytes           ...
Materialize                               (deserialize)                   Input: stream of bytes                          ...
Once serialized...                              Stream of bytes            Database           File       Memory   Socket  ...
Fuel’s main goals               Provide fast object serialization and materialization.               Be flexible and easy t...
Key features                          Fast serialization and materialization.                          Class reshape suppo...
Key Characteristics                          Pickle format.                          Objects grouped in clusters.         ...
Pickle format      Invest more time in serialization so that objects can then be                      materialized much fa...
Grouping objects in                        clusters “Similar” objects (they share writing/loading information) aregrouped ...
14Monday, August 22, 2011
Each jar has   a specific    type of    element                          14Monday, August 22, 2011
Each jar has   a specific    type of    element    Jars are in      order                          14Monday, August 22, 2011
Label:                               - What’s inside?  Each jar has                   - How much?   a specific    type of  ...
Label:                               - What’s inside?  Each jar has                   - How much?   a specific    type of  ...
s/jar/cluster                                               Label:                                          - What’s insid...
s/jar/cluster                                               Label:                                          - What’s insid...
s/jar/cluster                                               Label:                                          - What’s insid...
s/jar/cluster                                               Label:                                          - What’s insid...
s/jar/cluster                                               Label:                                          - What’s insid...
s/jar/cluster                                               Label:                                          - What’s insid...
s/jar/cluster                                              Label:                                           - Cluster ID  ...
s/jar/cluster                                              Label:                                           - Cluster ID  ...
s/jar/cluster                                              Label:                                           - Cluster ID  ...
Pickle format basic                           Stream                               15Monday, August 22, 2011
Pickle format basic                           Stream                               15Monday, August 22, 2011
Pickle format basic                           Stream                               15Monday, August 22, 2011
Pickle format basic                           Stream                               15Monday, August 22, 2011
Why the pickle format is so fast in materialization?            Standard serializers         Fuel pickle format           ...
Why the pickle format is so fast in materialization?            Standard serializers         Fuel pickle format           ...
Pickle advantages                          Batch/Bulk/Iterative materialization.                           Efficient since ...
There is no silver                              bullet...                                               ation             ...
Fuel requires                          Traversing the object graph.                          Mapping each object to a spec...
Analysis phase                                              1) Traverse                                              (#fue...
Serialization              cluster1     aPoint     ...              cluster2       x         y            ...             ...
Stack over recursion To traverse the object graph, Fuel uses a custom  stack implementation rather than a recursion.      ...
Basic steps                 Serialization                        1. Analyze.                        2. Serialize header.  ...
Fuel for far)                                 (so                                     software                          Mo...
Future Work                 Continue efforts on performance optimization.                Create a tool for loading class a...
Future work 2                          Enable to deploy serialization and                          materialization behavio...
Serialization of primitive objects                Memory based stream                                      27Monday, Augus...
Serialization of primitive objects                Memory based stream               StOMP               SRP               ...
Serialization of primitive objects                Memory based stream        File based stream               StOMP        ...
Serialization of primitive objects                Memory based stream        File based stream               StOMP        ...
Materialization                          of primitive objects                                   28Monday, August 22, 2011
Materialization                          of primitive objects                                             Fuel            ...
Non primitive objects                              29Monday, August 22, 2011
Non primitive objects               StOMP               SRP               Fuel           29Monday, August 22, 2011
Non primitive objects               StOMP               SRP               Fuel           29Monday, August 22, 2011
Non primitive objects               StOMP                Fuel               SRP                  ImageSegment             ...
Links         Website: http://rmod.lille.inria.fr/web/pier/         software/Fuel         Issue tracker: http://code.googl...
Conclusion                               for us                     Excellent performance without special support         ...
Conclusion                                   for YOU                          Fuel is a vehicle. It is infrastructure. You...
Thanks!                                Mariano Martinez Peck                                marianopeck@gmail.com         ...
Concrete example                                 34Monday, August 22, 2011
35Monday, August 22, 2011
Analysis phase                     Analysis      cluster1     aRectangle                      phase        cluster2     an...
cluster1               aRectangle         Serialization   cluster2               anOrigin aCorner    instances   cluster3 ...
cluster1               aRectangle         Serialization   cluster2               anOrigin aCorner    instances   cluster3 ...
cluster1               aRectangle            Serialization   cluster2               anOrigin aCorner       instances   clu...
cluster1               aRectangle            Serialization   cluster2               anOrigin aCorner       instances   clu...
cluster1               aRectangle                Serialization   cluster2               anOrigin aCorner           instanc...
cluster1               aRectangle                   Serialization   cluster2               anOrigin aCorner              i...
cluster1               aRectangle                       Serialization   cluster2               anOrigin aCorner           ...
cluster1               aRectangle                       Serialization   cluster2               anOrigin aCorner           ...
cluster1               aRectangle                       Serialization   cluster2               anOrigin aCorner           ...
cluster1               aRectangle                       Serialization   cluster2               anOrigin aCorner           ...
final stream                          {                               1 Rectangle 2 origi corne 1             instances   ...
final stream                                          objectsCounts                header    {    7 3           clustersCo...
Materialization                                                                      objectsCounts                        ...
41Monday, August 22, 2011
Upcoming SlideShare
Loading in …5
×

Fuel

1,304 views

Published on

ESUG 2011, Edinburgh

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,304
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
16
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Fuel

  1. 1. Mariano Martinez Peck marianopeck@gmail.com http://marianopeck.wordpress.com/ 1Monday, August 22, 2011
  2. 2. THANKS a lot!!! SummerTalk 2011 Student: Martin Dias Mentor: Mariano Martínez Peck 2Monday, August 22, 2011
  3. 3. Object references 3Monday, August 22, 2011
  4. 4. Object graph 4Monday, August 22, 2011
  5. 5. An objet graph serializer. 5Monday, August 22, 2011
  6. 6. Serialize Input: an object graph Output: stream of bytes 6Monday, August 22, 2011
  7. 7. Materialize (deserialize) Input: stream of bytes Output: an object graph 7Monday, August 22, 2011
  8. 8. Once serialized... Stream of bytes Database File Memory Socket 8Monday, August 22, 2011
  9. 9. Fuel’s main goals Provide fast object serialization and materialization. Be flexible and easy to customize. Have a good OO design, well tested and benchmarked. No need of special support from the VM. Be a general purpose serializer. Allow tools to be built on top of Fuel. 9Monday, August 22, 2011
  10. 10. Key features Fast serialization and materialization. Class reshape support. Serialization of any kind of object. Cycles support. Global objects references. Buffered writing. Support for some “hook methods”. 10Monday, August 22, 2011
  11. 11. Key Characteristics Pickle format. Objects grouped in clusters. Analysis phase before writing. Stack over recursion. Two phases for writing instances and references. Iterative graph recreation. 11Monday, August 22, 2011
  12. 12. Pickle format Invest more time in serialization so that objects can then be materialized much faster. 12Monday, August 22, 2011
  13. 13. Grouping objects in clusters “Similar” objects (they share writing/loading information) aregrouped together in clusters. The most common case, yet not the only one, takes place when a class is a cluster for its instances. 13Monday, August 22, 2011
  14. 14. 14Monday, August 22, 2011
  15. 15. Each jar has a specific type of element 14Monday, August 22, 2011
  16. 16. Each jar has a specific type of element Jars are in order 14Monday, August 22, 2011
  17. 17. Label: - What’s inside? Each jar has - How much? a specific type of element Jars are in order 14Monday, August 22, 2011
  18. 18. Label: - What’s inside? Each jar has - How much? a specific type of element Different sizes and different Jars are in amounts of order elements 14Monday, August 22, 2011
  19. 19. s/jar/cluster Label: - What’s inside? Each jar has - How much? a specific type of element Different sizes and different Jars are in amounts of order elements 14Monday, August 22, 2011
  20. 20. s/jar/cluster Label: - What’s inside?Each jar has - How much? a specific type of element Different sizes and different Jars are in amounts of order elements 14Monday, August 22, 2011
  21. 21. s/jar/cluster Label: - What’s inside? - How much? Each cluster has a specific type of object Different sizes and different Jars are in amounts of order elements 14Monday, August 22, 2011
  22. 22. s/jar/cluster Label: - What’s inside? - How much? Each cluster has a specific type of object Different sizes and differentJars are in amounts of order elements 14Monday, August 22, 2011
  23. 23. s/jar/cluster Label: - What’s inside? - How much? Each cluster has a specific type of object Different sizes and different Clusters are amounts of in order elements 14Monday, August 22, 2011
  24. 24. s/jar/cluster Label: - What’s inside? Each cluster - How much? has a specific type of object Different sizes and different Clusters are amounts of in order elements 14Monday, August 22, 2011
  25. 25. s/jar/cluster Label: - Cluster ID Each cluster - Amount of has a specific objects type of object Different sizes and different Clusters are amounts of in order elements 14Monday, August 22, 2011
  26. 26. s/jar/cluster Label: - Cluster ID Each cluster - Amount of has a specific objects type of object Different sizes and different Clusters are amounts of in order elements 14Monday, August 22, 2011
  27. 27. s/jar/cluster Label: - Cluster ID Each cluster - Amount of has a specific objects type of object Different sizes and different Clusters are amounts of in order objects 14Monday, August 22, 2011
  28. 28. Pickle format basic Stream 15Monday, August 22, 2011
  29. 29. Pickle format basic Stream 15Monday, August 22, 2011
  30. 30. Pickle format basic Stream 15Monday, August 22, 2011
  31. 31. Pickle format basic Stream 15Monday, August 22, 2011
  32. 32. Why the pickle format is so fast in materialization? Standard serializers Fuel pickle format 16Monday, August 22, 2011
  33. 33. Why the pickle format is so fast in materialization? Standard serializers Fuel pickle format ive ive n rat tio urs tion ec za Ite iza R ial eri ali at er m at m 16Monday, August 22, 2011
  34. 34. Pickle advantages Batch/Bulk/Iterative materialization. Efficient since types are stored and fetch only once. Fast because at materialization we know the size of everything. The generated stream is smaller. More next.... 17Monday, August 22, 2011
  35. 35. There is no silver bullet... ation rializ e) Fa st se ickl (with out p Slow seri (wit aliza h pi tion ckle ) 18Monday, August 22, 2011
  36. 36. Fuel requires Traversing the object graph. Mapping each object to a specific cluster. This is done in a phase before serialization called “Analysis”. 19Monday, August 22, 2011
  37. 37. Analysis phase 1) Traverse (#fuelAccept: aVisitor) 2) Fill dictionary (#fuelSerializer) cluster1 aPoint ... cluster2 x y ... ... ... key (a cluster) value (a set) 20Monday, August 22, 2011
  38. 38. Serialization cluster1 aPoint ... cluster2 x y ... ... ... key (a cluster) value (a set) Cluster ID serialize: anObject on: aStream materializeFrom: aStream A cluster defines how its objects are serialized and materialized. 21Monday, August 22, 2011
  39. 39. Stack over recursion To traverse the object graph, Fuel uses a custom stack implementation rather than a recursion. 22Monday, August 22, 2011
  40. 40. Basic steps Serialization 1. Analyze. 2. Serialize header. 3. Serialize instances. 4. Serialize references. 5. Serialize root. Materialization 1. Materialize header. 2. Materialize instances. 3. Materialize references. 4. Materialize root. 23Monday, August 22, 2011
  41. 41. Fuel for far) (so software Moose export utility. SandstoneDB persistence. Pier kernel persistence. Newspeak language. Marea (my own research project!). 24Monday, August 22, 2011
  42. 42. Future Work Continue efforts on performance optimization. Create a tool for loading class and trait packages. Support user-defined Singletons. Fast statistics/brief info extraction of a stored graph. Partial loading of a stored graph. 25Monday, August 22, 2011
  43. 43. Future work 2 Enable to deploy serialization and materialization behavior independently. Support object replacement for serialization and materialization. Allow cycle detections to be disabled. Partial loading. 26Monday, August 22, 2011
  44. 44. Serialization of primitive objects Memory based stream 27Monday, August 22, 2011
  45. 45. Serialization of primitive objects Memory based stream StOMP SRP Fuel 27Monday, August 22, 2011
  46. 46. Serialization of primitive objects Memory based stream File based stream StOMP SRP Fuel 27Monday, August 22, 2011
  47. 47. Serialization of primitive objects Memory based stream File based stream StOMP Fuel SRP SRP Fuel 27 StOMPMonday, August 22, 2011
  48. 48. Materialization of primitive objects 28Monday, August 22, 2011
  49. 49. Materialization of primitive objects Fuel StOMP ImageSegment 28Monday, August 22, 2011
  50. 50. Non primitive objects 29Monday, August 22, 2011
  51. 51. Non primitive objects StOMP SRP Fuel 29Monday, August 22, 2011
  52. 52. Non primitive objects StOMP SRP Fuel 29Monday, August 22, 2011
  53. 53. Non primitive objects StOMP Fuel SRP ImageSegment Fuel 29 StOMPMonday, August 22, 2011
  54. 54. Links Website: http://rmod.lille.inria.fr/web/pier/ software/Fuel Issue tracker: http://code.google.com/p/fuel Source repository: http://www.squeaksource.com/ Fuel Continuous integration server: https://pharo- ic.lille.inria.fr/hudson/job/Fuel/ 30Monday, August 22, 2011
  55. 55. Conclusion for us Excellent performance without special support from VM and good OO design. 31Monday, August 22, 2011
  56. 56. Conclusion for YOU Fuel is a vehicle. It is infrastructure. You can build cool stuff on top of it. 32Monday, August 22, 2011
  57. 57. Thanks! Mariano Martinez Peck marianopeck@gmail.com http://marianopeck.wordpress.com/Monday, August 22, 2011
  58. 58. Concrete example 34Monday, August 22, 2011
  59. 59. 35Monday, August 22, 2011
  60. 60. Analysis phase Analysis cluster1 aRectangle phase cluster2 anOrigin aCorner cluster3 10 20 30 40 key (a cluster) value (a IdentitySet) 20 36Monday, August 22, 2011
  61. 61. cluster1 aRectangle Serialization cluster2 anOrigin aCorner instances cluster3 10 20 30 40 phaseMonday, August 22, 2011
  62. 62. cluster1 aRectangle Serialization cluster2 anOrigin aCorner instances cluster3 10 20 30 40 phaseMonday, August 22, 2011
  63. 63. cluster1 aRectangle Serialization cluster2 anOrigin aCorner instances cluster3 10 20 30 40 phase instance variable count instance variable names instance cluster ID 1 Rectangle 2 origi corne 1 count class nameMonday, August 22, 2011
  64. 64. cluster1 aRectangle Serialization cluster2 anOrigin aCorner instances cluster3 10 20 30 40 phase instance variable count instance variable names instance cluster ID 1 Rectangle 2 origi corne 1 count class nameMonday, August 22, 2011
  65. 65. cluster1 aRectangle Serialization cluster2 anOrigin aCorner instances cluster3 10 20 30 40 phase instance variable count instance variable names instance cluster ID 1 Rectangle 2 origi corne 1 count class name 1 Point 2 x y 2Monday, August 22, 2011
  66. 66. cluster1 aRectangle Serialization cluster2 anOrigin aCorner instances cluster3 10 20 30 40 phase instance variable count instance variable names instance cluster ID 1 Rectangle 2 origi corne 1 count class name 1 Point 2 x y 2 54 10 20 30 40Monday, August 22, 2011
  67. 67. cluster1 aRectangle Serialization cluster2 anOrigin aCorner references cluster3 10 20 30 40 phase instancesIndex aRectangle 20 40 aCorner anOrigi 10 30(IdentityDictionary) 1 5 7 3 2 4 6 54 10 20 30 40 instances referencesMonday, August 22, 2011
  68. 68. cluster1 aRectangle Serialization cluster2 anOrigin aCorner references cluster3 10 20 30 40 phase instancesIndex aRectangle 20 40 aCorner anOrigi 10 30(IdentityDictionary) 1 5 7 3 2 4 6 54 10 20 30 40 instances 2 3 referencesMonday, August 22, 2011
  69. 69. cluster1 aRectangle Serialization cluster2 anOrigin aCorner references cluster3 10 20 30 40 phase instancesIndex aRectangle 20 40 aCorner anOrigi 10 30(IdentityDictionary) 1 5 7 3 2 4 6 54 10 20 30 40 instances 2 3 references 4 5 6 7Monday, August 22, 2011
  70. 70. cluster1 aRectangle Serialization cluster2 anOrigin aCorner references cluster3 10 20 30 40 phase instancesIndex aRectangle 20 40 aCorner anOrigi 10 30(IdentityDictionary) 1 5 7 3 2 4 6 54 10 20 30 40 instances 2 3 references 4 5 6 7Monday, August 22, 2011
  71. 71. final stream { 1 Rectangle 2 origi corne 1 instances 1 Point 2 x y 2 54 10 20 30 40 references { 2 3 4 5 6 7Monday, August 22, 2011
  72. 72. final stream objectsCounts header { 7 3 clustersCount { 1 Rectangle 2 origi corne 1 instances 1 Point 2 x y 2 54 10 20 30 40 references { 2 3 4 5 6 7 trailer { 1 rootMonday, August 22, 2011
  73. 73. Materialization objectsCounts header { 7 3 clustersCount 1 Rectangle 2 origin corner 1 instances { 1 Point 2 x y 2 54 1 2 3 4 2 3 references { 4 5 6 7 trailer { 1 rootMonday, August 22, 2011
  74. 74. 41Monday, August 22, 2011

×