upfront diet for plone
Roché Compaan
About me
● Owner of Upfront Systems
● Founded Upfront Systems in 1998
● Zope developer for the past 10 years
Experience
● Largest Plone deployments to date: 5FM & MetroFM
● 5FM
– 20 million page views per month
– 70 thousand member...
Can Plone scale to 10 million documents?
ZODB Benchmarks
● Most common myth spread: “ZODB is a high
read, (s)low write database”
● No published ZODB benchmarks
● W...
ZODB Write Speed
0 500 thousand 2,5 million 5 million 7,5 million 8,5 million 10 million
0
1000
2000
3000
4000
5000
6000
7...
ZODB Lookup Speed
Number of Objects Lookup Time
100,000 0.003 ms
1,000,000 0.006 ms
10,000,000 2 ms
Benchmark Conclusions
● Thanks Jim, ZODB Rocks!
● Large cache size important for writes too, not
only reads
● Need a solut...
A closer look at Plone
● Added 10,000 documents to Plone, while
profiling with Dieter Maurer's ZopeProfiler
● Timed “apply...
Plone Benchmark Conclusions
● portal_catalog remains a bottleneck to scaling a
Plone site.
● 1 catalog == many many confli...
What's in a pickle?
2,51755
81,564
9,872
20,024
85
655
1,544
30,455
182,708
7798,088122
Pickle contents of a single transa...
Think before you index!
“Waking up objects is expensive” - this is an
irrational fear. And adding another catalog in
respo...
The portal_catalog problem
● No navigation, portal tabs, folder contents, review
queue if your content type is not in the ...
What is upfront.diet?
● It's a healthier and happier approach to Plone
development: index only when you really really need...
More about upfront.diet
● upfront.navportlet
● Separate catalog for navportlet
● upfront.simplereferencefield
●
Store refe...
upfront.diet todo list
● Packages:
● Portal tabs (done but not a separate package yet)
● Sitemap
● Review queue
● News and...
upfront.diet deployments
● Very few at the moment but looks promising
● Fullmarks.org.za (half a million objects, Data.fs
...
Contact Me
● Email: roche@upfrontsystems.co.za
● Blog:
http://upfrontsystems.co.za/Members/roche
● facebook.com/rochecompa...
Questions?
upfront diet plone conf 2010
upfront diet plone conf 2010
upfront diet plone conf 2010
Upcoming SlideShare
Loading in …5
×

upfront diet plone conf 2010

1,627
-1

Published on

If you are concerned that your Plone site will grow up to be a fatty, you need to seriously consider upfront.diet. upfront.diet is a collection of fat burning code snippets and disciplines that will keep your Plone site fit and in shape. If you want to write Plone apps that need to scale to millions of objects, you'll need a fundamental understanding of what is inside each pickle in the ZODB. This talk will go over the fundamentals and explore the tools that abstract it right away so that you can forget about it, and focus on development.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,627
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
22
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

upfront diet plone conf 2010

  1. 1. upfront diet for plone Roché Compaan
  2. 2. About me ● Owner of Upfront Systems ● Founded Upfront Systems in 1998 ● Zope developer for the past 10 years
  3. 3. Experience ● Largest Plone deployments to date: 5FM & MetroFM ● 5FM – 20 million page views per month – 70 thousand members – 6 million objects in ZODB ● Metro FM – 15 million page views per month – 20 thousand members – 2 million objects in ZODB ● Helping Connexions with performance ● http://www.markhorner.net/2010/04/29/connexions-authoring- performance/
  4. 4. Can Plone scale to 10 million documents?
  5. 5. ZODB Benchmarks ● Most common myth spread: “ZODB is a high read, (s)low write database” ● No published ZODB benchmarks ● Wrote some benchmarks that test ZODB read/write speed in 2008 ● collective.zodbbench
  6. 6. ZODB Write Speed 0 500 thousand 2,5 million 5 million 7,5 million 8,5 million 10 million 0 1000 2000 3000 4000 5000 6000 7000 8000 Postgres ZODB Number of records Inserts/second
  7. 7. ZODB Lookup Speed Number of Objects Lookup Time 100,000 0.003 ms 1,000,000 0.006 ms 10,000,000 2 ms
  8. 8. Benchmark Conclusions ● Thanks Jim, ZODB Rocks! ● Large cache size important for writes too, not only reads ● Need a solution for more than 10 million objects ● If the ZODB can handle 250 inserts/second at peak volume why does it take a full minute to add 250 ATDocuments?
  9. 9. A closer look at Plone ● Added 10,000 documents to Plone, while profiling with Dieter Maurer's ZopeProfiler ● Timed “apply_index” method to find indexes performing badly with and without experimental.catalogqueryplan ● Batch import of content with an without references. ● Used fsdump to inspect pickle contents.
  10. 10. Plone Benchmark Conclusions ● portal_catalog remains a bottleneck to scaling a Plone site. ● 1 catalog == many many conflicts ● reference_catalog is a major performance bottleneck ● Indexing 10,000 documents shows that 90% of the objects in Data.fs relate to indexing.
  11. 11. What's in a pickle? 2,51755 81,564 9,872 20,024 85 655 1,544 30,455 182,708 7798,088122 Pickle contents of a single transaction adding an ATDocument object size in bytes IIBTree Folder IOBucket OIBucket IIBucket IOBTree BTrees.Length ATDocument OOBTree IISet Archetypes.BaseUnit OOBucket IITreeSet
  12. 12. Think before you index! “Waking up objects is expensive” - this is an irrational fear. And adding another catalog in response to this fear requires therapy.
  13. 13. The portal_catalog problem ● No navigation, portal tabs, folder contents, review queue if your content type is not in the portal_catalog. ● Your content type is automatically subscribed to 20+ indexes in the portal_catalog. Bad for new developers with hopes of building an enterprise CMS. ● Unless you have 5+ years experience with Zope – it's not so easy to “opt out” ● portal_catalog should only be used for content searches – no functionality must depend on it.
  14. 14. What is upfront.diet? ● It's a healthier and happier approach to Plone development: index only when you really really need to. ● Set of packages that re-implement plone functionality without depending on the portal_catalog and reference_catalog. ● Includes replacements for: portal tabs, navigation portlet, folder contents, archetype references. ● Provides a catalog blacklist/whitelist to customise indexing per content type.
  15. 15. More about upfront.diet ● upfront.navportlet ● Separate catalog for navportlet ● upfront.simplereferencefield ● Store references directly on object using annotations ● still depends uid_catalog ● upfront.foldercontents ● Adapter for IFolderish ● Index allowedRolesAndUsers directly on folder – less conflicts ● upfront.catalogblacklist ● Generic Setup profile with a whitelist and blacklist of indexes per content type ● patches IndexableObjectWrapper, ugh!
  16. 16. upfront.diet todo list ● Packages: ● Portal tabs (done but not a separate package yet) ● Sitemap ● Review queue ● News and Events Smart Folders ● Site Search ● References without uid_catalog ● Comparative benchmarks with and without it
  17. 17. upfront.diet deployments ● Very few at the moment but looks promising ● Fullmarks.org.za (half a million objects, Data.fs < 1GB)
  18. 18. Contact Me ● Email: roche@upfrontsystems.co.za ● Blog: http://upfrontsystems.co.za/Members/roche ● facebook.com/rochecompaan ● twitter.com/rochecompaan
  19. 19. Questions?
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×