Migrations With Transmogrifier
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

Migrations With Transmogrifier

  • 3,339 views
Uploaded on

 

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
3,339
On Slideshare
2,509
From Embeds
830
Number of Embeds
11

Actions

Shares
Downloads
18
Comments
2
Likes
1

Embeds 830

http://www.garbas.si 555
http://garbas.si 204
http://www.slideshare.net 20
http://admin.garbas.si 19
http://manage.garbas.si 13
http://localhost 9
http://www.linkedin.com 4
http://127.0.0.1 3
http://131.253.14.66 1
http://jeremycapo 1
https://www.linkedin.com 1

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Migrations with Transmogrifier Rok Garbas, http://www.garbas.si , European Plone Symposium May 2010 - Sorrento
  • 2. What is transmogrifier? Rok Garbas, http://www.garbas.si , European Plone Symposium May 2010 - Sorrento original intent was to provide pluggable way to import content into plone Migrations with Transmogrifier help us construct pipeline , from ordered sections (blueprints), where each section proccesses data in pipe can be also used for other tasks that require data proccessing tools. nice example which shows power of transmogrifier is collective.funnelweb
  • 3. Simple Example of Config Rok Garbas, http://www.garbas.si , European Plone Symposium May 2010 - Sorrento [transmogrifier] pipeline = source transform 1 transform 2 create [source] blueprint = example. transmogrifier .source size = 5 [transform 1] blueprint = example.transmogrifier.transform from_field = text to_ field = text transform = rst condition = python:item['filename'].endswith('.rst') [transform 2] blueprint = example.transmogrifier.transform from_field = text to_ field = description transform = get_first_paragraph [create] blueprint = collective.transmogrifier.constructor Migrations with Transmogrifier
  • 4. Create your own blueprint /1 Rok Garbas, http://www.garbas.si , European Plone Symposium May 2010 - Sorrento class ExampleSource(object): classProvides(ISectionBlueprint) implements(ISection) def __init__(self, transmogrifier, name, options, previous): self.previous = previous self.name = name def __iter__(self): for item in self.previous: yield item for record in ReadJSONFile(): yield dict(_path=normalize(record['title']), _type=”Document”, title =record['title'], text=record['text']) Migrations with Transmogrifier
  • 5. Create your own blueprint /2 Rok Garbas, http://www.garbas.si , European Plone Symposium May 2010 - Sorrento from collective.transmogrifier.utils import Condition class ExampleTranform(object): classProvides(ISectionBlueprint) implements(ISection) def __init__(self, transmogrifier, name, options, previous): self.previous = previous self.from_ field = options['from_field'] self.to_ field = options['to_field'] self.tranform = options['tranform'] self.condition = Condition(condition, transmogrifier, name, options) def __iter__(self): for item in self.previous: if self.condition(item): if 'rst' == self.param: item[self.to_field] = TransformFromRSTToHTML(item[self.from_field ]) elif 'get_first_paragraph' == self.param: item[self.to_field] = GetFirstParagraph(item[self.to_field]) yield item Migrations with Transmogrifier
  • 6. Register custom Blueprint Rok Garbas, http://www.garbas.si , European Plone Symposium May 2010 - Sorrento <configure xmlns=&quot;http://namespaces.zope.org/zope&quot; xmlns:tg=&quot;http://namespaces.plone.org/transmogrifier&quot; i18n_domain=&quot;example.transmogrifier&quot;> <tg:registerConfig name=&quot;exampleconfig&quot; title=&quot;Example&quot; description=&quot;Example pipeline config&quot; configuration=&quot; example .cfg&quot; /> <utility component=&quot;.source.ExampleSource&quot; name=&quot;example.transmogrifier.source&quot; /> <utility component=&quot;.transform.ExampleTransform&quot; name=&quot;example.transmogrifier.transform&quot; /> </configure> Migrations with Transmogrifier
  • 7. Base Blueprints /1 Rok Garbas, http://www.garbas.si , European Plone Symposium May 2010 - Sorrento collective.transmogrifier.sections.constructor Migrations with Transmogrifier collective.transmogrifier.sections.savepoint contstructs plone content (_type, _path) commits a savepoint which as result free up memory collective.transmogrifier.sections.savepoint commits a savepoint which as result free up memory collective.transmogrifier.sections.csvsource reads csv file and puts lines of it in pipeline collective.transmogrifier.sections.splitter splits pipeline, which lets you proccess data flow separatly
  • 8. Base Blueprints /2 Rok Garbas, http://www.garbas.si , European Plone Symposium May 2010 - Sorrento collective.transmogrifier.sections.manipulator Migrations with Transmogrifier rename, copy, delete data in pipeline collective.transmogrifier.sections.condition Lets you discard data from pipeline collective.transmogrifier.sections.inserter Lets you create new fields for data (from other fields) collective.transmogrifier.sections.codec Changes encoding for data in pipeline collective.transmogrifier.sections.constructor Creates parent folders when item needs them
  • 9. Some Useful Bluprints Rok Garbas, http://www.garbas.si , European Plone Symposium May 2010 - Sorrento plone.app.transmogrifier Migrations with Transmogrifier collective.funellweb transmogrify.siteanalyser transmogrify.filesystem transmogrify.ploneremote transmogrify.webcrawler transmogrify.htmltesting transmogrify.htmlcontentextractor transmogrify.pathsorter quintagroup.transmogrifier collective.blueprint.dancing transmogrify.sqlalchemy collective.blueprint.translationlinker
  • 10. Plone 2.0 > Plone 4.0 Rok Garbas, http://www.garbas.si , European Plone Symposium May 2010 - Sorrento >100.000 content objects Migrations with Transmogrifier >40GB in size many custom products usual migration/upgrade failed
  • 11. Lessons Learned Rok Garbas, http://www.garbas.si , European Plone Symposium May 2010 - Sorrento make migration repeatable (eg: using GenericSetup upgrade steps) Migrations with Transmogrifier migrate parts of page separatly split migration into export and import part monitor and time migration don't complicate in making blueprints extra reusable
  • 12. Plone 2.0 > Plone 4.0 Rok Garbas, http://www.garbas.si , European Plone Symposium May 2010 - Sorrento Migrations with Transmogrifier <10GB of “Data.fs” (RelStorage) + 20GB of blobs less custom code, better re-usage of Plone's default features using less resources, noticably faster thanks to Plone4 happy client
  • 13. Whats missing? Wishlist!? Future! Rok Garbas, http://www.garbas.si , European Plone Symposium May 2010 - Sorrento Migrations with Transmogrifier Better support for monitoring what goes inside pipeline Web GUI for base blueprints Making it not dependable on plone
  • 14. Thank you Rok Garbas, http://www.garbas.si , European Plone Symposium May 2010 - Sorrento Migrations with Transmogrifier Rok Garbas, rok@garbas.si http://www.garbas.si http://www.twitter.com/garbas