Transmogrifier
Lennart Regebro
Plone Conference 2009, Budapest

collective.transmogrifier
Created by
Martijn Pieters, Jarn
in 2008
Released in
version 1.0
August 2009

collective.transmogrifier
Import/migration
framework
Both from Plone 2
and from other
websites or data

Super-simple architecture
You set up a
pipeline of sections
Each section send
items to the next
section.
Sections do
something with the
items (or not)

What are you talking about?

A simple example
1. A section that reads a CSV file with a list of
events and creates items based on that data
2. A section that will generate some plone-specific
data for each item, like path and type.
3. A section that creates the Plone event
4. A section that updates the Archetypes fields
5. A section that publishes the event

The CSV file
title,startDate,endDate
Plone Conference,2009-10-28 09:00,2009-10-30 18:00
Christmas,2009-12-24 00:00,2009-12-26:23:59

Setting it up
1. Define a pipeline in a .cfg file
2. Register it as a name in ZCML
3. Call Transmogrifier from Python

The configuration file
[transmogrifier]
pipeline =
section1
section2
[section1]
blueprint = name.of.the.blueprint
size = 5
[section2]
blueprint = another.blueprint

ZCML
<configure
xmlns:transmogrifier=
"http://namespaces.plone.org/transmogrifier">
<transmogrifier:registerConfig
name="Zap event import"
title="Import of events"
description=""
configuration="transmogrifier.cfg"
/>
</configure>

setuphandlers.py
from collective.transmogrifier.transmogrifier import
Transmogrifier
def setupVarious(context):
if context.readDataFile('zap_various.txt') is None:
return
transmogrifier = Transmogrifier(context.getSite())
transmogrifier('Zap event import')

What is a generator
def my_generator():
i=1
while i < 1000:
yield i
i=i*2
for x in my_generator():
print x

Never ending story
def my_generator():
i=1
while True:
yield i
i=i*2
for x in my_generator():
print x

EventSource
class EventSource(object):
def __iter__(self):
for event is csv.DictReader(open(self.options['csvfile']))
yield event
for item in self.previous:
yield item

EventSource
class EventSource(object):
def __iter__(self):
for event is csv.DictReader(open(self.options['csvfile']))
yield event
for item in self.previous:
yield item

EventSource
class EventSource(object):
def __iter__(self):
for event is csv.DictReader(open(self.options['csvfile']))
yield event
for item in self.previous:
yield item

The item
{
'title': 'Plone Conference',
'startDate': '2009-10-27 09:00',
'endDate': '2009-10-30 18:00',
}

EventUpdater
class EventUpdater(object):
def __iter__(self):
for item in self.previous:
id = item['title'].lower().replace(' ', '-')
item['_path'] = '/events/'+id
item['_type'] = 'Event'
item['_transitions'] = ('publish',)
yield item

The item
{
'_type': 'Event',
'_path': '/events/plone-conference',
'_transitions': ('publish,),
'title': 'Plone Conference',
'startDate': '2009-10-27 09:00',
'endDate': '2009-10-30 18:00',
}

Registering the section
<utility
component=".eventsource.EventSource"
name="zap.eventsource"
/>
<utility
component=".eventsource.EventUpdater"
name="zap.eventupdater"
/>

The configuration file
[transmogrifier]
pipeline =
eventsource
eventupdater
[eventsource]
blueprint = zap.eventsource
csvfile = /path/to/events.csv
[eventupdater]
blueprint = zap.eventupdater

The configuration file cont.
[constructor]
blueprint = collective.transmogrifier.sections.constructor
[schemaupdater]
blueprint = plone.app.transmogrifier.atschemaupdater
[workflow]
blueprint = plone.app.transmogrifier.workflowupdater

The configuration file cont.
[transmogrifier]
pipeline =
eventsource
eventupdater
constructor
schemaupdater
workflow

Sections
Create items
Modify items
Drop items
Split the pipeline in
two
Construct content
Modify content

Migrating from another CMS
Similar to the simple example
but using SQL instead of CSV

Migration from Plone 2
1. Write an export script
2. Write an import section

Migrate from Plone 2
importsource
constructor
criterionadder
schemaupdater
browserdefault
workflow
savepoint

Migrating from HTML
1. Get HTML from web
2. Extract fields like title, publish date, etc.
3. Make all urls either relative or into UID's

From small to huge
Largest migration:
28 sections
527 lines of configuration
16 custom sections

Caveat Emptor
Don't migrate from HTML on disk

Caveat Emptor
Newly created objects are not indexed.

Caveat Emptor
Debugging is backwards

Lennart Regebro
regebro@gmail.com
IRC nick: regebro