1. A Multi-submission
importer for EasyForm

2. Annette Lewis
Developer @ Six Feet Up, Inc.
annette@sixfeetup.com
2

3. The Setup
✖ Created several lengthy forms to register
attendees for an event
✖ Needed a way to mass import attendee
registrations
✖ Allow non Site Admins to prepare import
data
3

4. The Solution should:
✖ Work with most forms
✖ Provide a template to be filled out
✖ Allow Site Admins to preview data before
submitting import
✖ Execute form actions
4

5. Let's see this
in action!
The REsulting Solution
5

6. What I know about EasyForm
✖ EasyForm uses Dexterity
✖ Dexterity uses the z3c.form library to
build its forms, via the plone.z3cform
integration package
✖ Dexterity also relies on plone.autoform,
in particular its AutoExtensibleForm base
class
6

7. Generating the CSV
7

8. CSV output?
8

9. 9
<browser:page
name="download-form-csv"
for="collective.easyform.interfaces.IEasyForm"
class="addon.policy.browser.views.GetCSVTemplateView"
permission="cmf.ModifyPortalContent"
layer="addon.policy.interfaces.IAddonPolicyLayer"
/>
<h2>Get csv template</h2>
<p>Use this file as a template for submitting your information.</p>
<form method="post" tal:attributes="action string:@@download-form-csv">
<input tal:replace="structure context/@@authenticator/authenticator" />
<input type="submit" value="Download CSV template" />
</form>
<br/>
Browser view and Download Button

10. 10from collective.easyform.api import get_schema
from collective.easyform.api import getFieldsInOrder
from six import StringIO
import csv
from DateTime import DateTime
class GetCSVTemplateView(BrowserView):
def __call__(self, *args, **kwargs):
# download csv template for all form fields
form = self.context
response = self.request.RESPONSE
schema = get_schema(form)
fields = getFieldsInOrder(schema)
fieldnames = []
for f in fields:
fieldnames.append(f[0])
now = DateTime().ISO().replace(" ", "-").replace(":", "")
self.request.RESPONSE.setHeader(
"Content-type",
"text/comma-separated-values")
self.request.RESPONSE.setHeader(
"Content-Disposition",
'attachment; filename="{0}_{1}.csv"'.format(form.__name__, now),
)
csv_file = StringIO()
writer = csv.DictWriter(csv_file, fieldnames=fieldnames)
writer.writeheader()
return csv_file.getvalue()

11. Sample CSV output
11

12. Upload the Import
12

13. CSV output?
13

14. 14
<browser:page
name="preview-csv-import"
for="collective.easyform.interfaces.IEasyForm"
class="addon.policy.browser.views.PreviewCsvDataView"
permission="cmf.ModifyPortalContent"
layer="addon.policy.interfaces.IAddonPolicyLayer"
/>
<h2>Import csv</h2>
<p>Upload the csv template here.</p>
<form action="@@preview-csv-import" method="post" enctype="multipart/form-data">
<input tal:replace="structure context/@@authenticator/authenticator" />
<input type=file name="file_attachment"><br>
<input type=submit value="Import csv data">
</form>
</div>
Import Button and Browser View

15. Reuse & REcycle
Reuse as much existing code as possible and
only add (override) what I need to
15

16. 16
class CrudForm(AbstractCrudForm, form.Form):
template = viewpagetemplatefile.ViewPageTemplateFile('crud-master.pt')
description = u''
editform_factory = EditForm
addform_factory = AddForm
def update(self):
super(CrudForm, self).update()
addform = self.addform_factory(self, self.request)
editform = self.editform_factory(self, self.request)
addform.update()
editform.update()
self.subforms = [editform, addform]
from plone.z3cform.crud import crud
class SavedDataForm(crud.CrudForm):
template = ViewPageTemplateFile("saveddata_form.pt")
addform_factory = crud.NullForm
Down the rabbit hole
plone.z3cform/crud.py
collective.easyform/browser/actions.py

17. REview the Import
17

18. 18

19. 19
from collective.easyform.browser.actions import SavedDataForm
from Products.Five.browser.pagetemplatefile import ViewPageTemplateFile
class csvimportcrud(SavedDataForm):
""" Use the SavedData Adapter form to review and validate csv import data
"""
template = ViewPageTemplateFile("templates/review_import.pt")
editform_factory = ImporterEditForm
from plone.z3cform import layout
PreviewCsvDataView = layout.wrap_form(csvimportcrud)
Building the Import Preview
<div class="crud-form"
tal:repeat="form view/subforms"
tal:content="structure form/render"
>
</div>

20. 20
def get_items():
"""Subclasses must a list of all items to edit.
This list contains tuples of the form ``(id, item)``, where
the id is a unique identifiers to the items. The items must
be adaptable to the schema returned by ``update_schema`` and
``view_schema`` methods.
"""
class SavedDataForm(crud.CrudForm):
[...]
def get_items(self):
return [
(key, DataWrapper(key, value, self.context))
for key, value in self.field.getSavedFormInputItems()
]
Formatting my import Data
plone.z3cform/crud.py
collective.easyform/browser/actions.py

21. To the Code
21

22. 22class csvimportcrud(SavedDataForm):
[...]
def importCSV(self, *args, **kwargs):
"""Import CSV information """
form = self.context
request = self.request
attachment = request.form['file_attachment']
data = self.getCSVdata(attachment, form)
return data
def getCSVdata(self, attachment, form):
# get form fields
schema = get_schema(form)
fields = getFieldsInOrder(schema)
fieldnames = []
for f in fields:
fieldnames.append(f[0])
# open csv file
if hasattr(attachment, 'file'):
data = attachment.file.read()
data = data.decode('UTF-8')
file = data.splitlines()
reader = csv.DictReader(file)
rows = list(reader)
# # format to match SavedDataForm expects
formatted_rows = []
for row in rows:
unsorted_data = dict(row)
unsorted_data = self.cleanup_values(form, row)
formatted_rows.append(unsorted_data)
return [(count, dict(formatted_rows))
for count, formatted_rows in enumerate(formatted_rows, 1)]

23. 23
def migrate_saved_data(ploneformgen, easyform):
[...]
for key, value in zip(cols, row):
field = schema.get(key)
value = value.decode('utf8')
if IFromUnicode.providedBy(field):
value = field.fromUnicode(value)
elif IDatetime.providedBy(field) and value:
value = DateTime(value).asdatetime()
elif IDate.providedBy(field) and value:
value = DateTime(value).asdatetime().date()
elif ISet.providedBy(field):
try:
value = set(literal_eval(value))
except ValueError:
pass
elif INamedBlobFileField.providedBy(field):
value = None
data[key] = value
Clean-up values
Inspired by collective.easyform/migration/data.py - migrate_saved_data
collective.easyform/api.py

24. 24
def cleanup(value):
"""Accepts lists, tuples or comma/semicolon-separated strings
and returns a list of native strings.
"""
if isinstance(value, six.string_types):
value = safe_unicode(value).strip()
value = value.replace(u",", u"n").replace(u";", u"n")
value = [s for s in value.splitlines()]
if isinstance(value, (list, tuple)):
value = [safe_unicode(s).strip() for s in value]
if six.PY2:
# py2 expects a list of bytes
value = [s.encode("utf-8") for s in value if s]
return value
Clean-up values pt 2
Also take advantage of easyform's cleanup()
collective.easyform/api.py

25. 25
def get_items(self):
""" Get csv data and format """
data = []
if hasattr(self.request, 'file_attachment'):
if self.request.form.get('file_attachment'):
data = self.importCSV()
return data
elif hasattr(self.request, 'crud-edit.form.buttons.edit'):
# get data from submitted form
if self.request.form.get('data_export'):
data_str = self.request.data_export
# ast cannot handle sets
data = ast.literal_eval(data_str)
else:
data = []
url = "{0}/@@import-forms".format(
self.context.absolute_url()
)
IStatusMessage(self.request).add(
_(u'Error: Missing file attachment.'))
self.request.response.redirect(url)
return data
The new get_items

26. REview the Import
26
Pt 2 - Handling the Data

27. 27
from collective.easyform.browser.actions import SavedDataForm
from Products.Five.browser.pagetemplatefile import ViewPageTemplateFile
class csvimportcrud(SavedDataForm):
""" Use the SavedData Adapter form to review and validate csv import data
"""
template = ViewPageTemplateFile("templates/review_import.pt")
editform_factory = ImporterEditForm
class ImporterEditForm(crud.EditForm):
template = ViewPageTemplateFile('templates/importer-crud-table.pt')
Displaying the Import Data
<input name="data_export" type="hidden" value="" tal:attributes="value view/getData|nothing">

28. To the Code
28

29. SUbmit the Data
29

30. 30
@button.buttonAndHandler(
PMF(u"Submit"), name="submit", condition=lambda form: not form.thanksPage
)
def handleSubmit(self, action):
unsorted_data, errors = self.extractData()
if errors:
self.status = self.formErrorsMessage
return
unsorted_data = self.updateServerSideData(unsorted_data)
errors = self.processActions(unsorted_data)
if errors:
return self.setErrorsMessage(errors)
data = OrderedDict(
[x for x in getFieldsInOrder(self.schema) if x[0] in unsorted_data]
)
data.update(unsorted_data)
[...]
EasyFOrm handleSubmit()

31. 31
class ImporterEditForm(crud.EditForm):
[...]
@button.buttonAndHandler(_('Import form Data'),
name='edit',
condition=lambda form: form.context.update_schema)
def handle_edit(self, action):
success = _(u"Successfully updated")
partly_success = _(u"Some of your changes could not be applied.")
status = no_changes = _(u"No changes made.")
for subform in self.subforms:
data, errors = subform.extractData()
if errors:
[...]
imported_rows = 0
form = self.context.context
rows = self.subforms
total_rows = len(rows)
for x in rows:
row = data
try:
csvimportcrud(self.context,
self.request).submitForm(form, row)
imported_rows += 1
except IndexError:
continue
# self.status = status
IStatusMessage(self.request).add(
_(u'Imported {0} out of {1} row(s).').format(imported_rows,
total_rows))
My entry point

32. 32
from collective.easyform.browser.view import EasyFormForm as easyformview
class csvimportcrud(SavedDataForm):
""" Use the SavedData Adapter form to review and validate csv import data
"""
def submitForm(self, form, row):
""" Process Easyform actions """
unsorted_data = row
errors = easyformview.processActions(self.context, unsorted_data)
if errors:
return form.setErrorsMessage(errors)
Submit the Subforms

33. The beginning?
A client request that taught me a lot and has
a pretty cool result.
33

34. Thanks For coming!!
Any questions?
You can find me at:
annette@sixfeetup.com
34