In 2016, a group of social scientists at the University of California Berkeley received a large grant to develop tools for rigorous social science research, initially focused on collective identity formation. Jazkarta has been helping them develop Dallinger, a tool to automate experiments that use large numbers of subjects recruited on platforms like Mechanical Turk. They chose Jazkarta because of our web development and project management expertise, but also because of our familiarity with large, open source software projects - which is a goal for Dallinger. At this 2017 Plone Conference presentation, members of the Jazkarta team (David Glick, Alec Mitchell, Matthew Wilkes, and Sally Kleinfeldt) describe how we've put the lessons of Plone to work setting up this new open source project. We also describe how the technology stack (Python, Redis, Web Sockets, Heroku, AWS/Mechanical Turk/boto, Flask, PostgreSQL/SQLAlchemy, Gunicorn, Pytest, gevent) has been working for us.

1. An Open Source Platform
for Social Science
Research
Sally Kleinfeldt
Barcelona
2017

2. The Program

3. Next Generation Social
Sciences
(NGS2)
Funded by:

4. "The program aims to build and evaluate new
methods and tools to advance rigorous,
reproducible social science studies at scales
necessary to develop and validate causal
models of human social behaviors."

5. Benefits for
• Public health
• Economics
• National security

6. Initial Focus
Identify causal mechanisms of
collective identity formation

7. Dallinger

8. Grant Awarded To

9. Dallinger
• Crowdsourced experiments
• Abstracted into single function calls
• Can be inserted into higher-order
algorithms, for example to progressively
refine experiment

10. Fully Automated
• Recruits participants (Mechanical Turk)
• Obtains informed consent
• Arranges participants into a network
• Runs experiement (Heroku)
• Coordinates communication

11. Fully Automated
• Records the data they produce
• Pays participants
• Recruits new batches of participants
contingent on the structure of the
experiment
• Validates and manages the resulting data

12. How Does It Work?
• Experiments are modeled as directed
graphs
• Experiments are like Plone add-ons being
run by the Dallinger system

13. Games!
• All teams coalesced on using a public goods
game
• Dallinger is the only team using a real-time
multiplayer game: Grid Universe

14. Demo

15. Repository
• github.com/Dallinger/Dallinger

16. Jazkarta’s Contribution

17. Why Jazkarta?
• Expertise in Python, Flask, PostgreSQL,
SQLAlchemy,Amazon Mechanical Turk,
boto, tox, pytest, Redis, Selenium,
PhantomJS, JavaScript, HTML, CSS…
• Expertise in project management
• Expertise in a mature open source
community - Plone!

18. Our Process
• Discovery meeting fall 2017
• Developed user stories
• Estimated using planning poker
• Implementing in a series of iteration

19. Our Team
• Alec Mitchell
• David Glick
• Matthew Wilkes
• Carlos de la Guardia
• Jesse Snyder

20. Lessons Learned from
Plone

21. Lessons
• Don’t over-engineer plugin architectures
(like recruiters)
• Support live editing as much as possible
• Break backwards compatibility when
needed
• Remove references to old ways of doing
things

22. Lessons
• Ship lots of useful demos
• Be diligent about code reviews
• Make important approach decisions
together
• People involved in decisions should know
user needs intimately (we miss Joel Burton)

23. Plone vs. Dallinger
Tech Stack

24. Tech Stack
• Web based, but Flask instead of Zope
• PostgreSQL instead of ZODB
• Real time websockets
• Built in deployment command

25. Writing Experiments

26. Helping Dallinger Users
• Documentation
• Slack channel
• Cookie cutter template
• Extendable base templates
• Javascript library
• Commands for local debugging

27. Ensuring Code Quality

28. Code Quality
• Automated lint checks
• Continuous integration with minimum
code coverage requirement
• Code review
• Regression testing of an experiment (GU)
against changes to core Dallinger

29. Fun Technical
Challenges

30. Fun Challenges
• Scaling selenium-based bots
• Getting access to track interactions with
3rd-party sites (Chrome extension)
• Testing multiple participants in parallel
without sharing cookies

31. Questions?