Large-scale, cross-platform synchronization using embedded python

Uploaded on

A talk I gave at PyCon in 2006

A talk I gave at PyCon in 2006

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads


Total Views
On Slideshare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide
  • Hi, glad to be at pycon06 (post-lunch comment) For the next 20 minutes, I'm going to present our experience at wgen about putting python in the hands of the most demanding users: teachers. ... without them knowing. It's a short, non-technical talk and I think we have about 10 minutes at the end of the presentation for questions, and I'm of course available after that time-slot elapses. Let's get started...
  • I'll take a few moments to describe the business to better understand our requirements. The name is somewhat misleading as we had hoped that classrooms would be equipped with wireless by now. When you go to the bank to talk to your banker, it's usually you, him/her and the computer. That does not work well with students. Hence we want small and cheap devices that work well around the classroom. No wireless so you need a computer for connectivity (at least for the next few years) That computer has to work reliably too, this is where python kicks in.
  • That's the basic flow of information. Teachers produce and consume data in schools Which we manage (means code and data) Aggregate, analyze and build reports at various levels You can see where python fits; I'll focus in this talk on the client-side of the picture Now I'd like to explain a bit the title of this presentation
  • Large scale because of the following. Listed users are active Each user has a PDA which we remotely manage Users might share sync stations, which run python All that is choreographed by a bunch of python servers that drive an oracle instance. So it's python talking to python...
  • On a variety of hardware/software combination If you remember I said that teachers are demanding users: they have better to do than to click through error screens, repair broken programs or debug the usb connection to their device. So the software has to be relatively robust, on hardware that's more or less up-to-date.
  • Picture this, our database has to synchronize with tens of thousands of small databases. Each small DB lives on a device, a device that was never really designed to handle a network connection on its own. So we rely on the sync station to perform all that synchronization work. It must keep the pda and the central db in sync, and make sure that the code on the pda is always up-to-date. For that we adopted a dialect of syncml. It started as syncml and evolved with our needs. It turns out that 2-phase commits are big need.
  • Truth be told we started in c++ and that's not fun... Once we wrapped palm libraries in python, we were able to have a nice lean body of code that behaves pretty much consistently across platforms. Network code (proxy handling, etc.) and GUI are exactly the same (except for adapters) When your complex logic is the same across all platforms you can focus on testing and fixing it in one environment and simply work out kinks on the python-host interface. And you avoid your usual buffer-overflow bugs
  • Python-wrapper around low-level device-interaction libraries (easy). Python can create/update/delete data on the device so more complex parts such as Caching, device data interpretation done in python, not hindered by trivial details. Given the network bandwidth/latencies the network stack is also implemented in python and performance not an issue
  • Since the building blocks are sound and well-isolated: memory management xml parsing Network, etc. we can focus on building the necessary services to meet requirements. Example: auto-upgrades, logic packaged in .py tarballs, no cross-platform headaches.


  • 1. Large-scale, cross-platform synchronization using embedded python
      • PyCON 2006
      • Alexis L ê -Qu ôc
  • 2. Wireless Generation
    • Education software for K-12 teachers
      • Most essential: teacher/student interaction
      • Handheld devices + “sync station”
      • Robust, simple software
  • 3. Architecture
  • 4. Large-scale
    • X.10 6 students
    • Y users
      • Y Palm(tm) handhelds
      • Y “sync stations”
    • Oracle DB with python, lots of SQL and gigabytes of assessments
  • 5. Cross-platform
    • Various breeds of Windows
    • Mac OS 9 + X
    • Palm's sync model
  • 6. Synchronization
    • 1-to-n database synchronization
      • Relational-to-relational mapping
    • Code synchronization
    • SyncML dialect
      • Transaction-oriented
      • Xml-encoded
  • 7. Why embedded python
    • Cross-platform
    • Nice interface with host system
      • Single-threaded network code using (+ openssl)
      • Simple GUI code based on winpython or macpython
    • Easy to test
    • Safe: no buffer-overflows, etc.
  • 8. Design
  • 9. Cross-platform upgrades
    • Isolated building blocks a benefit
    • Remote upgrades by swapping zipped libs
  • 10. From n to n+2 platforms
    • From windows to OS X + linux
      • Write pilot-link adapter
        • much much better than Palm's original HotSync
      • Write GUI adapter
      • Use different package maker
      • ... and we're done
  • 11. Issues
    • Memory-management can be tricky
      • Used too many object allocations
    • Profiling tools a bit lacking
  • 12. Conclusion
    • High-level problems deserve a high-level language
    • Cross-platform coding is fun again...
  • 13.
      • Thank you