Improving Rollback in Linux via DSL approach & distributing

Energia Open Source Improving Rollback in Linux via DSL approach & distributing Sunday, 6th February 2011 Presented by: John Thomson: John.Thomson@caixamagica.pt Researcher Paulo Trezentos: Paulo.Trezentos@caixamagica.pt http://twitter.com/PauloTrezentos R&D Director

Energia Open Source Overview  An overview of what Caixa Mágica does. – MANCOOSI project: European 7th Framework Programme  Roll-back – Definition – Types of roll-back – As part of a bigger system  DSL – Definition – Example  Approach  Implementation  Distributing data (kad/p2p)  Conclusion06.02.2011 Improving Rollback in Linux via DSL approach & distributing 2

Energia Open Source A brief summary of Caixa Mágica Software and an overview of what we do:  Caixa Mágica is a FOSS software and solutions provider, based in Lisbon, Portugal  Linux- Caixa Mágica 15 is the main Linux Distribution available in Portugal  Caixa Mágica work with national companies, Government and with European partners to promote Open Source in all aspects of work  Over 800,000 Linux CM systems installed (single & dual boot)  Interesting development → CM16 moving to Debian base06.02.2011 Improving Rollback in Linux via DSL approach & distributing 3

Energia Open Source European based research projects  Caixa Mágica works in many multi-national European based research projects. After EDOS (EU FP6 STREP) based project it is now actively working on MANCOOSI-FP7 Project  Also working on ULOOP and Timbus projects  Work with a multitude of top-tier Universities and research institutions06.02.2011 Improving Rollback in Linux via DSL approach & distributing 4

Energia Open Source About MANCOOSI  MANCOOSI - Managing the complexity of open- source software. www.mancoosi.org Solving package management issues that have been identified through EDOS project Other talks  Talks yesterday by Ralf and Jaap about solvers + tools  André Guerreiro presents Gumby in room H.1302 at 15:30 – Visualise package meta-data  Paulo Trezentos presents APT-PBO in room H.1302 at 16:00 – A better class of solver  Aim is to provide better tools for package management06.02.2011 Improving Rollback in Linux via DSL approach & distributing 5

Energia Open Source Roll-back as simple as traversing time?06.02.2011 Improving Rollback in Linux via DSL approach & distributing 6

Energia Open Source What does roll-back mean, really?  In terms of package configuration, “roll-back is the process of inverting the changes to the system made by package upgrades to get back to a particular system state.”  Many other mechanisms out there that work on using file system snapshots/saving the state, (next slide) S1 S2 PkgFoo v 1.00 Roll-back, PkgFoo v 2.00 Time: 10.00pm possible? Time: 10.20pm Installation Timeline  The mechanism is one part of Transactionally Protected Package Management as spoken about by Jeff Johnson last year at FOSDEM 2010.06.02.2011 Improving Rollback in Linux via DSL approach & distributing 7

Energia Open Source Different types of roll-back  Conary, a 2nd Generation Package Manager, aims to improve current meta-installers. Used in Linux distributions such as Foresight Linux.  Augeas, is a configuration management tool that makes manipulating config files much easier  ZFS, used by Nexenta is an example of a file-system, snapshot mechanism to snapshot several system states.  NixOS, a revolutionary system that re-thinks how files and resources are used to make them purely functional  Other mechanisms e.g. etckeeper being developed by Fedora also try and capture configuration files into a VCS. BTRFS – another snapshot based mechanism.06.02.2011 Improving Rollback in Linux via DSL approach & distributing 8

Energia Open Source Drawbacks of the various approaches  File-system Snapshot based mechanisms – Capture state of the whole system → down to individual files – Agnostic of packages – Granularity is not well matched for packages  Package Based – Same as installing an older version – Maintain configurations and settings  System Call based – Syscall trapping - records changes to the file-system – Capturing files alone is not always enough  Aim is to combine the complimentary parts of these systems to cover a wide range of situations and events06.02.2011 Improving Rollback in Linux via DSL approach & distributing 9

Energia Open Source Difficulties of Roll-back  Package maintainers and developers tend to think in the forward direction of upgrading packages  Working in the reverse direction is a relatively unheard of concept. Think of a toy slinky  Downgrade is counter-intuitive  Already have been done if it was necessary?  Rolling-back changes is only needed when a package fails to work on the system  There may be cases where roll-back is impossible using the techniques that we have investigated, or possibly at all06.02.2011 Improving Rollback in Linux via DSL approach & distributing 10

Energia Open Source Roll-back is one part of a bigger picture  Ability to undo package upgrade/install is just one benefit of having a system that can capture the package configuration state and maintain a deterministic, state transition model of the system.  By examining the current maintainer scripts and templates provided by deb/rpm-helper, we defined a language that assists with problems not addressed now by meta-installers.  Transactionaly Protected Package Management (TPPM) is what we are aiming to move towards. TPPM roll-back roll-back06.02.2011 Improving Rollback in Linux via DSL approach & distributing 11

Energia Open Source Advantages and benefits of Roll-back/TPPM  By building a framework in which R/B + TPPM is possible we can capture the functional elements of maintainer scripts Error Detection  Simulate/test the results of an upgrade on a model of system  Detect errors a-priori or potential slow errors after upgrades  Detect/(correct) sequencing orders in package installations Roll-back  Framework → drive roll-back. Now provide more information to a roll-back utility than available to current meta-installers  Identify which packages can/cannot perform roll-back  Allow individual packages to change versions and show the dependencies for those changes06.02.2011 Improving Rollback in Linux via DSL approach & distributing 12

Energia Open Source Domain Specific Language (DSL) -package upgrades  Our DSL is a language used to abstract from the system and model to solve particular, identified problems  In our case, the DSL is focused on analysing package maintainer scripts  Designed not to be a Turing Complete Language like BASH but rather something which can resolve problems  It is an evolutionary language designed to capture the details of the vast majority of common maintainer scripts  Increase the coverage of DSL by examining new scripts Many thanks to University of LAquila for their creation of the language and for their support in using it06.02.2011 Improving Rollback in Linux via DSL approach & distributing 13

Energia Open Source System Integration  For roll-back we will have a log of Apt the DSL commands executed in- sequence model_simulator (( )) model_simulator  Perform roll-back → run inverse statements associated with those DSL_rollback_pre (( )) DSL_rollback_pre commands in reverse order  Have the simulator to pre-check if run_transaction () run_transaction () the package configuration can be rolled-back; whether it will leave it DSL_rollback_post (( )) in an erroneus state DSL_rollback_post  Performing LIFO style roll-back we run post commands before we run the pre statements06.02.2011 Improving Rollback in Linux via DSL approach & distributing 14

Energia Open Source Domain Specific Language (DSL) Example  The DSL has been created to assist with some of the problems discovered and analysed by prior research Using cups.spec %post example from CUPS-1.4.2 %post  dslstart postinst_init(cups) /sbin/chkconfig add cups Matched DSL Pair /sbin/chkconfig cups on  dslstop postinst_init(cups) # Restart cupsd if we are upgrading...  dslstart post_init_restart(cups) if test $1 gt 1; then /sbin/service cups stop Matched DSL Pair /sbin/service cups start fi  dslstop post_init_restart(cups)06.02.2011 Improving Rollback in Linux via DSL approach & distributing 15

Energia Open Source Example continued, Log rbHist dslID TID parentID DSL_CMD bhINVERSE 1 1 1 start postinst_init(cups) TRUE 2 1 1 stop postinst_init(cups) TRUE 3 1 1 start post_init_restart(cups) TRUE 4 1 1 stop post_init_restart(cups) TRUE pkgHist id parent op pkgName pkgVer1 pkgVer2 dateTime 1 1 inst cups 0 1.4.2 2010-01-3006.02.2011 Improving Rollback in Linux via DSL approach & distributing 16

Energia Open Source Transactions- what happens if a maintainer script fails?  If a maintainer script fails in the middle of one of the operations will have a log like this dslID TID parentID DSL_CMD bhINVERSE 1 1 1 start postinst_init(cups) TRUE 2 1 1 stop postinst_init(cups) TRUE 3 1 1 start post_init_restart(cups) TRUE  Transaction has quite obviously failed. No matching end for a DSL command reached. Odd number of elements etc.  Perform a roll-back for all matching sub-transaction ID elements, but in the reverse order with certain constraints.  If a set of script elements cannot perform roll-back in the middle of operating, then dont create a dsl tag.06.02.2011 Improving Rollback in Linux via DSL approach & distributing 17

Energia Open Source Performing the roll-back  Identify the sub-transaction/package upgrade to roll-back.  Check that all commands in the database have an inverse for the particular transaction.  Run them in reverse-operation order.  dslstart postinst_init(cups) [1]  dslstop postinst_init(cups) [2]  dslstart post_init_restart(cups) [3] //Restart not needed postrm becomes  postrm_init_restart(cups) [3] //Remove nonneeded element  prerm_init_stop(cups) [4] //From lookup table.  prerm_init(cups) [1+2]  In other cases the order might be [3,2,1]. Identified by tags.06.02.2011 Improving Rollback in Linux via DSL approach & distributing 18

Energia Open Source Architecture  Modified Package as input → DSL extracted → Log storage and simulator → Mechanism for executing roll-backs → Maintenance of scripts etc. User I/P Inject DSL dslstart postinst_init(cups) dslend postinst_init(cups) Logs Roll-back SQLite DB .spec file Apt-rpm %post /sbin/chkconfig --add cups Modified System State /sbin/chkconfig cups on WP2 ... Simulator06.02.2011 Improving Rollback in Linux via DSL approach & distributing 19

Energia Open Source Overall Approach Analyse maintainer scripts in standard systems Identify common themes in maintainer scripts Create first version of the DSL Modify maintainer scripts to contain DSL Log DSL into SQLite DB Create roll-back mechanism Logged Data + Stored Info → Perform Roll-back06.02.2011 Improving Rollback in Linux via DSL approach & distributing 20

Energia Open Source A state/time perspective of roll-back apt-get [Maintainer Scripts] T0 install cups In terms of DSL apt-get DSL cups pkg rollback apt-rpm DSL stmts Log cups 1.2 apt-get T0+n rollback TID-028 [Maintainer Scripts]-1 apt-get rollback roll-back 2010-02-0606.02.2011 Improving Rollback in Linux via DSL approach & distributing 21

Energia Open Source Distributing Rollback-Information  Rollback data can be distributed – Transceive data onto Kademlia network. – Distribute rpms/rollback data/packages state to network – Private data submission to network? Opt-in mechanism Kad/P2P Network Logs Config cups pkg SQLite DB data Normal upgrades Initial System State Modified System State06.02.2011 Improving Rollback in Linux via DSL approach & distributing 22

Energia Open Source Todo list:  Completed: – Single package rollback with DSL – Supporting framework e.g. SQLite3 and script injectors.  Now in the process of: – Integrating Simulator/Failure Detector into package upgrade – Generating tool suite for automatic detection/generation of DSL commands for a given set of packages and their scripts – Creating web interface for changing the auto-created DSL pkgs – Completing rollback mechanism for multiple packages txns. – Testing – Documentation – Release + revisions – Integration into build systems06.02.2011 Improving Rollback in Linux via DSL approach & distributing 23

Energia Open Source Conclusion  The need for Roll-back of packages is growing  Not a trivial problem  Need to consider the problems of complex upgrade paths  Using the DSL we aim to remove a lot of the failure cases in package upgrades.06.02.2011 Improving Rollback in Linux via DSL approach & distributing 24

Energia Open Source Questions?  Thank you for listening. I hope that it was interesting and that you have some questions to ask.06.02.2011 Improving Rollback in Linux via DSL approach & distributing 25

http://pinguinsmagicos.blogs.sapo.pt	10
http://localhost	7
http://push.test.web	1
http://www.takeoverapp.com	1
http://spondulix9.ignisi.com	1
http://www.linkedin.com	1

Improving Rollback in Linux via DSL approach & distributing

by johngt

Accessibility

Categories

Upload Details

Usage Rights

6 Embeds 21

Statistics

Improving Rollback in Linux via DSL approach & distributing Presentation Transcript