1) Managing large scale OpenMRS implementations with many sites presents challenges around maintaining consistency between sites and enabling simple upgrades and maintenance. 2) One approach is to develop an OpenMRS "distribution" that defines a consistent set of components including OpenMRS version, modules, and metadata for all sites. 3) Key aspects of developing a distribution include using a "distribution module" to define required component versions, continuous integration testing, and managing metadata installation and consistency programmatically rather than relying on users.

1. Building an OpenMRS
Distribution
Developer lessons from KenyaEMR

2. Distribution?
• Thinking in terms of “distributions” appears a
recent development in OpenMRS
• Nationwide implementations of OpenMRS are
becoming more common (e.g. Rwanda)
• Managing 100 sites is a
very different challenge
to managing 10 sites

3. Challenges of scale
• Need for consistency across site installations
• Need for simple or automated upgrades and
maintenance
• Need for scalable user support

4. Site inconsistency
• What we want to avoid:
Site
OpenMRS Concepts
Module 1
Module 2
Clinic 1
1.9.3
20140714 2.5.3
0.3
Clinic 2
1.9.3
20131215 2.4.1
1.0-BETA
Clinic 3
1.9.7
20131215 2.4
…
0.3.1
…
• Endless different environments, most of which
won’t have been tested
• Becomes impossible to manually track

5. Site inconsistency
• More chance of site-specific bugs
– Development team might not have tested a site’s
particular environment
• Complex site-specific upgrade processes
– Single components upgraded individually
• Confusion for users trained in different
environments
– National implementations often rely on
centralised training events

6. Site consistency
• A distribution should define a consistent
environment
• We can break that down into components:
– A version of OpenMRS core
– A set of modules with specified versions
– A set of metadata objects
• Might be versioned SQL dumps or MDS packages
• Might include a separately versioned concept
dictionary

7. Distribution example
• An implementation’s own custom modules
will only be a part of the larger distribution:
OpenMRS
MyModule
Reporting
MyDistro
Idgen
HtmlFormEntry
CIEL

8. Distribution versioning
• Specific versioned releases of all the components make
up a single versioned release of the distribution, e.g.
OpenMRS 1.9.3
OpenMRS 1.9.7
MyModule 1.0
MyModule 1.1
Reporting 0.8
Reporting 0.8.1
MyDistro 1.0
MyDistro 2.0
Idgen 2.6
Idgen 2.6
HtmlFormEntry 2.5
HtmlFormEntry 2.5.1
CIEL 20120931
CIEL 20140107

9. Distribution versioning
• At every stage of the development lifecycle,
we should be working with a version of the
complete distribution, e.g.
Development with an
“in-progress” version
mydistro-2.0-SNAPSHOT
mydistro-2.0-RC1
Testing with release
candidate versions
mydistro-2.0-RC2
Installations and upgrades
with released versions
mydistro-2.0

10. Distribution modules
• Rather than manage a distribution as a
separate project, it can be easier to tie it to a
“distribution module”
• Its version is the distribution version
• Defines the required versions of all other
components
• Depends on all of the other modules

11. Distribution modules
• By requiring all of
the other modules
the, distribution
module ensures
that the
distribution is
always run as a
whole
<require_modules>
<require_module version="0.8.1">
org.openmrs.module.reporting
</require_module>
<require_module version="2.6">
org.openmrs.module.idgen
</require_module>
<require_module version="2.5.1">
org.openmrs.module.htmlformentry
</require_module>
</require_modules>

12. Continuous integration
• Don’t want to develop different components
in insolation and only realise integration
problems during testing
• Developers should work with the latest
version of the distribution
• CI server should be used to
keep a testing server up to
date with changes to any component

13. Buildable distributions
mydistro-2.0-SNAPSHOT
mydistro-2.0-distro.zip
• Need to make it easy for developers (and CI
servers) to deploy a particular version of a
distribution
• Useful to have a zip archive of the different
component modules
mymodule-1.1.omod
reporting-0.8.1.omod
idgen-2.6.omod
htmlformentry-2.5.1.omod

14. Buildable distributions
• Maven provides a convenient mechanism to
produce an archive of a project with its
dependencies – called an assembly
• Thus our distribution module can have two
build outputs:
– A regular omod
– A distribution zip archive
For an example of how to implement this, see: https://github.com/ITECH/openmrs-module-kenyaemr/blob/master/distro

15. Metadata consistency
• The idea of site consistency should apply also
to metadata
• For example:
– The distribution does patient registration
– This saves encounters of type “Registration”
– The distribution can’t function if that encounter
type doesn’t exist
– Can we guarantee that the encounter type means
the same thing in different installations?

16. User managed metadata
• Non-distribution modules often expect the
user to manage metadata, e.g.
– Every time some code tries to access the
“Registration” encounter type, check for null
– If it doesn’t exist, show the user an error message
to tell them to create it
– Tell user to set the
mydistro.registrationEncounterType global
property to reference the new object

17. Distribution managed metadata
• If we want to be sure that metadata is the
same across all sites, we manage it via code
rather than users
• Distributions should install required metadata
automatically
• Distribution code should assume that the
metadata exists
– If it doesn’t, it is a developer problem rather than
a user problem

18. Fail fast assumptions
• If we assume that metadata always exists, we
should fail-fast if that assumption turns out to
be incorrect, e.g.
EncounterType ret = Context.getEncounterService().getEncounterTypeByUuid(uuid);
if (ret == null) {
throw new IllegalArgumentException("No such encounter type with uuid " + uuid);
}
• Helps developers find problems right away
• Easier than tracking down source of a NPE
The Metadata Deploy module provides fail-fast fetch methods for most
metadata classes: https://github.com/I-TECH/openmrs-module-metadatadeploy

19. Metadata identity
• Database ids are not reliable for identifying
the same metadata in different installations
• Anytime distribution code references
metadata it should use one of:
– UUID: all OpenMRS classes have these and they
can be kept consistent across installations
– Reference terms: these are globally consistent
and unique identifiers for concepts (and soon also
drugs)

20. Metadata installation
• One approach is to bundle metadata packages
with the distribution and install these on
startup
• Weaknesses of this approach:
– Metadata is not easily readable or editable
– Packages typically have to be managed on an
external server, exported and embedded into the
code
– Package installation is slow so usually not
appropriate to use the same packages in unit tests

21. Metadata deploy
• Module was developed to address these
issues
– Allows metadata to be defined in code
– Metadata is easy to read and edit
– Fast installation suitable for unit tests
For more information about the Metadata Deploy module go to
https://wiki.openmrs.org/display/docs/Metadata+Deploy+Module

22. Metadata deploy
• Metadata that’s too lengthy to be described in
code can be loaded from CSV files etc
– Still more readable than zip archives
• Support for synchronization of large sets
– Used to synchronize OpenMRS locations with all
9500 facilities in the Kenya Master Facility List:
• Clean database synchronization: 1min
• Subsequent synchronization: 3-4secs
• Previous MDS package load: 20-25mins

23. Concepts?
• So far SQL dumps have been best for these:
– We don’t manage/edit them directly (we use CIEL)
– Database dump is only quick way to install
50,000+ concepts
• Groovy script used used to generate XML
dataset file of just those concepts used by
KenyaEMR
– Used for unit tests that need access to same
concepts as production environment

24. Example metadata bundle
@Component
@Requires({ BaseMetadata.class })
public class MyMetadata extends AbstractMetadataBundle {
This “bundle” installs
an encounter type
and two forms
public static final class _EncounterType {
public static final String ENCOUNTER_TYPE1 = "d3e3d723-7458-4b4e-8998-408e8a551a84";
}
public static final class _Form {
public static final String FORM1 = "4b296dd0-f6be-4007-9eb8-d0fd4e94fb3a";
public static final String FORM2 = "89994550-9939-40f3-afa6-173bce445c79";
}
@Override
public void install() {
install(encounterType("Encounter Type #1", "Something...", _EncounterType.ENCOUNTER_TYPE1));
install(form("Form #1", null, _EncounterType.ENCOUNTER_TYPE1, "1", _Form.FORM1));
install(form("Form #2", null, _EncounterType.ENCOUNTER_TYPE1, "1", _Form.FORM2));
// A form that should be retired if it exists
uninstall(possible(Form.class, "73d34479-2f9e-4de3-a5e6-1f79a17459bb"), "Because...");
}
}
Also retires a form
that’s no longer needed

25. Enforcing consistency
• Current OpenMRS UI wasn’t made for this
idea of a distribution
• Ideally we want to prevent even the super
user account from doing things like:
– Stopping modules
– Deleting or modifying metadata
• KenyaEMR overrides the regular
UI and provides a custom UI
without this functionality

26. Installations and upgrades
• OpenMRS itself is usually only one part of a
functioning EMR installation
• Other parts might be:
– The database client and server
– The JVM
– Tomcat or another Java web app server
– Database backup scripts run by cron jobs
– Help and training materials

27. Installations and upgrades
• For small or single site implementations,
developers often do these
• Not feasible for large implementations:
– Developers can’t physically visit every site
– Sites often have connectivity issues
– Upgrades need to be performed by less technical
users

28. Installation via virtual machine
• Working installations require correct
configuration of all those parts
• Easier to configure once at the office than
100s of times at each site
• Can build virtual machine
images and clone for all sites

29. Installation via virtual machine
• Ongoing experiments with different ways of
retaining patient data during upgrade
– Moving data from old to new VM via SQL dump
– Separate VMs for database (not replaced) and
web app (replaced)
• Other experiments using bittorrent to
download VM images to sites
– Works even when connectivity is very poor

30. Maintenance
• Things inevitably go wrong, e.g.
– Sites could end up with invalid data due to a
software bug
– Might need to convert data as software changes
• Modules can provide liquibase files to make
one time changes
– Not always easy or possible to do something with
just SQL

31. Automated fixes
• Requiring user intervention should be a last
resort
• KenyaEMR provides its own automated way of
making fixes called chores:
– Java classes which perform a one-time job
– Run at the end of KenyaEMR startup so have
access to all metadata and services
See KenyaEMR source for examples of chore classes: https://github.com/ITECH/openmrs-modulekenyaemr/tree/master/api/src/main/java/org/openmrs/module/kenyaemr/chor
e

32. User support
• Need clear processes for issue tracking:
– More technical site staff can create support tickets
directly in ticketing system
– Less technical site staff can use email or phone
and then support staff will create ticket
• Help materials need to be
easily accessible to site users

33. User support
• KenyaEMR integrates
with external help site
• Users can lookup help
documents and videos
• Can provide context
sensitive help within
different apps
Help site code available at
https://github.com/I-TECH/helpsite

34. Bahati nzuri!