Atlassian products contain lots of data, and often there isn't just one Jira system in use. Be it moving to or from the Cloud, or between instances - merging and migrating data can be hard. Dan Hardiker from Adaptavist will share the challenges they face and solutions they've found to common data portability issues. Learn some best practices, including their standardised Export-Transform-Load approach, and uncover many hidden gremlins you may trip over along the way. After this sessions you'll be ready to fearlessly move data from one Jira instance to another!

1. Data Portability
Tales from the Trenches
DAN HARDIKER | CTO | ADAPTAVIST | @DHARDIKER
BRITTANY WISPELL | CONSULTANT | ADAPTAVIST | @REDHOTJIRAPEPPS

5. Agenda
Tools & Support
Scenarios
Atlassian
Q&A

6. I have a
cunning plan
BALDRICK, BLACKADDER http://www.fansshare.com/celebrities/blackadder-goes-forth/

7. APPROACHES CHECKLISTS
✅ Task1
✅ Task2
✅ Task3
...

8. Business as
Usual
Infrastructure
Change
Atlassian
Cloud
Data
Shaping
Scenarios

9. Business as Usual
Scenarios

10. Copy & Clone
Driven by Operational Needs
Backup & Restore
Driven by Business Requirements
Scenarios
Business as Usual

11. Managing
Data
Business as Usual

12. XML
Backups
Business as Usual > Managing Data

13. Backup
Restoration
Gotchas
Contents
A zipped archive containing most of the data from
your Jira system.
Location
Found in the Jira home directory,
within the exports folder.
Default
Enabled by default, running nightly.
XML
Backups

15. XML
Backups
Backup
Restoration
Gotchas
Jira Admin UI
Restoration is started through the Jira user interface.
Resource Intensive
Intensive of use of Java memory, network & disk I/O.
Index Rebuilt
After the restoration, a full reindex will be triggered.

16. XML
Backups
Backup
Restoration
Gotchas
Missing in Action
• Attachments
• Bespoke Plugin Data
• Plugin Installation Files
Never quite the same
• Database record IDs are regenerated.
• Application keys are unchanged.
Not Production Safe
If Jira is updated during an XML backup, the resulting
backup can be inconsistent or even corrupt.

17. File System
& Database
Snapshots
Business as Usual > Managing Data

18. Backup
Restoration
Gotchas
Contents
Two separate snapshots taking the file system and
database state, at a synchronised point in time.
Knowledge & Tooling
Infrastructure and DBA knowledge required.
Process & tooling is technology specific.
No Maintenance Mode
Execute during an outage, or when minimal activity is
running through Jira.
File System
& Database
Snapshots

19. File System
& Database
Snapshots
Backup
Restoration
Gotchas
Technology Specific
Restore the entire snapshots of the file system &
database directly.
Outage Required
Jira must be offline for the restoration.
No Index Rebuilding
The system will now be identical to that backed up.

20. File System
& Database
Snapshots
Backup
Restoration
Gotchas
Identical in every way
• Base URL
• Application Links
• Inbound Email
Work together
Synchronising the snapshots can get tough.

21. ✅ Method
• XML
• Snapshot
✅ Storage
• Frequency & Retention
• Constrains your RPO
✅ Security (Access)
• Sensitive data
• Availability
✅ Security (Crypto)
• In Transit / At Rest / In Use
• Key Management
✅ Data Sovereignty
• Where is your data?
• DR / Backups / Support
✅ Compliance
• ISO 27001 / HIPAA
• Privacy Shield / DPA / GDPR
Business as Usual
Checklist

22. Business as
Usual
Infrastructure
Change
Atlassian
Cloud
Data
Shaping
Scenarios

23. Infrastructure Change
Scenarios

24. File System
Changing system architecture
Database
Moving between vendors / systems
Scenarios
Infrastructure Change

25. Transform
Filter, modify and translate
the data as needed.
ETL
Load
Hydrate the data into the
target system
Export
Extract the data from the
source system
Infrastructure Change

26. utf8mb4

27. JRASERVER-36135
MySQL 4-byte character support

28. Transform
Filter, modify and translate
the data as needed.
ETL
Load
Hydrate the data into the
target system
Export
Extract the data from the
source system
Infrastructure Change

29. ✅ Method
• XML for DB changes
• Snapshot for Lift & Shift
✅ Revisit Approach
• Backups & Resilience Processes
• Snapshot Consistency
✅ Ownership
• Access & Control
• DBA? Service Desks?
✅ Technology
• Drivers, Firewalls, Tooling
• Direct Integrations with FS & DB
✅ Conversion
• DB schema changes
• Indexes & Character Sets
✅ Dry Runs
• Multiple iterations required
• Outage sizing
Infrastructure Change
Checklist

30. KEEP
CALM
&
DRY RUN

31. ✅ Method
• XML for DB changes
• Snapshot for Lift & Shift
✅ Revisit Approach
• Backups & Resilience Processes
• Snapshot Consistency
✅ Ownership
• Access & Control
• DBA? Service Desks?
✅ Technology
• Drivers, Firewalls, Tooling
• Direct Integrations with FS & DB
✅ Conversion
• DB schema changes
• Indexes & Character Sets
✅ Dry Runs
• Multiple iterations required
• Outage sizing
✅ Immutability
• Prefer new & migrate
• How do we back out?
✅ Monitoring
• Benchmark latency & responsiveness
• Validate in all environments
Infrastructure Change

32. Business as
Usual
Infrastructure
Change
Atlassian
Cloud
Data
Shaping
Scenarios

33. Atlassian Cloud
Scenarios

34. Scenarios
Atlassian Cloud
XML

35. http://www.pngmart.com/image/tag/terminator
http://www.mcgamer.net/threads/gameover-us-clan.95893/

36. ✅ Method
• XML only
• No Infrastructural Access
✅ XML Backing Up
• Are more resources needed?
• Is an outage required?
✅ Duration
• How long does it take?
• Backup, Transfer, Restore
✅ Versions
• What versions are needed?
• What change is expected?
✅ Complexity
• How simple is your case?
• Is transformation needed?
✅ Atlassian ID
• What is your username schema?
• Does any mapping need to happen?
Atlassian Cloud
✅ Functionality
• Map the differences
• Include add-ons
✅ Licensing
• What are the commercial changes?
• Include add-ons and environments
Checklist

37. Business as
Usual
Infrastructure
Change
Atlassian
Cloud
Data
Shaping
Scenarios

38. Data Shaping
Scenarios

39. MergingSplitting
Data Shaping
Scenarios

40. Splitting
Data Shaping

41. Snapshot Backup & Restore
Splitting

42. Snapshot Backup & Restore
Duplicate the system from snapshot backups.
Splitting

43. Snapshot Backup & Restore
Duplicate the system from snapshot backups.
Remove unwanted data from either system.
Splitting

44. Snapshot Backup & Restore
Duplicate the system from snapshot backups.
Remove unwanted data from either system.
Constraints
Splitting

45. Snapshot Backup & Restore
Duplicate the system from snapshot backups.
Remove unwanted data from either system.
Constraints
Target system must not exist.
Splitting

46. Snapshot Backup & Restore
Duplicate the system from snapshot backups.
Remove unwanted data from either system.
Constraints
Target system must not exist.
Same technologies in both systems.
Splitting

47. Snapshot Backup & Restore
Duplicate the system from snapshot backups.
Remove unwanted data from either system.
Constraints
Target system must not exist.
Same technologies in both systems.
Use Cases
Splitting

48. Snapshot Backup & Restore
Duplicate the system from snapshot backups.
Remove unwanted data from either system.
Constraints
Target system must not exist.
Same technologies in both systems.
Use Cases
Where the source system cannot be impacted.
Splitting

49. Snapshot Backup & Restore
Duplicate the system from snapshot backups.
Remove unwanted data from either system.
Constraints
Target system must not exist.
Same technologies in both systems.
Use Cases
Where the source system cannot be impacted.
Where cloning the system is straight forward.
Splitting

50. Tooling Based Copy
Splitting

51. Tooling Based Copy
Setup a new system.
Splitting

52. Tooling Based Copy
Setup a new system. Copy global configuration.
Splitting

53. Tooling Based Copy
Setup a new system. Copy global configuration.
Use tooling to copy across project config & data.
Splitting

54. Tooling Based Copy
Setup a new system. Copy global configuration.
Use tooling to copy across project config & data.
Remove copied data from source system.
Splitting

55. Tooling Based Copy
Setup a new system. Copy global configuration.
Use tooling to copy across project config & data.
Remove copied data from source system.
Constraints
Splitting

56. Tooling Based Copy
Setup a new system. Copy global configuration.
Use tooling to copy across project config & data.
Remove copied data from source system.
Constraints
Target system global configuration must match.
Splitting

57. Tooling Based Copy
Setup a new system. Copy global configuration.
Use tooling to copy across project config & data.
Remove copied data from source system.
Constraints
Target system global configuration must match.
Appropriate tooling must be purchased.
Splitting

58. Tooling Based Copy
Setup a new system. Copy global configuration.
Use tooling to copy across project config & data.
Remove copied data from source system.
Constraints
Target system global configuration must match.
Appropriate tooling must be purchased.
Plugins data may be missed.
Splitting

59. Tooling Based Copy
Setup a new system. Copy global configuration.
Use tooling to copy across project config & data.
Remove copied data from source system.
Constraints
Target system global configuration must match.
Appropriate tooling must be purchased.
Plugins data may be missed.
Use Cases
Splitting

60. Tooling Based Copy
Setup a new system. Copy global configuration.
Use tooling to copy across project config & data.
Remove copied data from source system.
Constraints
Target system global configuration must match.
Appropriate tooling must be purchased.
Plugins data may be missed.
Use Cases
Where the source system must remain live.
Splitting

61. Tooling Based Copy
Setup a new system. Copy global configuration.
Use tooling to copy across project config & data.
Remove copied data from source system.
Constraints
Target system global configuration must match.
Appropriate tooling must be purchased.
Plugins data may be missed.
Use Cases
Where the source system must remain live.
Where a relatively small amount of data needs
to be split off.
Splitting

62. XML Backup & Restore
Splitting

63. XML Backup & Restore
Setup a new system, restoring the XML backup.
Splitting

64. XML Backup & Restore
Setup a new system, restoring the XML backup.
Install plugins.
Splitting

65. XML Backup & Restore
Setup a new system, restoring the XML backup.
Install plugins. Copy over the attachments.
Splitting

66. XML Backup & Restore
Setup a new system, restoring the XML backup.
Install plugins. Copy over the attachments.
Remove unwanted data from either system.
Splitting

67. XML Backup & Restore
Setup a new system, restoring the XML backup.
Install plugins. Copy over the attachments.
Remove unwanted data from either system.
Constraints
Splitting

68. XML Backup & Restore
Setup a new system, restoring the XML backup.
Install plugins. Copy over the attachments.
Remove unwanted data from either system.
Constraints
Source system must have an outage for backup.
Splitting

69. XML Backup & Restore
Setup a new system, restoring the XML backup.
Install plugins. Copy over the attachments.
Remove unwanted data from either system.
Constraints
Source system must have an outage for backup.
Typically consumes the most resources and
takes the longest.
Splitting

70. XML Backup & Restore
Setup a new system, restoring the XML backup.
Install plugins. Copy over the attachments.
Remove unwanted data from either system.
Constraints
Source system must have an outage for backup.
Typically consumes the most resources and
takes the longest.
Use Cases
Splitting

71. XML Backup & Restore
Setup a new system, restoring the XML backup.
Install plugins. Copy over the attachments.
Remove unwanted data from either system.
Constraints
Source system must have an outage for backup.
Typically consumes the most resources and
takes the longest.
Use Cases
Where transformation of the backup is needed.
Splitting

72. XML Backup & Restore
Setup a new system, restoring the XML backup.
Install plugins. Copy over the attachments.
Remove unwanted data from either system.
Constraints
Source system must have an outage for backup.
Typically consumes the most resources and
takes the longest.
Use Cases
Where transformation of the backup is needed.
When the new system architecture is different.
Splitting

73. XML Backup & Restore
Setup a new system, restoring the XML backup.
Install plugins. Copy over the attachments.
Remove unwanted data from either system.
Constraints
Source system must have an outage for backup.
Typically consumes the most resources and
takes the longest.
Use Cases
Where transformation of the backup is needed.
When the new system architecture is different.
Can restore specific projects. (Server/DC only)
Splitting

75. X X X

76. Merging
Data Shaping

77. Merging

78. Snapshot Backup & Restore
Not possible as target system already exists.
Merging

79. Tooling Based Copy
Merging

80. Tooling Based Copy
Align Jira & add-on versions in both systems.
Merging

81. Tooling Based Copy
Align Jira & add-on versions in both systems.
Sync target system global & project configuration.
Merging

82. Tooling Based Copy
Align Jira & add-on versions in both systems.
Sync target system global & project configuration.
Conflict Resolution
Resolve any configuration differences:
Merging

83. Tooling Based Copy
Align Jira & add-on versions in both systems.
Sync target system global & project configuration.
Conflict Resolution
Resolve any configuration differences:
Globals
Users & Groups, Resolutions, Statuses
Merging

84. Tooling Based Copy
Align Jira & add-on versions in both systems.
Sync target system global & project configuration.
Conflict Resolution
Resolve any configuration differences:
Globals
Users & Groups, Resolutions, Statuses
Project keys
Including historic
Merging

85. Tooling Based Copy
Align Jira & add-on versions in both systems.
Sync target system global & project configuration.
Conflict Resolution
Resolve any configuration differences:
Globals
Users & Groups, Resolutions, Statuses
Project keys
Including historic
Dashboards & Agile Rapid Boards & Filters
Compound key: item name + owning user
Merging

86. Strategies
Conflict Resolution
Resolve any configuration differences:
Globals
Users & Groups, Resolutions, Statuses
Project keys
Including historic
Dashboards & Agile Rapid Boards & Filters
Compound key: item name + owning user
Merging

87. Strategies
Technical
Conflict Resolution
Resolve any configuration differences:
Globals
Users & Groups, Resolutions, Statuses
Project keys
Including historic
Dashboards & Agile Rapid Boards & Filters
Compound key: item name + owning user
Merging

88. Strategies
Technical
Process
Conflict Resolution
Resolve any configuration differences:
Globals
Users & Groups, Resolutions, Statuses
Project keys
Including historic
Dashboards & Agile Rapid Boards & Filters
Compound key: item name + owning user
Merging

89. XML Backup & Restore
Merging

90. XML Backup & Restore
Transform the XML backup in-flight.
Merging

91. XML Backup & Restore
Transform the XML backup in-flight.
Constraints
Merging

92. XML Backup & Restore
Transform the XML backup in-flight.
Constraints
Sensitive to change.
Merging

93. XML Backup & Restore
Transform the XML backup in-flight.
Constraints
Sensitive to change.
Edge cases require testing.
Merging

94. XML Backup & Restore
Transform the XML backup in-flight.
Constraints
Sensitive to change.
Edge cases require testing.
Use Cases
Merging

95. XML Backup & Restore
Transform the XML backup in-flight.
Constraints
Sensitive to change.
Edge cases require testing.
Use Cases
Where complex migrations are needed.
Merging

96. XML Backup & Restore
Transform the XML backup in-flight.
Constraints
Sensitive to change.
Edge cases require testing.
Use Cases
Where complex migrations are needed.
Extreme changes in data structure and
architecture.
Merging

97. ✅ Versions
• See: Atlassian Cloud
• Add-ons too
✅ Add-ons
• Where is the data?
• How is it managed?
✅ Scope
• Whole or partial?
• New or merge in existing?
✅ Conflict Resolution
• Full assessment upfront
• Tooling assistance helpful
✅ Transformation
• What needs to change?
• Users, Keys, Globals
✅ Validation
• Missing Data (Script Fields)
• Smoke Test (config item count)
Data Shaping
Checklist

98. Data Shaping

99. Tools & Support
Scenarios
Atlassian
Q&A

100. Tools & Support
Scenarios
Atlassian
Q&A

101. http://wheelhouseworkshop.com/2017/04/06/its-dangerous-to-go-alone-take-this/

102. Botron Software
Configuration Manager
for Jira
Project Configurator
for Jira
Atlassian Recommends
Tools & Support
AdaptavistAwnaba

103. Atlassian Support
Standard or Premium.
Consider a TAM.
You are not alone!
Solution Partners
Professionals who have done
this many times before.
Atlassian Community
Documentation, Q&A,
support from your peers.
Tools & Support

104. Start off Simple
Tools & Support

105. Tools & Support
Scenarios
Atlassian
Q&A

106. Tools & Support
Scenarios
Atlassian
Q&A

107. • Merging and migrating are
really tough problems
• We're incredibly thankful for
our solution partners
• We recognize your pain and
are finally ready to help

108. Announcing a dedicated migration team!
• Migration (not just export/import)
• Multiple products (not just Jira)
• Rome wasn't built in a day
• Server->Cloud to start

109. Cloud migration, worthy of Atlassian
Inviting
Make migration feel
like the natural next
step in the customer
journey
Scalable
Migrate everything
from single projects
to massive instances
with confidence
Reliable
Tell customers what
is happening, before,
during, and after (and
make sure it does)
Flexible
Support multiple
migration modes and
approaches based on
customer needs

110. Tools & Support
Scenarios
Atlassian
Q&A

111. Tools & Support
Scenarios
Atlassian
Q&A
?

112. Thank you!
DAN HARDIKER | CTO | ADAPTAVIST | @DHARDIKER
BRITTANY WISPELL | CONSULTANT | ADAPTAVIST | @REDHOTJIRAPEPPS