Last year, our documentation team’s cloud-based authoring tool collapsed. The authoring tool is our team’s core tool. It manages our version control, source control, collaboration, review status, and publishing. This tool collapsing was one of our worst-case-scenario situations. However, we had prepared for this possibility because of our pre-purchase risk assessment, and our mitigation allowed us to get our release out on time, within two weeks of the tool’s collapse. In this talk, I walk through the warning signs we caught and the ones we missed as well as the mitigation strategies we had implemented that allowed us to recover with no disruption to our end-users. While this case study features an authoring tool, the strategies presented herein apply to any tool you rely on as a key part of your workflow.

1. We Didn’t Know Until We Knew
A Journey Through End Times
Margaret Fero
Technical Writer | Degreed
Maggie@degreed.com
@maggiefero1

2. Agenda
Agenda Slide
Map © OpenStreetMap contributors (CC BY-SA)
https://www.openstreetmap.org/copyright
2
Proposal
The Incident
Recovery
Takeaways

3. Proposal

4. Planning for Purchasing
“Let’s do a full process
for practice”
"Fields of Flowers" by Edmund Garman is licensed under CC BY 2.04

5. Our Purchasing Process
Identify Criteria
"red-checklist freelancer credit to https://1dayreview.com" by 1DayReview and "VAPOR" by AЯMEN are licensed under CC BY 2.0; "Fall for Writing" by
cogdogblog is licensed under CC0 1.0; "$5700" by AMagill is licensed under CC BY 2.0
5
What do we need from
this tool?
Matrix Possible
Vendors
Who can give us as
much of what we need
as possible?
Draft a Proposal
What are the risks and
benefits of going with
our first-choice, and
who else comes close?
Request Funding
Money for vendor, pls?

6. Our Criteria
6
• Source Control
• Content Security
• Output can easily become PDF
• Supports accessibility
• Easy to host
• Modular Publishing
• Templateable or otherwise visually-customizable
• Images can be updated in all locations at once
• Content reuse from a single source of truth
• Variables
• Can add users easily
• Internationalization-friendly

7. Our Matrix
7
Vendor One Vendor Two
Source Control
Content Security
…
Internationalization

8. Our Proposal
8
• Executive Summary
• Program Goals
• Description
• Cost-Benefit Analysis
• Assumptions and Risks
• Alternatives Considered

9. Specifically:
9
• Executive Summary
• Program Goals
• Description
• Cost-Benefit Analysis
• Assumptions and Risks
• Alternatives Considered

10. Specifically:
10
Assumptions and Risks
This assumes that upcoming features will successfully launch, and that
[Vendor] (which is a startup) will not stop business. Given their current state
of funding, the near-term acquisition or shutdown risk seems relatively low.
The risk of their road map falling behind is more severe, but releases occur
more frequently than alternates’ releases, which mitigates the risk of
features slipping by more than we can tolerate. Also present is the risk of
relying on external hosting, which we can mitigate slightly by keeping local
copies of documentation and which will also be a risk to varying degrees of
any authoring tool we pursue.

11. Funding Request
Success!
"Money" by aresauburn™ is licensed under CC BY-SA 2.011

12. Migration and Use
Happy Days!
"Bright flowers in a planting bed" by Ross A Hall is licensed under CC BY 2.012

13. The Incident

14. First Signs of Trouble
"Banana (variety 'Cocos'): Fusarium wilt (Panama disease)”
by Plant pests and diseases is licensed under CC PDM 1.0
14

15. Incident Timeline
Glitching
15
Custom CSS didn’t
appear, some images
didn’t load correctly
Full outage
AWS Billing Error
Attempts to Contact
Twitter
Slack
International phone call
Official Incident Reports
Warning Our Team

16. Other Signs We Should
Have Noticed
Hindsight is 20/20
16
• Release Notes weren’t being updated
• Vendor team had become less responsive
• Updates on their social media shifted from discussing progress
• Expected runway was coming to an end

17. Recovery

18. Good "morning". I see the [vendor] web
site is running again. Yay! Are you back
in business?
Supportive Colleague
"owl" by Mathias Appel is licensed under CC0 1.018

19. Our Mini-fied Purchasing Process
Check Criteria
"red-checklist freelancer credit to https://1dayreview.com" by 1DayReview and "VAPOR" by AЯMEN are licensed under CC BY 2.0; "Iowa City, IA" by Alan Light
is licensed under CC BY 2.0 ; "$5700" by AMagill is licensed under CC BY 2.0
19
Do we still have the
same needs for this
tool?
Verify Matrix of
Possible Vendors
Is anybody missing? Are
the feature sets still
accurate?
Propose to
Stakeholders
What works for our
very-urgent need right
this second?
Request Funding
Money for vendor, pls?

20. Migration
"Geese" by mitchell3417 is licensed under CC BY 2.020

21. (and cleanup)
"Cleaning stock photo" by bmstores is licensed under CC BY 2.021

22. A Digression on Backups
"S6300401" by twingly is licensed under CC BY 2.022

23. Takeaways

24. • Who else could fill your need?
• Have a BATNA
Have a Plan B
"Open Tools4Wisdom Planner, white pages, monthly view" by yourbestdigs is licensed under CC BY 2.024

25. • Set a plan for backups
• Stick to that plan
• Do not cut corners
• Do not diverge from the plan
Have a Backup Plan
"Walking the beam." by chrisjtse is licensed under CC BY-ND 2.025

26. • Helps with maintenance
• Helps with delegation
• Enables distributed migrations
Keep tasks broken down.
"Lego bricks" by EEPaul is licensed under CC BY 2.026

27. • You’re partnering on a critical part of
your business
• Even if they’re pretty reliable, you
want backups
Your source control is only as reliable as the
vendor.
"Family holding hands together closeup" by agilemktg1 is licensed under CC PDM 1.027

28. • Decreased responsiveness
• Missing or late release notes
• Shortened roadmap
• Decreased or missing hiring
Watch for warning signs.
"‫באלכימיה‬ ‫המים‬ ‫סמל‬" by zeevveez is licensed under CC BY 2.028

29. Thank you!
@maggiefero
You can ask questions in the hallway now, or on Twitter later!
Maggie@degreed.com
Degreed.com/maggiefero
Linkedin.com/in/margaretfero
29