Download to read offline
Uploaded byKM Institute
KM SHOWCASE 2019 - USAA Knowledge Base Case Study: Improving the KM Architecture
The document summarizes the USAA Knowledge Management team's project to improve search and findability on their knowledge base. The project goals were to organize content intuitively, allow filtering of search results, customize the search engine to their language, and quickly locate procedural information. They hired a vendor to help achieve these goals faster. Insights included realizing auto-tagging isn't perfect and communication is key. Outcomes included organized content with a common vocabulary, a customized search engine, intuitive browsing, reduced help line calls, and improved click-through rates.
More Related Content
Similar to KM SHOWCASE 2019 - USAA Knowledge Base Case Study: Improving the KM Architecture
KM SHOWCASE 2019 - USAA Knowledge Base Case Study: Improving the KM Architecture
- 1. 1 Search and Findability USAAKnowledge Base Case Study: Improving the KM Architecture Lead Development Product Owner, Knowledge Management Jennifer Jensen, CKM, CPM, Scaled Agilist April 5th, 2019
- 2. 2 Our Mission The missionof the association is to facilitate the financial security of its members, associates, and their families through provision of a full range of highly competitive financial products and services; in so doing, USAA seeks to be the provider of choice for the military community. Our Core Values Service Loyalty Honesty Integrity Passionate Member Advocacy Financial Strength & Wisdom Shared Military Values Our Brand Pillars GOING ABOVE Our Brand Promise FOR THOSE WHO HAVE GONE BEYOND Who We Are
- 3. 3 Former State ofKM at USAA Creation of a Team of Teams Project Goals for Search and Findability Why Hire a Vendor? Insights and Aha Moments Project Outcomes – Delivering Results
- 4. 4 Did you know... Todaya person is subjected to more new information in a day than a person in the middle ages in their entire life!
- 5. 5 Former State ofKM USAA – Many lines of business Distributed Models Who owns what? Controls
- 6. 6 Creation of aTeam of Teams A formal Knowledge Management team was established to support the enterprise with several key functions: › Search and Findability › Governance › Knowledge Management › Administration
- 7. 7 Creation of aTeam of Teams (continued) This Team of Teams was set up to run in a scaled agile construct, holding both individual and combined ceremonies, and working in a cross matrixed environment Each team reviewed Voice of the Customer, and performed stakeholder interviews in order to uncover gaps that were deemed highest risk in order to decide what to tackle first Improving Knowledge Architecture was a high priority for our team as it greatly impacted our Member Service Representative’s ability to serve our members with relevant compliant information in their moment of need
- 8. 8 Knowledge Architecture Evolution BaseLanguage Taxonomy The language used to identify, describe, label, and organize knowledge. Organized Content and Smarter Search Optimize the native components of the KD information architecture. Super-Intelligent Search KD knows and tailors presentation and results to user attributes. Larger Knowledge Architecture Identification and documentation of tangible knowledge domains of users, content, and terms. Findability of the Future Automated search optimization and strategic contextual push. Fix the Foundation Optimize the Experience Build for the Future Scalable World Class Findability
- 9. 9 Project Goals forSearch and Findability Problem Statement Representatives are unable to locate critical procedural information they rely on daily to serve our members. This results in increased operational costs and risk, poor member and employee experiences, and reduces USAA’s ability to be a compliant company. Goals • The content is organized in way that is intuitive to representatives. • Representatives can narrow search results based on filters meaningful to them. • The KB search engine Natural Language Processor is customized to the language of members and representatives. • Representatives are able to quickly locate critical procedural information in a way that is intuitive and emulates the way they expect to be able to look for information.
- 10. 10 Familiar Experience Browsing Faceting Search *Ebay, Google,Amazon are registered trademarks of their respective owners and their us here is not authorized by, sponsored by, or associated with the owner. They are used here as non-specific examples of general non-proprietary concepts.
- 11. 11 Why Hire aVendor? In the early days of our team we were small but mighty! USAA and EK are both focused on utilizing Human Centered Design to solve our most complex problems We realized that we could go faster and further together for the good of our users and members, providing compliant, relevant information in their moment of need
- 12. 12 Insights and AhaMoments Wagile Single vs. Phased Pilot Authors, content owners, and representatives helped us in creative ways we didn’t anticipate Auto-tagging isn’t perfect! A solid communication plan was at the center of our effort
- 13. 13 Insights and AhaMoments Creating a Low-fi Neural Network https://www.youtube.com/watch?v=rEDzUT3ymw4
- 14. 14 Outcomes Consectetur Adipiscing Voluptatem Incididunt Consequat laudantium Nemo tempora quis veritatis Organized Content Common Vocabulary Custom USAA NLP ThisPhoto by Unknown Author is licensed under CC BY-SA-NC This Photo by Unknown Author is licensed under CC BY Intuitive Browsing
- 15.
- 16. 16 Outcomes Representatives are statingthey can now find what they need to serve our members more quickly with higher confidence Feedback from our Helpline is that calls have been reduced related to help with searching our Knowledge Base Search results indicate an improved click through rate Our work is not done! We continue the work to improve search and findability, teaching users how to help us help them
- 17. 17 Summary Former and CurrentState of KM at USAA Creation of a Team of Teams Project Goals for Search and Findability Why Hire a Vendor? Insights and Aha Moments Project Outcomes – Delivering Results
- 18.
Editor's Notes
- #2 Welcome to our pre lunch topic! Please stay with me here, I promise I won’t go over time so that we can get to lunch on time. My name is Jennifer Jensen and I work as a lead DPO for our KM search and findability team. If you have any questions as we work our way through the presentation I ask that you please note them down and I’ll leave time for a quick Q&A session at the end. I’m also available throughout the conference if you have further questions for me! First of all, how many of you have heard of USAA? Do you know what we do? (allow someone in audience to answer) We are a member owned organization that serves the military community, most people know us to be a bank and insurance company, but we have a whole host of products we serve our membership with which can make km especially challenging!! (move to mission slide and can explain further what we do)
- #3 At USAA with every internal meeting and presentation we always start with our mission, and that is no different when we present externally. We want you to understand who we are, and how we serve members. Our part as a KM team in helping our employees to deliver on our mission is critical. Without being able to find the right information at the right time, our employees wouldn’t be able to do any of these things! What you see to the right through our brand promise and our brand pillars are part of how we hold ourselves accountable to the mission. So I’m going to ask you a question. Take a look at this, and consider it in the context of KM and a knowledge base. Can anyone share with me what you think about the importance of solid knowledge architecture and how those things intersect? (allow someone in audience to try to answer, help them along with this:) Very good, it’s how we share that wisdom so that we can be advocates for our members. (Pause and take a breath and allow that to soak in) In today’s current financial culture, compliance and risk also play a very big role in how we provision our range of products and services, and for good reason. Those controls are put in place to protect consumers. Good KM ensures that we can do this on every call.
- #4 Here is a high level agenda of our 50 minutes together. I will spend about 5-7 minutes sharing our former state of KM and level setting so that you understand what we do at USAA and how we used to manage knowledge before our team was put in place. Then we will talk( for about 5-7min) about how the KM team was stood up and the different parts of KM that we define and manage for the enterprise. I will share our project goals for my team’s large scale year long (and counting) effort to improve search and findability (10-15 min), and then we will talk about how you might determine when to hire a vendor in your own projects (5). I’ll share with you a few insights and AHA moments (3-5), and then we will discuss our project outcomes (5 min). After which, we will have about 5-10 min for Q&A. Sound good? I don’t think I heard you! Sound good? Everyone with me? Alright, let’s get going!
- #5 I know you all feel this, we have so much information coming at us every day. What do we do with all of this information? When an enterprise grows, can you imagine with regulatory rules and compliance how much more important being able to sort through and find just that right piece of information becomes? And as our employees get accustomed to better user interfaces and more intuitive ways to find information out in the world, they expect the same experience at work, and why shouldn’t they?
- #6 At USAA we are not only an Insurance company, but also a bank. We have many lines of business and many product offerings under the USAA enterprise. Yet, we didn’t’ have one centralized team who was the shepherd of our Knowledge Management function. Our knowledge management was left to the lines of business to figure out. They were responsible for creating content, properly tagging it, and storing it in our knowledge base. We had a publishing team who would ensure that the documents were published according to the rules for the types of knowledge we stored, but their main function was not knowledge management. Because some areas closely overlapped it was confusing for the lines of business to determine who should own what, and there were no formal centralized controls in place for reviews of content, updates, etc. We have so many different types of employees as well! We have those that serve on the front line, such as Member service representatives or msrs, as well as back office support staff that enable those front line employees with areas like underwriting and fulfillment. Then we have employees that work in non-msr facing roles that we consider staff roles. Almost all areas utilize our KB in some way. Our third party vendors rely ONLY on our KB. So, you can see how important it was for us to face this challenge head on. We used this photo in the early stages of our team as an example and it still makes me laugh. This looks like a hot mess! I think it’s safe to say that our KM wasn’t quite this bad, but it sure wasn’t pretty either! Our employees are resilient and they found work arounds for work arounds, but they deserved a better system and we were determined to step up to the plate and make some big changes to improve their user experience. We had to clean up our library.
- #7 The four key functions that we determined needed support across the enterprise are shown here. Each area has several employees dedicated to improving and in some cases establishing the way we handle KM at USAA. The first one is likely obvious, I’m proud of this one because it’s my own team Governance helps with establishing Service Level Objectives and ensuring we adhere to compliance for review of articles (for example) KM works on the practice of KM across the enterprise such as: templates, forums, a playbook for authors, and Administration works on reporting for our own team as well as other authoring teams and individual users(just to name a few). The lines of business still own the actual content, but now our KM team helps them better manage that responsibility with an eye on compliance and minimizing risk to the enterprise.
- #8 Ceremonies: daily standups, demos, iteration planning and reviews 1. Our teams work in a cross matrixed environment, that means when we need additional help beyond the capabilities of our team, we can pull individuals from other teams into out projects with the right skillsets. It’s a very powerful way to work when we share resources across teams without having to permanently reassign anyone. They get the opportunity to shine, doing what they do best, contributing to some pretty exciting efforts which gives them a chance to grow their skillset, and we benefit from their knowledge, diversity, and expertise. Show of hands, how many of you work in an environment like this? That’s great! Your organizations are harnessing the full potential of their workforce when they encourage cross collaboration! KB users were interviewed, our member service representatives are the most demanding users of our system, we spent time listening to them and their needs and performed y cording sessions where we were able to watch them interact with our members and the knowledge base. This exercise was equal parts exciting and scary! We discovered that they were beyond excited to have dedicated folks with whom to directly share their feedback AND WE DISCOVERED THAT THEY WERE BEYOND EXCITED TO DIRECTLY SHARE THEIR FEEDBACK. To say there was a lot of feedback could be an understatement! But we expected this and we welcomed it and just by doing this one simple thing with our employees we gained a lot of trust early on, that served us so well over the course of the projects our team undertook. As we worked we continually asked for feedback and we imbued that into every project. I have folders upon folders brimming with this goodness! 3. So because of this, we embarked on our initial project with the hope that in one year, we could make a huge difference to employees across the enterprise. Are you with me?
- #9 (read slide first two boxes) We started our journey with “fix the foundation”, which began late in 2017 and although the formal project ended at the end of 2018, we still continue to improve upon that foundation (I lovingly call this, flx the broken things) We knew we had to focus our efforts here before we could do anything slick and fancy! We had to get this part right in order to enable the future state of KM at USAA which you see in the boxes to the right. This is the logical progression of steps to reach World Class Findability, our search and findability Nirvana if you will. Has anyone attempted to do these steps in a different order? If so, how did it work?
- #10 These are the real Problem and Goal statements from our project. I wanted to share it with you because sometimes I think problem and goal statements can be quite challenging to agree on as a team. I can’t tell you how many iterations it took to get to this problem statement but I can tell you it all came from our users. We took their own words and we combined that with what we knew we needed to do as an enterprise team. We asked ourselves “WHY?’ probably a million times (ok, I may be overstating, but our team is a big fan of Simon Sinek and his work). We really wanted to get to the root of our problem so that we could ensure we were working on the right things. The goals came much easier! Our employees also helped us with this part, these are the things they told us they wanted (in less technical terms of course) Quick stats: 20,000 users 13,000 content records Hundreds of products Dozens of roles Five lines of business + Enterprise Each division is related enough to use the same terms, but with different meanings and for different content. THE TOOL: 6 non-it applications 30+ feature sets Hundreds of features Thousands of options 23 product manuals 6000 pages One new language
- #11 We wanted our users to have a more familiar intuitive user experience like they might have on other commonly used websites, but at the time, I was a lone ranger! I was a program manager with a lot of leadership experience and project and programmatic experience, but no formal KM experience. To say I was a bit terrified might be an understatement, but I was determined. I have an extensive background in Human Centered design having been to Stanford d. School and worked in our Experience Architecture department, so with that knowledge, I moved forward into uncharted waters. I was excited for the challenge. I didn’t stay a lone ranger for long…. (next slide)
- #12 When our team was stood up, we were cross matrixed and had to share resources with other areas. In short, in order for us to begin on this journey we knew we not only needed experts in the field (as I mentioned, I was a lead project/program manager with no formal KM experience) but we also needed a dedicated extended team to help us get there fast. in order to tackle our biggest hurdle, we entered into a partnership with Enterprise Knowledge. With their help we were able to accomplish what one person (me!) couldn’t do alone! We went further faster with EK and a cross matrixed team and learned a great deal along the way. We performed countless stakeholder interviews, card sorting exercises, and co-design activities to engage our users in the taxonomy re-design process. No single term was created from our own team! We only facilitated these sessions, our users did the heavy lifting! Through this process we uncovered over 600 search terms that were never added to the architecture, so new articles could not be tagged with language that made sense to the user, making it increasingly difficult to find. They would memorize difficult workarounds and once found would save articles to their favorites for future reference. New representatives coming in had to be trained how to find things in our KB! With the help of our users, we had them sort all the generated search terms into buckets that we call categories. We helped them understand that buckets inside buckets were to be avoided as much as possible unless it made the most sense and showed them why. We essentially taught them all about knowledge architecture, so now we have tons of folks who can help us continue to tweak this to improve it over time!
- #13 You have to play to your strengths. We jokingly call this WAGILE. It took a combination of project management and agile to get our project delivered on time We thought we would have a single pilot, but realized a phased pilot made more sense so that we could include a more diverse set of users. We utilized users in a lab environment to help us with our pilot as they are used to (and trained in) giving generous feedback. However, we also went outside of the lab with other users who had never participated in a pilot. We also included more than one functional area to ensure we weren’t focused only on our most vocal users. We used our authors, content owners, and representatives to help us check and tag articles with the use of a custom designed interface created by EK in order to train the computer (machine learning) to auto tag our articles. We checked and cross checked every term against multiple types of users. We had some users generate terms and others sort them into buckets and add what they thought might be missing. Then after all of that we had them all review the tags and we are continually having them review to ensure nothing was missed or mistagged. They learned KA! They would come to us with creative solutions that worked well for their line of business. It was awesome having them as partners on our journey and allowed for a lot of great collaboration between all the teams even beyond the project and pilot. They continue to reach out when they solve problems and we continue to tweak things to improve. We knew this, so we came up with a MVP solution prior to rollout. We had a plan for retagging articles that might have landed in the wrong place through either human or machine error. At the heart of any change is a good change management plan. We communicated early and often with our users. We told them what to expect and we were honest about the imperfections. They were just so excited we were finally fixing it we were surprised at how understanding they were! They wanted to help us help them!
- #14 Explain the concept of neural networks….
- #15 Through a common vocabulary and utilizing our user inputs to organize content we were able to create a low fi neural network to design a custom USAA NLP. Our new architecture enables a more intuitive browsing structure that is unique for each line of business, the structure itself is much flatter. Can anyone help explain what a “flat browsing structure looks like?” (allow someone from the audience to help out, prompt someone you know to help out in advance if no one volunteers). Great! So with a flatter browsing structure our representatives are able to quickly find what they need without having to dig deeply through a series of folders, unsure if they will eventually get there. I think of it like the stacking dolls inside one another. What we did was to take them apart and put them beside each other, so if you need a medium doll, you can grab that one and off you go!
- #16 This is a diagram of how we view knowledge architecture. We begin with concepts that are relevant to each line of business, we then classify that information in a hierarchy or type, we then look at the ontology of our taxonomy, or how those concepts relate to one another, we derive meaning from those relationships that inform the rules that are behind our Intelligent search ontology If you look at the break down of how this works to the right we show a typical search string, the purple denotes a piece of our unique vocab and how we use that vocab to then classify and make relationships between search terms to enable our search engine to derive meaning and serve back the expected result. This is essentially our low fi neural network explained.
- #17 We are still in the process of determining the final outcomes of our project as we just delivered it at the end of last year. Any major change will take several months to see results that we can quantify beyond qualitative feedback. So far, we see a positive trend even with our quantitative numbers, however we expect those numbers to improve as our users unlearn the habits they had to learn in order to make the correct article appear in search. We are not only changing the way search works, but how our users search for information. To this end we are enlisting the help of our power users to help us help our users UNLEARN their old search habits!
- #18 Ok, we’ve covered a great deal in our time together and I bet you’re all chomping at the bit to get to lunch. I’d like to thank you so much for allowing me to share our project with you and for being such a great crowd! This is the first time we’ve ever shared this outside of USAA, so I hope that it was clear with actionable steps you can take when trying to solve your own KM challenges. In the remaining time that we have left, what questions do you have for me?
- #19 Ok,