Everyone has blindspots. For developers it is often taking for granted certain technical skill sets. So what happens when you need to build a system for people without those skills? How do you adjust? What do you use as a reference point for common knowledge? How do you get out of your own head?
This talk will tell a story of a project where we attempted to do just that. What we learned. What challenges we faced. And how our final system took into account all of our experiences.
We were working with a company that had two teams. A technical development team, and a data team. Within the company there was an analytics application that was currently in use, subject to lots of worldwide scrutiny and about to fail under the weight of what had been crammed into it. That was problem number one. Problem number two was that the system was one giant piece of functionality that spit out a solution. You either ran everything or nothing and that was not going to work anymore. Then came problem number three, arguably the biggest issue with the system. The data that went into the system was owned by the data team, the results of what came out of the system were owned by the data team, but the system itself was owned and operated by the technical team. That needed to change.
So we went to work designing a new system. The system needed to work exactly as the old one did. The data that went in needed to produce the same results when it came out that the old system had. But it needed to do more than that. It could no longer be a black box. The data team wanted visibility into what was happening at every step of the process. That meant that the processes needed to have steps! Finally, the system needed to be owned, operated, and updated by the data team, a non-technical, though incredibly mathy, division of the organization.
We had an existing system with known requirements, a clear customer with specific needs and we were good to go! The technical team would work to pull out the details of the existing blackbox into a useable maintainable tool and everything would be good to go.
As we all know, things aren’t that simple. And in this case, the biggest road block was not the data team and their needs, but the technical team. Let me show you what I mean.
Not an ideal way to start the project. This debate brought up two important things.
Pick the right tool. We went with something called Knime that works as a visualization of an algorithm as it operates on entire database tables. You’ll see some examples later.
2. Don’t underestimate your non-technical resources. The data team weren’t developers, but they knew had to use plenty of tools. They were fluent in things like Excel and SQL queries such as in Navicat. We needed to leverage that knowledge and comfort level to build something they would pick up easily and be able to run with.
Step Two: The blackbox. Like it or not, part of the reason we as developers are so valuable is that we make magic happen that no one can see. Throw some information at a little black box and out comes exactly what a customer wanted! Yay! And for years the data team had told them what the black box should do, given them the data, and confirmed the output. But since they were the ones who knew what should be happening, and were in the business of making changes to workflows and algorithms inside the system, this new system needed to provide them the ability to do it themselves.
Break it down. Make it clear where the data comes in, how it is altered while in the system, and where it leaves. When you think you can’t break it down anymore? Try again.
Show this! Visual cues are extremely effective at conveying information and abstracting away technical ideas into process steps.
But this was only the start. The biggest gains came in the workflow files themselves.
If you’re going to leverage what your non-technical users know, don’t forget to remember what your developers know, even if it manifests itself a little differently. This was a sophisticated tool of fully repeatable closures, but it was incredibly accessible. Label things, write comments, lay things out in a way that they’re easy to find and it’s easier to understand what they do.
Make your errors clear. One of the hardest things about computer systems is understanding what went wrong. Visual colored statuses on every node made it clear which steps had been successful and which, if any, have failed. If you want non-technical users to own and operate a system, they need to be able to debug it too.
Step Four: Focus on the goal
The data team wanted and needed to own this system because they were in charge of the data and this was a data manipulation tool. So we took that to heart. At every step, after the functionality and transformation steps of every node, the entire data set was available for view in its new form.
1. Share as much information as possible. Making the input and output of every step clearer made it easier and more obvious to understand what was happening.
Step Five: Be Consistent
All of these recommendations are important. As the data team went through the workflows and looked at the patterns they grew more comfortable with the system. But that only worked if things were done the same way every time. Once there was a pattern to work off, it was easier for them to adapt that and make changes going forward. To own the system and alter its algorithm, calculations and results.
Having walked through the project and our solution to the challenges of non-technical system owners here are a few of the main takeaways I hope you come away with from our experience:
Take yourself out of the developer mindset, don’t build a fast node app just because it will solve the immediate problem, see the whole picture
Learn what they know, what tools they use, there are clues there of what behavior they are comfortable with
Technical tools can be developed by other people, but you have to set them up for it, this talk has shown you just a few ways we did that
There are a number of lessons from this project. Some are big picture concepts and some are smaller details that may not apply to every project. But the hope is that many of the solutions we developed have parallels in work you do and see. Who doesn’t want one less project for the engineering department to have to constantly update and maintain over the long term?
And if you’re interested in learning more about how the whole system worked, please come find me. I’ll be around all day.
Tech For the Non-Technical
BUILDING TECH FOR THE
By Laurie Barth
Ten Mile Square Technologies
• Brittle system
• Owned and operated by technical team
• PLSQL 2000+ line function
• Remove the blackbox
• New system must be maintained and
developed by data team (non-technical)
• Future tech support for infrastructure only
• Pick the right tool
• Don’t underestimate non-technical resources
• Break it down
• Use visuals to your advantage
• Use development best practices
• Make your error states clear
• Don’t hide information, make everything accessible that you can