Fitting Agility and Quality Under One Roof
•Download as PPTX, PDF•
0 likes•95 views
It’s the million-dollar question: How can you release with the speed and agility that the pace of business demands while maintaining the quality consumers expect? Typically, brands are only able to deliver on one or two of these expectations. Realtor.com, the pioneer of digital real estate, found out these problems firsthand. When it put emphasis on quality, its development was slow for both web and mobile. When it focused on speed and agility, quality suffered. Realtor.com found its solution in a combination approach, revamping its engineering capability, investing in automation, and leveraging crowdsourced testing to ensure the human side of quality was achieved with all its constraints. Discover the advantages of real-world testing and how Realtor.com achieves speed, agility, and quality, all at once.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Fitting Agility and Quality Under One Roof
Similar to Fitting Agility and Quality Under One Roof (20)
More from Applause
More from Applause (20)
Recently uploaded
Recently uploaded (20)
Fitting Agility and Quality Under One Roof
- 1. FITTING AGILITY & QUALITY UNDER ONE ROOF Webinar with Realtor.com
- 2. 2 FARIDA BUDIWARMAN Sr. Director, Engineering Realtor.com CHRIS SHEEHAN VP, Enterprise Customer Success & Renewals Applause
- 3. The Competing Priorities 3 AGILITY VELOCITY QUALITY vs vs
- 6. Drivers For Change 6 Data-Driven Experiences High Consumer Expectations Heavy Competition
- 7. The Situation 7 THE GOOD THE BAD THE UGLY
- 10. A Revamped Approach 10 Process | Tooling | Skillset Initial Limitations Unclear team guidelines and ownership Success based on the wrong KPIs Path to Success Standardized testing processes and strategies • Elimination of silos • Harden release process • Accountability for team members Establish clear quality bar • Zero Out of SLA program • CSAT • Test code quality • Support from Applause
- 11. A Revamped Approach 11 Process | Tooling | Skillset Initial Limitations Scattered technology stack Limited Automation Capabilities Path to Success One common framework Extendable Integrates easily with other tools Easily scalable
- 12. A Revamped Approach 12 Process | Tooling | Skillset Initial Limitations Redundancy of work Lack of on-staff experts Path to Success Diversifying skillsets of team members Hiring the right people
- 13. WHAT DOES TESTING LOOK LIKE NOW?
- 14. Go Agile 14
- 15. USE CASES
- 16. Data-Powered Experiences 16 Sign Snap Street Peek
- 17. Test In The Wild 17
- 18. RESULTS
- 19. Increased Velocity 19 Reduction in manual regression testing Faster time to sign off release candidate Increase in ability to support business to release 70% 80% +
- 20. Increased Agility 20 API test coverage Increase in ability to catch defects Increase in tooling 60% 1.5x +
- 21. Increased Quality 21 Success rate of Zero Out of SLA program Bug prevention rate Decrease in P1/P2 production bugs 90% 70% 40%
- 22. LESSONS LEARNED
- 23. #1 ADOPT THE STRATEGY THAT “WORKS” FOR YOUR CULTURE
- 24. #2 IDENTIFY AND FIX YOUR BLIND SPOT
- 25. #3 LET GO OF YOUR SUNK COST
- 26. #4 GET HELP WHEN YOU NEED HELP
- 27. #5 BE A LANNISTER: ALWAYS PAY YOUR DEBTS
Editor's Notes
- <SARA>
- Thanks Sara. By way of introduction, my name is Chris Sheehan and I manage Applause’s enterprise customer success and renewal team.
- Sheehan: So lets set that table for today’s discussion A wonderful part of my job is that I get to sit down and talk to many of our customers about their challenges and opportunities. And almost without exception, I hear a very common theme. And that is, We must deliver compelling, innovative digital experiences to our customers or we risk losing business. This means being more agile, releasing features faster, and, at the same time, improve quality. But this is a very hard balancing act that many companies struggle with. Farida, Realtor.com has managed to do it, but you needed to drive a significant amount of change over the last couple of years So lets dive into that journey…..
- Sheehan: Farida, a great place to start is to introduce us to Realtor.com. What is Realtor.com and who uses the platform? Farida Realtor.com is an online real estate platform which has been trusted resource for home buyers, sellers, agents and dreamers since 20 years ago. We offer most comprehensive source of for-sale properties among competing national sites, and the information, tools and professional expertise to help people move confidently through every step of their home journey. As one of the fastest growing digital real estate company, we serve more than 60 millions unique users monthly. Our metrics have consistently shown how we lead most consumer sites in engagement, based on time on site and page views. Our users view 1.7X more pages and spend 1.3X more time per visit on realtor.com than our nearest competitor. This year we were recognized as the Gold Stevie Award winner for Best Real Estate Mobile App in 2018, which was an accomplishment that showcased how a great team work together to make a difference in the industry and create product that make positive impact in our user’s life
- Sheehan: Congratulations for all your success and winning the best real estate mobile app! That is a great achievement, ……..but lets rewind a couple years –
- Sheehan: …..can you share, at the time, what was pushing the team to change..…that is why did you need to become more agile, why did you need to improve quality? Farida: The team has the mission to grow the business from good to great. But it’s not easy as digital real estate competitive landscape is very fierce. We have a lot of data that our competitors don’t have. We also continue to see growth in traffic and user engagement. So the key is to leverage both and build unique data driven experience that would help users find their future home anywhere anytime. This experience has to be excellent, lightening fast and relevant. Our users can be in different stages of their home search or home ownership journey. We need to build multiple products that can cater to their needs, we need to support in multiple platforms and environment. And we need to do it fast. However just like any other growing company, sometimes we suffer from growing pains that hindered team’s ability to execute and deliver innovative offerings to our users. We also realize that as our product becomes more sophisticated, we started seeing inconsistencies in the experience (across platform) and data quality issues The latter is harder to solve for my team in particular, because of our limited capacity
- Sheehan: So you had increasing traffic, an expanding product portfolio, more platforms to support, opportunities to use your data, ….but you were also dealing with a lot of growing pains As you stepped back and looked at the bigger picture, what was working and what did you feel needed to change? Farida: Well, first of all, let’s take a look at “The Good” We have massive amounts of data and with those How can we validate this wealth of data more quickly and efficiently? How can we validate the algorithm that’s being built? How can we deliver this content in a more consumable way? We have competitive advantage of high traffic to web and mobile We have unique products that are market disruptor The Bad Just like any other company, there is always competing priority between agility and quality: teams want to move fast and release often. We just moved to agile but if you are not careful, agile can turn into “fragile” which was exactly the situation back then Quality suffered due to multiple issues: #1: Agile encouraged team members to pick up assignments based on bandwidth, however if not managed properly it can turn into lack of ownership. And what happened it’s not clear who owns what, because assignment was randomly distributed across the globe and as the result, when things go wrong, no one takes accountability. Without having accountability, it was hard to drive for corrective actions. #2: There was a lack of clear release criteria, testing was given “budgeted time” and release will happen regardless of the test result. Hotfixes happened often and it was becoming a norm rather than exception. We ended up in vicious cycle of fixing and breaking. With high rate of bugs in production and site disruptions, leadership had to attend site incident meetings quite often. The Ugly We have toolings and automation but they took forever to run. Our business and partners did not have the luxury to wait. Since there was no test strategy, everything was tested from the front end hence we had a thick suite of UI tests that took hours to run. To complicate things more, we also suffer from code quality issue. We had lots of tests but they weren’t effective in finding regression issues. Therefore, the team was heavily relying on manual testing which is error prone and takes longer. The team was not aware of their own blind spot. We got too busy executing, and no one took the time to understand the test coverage or most importantly what coverage is missing.
- Sheehan: so quality was suffering, automation wasn’t effective, agile had turned to fragile ….. all of this meant you needed a new approach…..
- Sheehan: So at a high level, what was your new approach? Farida: I decided to go back to three basic needs: Process Tooling Skill set First, we need to assess where we are with the basics. Without any of the three, no team or company would be able to compete and win. You need skillset (or people with the right skill set) but they are not going to be successful without the right tooling and processes. Likewise, you can have the most sophisticated tooling and mature processes, but without the right talent, you are still going to fail. Once we identified our bottleneck, we can then start the transformation
- Sheehan: ok, so you identified the overall process as being broken. What specifically was broken and how did you go about fixing it? And can you tell us where Applause fits into the new process? FARIDA: The team did not have the right processes in place to enable them to be successful. Things like: lack of ownership and accountability that I’ve mentioned previously. I was not convinced that the KPIs we were using were the right KPI to measure ourselves. Are we measuring success by the number of releases we do? The number of story points we delivered each sprint? Or should we measure it differently? For Quality organization, is there clear testing strategy? What’s our quality bar? Or do we even have any? To address ownership and team’s guidelines We standardized our testing processes and strategies. Engineer started working more closely with other engineers. Each team started working more closely with other teams. Slowly we removed “siloed” working model and team morale went up. All team members are now held accountable for the area they are working on. Instead of random assignment, everyone has goals and areas that they own end to end We also continued to harden our release process. By end of last fiscal year, the team has been able to reduce manual regression effort and regression cycle was reduced to less than a day. We established our quality bar through multiple KPIs. We introduced Zero OOSLA program (Out of SLA program) so that bugs get addressed within defined SLA. We started with having more than 500 production bugs laying around not being addressed and by month #6, more and more teams paid attention to their production defects and addressing them in timely manner. To measure the impact on our users we used CSAT, user feedback and the number of production issues On engineering side, we introduced enhanced code review process where purpose of review was not only to look at code quality but also to reduce the risk having tests that are not valuable or helping us to achieve our goal. We cannot avoid tech debts, but we try to manage them as early as possible. How does Applause come in? Applause testing is a complementary to our testing strategy. There are few testing that we simply cannot do given our team size and location and we gave this task to applause so we can focus on other things. It’s kind of like divide and conquer approach.
- Sheehan: lets turn to automation. We all know that in order to support agility and increased velocity, automation is a key part of any testing strategy. But It sounds like you started with an automation approach that simply was not working. Can you tell us what you did to make it work….and I’m particularly interested in what you decided to do regarding an automation framework……and how automation impacted your manual testing approach. Farida: We knew we can’t support the required amount of agility without having a robust, scalable and stable automated test suite. But we had following problems – Scattered technology stack was posing a big problem for us. In some team selection of technology was driven by individual choice, comfort level with a particular technology, available resources. Not everyone was giving adequate thoughts regarding the long term effects such as resource mobilization, short coming of the technology based on its application towards our growing testing needs, ease of integration with other tool sets, ease of hiring, scalability, reusability, pluggability etc. Existing automation framework had a limited capability to support UI automation. It was slow and hard to extend to support other types of automation such as API, Native App, Mobile browser etc. How did we decide what kind of framework we need? Since cost and time was the major factor while choosing the new framework, even though we considered Java Script as an option we decided to stick with Java since the existing automation framework was based on Java and was used by the largest team within my org. It helped us reducing the onboarding cost of new framework drastically. Our requirements were very clear- We need to have one common framework which can supports most of our automation needs Should be extendable, pluggable with other tools, Should have decent reporting Scalability and performance should not be an issue And it needs to easily integrate with any sort of CI/CD pipeline Now since cost and time were the major factors while choosing the new framework, we decided to stick with already proven industry standard framework but we made few tweaks to improve the performance and cater to our company direction Since the technology stack used was something that the team was already familiar with, it helped us reducing the onboarding cost of new framework drastically. How did it help with manual testing? The only way we could improve our agility is by improving our automation quality, stability and coverage. It was also very important for us that we put the right amount of testing in each layer of test pyramid and find a way to measure the coverage in each test layer so that we can define and predict the cost of our manual test suite accurately. One big change that we did was to start writing automated tests at API level. The reason is pretty self explanatory. Not only it’s cheaper and faster to do, but it’s also more reliable We integrated our automation framework with our test case management tool. This helped us figuring out how much we have automated and how much manual testing we still need and hence we were able to predict the cost of manual testing/regression time more accurately. By end of the fiscal year, we were able to save significant amount of time doing manual work and nowadays, we spend less than 10% of our time, doing manual test.
- Sheehan: So the third area you mentioned that you needed to improve was the skillset of the team Can you tell us why you needed to change the skillset and how you went about doing that? Farida: When I joined the team, all our Full time engineers did manual testing and all our contractors did test automation. This has created inefficiencies within the team and it was as if 2 teams doing the same thing. So we started the journey to train our FTE to do test automation. The idea is everyone own quality end to end and as we get fluent in test automation, naturally the amount of manual work would decrease. However, strengthening our existing team alone, will not suffice. We need to get more talents to mentor and grow the team as well. The hiring journey was not easy. Especially when you are looking for skill set that are a hybrid of development and test. We did spent a big chunk of our time in hiring effort, including networking, crafting compelling job description, and talking to lots of candidates explaining the challenges and opportunities that we have. Our company is far from being traditional. We are fast paced, and we need people who can think on their feet and can grab the next opportunity. I also wanted to have well diverse team of engineers. Each team member has skill set that is unique to the team and complementing the team. One thing though, they all need to have obsession for quality, empathy for our users, and persistence to break developer’s code. This is the holy grail of software testing and a must have for us.
- Sheehan: You’ve clearly come a long way in two years. What does your overall testing process look like now? Farida: For most companies, test automation is done as a catch up work. But for us, test automation is being built before we release the product to our users. To make it even easier, we have rolled out standard design pattern across teams. This has helped us reduced our manual regression effort tremendously. We are measuring our automation more effectively to help us estimate our manual test effort including how to best leveraging applause Our new automation suite executes in minutes compared to 2 yrs ago when it executed in 3-4 hours With the new ownership model, we eliminated redundancy of work and instead do divide and conquer approach Being centralized Quality organization with standardized process and tooling, helped us to break the silo and improve engagement and sense of belonging across different domain teams Our Testing is now streamlined. Everyone knows what to do, how to do and when to do. They know their blind spot and continue to work on removing that blind spot.
- Sheehan: Lets look at your mobile release process. I understand regression testing used to take a long time and you have cut this down substantially. Obviously, automation has helped a lot, but can you talk to us about how you have used Applause to improve the velocity in your mobile release process Farida: Finding a way to speed up our testing processes was critical to Realtor.com crafting ideal consumer experience. In that same vein, we needed to ensure that our quality was not degraded as a result of the speed. Let me take example of our Mobile team. The team used to have longer release cycle. Regression testing was often taking more than one week to complete. There are different combination of OS and platform to test. Applause helped test incrementally before the regression build was pushed. We involved Applause in earlier phases of SDLC now to shift left and achieve goal of providing sign off in one day. We also leveraged Applause network to help us being agile by involving them in algorithm validation during early development cycle. Instead of waiting to get full user feedback (which is going to take much longer cycle), we use Applause feedback to refine the product until it meets our release criteria The other great example of how our partnership with applause work wonderfully was when we did backend infrastructure upgrade. Sometimes the harder testing to do is to make sure everything is working as is, as if nothing has changed. With such huge volume of properties in all location within US, it’s not humanly possible for our small team to test everything. Applause team has done a great job identifying the discrepancies in our system and helped to rollout the change successfully.
- Sheehan: You now have a much more efficient and effective testing process. And from the work we do with you, I know this has enabled you to release some innovative, high quality features for customers.
- Sheehan: Why don’t we touch on two of these features and talk about the testing approach for each To start, can you give us a brief overview of – sign snap, street peek, and property insights Farida: If you have our app, some of the unique functionalities that we built such as Sign Snap would enable users to be on the go and view the properties information from their phone by taking a snap of the for-sale sign. If you’re shopping for a neighboorhood, you can use Street Peek to scan the entire neighborhood to see how much each home is worth. Simply tap the feature on your app and point your camera to span the neighborhood street and via augmented reality, you’ll get property details about any of the houses on the street. We also do a lot of work to leverage our wealth of data by providing unique property insights to our users. Since buying a home is a serious commitment, we need to help our users by providing more and more insights to help them make informed decision
- Sheehan: So Sign Snap and Street Peek are features used by customers when they are out and about….. What was your approach to making sure that these were not only fully tested before being released but also were going to provide accurate data to users? Farida: Features like Sign Snap and Street Peek are designed for the on-the-go customer to use whether they are in the car or out for a walk. Testing new features like these cannot be done in a lab setting, but really require testing done in real-world conditions to mirror the unpredictable conditions of each GEO location and the unpredictable nature of the consumer. We don’t have enough in-house resources available to test the desired locations in the wild, yet it’s critical that the product is fully tested. Applause partnered with us not only to add testers at various locations around the country and expanded our device coverage across locations but also to help providing qualitative feedback that we use to refine our algorithm.
- Sheehan: Lets recap… - We have talked about the changes you made to your process, tools, and skillset of the team. - We also covered some practical examples of your updated testing approach . I’d like to turn now to metrics.
- Sheehan: How much improvement have you seen in velocity? Farida: The team was able to build capability to release faster by reducing manual regression test by 70% and faster time to sign off release candidate by 80%. This varies from team to team, but for example, a team used to spend few days now they can sign off in hours. A team that used to spend a week in effort, now can provide sign off in a day or two.
- Sheehan: How have you improved agility? Farida: We invested significantly in automated tests and I am happy to report that our investment is finally paying off. By the end of the fiscal year, we saw a jump of 1.5x in our ability to catch defects. As a team, we are all more effective in supporting ever changing business requirement and no longer a bottleneck in delivering solutions
- Sheehan: How has quality improved? Farida: About 90% of teams were able to comply with our OOSLA program and as the result, we have less complains from our users. By catching more defects before shipping, eventually you’ll see less and less bug leakage to production. this also translates to our Consumer satisfaction numbers which has been increasing steadily in the last 2 yrs.
- Sheehan: I know folks listening on this webinar will be at various stages of their own transformation into a more agile, higher quality testing organization So I’d like to close by asking you to share your top 5 lessons learned from your journey over the last couple of years
- Every organization has their own culture and mindset. Changing a culture is always challenging. Rather than stirring the pot, one needs to be “considerate” to the situation and organization culture surrounding the workplace. Take the time to adjust, and tweak your strategy as an ongoing process. periodically evaluate the result and again tweak as necessary. Do get a lot of feedback from everyone along the way to understand the pulse and tweak again as necessary.
- Hard to identify, blind spot can be fatal to any drivers in the middle of busy roads. It’s elusive, and almost unavoidable. Just like a busy road, drivers, and their blind spot, organization can suffer from fatal consequences if they don’t carefully look over their shoulder. When things go wrong, the first thing to do is to identify the blind spot, and plan for crossing the other lane successfully. In our example, it was easy to blame it on how slow the automation was, or the lack of resources, but they were not necessarily the real culprit. The danger of not identifying it correctly can be destructive because organization may be spending resources without gaining any benefit. In this case, even if our team gets more resources, we may still not be able to release fast enough or with the quality that we want due to lack of processes, correct KPIs and test strategy.
- Throwing away what you already have is always a painful and hard decision. But, sometimes keeping the old stuff could be an expensive deal. So how did we decide what to throw and why? Throw existing automation High maintenance cost Long execution time Not effective Low confidence Throw existing automation framework Not expandable (Could not support API testing) Not scalable (Took too long to execute) Not Pluggable (Was not easy to integrate with other tools) From processes We decided to remove the siloed way of working because we had too many integration defects. Consolidated releases into more meaningful releases (and therefore we were able to cut overhead cost) while keeping the same outcome. NOTE ONLY: We decided to keep releasing the business critical features while team’s primary focus was migration The architecture team conducted multiple class room trainings followed by daily office hours in order to make sure that no team member is stuck if they have a question or technical issue All the PR’s were mandatory to get merged within 48 hrs so that the test code we were writing can start supporting the regression for business critical releases While we were migrating our automation we decided to use Applause for regression to its maximum capacity
- You and your team are not expert of everything - So look for the expertise around you and delegate the work. Ex - We used applause for our field testing for things that we could not do it in a lab setup. You can’t build everything - You can’t build everything in house so looking towards open source or even a paid tool is not a bad option. Ex - We picked up open source testing framework, bought licenses for some toolings that we are using. Use top down approach when needed - Sometime it is hard to convince partner to accept a goal which does not have direct monetization attached to it. Most of Engineering hygiene goals falls into this category where there is no direct business impact and hence people are less interested in signing up for them. Ask for upper management support to roll out these kind of goals as organization level goal and everybody will sign up rather than rolling out the goal at your level. For example: when we implemented Zero OOSLA goals, I knew it won’t be successful without the buy in from every other teams. So I went to my boss to get his buy in, and he in turn, helped me to set this goal as organization goal and get buy in from other teams.
- Technical debts always bears compound interests, just like your financial advisor will tell you: the sooner you pay it off, the better it is for you. Try to shift left as much as you can with your automated test and do it before, NOT after code is rolled to production. It may sound counter intuitive and I did get a lot of questions about the ROI of doing this. However we learnt it the hard way that human has limited ability to keep track of changes, and we got bitten multiple times for missing features because we only rely on manual work. Nothing prevents bugs better than automated tests and continuous integration. Don’t let your customers experience death by a thousand cuts caused by un-fixed defects. The more bugs you have, the harder and longer it is to fix them. For A/B test features make a very conscious decision, when to automate. This depends on your business requirement, product stability and the goal. At the very least, when you can, cover your P1 scenarios. Last but not least, try to build any product right since the beginning. Dirty and quick approach may work in the beginning to grab market opportunity, but it’s not always sustainable. So know your timing and and manage your debts wisely
- Sheehan: Thank you Farida. Why did you decide to go with Applause for crowd testing – did you look at a variety of options or did you only look at crowdtesting? I’m just getting started on building out our automation framework, what would your advice be for getting started? Did you have pushback from your team or other groups when you started this new QA approach? If so, how did you overcome it? Lets turn it over to Sara for questions.