Downloaded 96 times
Uploaded byTasktop
Flow Metrics: What They Are & Why You Need Them
The document discusses flow metrics, which are essential for measuring speed, productivity, efficiency, and capacity in business processes, emphasizing their connection to business value and decision-making. Key metrics include flow time, flow velocity, flow efficiency, flow load, and change failure rate, which provide insights into work performance and identify areas for improvement. It stresses the importance of measuring these factors to manage workflows effectively and enhance organizational productivity.
More Related Content
Similar to Flow Metrics: What They Are & Why You Need Them
![[Talk] Manage flow - Metrics and Analytics for predictability and flow](https://cdn.slidesharecdn.com/ss_thumbnails/lkna-manageflow-metricsandanalyticsforpredictabilityandflow-190527041051-thumbnail.jpg?width=640&height=640&fit=bounds)
Flow Metrics: What They Are & Why You Need Them
- 1. ddegrandis.com @dominicad Flow Metrics: Whatthey are & why you need them Dominica DeGrandis Director, Digital Transformation
- 2. ddegrandis.com @dominicad • Flowmetrics are tied to business value • Flow metrics are based on outcomes • Flow metrics provide a feedback loop to improve decisions WHAT’s THE POINT?
- 3. ddegrandis.com @dominicad 1. FlowTime – a measure of speed Flow Time: The duration from when work enters the value stream to its completion.
- 4. ddegrandis.com @dominicad 90th percentilefiltered on business requests Measuring Flow Time helps with predictability date Percentiles answers Q: “What’s the probability of completing work in x days?” Inspired by Troy Magennis http://focusedobjective.com/team-metrics-right/
- 5. ddegrandis.com @dominicad Look atFlow time How fast? Flow Time date Unplanned work delays Planned work Inspired by Troy Magennis http://focusedobjective.com/team-metrics-right/ . What we measure impacts people b/c people value what is measured.
- 6. ddegrandis.com @dominicad date 2.Flow Velocity – a measure of productivity Flow Velocity: The number of items completed during a given time frame.
- 7. ddegrandis.com @dominicad date FVtrend over time provides data for teams to see delivery rates Question: Does Flow Velocity improve with faster Flow Time?
- 8. ddegrandis.com @dominicad Dominica DeGrandis ThiefUnplanned Work Transaction costs: Low for a one-time 6 month supply High for a one day supply. Knowledge work is perishable Consider Reducing Batch Size
- 9. ddegrandis.com @dominicad Dominica DeGrandis ThiefUnplanned Work Unplanned Work: Interruptions that prevent you from finishing something or from stopping at a better breaking point. Unplanned Work is a time thief b/c unplanned work usurps planned work @dominicad While economies of scale can reduce costs in manufacturing, software is a different story. Two things to consider: • Transaction cost • Holding cost
- 10. ddegrandis.com @dominicad 3. FlowEfficiency - a measure to expose wait time Flow Efficiency: The percentage of time where work is in an active state vs. a wait state.
- 11. ddegrandis.com @dominicad “Ifyou measure anything, measure wait time.”
- 12.
- 13. ddegrandis.com @dominicad “Things taketoo long” is a universal problem work request How many get work intake requests via email? Disconnects in communication interferes with collaboration & delays delivery
- 14. ddegrandis.com @dominicad 4. FlowLoad – a measure to balance demand & capacity Flow Load: All the partially completed work. All the work-in- progress (WIP) in the value stream
- 15. ddegrandis.com @dominicad Dominica DeGrandis WIPisa leading indicator
- 16. ddegrandis.com @dominicad Dominica DeGrandis ThiefToo much Work-in-progress (WIP) High WIP means that other items sit waiting for service longer. The single most important factor that affects queue size is capacity utilization.
- 17. ddegrandis.com @dominicad Dominica DeGrandis ThiefToo much Work-in-progress (WIP) Queuing Theory allows us to quantify the relationship between wait times and capacity utilization. Wait times increase exponentially as utilization approaches 100%. Queuing Theory: Applied statistics that studies waiting lines If the goal is speed, consider managing work by queues. http://reinertsenassociates.com/books/
- 18. ddegrandis.com @dominicad Dominica DeGrandis Projectsand tasks that compete with each other. Conflicting Priorities is a time thief b/c people take on too much WIP when priorities are unclear. • Visualize WIP • WIP limit as enabling constraint • Primary factor of speed is the amount of WIP Set WIP Limits to balance demand against capacity
- 19. ddegrandis.com @dominicad 5. FlowDistribution: a measure to make tradeoffs clear A decision to do one thing is a decision to delay something else. The proportion of work items in a value stream, adjusted depending on the need to maximize business value.
- 20. ddegrandis.com @dominicad Flow Distributionallocation example (3) (1) (5) (5) WIP limit
- 21. ddegrandis.com @dominicad Sample ValueStream Dashboard Project to Product: How Value Stream Networks Will Transform IT & Business (Mik Kersten) https://itrevolution.com/book/project-to-product/ https://go.tasktop.com/DevOps-Enterprise-Summit-Las-Vegas-Speaker-Presentations.html
- 22. ddegrandis.com @dominicad What wemeasure impacts people because people value what is measured.
- 23. ddegrandis.com @dominicad 6. Ametric to gage happiness “How likely are you to recommend working for this company to a friend?” Safety Example Q’s: • On my team, failure causes inquiry and not blame. • Our leadership is open to hearing bad news. • In my org, failures are learning opportunities and messengers are not punished. @nicolefv https://www.youtube.com/watch?v=avauW5FAWCw promoterspassivesdetractors
- 24. ddegrandis.com @dominicad 7. TheAging Report
- 25. ddegrandis.com @dominicad 7. ChangeFailure Rate (CFR) Answers the Q: What % of changes to prod fail? CFR = # of failures / total # of work items completed Ex: 60 items completed in Dec, 20 of them resulted in a defect. Dec CFR is 33 %. Failure - a change resulting in an outage or degraded service where hotfix, rollback or patch required.
- 26.
- 27. ddegrandis.com @dominicad Change FailureRate – a measure of Quality Change Failure Rate # Failures # total done items date Oh - ok – I see what you mean!!! What we measure impacts people b/c people value what is measured.
- 28. ddegrandis.com @dominicad 53,308 securityincidents, 2,216 data breaches, 2,216 https://www.verizonenterprise.com/resources/report s/rp_DBIR_2018_Report_execsummary_en_xg.pdf Allocate capacity for security
- 29. ddegrandis.com @dominicad WHAT’s THEPOINT AGAIN? • Flow metrics are tied to business value • Flow metrics are based on outcomes • Flow metrics provide a feedback loop to improve decisions
- 30. ddegrandis.com @dominicad MIK KERSTEN CEO& Co-Founder Tasktop SUZETTE JOHNSON Agile Center of Excellence Lead Northrop Grumman ROBIN YEMAN Lockheed Martin Fellow Lockheed Martin MARK FORD Lead Developer, TIAA Community Director, Vivit KRISTEN BIDDULPH Certified Agile Leader Nationwide Insurance KYLE WILLIAMS Manager Deloitte DAVE WEST CEO Scrum.org DOMINICA DEGRANDIS Director, Digital Transformation Tasktop JEFF ZAHORCHAK Manager, Enterprise Applications Select Medical NICOLE BRYAN VP, Product Tasktop FABIO ECHAVARRIA Senior Consultant Deloitte CARMEN DEARDO Senior VSM Strategist Tasktop Register at: connect.tasktop.com
- 31. ddegrandis.com @dominicad Email: dominica@SendYourSlides.com Subject:flow To receive: • copy of this presentation deck • excerpts of Making Work Visible • Tasktop video on JIRA/SN tool integration • Forrester article: Agile-Plus-DevOps With Value Stream Management • IT Rev paper: Overcoming inefficiencies in work mgmt. systems
Editor's Notes
- #2 Today – going to describe, explain, clarify 5 flow metrics – what they are, why they matter, how you measure them, how you game them and what benefits they can offer you. _______________________ When it comes to assessing an IT transformation (such as Agile and DevOps), performance metrics have come under intense scrutiny. Traditional performance metrics, such as counting the number of lines of code and the number of software bugs should be used with caution, because there are bugs that are not worth fixing and code that is not worth maintaining. These old-school performance metrics represent activities, not outcomes. To visualize and optimize the business value of your software delivery, you need to find a way to measure business outcomes. To do that, we need flow metrics. During this webinar, Dominica DeGrandis will present five key flow metrics that reveal trends on desirable business outcomes – such as faster time-to-market, responsiveness to customers, and predictable release timeframes – and explain how to implement them at your organization to measure and improve the impact and value of software products on your business.
- #3 We have no lack of data collection & info visualization approaches at our disposal. The problem is that at the biz level, there’s no compelling set of abstractions for what to visualize. Contrast this avail telemetry w/ teams practicing DevOps – they know their MTTR & Chg success rates that helps them to identify opportunities to improve. Look at dev teams, using scrum/kanban boards – they can make their WIP visible across the entire team, to help them see if teams are overloaded or not. It’s the biz-level visibility that orgs lack. This is what flow metrics provide, b/c tied to biz value. Flow metrics bring visibility to outcomes. Ex: Oh - only feature work got done last month – didn’t fix any tech debt. Oh - and that feature was delivered 1 week later than big customer expected. Outcome-based metrics provide a feedback loop to improve decisions. Should we allocate more capacity to fix tech debt next month? Or should we continue to allocate teams to do more features? Don’t know – depends on the strategy and context of the org. But these are the kinds of important Q’s that Flow metrics provoke. IT can provide biz leaders w/ metrics to better understand the tradeoffs from decisions they make. If we go another release w/o fixing any tech tebt, what might that do to the happiness factor of our employees? It’s important to note that Flow metrics use language that both IT and biz ppl can understand. TRANS: Like time – “When can big customer use this feature?” That’s a speed Q. Which brings us to our 1st flow metric – flow time. __________________
- #4 Flow time is a measure of speed. Flow time is the duration, the elapsed time it takes from when work enters the VS to when it’s considered done, which might be diff between different VSs. 1 team might consider something done when it’s delivered to the customer. Another team might not consider it done until the value derived from the change is determined 3 months later. It depends. This viz shows 3 diff types of speed measures, DLT, FL, LT. LT is measured from when a customer 1st made a request. This is the time customers care abt. DLT is measured from code commit to done – this is the time metric you’ll find in the Accelerate book by Nicole Forsgren and company. A good measure to see how your delivery pipeline automation is improving (or not). It measures the right side of the VS. FT is a term I started using 3-4 yrs ago? I didn’t make it up – it’s been around since John Little proofed out Little’s Law, in 1960’s. I just got tired of hearing people go to war over twitter & at the Lean kanban conferences on CT & LT. CT is an ambiguous term. Has multiple definitions depending on context (manufacturing or software). So, in my workshops while helping ppl MWV, find bottlenecks & optimize their workflow, I thought, just start the clock where it makes sense to in your context. This is your VS – at what point do you want to the clock to start? Learned that many Product Owners /biz ppl were interested in how long it took to do something once they decided that it would be a good thing to do. Which is what you see in this illustration – Once ppl decide, Yes, let’s do this! So, that’s why I started using Flow Time when talking abt speed metrics in general, b/c it starts the discussion on what timeframe is most important to measure for your org. Also, Flow is attuned with Lean & kanban –I’m a big kanban fan. Flow is a pillar of Lean. Define FLOW: Flow is value pulled thru a VS smoothly & predictably. Flow is 1st way of DevOps “DevOps relies on bodies of knowledge from Lean, Theory of Constraints (study of bottlenecks) Enough abt what Flow time is. Why do we measure it? __________________________ “DevOps relies on bodies of knowledge from Lean, Theory of Constraints, the Toyota Production System, resilience engineering, learning organizations, & human factors.”
- #5 For a number of reasons. 1. B/c ppl complain that things take too long. If want to improve that, then measure to see how long, and if indeed way too long, measuring Flow time trends over time shows you if things are improving (or not). It’s useful to test opinions against data. This is a run time chart used to capture flow time. Vertical axis is # of days to complete a work item. Horizontal axis is date work completed. 4 week ins Sep. Legend on right. The charts shows flow time of rev gen work – features completed. Also shows which items generated some failure – call that failure demand. Can see that the 1st two weeks of Sept, work completed in 10 days or less. Then in week 3, FT increases to 15 days. Wk 4 FT jumps up to 20 days. So seeing this data, should prompt discussion on why things took longer to complete toward end of the month. We don’t know why from looking at this data – but it’s prompts the Q, “What happened?” to start learning more. It also helps set expectations for how long things might take next month. This small blue dots show the 90th percentile. In other words, based on this Sep data, 90% of the time (9 times out of 10) feature requests take no longer than 20 days. If you use the 50th percentile, you get the avg, which makes your estimates wrong – at least 50% of the time. Book called “The Flaw of averages” – opening cartoon is picture of 6 ft tall statistician drowning in lake with an average depth of 3 ft. Percentiles answer Q…… Well, FT metrics shows you just that. Depending on your tool functionality, configure settings or filter on different percentiles to help you be more predictable. Trying to be approx. right instead of exactly wrong. TRANS: We don’t have the whole story here. There’s something else going on. ______________________________ Focus on capabilities, not maturity levels Ppl said things take too long – useful when ppl complain [..] As bld engr, used to rant, no auto testing! But then measured CT. Blew me away. Ppl listened, budget, headcount, empathy,
- #6 And that’s unplanned work (yellow trianges) & revenue protection work dots inside circles (security vulnerabilities, patches, maintenance, KLO) If 3 emergencies pop-up 1 week, then planned work likely delayed. Thief Unplanned work sneaks in and steals time away from planned work – purple arrow. If tracking unplanned work, then it will show up in this chart. Keep in mind that “What we measure impacts ppl, b/c…..” Tell me how you measure me…. Merit review is based on LT target. Trying to game metrics by excluding weekends. How gamed? 1. Exclude weekends 2. Only pull small simple work items into WIP. Have discussions with your teams on how ppl will game your metrics – useful, interesting exercise to bring transparency to the reality of the situation. To sum up FT benefits, b4 moving on, FT helps ppl understand why work was delayed, can see what else competes w/ planned work. Helps ppl understand time-to-market for completing items, and look for ways to reduce unnecessary steps & waste. TRANS: This chart btw is part of an exercise I created and will give you a link to. It wasInspired by Troy Magennis/Larry Maccherone, “Doing Team Metrics Right,” Focused Objective blog
- #7 TP – how measured, why measured (productivity measure) This is your TP. Answers Q , “How many WI’s got done?” TP totaled at the top of the chart in green. Here 7 WI’s finished 1st week, 3 on wk 2, 3 on wk 3 and 4 on wk 4. Why do we measure it? TRANS: Tracking FV trend over time provides historical data for teams to see delivery rates. This can help teams improve estimates/forecasting how much work they can deliver.
- #8 FV gets gamed by breaking down work into smaller bits – which may not be a bad thing – show you why in a min. You can experiment w/ the relationship between FT & FV. Q…””? Or does FV diminish when FT increases? If capturing & visualizing these flow metrics, then can see possible correlation. We only delivered 3 feature work items this week and it took 5 days longer than previous week. It’s not so easy to detect this kind of info in a spreadsheet. Leave the spreadsheets to the accountants and start making your metrics visible in a way that is easy for the human eye to comprehend. TRANS: Here’s why breaking down work items into smaller bits may not be a problem. In fact, why you may want to consider reducing batch size. ______________________________ Bringing visibility of problems & risks to the table provokes necessary conversations that enable change.
- #9 Imagine shopping for bananas. If you buy 6 mon supply of bananas, transaction cost is low, but most of the bananas will be rotten within 10 days, so waste money. If you buy 1-day supply of bananas, they won’t rot, but transaction costs are high b/c have to go grocery store every day. TRANS: Somewhere in between is the right batch size.
- #10 2 things to consider. Transaction costs & holding costs. In knowledge work, holding costs go up w/ lg batch sizes b/c delays fr dependencies, late feedback, code merges. Lg deployments have high holding costs b/c software not realizing value yet. Corn flakes still sitting on the shelf. It’s diff w/ manufacturing. Transaction costs go down w/ large batch-size projects like book publishing – don’t print just 1 book. Assumptions abt economies of scale rarely apply to software dev. TRANS: A small quick deploy goes a long way toward a speedy recovery, which is why managing by VS is the better way to go. Project oriented management models incentivize lg batch size b/c of the capitalization of software development.
- #11 How measured – see equation Why we measure it - b/c helps team see just how much wait time in the system & make decisions to do something abt it. Much of flow time is simply wait time. You are lucky if your FE is > 15%. 5% is common. Often, the emphasis is on estimating how long things will take – assuming no interruptions or dependencies..
- #12 Flow efficiency needs wait states in your workflow design – otherwise very hard to track them. For ex: walk thru this board…..backlog (options), might do them, might not – 1 reason start FT clock later – b/c might not ever get to stuff in the backlog. Then investigate – How much design capacity there is?, what’s the capacity of design team? prioritize it against the other 100 requests, before it moves into Dev. Whatever the states in your VS, FE requires you to identify when work sits idle – waiting. This is a challenge for a lot of tools – except for real kanban tools. _____________________________________ In Viz – in jira – if take a jira item, if card is in new or in progress, tt will record that. In Viz, can say that something is an active state. We can map their wait states to Viz. In Hub, would need to do some ETL to do the same thing. Same as LK functionality – where wait states are aggregated to calc FE
- #13 Some tools allow you to set a work item to blocked, but it throws all blocked items into one queue to the right of the board, so you lose visibility of which state the work was in when it got blocked. It’s better than nothing, at least ppl know work is in a wait state. “Blocked” word reaction…. If wait state is one common area – then can’t measure individual start & finish dates for each queue. (CT). So – consider when you design your kanban boards – Where is your bottleneck – it will likely have a big wait queue in front of it – but if all the work in waiting is moved to the right, then diminishes the benefit of visualizing the bottleneck. Benefit of FE: helps team see the wait in the system & where work get stuck. Can visualize bottlenecks and so you can do something abt it. And don’t fret if you or your friend are the bottleneck – you can begin to get the help you need. When I managed the build & release team at Corbis, we didn’t have automated testing and it took a long time to do manual smoke tests – especially when functionality was broken. Once I shared metrics with mgmt., I got budget & headcount.
- #14 The handoffs aren’t as clear or clean as they should be. Details lost in email. “impacts communication & collaboration & delays delivery. Things take too long is a problem every company grumbles abt.
- #15 How measured – All the partially completed work – all work in system, in paper post-it notes, unplanned work. Why measured? – WIP is the primary factor of all speed metrics. FT = WIP /TP (FV) – little’s btw. WIP is a leading indicator. _____________________ WIP balance of demand vs. capacity
- #16 MTTR, velocity LT, CT, TP – all trailing indicators. High WIP levels means that other work items sit waiting for service longer, like this freeway.
- #17 Only so many vehicles fit on the road. Freeways clog up & slow way down when at full capacity. The more cars, the longer the commute. Same thing with Computers - stop responding.. Loading ppl to 100% capacity.. The higher the utilization, the longer the wait – esp in areas of high variability areas - like IT – so much unplanned work, and conflicting priorities and dependencies – all the time thieves. TRANS: THE SINGLE MOST IMPORTANT Factor that affects wait time (queue size) is WIP.
- #18 When we look at the math, can see that wait times increase exponentially when utilization approaches 100%. QT is what allows us to quantify relationship between [..] Something that Gantt charts aren’t especially good at, b/c don’t consider the wait & blocked times that occur when ppl are allocated at high levels. Keeping ppl busy all the time isn’t efficient or effective – b/c CP - ppl overbooked. TRANS: This is the problem with high utilization levels —important work takes longer & flow time increases because people aren't avail. Teams rarely work in isolation. If a skillset is needed from a different team, then hand-offs between the two teams occur. A developer wonders, “Are there unknown vulnerabilities in this code?” while waiting on feedback from a security expert. But the security expert is busy discovering how someone hacked into the now unsecure database. A question waits on input from a database architect: “Is the data in the test environment wrong? Can they please check it out?” But the database architect is busy helping the security expert. When you are the only one on the team with a special skill set, you can be a bottleneck pulled in many conflicting directions. Expert skills in high demand are often unavailable when you need them.
- #19 When look at work like this, can see million incoming requests. Can see competing work items - can see security vulnerability fix not done. How WIP is gamed – work done under the table – invisible work – really not done cause going to finish it over the weekend. Benefit – WIP limit is enabling constraint.
- #20 How measured, Why measured – balance necessary work. A decision to do one thing is a decision to delay…. Benefit can begin to see how work is allocated. How gamed – see “More on Flow distribution” Depending on context, may want to chg it up. If you do more feature work, you can’t expect that it won’t take away from doing risk work. Biz and IT need to understand this. TRANS: If categorize work items, can visualize them in progress, use WIP limits to allocate capacity. Make tradeoffs clear – set strategic direction with Flow Distribution
- #21 Could show diff WIP by work type distribution In this example, WIP limits are used to allocated capacity to fix debts. Can have 3 team improvements and where on the board. If team has capacity to work on 14 things at a time, and situation prioritizes fixing debt & defects, then feature & risk work should be reduced. If keep working on features, then you put defects and debt work at risk. TRANS: A decision to one thing is a decision to delay something else. Help others see the impacts of conflicting priorities across Team/org. ___________ Diff schools of thought on def of defect
- #22 In my experience, metrics have been invaluable to get leadership agreement & buy-in, I used to rant. Then I showed metrics and got budget and headcount. If you want to convince others suffering from misconception, useful to test their opinion against the data. Ppl listen to data – it helps with credibility. They help you become the voice of reason in the org. Someone is going to influence boss.
- #23 If measure an activity metrics like LoC, then ppl write a lot of code, when maybe a 10-line solution is better than 1000-lines. Ideally, we should reward devs for solving biz probs w/ min amt of code – or no code! Rewarding devs for writing LoC leads to bloated software that incurs high maint costs & high costs to change.
- #24 The standard statement for NPS is normally, “How likely are you to recommend working for this company to a friend?” We use it measure employee happiness at Tasktop. Not customer happiness, but employee happiness. Also used from a DevOps perspective to measure safety Examples: On my team, failure causes inquiry and not blame. Our leadership is open to hearing bad news. In my org, failures are learning opportunities and messengers are not punished. People on our team trust one another
- #25 Query your ticketing tool or work flow process system – show all …for 30 days Turn this into leading indicator by comparing age of work with avg age of that work type distribution
- #26 Diff schools of thought on def of defect. Why “Failure”. Quality is difficult to measure. CFR takes a decent stab at it. It answers Q… It’s a ratio – calculated by dividing # of failures by the total number of work items completed. Failure defined as a change that caused an outage where hotfix, rollback or patch needed, or some degraded service. In this example [..] TRANS: CFR matters b/c DevOps is shooting for “world-class quality”.
- #27 Here are some failures on the right side of the screen. Might recognize some of these companies. For ea event celebrated in the press, 100’s more that we don’t hear abt. It’s not surprising that your systems sometimes fails. Dr. Richard Cook (works in area of resilience engineering, say’s that it’s surprising our systems work at all. TRANS: Failures are inevitable.
- #28 Easy to calc once you have TP. Note failure demand in pink. FD is work done b/c of error (degrated service, SQL server went down) Now, this begins to tell story. CFR trending up, TP up, FT increasing? The experiment provides link between seeing & relating that opens the door to new understanding. Looking at this diagram is less threatening – “Oh, I see where you’re coming from” The data becomes the thing to change, not “Them”. TRANS: Licensed under Creative Commons Attribution share alike. .
- #29 complicated threat landscape Medical info worth > credit card #s b/c can buy pharmaceuticals, make claims, takes time to discover…
- #30 "Teams are not paid to deliver software, they are paid to deliver impact from which value can be derived." https://hamletarable.com/2018/10/06/impactmetrics/ … @hamletarable Paul Shepheard Flow metrics bring visibility to biz outcomes. Are we more predictable this qtr than last qtr? What’s the probability of delivering a new feature when needed? There’s a desire to look at real biz outcomes over activity based metrics such as # LoC Biz ppl don’t care abt Story points or containers or kubernetes. They care abt – did customers get the changes they needed when it was promised to them? Is this change going to generate revenue or protect revenue? How can we be more predictable?
- #31 Tasktop Connect is the best place to connect with software delivery pros. Speaker lineup includes Lockheed Martin, Northrup Grunman, Nationwide, TIAA, Deloitte, Select Medical, Scrum.org and Tasktop. Tasktop Connect 2018 - Tasktop's 2nd annual conference on Thursday, Dec 6th at the Knight Conference Center at the Newseum in Washington DC. There will be talks from Tasktop customers (Lockheed Martin, Nationwide, Select Medical, TIAA, and others) on their transformations - how they scale initiatives across their value streams. Other talks: Mik on Flow Framework Carmen on how to Apply Flow Framework to your Value Streams Yours truly on Flow Metrics. It's a way to meet & talk with other enterprise IT leaders, and we'd love to see you there.