How to introduce Apache Pulsar into your organization successfully - Devin Bost
•Download as PPTX, PDF•
0 likes•103 views
This presentation covers principles and best practices based on significant research regarding how to maximize the likelihood of success with any technological adoption or revolution in your company, especially involving Apache Pulsar.
More Related Content
Similar to How to introduce Apache Pulsar into your organization successfully - Devin Bost
Similar to How to introduce Apache Pulsar into your organization successfully - Devin Bost (20)
More from Devin Bost
More from Devin Bost (6)
Recently uploaded
Recently uploaded (20)
How to introduce Apache Pulsar into your organization successfully - Devin Bost
- 1. How to introduce Pulsar into your organization (successfully) Email: dbost@overstock.com Twitter: DevinBost LinkedIn: https://www.linkedin.com/in/devinbost/
- 2. Step 1: Think like an entrepreneur
- 3. The beginning is always to nail the problem
- 4. That means finding a true pain point
- 5. Problem discovery happens in steps.
- 6. Finding the right solution requires deep exploration of the problem
- 8. Finding black swans requires building empathy through “active listening”
- 9. Make time every day to develop your communication skills and business savvy
- 10. See my recommended reading list at the end of this presentation Easy way to do this: Listen to audiobooks whenever you’re driving
- 11. Example:
- 15. Scenario 2:
- 16. Once you’ve nailed the problem, rapidly iterate towards a solution
- 17. Ensure your agile processes don’t operate in a vacuum!
- 18. Fire bullets before you fire the cannonball
- 19. Firing uncalibrated cannonballs will burn up your runway
- 21. The first step to firing a bullet is to aim
- 23. ?
- 24. If z-score < -2, then add studentID to studentsAtRisk Aggregate scores by school by module Enrich individual scores with school averages Compute Z-score
- 26. Don’t boil the ocean on Day 1
- 29. On the flip side, it’s possible to hard-code yourself in a way that kills your agility.
- 34. Leverage the Helm charts! Or use a managed service!
- 35. Build solid alerting systems with Grafana/Prometheus
- 37. Build success stories and share them
- 38. Stay disciplined
- 39. Recommended reading: Building a great tech company: • Nail It then Scale It • Good to Great • Great by Choice • Crossing the Chasm • Inside the Tornado • Enterprise Architecture as Strategy • Measure What Matters • Start With Why • The Four Steps to the Epiphany Communication / Persuasion: • Never Split The Difference (especially the chapters on building empathy) • Influence • Switch Email: dbost@overstock.com Twitter: DevinBost LinkedIn: https://www.linkedin.com/in/devinbost/
Editor's Notes
- In this presentation, I’m going to focus primarily on talking about principles that you can utilize. They’re especially helpful with Pulsar, but they can be applied to many other technologies or situations.
- When introducing something new to your organization that you believe is technologically superior or solves a critical business problem, you need to think like an entrepreneur mini-startup going out of business = getting fired
- You need to nail the problem to build the right solution
- Look for the shark bite problem, not a mosquito bite problem. If the problem is just annoying, people won’t have the motivation to risk moving to a new solution. You want to ensure your implementation will be adopted. If you build a platform that nobody uses, that will do you no good. To get adoption, you need to cure a major pain.
- You want to get a deep level of understanding to really find the underlying pain and the root cause. Otherwise, it’s easy for assumptions to cloud your success and cause you to miss the target.
- The goal is to discover the true underlying pain point. Once upon a time, people didn’t know that black swans exited. The idea of finding a black swan is about finding the hidden truths counter to expectations that influence decisions or the success of an implementation of a solution. If you can find these, you can build a much more effective solution. The trick is learning how to find them.
- The information about black swans is stored in people, not systems. You can't just dig through code and expect to find a black swan. So, we need to take a people approach to information gathering. Active listening is about repeating back what you’ve heard to validate the speaker. This helps the speaker know that you understand what they’ve said so far, which builds trust. It also allows you to test your understanding. It’s like Test Driven Development for communication. It also helps the speaker know they can move into deeper content, which can reveal critical information. They might not even realize that information is so critical.
- Remember, if you’re going to be a leader in creating a micro-startup in your company, you need to develop the skills of a successful entrepreneur, and that means improving your communication! This can be hard for a programmer, but it’s worth it. It will pay off by saving you time that otherwise would be wasted on building the wrong things. It will help you focus your time on what really matters the most.
- At the end of this presentation, I’ll give you a list of books that are available in hard copy or on audiobook. Find time to study them. PRACTICE The skills you learn Every day Then it will become more natural and automatic even if you spend 30 seconds a day
- Example of information gathering process You’re the doctor in this situation.
- We start with level 1 data. We get level 2 data.
- Level 3 data
- Level 4 data. What if we just stopped at the level 1 course of action (pain killers and time)? We could have lost the opportunity to obtain the ideal course of action.
- At this point, do we really need to engineer a Kevlar thorn-proof shoe? Probably not. It’s better to look for a bigger problem to solve.
- The goal is to get quick feedback to identify how well it solves the problem. When you iterate, keep in mind that you should be iterating towards A solution, not THE solution. Discovery should be a process of refinement and improvement. QUICKLY PIVOT which requires humility
- Customer feedback is critical. Most successful companies work very closely with their customers Agile teams can sometimes quickly iterate on their own ideas without getting enough outside involvement in the process. That can make them blind to oversights At this stage, that’s a big mistake! Be sure you’re working closely with other teams to validate and refine quickly. working towards curing the underlying pain
- Bullets are low cost, low risk, and low distraction initiatives/project. For example, using a Pulsar function in a new app is probably a bullet whereas a cannonball might be a bulk migration of a thousand Kafka apps to Pulsar. I’m not saying don’t fire cannonballs. Just make sure you fire bullets first. The reason is because bullets calibrate your aim and allow you to perfect your delivery. If you fire your cannonball before EMPIRICAL VALIDATION of your solution, you’re going to miss! People get so impatient, they miss the mark, put all their resources into something (which makes it a cannonball) and ruin their opportunity. Some companies have a culture of firing cannonballs before firing bullets.
- When cannonballs are fired before bullets, the likelihood of failure is MUCH greater! Since Pulsar can bring a lot of benefit to your organization if used right, you don’t want to miss your shot. You don't want them to say, "Well, Pulsar was a failure, so we're not doing that" Due to curing wrong pain or curing in the wrong way
- It’s a big mistake to build a car to test your experimental engine design when you haven’t yet validated your engine! Start small and grow from there
- When you aim, you’re lining up your aperture, or source, with your target, or your sink.
- In a stream messaging context, that looks like this. You look at your source data (CLICK) and your destination (CLICK). [Data into API or web service] The challenge is to line them up to get the payload across (CLICK). If it turns out that your source is missing critical data, then you either need to update the producer (CLICK) to send the missing data or use stream enrichment to add the data to the message within the flow.
- From there, the engineering task is to determine what transformations must happen in the data to get from your source to destination. (CLICK) In this case, we first identify that to compute students at risk, we must compute a Z-score that compares a student’s performance to their school averages. (CLICK) That tells us we need to have the school averages and standard deviations to get the Z-score. (CLICK) In order to do that, we determine that we need to compute the school statistics, (CLICK) which are derived from the original scores (CLICK), and then we need to enrich incoming messages with these summary statistics. At this point, we can determine that Pulsar functions are a good fit for the simple transformations (CLICK) and pulsar topics can glue all the connections, and depending on our other requirements (such as latency), we can identify what technologies will work best for the stateful compute tasks and build Pulsar connectors to bridge the gaps or use out of the box connectors.
- By modeling flows based on the data first, you will have a much easier time reaching agreements with other developers and teams. (Avoids contention around Pulsar vs Kafka) It also ensures that glaring issues become obvious more quickly.
- Boiling the ocean is equivalent to firing uncalibrated cannonballs. Gaining great success with your first solution will allow you to gain synergy and momentum, so it's important to keep it small to ensure your bullet nails the target. If you're firing bullets and need to pivot, it's not a big deal; you can try again.
- Automation is great, but until you’ve calibrated your delivery by firing bullets, you can sink all your time into optimizing the wrong things. MODEL YOUR DATA FLOW FIRST after you've identified the black swan or the underlying pain of a major problem
- Webvan is a famous dot com startups that pioneered home grocery delivery. They thought through every aspect of the delivery process, logistics and fulfillment, of warehouse optimization, and built the most advanced robotic-driven warehouse automation that was ever invented with custom algorithms for every step, with advanced tracking and inventory management. But, they ran out of money. $1.2 billion to be precise. They focused on the wrong things and couldn’t get enough business to make it sustain.
- On the flip side, you can very easily go too far down the path of not thinking about scalability. This is probably the more common trap people fall into. Failing to think about scalability of developer time can suck all the time right out of a team. THINK OF SELF-SERVICE and generalizability Look for patterns
- The first key is to focus on building automation after you’ve calibrated your delivery by firing bullets. If you automate after you calibrate, your operations will scale as new demands are placed upon you. It allows you to automate the right things and not get distracted.
- If you can anticipate a clear problem in your ability to scale, the key is to find a way to incrementally automate or create a path that will give you a roadmap of how you can immediately deliver while creating a clear plan of action to enable you to scale as you move forward.
- Build the interfaces and then incrementally automate what’s in the middle. When you find yourself spending too much time on the busy work, invest some time into automating those pain points in your operations.
- Pulsar has fantastic APIs. Be sure to tap into them!
- Unless you need the latency of bare metal, the Helm charts will make your lives easier. If you’re not at that scale yet, using a managed service is the best way to get off the ground quickly. So, make use of the resources that are available!
- Pulsar also has nice integration with Grafana, so make sure you’re making use of it. You can easily build automation around Grafana to make it more self-service as you identify the most common needs of your tenants so you're not building things that people won't use.
- Most of you have cloud support available, either through GCP, AWS, or Azure. Or, if you have a managed Pulsar provider, get their help! At a previous company, we were using AWS, and I literally created a support ticket for every piece of the flow I needed to build. My boss said I got a year’s worth of work done in 2 weeks. What he didn’t realize is that I had a team of experts working to help me accomplish the required tasks. By standing on their shoulders, you can do more with less.
- Testimonies inside your organization are a thousand times more valuable than those from outside the company. The flip side is also true. If you have a loud spoken adversary inside your company, they can do 1000 times more damage than such a person outside the company. This is why getting that initial delivery right is so important. You want to obtain empirical validation of your platform. By firing bullets before cannonballs, you will build successes incrementally. Don't let it go unnoticed. Let your light shine.
- Keep your focus on what really matters most. DISCIPLINE IS A KEY CHARACTERISTIC OF TOP PERFORMING COMPANIES. The most successful companies are the ones that stay disciplined.
- Screenshot this slide. These skills will help you be more effective even if you're purely a developer and not in management. Of course, if you're in management, then you really need to learn these skills. And that's all I got. Questions?