Recommended
Recommended
More Related Content
Similar to Integrating AI into Your Startup's Workflow
Similar to Integrating AI into Your Startup's Workflow (20)
Recently uploaded
Recently uploaded (20)
Integrating AI into Your Startup's Workflow
- 1. FROM NEURONS TO A BRAIN: INTEGRATING AI INTO YOUR STARTUP’S WORKFLOW
- 2. I am Emily Dresner I am the CTO of Upside Travel. You can find me at @multiplexer.
- 3. ▪ AI-powered Business Travel Startup in Washington, DC. ▪ Travel is a huge ML opportunity and a big data business. ▫ Shopping and Booking ▫ Aggregated Itineraries ▫ Fare Construction ▪ Millions to billions of itineraries daily.
- 4. Our goal is to learn from real people doing real work on our platform.
- 5. ▪ Air and Hotel Sort Algorithms ▪ Recommendation Engines ▪ Personalization ▪ Proactively Detect, Alert and Adjust ▪ Optimum Pricing both for Customer and for Upside (steering) ▪ Margin Calculations ▪ Fare Construction ▪ Upsell/Cross-sell We are in constant tension between supply, profitability and our customer. We want to make sure everyone is somewhat happy. It’s realistic ML.
- 6. ▪ Machine Learning Engineering - Shipping Models, Auto-Training models, A/B testing ▪ Data Pipelining and the ETL - Make Data Useable and Useful ▪ Product Management - Understanding the Models and Turning them into Product that Delights and Finds “Model-Market Fit."
- 7. A system to derive hypotheses, run Machine Learning experiments, and learn quickly with live customers without negatively impacting the product.
- 8. DevOps/Data Fully automated CI/CD infrastructure. ML model training pipeline and deployment fully automated. ETL’d Tracks and travel itinerary data flows automatically into data warehouse for continuous training. (New!) Experiment data mixes with track data for end to end product analysis. Data Science Hypothesis construction. Data selection for research and development. Algorithm development, model training and data analysis. Extracts meaning from experimental results. Communication with visualization (Looker) Product Engineering Builds tools and services for deploying models to labs production. Builds data pipelines and schemas for the collection and extraction of data. Implements experiments in the product flow. Brings successful experiments to the product and customers.
- 10. Hypothesis Idea! Standard Intake Form Wiki Page • Testable Hypothesis • Requirements • Launch Checklist • Priori Work Booking Pre-Trip On-Trip Post-Trip Total Trip Profit Product Process Goal: Capture every idea we can and categorize it at the intersection of the UX Journey x Product Theme to help better understand stakeholders and visibility.
- 11. Fields in a Concept Document Customer Hypothesis Technical Hypothesis Target Audience MVP Description Requirements – Strategic Fit Requirements – Experimental Design Measurements and Metrics Tasks Prototyping -> Testing Checklist Polish (if needed)
- 12. Categorized Ideas Booking Pre-Trip On-Trip Post-Trip Total Trip Profit Product Concept Document Ideation + Backlog Prototyping Testing Review Measurement
- 13. Labs Process Categorized Ideas Booking Pre-Trip On-Trip Post-Trip Total Trip Profit Product Concept Document Ideation + Backlog Prototyping Testing Review Measurement Archive Needs Product Research Product Candidate (Win!) Iterate on Current Experimentation Failure and Re- evaluate Bi-Weekly Council Product Ideation
- 14. Useful for any industry with customer-facing product.
- 15. ▪ Culture of experimentation across Engineering and Product ▪ Faster learning – “pruning the tree.” ▪ Continuous improving our ETL processes for cleaner, better data ▫ QA-ing the data in the pipeline ▫ Better alerting/monitoring
- 16. ▪ We are at upside.com! ▪ We host an Engineering and Data Science blog at engineering.upside.com ▪ Check out our experiments at labs.upside.com ▪ You can reach me at emily@upside.com.
Editor's Notes
- Hello! This is me!
- Everything in the travel industry is data. Inventory is data. Pricing is data. Packaging is data. We’re heavily awash in data, and it’s not always coherent data. It comes from dozens of sources, and every source has its own file and transport format with its own data and its own priorities. We have invested heavily in ETL (Matillion and Snowflake) to help smooth out that problem.
- This is me and what I believe. Our goal is to build engines of innovation around our data and ML to learn, grow, and give customers what they want. Our process is an ML Innovation Engine. Customer empathy is paramount.
- A nice selection of our current ML research work. We’re heavily focused on pricing, margins and personalization. We have limited resources, so we want to apply our people where we’re going to get. And we have a constant tensions between supply, profitability and our customer. Everyone ends up somewhat happy. It’s not optimized laboratory ML. It’s realistic, real-world ML.
- This is a list of the challenges directly out of the posting for the talk. We do like the phrase “model-market” fit! It means finding a model that works so well for the customer that the customer wants more of your product.
- * It is a way to validate product hypothesis against real customers. * It forces us to curate the hypothesis so that they’re measurable to the customer. * And these environments are usually only available to large environments ie IBM. We built it small and lean and effective on few people.
- One thing to emphasize – you cannot have a post doc in a corner as your AI launching system. This took a small army. This does not happen in silos. We put them all in a barn. Don’t spend much time here. If people want to learn more, they can get the deck after the fact.
- Capturing context and learning from the context. We have difficulty understanding the context of the customer at the time of purchase. We need to be as deliberate as possible. We have to marginalize against the context the data was generated in. HMMM is ML DEBT. What are the tools we can provide anyone that they can go and run experiments on their own or with little handholding? It’s not autoML but it is contextual ML and templatized ML in that it is contextual for our problem. Travel is very domain specific ML problem.
- This is our Innovation Filtering Process. Also gives us WHAT IT IS NOT. Gives us clear guidelines on experimentation. Can use the hypothesis as defense against scope creep. You can get velocity through scope management. Controlling scope is key
- #1 is the customer. These fields make it very clear what the MVP is. We need a document that lives beyond the team. Polish is very important. That’s where all the scope creep goes. Anything not necessary to measure the hypothesis is scope creep/ Launch Checklist is a big innovation. It forces a pattern to notify, measure, and does our measurement work? We found standardization helped us control scope.
- Innovation Engine in Action Take your hypothesis and break it up. An experiment really should not be more than two hours. Think about your smallest possible unit and make it smaller. Then pass through this cycle until you learn something useful. You can pop out at any of these points. It’s ok to bail if this not going to work!
- This is the evaluation! Does it get into the product?!? Exit points! There are feedback loops to this process. Even if it fails, there is a feedback loop. We have lots of opportunity for improvement and optimization You have to have someone looking over all of it We have taught non-data scientists this process for innovation
- Exactly what it says on the slide. We are accelerating research by controlling scope, leaning into customer empathy, focusing on learning, feedback cycles, and focusing our time on results. Despite dirty data and travel industry issues, we have pushed 50+ ML experiments to customers live. This works for any industry anywhere.
- Just what it says on the slide.
- Hey us!