Prefect: tools for interacting with complex system. Prefect is the flexible and scalable Python data orchestrator. And introducing Marvin, the batteries-included library for building AI-powered software.

1. Prefect
Tools for dealing with complex systems
Jeff Hale prefect.io
April 2023

2. What?

3. Prefect
Tools for interacting with complex
systems
Prefect
Flexible & scalable
Python data
orchestrator 🐍📊🏗
Marvin
Batteries-included library
for building
AI-powered software
🤖🪄

4. Prefect
Tools for interacting with complex
systems

5. Prefect
The flexible and scalable
Python data orchestrator 🐍📊🪄

7. Who’s talking?

8. Jeff Hale
Developer Advocate
● Teach 🍎
● Write ✍
● Organize events 🫂

9. Jeff + Marvin
Also bad puns - courtesy of Marvin
Why did the data scientist break up with her
boyfriend?
Because he couldn't handle her discrete values!

10. Why an
orchestrator?

11. “I have some tasks, I need
to make sure they happen.”
- Jeremiah Lowin, Prefect CEO

12. “Everything fails all the
time.”
- Werner Vogels, AWS CTO

13. Logging
● Logging
● UI visibility
● Automatic retries
● Notifications
● Caching
● Scheduling
● Event-driven workflows

14. Observability Orchestration
Event monitoring
Logging
Automations
Retries
Concurrency
limits
Parallelization
Data dependencies
Scheduling

15. Background

16. Prefect Origins
Jeremiah Lowin
● Ported Airflow to Python 3
● Former Airflow PMC member
● Wrote XComs

17. Prefect Origins
Founded Prefect to:
● Handle workloads Airflow wasn’t designed for
● Treat data in a first class way
● Improve user experience

18. Background
● 2018 PyData talk
● 2019 Prefect released
● 2022 Prefect 2.0 released

19. Prefect Today
🐍 Open source Prefect library
☁ Cloud platform
AI assistant in your code
📈 Strong and growing company

20. Stats

21. GitHub Forks 🍴
1,000+

22. GitHub Stars ⭐
10,000+

23. Community Slack Members
20,000+
prefect.io/slack

24. Weekly Downloads
200,000+

25. Cloud task runs
2,000,000,000+

26. Features

27. Features
1. Scalable
2. Flexible
3. Observability
4. Event driven

28. 1. Scalable
Can run and observe thousands of flows in the UI
YouTube Video: Orchestrate Your Data Science Workflows at
Scale using Anyscale and Prefect and Featuring LiveEO

29. Scalable

30. 2. Flexible
Just bring your Python code
(even 3.11)
Image: www.python.org, GPL, https://commons.wikimedia.org/w/index.php?curid=34991651

31. 2. Flexible
● DAGs are fine, but also restricting.
● Prefect doesn’t require you to predefine a DAG.

32. ● Logging in the CLI & UI
● Timeline view
● Collaborators
● Notifications
3. Observability

34. 4. Event-driven workflows
Events provide observability and drive Automations
Automations = Trigger then Action

35. 4. Event-driven workflows

36. 4. Event-driven workflows

37. One more thing: DX ❤
● Clear, well-documented API
● Blocks: configuration + code
● Fast release velocity
● Supportive community
prefect.io/guide/blog/blocks-connectors-for-code

38. API

39. Blocks 🧱

40. Frequent Releases 🚀

41. Supportive Community 🤗

42. Server or Cloud?

43. Prefect
1. Automatic logging
2. Automatic retries
3. Scheduling
4. Notifications
5. Caching
6. Simplified async
7. Integrations
8. Pythonic incremental adoption
9. Informative UI
10. Blocks

44. Prefect Cloud
1. SSO
2. RBAC
3. Collaborative UI
4. Hybrid model
5. Automations
6. Events
7. Super scalable
8. Strong security
9. Free forever tier
10. Self serve tier

45. Hybrid model

46. Why Server?
1. You really enjoy setting up and maintaining
server infrastructure 😅
2. Particular needs that mean you can’t use a
cloud solution
Solution for large users: Prefect Dedicated
Cloud

47. How?

48. Install
pip install -U prefect

49. How?
● Sprinkle flow & task decorators into Python
● Log in to Prefect Cloud
● Invite collaborators
● Create a deployment
● Start an agent/worker

50. Integrations

53. Marvin
Why do programmers prefer dark mode?
Because light attracts too many data
bugs! 🐛

54. Doing it

55. Plain Python 🐍
Writes out cat fact and prints message
(if all goes well)

57. Enlightening cat fact 🐱

58. Prefectionized Python 🐍
● Add flow and task decorators
● Writes out cat fact and prints message +
helpful logs

60. Logs

61. Want retries? 🪄

62. Want retries? 🪄

63. Bundle flow into a deployment
Deployment
● Specify flow code storage (Cloud providers/repos)
● Define infrastructure (Docker, K8s, ECS, etc.)
● Schedule

64. Doing it at scale

65. Schedule

66. Scalable
2,000 tasks

68. Scalable
● Parallelize with dask
● Can run locally or distributed

69. Scalable

70. Scalable

71. Scalable

72. Marvin
Why do data analysts like looking at pie
charts?
Because they're sweet and delicious! 🥧

73. Transitioning from
Airflow

74. Airflow

75. Option 1
Airflow with Prefect Event
Observation

76. Airflow

77. Airflow with Prefect Event Observation

79. Option 2
Airflow with Prefect Flows

80. Airflow with
Prefect Flows
(Part 1)

81. Airflow with
Prefect Flows
(Part 2)

82. Airflow with Prefect Flows
In Airflow

83. Airflow with Prefect Flows
In Prefect

84. Why Prefect?

85. Prefect Customer Quotes
● “I can go from thought to production 5x
faster.”
● “We used to spend 20% of our time (and up
to 80% at times) on errors. [With Prefect,]
we're approaching 5%, a 75% reduction.”

86. Prefect Customer Quotes
● “...can't imagine how anyone wouldn't like it.”
● “We love prefect, it works and makes our
lives easier.”
● “ It addresses many of the pain points
common to more complicated tools like
Airflow.”

87. Advice
● Use what’s working
● Need scale?
● Need flexibility?
● Need observability across your stack?
● Need event-driven workflows?
● Need nicer UX?
Check out Prefect

88. Prefect
Observe everything 🧐
Orchestrate anything 🪄

89. Prefect Training Opportunities 🍎
San Francisco
eventbrite.com/e/prefect-associate-certification-course-san-francisco-tickets-450754347387
Online
eventbrite.com/e/prefect-practitioner-certification-course-live-online-june-14-2023-tickets-600839907337
London
eventbrite.com/e/prefect-associate-certification-course-london-tickets-476657915657
Discount Code for 20% off: AirflowParis20

90. One more thing…

91. Introducing Marvin

92. Marvin
pip install marvin
askmarvin.ai

93. Marvin

94. Marvin

95. Marvin AI Functions
pip install marvin
askmarvin.ai

96. Prefect
Tools for interacting with complex
systems
Prefect
Flexible & scalable
Python data
orchestrator 🐍📊🏗
Marvin
Batteries-included library
for building
AI-powered software
🤖🪄

97. Thank you!
Jeff Hale prefect.io
April 2023 linkedin.com/in/-jeffhale/