Successfully reported this slideshow.
Your SlideShare is downloading. ×

Better Python Coding with Prefect Blocks

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad

Check these out next

1 of 96 Ad
Advertisement

More Related Content

Recently uploaded (20)

Advertisement

Better Python Coding with Prefect Blocks

  1. 1. Better Python Coding with Prefect Blocks 🧱 Jeff Hale prefect.io PyData Global 2022 linkedin.com/in/-jeffhale/
  2. 2. Jeff Hale ● Hawaiian shirts 🌺 ● Hiking 🥾 ● Ultimate frisbee 🥏
  3. 3. Jeff Hale 📊 + 🐍 ● Teach 🍎 ● Write ✍ ● Organize events 🫂
  4. 4. I ❤ Prefect because it solves common pain points elegantly
  5. 5. 🐍 open source library 🙂 growing company behind it
  6. 6. Why Prefect? Usual answer: Coordinate your dataflow! 🚀
  7. 7. Observation + Orchestration 🧐 + 🪄
  8. 8. How? ● Sprinkle in flow & task decorators ● Turn into a deployment ● Share with workspace users
  9. 9. Today: Blocks
  10. 10. Why blocks? ● 🤗 More collaboration ● 🏛 Fewer errors ● 🔐 Increased security ● 🍴 Easier extendability ● 🕰 Time saved
  11. 11. What are blocks? + front-end magic + server + access controls
  12. 12. Stored in server database ● OSS Orion DB (SQLite or Postgres) ● Prefect Cloud
  13. 13. Problem ☹
  14. 14. “Data practitioners have historically been forced to choose between the simple setup experience of low-code data integration tools and the flexibility of an all-Python pipeline.”
  15. 15. Simple form vs. Flexible 🐍
  16. 16. Blocks offer choices 🙂
  17. 17. Prefect Blocks are like a mullet Structured form in front Flexible code in back
  18. 18. Low-code connectors with a first class code experience ✨
  19. 19. How do I get this Block awesomeness?
  20. 20. Install pip install -U prefect
  21. 21. Option 1 Create with HTML form interface
  22. 22. Option 1
  23. 23. Option 1
  24. 24. Use it
  25. 25. Option 2 Create with 🐍 code
  26. 26. Use it
  27. 27. Prefect Blocks 🧱 Flexible creation and editing
  28. 28. Use case You want to share cloud provider configuration securely 🔐
  29. 29. 🔐 Options 1. Hard code secrets 🚫 Just don’t 😅
  30. 30. 2. Environment variables ● Viewable in logs ● Tied to machine
  31. 31. 3. Secrets manager ● Vendor lock-in ☁ ● Steep learning curve 📈 ● Missing configuration 🧭
  32. 32. 4. Build custom secrets manager ● Add configuration ● Encryption ● Server sync ● Bunch of code
  33. 33. Prefect Secret Blocks 🔐🧱 ● Obfuscated pydantic SecretStr ● Encrypted at rest ● Can use HashiCorp Vault with prefect-vault collection
  34. 34. Historical context
  35. 35. 1791 Charles Babbage ● Mathematician ● Philosopher ● Code breaker ● Inventor By anonymous - Art UK, Public Domain, https://commons.wikimedia.org/w/index.php?curid=97545883
  36. 36. Babbage’s Inventions Ophthalmoscope Cowcatcher
  37. 37. Analytical engine By Charles Babbage - Upload by Mrjohncummings 2013-08-28 15:10, CC BY-SA 2.0, https://commons.wikimedia.org/w/index.php?curid=28024313
  38. 38. Punch cards By Arnold Reinhold - I took this picture of an artifact in my possession. The card was created in the late 1960s or early1970s and has no copyright notice., CC BY-SA 2.5, https://commons.wikimedia.org/w/index.php?curid=775153
  39. 39. 1842 Ada Lovelace First computer program By Margaret Sarah Carpenter - UK Government Art Collection website.Original upload was at English wikipedia at en:File:Ada_Lovelace.jpg, Public Domain, https://commons.wikimedia.org/w/index.php?curid=35407
  40. 40. 1991 Guido van Rossum Python released By www.python.org - www.python.org, GPL, https://commons.wikimedia.org/w/index.php?curid=34991651
  41. 41. 2018 Streamlit
  42. 42. 2018 Streamlit Takes HTML/JavaScript/Jinja out of the Python developer’s required skill set for data website development! 🎉
  43. 43. 2018 Jeremiah Lowin PyData DC talk Task Failed Successfully
  44. 44. 2019 Prefect released
  45. 45. 2022 Prefect 2 released
  46. 46. Similar to Streamlit Prefect Blocks allow Python developers to build an HTML interface easily ⭐
  47. 47. Different than Streamlit Prefect Blocks save configuration information for reuse and sharing in Python. 🐍
  48. 48. Block types
  49. 49. Prefect ships with many block types
  50. 50. Basic Block Types ● String 🔤 ● Date Time 📆 ● JSON 📖 ● Secret 🔐
  51. 51. Code Storage Block Types ● GitHub ● S3 ● Azure ● GCS
  52. 52. Infrastructure Block Types ● DockerContainer ● KubernetesJob
  53. 53. Notification Block Types ● Email ● Slack ● Microsoft Teams ● Twillio ● PagerDuty
  54. 54. Zoom in 🔬 AWS S3 Block Stores code to run on a schedule
  55. 55. A colleague who doesn’t want to touch Python can add exactly the configuration info needed! 🎉
  56. 56. Slack notifications
  57. 57. Need more block types?
  58. 58. Collections 🎉 ● Python libraries ● Built by Prefect/community
  59. 59. Blocks from Collections ● Snowflake ● dbt ● GitLab ● Hex ● Great Expectations ● Fugue
  60. 60. Need even more block types?
  61. 61. Create your own!
  62. 62. Register them with the server prefect block register
  63. 63. Block Type Shows in UI!
  64. 64. Prefect has helpful: ● Docs 📃 ● Slack Community 📣 ● Recipes 🍛 ● Blog posts 📝 ● Docstrings 📒 ● CLI - - help 🔤
  65. 65. Block benefits
  66. 66. Fewer Errors Blocks take the guesswork out of configuring a connection
  67. 67. Composable Can use multiple blocks together in the same code
  68. 68. Nestable
  69. 69. Nestable
  70. 70. Nestable
  71. 71. Versatile Same core concept - many different roles ● Storage ● Infrastructure ● Notifications
  72. 72. Reusable Reuse a block in multiple flows
  73. 73. Shareable Workspace users can share blocks
  74. 74. Extendable “With blocks, people don't need to fork Prefect to extend it.”
  75. 75. What else should I know about Prefect? 🤔
  76. 76. Best option for dataflow coordination 1. Automatic logging 2. Automatic retries 3. Scheduling 4. Notifications 5. Caching 6. Easy async 7. Collaborative UI 8. Hybrid model 9. Integrations 10. Incremental adoption
  77. 77. Easy start
  78. 78. Add Prefect flow decorator
  79. 79. Want retries? 🪄
  80. 80. Want notifications? 🪄
  81. 81. Prefect Cloud ● Hosted ● Free tier with lots of goodies ● Enterprise tools
  82. 82. Prefect Cloud
  83. 83. Hybrid model ●
  84. 84. Prefect Cloud 2 billion tasks
  85. 85. One more blocks metaphor
  86. 86. ● System of pulleys & ropes ● Mechanical advantage ● Force multiplier 💪 Block and tackle
  87. 87. Easier to build great things 🎉 By CR - Wikipedia en Español, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=38727
  88. 88. Why Blocks? 🧱 1. Create with HTML form or code 2. Use in code 3. Fewer errors 4. Secure 5. Composable 6. Nestable 7. Versatile 8. Reusable 9. Shareable 10. Extendable
  89. 89. How could your next Python project benefit from Blocks?
  90. 90. Thank you! Better Python Coding with Prefect Blocks 🧱 Jeff Hale prefect.io PyData Global 2022 linkedin.com/in/-jeffhale/

×