"Implementing real-time data pipelines is still a challenge. Even more so for data scientists who have often been brought up with batch processing and files and typically have only heard of Kafka, but never really used it. Now if your team consists only of data scientists and you want them to implement a real-time data pipeline, your fate seems to be sealed. You need to hire real-time and streaming experts first - and you are guaranteed to lose months before you can even start implementing. But is there really no other way? At Forecasty.AI, developer of a SaaS platform for commodity price forecasting called Commodity Desk, we thought again. And came up with a way how to bridge the gap between the batch and file-based world of most data scientists and the world of real-time and streaming. Our solution is a new Open Source library allowing any Python programmer, including the aforementioned data scientists, to access the Kafka API in an easier way than ever before. kash.py (""Kafka Shell for Python"") offers a large number of easy-to-use abstractions on top of the Kafka API, including bash-inspired commands like ""ls"" or ""l"" for listing Kafka topics and ""cat"", ""head"" or ""tail"" for displaying the content of topics. kash.py bridges the gap between the file and the streaming worlds with commands like ""upload"" to upload a file to a topic, or ""download"" to download a topic to a file, and offers commands inspired by functional programming to do Kafka-Kafka/File-Kafka or Kafka-File stream processing (""map"", ""flatMap"", ""filter"" etc.) in one-liners - of course with full support for JSON Schema, Avro and Protobuf. In this session, we show you how kash.py can be used to bring the two disparate worlds of files and streaming together - and thus not only save a lot of time and money hiring real-time and streaming experts, but also make your data scientists start loving real-time."

1. kash.py - How to Make Your Data Scientists
Love Real-time
Ralph M. Debusmann

2. What is kash.py?
●
Successor to Streampunk (presented at Kafka Summit London 2022)
●
Open Source module for Python – the language for Data Science/AI
(https://github.com/xdgrulez/kash.py)
●
Wrapper around confluent_kafka which is a wrapper around
librdkafka – both by Magnus Edenhill – he made it all possible :)

3. The Two Purposes of kash.py
1) Provide a shell for Kafka (kash.py = Kafka Shell for Python) that is
easier to use than the Apache Kafka CLI tools and goes beyond
kcat (again by Magnus Edenhill)
2) Provide a Python module for coding Kafka clients that is more
pythonic than confluent_kafka, requires less boilerplate code and
supports basic stream processing out of the box
=> kash.py is your Python-based swiss army knife for Kafka

4. Why Do We Need This?
●
Working with Kafka is still harder than it should be
●
For example, imagine you have a team of Data Scientists with no prior
exposure to Kafka that have to build a real-time data pipeline
●
Before kash.py, you would have had to hire additional streaming
experts before you could even start implementing
●
With kash.py in hand, you can make your Data Scientists build the
real-time data pipeline yourself and make them ♥ real-time :)

5. Seeing Is Believing
●
Let’s build a real-time data pipeline for a real-world use case from the
late Forecasty.AI – and, it’s 2023, the time of ChatGPT, so can’t go
without some trendy AI/LLM stuff in here ;)
Scrape tweets Score tweets Make
queryable

6. Seeing Is Believing
●
Let’s build a real-time data pipeline for a real-world use case from the
late Forecasty.AI – and, it’s 2023, the time of ChatGPT, so can’t go
without some trendy AI/LLM stuff in here ;)
Scrape tweets Score tweets Make
queryable

7. Bring Scraped Data to Kafka

8. Bring Scraped Data to Kafka

9. Bring Scraped Data to Kafka

10. Bring Scraped Data to Kafka

11. Bring Scraped Data to Kafka

12. Bring Scraped Data to Kafka

13. Bring Scraped Data to Kafka

14. Bring Scraped Data to Kafka

15. Bring Scraped Data to Kafka

16. Bring Scraped Data to Kafka

17. Bring Scraped Data to Kafka

18. Bring Scraped Data to Kafka

19. Bring Scraped Data to Kafka

20. Bring Scraped Data to Kafka

21. Bring Scraped Data to Kafka

22. Bring Scraped Data to Kafka

23. Bring Scraped Data to Kafka

24. Bring Scraped Data to Kafka

25. Bring Scraped Data to Kafka

26. Bring Scraped Data to Kafka

27. Bring Scraped Data to Kafka

28. Bring Scraped Data to Kafka

29. Bring Scraped Data to Kafka

30. Bring Scraped Data to Kafka

31. Bring Scraped Data to Kafka

32. Implement Sentiment Analysis (Scoring)

33. Implement Sentiment Analysis (Scoring)

34. Run Sentiment Analysis (Scoring)

35. Run Sentiment Analysis (Scoring)

36. Monitor Sentiment Analysis (Scoring)

37. Monitor Sentiment Analysis (Scoring)

38. Monitor Sentiment Analysis (Scoring)

39. Thank You ♥
●
kash.py can make your Data Scientists ♥ real-time :)
●
Thanks to the Forecasty.AI Data Scientists:
Jamsheeda Kadengal, Subhashree Mangaraj, Grant Sutcliffe, and Tuhin Mallick :)
●
kash.py is also very useful for Kafka DevOps engineers and
developers – e.g. for building PoCs, debugging…
●
To use it, just import kash.py from PyPI or contribute at GitHub
(https://github.com/xdgrulez/kash.py) – a star would be nice
●
Watch out for the next version of kash.py with support for:
●
Kafka REST Proxy
●
Blob Storage (S3 etc.)
●
...