Download to read offline
Pyconke2018
- 1. Speed up Pythonusing Rust Kent Marete @kentccs maretekent
- 2. Table of content Whatwe will talk about today • Why Python is awesome • Python applications • Speeding python using Rust library • Examples – Sentry, Csv CLI application • What is Rust? • Why Rust? • How Rust looks? –syntax, similarities • Rust trifecta – Speed, Concurrency, Safety • Rust tooling • Rust Pipeline • Who is using Rust? • Rust Meetup • Q & A
- 3. What my talkis not Introduction • It’s not about which language is great/ better • Why a certain language is not good • Comparing languages • Promoting a language
- 4. What my talkis … Introduction • Leverage on different technologies • Python and Rust power • Introduce Rust programming language
- 5. Programming languages asIngredients Introduction
- 6. Why Python isawesome…
- 7. Why Python isawesome … Why python • Readability - resembles the English language • Libraries - Python has been around for over 20 years • Community - has user groups everywhere • Dynamically typed language – no types declaring variable • Simple syntax – less debugging • One liner – elegant 1-liner solution i.e. x, y = y, x • Intuitive data structures – List, tuples, dictionaries, mixed data types
- 8.
- 9.
- 10. Python applications Why python DesktopGUIs SoftwareDevelopment Business Applications
- 11.
- 12. What is Rust? •Systems programming language • Trifecta – Speed, concurrency and Safety • Rust is considered a C-like language with more features – advantage
- 13. History • Graydon Hoaredev 2006 • Mozilla starts sponsoring Rust in 2009 • Version 1.0.0 (2015-) • > 19,567 crates – libraries & 584,987,337 Downloads • >2,211 contributors on Github. Compiler • Big areas: game dev, operating systems, web development, block chain
- 14. Csvterm CLI project MattGathu @swissgathu
- 15. About csvterm cliproject Csvterm cli project • Count the number of rows in the CSV data • Show the last N rows of CSV data. • Search CSV data using regex • Sort CSV data Commands sort search count head headers
- 16. Demo time Csvterm cliproject
- 17. Bar graph showingsort in Python vs Python with Rust lib
- 18. Line graph showingsort in Python vs Python with Rust lib
- 19. Bar graph showingbig csv file sort in Python vs Python with Rust lib Rsort: 35 secs Sort: 6 minutes 22.551s
- 20. Csvterm CLI project CodeAnatomy
- 21. Extending Python withC Why python Useful libraries: • Cffi
- 22.
- 23. Extending Python withRust Why python Useful libraries: • Cffi • Milksnake Other libs • Click • terminaltables Procedure 1. build the Rust code into libs (crate) 2. build a new crate that exposes a nice C-ABI 3. use cbindgen to generate C headers 4. use Milksnake to automatically generate low-level bindings from this crate and headers 5. generate high-level Python wrappers around the low- level bindings
- 24. Extending Python withRust Why python Procedure 1. build the Rust code into libs (crate) 2. build a new crate that exposes a nice C-ABI 3. Use cbindgen to generate C headers 4. use Milksnake to automatically generate low-level bindings from this crate and headers 5. generate high-level Python wrappers around the low- level bindings cargo new crate_name --lib cargo new crate_name --bin
- 25. Extending Python withRust Why python Procedure 1. build the Rust code into libs (crate) 2. build a new crate that exposes a nice C-ABI 3. Use cbindgen to generate C headers 4. use Milksnake to automatically generate low- level bindings from this crate and headers 5. generate high-level Python wrappers around the low- level bindings
- 26. Extending Python withRust To explain further
- 27. Extending Python withRust Why python Procedure 1. build the Rust code into libs (crate) 2. build a new crate that exposes a nice C-ABI 3. Use cbindgen to generate C headers 4. use Milksnake to automatically generate low-level bindings from this crate and headers 5. generate high-level Python wrappers around the low- level bindings
- 28. Extending Python withRust Why python Procedure 1. build the Rust code into libs (crate) 2. build a new crate that exposes a nice C-ABI 3. Use cbindgen to generate C headers 4. use Milksnake to automatically generate low-level bindings from this crate and headers 5. generate high-level Python wrappers around the low- level bindings
- 29. Extending Python withRust Why python Procedure 1. build the Rust code into libs (crate) 2. build a new crate that exposes a nice C-ABI 3. Use cbindgen to generate C headers 4. use Milksnake to automatically generate low- level bindings from this crate and headers 5. generate high-level Python wrappers around the low- level bindings
- 30.
- 31. Sentry extends Pythonwith Rust Fixed performance with Rust
- 32. Sentry extends Pythonwith Rust CPUs processing
- 33. Sentry extends Pythonwith Rust The average processing time
- 34.
- 35. Why Rust? Speed Rust has great functionality – raw binary data Reliability – rare to break, once it runs Control limited resources Concurrency Type inference - let b= 5u8; let a=5; High level abstraction – minimizes code
- 36.
- 37.
- 38.
- 39. • Run pieceof code concurrently • leverages ownership safety • message-passing between threads • One data owner at a time • more than one pointer to the same memory no changing of the data • Uses Ownership and borrowing concept • No Garbage Collection • LLVM - compiler infrastructure • Zero Cost Abstractions • Minimal Runtime – No GC, can compile without stdlib Speed, Safety and Concurrency SUB-HEADING
- 40. Rust tooling &pipeline
- 41. Rust tooling andpipeline Cargo rustup Rustfmt Clippy RFC Nightly Beta Stable
- 42. Who is usingRust? https://www.rust-lang.org/en-US/friends.html Armin Ronacher
- 43. Rust Nairobi Usergroup Meetup https://www.meetup.com/Rust-Nairobi/ https://twitter.com/RustNairobi https://github.com/rust-nairobi/
- 44.
Editor's Notes
- #5 How could we leverage on different technologies to solve our issues and achieve solutions in optimal way
- #10 Web and Internet Development – django, flask, pyramid, bottle Scientific and Numeric – SciPy, Pandas- Data science Education – Deep Learning- Natural Language Processing - like detect spam emails, write poetry, spin articles, and group together similar words. Scripting & Analysis – Machine learning
- #11 Desktop GUIs – pyqt, Tk Software Development – build control Business Applications – ERPs
- #13 Rust is a systems programming language that runs blazingly fast, prevents segfaults, and guarantees thread safety. Avoiding data races. Rust is a systems programming language focused on three goals: safety, speed, and concurrency. Segfaults are caused by a program trying to read or write an illegal memory location. A data race occurs when: two or more threads in a single process access the same memory location concurrently, and. at least one of the accesses is for writing, and. the threads are not using any exclusive locks to control their accesses to that memory.
- #14 Abstract Machine Model : C Data types : C, SML, OCaml, Lisp, Limbo Optional Bindings : Swift Hygienic Macros : Scheme Functional Programming : Haskell, OCaml, F# Attributes : ECMA-335 Memory Model and Memory Management : C++, ML Kit, Cyclone Type Classes : Haskell Crate : Assembly in the ECMA-335 CLI model Channels and Concurrency : Newsqueak, Alef, Limbo Message passing and Thread failure : Erlang
- #22 C Foreign Function Interface - provides a simple to use mechanism for interfacing with C from both CPython and PyPy. It supports two modes: an inline ABI compatibility mode an inline ABI compatibility mode (example provided below) which allows you to dynamically load and run functions from executable modules (essentially exposing the same functionality as LoadLibrary or dlopen) --------------------------------- ctypes is the de facto library for interfacing with C/C++ from CPython, and it provides not only full access to the native C interface of most major operating systems (e.g., kernel32 on Windows, or libc on *nix), but also provides support for loading and interfacing with dynamic libraries, such as DLLs or shared objects at runtime. What is ABI: application binary interface (ABI) is an interface between two binary program modules; often, one of these modules is a library or operating system facility, and the other is a program that is being run by a user.
- #23 This is using ctypes Load dynamic library in rust Generally import the function as it is pass in the arguments/ params Does computation inside rust and give the results
- #24 These are some useful libraries Milksnake is an extension for setuptools that allows you to distribute dynamic linked libraries in Python wheels in the most portable way imaginable. ========================= Dylib – dynamic library What is ABI: application binary interface (ABI) is an interface between two binary program modules; often, one of these modules is a library or operating system facility, and the other is a program that is being run by a user.
- #25 Lib creates a crate (reusable library) Bin creates a binary project
- #26 Sometimes, your Rust code may need to interact with code written in another language. Because that is the way we interact with the python through c. To do this, Rust has a keyword, extern, that facilitates creating and using a Foreign Function Interface (FFI). No_mangle is used in mangling mangling is the encoding of function and variable names into unique names so that linkers can separate common names in the language. Linkers - a program used with a compiler or assembler to provide links to the libraries needed for an executable program What is ABI: application binary interface (ABI) is an interface between two binary program modules; often, one of these modules is a library or operating system facility, and the other is a program that is being run by a user.
- #28 Cargo new –bin <name-project>
- #29 Cargo new –bin <name-project>
- #30 Cargo new –bin <name-project>
- #31 Sentry - error tracking that helps developers monitor and fix crashes in real time.
- #32 cut down the processing time from >20 seconds to <0.5 sec. This meant that even ignoring any optimizations, just replacing Python with Rust could relieve our chokepoint CPUs efficiently processing, Their worst source map times reduced to a tenth of their original time.
- #33 Actual graph of CPU processing CPUs efficiently processing
- #34 - The average processing time reduced to ~400ms.
- #36 Rust has great functionality which we don’t find in python – raw binary data If you needed to control of your machine and make most of your resource in the past you had one language C/ C++. Great power comes with great responsibility. Any small mistake can have catastrophic consequences or do the unexpected. The best scenario being a crash or worse open you to great security vulnerability. This is why systems programming has come to have horror of being a super hero. - Rust focuses on minimal core and extended by powerful libraries - Making it possible to incorporate other libraries - Great package management – crates.io - Open governance model - everyone can contribute to code High level abstraction – minimizes code, if you were to write a code to remove spaces Type inference - if you declare a variable as Unsigned 8 bit integer and you have pushed it to a vector which you didn’t define it will automatically give the vector unsigned 8bit integer datatype Reliability – rare to break – rust code is statically checked at compile time and also compile enforces the 2 concepts used in rust called ownership and borrowing. Interoperability with other languages using a Foreign Function Interface (FFI)
- #38 - Simple main function
- #42 Nightly – unstable features 6 weeks release cycle : only stable features A Request for Comments (RFC) is a formal document from the Internet Engineering Task Force ( IETF ) that is the result of committee drafting and subsequent review by interested parties. RFC becomes the standard and no further comments or changes are permitted.
- #43 - I have referenced him since he is one python builder and currently building sentry - He created the python micro framework and most importantly he found rust efficient and useful in his work - He blended python and rust in an amazing way by also creating and contributing to the milksnake library
- #44 Rust has great functionality which we don’t find in python – raw binary data If you needed to control of your machine and make most of your resource in the past you had one language C/ C++. Great power comes with great responsibility. Any small mistake can have catastrophic consequences or do the unexpected. The best scenario being a crash or worse open you to great security vulnerability. This is why systems programming has come to have horror of being a super hero. - Rust focuses on minimal core and extended by powerful libraries - Making it possible to incorporate other libraries - Great package management – crates.io - Open governance model - everyone can contribute to code High level abstraction – minimizes code, if you were to write a code to remove spaces Type inference - if you declare a variable as Unsigned 8 bit integer and you have pushed it to a vector which you didn’t define it will automatically give the vector unsigned 8bit integer datatype Reliability – rare to break – rust code is statically checked at compile time and also compile enforces the 2 concepts used in rust called ownership and borrowing.