SlideShare a Scribd company logo
Try-Monads With Big Data
Using PySpark
Jordan Yaker (@jordanyaker)
CEO/ Co-Founder of hireross.com (@hireross)
We all want this…
But sometimes we end up with this…
So, what exactly is a Monad?
• A thing.
• A thing to encapsulate other things.
• A thing to encapsulate other things and do stuff to.
• A thing to encapsulate other things and do other stuff to and then do
more stuff to.
• A thing to encapsulate other things and do other stuff to and then do
more stuff to and find out the results at the end.
Try Monad: Success or Exception Monad
How does this help stream processing?
• Per-Function Errors ➞
Per-Record Errors
• Defer Error Handling
• Cleaner Function
Implementations
How do I add a Try-Monad to my app?
• Pick a Python Monad implementation:
• Ø by Dag Brattli (https://github.com/dbrattli/Oslash)
• PyMonad by Jason DeLaat (https://bitbucket.org/jason_delaat/pymonad)
• Use an Either-Monad implementation or,
• Roll your own implementation or,
• Just copy mine (based on PyMonad)
• https://gist.github.com/jordanyaker/2850956fea18a3f3b65b13ef0c0cbe7d
How do I test my Monad-ified code?
Is it really dangerous to go alone?
We won’t know until we do.
• Ensure you’re passing
serializable objects.
• If you’re switching back and
forth between map and flat-map
operations act accordingly.
• Do something with the errors!
Acknowledgements
• Dag Brattli - github.com/dbrattli/oslash
• Jason DeLaat - bitbucket.org/jason_delaat/pymonad
• Be a Bird Comics - instagram.com/imthebirdguy
Thanks PyGotham 2017!
Jordan Yaker @jordanyaker
R.O.S.S. @hireross

More Related Content

Similar to Try-Monads With Big Data Using PySpark

MarkLogicWorld 2013 - Automate your deployments
MarkLogicWorld 2013 - Automate your deploymentsMarkLogicWorld 2013 - Automate your deployments
MarkLogicWorld 2013 - Automate your deployments
Geert Josten
 
Measure camp tools of the cro rabble
Measure camp   tools of the cro rabbleMeasure camp   tools of the cro rabble
Measure camp tools of the cro rabble
Craig Sullivan
 
Mongo NYC PHP Development
Mongo NYC PHP Development Mongo NYC PHP Development
Mongo NYC PHP Development
Fitz Agard
 
Yoast SEO for TYPO3 - TYPO3 Developer Days 2017
Yoast SEO for TYPO3 - TYPO3 Developer Days 2017Yoast SEO for TYPO3 - TYPO3 Developer Days 2017
Yoast SEO for TYPO3 - TYPO3 Developer Days 2017
Richard Haeser
 
Dev and Ops Collaboration and Awareness at Etsy and Flickr
Dev and Ops Collaboration and Awareness at Etsy and FlickrDev and Ops Collaboration and Awareness at Etsy and Flickr
Dev and Ops Collaboration and Awareness at Etsy and Flickr
John Allspaw
 
Lecture #3 activities and intents
Lecture #3  activities and intentsLecture #3  activities and intents
Lecture #3 activities and intents
Vitali Pekelis
 
Hacker vs company, Cloud Cyber Security Automated with Kubernetes - Demi Ben-...
Hacker vs company, Cloud Cyber Security Automated with Kubernetes - Demi Ben-...Hacker vs company, Cloud Cyber Security Automated with Kubernetes - Demi Ben-...
Hacker vs company, Cloud Cyber Security Automated with Kubernetes - Demi Ben-...
Demi Ben-Ari
 
Practical DMD Scripting
Practical DMD Scripting Practical DMD Scripting
Practical DMD Scripting
Zenoss
 
Hacking for fun & profit - The Kubernetes Way - Demi Ben-Ari - Panorays
Hacking for fun & profit - The Kubernetes Way - Demi Ben-Ari - PanoraysHacking for fun & profit - The Kubernetes Way - Demi Ben-Ari - Panorays
Hacking for fun & profit - The Kubernetes Way - Demi Ben-Ari - Panorays
Demi Ben-Ari
 
The Good, the Bad and the Ugly things to do with android
The Good, the Bad and the Ugly things to do with androidThe Good, the Bad and the Ugly things to do with android
The Good, the Bad and the Ugly things to do with android
Stanojko Markovik
 
How go makes us faster (May 2015)
How go makes us faster (May 2015)How go makes us faster (May 2015)
How go makes us faster (May 2015)
Wilfried Schobeiri
 
All about that reactive ui
All about that reactive uiAll about that reactive ui
All about that reactive ui
Paul van Zyl
 
Refactoring to GO modules
Refactoring to GO modulesRefactoring to GO modules
Refactoring to GO modules
Elad Hirsch
 
Big data made easy with a Spark
Big data made easy with a SparkBig data made easy with a Spark
Big data made easy with a Spark
Jean-Georges Perrin
 
What is "Domain Driven Design" and what can you expect from it?
What is "Domain Driven Design" and what can you expect from it?What is "Domain Driven Design" and what can you expect from it?
What is "Domain Driven Design" and what can you expect from it?
Tom Janssens
 
Continuous Deployment at Etsy: A Tale of Two Approaches
Continuous Deployment at Etsy: A Tale of Two ApproachesContinuous Deployment at Etsy: A Tale of Two Approaches
Continuous Deployment at Etsy: A Tale of Two Approaches
Ross Snyder
 
רועי פלד
רועי פלדרועי פלד
רועי פלד
Netcraft
 
Automation with OS X
Automation with OS XAutomation with OS X
Automation with OS X
Kirschen Seah
 
Journeys with Transmogrifier and friends or How not to get stuck in the Plone...
Journeys with Transmogrifier and friends or How not to get stuck in the Plone...Journeys with Transmogrifier and friends or How not to get stuck in the Plone...
Journeys with Transmogrifier and friends or How not to get stuck in the Plone...
Daniel Jowett
 
So You Just Inherited a $Legacy Application...
So You Just Inherited a $Legacy Application...So You Just Inherited a $Legacy Application...
So You Just Inherited a $Legacy Application...
Joe Ferguson
 

Similar to Try-Monads With Big Data Using PySpark (20)

MarkLogicWorld 2013 - Automate your deployments
MarkLogicWorld 2013 - Automate your deploymentsMarkLogicWorld 2013 - Automate your deployments
MarkLogicWorld 2013 - Automate your deployments
 
Measure camp tools of the cro rabble
Measure camp   tools of the cro rabbleMeasure camp   tools of the cro rabble
Measure camp tools of the cro rabble
 
Mongo NYC PHP Development
Mongo NYC PHP Development Mongo NYC PHP Development
Mongo NYC PHP Development
 
Yoast SEO for TYPO3 - TYPO3 Developer Days 2017
Yoast SEO for TYPO3 - TYPO3 Developer Days 2017Yoast SEO for TYPO3 - TYPO3 Developer Days 2017
Yoast SEO for TYPO3 - TYPO3 Developer Days 2017
 
Dev and Ops Collaboration and Awareness at Etsy and Flickr
Dev and Ops Collaboration and Awareness at Etsy and FlickrDev and Ops Collaboration and Awareness at Etsy and Flickr
Dev and Ops Collaboration and Awareness at Etsy and Flickr
 
Lecture #3 activities and intents
Lecture #3  activities and intentsLecture #3  activities and intents
Lecture #3 activities and intents
 
Hacker vs company, Cloud Cyber Security Automated with Kubernetes - Demi Ben-...
Hacker vs company, Cloud Cyber Security Automated with Kubernetes - Demi Ben-...Hacker vs company, Cloud Cyber Security Automated with Kubernetes - Demi Ben-...
Hacker vs company, Cloud Cyber Security Automated with Kubernetes - Demi Ben-...
 
Practical DMD Scripting
Practical DMD Scripting Practical DMD Scripting
Practical DMD Scripting
 
Hacking for fun & profit - The Kubernetes Way - Demi Ben-Ari - Panorays
Hacking for fun & profit - The Kubernetes Way - Demi Ben-Ari - PanoraysHacking for fun & profit - The Kubernetes Way - Demi Ben-Ari - Panorays
Hacking for fun & profit - The Kubernetes Way - Demi Ben-Ari - Panorays
 
The Good, the Bad and the Ugly things to do with android
The Good, the Bad and the Ugly things to do with androidThe Good, the Bad and the Ugly things to do with android
The Good, the Bad and the Ugly things to do with android
 
How go makes us faster (May 2015)
How go makes us faster (May 2015)How go makes us faster (May 2015)
How go makes us faster (May 2015)
 
All about that reactive ui
All about that reactive uiAll about that reactive ui
All about that reactive ui
 
Refactoring to GO modules
Refactoring to GO modulesRefactoring to GO modules
Refactoring to GO modules
 
Big data made easy with a Spark
Big data made easy with a SparkBig data made easy with a Spark
Big data made easy with a Spark
 
What is "Domain Driven Design" and what can you expect from it?
What is "Domain Driven Design" and what can you expect from it?What is "Domain Driven Design" and what can you expect from it?
What is "Domain Driven Design" and what can you expect from it?
 
Continuous Deployment at Etsy: A Tale of Two Approaches
Continuous Deployment at Etsy: A Tale of Two ApproachesContinuous Deployment at Etsy: A Tale of Two Approaches
Continuous Deployment at Etsy: A Tale of Two Approaches
 
רועי פלד
רועי פלדרועי פלד
רועי פלד
 
Automation with OS X
Automation with OS XAutomation with OS X
Automation with OS X
 
Journeys with Transmogrifier and friends or How not to get stuck in the Plone...
Journeys with Transmogrifier and friends or How not to get stuck in the Plone...Journeys with Transmogrifier and friends or How not to get stuck in the Plone...
Journeys with Transmogrifier and friends or How not to get stuck in the Plone...
 
So You Just Inherited a $Legacy Application...
So You Just Inherited a $Legacy Application...So You Just Inherited a $Legacy Application...
So You Just Inherited a $Legacy Application...
 

Recently uploaded

A Study of Variable-Role-based Feature Enrichment in Neural Models of Code
A Study of Variable-Role-based Feature Enrichment in Neural Models of CodeA Study of Variable-Role-based Feature Enrichment in Neural Models of Code
A Study of Variable-Role-based Feature Enrichment in Neural Models of Code
Aftab Hussain
 
Graspan: A Big Data System for Big Code Analysis
Graspan: A Big Data System for Big Code AnalysisGraspan: A Big Data System for Big Code Analysis
Graspan: A Big Data System for Big Code Analysis
Aftab Hussain
 
SMS API Integration in Saudi Arabia| Best SMS API Service
SMS API Integration in Saudi Arabia| Best SMS API ServiceSMS API Integration in Saudi Arabia| Best SMS API Service
SMS API Integration in Saudi Arabia| Best SMS API Service
Yara Milbes
 
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian CompaniesE-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
Quickdice ERP
 
GreenCode-A-VSCode-Plugin--Dario-Jurisic
GreenCode-A-VSCode-Plugin--Dario-JurisicGreenCode-A-VSCode-Plugin--Dario-Jurisic
GreenCode-A-VSCode-Plugin--Dario-Jurisic
Green Software Development
 
Why Choose Odoo 17 Community & How it differs from Odoo 17 Enterprise Edition
Why Choose Odoo 17 Community & How it differs from Odoo 17 Enterprise EditionWhy Choose Odoo 17 Community & How it differs from Odoo 17 Enterprise Edition
Why Choose Odoo 17 Community & How it differs from Odoo 17 Enterprise Edition
Envertis Software Solutions
 
Automated software refactoring with OpenRewrite and Generative AI.pptx.pdf
Automated software refactoring with OpenRewrite and Generative AI.pptx.pdfAutomated software refactoring with OpenRewrite and Generative AI.pptx.pdf
Automated software refactoring with OpenRewrite and Generative AI.pptx.pdf
timtebeek1
 
Hand Rolled Applicative User Validation Code Kata
Hand Rolled Applicative User ValidationCode KataHand Rolled Applicative User ValidationCode Kata
Hand Rolled Applicative User Validation Code Kata
Philip Schwarz
 
SWEBOK and Education at FUSE Okinawa 2024
SWEBOK and Education at FUSE Okinawa 2024SWEBOK and Education at FUSE Okinawa 2024
SWEBOK and Education at FUSE Okinawa 2024
Hironori Washizaki
 
Atelier - Innover avec l’IA Générative et les graphes de connaissances
Atelier - Innover avec l’IA Générative et les graphes de connaissancesAtelier - Innover avec l’IA Générative et les graphes de connaissances
Atelier - Innover avec l’IA Générative et les graphes de connaissances
Neo4j
 
Transform Your Communication with Cloud-Based IVR Solutions
Transform Your Communication with Cloud-Based IVR SolutionsTransform Your Communication with Cloud-Based IVR Solutions
Transform Your Communication with Cloud-Based IVR Solutions
TheSMSPoint
 
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOMLORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
lorraineandreiamcidl
 
What is Augmented Reality Image Tracking
What is Augmented Reality Image TrackingWhat is Augmented Reality Image Tracking
What is Augmented Reality Image Tracking
pavan998932
 
LORRAINE ANDREI_LEQUIGAN_HOW TO USE WHATSAPP.pptx
LORRAINE ANDREI_LEQUIGAN_HOW TO USE WHATSAPP.pptxLORRAINE ANDREI_LEQUIGAN_HOW TO USE WHATSAPP.pptx
LORRAINE ANDREI_LEQUIGAN_HOW TO USE WHATSAPP.pptx
lorraineandreiamcidl
 
socradar-q1-2024-aviation-industry-report.pdf
socradar-q1-2024-aviation-industry-report.pdfsocradar-q1-2024-aviation-industry-report.pdf
socradar-q1-2024-aviation-industry-report.pdf
SOCRadar
 
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI AppAI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
Google
 
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...Introducing Crescat - Event Management Software for Venues, Festivals and Eve...
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...
Crescat
 
Using Xen Hypervisor for Functional Safety
Using Xen Hypervisor for Functional SafetyUsing Xen Hypervisor for Functional Safety
Using Xen Hypervisor for Functional Safety
Ayan Halder
 
Microservice Teams - How the cloud changes the way we work
Microservice Teams - How the cloud changes the way we workMicroservice Teams - How the cloud changes the way we work
Microservice Teams - How the cloud changes the way we work
Sven Peters
 
Need for Speed: Removing speed bumps from your Symfony projects ⚡️
Need for Speed: Removing speed bumps from your Symfony projects ⚡️Need for Speed: Removing speed bumps from your Symfony projects ⚡️
Need for Speed: Removing speed bumps from your Symfony projects ⚡️
Łukasz Chruściel
 

Recently uploaded (20)

A Study of Variable-Role-based Feature Enrichment in Neural Models of Code
A Study of Variable-Role-based Feature Enrichment in Neural Models of CodeA Study of Variable-Role-based Feature Enrichment in Neural Models of Code
A Study of Variable-Role-based Feature Enrichment in Neural Models of Code
 
Graspan: A Big Data System for Big Code Analysis
Graspan: A Big Data System for Big Code AnalysisGraspan: A Big Data System for Big Code Analysis
Graspan: A Big Data System for Big Code Analysis
 
SMS API Integration in Saudi Arabia| Best SMS API Service
SMS API Integration in Saudi Arabia| Best SMS API ServiceSMS API Integration in Saudi Arabia| Best SMS API Service
SMS API Integration in Saudi Arabia| Best SMS API Service
 
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian CompaniesE-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
 
GreenCode-A-VSCode-Plugin--Dario-Jurisic
GreenCode-A-VSCode-Plugin--Dario-JurisicGreenCode-A-VSCode-Plugin--Dario-Jurisic
GreenCode-A-VSCode-Plugin--Dario-Jurisic
 
Why Choose Odoo 17 Community & How it differs from Odoo 17 Enterprise Edition
Why Choose Odoo 17 Community & How it differs from Odoo 17 Enterprise EditionWhy Choose Odoo 17 Community & How it differs from Odoo 17 Enterprise Edition
Why Choose Odoo 17 Community & How it differs from Odoo 17 Enterprise Edition
 
Automated software refactoring with OpenRewrite and Generative AI.pptx.pdf
Automated software refactoring with OpenRewrite and Generative AI.pptx.pdfAutomated software refactoring with OpenRewrite and Generative AI.pptx.pdf
Automated software refactoring with OpenRewrite and Generative AI.pptx.pdf
 
Hand Rolled Applicative User Validation Code Kata
Hand Rolled Applicative User ValidationCode KataHand Rolled Applicative User ValidationCode Kata
Hand Rolled Applicative User Validation Code Kata
 
SWEBOK and Education at FUSE Okinawa 2024
SWEBOK and Education at FUSE Okinawa 2024SWEBOK and Education at FUSE Okinawa 2024
SWEBOK and Education at FUSE Okinawa 2024
 
Atelier - Innover avec l’IA Générative et les graphes de connaissances
Atelier - Innover avec l’IA Générative et les graphes de connaissancesAtelier - Innover avec l’IA Générative et les graphes de connaissances
Atelier - Innover avec l’IA Générative et les graphes de connaissances
 
Transform Your Communication with Cloud-Based IVR Solutions
Transform Your Communication with Cloud-Based IVR SolutionsTransform Your Communication with Cloud-Based IVR Solutions
Transform Your Communication with Cloud-Based IVR Solutions
 
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOMLORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
 
What is Augmented Reality Image Tracking
What is Augmented Reality Image TrackingWhat is Augmented Reality Image Tracking
What is Augmented Reality Image Tracking
 
LORRAINE ANDREI_LEQUIGAN_HOW TO USE WHATSAPP.pptx
LORRAINE ANDREI_LEQUIGAN_HOW TO USE WHATSAPP.pptxLORRAINE ANDREI_LEQUIGAN_HOW TO USE WHATSAPP.pptx
LORRAINE ANDREI_LEQUIGAN_HOW TO USE WHATSAPP.pptx
 
socradar-q1-2024-aviation-industry-report.pdf
socradar-q1-2024-aviation-industry-report.pdfsocradar-q1-2024-aviation-industry-report.pdf
socradar-q1-2024-aviation-industry-report.pdf
 
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI AppAI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
 
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...Introducing Crescat - Event Management Software for Venues, Festivals and Eve...
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...
 
Using Xen Hypervisor for Functional Safety
Using Xen Hypervisor for Functional SafetyUsing Xen Hypervisor for Functional Safety
Using Xen Hypervisor for Functional Safety
 
Microservice Teams - How the cloud changes the way we work
Microservice Teams - How the cloud changes the way we workMicroservice Teams - How the cloud changes the way we work
Microservice Teams - How the cloud changes the way we work
 
Need for Speed: Removing speed bumps from your Symfony projects ⚡️
Need for Speed: Removing speed bumps from your Symfony projects ⚡️Need for Speed: Removing speed bumps from your Symfony projects ⚡️
Need for Speed: Removing speed bumps from your Symfony projects ⚡️
 

Try-Monads With Big Data Using PySpark

  • 1. Try-Monads With Big Data Using PySpark Jordan Yaker (@jordanyaker) CEO/ Co-Founder of hireross.com (@hireross)
  • 2. We all want this…
  • 3. But sometimes we end up with this…
  • 4. So, what exactly is a Monad? • A thing. • A thing to encapsulate other things. • A thing to encapsulate other things and do stuff to. • A thing to encapsulate other things and do other stuff to and then do more stuff to. • A thing to encapsulate other things and do other stuff to and then do more stuff to and find out the results at the end.
  • 5. Try Monad: Success or Exception Monad
  • 6. How does this help stream processing? • Per-Function Errors ➞ Per-Record Errors • Defer Error Handling • Cleaner Function Implementations
  • 7. How do I add a Try-Monad to my app? • Pick a Python Monad implementation: • Ø by Dag Brattli (https://github.com/dbrattli/Oslash) • PyMonad by Jason DeLaat (https://bitbucket.org/jason_delaat/pymonad) • Use an Either-Monad implementation or, • Roll your own implementation or, • Just copy mine (based on PyMonad) • https://gist.github.com/jordanyaker/2850956fea18a3f3b65b13ef0c0cbe7d
  • 8. How do I test my Monad-ified code?
  • 9. Is it really dangerous to go alone? We won’t know until we do. • Ensure you’re passing serializable objects. • If you’re switching back and forth between map and flat-map operations act accordingly. • Do something with the errors!
  • 10. Acknowledgements • Dag Brattli - github.com/dbrattli/oslash • Jason DeLaat - bitbucket.org/jason_delaat/pymonad • Be a Bird Comics - instagram.com/imthebirdguy
  • 11. Thanks PyGotham 2017! Jordan Yaker @jordanyaker R.O.S.S. @hireross