Recommended
Recommended
More Related Content
Similar to Oops! I Wrote my Data Science in COBOL
Similar to Oops! I Wrote my Data Science in COBOL (20)
Recently uploaded
Recently uploaded (20)
Oops! I Wrote my Data Science in COBOL
- 1. 2019-05-25 Oops! I wrote my data science in COBOL
- 4. Program
- 5. Data Science for us The use of Advanced Analytics to help with complex business decisions or problems
- 6. Advanced Analytics for us Handle Large, Complex Datasets Statistical Algorithms Visualizations, Heuristics Artificial Intelligence Machine Learning Prediction Machine
- 7. What languages should we choose from? Our Question so far ? ? ? ? ? ? ? ? ? ? ? ? ? ?
- 8. GitHub – Top Languages Javascript Java Python PHP C++ C# TypeScript Shell C Ruby Source:Octoverse2018
- 9. Stack Overflow – Top Languages Source:StackoverflowInsights2019
- 10. C C++ C# COBOL GO Java Julia Kotlin Python R Ruby Rust Scala SQL Our Question so far What languages should we choose from?
- 14. Flat Tables Related Tables CSV, TSV, fixed-width JSON, XML HTML, CSS O S E M N
- 15. Our Question so far What language is good for: - Querying related tables? C C++ C# COBOL GO Java Julia Kotlin Python R Ruby Rust Scala SQLSQL
- 16. O S E M N Flat Tables Related Tables CSV, TSV, fixed-width JSON, XML HTML, CSS
- 17. C C++ C# COBOL GO Java Julia Kotlin Python R Ruby Rust Scala SQL What language is good for: - Querying related tables? - Handling Excel, CSV, JSON, XML and scraping websites? Our Question so far COBOL SQLSQL
- 18. What we have What we need O S E M N
- 19. O S E M N Inconsistencies Alberta = AB Customer Province Wonka Industries Alberta Stark Industries AB Wayne Enterprises BC
- 20. O S E M N Categorical Data Customer Death By Wonka Industries Chocolate Stark Industries Plasma Burns Wayne Enterprises Multiple Contusions Customer Death By Chocolate Death by Plasma Burns Death by Multiple Contusions Wonka Industries 1 0 0 Stark Industries 0 1 0 Wayne Enterprises 0 0 1
- 21. O S E M N Flatten (Denormalize) Customer Province Wonka Industries Alberta Stark Industries AB Wayne Enterprises BC Customer Item Price Date Wonka Industries Toffee 5.00 2018-12-31 Stark Industries Iron 15.00 2018-03-30 Wayne Enterprises Vitamin D 25.00 2018-07-31 Wonka Industries Toffee 5.00 2019-01-04 Stark Industries Iron 15.00 2018-04-15 Wayne Enterprises Vitamin D 25.00 2018-08-01 Customer Death By Wonka Industries Chocolate Stark Industries Plasma Burns Wayne Enterprises Multiple Contusions
- 22. O S E M N Flatten (Denormalize) Customer Item Price Date Province Death By Wonka Industries Toffee 5.00 2018-12-31 Alberta Chocolate Stark Industries Iron 15.00 2018-03-30 AB Plasma Burns Wayne Enterprises Vitamin D 25.00 2018-07-31 BC Multiple Contusions Wonka Industries Toffee 5.00 2019-01-04 Alberta Chocolate Stark Industries Iron 15.00 2018-04-15 AB Plasma Burns Wayne Enterprises Vitamin D 25.00 2018-08-01 BC Multiple Contusions
- 23. C C++ C# COBOL GO Java Julia Kotlin Python R Ruby Rust Scala SQL What language is good for: - Querying related tables? - Handling Excel, CSV, JSON, XML and scraping websites? - Manipulating DataTables / DataFrames? Our Question so far C COBOL SQLSQL
- 24. Define or observe process Codify process Repeatable Outcome Verify Result O S E M N Programming an Application
- 25. Define or observe problem Experiment Observe Result Exp. Exp. Exp. Strong library of math algorithms and visualizations O S E M N Programming in Data Science Interactive (REPL) languages
- 26. C C++ C# COBOL GO Java Julia Kotlin Python R Ruby Rust Scala SQL What language is good for: - Querying related tables? - Handling Excel, CSV, JSON, XML and scraping websites? - Manipulating DataTables / DataFrames? - REPL Interactivity? - Libraries for math analysis and in-flight visualizations? Our Question so far Source:Wikipedia–InteractiveLanguages C C# COBOL GO Kotlin Rust C++ Java SQL
- 27. O S E M N O S E M N y = ax2 + bx + c y = 10x2 + 5x + 12
- 29. C C++ C# COBOL GO Java Julia Kotlin Python R Ruby Rust Scala SQL Our Question so far What language is good for: - Querying related tables? - Handling Excel, CSV, JSON, XML and scraping websites? - Manipulating DataTables / DataFrames? - REPL Interactivity? - Libraries for statistical analysis and in-flight visualizations? - Libraries for machine learning? - Distributed modeling on Spark? Python R Scala C C# COBOL GO Java Kotlin Rust C++ SQLSQL Ruby
- 31. Metcalfe’s Law The value of a network grows as the square of the number of its users Network Effects
- 36. Network Effects Source:StackoverflowTagsandGithubStars 7 years old 15 years old 25 years old 29 years old
- 37. C C++ C# COBOL GO Java Julia Kotlin Python R Ruby Rust Scala SQL Our Question so far What language is good for: - Querying related tables? - Handling Excel, CSV, JSON, XML and scraping websites? - Manipulating DataTables / DataFrames? - REPL Interactivity? - Libraries for statistical analysis and in-flight visualizations? - Distributed modeling on Spark? - Reducing the amount of time spent debugging and writing code that already exists? C C# COBOL GO Java Julia Kotlin Ruby Rust SQL C++ Python R Scala
- 39. Supply and Demand Source:Supply&DemandbyVilmosMüller Supply Demand Fulfillment Python 86% 34% 2.5 x R 38% 8% 4.8 x Scala <10% 12% 0.8 x
- 40. Source:StackoverflowInsights2019 Supply and Demand Cost Premium Python $63k baseline R $64k 0.01% Scala $78k 24%
- 41. C C++ C# COBOL GO Java Julia Kotlin Python R Ruby Rust Scala SQL Our Question What language is good for: - Querying related tables? - Handling Excel, CSV, JSON, XML and scraping websites? - Manipulating DataTables / DataFrames? - REPL Interactivity? - Libraries for statistical analysis and in-flight visualizations? - Distributed modeling on Spark? - Boosting productivity and efficiency? - Reducing the supply premium? Python R C C# COBOL GO Java Julia Kotlin Ruby Rust SQL C++ ScalaScala
- 42. Time Knowledge Learning Curve Fast Learning Curve Typical Learning Curve Practitioner Novice Expert Time Savings = Cost Savings
- 45. Our Question What language is good for: - Querying related tables? - Handling Excel, CSV, JSON, XML and scraping websites? - Manipulating DataTables / DataFrames? - REPL Interactivity? - Libraries for statistical analysis and in-flight visualizations? - Distributed modeling on Spark? - Boosting productivity and efficiency? - Reducing the supply premium? - Reducing training costs? Python R C C# COBOL GO Java Julia Kotlin Ruby Rust SQL C++ Scala R
- 46. Python
- 47. Solution
- 49. Python | SQL | Algorithms
- 52. Python | SQL | Algorithms Storytelling
Editor's Notes
- An individual can
- Led to the question, what is the right language for data science?
- We’ll start by exploring some technical aspects, then some economic ones … but first, some context
- Building a program, and building a team around the program. This is not about you as an individual. Will need to consider not just the current business problem, but the many business problems you will face.
- Needs refinement
- Underlying these are statistical analyses, algorithms, models, visualizations … anything that results in a prediction machine
- Based on Github and SO, here’s our list. Now I’ll ask your indulgence here, I’ve added Julia and Scala because they are highly relevant in the context of data science. And we can’t forget COBOL, which we are already starting to see was a mistake in my fictitious data science project. Let’s continue to build our question
- Remove anything dedicated to web-programming. Remove anything dedicated to shell scripting.
- Let’s get rid of: What we already cleared out Exclusive for web or mobile app Take everything above VBA because I hate VBA and I never want to talk about VBA again and I’ve already said VBA too many times in this sentence.
- Based on Github and SO, here’s our list. Now I’ll ask your indulgence here, I’ve added Julia and Scala because they are highly relevant in the context of data science. And we can’t forget COBOL, which we are already starting to see was a mistake in my fictitious data science project. Let’s continue to build our question
- …with a focus on some technical elements. We’ll use the OSEMN model from earlier to walk through the technical gauntlet.
- OSEMN model “Awesome” 2010 by Hilary Mason and Chris Wiggins Simplified, but it does a good job of capturing the essence of datasci http://www.dataists.com/2010/09/a-taxonomy-of-data-science/ https://towardsdatascience.com/5-steps-of-a-data-science-project-lifecycle-26c50372b492 https://medium.com/@randylaosat/life-of-data-data-science-is-osemn-f453e1febc10
- OBTAIN Although there are many datasets obtained from APIs and from scraping websites, the vast majority still comes from databases that house the data in a structured form. These might be application databases, ODS, data warehouses, semantic layers ... Regardless, they’re treated as structured databases Databases contain almost all of our contextual (reference) data, and almost all of our industry-secret data Websites contain a wealth of data when trying to extrapolate information that the world, in general, has to offer
- SQL was designed to query tables! In fact, most languages have abstraction libraries that allow you to write SQL or almost-SQL … and most of those are translated into SQL when executed against databases. It is the de facto standard for extracting data from databases, and this point must not be understated.
- All of them can handle Excel / CSV / JSON / XML, even COBOL! Not a helpful question to ask. Let’s ignore it and carry on.
- Reduce Clean Transform Categorize / Label Observe / Take notes … what might be a good feature? What is unnecessary noise? What might be an outcome?
- 1-hot encoding / binarize If we simply provide a numerical category, then the average of “Chocolate” and “Multiple Contusions” = “Plasma Burns”
- http://elitedatascience.com/data-cleaning Reduce to what you need Remove outliers Handle missing data
- Highlight SQL for O and S… it’s so valuable that in the very early days of big data, SQL interpreters were quintessential to adoption. This is a show-stopper. If there aren’t native objects or generally accepted libraries that help a language manage data native as a table, then there’s nowhere to go. C is really close to bare metal (i.e. low-level language), making it non-ideal. It’s possible, not pragmatic.
- Very All about workflows
- Could write our own libraries, but this is an immensely costly effort… and our objective is make this a cost-effective team / program. Read – Evaluate – Print – Loop https://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop https://en.wikipedia.org/wiki/List_of_programming_languages_by_type#Interactive_mode_languages
- What is modeling? A trained model = a populated algorithm
- Knowing the algorithm and the purpose; Applying the right one to the problem at hand; Training a model Testing the model’s validity Tuning the parameters Training and testing need large datasets Some algorithms are complex and need mucho data These require a distributed environment Distributed data frames The person filling this role needs the skill of knowing which algorithm are available, and when to apply the appropriate ones
- Julia could, but it’s still really, really new. We are now down to the most relevant languages for data science. For anyone who’s familiar with the field, this is where the question gets difficult. Good time to call out productionalization of a trained model… can rewrite it in a low-level language for efficiency, or can scale the solution in the cloud. I’ve intentionally ignored that
- Let’s see if we can use some economic principles to help us expand our question. Network Effects Supply / Demand Learning Curve
- Network effect (value of X is amplified by Y connected nodes) Number of developers that know the language (SO survey + google searches) Number of google pages / SO answers / github libraries in the language Number of libraries Number of developers
- Network effect (value of X is amplified by Y connected nodes) Number of developers that know the language Number of SO answers Number of github libraries in the language
- Observations: Python’s network with frameworks Python’s response size (network of people using it) Relationship to data-science frameworks Pandas, PyTorch, Tensorflow All interlinked with Jupyter acting as a node
- Observations: Python’s network with frameworks Python’s response size (network of people using it) Relationship to data-science frameworks Pandas, PyTorch, Tensorflow All interlinked with Jupyter acting as a node
- Y: Github Repos (including data-sci / machine-learning libraries, categorized by target language) X: StackOverflow questions (including libraries, categorized by target language) Bubblesize: Language popularity Why no SQL?
- I don’t have to re-invent what I can re-use Someone else is bound to have hit the problem I’m facing
- Data based on Kaggle datasets including the 2018 Kaggle survey and a job-demand dataset. We can’t fulfill Scala … in economic theory, if supply is below demand, we have to pay a premium to get it. This is really important, let’s validate this.
- Here’s SO’s pay-by-technology breakdown. Let’s zoom in on the relevant entries Note: Doesn’t account for cross-training.
- Also, timely staffing when turnover occurs, and reduced poaching
- On the fast learning curve, we get to being a practitioner much faster. Even if reaching expert takes around the same time, the developer can be useful much sooner. The faster something can be learned: The lesser the up front cost; The lower the barrier to entry; The greater the adoption Leading to an amplified network effect Virtuous cycle!
- The faster something can be learned: The lesser the up front cost; The lower the barrier to entry; The greater the adoption Leading to an amplified network effect Virtuous cycle! https://www.codingdojo.com/blog/python-perfect-beginners
- The faster something can be learned: The lesser the up front cost; The lower the barrier to entry; The greater the adoption Leading to an amplified network effect Virtuous cycle! https://www.codingdojo.com/blog/python-perfect-beginners
- Don’t need a homogeneous team! Remember! Not mutually exclusive! Depending on the size of the team and the problem at hand … team makeup can vary significantly Technical conclusion: Roles <-> Languages and Knowledges Which language has the combination of features and most pliable across the data science process
- Our team, as a team, needs to know The syntax, patterns, principles and utilization of Python To understand which algorithms are appropriate to the problem
- Remember the OSEMN model? We didn’t talk about the last step – Interpreting! This is what makes it real for stakeholders. If you can’t explain what you did, why you did it, and what the results imply … then it was all for nought.
- TRUST Stakeholders and users of our model want to trust it. If they don’t understand it, they don’t trust it. What data did we obtain? What did we do to scrub it? Why did we choose these algorithms, and this training data? What biases could remain? Under what conditions does this start to break down?
- The answer is actually this Story telling is clear communication in a natural language (English). I hope you have enjoyed my storytelling today. Thank you.
- Interesting chart: Stackoverflow ^ | | |______________> Github
- https://www.tiobe.com/tiobe-index/
- https://github.blog/2019-01-24-the-state-of-the-octoverse-machine-learning/
- https://insights.stackoverflow.com/survey/2019#technology