Paco Nathan, profile picture
Uploaded byPaco Nathan
PDF, PPTX31,704 views

Data Workflows for Machine Learning - Seattle DAML

The document discusses the evolution and significance of data workflows for machine learning in production applications, emphasizing the importance of frameworks over intricate algorithms. It highlights a variety of popular frameworks and methodologies for building effective data workflows, detailing their capabilities and features. Additionally, it proposes a 'scorecard' for evaluating workflow tools based on their suitability for specific machine learning use cases.

Embed presentation

Download as PDF, PPTX
Data Workflows for Machine Learning: 
 Seattle,WA 2014-01-29 ! Paco Nathan
 @pacoid http://liber118.com/pxn/ meetup.com/Seattle-DAML/events/159043422/
Why is this talk here? Machine Learning in production apps is less and less about algorithms (even though that work is quite fun and vital) ! Performing real work is more about: ! socializing a problem within an organization ! feature engineering (“Beyond Product Managers”) ! tournaments in CI/CD environments ! operationalizing high-ROI apps at scale ! etc. ! So I’ll just crawl out on a limb and state that leveraging great frameworks to build data workflows is more important than 
 chasing after diminishing returns on highly nuanced algorithms. ! Because Interwebs!
Data Workflows for Machine Learning Middleware has been evolving for Big Data, and there are some great examples — we’ll review several. Process has been evolving too, right along with the use cases. ! Popular frameworks typically provide some Machine Learning capabilities within their core components, or at least among their major use cases. ! Let’s consider features from Enterprise Data Workflows as a basis 
 for what’s needed in Data Workflows for Machine Learning. ! Their requirements for scale, robustness, cost trade-offs, interdisciplinary teams, etc., serve as guides in general.
Caveat Auditor I won’t claim to be expert with each of the frameworks and environments described in this talk. Expert with a few of them perhaps, but more to the point: embroiled in many use cases. ! This talk attempts to define a “scorecard” for evaluating important ML data workflow features: 
 what’s needed for use cases, compare and contrast of what’s available, plus some indication of which frameworks are likely 
 to be best for a given scenario. ! Seriously, this is a work in progress.
Outline • Definition: Machine Learning • Definition: Data Workflows • A whole bunch o’ examples across several platforms • Nine points to discuss, leading up to a scorecard • A crazy little thing called PMML • Questions, comments, flying tomatoes…
Data Workflows for Machine Learning: 
 Frame the question… ! 
 “A Basis for What’s Needed”
Definition: Machine Learning “Machine learning algorithms can figure out how to perform
 important tasks by generalizing from examples.This is often
 feasible and cost-effective where manual programming is not. 
 As more data becomes available, more ambitious problems 
 can be tackled. As a result, machine learning is widely used 
 in computer science and other fields.” Pedro Domingos, U Washington
 A Few Useful Things to Know about Machine Learning [learning as generalization] =
 [representation] + [evaluation] + [optimization] • • • overfitting (variance) underfitting (bias) “perfect classifier” (no free lunch)
Definition: Machine Learning … Conceptually
 ! ! 1. real-world data 2. graph theory for representation 3. convert to sparse matrix for production work 
 leveraging abstract algebra + func programming 4. cost-effective parallel processing for ML apps 
 at scale, live use cases
Definition: Machine Learning … Conceptually
 ! ! f(x): loss function g(z): regularization term 1. real-world data [generalization] 2. graph theory for representation [representation] 3. convert to sparse matrix for production work 
 leveraging abstract algebra + func programming [evaluation] 4. cost-effective parallel processing for ML apps 
 at scale, live use cases [optimization]
Definition: Machine Learning … Interdisciplinary Teams, Needs × Roles very very sco iisco d d ng lliing ode ode m m n n atiio at o tegr tegr n iin pps pps a a s s tem tem sys sys business process, stakeholder Domain Expert data science Data Scientist App Dev Ops data prep, discovery, modeling, etc. software engineering, automation systems engineering, availability
Definition: Machine Learning … Process, Feature Engineering feature engineering data data sources data sources sources data prep pipeline hold-outs train learners classifiers classifiers classifiers scoring test representation evaluation optimization use cases
Definition: Machine Learning … Process, Tournaments feature engineering data data sources data sources sources data prep pipeline tournaments hold-outs train learners classifiers classifiers classifiers scoring test representation • obtain other data? • improve metadata? • refine representation? • improve optimization? evaluation • iterate with stakeholders? • can models (inference) inform first principles approaches? optimization use cases quantify and measure: • benefit? • risk? • operational costs?
Definition: Machine Learning … Invasion of the Monoids monoid (an alien life form, courtesy of abstract algebra): ! binary associative operator ! closure within a set of objects ! unique identity element what are those good for? ! ! ! ! ! composable functions in workflows compiler “hints” on steroids, to parallelize reassemble results minimizing bottlenecks at scale reusable components, like Lego blocks think: Docker for business logic Monoidify! Monoids as a Design Principle for Efficient MapReduce Algorithms
 Jimmy Lin, U Maryland/Twitter kudos to Oscar Boykin, Sam Ritchie, et al.
Definition: Machine Learning feature engineering data data sources data sources sources data prep pipeline tournaments hold-outs train learners classifiers classifiers classifiers scoring test representation • obtain other data? • improve metadata? • refine representation? • improve optimization? evaluation optimization • iterate with stakeholders? quantify and measure: • benefit? • risk? • operational costs? • can models (inference) inform first principles approaches? y y over over diisc d sc elliing e ng mod mod use cases n on ratiio grat nteg iinte apps apps ems ems syst syst Domain Expert data science Data Scientist App Dev Ops introduced capability
Definition feature engineering data data sources data sources sources data prep pipeline tournaments hold-outs train learners classifiers classifiers classifiers scoring test Can we fit the required 
 process into generalized 
 workflow definitions? representation • obtain other data? • improve metadata? • refine representation? • improve optimization? evaluation optimization • iterate with stakeholders? quantify and measure: • benefit? • risk? • operational costs? • can models (inference) inform first principles approaches? y y over over diisc d sc elliing e ng mod mod use cases n on ratiio grat nteg iinte apps apps ems ems syst syst Domain Expert data science Data Scientist App Dev Ops introduced capability
Definition: Data Workflows Middleware, effectively, is evolving for Big Data and Machine Learning…
 The following design pattern — a DAG — shows up in many places, 
 via many frameworks: ETL data sources data prep predictive model end uses
Definition: Data Workflows definition of a typical Enterprise workflow which crosses through multiple departments, languages, and technologies… ANSI SQL for ETL ETL data sources data prep predictive model end uses
Definition: Data Workflows definition of a typical Enterprise workflow which crosses through multiple departments, languages, and technologies… ETL data sources data prep Java, Pig for business logic predictive model end uses
Definition: Data Workflows definition of a typical Enterprise workflow which crosses through multiple departments, languages, and technologies… SAS for predictive models
 ETL data sources data prep predictive model end uses
Definition: Data Workflows definition of a typical Enterprise workflow which crosses through multiple departments, languages, and technologies… ANSI SQL for ETL most of the licensing costs… ETL data sources data prep SAS for predictive models predictive model end uses
Definition: Data Workflows definition of a typical Enterprise workflow which crosses through multiple departments, languages, and technologies… most of the project costs… ETL data sources data prep Java, Pig for business logic predictive model end uses
Definition: Data Workflows definition of a typical Enterprise workflow which crosses through multiple departments, languages, and technologies… most of the project costs… Something emerges 
 to fill these needs, 
 for instance… ETL data sources data prep Java, Pig for business logic predictive model end uses
Definition: Data Workflows For example, Cascading and related projects implement the following components, based on 100% open source: Lingual: DW → ANSI SQL ETL data sources business logic in Java, Clojure, Scala, etc. data prep a compiler sees it all…
 one connected DAG: • troubleshooting cascading.org Pattern: SAS, R, etc. → PMML predictive model end uses • exception handling source taps for Cassandra, JDBC, Splunk, etc. • notifications • some optimizations sink taps for Memcached, HBase, MongoDB, etc.
Definition: Data Workflows Cascading allows multiple departments to combine their workflow components into an integrated app – one among many, typically – based on 100% open source Lingual: DW → ANSI SQL cascading.org ETL data sources source taps for Cassandra, JDBC, Splunk, etc. business logic in Java, ! Clojure, Scala, etc. Pattern: SAS, R, etc. → PMML ! FlowDef flowDef = FlowDef.flowDef()! .setName( "etl" )! predictive data .addSource( "example.employee", emplTap )! model prep .addSource( "example.sales", salesTap )! .addSink( "results", resultsTap );!  ! SQLPlanner sqlPlanner = new SQLPlanner()! .setSql( sqlStatement );! end uses  ! flowDef.addAssemblyPlanner( sqlPlanner );! ! sink taps for ! Memcached, HBase, MongoDB, etc.
Definition: Data Workflows Cascading allows multiple departments to combine their workflow components into an integrated app – one among many, typically – based on 100% open source Lingual: DW → ANSI SQL business logic in Java, Clojure, Scala, etc. ! ! FlowDef flowDef = FlowDef.flowDef()! .setName( "classifier" )! .addSource( "input", inputTap )! data ETL .addSink( "classify", classifyTap prep );!  ! PMMLPlanner pmmlPlanner = new PMMLPlanner()! .setPMMLInput( new File( pmmlModel ) )! data .retainOnlyActiveIncomingFields();! sources  ! flowDef.addAssemblyPlanner( pmmlPlanner );! ! source taps for ! Cassandra, JDBC, Splunk, etc. Pattern: SAS, R, etc. → PMML predictive model end uses sink taps for Memcached, HBase, MongoDB, etc.
Enterprise Data Workflows with Cascading O’Reilly, 2013 shop.oreilly.com/product/ 0636920028536.do ETL data sources data prep predictive model end uses
Enterprise Data Workflows with Cascading O’Reilly, 2013 shop.oreilly.com/product/ 0636920028536.do This begs an update… ETL data sources data prep predictive model end uses
Enterprise Data Workflows with Cascading O’Reilly, 2013 shop.oreilly.com/product/ 0636920028536.do Because… ETL data sources data prep predictive model end uses
Data Workflows for Machine Learning: 
 Examples… ! 
 “Neque per exemplum”
Example: KNIME “a user-friendly graphical workbench for the entire analysis
 process: data access, data transformation, initial investigation,
 powerful predictive analytics, visualisation and reporting.” • • • • • • large number of integrations (over 1000 modules) ranked #1 in customer satisfaction among 
 open source analytics frameworks visual editing of reusable modules leverage prior work in R, Perl, etc. Eclipse integration easily extended for new integrations knime.org
Example: Python stack Python has much to offer – ranging across an 
 organization, no just for the analytics staff • • • • • • • • • ipython.org pandas.pydata.org scikit-learn.org numpy.org scipy.org code.google.com/p/augustus continuum.io nltk.org matplotlib.org
Example: Julia julialang.org “a high-level, high-performance dynamic programming language
 for technical computing, with syntax that is familiar to users of
 other technical computing environments” • • • significantly faster than most alternatives built to leverage parallelism, cloud computing still relatively new — one to watch! importall Base! ! type ! BubbleSort <: Sort.Algorithm end! function sort!(v::AbstractVector, lo::Int, hi::Int, ::BubbleSort, o::Sort.Ordering)!     while true!         clean = true!         for i = lo:hi-1!             if Sort.lt(o, v[i+1], v[i])!                 v[i+1], v[i] = v[i], v[i+1]!                 clean = false!             end!         end!         clean && break!     end!     return v! end!
Example: Summingbird github.com/twitter/summingbird “a library that lets you write streaming MapReduce programs
 that look like native Scala or Java collection transformations 
 and execute them on a number of well-known distributed
 MapReduce platforms like Storm and Scalding.” • switch between Storm, Scalding (Hadoop) • Spark support is in progress • leverage Algebird, Storehaus, Matrix API, etc. def wordCount[P <: Platform[P]]! (source: Producer[P, String], store: P#Store[String, Long]) =! source.flatMap { sentence => ! toWords(sentence).map(_ -> 1L)! }.sumByKey(store)
Example: Scalding import com.twitter.scalding._!  ! class WordCount(args : Args) extends Job(args) {! Tsv(args("doc"),! ('doc_id, 'text),! skipHeader = true)! .read! .flatMap('text -> 'token) {! text : String => text.split("[ [](),.]")! }! .groupBy('token) { _.size('count) }! .write(Tsv(args("wc"), writeHeader = true))! } github.com/twitter/scalding
Example: Scalding • extends the Scala collections API so that distributed lists become “pipes” backed by Cascading • code is compact, easy to understand • nearly 1:1 between elements of conceptual flow diagram 
 and function calls • extensive libraries are available for linear algebra, abstract algebra, machine learning – e.g., Matrix API, Algebird, etc. • significant investments by Twitter, Etsy, eBay, etc. • less learning curve than Cascalog • build scripts… those take a while to run : github.com/twitter/scalding
Example: Cascalog (ns impatient.core!   (:use [cascalog.api]!         [cascalog.more-taps :only (hfs-delimited)])!   (:require [clojure.string :as s]!             [cascalog.ops :as c])!   (:gen-class))! ! (defmapcatop split [line]!   "reads in a line of string and splits it by regex"!   (s/split line #"[[](),.)s]+"))! ! (defn -main [in out & args]!   (?<- (hfs-delimited out)!        [?word ?count]!        ((hfs-delimited in :skip-header? true) _ ?line)!        (split ?line :> ?word)!        (c/count ?count)))! ! ; Paul Lam! ; github.com/Quantisan/Impatient cascalog.org
Example: Cascalog • implements Datalog in Clojure, with predicates backed 
 by Cascading – for a highly declarative language • run ad-hoc queries from the Clojure REPL –
 approx. 10:1 code reduction compared with SQL • composable subqueries, used for test-driven development (TDD) practices at scale • Leiningen build: simple, no surprises, in Clojure itself • more new deployments than other Cascading DSLs – 
 Climate Corp is largest use case: 90% Clojure/Cascalog • has a learning curve, limited number of Clojure developers • aggregators are the magic, and those take effort to learn cascalog.org
Example: Apache Spark spark-project.org in-memory cluster computing, 
 by Matei Zaharia, Ram Venkataraman, et al. • • • • • intended to make data analytics fast to write and to run load data into memory and query it repeatedly, much more quickly than with Hadoop APIs in Scala, Java and Python, shells in Scala and Python Shark (Hive on Spark), Spark Streaming (like Storm), etc. integrations with MLbase, Summingbird (in progress) // word count! val file = spark.textFile(“hdfs://…”)! val counts = file.flatMap(line => line.split(” “))! .map(word => (word, 1))! .reduceByKey(_ + _)! counts.saveAsTextFile(“hdfs://…”)
Example: MLbase “distributed machine learning made easy” • • • • • sponsored by UC Berkeley EECS AMP Lab MLlib – common algorithms, low-level, written atop Spark MLI – feature extraction, algorithm dev atop MLlib ML Optimizer – automates model selection, 
 compiler/optimizer see article:
 http://strata.oreilly.com/2013/02/mlbase-scalablemachine-learning-made-accessible.html ! data = load("hdfs://path/to/als_clinical")! // the features are stored in columns 2-10! X = data[, 2 to 10]! y = data[, 1]! model = do_classify(y, X) mlbase.org
Example: Titan thinkaurelius.github.io/titan distributed graph database, 
 by Matthias Broecheler, Marko Rodriguez, et al. • • • • • • scalable graph database optimized for storing and querying graphs supports hundreds of billions of vertices and edges transactional database that can support thousands of concurrent users executing complex graph traversals supports search through Lucene, ElasticSearch can be backed by HBase, Cassandra, BerkeleyDB TinkerPop native graph stack with Gremlin, etc. // who is hercules' paternal grandfather?! g.V('name','hercules').out('father').out('father').name
Example: MBrace “a .NET based software stack that enables easy large-scale
 distributed computation and big data analysis.” • declarative and concise distributed algorithms in 
 F# asynchronous workflows • scalable for apps in private datacenters or public 
 clouds, e.g., Windows Azure or Amazon EC2 • tooling for interactive, REPL-style deployment, monitoring, management, debugging in Visual Studio • leverages monads (similar to Summingbird) • MapReduce as a library, but many patterns beyond m-brace.net let rec mapReduce (map: 'T -> Cloud<'R>)! (reduce: 'R -> 'R -> Cloud<'R>)! (identity: 'R)! (input: 'T list) =! cloud {! match input with! | [] -> return identity! | [value] -> return! map value! | _ ->! let left, right = List.split input! let! r1, r2 =! (mapReduce map reduce identity left)! <||>! (mapReduce map reduce identity right)! return! reduce r1 r2! }
Data Workflows for Machine Learning: 
 Update the definitions… ! “Nine Points to Discuss”

Data Workflows Formally speaking, workflows include people (cross-dept) 
 and define oversight for exceptional data 1 …otherwise, data flow or pipeline would be more apt …examples include Cascading traps, with exceptional tuples 
 routed to a review organization, e.g., Customer Support Customers Web App logs logs Logs Cache Support trap tap Modeling PMML source tap Data Workflow source tap sink tap Analytics Cubes Reporting sink tap customer Customer profile DBs Prefs Hadoop Cluster includes people, defines oversight for exceptional data
Data Workflows Workflows impose a separation of concerns, allowing 
 for multiple abstraction layers in the tech stack …specify what is required, not how to accomplish it …articulating the business logic … Cascading leverages pattern language …related notions from Knuth, literate programming …not unlike BPM/BPEL for Big Data …examples: IPython, R Markdown, etc. 2 separation of concerns, allows for literate programming
3 Data Workflows Multiple abstraction layers in the tech stack are 
 needed, but still emerging …feedback loops based on machine data …optimizers feed on abstraction Portable Models …metadata accounting: Reusable Components track data lineage • propagate schema • model / feature usage • ensemble performance …app history accounting: • • • • util stats, mixing workloads heavy-hitters, bottlenecks throughput, latency Business Process multiple abstraction layers 
 for metadata, feedback, 
 and optimization • • • • data lineage schema propagation feature selection tournaments DSLs Planners/ Optimizers Mixed Topologies Cluster Scheduler • app history • util stats • bottlenecks Machine Data Clusters
3 Data Workflows Multiple needed, but still emerging Business Process …feedback loops based on …optimizers feed on abstraction Portable Models Um, will compilers 
 track data lineage propagate ever look like this?? model / feature usage …metadata accounting: • • • Reusable Components ensemble performance …app history accounting: • • • • • • data lineage schema propagation feature selection tournaments DSLs • • multiple abstraction layers 
 for metadata, feedback, 
 and optimization util stats, mixing workloads heavy-hitters, bottlenecks throughput, latency Planners/ Optimizers Mixed Topologies Cluster Scheduler • app history • util stats • bottlenecks Machine Data Clusters
4 Data Workflows Workflows must provide for test, which is no simple 
 matter testing: model evaluation, TDD, app deployment …testing is required on three abstraction layers: model portability/evaluation, ensembles, tournaments • TDD in reusable components and business process • continuous integration / continuous deployment …examples: Cascalog for TDD, PMML for model eval • …still so much more to improve Business Process Portable Models Reusable Components • • • • data lineage schema propagation feature selection tournaments DSLs …keep in mind that workflows involve people, too Planners/ Optimizers Mixed Topologies Cluster Scheduler • app history • util stats • bottlenecks Machine Data Clusters
5 Data Workflows Workflows enable system integration …future-proof integrations and scale-out future-proof system integration, scale-out, ops …build components and apps, not jobs and command lines …allow compiler optimizations across the DAG,
 i.e., cross-dept contributions Customers Web App …minimize operationalization costs: troubleshooting, debugging at scale • exception handling • notifications, instrumentation …examples: KNIME, Cascading, etc. • logs logs Logs Cache Support trap tap Modeling ! PMML source tap Data Workflow source tap sink tap Analytics Cubes Reporting sink tap customer Customer profile DBs Prefs Hadoop Cluster
Data Workflows Visualizing workflows, what a great idea. …the practical basis for: collaboration • rapid prototyping • component reuse …examples: KNIME wins best in category • …Cascading generates flow diagrams, 
 which are a nice start 6 visualizing allows people 
 to collaborate through code
Data Workflows Abstract algebra, containerizing workflow metadata …monoids, semigroups, etc., allow for reusable components 
 with well-defined properties for running in parallel at scale …let business process be agnostic about underlying topologies, 
 with analogy to Linux containers (Docker, etc.) …compose functions, take advantage of sparsity (monoids) …because “data is fully functional” – FP for s/w eng benefits …aggregators are almost always “magic”, now we can solve 
 for common use cases …read Monoidify! Monoids as a Design Principle for Efficient MapReduce Algorithms by Jimmy Lin …Cascading introduced some notions of this circa 2007, 
 but didn’t make it explicit …examples: Algebird in Summingbird, Simmer, Spark, etc. 7 abstract algebra and functional programming containerize business process
Data Workflows Workflow needs vary in time, and need to blend time …something something batch, something something low-latency 8 blend results from different time scales: batch plus lowlatency …scheduling batch isn’t hard; scheduling low-latency is 
 computationally brutal (see the Omega paper) …because people like saying “Lambda Architecture”, 
 it gives them goosebumps, or something …because real-time means so many different things …because batch windows are so 1964 …examples: Summingbird, Oryx Big Data
 Nathan Marz, James Warren
 manning.com/marz
Data Workflows Workflows may define contexts in which model selection 
 possibly becomes a compiler problem …for example, see Boyd, Parikh, et al. …ultimately optimizing for loss function + regularization term …probably not ready for prime-time tomorrow …examples: MLbase f(x): loss function g(z): regularization term 9 optimize learners in context, to make model selection potentially a compiler problem
Data Workflows For one of the best papers about what workflows really
 truly require, see Out of the Tar Pit by Moseley and Marks bonus
Nine Points for Data Workflows — a wish list 1. includes people, defines oversight for exceptional data 2. separation of concerns, allows for literate programming 3. multiple abstraction layers for metadata, feedback, and optimization Business Process Portable Models Reusable Components 4. testing: model evaluation, TDD, app deployment 6. visualizing workflows allows people to collaborate through code Planners/ Optimizers Mixed Topologies 7. abstract algebra and functional programming containerize business process 9. optimize learners in context, to make model 
 selection potentially a compiler problem data lineage schema propagation feature selection tournaments DSLs 5. future-proof system integration, scale-out, ops 8. blend results from different time scales: 
 batch plus low-latency • • • • Cluster Scheduler • app history • util stats • bottlenecks Machine Data Clusters
Nine Points for Data Workflows — a scorecard Spark, MLbase Oryx Summing! bird Cascalog Cascading ! KNIME Py Data R Markdown ✔ ✔ ✔ ✔ includes people, exceptional data separation of concerns multiple 
 abstraction layers testing in depth future-proof 
 system integration visualize to collab can haz monoids blends batch + 
 “real-time” optimize learners 
 in context can haz PMML ✔ MBrace
Data Workflows for Machine Learning: 
 PMML… ! 
 “Oh, by the way…”
PMML – an industry standard • • established XML standard for predictive model markup • members: IBM, SAS, Visa, FICO, Equifax, NASA,
 Microstrategy, Microsoft, etc. • PMML concepts for metadata, ensembles, etc., translate 
 directly into workflow abstractions, e.g., Cascading organized by Data Mining Group (DMG), since 1997 
 http://dmg.org/ ! “PMML is the leading standard for statistical and data mining models and
 supported by over 20 vendors and organizations.With PMML, it is easy 
 to develop a model on one system using one application and deploy the
 model on another system using another application.” wikipedia.org/wiki/Predictive_Model_Markup_Language
PMML – vendor coverage
PMML – model coverage • • • • • • • • • • • Association Rules: AssociationModel element Cluster Models: ClusteringModel element Decision Trees: TreeModel element Naïve Bayes Classifiers: NaiveBayesModel element Neural Networks: NeuralNetwork element Regression: RegressionModel and GeneralRegressionModel elements Rulesets: RuleSetModel element Sequences: SequenceModel element Support Vector Machines: SupportVectorMachineModel element Text Models: TextModel element Time Series: TimeSeriesModel element ibm.com/developerworks/industry/library/ind-PMML2/
PMML – further study ! PMML in Action
 Alex Guazzelli, Wen-Ching Lin, Tridivesh Jena
 amazon.com/dp/1470003244 ! See also excellent resources at: zementis.com/pmml.htm
Pattern – model scoring • • • • • a PMML library for Cascading workflows Customers migrate workloads: SAS,Teradata, etc., 
 exporting predictive models as PMML Web App great open source tools – R, Weka, 
 KNIME, Matlab, RapidMiner, etc. leverage PMML as another kind of DSL logs logs Logs Cache Support only for scoring models, not training trap tap Modeling PMML source tap Data Workflow source tap sink tap Analytics Cubes Reporting sink tap customer Customer profile DBs Prefs Hadoop Cluster
Pattern – model scoring • originally intended as a general purpose, tuple-oriented model scoring engine for JVM-based frameworks • library plugs into Cascading (Hadoop), and ostensibly Cascalog, Scalding, Storm, Spark, Summingbird, etc. • dev forum:
 https://groups.google.com/forum/#!forum/pattern-user • original GitHub repo:
 https://github.com/ceteri/pattern/ • somehow, the other fork became deeply intertwined 
 with Cascading dependencies… • work in progress to integrate with other frameworks, 
 add models, and also train models at scale using Spark
Pattern – create a model in R ## train a RandomForest model!  ! f <- as.formula("as.factor(label) ~ .")! fit <- randomForest(f, data_train, ntree=50)!  ! ## test the model on the holdout test set!  ! print(fit$importance)! print(fit)!  ! predicted <- predict(fit, data)! data$predicted <- predicted! confuse <- table(pred = predicted, true = data[,1])! print(confuse)!  ! ## export predicted labels to TSV!  ! write.table(data, file=paste(dat_folder, "sample.tsv", sep="/"), 
 quote=FALSE, sep="t", row.names=FALSE)!  ! ## export RF model to PMML!  ! saveXML(pmml(fit), file=paste(dat_folder, "sample.rf.xml", sep="/"))!
Pattern – capture model parameters as PMML <?xml version="1.0"?>! <PMML version="4.0" xmlns="http://www.dmg.org/PMML-4_0"!  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"!  xsi:schemaLocation="http://www.dmg.org/PMML-4_0
 http://www.dmg.org/v4-0/pmml-4-0.xsd">!  <Header copyright="Copyright (c)2012 Concurrent, Inc." description="Random Forest Tree Model">!   <Extension name="user" value="ceteri" extender="Rattle/PMML"/>!   <Application name="Rattle/PMML" version="1.2.30"/>!   <Timestamp>2012-10-22 19:39:28</Timestamp>!  </Header>!  <DataDictionary numberOfFields="4">!   <DataField name="label" optype="categorical" dataType="string">!    <Value value="0"/>!    <Value value="1"/>!   </DataField>!   <DataField name="var0" optype="continuous" dataType="double"/>!   <DataField name="var1" optype="continuous" dataType="double"/>!   <DataField name="var2" optype="continuous" dataType="double"/>!  </DataDictionary>!  <MiningModel modelName="randomForest_Model" functionName="classification">!   <MiningSchema>!    <MiningField name="label" usageType="predicted"/>!    <MiningField name="var0" usageType="active"/>!    <MiningField name="var1" usageType="active"/>!    <MiningField name="var2" usageType="active"/>!   </MiningSchema>!   <Segmentation multipleModelMethod="majorityVote">!    <Segment id="1">!     <True/>!     <TreeModel modelName="randomForest_Model" functionName="classification" algorithmName="randomForest" splitCharacteristic="binarySplit">!      <MiningSchema>!       <MiningField name="label" usageType="predicted"/>!       <MiningField name="var0" usageType="active"/>!       <MiningField name="var1" usageType="active"/>!       <MiningField name="var2" usageType="active"/>!      </MiningSchema>! ...!
Pattern – score a model, within an app public static void main( String[] args ) throws RuntimeException {! String inputPath = args[ 0 ];! String classifyPath = args[ 1 ];! // set up the config properties! Properties properties = new Properties();! AppProps.setApplicationJarClass( properties, Main.class );! HadoopFlowConnector flowConnector = new HadoopFlowConnector( properties );!   // create source and sink taps! Tap inputTap = new Hfs( new TextDelimited( true, "t" ), inputPath );! Tap classifyTap = new Hfs( new TextDelimited( true, "t" ), classifyPath );!   // handle command line options! OptionParser optParser = new OptionParser();! optParser.accepts( "pmml" ).withRequiredArg();!   OptionSet options = optParser.parse( args );!  ! // connect the taps, pipes, etc., into a flow! FlowDef flowDef = FlowDef.flowDef().setName( "classify" )! .addSource( "input", inputTap )! .addSink( "classify", classifyTap );!  ! if( options.hasArgument( "pmml" ) ) {! String pmmlPath = (String) options.valuesOf( "pmml" ).get( 0 );! PMMLPlanner pmmlPlanner = new PMMLPlanner()! .setPMMLInput( new File( pmmlPath ) )! .retainOnlyActiveIncomingFields()! .setDefaultPredictedField( new Fields( "predict", Double.class ) ); // default value if missing from the model! flowDef.addAssemblyPlanner( pmmlPlanner );! }!  ! // write a DOT file and run the flow! Flow classifyFlow = flowConnector.connect( flowDef );! classifyFlow.writeDOT( "dot/classify.dot" );! classifyFlow.complete();! }
Pattern – score a model, using pre-defined Cascading app Customer Orders Classify Scored Orders Assert GroupBy token M R PMML Model Count Failure Traps Confusion Matrix
Pattern – score a model, using pre-defined Cascading app ## run an RF classifier at scale!  ! hadoop jar build/libs/pattern.jar ! data/sample.tsv out/classify out/trap ! --pmml data/sample.rf.xml!  ! !
Data Workflows for Machine Learning: 
 Summary ! 
 “feedback and discussion” ?
Nine Points for Data Workflows — a scorecard Spark, MLbase Oryx Summing! bird Cascalog Cascading ! KNIME Py Data R Markdown ✔ ✔ ✔ ✔ includes people, exceptional data separation of concerns multiple 
 abstraction layers testing in depth future-proof 
 system integration visualize to collab can haz monoids blends batch + 
 “real-time” optimize learners 
 in context can haz PMML ✔ MBrace
PMML – what’s needed? in the language standard: ! data preparation ! data sources/sinks as parameters ! track data lineage ! more monoid, less XML ! handle data exceptions => TDD ! updates/active learning? feature engineering data data sources data sources sources data prep pipeline tournaments hold-outs train learners classifiers classifiers classifiers ! tournaments? scoring test representation evaluation optimization use cases ! in the workflow frameworks: ! include feature engineering w/ model? ! support evaluation better ! build ensembles • obtain other data? • improve metadata? • refine representation? • improve optimization? • iterate with stakeholders? • can models (inference) inform first principles approaches? quantify and measure: • benefit? • risk? • operational costs?
Enterprise Data Workflows with Cascading O’Reilly, 2013 shop.oreilly.com/product/ 0636920028536.do ! monthly newsletter for updates, events, conference summaries, etc.: liber118.com/pxn/

More Related Content

PDF
10 Lessons Learned from Building Machine Learning Systems
byXavier Amatriain
 
PDF
Square's Machine Learning Infrastructure and Applications - Rong Yan
byHakka Labs
 
PDF
OSCON 2014: Data Workflows for Machine Learning
byPaco Nathan
 
PDF
Data Workflows for Machine Learning - SF Bay Area ML
byPaco Nathan
 
PPTX
Machine learning 101 dkom 2017
byfredverheul
 
PDF
Azure Machine Learning
byMostafa
 
PPTX
Production and Beyond: Deploying and Managing Machine Learning Models
byTuri, Inc.
 
PDF
Machine learning the high interest credit card of technical debt [PWL]
byJenia Gorokhovsky
 
10 Lessons Learned from Building Machine Learning Systems
byXavier Amatriain
 
Square's Machine Learning Infrastructure and Applications - Rong Yan
byHakka Labs
 
OSCON 2014: Data Workflows for Machine Learning
byPaco Nathan
 
Data Workflows for Machine Learning - SF Bay Area ML
byPaco Nathan
 
Machine learning 101 dkom 2017
byfredverheul
 
Azure Machine Learning
byMostafa
 
Production and Beyond: Deploying and Managing Machine Learning Models
byTuri, Inc.
 
Machine learning the high interest credit card of technical debt [PWL]
byJenia Gorokhovsky
 

What's hot

PDF
10 more lessons learned from building Machine Learning systems
byXavier Amatriain
 
PPTX
Towards a Comprehensive Machine Learning Benchmark
byTuri, Inc.
 
PPTX
Production machine learning_infrastructure
byjoshwills
 
PDF
Staying Shallow & Lean in a Deep Learning World
byXavier Amatriain
 
PPTX
Machine Learning for .NET Developers - ADC21
byGülden Bilgütay
 
PDF
DutchMLSchool. ML Automation
byBigML, Inc
 
PDF
Machine Learning Goes Production
byMichał Łopuszyński
 
PDF
Target Leakage in Machine Learning
byYuriy Guts
 
PDF
Automated Machine Learning
byYuriy Guts
 
PDF
Automatic machine learning (AutoML) 101
byQuantUniversity
 
PDF
MLconf seattle 2015 presentation
byehtshamelahi
 
PDF
Data Science, Machine Learning and Neural Networks
byBICA Labs
 
PDF
Building a performing Machine Learning model from A to Z
byCharles Vestur
 
PPTX
Day 2 (Lecture 5): A Practitioner's Perspective on Building Machine Product i...
byAseda Owusua Addai-Deseh
 
PDF
General Tips for participating Kaggle Competitions
byMark Peng
 
PDF
The Power of Auto ML and How Does it Work
byIvo Andreev
 
PDF
Setting up Machine Learning Projects - Full Stack Deep Learning
bySergey Karayev
 
PPTX
Hadoop and Machine Learning
byjoshwills
 
PDF
Knowledge Discovery
byAndré Karpištšenko
 
PDF
L11. The Future of Machine Learning
byMachine Learning Valencia
 
10 more lessons learned from building Machine Learning systems
byXavier Amatriain
 
Towards a Comprehensive Machine Learning Benchmark
byTuri, Inc.
 
Production machine learning_infrastructure
byjoshwills
 
Staying Shallow & Lean in a Deep Learning World
byXavier Amatriain
 
Machine Learning for .NET Developers - ADC21
byGülden Bilgütay
 
DutchMLSchool. ML Automation
byBigML, Inc
 
Machine Learning Goes Production
byMichał Łopuszyński
 
Target Leakage in Machine Learning
byYuriy Guts
 
Automated Machine Learning
byYuriy Guts
 
Automatic machine learning (AutoML) 101
byQuantUniversity
 
MLconf seattle 2015 presentation
byehtshamelahi
 
Data Science, Machine Learning and Neural Networks
byBICA Labs
 
Building a performing Machine Learning model from A to Z
byCharles Vestur
 
Day 2 (Lecture 5): A Practitioner's Perspective on Building Machine Product i...
byAseda Owusua Addai-Deseh
 
General Tips for participating Kaggle Competitions
byMark Peng
 
The Power of Auto ML and How Does it Work
byIvo Andreev
 
Setting up Machine Learning Projects - Full Stack Deep Learning
bySergey Karayev
 
Hadoop and Machine Learning
byjoshwills
 
Knowledge Discovery
byAndré Karpištšenko
 
L11. The Future of Machine Learning
byMachine Learning Valencia
 

Viewers also liked

PDF
Big Data Tutorial For Beginners | What Is Big Data | Big Data Tutorial | Hado...
byEdureka!
 
PDF
Telco Big Data Workshop Sample
byAlan Quayle
 
PPTX
Big data and its impact on indian business
byMegha Shree Shivalingaiah
 
PDF
Digital Pragmatism with Business Intelligence, Big Data and Data Visualisation
byJen Stirrup
 
PPT
3.1.2 classification of network
byKMJ Science Computer
 
PDF
Big Data Step-by-Step: Infrastructure 3/3: Taking it to the cloud... easily.....
byJeffrey Breen
 
PPTX
Big data ppt
byThirunavukkarasu Ps
 
PPTX
Big Data with Not Only SQL
byPhilippe Julio
 
PPTX
Testing Big Data: Automated Testing of Hadoop with QuerySurge
byRTTS
 
PDF
Big Data visualization with Apache Spark and Zeppelin
byprajods
 
PPT
Evolution Of The Computers
bypanitiaict
 
PPTX
A Brief History of Big Data
byBernard Marr
 
PPTX
Layered Software Architecture
byLars-Erik Kindblad
 
PPTX
Software Architecture and Design - An Overview
byOliver Stadie
 
PDF
Streaming Big Data with Spark, Kafka, Cassandra, Akka & Scala (from webinar)
byHelena Edelson
 
PPTX
What is big data?
byDavid Wellman
 
KEY
Big Data Trends
byDavid Feinleib
 
PPTX
Big Data - The 5 Vs Everyone Must Know
byBernard Marr
 
PPT
Architectural Patterns and Software Architectures: Client-Server, Multi-Tier,...
bySvetlin Nakov
 
PPTX
Big data architectures and the data lake
byJames Serra
 
Big Data Tutorial For Beginners | What Is Big Data | Big Data Tutorial | Hado...
byEdureka!
 
Telco Big Data Workshop Sample
byAlan Quayle
 
Big data and its impact on indian business
byMegha Shree Shivalingaiah
 
Digital Pragmatism with Business Intelligence, Big Data and Data Visualisation
byJen Stirrup
 
3.1.2 classification of network
byKMJ Science Computer
 
Big Data Step-by-Step: Infrastructure 3/3: Taking it to the cloud... easily.....
byJeffrey Breen
 
Big data ppt
byThirunavukkarasu Ps
 
Big Data with Not Only SQL
byPhilippe Julio
 
Testing Big Data: Automated Testing of Hadoop with QuerySurge
byRTTS
 
Big Data visualization with Apache Spark and Zeppelin
byprajods
 
Evolution Of The Computers
bypanitiaict
 
A Brief History of Big Data
byBernard Marr
 
Layered Software Architecture
byLars-Erik Kindblad
 
Software Architecture and Design - An Overview
byOliver Stadie
 
Streaming Big Data with Spark, Kafka, Cassandra, Akka & Scala (from webinar)
byHelena Edelson
 
What is big data?
byDavid Wellman
 
Big Data Trends
byDavid Feinleib
 
Big Data - The 5 Vs Everyone Must Know
byBernard Marr
 
Architectural Patterns and Software Architectures: Client-Server, Multi-Tier,...
bySvetlin Nakov
 
Big data architectures and the data lake
byJames Serra
 

Similar to Data Workflows for Machine Learning - Seattle DAML

PDF
Intro to machine learning
byTamir Taha
 
PPTX
Introduction-to-ML.--Module-1-section-1-.pptx
byhellenmuringi
 
PDF
Machine Learning: Past, Present and Future - by Tom Dietterich
byBigML, Inc
 
PPTX
AI hype or reality
byAwantik Das
 
PDF
Fantastic Problems and Where to Find Them: Daryl Weir
byFuturice
 
PPTX
Integrating Machine Learning Capabilities into your team
byCameron Vetter
 
PPTX
Machine Learning AND Deep Learning for OpenPOWER
byGanesan Narayanasamy
 
PPTX
UNIT 1 Early Trends AND TECHNOLOGY in ML.pptx
bylakshithashree002
 
PPTX
Introduction-to-ML Presentation from Minors course of LDCe
bytaralonyt
 
PPTX
Machine learning at scale - Webinar By zekeLabs
byzekeLabs Technologies
 
PDF
Efficient Learning Machines Theories Concepts And Applications For Engineers ...
bycojanstoufa
 
PDF
ML MODULE 1_slideshare.pdf
byShiwani Gupta
 
PPTX
recent.pptx
byaddisuaddaaa
 
PDF
Choosing a Machine Learning technique to solve your need
byGibDevs
 
PDF
The Art of Intelligence – A Practical Introduction Machine Learning for Orac...
byLucas Jellema
 
PDF
Azure Engineering MLOps
byMinesh A. Jethva
 
PPTX
L15.pptx
byImonBennett
 
PPTX
Moving from BI to AI : For decision makers
byzekeLabs Technologies
 
PDF
Machine Learning: The First Salvo of the AI Business Revolution
byCognizant
 
PDF
Agentic AI, Interoperability, and Automation: Lessons from the Past, Visions ...
byAll Things Open
 
Intro to machine learning
byTamir Taha
 
Introduction-to-ML.--Module-1-section-1-.pptx
byhellenmuringi
 
Machine Learning: Past, Present and Future - by Tom Dietterich
byBigML, Inc
 
AI hype or reality
byAwantik Das
 
Fantastic Problems and Where to Find Them: Daryl Weir
byFuturice
 
Integrating Machine Learning Capabilities into your team
byCameron Vetter
 
Machine Learning AND Deep Learning for OpenPOWER
byGanesan Narayanasamy
 
UNIT 1 Early Trends AND TECHNOLOGY in ML.pptx
bylakshithashree002
 
Introduction-to-ML Presentation from Minors course of LDCe
bytaralonyt
 
Machine learning at scale - Webinar By zekeLabs
byzekeLabs Technologies
 
Efficient Learning Machines Theories Concepts And Applications For Engineers ...
bycojanstoufa
 
ML MODULE 1_slideshare.pdf
byShiwani Gupta
 
recent.pptx
byaddisuaddaaa
 
Choosing a Machine Learning technique to solve your need
byGibDevs
 
The Art of Intelligence – A Practical Introduction Machine Learning for Orac...
byLucas Jellema
 
Azure Engineering MLOps
byMinesh A. Jethva
 
L15.pptx
byImonBennett
 
Moving from BI to AI : For decision makers
byzekeLabs Technologies
 
Machine Learning: The First Salvo of the AI Business Revolution
byCognizant
 
Agentic AI, Interoperability, and Automation: Lessons from the Past, Visions ...
byAll Things Open
 

More from Paco Nathan

PDF
Human in the loop: a design pattern for managing teams working with ML
byPaco Nathan
 
PDF
Human-in-the-loop: a design pattern for managing teams that leverage ML
byPaco Nathan
 
PDF
Human-in-a-loop: a design pattern for managing teams which leverage ML
byPaco Nathan
 
PDF
Humans in a loop: Jupyter notebooks as a front-end for AI
byPaco Nathan
 
PDF
Humans in the loop: AI in open source and industry
byPaco Nathan
 
PDF
Computable Content
byPaco Nathan
 
PDF
Computable Content: Lessons Learned
byPaco Nathan
 
PDF
SF Python Meetup: TextRank in Python
byPaco Nathan
 
PDF
Use of standards and related issues in predictive analytics
byPaco Nathan
 
PDF
Data Science in 2016: Moving Up
byPaco Nathan
 
PDF
Data Science Reinvents Learning?
byPaco Nathan
 
PDF
Jupyter for Education: Beyond Gutenberg and Erasmus
byPaco Nathan
 
PDF
GalvanizeU Seattle: Eleven Almost-Truisms About Data
byPaco Nathan
 
PDF
Microservices, containers, and machine learning
byPaco Nathan
 
PDF
GraphX: Graph analytics for insights about developer communities
byPaco Nathan
 
PDF
Graph Analytics in Spark
byPaco Nathan
 
PDF
Apache Spark and the Emerging Technology Landscape for Big Data
byPaco Nathan
 
PDF
QCon São Paulo: Real-Time Analytics with Spark Streaming
byPaco Nathan
 
PDF
Strata 2015 Data Preview: Spark, Data Visualization, YARN, and More
byPaco Nathan
 
PDF
A New Year in Data Science: ML Unpaused
byPaco Nathan
 
Human in the loop: a design pattern for managing teams working with ML
byPaco Nathan
 
Human-in-the-loop: a design pattern for managing teams that leverage ML
byPaco Nathan
 
Human-in-a-loop: a design pattern for managing teams which leverage ML
byPaco Nathan
 
Humans in a loop: Jupyter notebooks as a front-end for AI
byPaco Nathan
 
Humans in the loop: AI in open source and industry
byPaco Nathan
 
Computable Content
byPaco Nathan
 
Computable Content: Lessons Learned
byPaco Nathan
 
SF Python Meetup: TextRank in Python
byPaco Nathan
 
Use of standards and related issues in predictive analytics
byPaco Nathan
 
Data Science in 2016: Moving Up
byPaco Nathan
 
Data Science Reinvents Learning?
byPaco Nathan
 
Jupyter for Education: Beyond Gutenberg and Erasmus
byPaco Nathan
 
GalvanizeU Seattle: Eleven Almost-Truisms About Data
byPaco Nathan
 
Microservices, containers, and machine learning
byPaco Nathan
 
GraphX: Graph analytics for insights about developer communities
byPaco Nathan
 
Graph Analytics in Spark
byPaco Nathan
 
Apache Spark and the Emerging Technology Landscape for Big Data
byPaco Nathan
 
QCon São Paulo: Real-Time Analytics with Spark Streaming
byPaco Nathan
 
Strata 2015 Data Preview: Spark, Data Visualization, YARN, and More
byPaco Nathan
 
A New Year in Data Science: ML Unpaused
byPaco Nathan
 

Recently uploaded

PDF
Why Many Smart Device Platforms Fail to Scale?
byPromeraki Developments
 
PDF
UiPath Automation Developer Associate Training Series 2026 - Session 2
byDianaGray10
 
PDF
Understanding Foldable 3-Wheel Electric Scooters for Everyday Use
byTopmate
 
PDF
Writing GPU-Ready AI Models in Pure Java with Babylon
byAna-Maria Mihalceanu
 
PPTX
Career-Opportunities in Industrial Arts Grade 8 PPT Lesson 2
byFSBTLEDNathanVince
 
PDF
Track Phone Location by Number (2026)_ What’s Actually Possible for Free (and...
byEmily Woods
 
PPTX
Retrieval Augmented Generation- The Synergistic Power of Prompt Engineering
bySemantic SEO BD
 
PDF
How to build hackthon projects from ZERO!
byishantyadav1111
 
PDF
Dev Dives: Build and deploy agentic automations - the unified way
byUiPathCommunity
 
PDF
AI and Zero Trust: What it takes to do it right
byMark Simos
 
PDF
UiPath Automation Developer Associate Training Series 2026 - Session 1
byDianaGray10
 
PDF
Traditional-Security-Models-No-Longer-Work.pptx (1).pdf
byOhhproJunction
 
PDF
Analyze and Preserve Logs - RHCSA (RH134).pdf
byLinuxCert Guru
 
PDF
Designing a Blog Using Wordpress
bymarkzubi50
 
PDF
Getting Started with Apache Spark: Big Data Made Simple [Free Meetup]
byHaim Michael
 
PDF
The Emergence of Empathic XR: AWE Asia 2026 keynote
byMark Billinghurst
 
PDF
The Enterprise Web3 Landscape - a view from Kaleido based on the past 10 year...
byPeter Broadhurst
 
PPTX
Authority After Search: How AI Systems Reconstruct Trust, Expertise, and Legi...
byJeff Howell
 
PDF
Title Installing Windows, Linux, MacOS.pdf
byGeraldAbadajos
 
PDF
Mesh WiFi Router: The Smart Solution for Fast, Seamless, Whole-Home Internet
byAeroMesh Systems
 
Why Many Smart Device Platforms Fail to Scale?
byPromeraki Developments
 
UiPath Automation Developer Associate Training Series 2026 - Session 2
byDianaGray10
 
Understanding Foldable 3-Wheel Electric Scooters for Everyday Use
byTopmate
 
Writing GPU-Ready AI Models in Pure Java with Babylon
byAna-Maria Mihalceanu
 
Career-Opportunities in Industrial Arts Grade 8 PPT Lesson 2
byFSBTLEDNathanVince
 
Track Phone Location by Number (2026)_ What’s Actually Possible for Free (and...
byEmily Woods
 
Retrieval Augmented Generation- The Synergistic Power of Prompt Engineering
bySemantic SEO BD
 
How to build hackthon projects from ZERO!
byishantyadav1111
 
Dev Dives: Build and deploy agentic automations - the unified way
byUiPathCommunity
 
AI and Zero Trust: What it takes to do it right
byMark Simos
 
UiPath Automation Developer Associate Training Series 2026 - Session 1
byDianaGray10
 
Traditional-Security-Models-No-Longer-Work.pptx (1).pdf
byOhhproJunction
 
Analyze and Preserve Logs - RHCSA (RH134).pdf
byLinuxCert Guru
 
Designing a Blog Using Wordpress
bymarkzubi50
 
Getting Started with Apache Spark: Big Data Made Simple [Free Meetup]
byHaim Michael
 
The Emergence of Empathic XR: AWE Asia 2026 keynote
byMark Billinghurst
 
The Enterprise Web3 Landscape - a view from Kaleido based on the past 10 year...
byPeter Broadhurst
 
Authority After Search: How AI Systems Reconstruct Trust, Expertise, and Legi...
byJeff Howell
 
Title Installing Windows, Linux, MacOS.pdf
byGeraldAbadajos
 
Mesh WiFi Router: The Smart Solution for Fast, Seamless, Whole-Home Internet
byAeroMesh Systems
 

Data Workflows for Machine Learning - Seattle DAML

  • 1.
    Data Workflows for MachineLearning: 
 Seattle,WA 2014-01-29 ! Paco Nathan
 @pacoid http://liber118.com/pxn/ meetup.com/Seattle-DAML/events/159043422/
  • 2.
    Why is thistalk here? Machine Learning in production apps is less and less about algorithms (even though that work is quite fun and vital) ! Performing real work is more about: ! socializing a problem within an organization ! feature engineering (“Beyond Product Managers”) ! tournaments in CI/CD environments ! operationalizing high-ROI apps at scale ! etc. ! So I’ll just crawl out on a limb and state that leveraging great frameworks to build data workflows is more important than 
 chasing after diminishing returns on highly nuanced algorithms. ! Because Interwebs!
  • 3.
    Data Workflows forMachine Learning Middleware has been evolving for Big Data, and there are some great examples — we’ll review several. Process has been evolving too, right along with the use cases. ! Popular frameworks typically provide some Machine Learning capabilities within their core components, or at least among their major use cases. ! Let’s consider features from Enterprise Data Workflows as a basis 
 for what’s needed in Data Workflows for Machine Learning. ! Their requirements for scale, robustness, cost trade-offs, interdisciplinary teams, etc., serve as guides in general.
  • 4.
    Caveat Auditor I won’tclaim to be expert with each of the frameworks and environments described in this talk. Expert with a few of them perhaps, but more to the point: embroiled in many use cases. ! This talk attempts to define a “scorecard” for evaluating important ML data workflow features: 
 what’s needed for use cases, compare and contrast of what’s available, plus some indication of which frameworks are likely 
 to be best for a given scenario. ! Seriously, this is a work in progress.
  • 5.
    Outline • Definition: Machine Learning • Definition:Data Workflows • A whole bunch o’ examples across several platforms • Nine points to discuss, leading up to a scorecard • A crazy little thing called PMML • Questions, comments, flying tomatoes…
  • 6.
    Data Workflows for MachineLearning: 
 Frame the question… ! 
 “A Basis for What’s Needed”
  • 7.
    Definition: Machine Learning “Machinelearning algorithms can figure out how to perform
 important tasks by generalizing from examples.This is often
 feasible and cost-effective where manual programming is not. 
 As more data becomes available, more ambitious problems 
 can be tackled. As a result, machine learning is widely used 
 in computer science and other fields.” Pedro Domingos, U Washington
 A Few Useful Things to Know about Machine Learning [learning as generalization] =
 [representation] + [evaluation] + [optimization] • • • overfitting (variance) underfitting (bias) “perfect classifier” (no free lunch)
  • 8.
    Definition: Machine Learning… Conceptually
 ! ! 1. real-world data 2. graph theory for representation 3. convert to sparse matrix for production work 
 leveraging abstract algebra + func programming 4. cost-effective parallel processing for ML apps 
 at scale, live use cases
  • 9.
    Definition: Machine Learning… Conceptually
 ! ! f(x): loss function g(z): regularization term 1. real-world data [generalization] 2. graph theory for representation [representation] 3. convert to sparse matrix for production work 
 leveraging abstract algebra + func programming [evaluation] 4. cost-effective parallel processing for ML apps 
 at scale, live use cases [optimization]
  • 10.
    Definition: Machine Learning… Interdisciplinary Teams, Needs × Roles very very sco iisco d d ng lliing ode ode m m n n atiio at o tegr tegr n iin pps pps a a s s tem tem sys sys business process, stakeholder Domain Expert data science Data Scientist App Dev Ops data prep, discovery, modeling, etc. software engineering, automation systems engineering, availability
  • 11.
    Definition: Machine Learning… Process, Feature Engineering feature engineering data data sources data sources sources data prep pipeline hold-outs train learners classifiers classifiers classifiers scoring test representation evaluation optimization use cases
  • 12.
    Definition: Machine Learning… Process, Tournaments feature engineering data data sources data sources sources data prep pipeline tournaments hold-outs train learners classifiers classifiers classifiers scoring test representation • obtain other data? • improve metadata? • refine representation? • improve optimization? evaluation • iterate with stakeholders? • can models (inference) inform first principles approaches? optimization use cases quantify and measure: • benefit? • risk? • operational costs?
  • 13.
    Definition: Machine Learning… Invasion of the Monoids monoid (an alien life form, courtesy of abstract algebra): ! binary associative operator ! closure within a set of objects ! unique identity element what are those good for? ! ! ! ! ! composable functions in workflows compiler “hints” on steroids, to parallelize reassemble results minimizing bottlenecks at scale reusable components, like Lego blocks think: Docker for business logic Monoidify! Monoids as a Design Principle for Efficient MapReduce Algorithms
 Jimmy Lin, U Maryland/Twitter kudos to Oscar Boykin, Sam Ritchie, et al.
  • 14.
    Definition: Machine Learning featureengineering data data sources data sources sources data prep pipeline tournaments hold-outs train learners classifiers classifiers classifiers scoring test representation • obtain other data? • improve metadata? • refine representation? • improve optimization? evaluation optimization • iterate with stakeholders? quantify and measure: • benefit? • risk? • operational costs? • can models (inference) inform first principles approaches? y y over over diisc d sc elliing e ng mod mod use cases n on ratiio grat nteg iinte apps apps ems ems syst syst Domain Expert data science Data Scientist App Dev Ops introduced capability
  • 15.
    Definition feature engineering data data sources data sources sources data prep pipeline tournaments hold-outs train learners classifiers classifiers classifiers scoring test Canwe fit the required 
 process into generalized 
 workflow definitions? representation • obtain other data? • improve metadata? • refine representation? • improve optimization? evaluation optimization • iterate with stakeholders? quantify and measure: • benefit? • risk? • operational costs? • can models (inference) inform first principles approaches? y y over over diisc d sc elliing e ng mod mod use cases n on ratiio grat nteg iinte apps apps ems ems syst syst Domain Expert data science Data Scientist App Dev Ops introduced capability
  • 16.
    Definition: Data Workflows Middleware,effectively, is evolving for Big Data and Machine Learning…
 The following design pattern — a DAG — shows up in many places, 
 via many frameworks: ETL data sources data prep predictive model end uses
  • 17.
    Definition: Data Workflows definitionof a typical Enterprise workflow which crosses through multiple departments, languages, and technologies… ANSI SQL for ETL ETL data sources data prep predictive model end uses
  • 18.
    Definition: Data Workflows definitionof a typical Enterprise workflow which crosses through multiple departments, languages, and technologies… ETL data sources data prep Java, Pig for business logic predictive model end uses
  • 19.
    Definition: Data Workflows definitionof a typical Enterprise workflow which crosses through multiple departments, languages, and technologies… SAS for predictive models
 ETL data sources data prep predictive model end uses
  • 20.
    Definition: Data Workflows definitionof a typical Enterprise workflow which crosses through multiple departments, languages, and technologies… ANSI SQL for ETL most of the licensing costs… ETL data sources data prep SAS for predictive models predictive model end uses
  • 21.
    Definition: Data Workflows definitionof a typical Enterprise workflow which crosses through multiple departments, languages, and technologies… most of the project costs… ETL data sources data prep Java, Pig for business logic predictive model end uses
  • 22.
    Definition: Data Workflows definitionof a typical Enterprise workflow which crosses through multiple departments, languages, and technologies… most of the project costs… Something emerges 
 to fill these needs, 
 for instance… ETL data sources data prep Java, Pig for business logic predictive model end uses
  • 23.
    Definition: Data Workflows Forexample, Cascading and related projects implement the following components, based on 100% open source: Lingual: DW → ANSI SQL ETL data sources business logic in Java, Clojure, Scala, etc. data prep a compiler sees it all…
 one connected DAG: • troubleshooting cascading.org Pattern: SAS, R, etc. → PMML predictive model end uses • exception handling source taps for Cassandra, JDBC, Splunk, etc. • notifications • some optimizations sink taps for Memcached, HBase, MongoDB, etc.
  • 24.
    Definition: Data Workflows Cascadingallows multiple departments to combine their workflow components into an integrated app – one among many, typically – based on 100% open source Lingual: DW → ANSI SQL cascading.org ETL data sources source taps for Cassandra, JDBC, Splunk, etc. business logic in Java, ! Clojure, Scala, etc. Pattern: SAS, R, etc. → PMML ! FlowDef flowDef = FlowDef.flowDef()! .setName( "etl" )! predictive data .addSource( "example.employee", emplTap )! model prep .addSource( "example.sales", salesTap )! .addSink( "results", resultsTap );!  ! SQLPlanner sqlPlanner = new SQLPlanner()! .setSql( sqlStatement );! end uses  ! flowDef.addAssemblyPlanner( sqlPlanner );! ! sink taps for ! Memcached, HBase, MongoDB, etc.
  • 25.
    Definition: Data Workflows Cascadingallows multiple departments to combine their workflow components into an integrated app – one among many, typically – based on 100% open source Lingual: DW → ANSI SQL business logic in Java, Clojure, Scala, etc. ! ! FlowDef flowDef = FlowDef.flowDef()! .setName( "classifier" )! .addSource( "input", inputTap )! data ETL .addSink( "classify", classifyTap prep );!  ! PMMLPlanner pmmlPlanner = new PMMLPlanner()! .setPMMLInput( new File( pmmlModel ) )! data .retainOnlyActiveIncomingFields();! sources  ! flowDef.addAssemblyPlanner( pmmlPlanner );! ! source taps for ! Cassandra, JDBC, Splunk, etc. Pattern: SAS, R, etc. → PMML predictive model end uses sink taps for Memcached, HBase, MongoDB, etc.
  • 26.
    Enterprise Data Workflowswith Cascading O’Reilly, 2013 shop.oreilly.com/product/ 0636920028536.do ETL data sources data prep predictive model end uses
  • 27.
    Enterprise Data Workflowswith Cascading O’Reilly, 2013 shop.oreilly.com/product/ 0636920028536.do This begs an update… ETL data sources data prep predictive model end uses
  • 28.
    Enterprise Data Workflowswith Cascading O’Reilly, 2013 shop.oreilly.com/product/ 0636920028536.do Because… ETL data sources data prep predictive model end uses
  • 30.
    Data Workflows for MachineLearning: 
 Examples… ! 
 “Neque per exemplum”
  • 31.
    Example: KNIME “a user-friendlygraphical workbench for the entire analysis
 process: data access, data transformation, initial investigation,
 powerful predictive analytics, visualisation and reporting.” • • • • • • large number of integrations (over 1000 modules) ranked #1 in customer satisfaction among 
 open source analytics frameworks visual editing of reusable modules leverage prior work in R, Perl, etc. Eclipse integration easily extended for new integrations knime.org
  • 32.
    Example: Python stack Pythonhas much to offer – ranging across an 
 organization, no just for the analytics staff • • • • • • • • • ipython.org pandas.pydata.org scikit-learn.org numpy.org scipy.org code.google.com/p/augustus continuum.io nltk.org matplotlib.org
  • 33.
    Example: Julia julialang.org “a high-level,high-performance dynamic programming language
 for technical computing, with syntax that is familiar to users of
 other technical computing environments” • • • significantly faster than most alternatives built to leverage parallelism, cloud computing still relatively new — one to watch! importall Base! ! type ! BubbleSort <: Sort.Algorithm end! function sort!(v::AbstractVector, lo::Int, hi::Int, ::BubbleSort, o::Sort.Ordering)!     while true!         clean = true!         for i = lo:hi-1!             if Sort.lt(o, v[i+1], v[i])!                 v[i+1], v[i] = v[i], v[i+1]!                 clean = false!             end!         end!         clean && break!     end!     return v! end!
  • 34.
    Example: Summingbird github.com/twitter/summingbird “a librarythat lets you write streaming MapReduce programs
 that look like native Scala or Java collection transformations 
 and execute them on a number of well-known distributed
 MapReduce platforms like Storm and Scalding.” • switch between Storm, Scalding (Hadoop) • Spark support is in progress • leverage Algebird, Storehaus, Matrix API, etc. def wordCount[P <: Platform[P]]! (source: Producer[P, String], store: P#Store[String, Long]) =! source.flatMap { sentence => ! toWords(sentence).map(_ -> 1L)! }.sumByKey(store)
  • 35.
    Example: Scalding import com.twitter.scalding._!  ! classWordCount(args : Args) extends Job(args) {! Tsv(args("doc"),! ('doc_id, 'text),! skipHeader = true)! .read! .flatMap('text -> 'token) {! text : String => text.split("[ [](),.]")! }! .groupBy('token) { _.size('count) }! .write(Tsv(args("wc"), writeHeader = true))! } github.com/twitter/scalding
  • 36.
    Example: Scalding • extendsthe Scala collections API so that distributed lists become “pipes” backed by Cascading • code is compact, easy to understand • nearly 1:1 between elements of conceptual flow diagram 
 and function calls • extensive libraries are available for linear algebra, abstract algebra, machine learning – e.g., Matrix API, Algebird, etc. • significant investments by Twitter, Etsy, eBay, etc. • less learning curve than Cascalog • build scripts… those take a while to run : github.com/twitter/scalding
  • 37.
    Example: Cascalog (ns impatient.core!   (:use[cascalog.api]!         [cascalog.more-taps :only (hfs-delimited)])!   (:require [clojure.string :as s]!             [cascalog.ops :as c])!   (:gen-class))! ! (defmapcatop split [line]!   "reads in a line of string and splits it by regex"!   (s/split line #"[[](),.)s]+"))! ! (defn -main [in out & args]!   (?<- (hfs-delimited out)!        [?word ?count]!        ((hfs-delimited in :skip-header? true) _ ?line)!        (split ?line :> ?word)!        (c/count ?count)))! ! ; Paul Lam! ; github.com/Quantisan/Impatient cascalog.org
  • 38.
    Example: Cascalog • implementsDatalog in Clojure, with predicates backed 
 by Cascading – for a highly declarative language • run ad-hoc queries from the Clojure REPL –
 approx. 10:1 code reduction compared with SQL • composable subqueries, used for test-driven development (TDD) practices at scale • Leiningen build: simple, no surprises, in Clojure itself • more new deployments than other Cascading DSLs – 
 Climate Corp is largest use case: 90% Clojure/Cascalog • has a learning curve, limited number of Clojure developers • aggregators are the magic, and those take effort to learn cascalog.org
  • 39.
    Example: Apache Spark spark-project.org in-memorycluster computing, 
 by Matei Zaharia, Ram Venkataraman, et al. • • • • • intended to make data analytics fast to write and to run load data into memory and query it repeatedly, much more quickly than with Hadoop APIs in Scala, Java and Python, shells in Scala and Python Shark (Hive on Spark), Spark Streaming (like Storm), etc. integrations with MLbase, Summingbird (in progress) // word count! val file = spark.textFile(“hdfs://…”)! val counts = file.flatMap(line => line.split(” “))! .map(word => (word, 1))! .reduceByKey(_ + _)! counts.saveAsTextFile(“hdfs://…”)
  • 40.
    Example: MLbase “distributed machinelearning made easy” • • • • • sponsored by UC Berkeley EECS AMP Lab MLlib – common algorithms, low-level, written atop Spark MLI – feature extraction, algorithm dev atop MLlib ML Optimizer – automates model selection, 
 compiler/optimizer see article:
 http://strata.oreilly.com/2013/02/mlbase-scalablemachine-learning-made-accessible.html ! data = load("hdfs://path/to/als_clinical")! // the features are stored in columns 2-10! X = data[, 2 to 10]! y = data[, 1]! model = do_classify(y, X) mlbase.org
  • 41.
    Example: Titan thinkaurelius.github.io/titan distributed graphdatabase, 
 by Matthias Broecheler, Marko Rodriguez, et al. • • • • • • scalable graph database optimized for storing and querying graphs supports hundreds of billions of vertices and edges transactional database that can support thousands of concurrent users executing complex graph traversals supports search through Lucene, ElasticSearch can be backed by HBase, Cassandra, BerkeleyDB TinkerPop native graph stack with Gremlin, etc. // who is hercules' paternal grandfather?! g.V('name','hercules').out('father').out('father').name
  • 42.
    Example: MBrace “a .NETbased software stack that enables easy large-scale
 distributed computation and big data analysis.” • declarative and concise distributed algorithms in 
 F# asynchronous workflows • scalable for apps in private datacenters or public 
 clouds, e.g., Windows Azure or Amazon EC2 • tooling for interactive, REPL-style deployment, monitoring, management, debugging in Visual Studio • leverages monads (similar to Summingbird) • MapReduce as a library, but many patterns beyond m-brace.net let rec mapReduce (map: 'T -> Cloud<'R>)! (reduce: 'R -> 'R -> Cloud<'R>)! (identity: 'R)! (input: 'T list) =! cloud {! match input with! | [] -> return identity! | [value] -> return! map value! | _ ->! let left, right = List.split input! let! r1, r2 =! (mapReduce map reduce identity left)! <||>! (mapReduce map reduce identity right)! return! reduce r1 r2! }
  • 43.
    Data Workflows for MachineLearning: 
 Update the definitions… ! “Nine Points to Discuss”

  • 44.
    Data Workflows Formally speaking,workflows include people (cross-dept) 
 and define oversight for exceptional data 1 …otherwise, data flow or pipeline would be more apt …examples include Cascading traps, with exceptional tuples 
 routed to a review organization, e.g., Customer Support Customers Web App logs logs Logs Cache Support trap tap Modeling PMML source tap Data Workflow source tap sink tap Analytics Cubes Reporting sink tap customer Customer profile DBs Prefs Hadoop Cluster includes people, defines oversight for exceptional data
  • 45.
    Data Workflows Workflows imposea separation of concerns, allowing 
 for multiple abstraction layers in the tech stack …specify what is required, not how to accomplish it …articulating the business logic … Cascading leverages pattern language …related notions from Knuth, literate programming …not unlike BPM/BPEL for Big Data …examples: IPython, R Markdown, etc. 2 separation of concerns, allows for literate programming
  • 46.
    3 Data Workflows Multiple abstractionlayers in the tech stack are 
 needed, but still emerging …feedback loops based on machine data …optimizers feed on abstraction Portable Models …metadata accounting: Reusable Components track data lineage • propagate schema • model / feature usage • ensemble performance …app history accounting: • • • • util stats, mixing workloads heavy-hitters, bottlenecks throughput, latency Business Process multiple abstraction layers 
 for metadata, feedback, 
 and optimization • • • • data lineage schema propagation feature selection tournaments DSLs Planners/ Optimizers Mixed Topologies Cluster Scheduler • app history • util stats • bottlenecks Machine Data Clusters
  • 47.
    3 Data Workflows Multiple needed, butstill emerging Business Process …feedback loops based on …optimizers feed on abstraction Portable Models Um, will compilers 
 track data lineage propagate ever look like this?? model / feature usage …metadata accounting: • • • Reusable Components ensemble performance …app history accounting: • • • • • • data lineage schema propagation feature selection tournaments DSLs • • multiple abstraction layers 
 for metadata, feedback, 
 and optimization util stats, mixing workloads heavy-hitters, bottlenecks throughput, latency Planners/ Optimizers Mixed Topologies Cluster Scheduler • app history • util stats • bottlenecks Machine Data Clusters
  • 48.
    4 Data Workflows Workflows mustprovide for test, which is no simple 
 matter testing: model evaluation, TDD, app deployment …testing is required on three abstraction layers: model portability/evaluation, ensembles, tournaments • TDD in reusable components and business process • continuous integration / continuous deployment …examples: Cascalog for TDD, PMML for model eval • …still so much more to improve Business Process Portable Models Reusable Components • • • • data lineage schema propagation feature selection tournaments DSLs …keep in mind that workflows involve people, too Planners/ Optimizers Mixed Topologies Cluster Scheduler • app history • util stats • bottlenecks Machine Data Clusters
  • 49.
    5 Data Workflows Workflows enablesystem integration …future-proof integrations and scale-out future-proof system integration, scale-out, ops …build components and apps, not jobs and command lines …allow compiler optimizations across the DAG,
 i.e., cross-dept contributions Customers Web App …minimize operationalization costs: troubleshooting, debugging at scale • exception handling • notifications, instrumentation …examples: KNIME, Cascading, etc. • logs logs Logs Cache Support trap tap Modeling ! PMML source tap Data Workflow source tap sink tap Analytics Cubes Reporting sink tap customer Customer profile DBs Prefs Hadoop Cluster
  • 50.
    Data Workflows Visualizing workflows,what a great idea. …the practical basis for: collaboration • rapid prototyping • component reuse …examples: KNIME wins best in category • …Cascading generates flow diagrams, 
 which are a nice start 6 visualizing allows people 
 to collaborate through code
  • 51.
    Data Workflows Abstract algebra,containerizing workflow metadata …monoids, semigroups, etc., allow for reusable components 
 with well-defined properties for running in parallel at scale …let business process be agnostic about underlying topologies, 
 with analogy to Linux containers (Docker, etc.) …compose functions, take advantage of sparsity (monoids) …because “data is fully functional” – FP for s/w eng benefits …aggregators are almost always “magic”, now we can solve 
 for common use cases …read Monoidify! Monoids as a Design Principle for Efficient MapReduce Algorithms by Jimmy Lin …Cascading introduced some notions of this circa 2007, 
 but didn’t make it explicit …examples: Algebird in Summingbird, Simmer, Spark, etc. 7 abstract algebra and functional programming containerize business process
  • 52.
    Data Workflows Workflow needsvary in time, and need to blend time …something something batch, something something low-latency 8 blend results from different time scales: batch plus lowlatency …scheduling batch isn’t hard; scheduling low-latency is 
 computationally brutal (see the Omega paper) …because people like saying “Lambda Architecture”, 
 it gives them goosebumps, or something …because real-time means so many different things …because batch windows are so 1964 …examples: Summingbird, Oryx Big Data
 Nathan Marz, James Warren
 manning.com/marz
  • 53.
    Data Workflows Workflows maydefine contexts in which model selection 
 possibly becomes a compiler problem …for example, see Boyd, Parikh, et al. …ultimately optimizing for loss function + regularization term …probably not ready for prime-time tomorrow …examples: MLbase f(x): loss function g(z): regularization term 9 optimize learners in context, to make model selection potentially a compiler problem
  • 54.
    Data Workflows For oneof the best papers about what workflows really
 truly require, see Out of the Tar Pit by Moseley and Marks bonus
  • 55.
    Nine Points forData Workflows — a wish list 1. includes people, defines oversight for exceptional data 2. separation of concerns, allows for literate programming 3. multiple abstraction layers for metadata, feedback, and optimization Business Process Portable Models Reusable Components 4. testing: model evaluation, TDD, app deployment 6. visualizing workflows allows people to collaborate through code Planners/ Optimizers Mixed Topologies 7. abstract algebra and functional programming containerize business process 9. optimize learners in context, to make model 
 selection potentially a compiler problem data lineage schema propagation feature selection tournaments DSLs 5. future-proof system integration, scale-out, ops 8. blend results from different time scales: 
 batch plus low-latency • • • • Cluster Scheduler • app history • util stats • bottlenecks Machine Data Clusters
  • 57.
    Nine Points forData Workflows — a scorecard Spark, MLbase Oryx Summing! bird Cascalog Cascading ! KNIME Py Data R Markdown ✔ ✔ ✔ ✔ includes people, exceptional data separation of concerns multiple 
 abstraction layers testing in depth future-proof 
 system integration visualize to collab can haz monoids blends batch + 
 “real-time” optimize learners 
 in context can haz PMML ✔ MBrace
  • 58.
    Data Workflows for MachineLearning: 
 PMML… ! 
 “Oh, by the way…”
  • 59.
    PMML – anindustry standard • • established XML standard for predictive model markup • members: IBM, SAS, Visa, FICO, Equifax, NASA,
 Microstrategy, Microsoft, etc. • PMML concepts for metadata, ensembles, etc., translate 
 directly into workflow abstractions, e.g., Cascading organized by Data Mining Group (DMG), since 1997 
 http://dmg.org/ ! “PMML is the leading standard for statistical and data mining models and
 supported by over 20 vendors and organizations.With PMML, it is easy 
 to develop a model on one system using one application and deploy the
 model on another system using another application.” wikipedia.org/wiki/Predictive_Model_Markup_Language
  • 60.
  • 61.
    PMML – modelcoverage • • • • • • • • • • • Association Rules: AssociationModel element Cluster Models: ClusteringModel element Decision Trees: TreeModel element Naïve Bayes Classifiers: NaiveBayesModel element Neural Networks: NeuralNetwork element Regression: RegressionModel and GeneralRegressionModel elements Rulesets: RuleSetModel element Sequences: SequenceModel element Support Vector Machines: SupportVectorMachineModel element Text Models: TextModel element Time Series: TimeSeriesModel element ibm.com/developerworks/industry/library/ind-PMML2/
  • 62.
    PMML – furtherstudy ! PMML in Action
 Alex Guazzelli, Wen-Ching Lin, Tridivesh Jena
 amazon.com/dp/1470003244 ! See also excellent resources at: zementis.com/pmml.htm
  • 63.
    Pattern – modelscoring • • • • • a PMML library for Cascading workflows Customers migrate workloads: SAS,Teradata, etc., 
 exporting predictive models as PMML Web App great open source tools – R, Weka, 
 KNIME, Matlab, RapidMiner, etc. leverage PMML as another kind of DSL logs logs Logs Cache Support only for scoring models, not training trap tap Modeling PMML source tap Data Workflow source tap sink tap Analytics Cubes Reporting sink tap customer Customer profile DBs Prefs Hadoop Cluster
  • 64.
    Pattern – modelscoring • originally intended as a general purpose, tuple-oriented model scoring engine for JVM-based frameworks • library plugs into Cascading (Hadoop), and ostensibly Cascalog, Scalding, Storm, Spark, Summingbird, etc. • dev forum:
 https://groups.google.com/forum/#!forum/pattern-user • original GitHub repo:
 https://github.com/ceteri/pattern/ • somehow, the other fork became deeply intertwined 
 with Cascading dependencies… • work in progress to integrate with other frameworks, 
 add models, and also train models at scale using Spark
  • 65.
    Pattern – createa model in R ## train a RandomForest model!  ! f <- as.formula("as.factor(label) ~ .")! fit <- randomForest(f, data_train, ntree=50)!  ! ## test the model on the holdout test set!  ! print(fit$importance)! print(fit)!  ! predicted <- predict(fit, data)! data$predicted <- predicted! confuse <- table(pred = predicted, true = data[,1])! print(confuse)!  ! ## export predicted labels to TSV!  ! write.table(data, file=paste(dat_folder, "sample.tsv", sep="/"), 
 quote=FALSE, sep="t", row.names=FALSE)!  ! ## export RF model to PMML!  ! saveXML(pmml(fit), file=paste(dat_folder, "sample.rf.xml", sep="/"))!
  • 66.
    Pattern – capturemodel parameters as PMML <?xml version="1.0"?>! <PMML version="4.0" xmlns="http://www.dmg.org/PMML-4_0"!  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"!  xsi:schemaLocation="http://www.dmg.org/PMML-4_0
 http://www.dmg.org/v4-0/pmml-4-0.xsd">!  <Header copyright="Copyright (c)2012 Concurrent, Inc." description="Random Forest Tree Model">!   <Extension name="user" value="ceteri" extender="Rattle/PMML"/>!   <Application name="Rattle/PMML" version="1.2.30"/>!   <Timestamp>2012-10-22 19:39:28</Timestamp>!  </Header>!  <DataDictionary numberOfFields="4">!   <DataField name="label" optype="categorical" dataType="string">!    <Value value="0"/>!    <Value value="1"/>!   </DataField>!   <DataField name="var0" optype="continuous" dataType="double"/>!   <DataField name="var1" optype="continuous" dataType="double"/>!   <DataField name="var2" optype="continuous" dataType="double"/>!  </DataDictionary>!  <MiningModel modelName="randomForest_Model" functionName="classification">!   <MiningSchema>!    <MiningField name="label" usageType="predicted"/>!    <MiningField name="var0" usageType="active"/>!    <MiningField name="var1" usageType="active"/>!    <MiningField name="var2" usageType="active"/>!   </MiningSchema>!   <Segmentation multipleModelMethod="majorityVote">!    <Segment id="1">!     <True/>!     <TreeModel modelName="randomForest_Model" functionName="classification" algorithmName="randomForest" splitCharacteristic="binarySplit">!      <MiningSchema>!       <MiningField name="label" usageType="predicted"/>!       <MiningField name="var0" usageType="active"/>!       <MiningField name="var1" usageType="active"/>!       <MiningField name="var2" usageType="active"/>!      </MiningSchema>! ...!
  • 67.
    Pattern – scorea model, within an app public static void main( String[] args ) throws RuntimeException {! String inputPath = args[ 0 ];! String classifyPath = args[ 1 ];! // set up the config properties! Properties properties = new Properties();! AppProps.setApplicationJarClass( properties, Main.class );! HadoopFlowConnector flowConnector = new HadoopFlowConnector( properties );!   // create source and sink taps! Tap inputTap = new Hfs( new TextDelimited( true, "t" ), inputPath );! Tap classifyTap = new Hfs( new TextDelimited( true, "t" ), classifyPath );!   // handle command line options! OptionParser optParser = new OptionParser();! optParser.accepts( "pmml" ).withRequiredArg();!   OptionSet options = optParser.parse( args );!  ! // connect the taps, pipes, etc., into a flow! FlowDef flowDef = FlowDef.flowDef().setName( "classify" )! .addSource( "input", inputTap )! .addSink( "classify", classifyTap );!  ! if( options.hasArgument( "pmml" ) ) {! String pmmlPath = (String) options.valuesOf( "pmml" ).get( 0 );! PMMLPlanner pmmlPlanner = new PMMLPlanner()! .setPMMLInput( new File( pmmlPath ) )! .retainOnlyActiveIncomingFields()! .setDefaultPredictedField( new Fields( "predict", Double.class ) ); // default value if missing from the model! flowDef.addAssemblyPlanner( pmmlPlanner );! }!  ! // write a DOT file and run the flow! Flow classifyFlow = flowConnector.connect( flowDef );! classifyFlow.writeDOT( "dot/classify.dot" );! classifyFlow.complete();! }
  • 68.
    Pattern – scorea model, using pre-defined Cascading app Customer Orders Classify Scored Orders Assert GroupBy token M R PMML Model Count Failure Traps Confusion Matrix
  • 69.
    Pattern – scorea model, using pre-defined Cascading app ## run an RF classifier at scale!  ! hadoop jar build/libs/pattern.jar ! data/sample.tsv out/classify out/trap ! --pmml data/sample.rf.xml!  ! !
  • 70.
    Data Workflows for MachineLearning: 
 Summary ! 
 “feedback and discussion” ?
  • 72.
    Nine Points forData Workflows — a scorecard Spark, MLbase Oryx Summing! bird Cascalog Cascading ! KNIME Py Data R Markdown ✔ ✔ ✔ ✔ includes people, exceptional data separation of concerns multiple 
 abstraction layers testing in depth future-proof 
 system integration visualize to collab can haz monoids blends batch + 
 “real-time” optimize learners 
 in context can haz PMML ✔ MBrace
  • 73.
    PMML – what’sneeded? in the language standard: ! data preparation ! data sources/sinks as parameters ! track data lineage ! more monoid, less XML ! handle data exceptions => TDD ! updates/active learning? feature engineering data data sources data sources sources data prep pipeline tournaments hold-outs train learners classifiers classifiers classifiers ! tournaments? scoring test representation evaluation optimization use cases ! in the workflow frameworks: ! include feature engineering w/ model? ! support evaluation better ! build ensembles • obtain other data? • improve metadata? • refine representation? • improve optimization? • iterate with stakeholders? • can models (inference) inform first principles approaches? quantify and measure: • benefit? • risk? • operational costs?
  • 74.
    Enterprise Data Workflowswith Cascading O’Reilly, 2013 shop.oreilly.com/product/ 0636920028536.do ! monthly newsletter for updates, events, conference summaries, etc.: liber118.com/pxn/