Dublin Ireland Spark Meetup October 15, 2015

Copyright © 2014 BAE Systems. All Rights Reserved.
BAE Systems is a trade mark of BAE Systems plc 1
Applied Intelligence
Company Confidential
APACHE SPARK DATA FRAMES
FOR FRAUD DETECTION
Eddie Baggott
15th October 2015

INTRODUCTION
• Functional and Data Architect
• BAE Systems , Norkom
• Anti Fraud ,AML , Compliance , Watch lists, Cyber
• Big data?
• Beginner in spark…No java or Scala experience
• All my own opinion

OVERVIEW
DataFrames

WHAT ARE DATAFRAMES
• Distributed collection of rows organized into named columns
• An abstraction for selecting, filtering, aggregating and plotting structured
data (R,Pandas)
• Schema RDD
• Named columns
• Spark SQL
• RDD-style methods such as Map, flatMap are available
• Full Spark Stack (MLLIb, GraphX) available
• Can switch between RDD and DF with .rdd and .toDF
• Many data sources and formats. Easy to read from:
• JSON ,Parquet, Hive , CSV ,HDFS ,S3 ,JDBC, Cassandra
• Evaluated lazily
• optimized
•

USING DATAFRAMES
• Selecting, Filtering, Aggregating structured data
• Write Less code
• Read Less data
• Let the optimizer do the hard work
• Old style
wordCount = data.flatMap(_.split(" ")).map(_ ->
1).reduceByKey(_+_)
• SQL
sqlContext.sql("select count(*) , AccountId FROM
TRANSACTIONS GROUP BY AccountId")
• Programmatically
Txn_df.groupBy(“AccountId").count()

CREATE A DATAFRAME
case class Account_View ( RunDate: String ,txnBaseAmount: Double
,TransactionType: String ,AccountId: String ,CreditDebitCode: String )
val df_txn = sc.textFile("TRANSACTIONS.txt").map(_.split('|')).map(p =>
Account_View(p(0), p(4).trim.toDouble, p(7), p(8), p(5))).toDF
val df_cash = df_txn.filter("TransactionType = 'TT0090'").groupBy("AccountId").sum()
df_cash. printSchema
|-- AccountId: string (nullable = true)
|-- SUM(txnBaseAmount): double (nullable = true)
df_cash.show
| AccountId | SUM(txnBaseAmount)|
|account8849-25| 54.0|
|account16705-2| 65.0|

SQL EXAMPLE
//register the table with hive metadata
df_txn.registerTempTable("TRANSACTIONS")
//run query
val df_agg = sqlContext.sql("select RunDate, AccountId , CreditDebitCode
, COUNT(TransactionId) AS COUNT_TXN , SUM(txnBaseAmount)
AS TOTAL_AMOUNT FROM TRANSACTIONS
GROUP BY RunDate, AccountId, CreditDebitCode ")

PROGRAMMATICALLY
// read from json or parquet
val df_json = sqlContext.read.json("txn_json")
//schema discovery
df_json.printSchema
//aggregate
df_json.filter(" CreditDebitCode = 'Credit'")
.groupBy("AccountId").agg(sum("txnBaseAmount"), max("txnBaseAmount") )
//join different data sources
df_json.join (df_txn.as('b), $"a.TransactionId" === $"b.TransactionId")
//sql
data.registerTempTable("TXN_JSON")
sqlContext.sql("select distinct a.TransactionType from TRANSACTIONS a JOIN
TXN_JSON b where a.TransactionId = b.TransactionId ")

NEW FEATURES
• Windows 1.4
• Analytic functions
• Ranking functions
.over(…)
.PartitionBy,.OrderBy, Frame( Row or Range)
• New in 1.5
• 90 functions
• Math/date/condition/string/etc.
• https://issues.apache.org/jira/browse/SPARK-8159
• Spark Packages http://spark-packages.org/
• spark-csv from DataBricks
• https://github.com/lift/lift/tree/master/framework/lift-base/lift-json/

APPLICATIONS
 Profiling
 Build aggregates
 Detection with CEP
• Charting
• Crossfilter and Other BI tools
• Data Quality
• Need to understand the data from a new Client
• Finding Networks
• Fraud and Ml rings
• GraphX and PageRank
• ETL
• Analytics
• Recommendations

PROFILING
//Data labels
{ "Name":“Credit", "Field": " CreditDebitCode ", "Value": " Credit " ,
"Operator":"=" } ] }
{ "Name":"Cash", "Field": " INSTRUMENT ", "Value": "Cash" , "Operator":"=" }
] }
//Profile Definition
{ "id":2, "ProfileName":“CashCredit" , "Operator" : "Accumulator" ,
“Data_Labels": [ { "Name":" Credit " } ,{Name:"Cash" } ] }
//CEP rules engine
If CashCredit.sum >10,000 Then Alert
//File layout
RunDate|x|x|AccountId|AmountBaseCurrency|
CreditDebitCode|TransactionID|TransactionType|UnusedField|…

CHARTING

CROSSFILTER CHARTS (1)
• Three open source JavaScript libraries
• D3.js
• Crossfilter.js
• DC.js
• https://dc-js.github.io/dc.js/docs/stock.html
• Data in JSON format from previous example
• Reduce 30m transactions down to 40k rows

• Define Dimension (x-axis)
• var dateDim = ndx.dimension(function(d) {return d.rundate;});
• var creditordebitDim = ndx.dimension(function (d) {return d.creditordebit;});
• var txnTypeDim = ndx.dimension(function (d) {return d.transactiontype;});
• Define Metrics ( groups)
• var amount = dateDim.group().reduceSum(function(d) {return d.total_amount/10000;});
• var txnCreditOrDebitCount = creditordebitDim.group().reduceSum(function (d) { return
d.count_txn; })
• var txnTypeCount = txnTypeDim.group().reduceSum(function (d) { return d.count_txn; })

• Create Charts
txnType_chart
.dimension(txnTypeDim)
.group(txnTypeCount)
.cap(10)
.width(500)
.height(600)
.colors(d3.scale.category10())
.ordering(function(d) { return -d.value })
.title(function (d) { return d.value; } )
.elasticX(true)
.xAxis().ticks(4).tickFormat(d3.format(".2s"));

DQ REPORT
• Early stages of New Data from a Client
• New to understand the data to create good scenarios
• New to be flexible , fast and easy to use
• Frequent values
.stat.freqItems

OTHER APPLICATIONS
• Finding Networks
• Fraud and ML rings
• GraphX and PageRank
• ETL
• Identity Resolution
• Analytics
• Ml Lib
• Recommendations

RECAP
• DataFrames make Spark easy to use
• Can reuse existing SQL code and skills
• Fast to write code with Scala REPL
• Lots of business applications for Spark
• New ways of thinking about working with data to get information
• We’re hiring!
• http://www.baesystems.com/en/cybersecurity
• Thanks to Brendan and his team here
• Encourage anyone to give a presentation
• Any Questions?

Dublin Ireland Spark Meetup October 15, 2015

Recommended

Recommended

More Related Content

What's hot

What's hot (12)

Similar to Dublin Ireland Spark Meetup October 15, 2015

Similar to Dublin Ireland Spark Meetup October 15, 2015 (20)

Recently uploaded

Recently uploaded (20)

Dublin Ireland Spark Meetup October 15, 2015