Recommended
More Related Content
What's hot
What's hot (18)
Similar to Data platform conference in Tokyo 2017 - Vupico - Credit scoring using Kafka, NiFi Spark, Hadoop, SAP Vora, SAP HANA and Tableau
Similar to Data platform conference in Tokyo 2017 - Vupico - Credit scoring using Kafka, NiFi Spark, Hadoop, SAP Vora, SAP HANA and Tableau (20)
Recently uploaded
Recently uploaded (20)
Data platform conference in Tokyo 2017 - Vupico - Credit scoring using Kafka, NiFi Spark, Hadoop, SAP Vora, SAP HANA and Tableau
- 6. © Data Platform Conference Tokyo 2017. All Rights Reserved6 What we see Suppliers サプライヤー Customers 顧客 Partners パートナー Affiliated firms / subsidiaries 子会社 Mother Company 親会社 Legacy layer(s) Current layer(s) R&D systems 開発部の 情報システム R&D Related system(s) CAD EBOM PLM Mainframe Invoice system Reporting Tool Quality System AR System SRM Pricing System EDI Productio n Planning Product Distributio n Delivery Mgt AP System Staging System Bank(s) 銀行 生産情報 システム Production Information System Production Related system(s) Quality Control Production Declaratio n Process Control Inventory Control PLC Mgt PLCPLC PLC I/F Server I/F Server ETL / ESB New ERP BI & Reporting CallCenter MarketingSales Forecast CRM HRM SRM DWH Payroll Route Planning Production Plan SCM CustomersSuppliers サプライヤー Multi-layer of information means lots of data exchange : § Through Point to point à 1 purpose = 1 Flow 2地点間構成 § multiple ESB / ETL tools § Batch & time driven for most cases § バッチとスケジューラで起動 § But sometimes highly segregated § 分離されてる
- 7. © Data Platform Conference Tokyo 2017. All Rights Reserved7 What we see Suppliers サプライヤー Customers 顧客 Partners パートナー Affiliated firms / subsidiaries 子会社 Mother Company 親会社 Legacy layer(s) Current layer(s) R&D systems 開発部の 情報システム R&D Related system(s) CAD EBOM PLM Mainframe Invoice system Reporting Tool Quality System AR System SRM Pricing System EDI Productio n Planning Product Distributio n Delivery Mgt AP System Staging System Bank(s) 銀行 生産情報 システム Production Information System Production Related system(s) Quality Control Production Declaratio n Process Control Inventory Control PLC Mgt PLCPLC PLC New ERP BI & Reporting CallCenter MarketingSales Forecast CRM HRM SRM DWH Payroll Route Planning Production Plan SCM Customers 顧客Suppliers サプライヤー Data everywhere: § Data replicated in each systems各システムで複製されたデータ § Data enriched based on requirements 要件に基づいて設定されたデータを エンリッチされ § Weak data / master data governance 弱いデータもしくはマスターデータガ バナンス § Multiple technologies implemented by multiple vendors 複数のベンダーに よって実装された複数のテクノロジ § Application built based on a specific requirement not necessarily transversal or with reusability in mind § 再利用性を考えずに一つの要求に対して一つのソリューションを作る
- 8. © Data Platform Conference Tokyo 2017. All Rights Reserved8 Vupico is a global entity to help companies through their Data journey u Supporting analytics in Japan since 2014 u 2014から日本の企業をサポートしており、 特にアナリティクス面では大きな実績を 上げております u 7 locations and analytics is all we Do! u ７カ国に支社があり u Focused on generating value to clients through Analytics u アナリティクスを中心になって： – Hadoop – SAP® HANA、SAP Vora, – Analytics u Official Consulting Partner of /正式のパトーナー：
- 9. © Data Platform Conference Tokyo 2017. All Rights Reserved9 We help you at every stage of your journey エンドツーエンドのサポート Strategy / Roadmap 戦略 / ロード マップ Business & Use case ビジネス＆ユー ス ケース Proof of Concept Maturity Assessment 相互 アセス メント KPI Definition KPI 定義 Data Discovery データ・ディ スカバリー Landscape Analysis ランドスケー プ分析 Process Evaluation プロセス 評 価 Capability Assessment ケイパビリ ティ評価 Project Mgt プロジェクト・ マネジメント Technology Assessment テクノロ ジー・アセス メント Auditing システム 監査 Governance ガバナンス Change Mgt 変更管理 Training トレー ニング Operation Monitoring オペレーショ ン・モニタリン グ Readiness Realization Operation Platform Enhancement プラットフォー ム 改善Predictive Modeling & Build 予測モデリン グ& 構築 Solution Design & Build ソリューション 設計& 構築 Platform Design & Build プラットフォー ム 設計&構築 Data ETL / Pipeline Design & Build Data ETL / Pipeline 設計& 構築 UI Design & Build UI 設計 & 構 築
- 10. © Data Platform Conference Tokyo 2017. All Rights Reserved10 Vupico main area of expertise / ヴピコの専門知識 Skill Set スキル Japan India Singapore U.S.A Australia / New Zealand / Philippines Verticals 業種別 Strategy, Governanceせん ◯ ◯ ◯ ◯ ◯ § Banking § Financial Administration § Food processing § Healthcare § Insurance § IT § Mortgage § Oil & Gas § Pharmaceutical § Retail & CPG Manufacturing § Semiconductor § Telecomm § Utilities – Gas & Electric § Warehousing Data Governance ◯ ◯ ◯ ◯ ◯ PMO, PM ◯ ◯ Solution Architecture ◯ ◯ Data Scientist ◯ ◯ ◯ AI / ML / Deep learning ◯ ◯ ◯ Hadoop ◯ ◯ Spark ◯ ◯ NiFi ◯ ◯ Other ETL (Informatica,…) ◯ ◯ SAP Functional ◯ ◯ ◯ SAP HANA ◯ ◯ SAP VORA ◯ ◯ SAP BW ◯ ◯
- 11. © Data Platform Conference Tokyo 2017. All Rights Reserved11 Generic Architecture Upstream downstreamHadoop Historical data Transactional data ExternalInternal Market data SNS / iOT data Master data Hardware HDFS Hive Kafka NiFi Impala Storm Samza Flink HBase Phoenix TensorFlow, scikit, pytorch, sparklingWater(H2O), Caffe, … Ambari, Cloudbreak, Knox, Oozie, Slider, Accumulo, falcon, Pig, … Flume Ranger Atlas Hana DBApps Spark ML SAP Vora Drill Tenso rFlow Knime RStudio Ambari Slider FalconPig Knox H2O Clouder a Mger Presto Zoo keeper AD / LDAP Zeppelin Yarn Real Time event , Streaming & Dataflow Druid Sqoop Solr Oozie
- 12. © Data Platform Conference Tokyo 2017. All Rights Reserved12 Use case – Credit scoring クレジットスコア § The Customer European bank wanted to learn more how they could redesign their credit scoring system as it is currently heavily relying on a paper based process. 欧州銀行のため、与信審査のプロセスを、書類ベースから自動化のフロー にするための提案 § The use case Show case to the customer a prototype that would enable them to manage their credit scoring process with the following capabilities: § Be able to give an answer right away for low risk customers § リスクの低い申し込みをすぐ判断 § Be able to review customer almost accepter through self-service analytics § リスクのちょっとある顧客をレビューできる、セルフサービス分析 § Flexible model that can be extended § 特徴を追加することは追加可能モデル Vupico added value & services § Data Exploration データ調査 § Architecting solution ソリューションの設計 § Platform configuration プラットフォーム構成 § Data modeling データモデリング § Model creation分析モデル § Rapid prototyping of the demoデモの準備 § Business knowledge of financial organization 金融機関のビジネス知識
- 13. © Data Platform Conference Tokyo 2017. All Rights Reserved13 Application Submission 申込書提出 Data Collection 関連情報収集 Reference Checking 情報真偽の確認 Evaluation 評価 Decision 最終判断 Client prepares necessary documents, answer forms and Operator confirms that all documents are properly provided, verify against credit reports (RMCR,…) Operator checks accuracy of personal data provided (client financial history,…) Credit analyst make a recommendation Credit officer / committee approves or rejects How long, how much Use case - Scoring standard process 一般的な与信審査のプロセス
- 14. © Data Platform Conference Tokyo 2017. All Rights Reserved14 Bank 銀行 Bank account Balance 口座残高 Credit history 支払い実績 Other Debtors / guarantors Other Installment plans 割賦プラン # of credits at this bank ローンの数 Creditability Accept Refuse Scoring Present employment since Occupation 職業 Foreign worker 外国人 Work 仕事 Marital situation & gender 性別 未婚 Age 年齢 # of depend people 扶養家族 Individual Property Housing 住宅 Telephone 携帯 Asset 資産 Duration (month) 期間 Purpose 目的 Credit Amount 与信枠 Installment Rate in % of dispo income Loan 融資 the model gives: probability of the applicant repaying the requested amount of loan on a scale between 225 and 900. ２２５から９００のスコアーレンジの支払い能 力の数値化 Use Case - Introducing the model’s Features 特徴の紹介
- 15. © Data Platform Conference Tokyo 2017. All Rights Reserved15 Use Case – Vupico Thought process Encoding Data Selecting potential classifier Tune hyper- parameters Cross validation Correlation Normalization Compare model performance A-B à 010401 A à 1 ~ 20 B à 1 ~ 25 f (2) = (x)2 +2(y) x =1, y=4 X=4, y=16 X=7, y= 18 1 Encoding the data データの記号化 Convert data from text based to numerical values to easily use it in a model モデルに簡単に使えるように文章のデータを数値化 e.g.: balance is coded as: 1 : ... < 0 DM 2 : 0 <= ... < 200 DM 3 : ... >= 200 DM / salary assignments for at least 1 year
- 16. © Data Platform Conference Tokyo 2017. All Rights Reserved16 Use Case - Vupico Thought process Encoding Data Selecting potential classifier Tune hyper- parameters Cross validation Correlation Normalization Compare model performance A-B à 010401 A à 1 ~ 20 B à 1 ~ 25 f (2) = (x)2 +2(y) x =1, y=4 X=4, y=16 X=7, y= 18 2 Selecting the classifier 分類器の選択 Choose the potential models that can be applied to the data at hand. In this case we chose 2 candidates: Naïve Bayes １/ナイーブベイズ：独立した（ナイーブ）特徴が必 要, Random Forests 2/ランダムフォレスト：決定木の特徴が強くは出た ため, 過剰適合となり、不規則なパターンが出てし まう
- 17. © Data Platform Conference Tokyo 2017. All Rights Reserved17 Use Case - Vupico Thought process Encoding Data Selecting potential classifier Tune hyper- parameters Cross validation Correlation Normalization Compare model performance A-B à 010401 A à 1 ~ 20 B à 1 ~ 25 f (2) = (x)2 +2(y) x =1, y=4 X=4, y=16 X=7, y= 18 3 Correlation 相関性 Verify correlation between features (especially for Naïve Bayes approach) 相関性ヒートマップ： ナイーブベイズは強い特徴 想定で一番うまく作用する相関性ヒートマップは相 関性マトリックスのグラフ表示です。 Correlation Heatmap 相関性ヒートマップ
- 18. © Data Platform Conference Tokyo 2017. All Rights Reserved18 Encoding Data Selecting potential classifier Tune hyper- parameters Cross validation Correlation Normalization Compare model performance A-B à 010401 A à 1 ~ 20 B à 1 ~ 25 f (2) = (x)2 +2(y) x =1, y=4 X=4, y=16 X=7, y= 18 4 Normalization Normalizing fields to use similar scale with a mean around 0 applying : データは多くの特徴があり、その結果、アルゴ リズムでの計算時間が掛かってしまう、その ため、ｘ／||ｘ||を適用し、標準化を使い全て のフィールドを同様なスケールにすることで、 バラツキを少なくします。 X / ||X|| Use Case - Vupico Thought process
- 19. © Data Platform Conference Tokyo 2017. All Rights Reserved19 Use Case - Vupico Thought process Encoding Data Selecting potential classifier Tune hyper- parameters Cross validation Correlation Normalization Compare model performance A-B à 010401 A à 1 ~ 20 B à 1 ~ 25 f (2) = (x)2 +2(y) x =1, y=4 X=4, y=16 X=7, y= 18 5 Comparing models performance With default settings デフォルト設定で比較モデルパフォーマンス Train and verify the performance of each candidate model ７０%のデータをトレーニングに使い、３０％をテストとモデ ル性能評価に使う。 70% for training 30% for testing & validation 私達のケースでは、重要なことは適正はポジティブレート とＦ１スコアを見つける回収（リコール）です。 Random forest is the best For our use case 1/ Naïve bayes (Gaussian Implementation): 2/ Random forest:
- 20. © Data Platform Conference Tokyo 2017. All Rights Reserved20 Encoding Data Selecting potential classifier Tune hyper- parameters Cross validation Correlation Normalization Compare model performance A-B à 010401 A à 1 ~ 20 B à 1 ~ 25 f (2) = (x)2 +2(y) x =1, y=4 X=4, y=16 X=7, y= 18 6 Tuning hyper-parameters ハイパーパラミターの調整 Fine tune user defined hyper-parameters: § N-estimators § min_samples_leaf § Oob_score § Random_state § Class-weight § n_jobs Use Case - Vupico Thought process
- 21. © Data Platform Conference Tokyo 2017. All Rights Reserved21 Use Case - Vupico Thought process Encoding Data Selecting potential classifier Tune hyper- parameters Cross validation Correlation Normalization Compare model performance A-B à 010401 A à 1 ~ 20 B à 1 ~ 25 f (2) = (x)2 +2(y) x =1, y=4 X=4, y=16 X=7, y= 18 7 Grid-search: cross validation グリッドサーチ：クロス検証 Grid search: cross validation to select the best Perform on the whole dataset all combination of specified hyper-parameters to determine the best parameter combination. 最善のパラメーター組み合わせを決めるため特定 されたハイパーパラメターの全ての組み合わせを 全データセットで実行するcombination of parameters Result Recall score as well as the F1 score are both at 98%therefore we have a very good score Using the model we can generate a probability score representing the risk F1スコアとリコール・スコア共に９８％となり、大変良い スコア結果を得ました。 このモデルを使い、リスクを示す確率スコアを生成す ることが出来ました
- 22. © Data Platform Conference Tokyo 2017. All Rights Reserved22 Use Case – Flow § The Architecture Upstream downstreamHadoop Historical data Transactional data ExternalInternal Market data SNS or iOT data Master data SAP Hana Hardware (NEC) HDFS Outbound datasetInbound dataset < 3 years 3年以上のデータ NiFi >= 3 years 1 2 4 Hive delta Scored dataset Spar kML 3 Kafka SAP VORA Whole dataset of scored data 全てのデータセット Last 3 years of scored data 3年以下のデータ
- 23. © Data Platform Conference Tokyo 2017. All Rights Reserved23 Vupico supports the deployment of multiple projects using SAP Hana & Vora in Japan ヴピコは現在SAP HanaまたVoraを使い複数の日本の プロジェクトの提供をサーポートしてます What is SAP Vora & SAP Hana, HanaとVoraとは？ SAP Vora SAP HANA In Memory Data-storage インメモリ データ保存 Integration with Ambari アンバリと統 合 Leverage your current Hadoop Resources 手元にあるリ ソースを使うAvailable from a SparkContext Spark Contextから 使用 OLAP hierarchies Structure OLAP構造 In Memory data-storage & processing インメモリ データ保存と アプリ More than a Database, host applications ,… Good integration with SparkContext SaprkContext と強い連携 はできる Business Integration ブジネスの 関連さは強 い Easy Access through JDBC JDBCアクセス Appliance アプライアン スSDA JDBC to Hive SAP SparkController
- 24. © Data Platform Conference Tokyo 2017. All Rights Reserved24 Role series Specs Sofware SAP Hana NEC NX7700ｘ A2010M- 60 § OS: RHEL 7.2 § CPU: 4 CPU 60 cores @ 2.8GHz § RAM: 1TB § Disk: x8 900GB HDD § SAP HANA 2.0 SPS02 Hadoop Master Node NEC Express 5800 R120e-1M § OS: RHEL 7.2 § CPU: 8 cores x 2 § RAM: 128GB § Disk: 2x 100GB § SAP VORA 1.4 § Hadoop HDP 2.6.2 Hadoop Data Node NEC Express 5800 R120e-1M § OS: RHEL 7.2 § CPU: 6 cores x2 § RAM: 256GB § Disk: 2x 1TB + 12x 4TB Tableau NEC Express 5800 R120e-1M § OS: Windows Server § CPU: 8 cores x2 § RAM: 128GB § Disk: 2x 100GB Use Case – Technical platform x1 x2 x3 x1
- 26. © Data Platform Conference Tokyo 2017. All Rights Reserved26 NiFi HDFS Hive GetFileExecStreamCmd data_gen.py Result.csv PutHDFS 1506424816479 Result.csv t_inbound_loan_request_txt_p /t_outbound_loan_request_txt_p 1506424816479 Result.csv Loan_scoring SAP VORA SAP HANA V + t_outbound_loan_request comsumeKafka LFS 1) Run scoring & save to HDFS 2) Ingest into SAP Hana 3) Ingest into SAP Vora /data/inbound/ /data/outbound/ t_outbound_loan_request_txt_p /home/vupico/genData T_inbound_loan_request_txt_p vora_t_outbound_loan_request t_outbound_loan_request cv_t_outbound _loan_request Tableau Step 1: inbound data in Hive (before scoring)
- 28. © Data Platform Conference Tokyo 2017. All Rights Reserved28 NiFi HDFS Hive GetFileExecStreamCmd data_gen.py Result.csv PutHDFS 1506424816479 Result.csv t_inbound_loan_request_txt_p /t_outbound_loan_request_txt_p 1506424816479 Result.csv Loan_scoring SAP VORA SAP HANA V + t_outbound_loan_request comsumeKafka LFS 1) Run scoring & save to HDFS 2) Ingest into SAP Hana 3) Ingest into SAP Vora /data/inbound/ /data/outbound/ t_outbound_loan_request_txt_p /home/vupico/genData T_inbound_loan_request_txt_p vora_t_outbound_loan_request t_outbound_loan_request cv_t_outbound _loan_request Tableau Hive – outbound data (after scoring)
- 30. © Data Platform Conference Tokyo 2017. All Rights Reserved30 NiFi HDFS Hive GetFileExecStreamCmd data_gen.py Result.csv PutHDFS 1506424816479 Result.csv t_inbound_loan_request_txt_p /t_outbound_loan_request_txt_p 1506424816479 Result.csv Loan_scoring SAP VORA SAP HANA V + t_outbound_loan_request comsumeKafka LFS 1) Run scoring & save to HDFS 2) Ingest into SAP Hana 3) Ingest into SAP Vora /data/inbound/ /data/outbound/ t_outbound_loan_request_txt_p /home/vupico/genData T_inbound_loan_request_txt_p vora_t_outbound_loan_request t_outbound_loan_request cv_t_outbound _loan_request Tableau VORA – outbound data (after scoring)
- 32. © Data Platform Conference Tokyo 2017. All Rights Reserved32 NiFi HDFS Hive GetFileExecStreamCmd data_gen.py Result.csv PutHDFS 1506424816479 Result.csv t_inbound_loan_request_txt_p /t_outbound_loan_request_txt_p 1506424816479 Result.csv loan_scoring SAP VORA SAP HANA V + t_outbound_loan_request comsumeKafka LFS 1) Run scoring & save to HDFS 2) Ingest into SAP Hana 3) Ingest into SAP Vora /data/inbound/ /data/outbound/ t_outbound_loan_request_txt_p /home/vupico/genData T_inbound_loan_request_txt_p vora_t_outbound_loan_request t_outbound_loan_request cv_t_outbound _loan_request Tableau SAP HANA – outbound data (after scoring)
- 33. © Data Platform Conference Tokyo 2017. All Rights Reserved33 SAP HANA – outbound data (after scoring) Vora Virtual table in Hana Physical table in Hana On Hana as well we can see that both tables, SAP Vora virtual table as well as Hana physical table have both the necessary records. The calc view will make sure that we take the necessary data from each system