SlideShare a Scribd company logo
1 of 26
使用開源資料庫 arangoDB的奇幻之旅
2020 / 08 / 01
11:30 ~ 12:00
TR410 范建銘
團隊介紹
范建銘 林韵馨
about me
2008~2009 JAVA信用卡M-FES
2010~2016 PHP 業績計算
2012~ PHP 金流串接
2016~ open source 內部系統使用
2017~ AWS Serverless 推播系統
2018 IThome Serviceless All-star
2018~ GraphDatabase - arangoDB
2018 IThome ModernWeb Graph Database應用思考模式
2019 IThome ModernWeb 使用Pentaho Kettle ETL做系統資料後盾
2019 COSCUP 用 ETL Kettle打造程序流水線
故事前景提要
某電商發出推播通知就成
用戶DDOS自己主機
RDB內上億筆資料成為最大負擔
NOSQL種類
圖片出處:https://www.simplilearn.com/introduction-to-nosql-databases-tutorial-video
Graph-Based
由節點、關係、屬性組合而成的資料結構
節點 節點
關係
屬性
name:B關係內可
定義屬性
參加者
活動
參加者
投稿:演講主題
成功:是
搶票:成功
參與講座:
[{class:TR410-1}]
身份:聽眾
身份:講者
主辦:
COSCUP
選擇適合的軟體
排名出處:https://db-engines.com/en/ranking/graph+dbms
多模型資料庫:Graph、Document、KeyValue
內建API Fromwork:FOXX(JavaScript)
AQL (Arango Query Language) 查詢語法
官方文件教學完整
社群討論熱烈
社群版能使用CLUSTER
導入困難!!
軟體知名度太低
資源有限,不可能直接買貴貴的付費DB
原商業邏輯超複雜
自己也沒太大把握/____
突破困境!!
**一定要先搞定商業邏輯!!**
POC直接測試上億資料量
不只自己學,更要教會長官
不要在意酸言酸語
如何實作呢?
ID 客戶
編號
ID 折價券
券號
客戶
ID
折價券
ID
使用
狀況
折價券客戶
使用狀況:
False
客戶編號:
A1111111
11
折價券編號:
FG20201028
折價券客戶
使用狀況:
False
定義Graph 方向性
INBOUND / OUTBOUND
from
to
AQL寫法 計算某人的折價券張數
For v,e,p IN 1..1 OUTBOUND concat(‘A_cust/A111111111’)
GRAPH ‘Coupon’
FILTER p.vertices[1].START <= DATE_NOW()
AND p.vertices[1].END >=DATE_NOW()
AND p.edges[0].GET==‘1’
COLLECT WITH COUNT INTO COUPON_COUNT
RETURN COUPON_COUNT
For v,e,p IN 1..1 OUTBOUND concat(‘A_cust/A111111111’)
GRAPH ‘Coupon’
v : vertex
e : edge
p : path
1..1 min..max
INBOUND / OUTBOUND
GRAPH graphName [0]
[1]
FILTER p.vertices[1].START <= DATE_NOW()
AND p.vertices[1].END >=DATE_NOW()
AND p.edges[0].GET==‘1’
COLLECT WITH COUNT INTO COUPON_COUNT
RETURN COUPON_COUNT
過濾節點或關係的條件
再回傳
JavaScript framework : FOXX
Apache JMeter壓測:
4 CPU+32G RAM PC
Query 2000 次/秒
2 毫秒/次
沒有一次到位的架構
F5
維運學問大!
● 要了解軟體底層特性
○ 軟體CONF檔內參數設定實驗
○ Linux kernel 參數搭配調整
● 一定要監控啊!!
○ 設定標準值(Memory 或CPU)
不斷重新檢視系統
調整AQL語法內容
作業系統與軟體各項參數長期觀察並統計
主機台數及瞬間高流量觀察
硬碟換SSD
穩了~!
2018 年 雙11 安全過關,但硬體效能跟
不太上,重啟主機釋放資源
2019 年 雙11 安全過關,喝咖啡聊天中
渡過
接班問題......
門檻太高,內部同仁不願意學
知名度低,找不到有經驗新人
感謝大家聆聽這段旅程

More Related Content

What's hot

Querying Linked Data with SPARQL
Querying Linked Data with SPARQLQuerying Linked Data with SPARQL
Querying Linked Data with SPARQLOlaf Hartig
 
Training Series: Build APIs with Neo4j GraphQL Library
Training Series: Build APIs with Neo4j GraphQL LibraryTraining Series: Build APIs with Neo4j GraphQL Library
Training Series: Build APIs with Neo4j GraphQL LibraryNeo4j
 
Non Relational Databases
Non Relational DatabasesNon Relational Databases
Non Relational DatabasesChris Baglieri
 
FIBO in Neo4j: Applying Knowledge Graphs in the Financial Industry
FIBO in Neo4j: Applying Knowledge Graphs in the Financial IndustryFIBO in Neo4j: Applying Knowledge Graphs in the Financial Industry
FIBO in Neo4j: Applying Knowledge Graphs in the Financial IndustryNeo4j
 
Building Lakehouses on Delta Lake with SQL Analytics Primer
Building Lakehouses on Delta Lake with SQL Analytics PrimerBuilding Lakehouses on Delta Lake with SQL Analytics Primer
Building Lakehouses on Delta Lake with SQL Analytics PrimerDatabricks
 
Intro to Cypher
Intro to CypherIntro to Cypher
Intro to CypherNeo4j
 
Introduction to MongoDB.pptx
Introduction to MongoDB.pptxIntroduction to MongoDB.pptx
Introduction to MongoDB.pptxSurya937648
 
Machine Learning using Apache Spark MLlib
Machine Learning using Apache Spark MLlibMachine Learning using Apache Spark MLlib
Machine Learning using Apache Spark MLlibIMC Institute
 
Databricks secure deployments and security baselines, doug march 2022
Databricks secure deployments and security baselines, doug march 2022Databricks secure deployments and security baselines, doug march 2022
Databricks secure deployments and security baselines, doug march 2022Henrik Brattlie
 
Data Quality With or Without Apache Spark and Its Ecosystem
Data Quality With or Without Apache Spark and Its EcosystemData Quality With or Without Apache Spark and Its Ecosystem
Data Quality With or Without Apache Spark and Its EcosystemDatabricks
 
Knowledge Graphs for Transformation: Dynamic Context for the Intelligent Ente...
Knowledge Graphs for Transformation: Dynamic Context for the Intelligent Ente...Knowledge Graphs for Transformation: Dynamic Context for the Intelligent Ente...
Knowledge Graphs for Transformation: Dynamic Context for the Intelligent Ente...Neo4j
 
Intro to Neo4j - Nicole White
Intro to Neo4j - Nicole WhiteIntro to Neo4j - Nicole White
Intro to Neo4j - Nicole WhiteNeo4j
 
Unified Big Data Processing with Apache Spark (QCON 2014)
Unified Big Data Processing with Apache Spark (QCON 2014)Unified Big Data Processing with Apache Spark (QCON 2014)
Unified Big Data Processing with Apache Spark (QCON 2014)Databricks
 
Querying the Wikidata Knowledge Graph
Querying the Wikidata Knowledge GraphQuerying the Wikidata Knowledge Graph
Querying the Wikidata Knowledge GraphIoan Toma
 
Knowledge Graphs - The Power of Graph-Based Search
Knowledge Graphs - The Power of Graph-Based SearchKnowledge Graphs - The Power of Graph-Based Search
Knowledge Graphs - The Power of Graph-Based SearchNeo4j
 
Azure Data Factory Data Flows Training v005
Azure Data Factory Data Flows Training v005Azure Data Factory Data Flows Training v005
Azure Data Factory Data Flows Training v005Mark Kromer
 
Data Quality Patterns in the Cloud with Azure Data Factory
Data Quality Patterns in the Cloud with Azure Data FactoryData Quality Patterns in the Cloud with Azure Data Factory
Data Quality Patterns in the Cloud with Azure Data FactoryMark Kromer
 
Considerations for Data Access in the Lakehouse
Considerations for Data Access in the LakehouseConsiderations for Data Access in the Lakehouse
Considerations for Data Access in the LakehouseDatabricks
 
Make your PySpark Data Fly with Arrow!
Make your PySpark Data Fly with Arrow!Make your PySpark Data Fly with Arrow!
Make your PySpark Data Fly with Arrow!Databricks
 
Announcing Databricks Cloud (Spark Summit 2014)
Announcing Databricks Cloud (Spark Summit 2014)Announcing Databricks Cloud (Spark Summit 2014)
Announcing Databricks Cloud (Spark Summit 2014)Databricks
 

What's hot (20)

Querying Linked Data with SPARQL
Querying Linked Data with SPARQLQuerying Linked Data with SPARQL
Querying Linked Data with SPARQL
 
Training Series: Build APIs with Neo4j GraphQL Library
Training Series: Build APIs with Neo4j GraphQL LibraryTraining Series: Build APIs with Neo4j GraphQL Library
Training Series: Build APIs with Neo4j GraphQL Library
 
Non Relational Databases
Non Relational DatabasesNon Relational Databases
Non Relational Databases
 
FIBO in Neo4j: Applying Knowledge Graphs in the Financial Industry
FIBO in Neo4j: Applying Knowledge Graphs in the Financial IndustryFIBO in Neo4j: Applying Knowledge Graphs in the Financial Industry
FIBO in Neo4j: Applying Knowledge Graphs in the Financial Industry
 
Building Lakehouses on Delta Lake with SQL Analytics Primer
Building Lakehouses on Delta Lake with SQL Analytics PrimerBuilding Lakehouses on Delta Lake with SQL Analytics Primer
Building Lakehouses on Delta Lake with SQL Analytics Primer
 
Intro to Cypher
Intro to CypherIntro to Cypher
Intro to Cypher
 
Introduction to MongoDB.pptx
Introduction to MongoDB.pptxIntroduction to MongoDB.pptx
Introduction to MongoDB.pptx
 
Machine Learning using Apache Spark MLlib
Machine Learning using Apache Spark MLlibMachine Learning using Apache Spark MLlib
Machine Learning using Apache Spark MLlib
 
Databricks secure deployments and security baselines, doug march 2022
Databricks secure deployments and security baselines, doug march 2022Databricks secure deployments and security baselines, doug march 2022
Databricks secure deployments and security baselines, doug march 2022
 
Data Quality With or Without Apache Spark and Its Ecosystem
Data Quality With or Without Apache Spark and Its EcosystemData Quality With or Without Apache Spark and Its Ecosystem
Data Quality With or Without Apache Spark and Its Ecosystem
 
Knowledge Graphs for Transformation: Dynamic Context for the Intelligent Ente...
Knowledge Graphs for Transformation: Dynamic Context for the Intelligent Ente...Knowledge Graphs for Transformation: Dynamic Context for the Intelligent Ente...
Knowledge Graphs for Transformation: Dynamic Context for the Intelligent Ente...
 
Intro to Neo4j - Nicole White
Intro to Neo4j - Nicole WhiteIntro to Neo4j - Nicole White
Intro to Neo4j - Nicole White
 
Unified Big Data Processing with Apache Spark (QCON 2014)
Unified Big Data Processing with Apache Spark (QCON 2014)Unified Big Data Processing with Apache Spark (QCON 2014)
Unified Big Data Processing with Apache Spark (QCON 2014)
 
Querying the Wikidata Knowledge Graph
Querying the Wikidata Knowledge GraphQuerying the Wikidata Knowledge Graph
Querying the Wikidata Knowledge Graph
 
Knowledge Graphs - The Power of Graph-Based Search
Knowledge Graphs - The Power of Graph-Based SearchKnowledge Graphs - The Power of Graph-Based Search
Knowledge Graphs - The Power of Graph-Based Search
 
Azure Data Factory Data Flows Training v005
Azure Data Factory Data Flows Training v005Azure Data Factory Data Flows Training v005
Azure Data Factory Data Flows Training v005
 
Data Quality Patterns in the Cloud with Azure Data Factory
Data Quality Patterns in the Cloud with Azure Data FactoryData Quality Patterns in the Cloud with Azure Data Factory
Data Quality Patterns in the Cloud with Azure Data Factory
 
Considerations for Data Access in the Lakehouse
Considerations for Data Access in the LakehouseConsiderations for Data Access in the Lakehouse
Considerations for Data Access in the Lakehouse
 
Make your PySpark Data Fly with Arrow!
Make your PySpark Data Fly with Arrow!Make your PySpark Data Fly with Arrow!
Make your PySpark Data Fly with Arrow!
 
Announcing Databricks Cloud (Spark Summit 2014)
Announcing Databricks Cloud (Spark Summit 2014)Announcing Databricks Cloud (Spark Summit 2014)
Announcing Databricks Cloud (Spark Summit 2014)
 

Similar to 2020 COSCUP 使用開源資料庫 arangoDB的奇幻之旅

2020 CloudEege Summit 用Serverless真的方便又省錢?
2020 CloudEege Summit 用Serverless真的方便又省錢?2020 CloudEege Summit 用Serverless真的方便又省錢?
2020 CloudEege Summit 用Serverless真的方便又省錢?Freddy Fan
 
Hadoop 與 SQL 的甜蜜連結
Hadoop 與 SQL 的甜蜜連結Hadoop 與 SQL 的甜蜜連結
Hadoop 與 SQL 的甜蜜連結James Chen
 
選擇正確的Solution 來建置現代化的雲端資料倉儲
選擇正確的Solution 來建置現代化的雲端資料倉儲選擇正確的Solution 來建置現代化的雲端資料倉儲
選擇正確的Solution 來建置現代化的雲端資料倉儲Herman Wu
 
AWS re:Invent Recap 2016 Taiwan part 1
AWS re:Invent Recap 2016 Taiwan part 1AWS re:Invent Recap 2016 Taiwan part 1
AWS re:Invent Recap 2016 Taiwan part 1Amazon Web Services
 
初探 Data API Builder:在幾分鐘內將資料庫轉換成 REST 和 GraphQL 不再是夢想
初探 Data API Builder:在幾分鐘內將資料庫轉換成 REST 和 GraphQL 不再是夢想初探 Data API Builder:在幾分鐘內將資料庫轉換成 REST 和 GraphQL 不再是夢想
初探 Data API Builder:在幾分鐘內將資料庫轉換成 REST 和 GraphQL 不再是夢想Alan Tsai
 
链家网大数据平台枢纽——工具链,吕毅
链家网大数据平台枢纽——工具链,吕毅链家网大数据平台枢纽——工具链,吕毅
链家网大数据平台枢纽——工具链,吕毅毅 吕
 
20110821 Web Development on Cloud Platform - PIXNET
20110821 Web Development on Cloud Platform - PIXNET20110821 Web Development on Cloud Platform - PIXNET
20110821 Web Development on Cloud Platform - PIXNETJui-Nan Lin
 
課程1 1:雲端運算初探
課程1 1:雲端運算初探課程1 1:雲端運算初探
課程1 1:雲端運算初探vaemon
 
Ibm dnt-dcos-v9-3
Ibm dnt-dcos-v9-3Ibm dnt-dcos-v9-3
Ibm dnt-dcos-v9-3Guangya Liu
 
Data Engineering in Taiwan: PAST, NOW and FUTURE
Data Engineering in Taiwan: PAST, NOW and FUTUREData Engineering in Taiwan: PAST, NOW and FUTURE
Data Engineering in Taiwan: PAST, NOW and FUTUREJazz Yao-Tsung Wang
 
Se shang hai_04_cloudybi
Se shang hai_04_cloudybiSe shang hai_04_cloudybi
Se shang hai_04_cloudybiTech2IPO
 
(宇宏)Talend工具說明
(宇宏)Talend工具說明(宇宏)Talend工具說明
(宇宏)Talend工具說明Sonny Chen
 
機器學習工具介紹 / 机器学习工具介绍 Demos for Machine Learning Tools
機器學習工具介紹 / 机器学习工具介绍 Demos for Machine Learning Tools機器學習工具介紹 / 机器学习工具介绍 Demos for Machine Learning Tools
機器學習工具介紹 / 机器学习工具介绍 Demos for Machine Learning ToolsWill Kuan 官大鈞
 
Analytics in a Day.pptx
Analytics in a Day.pptxAnalytics in a Day.pptx
Analytics in a Day.pptxLigangJin
 
Big Data, NoSQL, and MongoDB
Big Data, NoSQL, and MongoDBBig Data, NoSQL, and MongoDB
Big Data, NoSQL, and MongoDBMonster Supreme
 
Using google appengine_1027
Using google appengine_1027Using google appengine_1027
Using google appengine_1027Wei Sun
 
如何利用 Amazon EMR 及Athena 打造高成本效益的大數據環境
如何利用 Amazon EMR 及Athena 打造高成本效益的大數據環境如何利用 Amazon EMR 及Athena 打造高成本效益的大數據環境
如何利用 Amazon EMR 及Athena 打造高成本效益的大數據環境Amazon Web Services
 
解读数据虚拟化支持的逻辑数据编织(Data Fabric)
解读数据虚拟化支持的逻辑数据编织(Data Fabric)解读数据虚拟化支持的逻辑数据编织(Data Fabric)
解读数据虚拟化支持的逻辑数据编织(Data Fabric)Denodo
 
Bluemix Node-Red Part I
Bluemix Node-Red Part IBluemix Node-Red Part I
Bluemix Node-Red Part IJoseph Chang
 

Similar to 2020 COSCUP 使用開源資料庫 arangoDB的奇幻之旅 (20)

2020 CloudEege Summit 用Serverless真的方便又省錢?
2020 CloudEege Summit 用Serverless真的方便又省錢?2020 CloudEege Summit 用Serverless真的方便又省錢?
2020 CloudEege Summit 用Serverless真的方便又省錢?
 
Hadoop 與 SQL 的甜蜜連結
Hadoop 與 SQL 的甜蜜連結Hadoop 與 SQL 的甜蜜連結
Hadoop 與 SQL 的甜蜜連結
 
選擇正確的Solution 來建置現代化的雲端資料倉儲
選擇正確的Solution 來建置現代化的雲端資料倉儲選擇正確的Solution 來建置現代化的雲端資料倉儲
選擇正確的Solution 來建置現代化的雲端資料倉儲
 
AWS re:Invent Recap 2016 Taiwan part 1
AWS re:Invent Recap 2016 Taiwan part 1AWS re:Invent Recap 2016 Taiwan part 1
AWS re:Invent Recap 2016 Taiwan part 1
 
初探 Data API Builder:在幾分鐘內將資料庫轉換成 REST 和 GraphQL 不再是夢想
初探 Data API Builder:在幾分鐘內將資料庫轉換成 REST 和 GraphQL 不再是夢想初探 Data API Builder:在幾分鐘內將資料庫轉換成 REST 和 GraphQL 不再是夢想
初探 Data API Builder:在幾分鐘內將資料庫轉換成 REST 和 GraphQL 不再是夢想
 
链家网大数据平台枢纽——工具链,吕毅
链家网大数据平台枢纽——工具链,吕毅链家网大数据平台枢纽——工具链,吕毅
链家网大数据平台枢纽——工具链,吕毅
 
20110821 Web Development on Cloud Platform - PIXNET
20110821 Web Development on Cloud Platform - PIXNET20110821 Web Development on Cloud Platform - PIXNET
20110821 Web Development on Cloud Platform - PIXNET
 
課程1 1:雲端運算初探
課程1 1:雲端運算初探課程1 1:雲端運算初探
課程1 1:雲端運算初探
 
Ibm dnt-dcos-v9-3
Ibm dnt-dcos-v9-3Ibm dnt-dcos-v9-3
Ibm dnt-dcos-v9-3
 
Data Engineering in Taiwan: PAST, NOW and FUTURE
Data Engineering in Taiwan: PAST, NOW and FUTUREData Engineering in Taiwan: PAST, NOW and FUTURE
Data Engineering in Taiwan: PAST, NOW and FUTURE
 
Se shang hai_04_cloudybi
Se shang hai_04_cloudybiSe shang hai_04_cloudybi
Se shang hai_04_cloudybi
 
(宇宏)Talend工具說明
(宇宏)Talend工具說明(宇宏)Talend工具說明
(宇宏)Talend工具說明
 
機器學習工具介紹 / 机器学习工具介绍 Demos for Machine Learning Tools
機器學習工具介紹 / 机器学习工具介绍 Demos for Machine Learning Tools機器學習工具介紹 / 机器学习工具介绍 Demos for Machine Learning Tools
機器學習工具介紹 / 机器学习工具介绍 Demos for Machine Learning Tools
 
Analytics in a Day.pptx
Analytics in a Day.pptxAnalytics in a Day.pptx
Analytics in a Day.pptx
 
Big Data, NoSQL, and MongoDB
Big Data, NoSQL, and MongoDBBig Data, NoSQL, and MongoDB
Big Data, NoSQL, and MongoDB
 
Using google appengine_1027
Using google appengine_1027Using google appengine_1027
Using google appengine_1027
 
Hadoop 介紹 20141024
Hadoop 介紹 20141024Hadoop 介紹 20141024
Hadoop 介紹 20141024
 
如何利用 Amazon EMR 及Athena 打造高成本效益的大數據環境
如何利用 Amazon EMR 及Athena 打造高成本效益的大數據環境如何利用 Amazon EMR 及Athena 打造高成本效益的大數據環境
如何利用 Amazon EMR 及Athena 打造高成本效益的大數據環境
 
解读数据虚拟化支持的逻辑数据编织(Data Fabric)
解读数据虚拟化支持的逻辑数据编织(Data Fabric)解读数据虚拟化支持的逻辑数据编织(Data Fabric)
解读数据虚拟化支持的逻辑数据编织(Data Fabric)
 
Bluemix Node-Red Part I
Bluemix Node-Red Part IBluemix Node-Red Part I
Bluemix Node-Red Part I
 

2020 COSCUP 使用開源資料庫 arangoDB的奇幻之旅