SlideShare a Scribd company logo
musiclopedia
♯ ♮ ♭ ♬ ♫ ♪ ♩
♩ ♪ ♫ ♬ ♭ ♮ ♯
discover the world of music
motivation
demo
(artists,dates)(1.4 TB, 240 million records)
pipeline
data sources
MusicBrainz
(artists,dates)
pipeline
Data Store & FrontEndStorage & Batch processing
data sources
MusicBrainz
(1.4 TB, 240 million records)
(artists,dates)
clusters
data sources
MusicBrainz
(1.4 TB, 240 million records)
HDFS datanode
Spark executor
HDFS datanode
Spark executor
HDFS datanode
Spark executor
HDFS namenode
Spark driver
Flask
server
OrientDB
master
OrientDB
master
4 x m4.large (8GB RAM ea. & 6TB SSD total) 3 x m4.large (32 GB SSD total)
data flow
content
header
WARC/1.0
WARC-Type: conversion
WARC-Target-URI: http://www.biography.com/people/ella-fitzgerald-9296210
WARC-Date: 2014-08-02T09:52:13Z
WARC-Record-ID:
WARC-Refers-To:
WARC-Block-Digest: sha1:JROHLCS5SKMBR6XY46WXREW7RXM64EJC
Content-Type: text/plain
Content-Length: 6724
Ella Fitzgerald, known as the "First Lady of Song" and "Lady Ella," was
an American jazz and song vocalist who interpreted much of the Great
American Songbook...
data flow
www.biography.com/people/ella-fitzgerald-9296210, Ella Fitzgerald
www.oldies.com/product-view/47037M.html, Louis Armstrong
bojack.org/2007/06/knock_a_few_bucks_off.html, John Coltrane
WARC/1.0
WARC-Type: conversion
WARC-Target-URI: http://www.biography.com/people/ella-fitzgerald-9296210
WARC-Date: 2014-08-02T09:52:13Z
WARC-Record-ID:
WARC-Refers-To:
WARC-Block-Digest: sha1:JROHLCS5SKMBR6XY46WXREW7RXM64EJC
Content-Type: text/plain
Content-Length: 6724
Ella Fitzgerald, known as the "First Lady of Song" and "Lady Ella," was
an American jazz and song vocalist who interpreted much of the Great
American Songbook...
data flow
challenges
- How to find the bands:
Air, The Clash, Chicago?
~1,4 TB, 274M websites,
1000 artists
- Norah Jones vs Miles
Davis?
challenges
- How to find the bands:
Air, The Clash, Chicago?
~1,4 TB, 274M websites,
1000 artists
- Norah Jones vs Miles
Davis?
challenges
- How to find the bands:
Air, The Clash, Chicago?
~1,4 TB, 274M websites,
1000 artists
- Norah Jones vs Miles
Davis?
about me
B.Sc. EE,
Universidad de
Chile
M.Mus. Music
Technology,
NYU
Artist catalog:
-MusicBrainz databaste (~1,000,000 entries)
→Jazz subset (1,000 entries)
Artist relationship metric:
-CommonCrawl July 2015 log (~145 TB)
→ Uncompressed '.wet' files (~1.5 TB)
data specs
John
Coltrane
W1
W10
W6 W5
Norah
Jones
W2
W3
W4
Miles
Davis
W9
W8
W7
W12
W13
W5
Miles John Norah Total
Miles 5 2 2 9
John 2 4 1 7
Norah 2 1 9 12
model
Miles John Norah Total
Miles 5 2 2 9
John 2 4 1 7
Norah 2 1 9 12
model
Avgerage links between any two artists “X” = (2+2+1)/3 = 1.667
Avgerage links for a single artist “Y”= (9+7+12)/3 = 9.333
=> Average percentage “Z” = X/Y = 17.8 %
bool areConnected(artist A, artist B){
aCountsInB = countLinks(A,B) / countLinks (B)
bCountsInA = countLinks(A,B) / countLinks (A)
if mean(aCountsInB, bCountsInA) > C *Z
return true
return false
}

More Related Content

What's hot

Rdf Overview Presentation
Rdf Overview PresentationRdf Overview Presentation
Rdf Overview Presentation
Ken Varnum
 
Rdf
RdfRdf
Getting triples from records: the role of ISBD
Getting triples from records: the role of ISBDGetting triples from records: the role of ISBD
Publishing and Using Linked Open Data - Day 2
Publishing and Using Linked Open Data - Day 2Publishing and Using Linked Open Data - Day 2
Publishing and Using Linked Open Data - Day 2
Richard Urban
 
Memento 101
Memento 101Memento 101
Vocabulary Development for Local Use: A DIY Introduction
Vocabulary Development for Local Use: A DIY IntroductionVocabulary Development for Local Use: A DIY Introduction
Vocabulary Development for Local Use: A DIY Introduction
Diane Hillmann
 
Another RDF Encoding Form
Another RDF Encoding FormAnother RDF Encoding Form
Another RDF Encoding Form
Jakob .
 
Wikidata for libraries and archives
Wikidata for libraries and archivesWikidata for libraries and archives
Wikidata for libraries and archives
_Emw
 
Up and running with Wikidata
Up and running with WikidataUp and running with Wikidata
Up and running with Wikidata
_Emw
 
An Ambitious Wikidata Tutorial
An Ambitious Wikidata TutorialAn Ambitious Wikidata Tutorial
An Ambitious Wikidata Tutorial
_Emw
 
#sod14 - ok, è un endpoint SPARQL non facciamoci prendere dal panico
#sod14 - ok, è un endpoint SPARQL non facciamoci prendere dal panico#sod14 - ok, è un endpoint SPARQL non facciamoci prendere dal panico
#sod14 - ok, è un endpoint SPARQL non facciamoci prendere dal panico
Diego Valerio Camarda
 
Intro to Linked, Dutch Ships and Sailors and SPARQL handson
Intro to Linked, Dutch Ships and Sailors and SPARQL handson Intro to Linked, Dutch Ships and Sailors and SPARQL handson
Intro to Linked, Dutch Ships and Sailors and SPARQL handson
Victor de Boer
 
RDA: Alive and Well and Still Speaking MARC
RDA: Alive and Well and Still Speaking MARCRDA: Alive and Well and Still Speaking MARC
RDA: Alive and Well and Still Speaking MARC
Diane Hillmann
 
HiBISCuS: Hypergraph-Based Source Selection for SPARQL Endpoint Federation
HiBISCuS: Hypergraph-Based Source Selection for SPARQL Endpoint FederationHiBISCuS: Hypergraph-Based Source Selection for SPARQL Endpoint Federation
HiBISCuS: Hypergraph-Based Source Selection for SPARQL Endpoint Federation
Muhammad Saleem
 
18 ° Nexa Lunch Seminar - Lo stato dell'arte dei Linked Open Data italiani
18 ° Nexa Lunch Seminar - Lo stato dell'arte dei Linked Open Data italiani18 ° Nexa Lunch Seminar - Lo stato dell'arte dei Linked Open Data italiani
18 ° Nexa Lunch Seminar - Lo stato dell'arte dei Linked Open Data italiani
Diego Valerio Camarda
 
Poster - Completeness Statements about RDF Data Sources and Their Use for Qu...
Poster - Completeness Statements about RDF Data Sources and Their Use for Qu...Poster - Completeness Statements about RDF Data Sources and Their Use for Qu...
Poster - Completeness Statements about RDF Data Sources and Their Use for Qu...
Fariz Darari
 
DLF 2015 Presentation, "RDF in the Real World."
DLF 2015 Presentation, "RDF in the Real World." DLF 2015 Presentation, "RDF in the Real World."
DLF 2015 Presentation, "RDF in the Real World."
Avalon Media System
 
NdFluents: An Ontology for Annotated Statements with Inference Preservation
NdFluents: An Ontology for Annotated Statements with Inference PreservationNdFluents: An Ontology for Annotated Statements with Inference Preservation
NdFluents: An Ontology for Annotated Statements with Inference Preservation
José M. Giménez-García
 
Fedora Migration Considerations
Fedora Migration ConsiderationsFedora Migration Considerations
Fedora Migration Considerations
Avalon Media System
 
Federated SPARQL query processing over the Web of Data
Federated SPARQL query processing over the Web of DataFederated SPARQL query processing over the Web of Data
Federated SPARQL query processing over the Web of Data
Muhammad Saleem
 

What's hot (20)

Rdf Overview Presentation
Rdf Overview PresentationRdf Overview Presentation
Rdf Overview Presentation
 
Rdf
RdfRdf
Rdf
 
Getting triples from records: the role of ISBD
Getting triples from records: the role of ISBDGetting triples from records: the role of ISBD
Getting triples from records: the role of ISBD
 
Publishing and Using Linked Open Data - Day 2
Publishing and Using Linked Open Data - Day 2Publishing and Using Linked Open Data - Day 2
Publishing and Using Linked Open Data - Day 2
 
Memento 101
Memento 101Memento 101
Memento 101
 
Vocabulary Development for Local Use: A DIY Introduction
Vocabulary Development for Local Use: A DIY IntroductionVocabulary Development for Local Use: A DIY Introduction
Vocabulary Development for Local Use: A DIY Introduction
 
Another RDF Encoding Form
Another RDF Encoding FormAnother RDF Encoding Form
Another RDF Encoding Form
 
Wikidata for libraries and archives
Wikidata for libraries and archivesWikidata for libraries and archives
Wikidata for libraries and archives
 
Up and running with Wikidata
Up and running with WikidataUp and running with Wikidata
Up and running with Wikidata
 
An Ambitious Wikidata Tutorial
An Ambitious Wikidata TutorialAn Ambitious Wikidata Tutorial
An Ambitious Wikidata Tutorial
 
#sod14 - ok, è un endpoint SPARQL non facciamoci prendere dal panico
#sod14 - ok, è un endpoint SPARQL non facciamoci prendere dal panico#sod14 - ok, è un endpoint SPARQL non facciamoci prendere dal panico
#sod14 - ok, è un endpoint SPARQL non facciamoci prendere dal panico
 
Intro to Linked, Dutch Ships and Sailors and SPARQL handson
Intro to Linked, Dutch Ships and Sailors and SPARQL handson Intro to Linked, Dutch Ships and Sailors and SPARQL handson
Intro to Linked, Dutch Ships and Sailors and SPARQL handson
 
RDA: Alive and Well and Still Speaking MARC
RDA: Alive and Well and Still Speaking MARCRDA: Alive and Well and Still Speaking MARC
RDA: Alive and Well and Still Speaking MARC
 
HiBISCuS: Hypergraph-Based Source Selection for SPARQL Endpoint Federation
HiBISCuS: Hypergraph-Based Source Selection for SPARQL Endpoint FederationHiBISCuS: Hypergraph-Based Source Selection for SPARQL Endpoint Federation
HiBISCuS: Hypergraph-Based Source Selection for SPARQL Endpoint Federation
 
18 ° Nexa Lunch Seminar - Lo stato dell'arte dei Linked Open Data italiani
18 ° Nexa Lunch Seminar - Lo stato dell'arte dei Linked Open Data italiani18 ° Nexa Lunch Seminar - Lo stato dell'arte dei Linked Open Data italiani
18 ° Nexa Lunch Seminar - Lo stato dell'arte dei Linked Open Data italiani
 
Poster - Completeness Statements about RDF Data Sources and Their Use for Qu...
Poster - Completeness Statements about RDF Data Sources and Their Use for Qu...Poster - Completeness Statements about RDF Data Sources and Their Use for Qu...
Poster - Completeness Statements about RDF Data Sources and Their Use for Qu...
 
DLF 2015 Presentation, "RDF in the Real World."
DLF 2015 Presentation, "RDF in the Real World." DLF 2015 Presentation, "RDF in the Real World."
DLF 2015 Presentation, "RDF in the Real World."
 
NdFluents: An Ontology for Annotated Statements with Inference Preservation
NdFluents: An Ontology for Annotated Statements with Inference PreservationNdFluents: An Ontology for Annotated Statements with Inference Preservation
NdFluents: An Ontology for Annotated Statements with Inference Preservation
 
Fedora Migration Considerations
Fedora Migration ConsiderationsFedora Migration Considerations
Fedora Migration Considerations
 
Federated SPARQL query processing over the Web of Data
Federated SPARQL query processing over the Web of DataFederated SPARQL query processing over the Web of Data
Federated SPARQL query processing over the Web of Data
 

Viewers also liked

Take Back Your Energy Dollars
Take Back Your Energy DollarsTake Back Your Energy Dollars
Take Back Your Energy Dollars
Derek Satnik
 
Evento texts
Evento textsEvento texts
Evento texts
Arianna Prosperi
 
Політичні партії у теоретичній спадщині ­Дж. Вашингтона
Політичні партії у теоретичній спадщині ­Дж. ВашингтонаПолітичні партії у теоретичній спадщині ­Дж. Вашингтона
Політичні партії у теоретичній спадщині ­Дж. Вашингтона
Maria Zhyhil
 
Trabajo de redes kjvg
Trabajo de redes kjvgTrabajo de redes kjvg
Trabajo de redes kjvg
jhossua victoria
 
Parque de atracciones mundo extremo
Parque de atracciones mundo extremoParque de atracciones mundo extremo
Parque de atracciones mundo extremo
insideshare
 
ICSTE 2015
ICSTE 2015ICSTE 2015
Sustainability Meets Reality - 70 Acres Net-Zero in London, ON
Sustainability Meets Reality - 70 Acres Net-Zero in London, ONSustainability Meets Reality - 70 Acres Net-Zero in London, ON
Sustainability Meets Reality - 70 Acres Net-Zero in London, ON
Derek Satnik
 
História dos Avivamentos
História dos AvivamentosHistória dos Avivamentos
História dos Avivamentos
Fernando Menezes
 
Parque de atracciones mundo extremo
Parque de atracciones mundo extremoParque de atracciones mundo extremo
Parque de atracciones mundo extremo
insideshare
 
Trends in Green Kitchens and Green Buildings
Trends in Green Kitchens and Green BuildingsTrends in Green Kitchens and Green Buildings
Trends in Green Kitchens and Green Buildings
Derek Satnik
 
Green Building Policy & Programs
Green Building Policy & ProgramsGreen Building Policy & Programs
Green Building Policy & Programs
Derek Satnik
 
Asif Shahab CSCM - Resume
Asif Shahab CSCM - ResumeAsif Shahab CSCM - Resume
Asif Shahab CSCM - Resume
Asif Shahab
 
CaGBC Toronto: Partners In Project Green
CaGBC Toronto: Partners In Project GreenCaGBC Toronto: Partners In Project Green
CaGBC Toronto: Partners In Project Green
Derek Satnik
 
Visita TERMAS Romanas Campo Valdés
Visita TERMAS Romanas Campo ValdésVisita TERMAS Romanas Campo Valdés
Visita TERMAS Romanas Campo Valdés
retruyes
 
Lk tugas kelompok
Lk tugas kelompokLk tugas kelompok
Lk tugas kelompok
WitAnggraini OS
 
Prakarya dan kewirausahaan: Potensi Bahan Baku Limbah
Prakarya dan kewirausahaan: Potensi Bahan Baku LimbahPrakarya dan kewirausahaan: Potensi Bahan Baku Limbah
Prakarya dan kewirausahaan: Potensi Bahan Baku Limbah
Rara Hanifatuzzahra
 
Edad Media
 Edad Media Edad Media
Edad Media
Tumente Empoderando
 
Edad media
Edad mediaEdad media
Edad media
Sergio Martínez
 
Ulangan uts gas buang smk tsm tkr
Ulangan uts gas buang smk tsm tkrUlangan uts gas buang smk tsm tkr
Ulangan uts gas buang smk tsm tkr
Tan Malaka
 
Cuidados no paciente asmático e abordagem do broncoespasmo
Cuidados no paciente asmático e abordagem do broncoespasmoCuidados no paciente asmático e abordagem do broncoespasmo
Cuidados no paciente asmático e abordagem do broncoespasmo
Fabricio Mendonca
 

Viewers also liked (20)

Take Back Your Energy Dollars
Take Back Your Energy DollarsTake Back Your Energy Dollars
Take Back Your Energy Dollars
 
Evento texts
Evento textsEvento texts
Evento texts
 
Політичні партії у теоретичній спадщині ­Дж. Вашингтона
Політичні партії у теоретичній спадщині ­Дж. ВашингтонаПолітичні партії у теоретичній спадщині ­Дж. Вашингтона
Політичні партії у теоретичній спадщині ­Дж. Вашингтона
 
Trabajo de redes kjvg
Trabajo de redes kjvgTrabajo de redes kjvg
Trabajo de redes kjvg
 
Parque de atracciones mundo extremo
Parque de atracciones mundo extremoParque de atracciones mundo extremo
Parque de atracciones mundo extremo
 
ICSTE 2015
ICSTE 2015ICSTE 2015
ICSTE 2015
 
Sustainability Meets Reality - 70 Acres Net-Zero in London, ON
Sustainability Meets Reality - 70 Acres Net-Zero in London, ONSustainability Meets Reality - 70 Acres Net-Zero in London, ON
Sustainability Meets Reality - 70 Acres Net-Zero in London, ON
 
História dos Avivamentos
História dos AvivamentosHistória dos Avivamentos
História dos Avivamentos
 
Parque de atracciones mundo extremo
Parque de atracciones mundo extremoParque de atracciones mundo extremo
Parque de atracciones mundo extremo
 
Trends in Green Kitchens and Green Buildings
Trends in Green Kitchens and Green BuildingsTrends in Green Kitchens and Green Buildings
Trends in Green Kitchens and Green Buildings
 
Green Building Policy & Programs
Green Building Policy & ProgramsGreen Building Policy & Programs
Green Building Policy & Programs
 
Asif Shahab CSCM - Resume
Asif Shahab CSCM - ResumeAsif Shahab CSCM - Resume
Asif Shahab CSCM - Resume
 
CaGBC Toronto: Partners In Project Green
CaGBC Toronto: Partners In Project GreenCaGBC Toronto: Partners In Project Green
CaGBC Toronto: Partners In Project Green
 
Visita TERMAS Romanas Campo Valdés
Visita TERMAS Romanas Campo ValdésVisita TERMAS Romanas Campo Valdés
Visita TERMAS Romanas Campo Valdés
 
Lk tugas kelompok
Lk tugas kelompokLk tugas kelompok
Lk tugas kelompok
 
Prakarya dan kewirausahaan: Potensi Bahan Baku Limbah
Prakarya dan kewirausahaan: Potensi Bahan Baku LimbahPrakarya dan kewirausahaan: Potensi Bahan Baku Limbah
Prakarya dan kewirausahaan: Potensi Bahan Baku Limbah
 
Edad Media
 Edad Media Edad Media
Edad Media
 
Edad media
Edad mediaEdad media
Edad media
 
Ulangan uts gas buang smk tsm tkr
Ulangan uts gas buang smk tsm tkrUlangan uts gas buang smk tsm tkr
Ulangan uts gas buang smk tsm tkr
 
Cuidados no paciente asmático e abordagem do broncoespasmo
Cuidados no paciente asmático e abordagem do broncoespasmoCuidados no paciente asmático e abordagem do broncoespasmo
Cuidados no paciente asmático e abordagem do broncoespasmo
 

Recently uploaded

一比一原版卡尔加里大学毕业证(uc毕业证)如何办理
一比一原版卡尔加里大学毕业证(uc毕业证)如何办理一比一原版卡尔加里大学毕业证(uc毕业证)如何办理
一比一原版卡尔加里大学毕业证(uc毕业证)如何办理
oaxefes
 
一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理
一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理
一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理
nyfuhyz
 
一比一原版(Unimelb毕业证书)墨尔本大学毕业证如何办理
一比一原版(Unimelb毕业证书)墨尔本大学毕业证如何办理一比一原版(Unimelb毕业证书)墨尔本大学毕业证如何办理
一比一原版(Unimelb毕业证书)墨尔本大学毕业证如何办理
xclpvhuk
 
一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理
一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理
一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理
nuttdpt
 
End-to-end pipeline agility - Berlin Buzzwords 2024
End-to-end pipeline agility - Berlin Buzzwords 2024End-to-end pipeline agility - Berlin Buzzwords 2024
End-to-end pipeline agility - Berlin Buzzwords 2024
Lars Albertsson
 
一比一原版美国帕森斯设计学院毕业证(parsons毕业证书)如何办理
一比一原版美国帕森斯设计学院毕业证(parsons毕业证书)如何办理一比一原版美国帕森斯设计学院毕业证(parsons毕业证书)如何办理
一比一原版美国帕森斯设计学院毕业证(parsons毕业证书)如何办理
asyed10
 
一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理
bmucuha
 
Jio cinema Retention & Engagement Strategy.pdf
Jio cinema Retention & Engagement Strategy.pdfJio cinema Retention & Engagement Strategy.pdf
Jio cinema Retention & Engagement Strategy.pdf
inaya7568
 
一比一原版(曼大毕业证书)曼尼托巴大学毕业证如何办理
一比一原版(曼大毕业证书)曼尼托巴大学毕业证如何办理一比一原版(曼大毕业证书)曼尼托巴大学毕业证如何办理
一比一原版(曼大毕业证书)曼尼托巴大学毕业证如何办理
ytypuem
 
How To Control IO Usage using Resource Manager
How To Control IO Usage using Resource ManagerHow To Control IO Usage using Resource Manager
How To Control IO Usage using Resource Manager
Alireza Kamrani
 
A presentation that explain the Power BI Licensing
A presentation that explain the Power BI LicensingA presentation that explain the Power BI Licensing
A presentation that explain the Power BI Licensing
AlessioFois2
 
DSSML24_tspann_CodelessGenerativeAIPipelines
DSSML24_tspann_CodelessGenerativeAIPipelinesDSSML24_tspann_CodelessGenerativeAIPipelines
DSSML24_tspann_CodelessGenerativeAIPipelines
Timothy Spann
 
Orchestrating the Future: Navigating Today's Data Workflow Challenges with Ai...
Orchestrating the Future: Navigating Today's Data Workflow Challenges with Ai...Orchestrating the Future: Navigating Today's Data Workflow Challenges with Ai...
Orchestrating the Future: Navigating Today's Data Workflow Challenges with Ai...
Kaxil Naik
 
一比一原版英国赫特福德大学毕业证(hertfordshire毕业证书)如何办理
一比一原版英国赫特福德大学毕业证(hertfordshire毕业证书)如何办理一比一原版英国赫特福德大学毕业证(hertfordshire毕业证书)如何办理
一比一原版英国赫特福德大学毕业证(hertfordshire毕业证书)如何办理
nyvan3
 
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
Timothy Spann
 
一比一原版爱尔兰都柏林大学毕业证(本硕)ucd学位证书如何办理
一比一原版爱尔兰都柏林大学毕业证(本硕)ucd学位证书如何办理一比一原版爱尔兰都柏林大学毕业证(本硕)ucd学位证书如何办理
一比一原版爱尔兰都柏林大学毕业证(本硕)ucd学位证书如何办理
hqfek
 
一比一原版南十字星大学毕业证(SCU毕业证书)学历如何办理
一比一原版南十字星大学毕业证(SCU毕业证书)学历如何办理一比一原版南十字星大学毕业证(SCU毕业证书)学历如何办理
一比一原版南十字星大学毕业证(SCU毕业证书)学历如何办理
slg6lamcq
 
Palo Alto Cortex XDR presentation .......
Palo Alto Cortex XDR presentation .......Palo Alto Cortex XDR presentation .......
Palo Alto Cortex XDR presentation .......
Sachin Paul
 
一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理
一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理
一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理
hyfjgavov
 
原版一比一爱尔兰都柏林大学毕业证(UCD毕业证书)如何办理
原版一比一爱尔兰都柏林大学毕业证(UCD毕业证书)如何办理 原版一比一爱尔兰都柏林大学毕业证(UCD毕业证书)如何办理
原版一比一爱尔兰都柏林大学毕业证(UCD毕业证书)如何办理
tzu5xla
 

Recently uploaded (20)

一比一原版卡尔加里大学毕业证(uc毕业证)如何办理
一比一原版卡尔加里大学毕业证(uc毕业证)如何办理一比一原版卡尔加里大学毕业证(uc毕业证)如何办理
一比一原版卡尔加里大学毕业证(uc毕业证)如何办理
 
一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理
一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理
一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理
 
一比一原版(Unimelb毕业证书)墨尔本大学毕业证如何办理
一比一原版(Unimelb毕业证书)墨尔本大学毕业证如何办理一比一原版(Unimelb毕业证书)墨尔本大学毕业证如何办理
一比一原版(Unimelb毕业证书)墨尔本大学毕业证如何办理
 
一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理
一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理
一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理
 
End-to-end pipeline agility - Berlin Buzzwords 2024
End-to-end pipeline agility - Berlin Buzzwords 2024End-to-end pipeline agility - Berlin Buzzwords 2024
End-to-end pipeline agility - Berlin Buzzwords 2024
 
一比一原版美国帕森斯设计学院毕业证(parsons毕业证书)如何办理
一比一原版美国帕森斯设计学院毕业证(parsons毕业证书)如何办理一比一原版美国帕森斯设计学院毕业证(parsons毕业证书)如何办理
一比一原版美国帕森斯设计学院毕业证(parsons毕业证书)如何办理
 
一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理
 
Jio cinema Retention & Engagement Strategy.pdf
Jio cinema Retention & Engagement Strategy.pdfJio cinema Retention & Engagement Strategy.pdf
Jio cinema Retention & Engagement Strategy.pdf
 
一比一原版(曼大毕业证书)曼尼托巴大学毕业证如何办理
一比一原版(曼大毕业证书)曼尼托巴大学毕业证如何办理一比一原版(曼大毕业证书)曼尼托巴大学毕业证如何办理
一比一原版(曼大毕业证书)曼尼托巴大学毕业证如何办理
 
How To Control IO Usage using Resource Manager
How To Control IO Usage using Resource ManagerHow To Control IO Usage using Resource Manager
How To Control IO Usage using Resource Manager
 
A presentation that explain the Power BI Licensing
A presentation that explain the Power BI LicensingA presentation that explain the Power BI Licensing
A presentation that explain the Power BI Licensing
 
DSSML24_tspann_CodelessGenerativeAIPipelines
DSSML24_tspann_CodelessGenerativeAIPipelinesDSSML24_tspann_CodelessGenerativeAIPipelines
DSSML24_tspann_CodelessGenerativeAIPipelines
 
Orchestrating the Future: Navigating Today's Data Workflow Challenges with Ai...
Orchestrating the Future: Navigating Today's Data Workflow Challenges with Ai...Orchestrating the Future: Navigating Today's Data Workflow Challenges with Ai...
Orchestrating the Future: Navigating Today's Data Workflow Challenges with Ai...
 
一比一原版英国赫特福德大学毕业证(hertfordshire毕业证书)如何办理
一比一原版英国赫特福德大学毕业证(hertfordshire毕业证书)如何办理一比一原版英国赫特福德大学毕业证(hertfordshire毕业证书)如何办理
一比一原版英国赫特福德大学毕业证(hertfordshire毕业证书)如何办理
 
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
 
一比一原版爱尔兰都柏林大学毕业证(本硕)ucd学位证书如何办理
一比一原版爱尔兰都柏林大学毕业证(本硕)ucd学位证书如何办理一比一原版爱尔兰都柏林大学毕业证(本硕)ucd学位证书如何办理
一比一原版爱尔兰都柏林大学毕业证(本硕)ucd学位证书如何办理
 
一比一原版南十字星大学毕业证(SCU毕业证书)学历如何办理
一比一原版南十字星大学毕业证(SCU毕业证书)学历如何办理一比一原版南十字星大学毕业证(SCU毕业证书)学历如何办理
一比一原版南十字星大学毕业证(SCU毕业证书)学历如何办理
 
Palo Alto Cortex XDR presentation .......
Palo Alto Cortex XDR presentation .......Palo Alto Cortex XDR presentation .......
Palo Alto Cortex XDR presentation .......
 
一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理
一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理
一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理
 
原版一比一爱尔兰都柏林大学毕业证(UCD毕业证书)如何办理
原版一比一爱尔兰都柏林大学毕业证(UCD毕业证书)如何办理 原版一比一爱尔兰都柏林大学毕业证(UCD毕业证书)如何办理
原版一比一爱尔兰都柏林大学毕业证(UCD毕业证书)如何办理
 

musiclopedia presentation

  • 1. musiclopedia ♯ ♮ ♭ ♬ ♫ ♪ ♩ ♩ ♪ ♫ ♬ ♭ ♮ ♯ discover the world of music
  • 4. (artists,dates)(1.4 TB, 240 million records) pipeline data sources MusicBrainz
  • 5. (artists,dates) pipeline Data Store & FrontEndStorage & Batch processing data sources MusicBrainz (1.4 TB, 240 million records)
  • 6. (artists,dates) clusters data sources MusicBrainz (1.4 TB, 240 million records) HDFS datanode Spark executor HDFS datanode Spark executor HDFS datanode Spark executor HDFS namenode Spark driver Flask server OrientDB master OrientDB master 4 x m4.large (8GB RAM ea. & 6TB SSD total) 3 x m4.large (32 GB SSD total)
  • 7. data flow content header WARC/1.0 WARC-Type: conversion WARC-Target-URI: http://www.biography.com/people/ella-fitzgerald-9296210 WARC-Date: 2014-08-02T09:52:13Z WARC-Record-ID: WARC-Refers-To: WARC-Block-Digest: sha1:JROHLCS5SKMBR6XY46WXREW7RXM64EJC Content-Type: text/plain Content-Length: 6724 Ella Fitzgerald, known as the "First Lady of Song" and "Lady Ella," was an American jazz and song vocalist who interpreted much of the Great American Songbook...
  • 8. data flow www.biography.com/people/ella-fitzgerald-9296210, Ella Fitzgerald www.oldies.com/product-view/47037M.html, Louis Armstrong bojack.org/2007/06/knock_a_few_bucks_off.html, John Coltrane WARC/1.0 WARC-Type: conversion WARC-Target-URI: http://www.biography.com/people/ella-fitzgerald-9296210 WARC-Date: 2014-08-02T09:52:13Z WARC-Record-ID: WARC-Refers-To: WARC-Block-Digest: sha1:JROHLCS5SKMBR6XY46WXREW7RXM64EJC Content-Type: text/plain Content-Length: 6724 Ella Fitzgerald, known as the "First Lady of Song" and "Lady Ella," was an American jazz and song vocalist who interpreted much of the Great American Songbook...
  • 10. challenges - How to find the bands: Air, The Clash, Chicago? ~1,4 TB, 274M websites, 1000 artists - Norah Jones vs Miles Davis?
  • 11. challenges - How to find the bands: Air, The Clash, Chicago? ~1,4 TB, 274M websites, 1000 artists - Norah Jones vs Miles Davis?
  • 12. challenges - How to find the bands: Air, The Clash, Chicago? ~1,4 TB, 274M websites, 1000 artists - Norah Jones vs Miles Davis?
  • 13. about me B.Sc. EE, Universidad de Chile M.Mus. Music Technology, NYU
  • 14.
  • 15. Artist catalog: -MusicBrainz databaste (~1,000,000 entries) →Jazz subset (1,000 entries) Artist relationship metric: -CommonCrawl July 2015 log (~145 TB) → Uncompressed '.wet' files (~1.5 TB) data specs
  • 16. John Coltrane W1 W10 W6 W5 Norah Jones W2 W3 W4 Miles Davis W9 W8 W7 W12 W13 W5 Miles John Norah Total Miles 5 2 2 9 John 2 4 1 7 Norah 2 1 9 12 model
  • 17. Miles John Norah Total Miles 5 2 2 9 John 2 4 1 7 Norah 2 1 9 12 model Avgerage links between any two artists “X” = (2+2+1)/3 = 1.667 Avgerage links for a single artist “Y”= (9+7+12)/3 = 9.333 => Average percentage “Z” = X/Y = 17.8 % bool areConnected(artist A, artist B){ aCountsInB = countLinks(A,B) / countLinks (B) bCountsInA = countLinks(A,B) / countLinks (A) if mean(aCountsInB, bCountsInA) > C *Z return true return false }