Fazendo mágica com ElasticSearch
Upcoming SlideShare
Loading in...5
×
 

Fazendo mágica com ElasticSearch

on

  • 776 views

Quando uma aplicação começa a ficar grande e complexa, fazer buscas nos seus models torna-se uma tarefa complicada. Efetuar as buscas diretamente no banco de dados é um processo lento, ineficiente ...

Quando uma aplicação começa a ficar grande e complexa, fazer buscas nos seus models torna-se uma tarefa complicada. Efetuar as buscas diretamente no banco de dados é um processo lento, ineficiente e que permite pouca ou nenhuma maleabilidade sobre a forma com que a busca é feita. Surge então o ElasticSearch, uma engine de busca utilizada por empresas como Github, Twitter e 4square para indexar e buscar literalmente milhões de documentos em tempo real. Nessa palestra, explicarei quando, como e porque utilizar o ElasticSearch para facilmente indexar e efetuar buscas complexas nos seus models.

Statistics

Views

Total Views
776
Views on SlideShare
741
Embed Views
35

Actions

Likes
6
Downloads
11
Comments
0

1 Embed 35

https://twitter.com 35

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Fazendo mágica com ElasticSearch Fazendo mágica com ElasticSearch Presentation Transcript

  • Fazendo mágica com ElasticSearch PEDROFRANCESCHI @pedroh96 pedro@pagar.me github.com/pedrofranceschi
  • Outubro/2010
  • Filters Full text search Sort Highlight Facets Pagination
  • Você vai precisar buscar dados.
  • Você vai precisar entender dados.
  • (My)SQL não é a solução. (… nem NoSQL)
  • O que é o ElasticSearch?
  • ElasticSearch • “Open Source Distributed Real Time Search & Analytics” • API RESTful para indexar/buscar JSONs (“NoSQL”) • NÃO é um banco de dados • Apache Lucene • Just works (and scales) • Full text search, aggregations, scripting, etc, etc, etc.
  • Nomes? MySQL ElasticSearch Database Index Table Type Row Document Column Field Schema Mapping Partition Shard
  • Como usar o ElasticSearch?
  • $ curl -XPUT 'http://localhost:9200/twitter/tweet/1' -d '{! "user" : “pedroh96",! "post_date" : "2009-11-15T14:12:12",! "message" : "trying out Elasticsearch"! }' Endpoint Index Type Document ID Document {! "_index" : "twitter",! "_type" : "tweet",! "_id" : "1",! "_version" : 1,! "created" : true! } PUT data
  • Endpoint Index Type $ curl -XGET 'http://localhost:9200/twitter/tweet/1' Document ID {! "_id": "1",! "_index": "twitter",! "_source": {! "message": "trying out Elasticsearch",! "post_date": "2009-11-15T14:12:12",! "user": "pedroh96"! },! "_type": "tweet",! "_version": 1,! "found": true! } Document GET data
  • GET data Endpoint Index $ curl -XGET 'http://localhost:9200/twitter/_search'! -d ‘{ query: . . . }! ! ! Query de busca ! ! ! ! ! ! ! Operador de busca
  • ActiveRecords class Tweet < ActiveRecord::Base! end
  • ActiveRecords require 'elasticsearch/model'! ! class Tweet < ActiveRecord::Base! include Elasticsearch::Model! include Elasticsearch::Model::Callbacks! end! !
  • Tweet.import
  • Tweet.search(“pedroh96”)
  • Por que usar o ElasticSearch?
  • DISCLAIMER
  • Post.where(:all, :author => "pedroh96") vs Post.search(query: { match: { author: "pedroh96" }}) Just Another Query Language?
  • 1) Full text search
  • ActiveRecords $ rails g scaffold Post title:string! source:string
  • GET /posts/5 Post.find(5) :-) ActiveRecords
  • ActiveRecords “Amazon to Buy Video Site Twitch for More Than $1B” Post.where(:all, :title => "Amazon to Buy Video Site Twitch for More Than $1B") :-)
  • “amazon” Post.where(["title LIKE ?", "%Amazon%"]) ??? ActiveRecords
  • “amazon source:online.wsj.com” Post.where(["title LIKE ? AND source = ?", "%Amazon%", "online.wsj.com"]) ?????? ActiveRecords
  • “amazon” Post.search("amazon") :-) ElasticSearch
  • ElasticSearch “amazon source:online.wsj.com” search = Post.search("amazon source:online.wsj.com") :-)
  • ElasticSearch “amazon source:online.wsj.com” search = Post.search( query:{ match: { _all: "amazon source:online.wsj.com", } } ) Full-text search
  • ElasticSearch “amazon source:online.wsj.com” search = Post.search( query:{ multi_match: { query: "amazon source:online.wsj.com", fields: ['title^10', 'source'] } } ) Full-text search Title boost
  • ElasticSearch “amazon source:online.wsj.com” search = Post.search( query:{ multi_match: { query: "amazon source:online.wsj.com", fields: ['title^10', 'source'] } }, highlight: { fields: { title: {} } } ) Title highlight Full-text search Title boost
  • ElasticSearch Title highlight > search.results[0].highlight.title => ["Twitch officially acquired by <em>Amazon</em>"]
  • 2) Aggregations (faceting)
  • Geo distance aggregation
  • ActiveRecords $ rails g scaffold Coordinate latitude:decimal longitude:decimal
  • ActiveRecords class Coordinate < ActiveRecord::Base! end
  • ActiveRecords class Coordinate < ActiveRecord::Base! def distance_to(coordinate)! # From http://en.wikipedia.org/wiki/Haversine_formula! rad_per_deg = Math::PI/180 # PI / 180! rkm = 6371 # Earth radius in kilometers! rm = rkm * 1000 # Radius in meters! ! dlon_rad = (coordinate.longitude.to_f - self.longitude.to_f) * rad_per_deg # Delta, converted to rad! dlat_rad = (coordinate.latitude.to_f - self.latitude.to_f) * rad_per_deg! ! lat1_rad = coordinate.latitude.to_f * rad_per_deg! lat2_rad = self.latitude.to_f * rad_per_deg! lon1_rad = coordinate.longitude.to_f * rad_per_deg! lon2_rad = self.longitude.to_f * rad_per_deg! ! a = Math.sin(dlat_rad/2)**2 + Math.cos(lat1_rad) * Math.cos(lat2_rad) * Math.sin(dlon_rad/2)**2! c = 2 * Math::atan2(Math::sqrt(a), Math::sqrt(1-a))! ! rm * c # Delta in meters! end! end > c1 = Coordinate.new(:latitude => -23.5532636, :longitude => -46.6528908) > c2 = Coordinate.new(:latitude => -23.5538488, :longitude => -46.6530035) > c1.distance_to(c2) => 66.07749735875552
  • ActiveRecords origin = Coordinate.new(:latitude => -23.5532636, :longitude => -46.6528908) buckets = [! {! :to => 100,! :coordinates => []! },! {! :from => 100,! :to => 300,! :coordinates => []! },! {! :from => 300,! :coordinates => []! }! ]! Coordinate.all.each do |coordinate|! distance = origin.distance_to(coordinate)! ! buckets.each do |bucket|! if distance < bucket[:to] and distance > (bucket[:from] || 0)! bucket[:coordinates] << coordinate! end! end! end ??????
  • ElasticSearch query = {! aggregations: {! Nome da aggregation rings_around_rubyconf: {! geo_distance: {! Field com localização Coordenadas da origem field: "location",! origin: "-23.5532636, -46.6528908",! ranges: [! { to: 100 },! { from: 100, to: 300 },! { from: 300 }! ]! }! Tipo da aggregation }! }! } Buckets para agregar search = Coordinate.search(query) :-)
  • (Extended) stats aggregation
  • ActiveRecords $ rails g scaffold Grade subject:string grade:decimal
  • ElasticSearch query = {! aggregations: {! Nome da aggregation grades_stats: {! Tipo da aggregation extended_stats: {! field: "grade",! }! }! }! }! ! search = Grade.search(query) Nome do field
  • ElasticSearch > search.response.aggregations.grades_stats! ! => #<Hashie::Mash avg=8.03 count=3 max=10.0 min=4.6 std_deviation=2.43 sum=24.1 sum_of_squares=211.41 variance=5.93>>
  • (Extended) stats aggregation + Scripting
  • ElasticSearch query = {! aggregations: {! grades_stats: {! extended_stats: {! field: "grade",! }! }! }! }
  • ElasticSearch query = {! aggregations: {! Nome da aggregation grades_stats: {! extended_stats: {! field: "grade",! script: "_value < 7.0 ? _value * correction : _value",! params: {! correction: 1.2! }! }! }! }! }! ! search = Grade.search(query) Nome do field JavaScript para calcular novo grade Tipo da aggregation
  • ElasticSearch > search.response.aggregations.grades_stats! ! => #<Hashie::Mash avg=8.34 count=3 max=10.0 min=5.52 std_deviation=2.00 sum=25.02 sum_of_squares=220.72 variance=4.01>>
  • Term aggregation
  • ElasticSearch query = {! aggregations: {! subjects: {! terms: {! Nome da aggregation field: "subject"! }! }! }! }! ! search = Grade.search(query) Nome do field Tipo da aggregation
  • ElasticSearch > search.response.aggregations.subjects! ! => #<Hashie::Mash buckets=[! #<Hashie::Mash doc_count=2 key=“math">,! #<Hashie::Mash doc_count=1 key="grammar">, #<Hashie::Mash doc_count=1 key=“physics">! ]>
  • Combined aggregations (term + stats)
  • ElasticSearch query = {! aggregations: {! subjects: {! terms: {! field: "subject"! }! }! }! }! ! search = Grade.search(query)
  • ElasticSearch query = {! aggregations: {! subjects: {! terms: {! Nome da parent aggregation field: "subject"! },! aggregations: {! grade_stats: {! stats: {! Nome da child aggregation field: "grade"! }! }! }! }! }! }! ! search = Grade.search(query) Field para parent aggregation Field para child aggregation
  • ElasticSearch > search.response.aggregations.subjects! ! #<Hashie::Mash buckets=[! #<Hashie::Mash doc_count=2 grade_stats=#<Hashie::Mash avg=9.0 count=2 max=10.0 min=8.0 sum=18.0> key="math">, #<Hashie::Mash doc_count=1 grade_stats=#<Hashie::Mash avg=4.6 count=1 max=4.6 min=4.6 sum=4.6> key="grammar">, #<Hashie::Mash doc_count=1 grade_stats=#<Hashie::Mash avg=9.5 count=1 max=9.5 min=9.5 sum=9.5> key=“physics">! ]>
  • Top Hits More like this Histogram Scripted metrics Geo bounds Stemmer (sinônimos) IPv4 ranges . . .
  • 3) Scoring
  • ActiveRecords $ rails g scaffold Post title:string! source:string likes:integer
  • “amazon” ElasticSearch search = Post.search( query: { match: { _all: "amazon", } } ) Full-text search search.results.results[0]._score => 0.8174651
  • “amazon” ElasticSearch search = Post.search( query: { custom_score: { query:{ match: { _all: "amazon", } }, script: "_score * doc['likes'].value" } } ) Full-text search Likes influenciam no score search.results.results[0]._score => 31.8811388
  • GET http://localhost:9200/post/_search?explain "_explanation": {! "description": "weight(tweet:honeymoon in 0)! [PerFieldSimilarity], result of:",! "value": 0.076713204,! "details": [! {! "description": "fieldWeight in 0, product of:",! "value": 0.076713204,! "details": [! {! "description": "tf(freq=1.0), with freq of:",! "value": 1,! "details": [! {! "description": "termFreq=1.0",! "value": 1! }! ]! },! {! "description": "idf(docFreq=1, maxDocs=1)",! "value": 0.30685282! },! {! "description": "fieldNorm(doc=0)",! "value": 0.25,! }! ]! }! ]! } Score explicado
  • 4) Indexando responses
  • $ rails g scaffold Post title:string! source:string likes:integer
  • class PostsController < ApplicationController! ! # ...! ! def show! @post = Post.find(params[:id])! ! render json: @post! end! ! # ...! ! end SELECT * FROM Posts WHERE id = params[:id]
  • class PostsController < ApplicationController! ! # ...! ! def show! @post = Post.search(query: { match: { id: params[:id] }})! ! render json: @post! end! ! # ...! ! end GET http://localhost:9200/posts/posts/params[:id]
  • ActiveRecords require 'elasticsearch/model'! ! class Post < ActiveRecord::Base! include Elasticsearch::Model! include Elasticsearch::Model::Callbacks! ! belongs_to :author! ! def as_indexed_json(options={})! self.as_json(! include: { author: { only: [:name, :bio] },! })! end! end Inclui um parent no JSON indexado
  • Expondo o ElasticSearch
  • http://localhost:9200/pagarme/_search https://api.pagar.me/1/search
  • Infraestrutura do Pagar.me ElasticSearch ElasticSearch Router api.pagar.me Servidor da API (Node.js) MySQL (transações e dados relacionais) MySQL (transações e dados relacionais) MongoDB (dados de clientes e não relacionais) Ambiente de testes (sandbox dos clientes) Servidor da API (Node.js) Ambiente de produção
  • Expondo o ElasticSearch • Endpoint do ElasticSearch -> Endpoint acessado pelo cliente… • … mas cuidado: dados precisam ser delimitados a conta do cliente (claro) • Vantagem: acesso às mesmas features do ElasticSearch (aggregations, statistics, scores, etc) • Segurança: desabilitar scripts do ElasticSearch
  • GET /search • Um único endpoint para todos os GETs • Todos os dados indexados e prontos para serem usados (no joins) • Queries complexas construídas no front-side (Angular.js) • Desenvolvimento front-end não dependente do back-end
  • Overall…
  • 1)Há uma ferramenta para cada tarefa. 2)Um martelo é sempre a ferramenta certa. 3)Toda ferramenta também é um martelo.
  • MySQL != NoSQL != ElasticSearch
  • Obrigado! :) PEDROFRANCESCHI @pedroh96 pedro@pagar.me github.com/pedrofranceschi
  • Perguntas? PEDROFRANCESCHI @pedroh96 pedro@pagar.me github.com/pedrofranceschi
  • Fazendo mágica com ElasticSearch PEDROFRANCESCHI @pedroh96 pedro@pagar.me github.com/pedrofranceschi