SlideShare a Scribd company logo
require 'rubygems'
require 'sinatra'

post '/' do
  url = params[:url]
  data = params[:data]
  store(url, data)
  next_url = process(url)
  next_url
end
//   ==UserScript==
//   @name      greasi_scraper
//   @namespace http://libelabo.jp/
//   @include   http://images.google.co.jp/*
//   @require   http://ajax.googleapis.com/ajax/libs/jquery/1.3.1/jquery.min.js
//   ==/UserScript==


function postData(data) {
  var postData = $.param({url: location.href, data: JSON.stringify(data)});
  GM_xmlhttpRequest({
    method: "POST",
    url:     "http://libelabo.jp/greasi/",
    headers: {'Content-type':'application/x-www-form-urlencoded'},
    data:    postData,
    onload: function(xhr){ location.href = xhr.responseText }
  });
}
Introduce of the parallel distributed Crawler with scraping Dynamic HTML
Introduce of the parallel distributed Crawler with scraping Dynamic HTML
Introduce of the parallel distributed Crawler with scraping Dynamic HTML
Introduce of the parallel distributed Crawler with scraping Dynamic HTML
Introduce of the parallel distributed Crawler with scraping Dynamic HTML
Introduce of the parallel distributed Crawler with scraping Dynamic HTML
Introduce of the parallel distributed Crawler with scraping Dynamic HTML

More Related Content

What's hot

AngularJS Services
AngularJS ServicesAngularJS Services
AngularJS Services
Eyal Vardi
 
Code igniter parameter passing techniques
Code igniter parameter passing techniquesCode igniter parameter passing techniques
Code igniter parameter passing techniques
Rakhitha Ratnayake
 
Let's Build A Gutenberg Block | WordCamp Europe 2018
Let's Build A Gutenberg Block | WordCamp Europe 2018Let's Build A Gutenberg Block | WordCamp Europe 2018
Let's Build A Gutenberg Block | WordCamp Europe 2018
Lara Schenck
 
Backbone
BackboneBackbone
Backbone
Glenn De Backer
 
Understanding backbonejs
Understanding backbonejsUnderstanding backbonejs
Understanding backbonejs
Nick Lee
 
Hack tutorial
Hack tutorialHack tutorial
Hack tutorial
Wakana Yoshizawa
 
Flask and Angular: An approach to build robust platforms
Flask and Angular:  An approach to build robust platformsFlask and Angular:  An approach to build robust platforms
Flask and Angular: An approach to build robust platforms
Ayush Sharma
 
[Srijan Wednesday Webinars] Routing in Drupal 8: Decoupling hook_menu
[Srijan Wednesday Webinars] Routing in Drupal 8: Decoupling hook_menu[Srijan Wednesday Webinars] Routing in Drupal 8: Decoupling hook_menu
[Srijan Wednesday Webinars] Routing in Drupal 8: Decoupling hook_menu
Srijan Technologies
 
Clojure Workshop: Web development
Clojure Workshop: Web developmentClojure Workshop: Web development
Clojure Workshop: Web development
Sytac
 
The hitchhiker's guide to the Webpack - Sara Vieira - Codemotion Amsterdam 2017
The hitchhiker's guide to the Webpack - Sara Vieira - Codemotion Amsterdam 2017The hitchhiker's guide to the Webpack - Sara Vieira - Codemotion Amsterdam 2017
The hitchhiker's guide to the Webpack - Sara Vieira - Codemotion Amsterdam 2017
Codemotion
 
Introduction to AngularJS For WordPress Developers
Introduction to AngularJS For WordPress DevelopersIntroduction to AngularJS For WordPress Developers
Introduction to AngularJS For WordPress Developers
Caldera Labs
 
Angular Promises and Advanced Routing
Angular Promises and Advanced RoutingAngular Promises and Advanced Routing
Angular Promises and Advanced Routing
Alexe Bogdan
 
Creating effective ruby gems
Creating effective ruby gemsCreating effective ruby gems
Creating effective ruby gems
Ben Zhang
 
Backbone js in drupal core
Backbone js in drupal coreBackbone js in drupal core
Backbone js in drupal core
Marcin Wosinek
 
Javascript - Beyond-jQuery
Javascript - Beyond-jQueryJavascript - Beyond-jQuery
Ruby on Rails Intro
Ruby on Rails IntroRuby on Rails Intro
Ruby on Rails Intro
zhang tao
 
Introducing AngularJS
Introducing AngularJSIntroducing AngularJS
Introducing AngularJS
Loc Nguyen
 
Express Presentation
Express PresentationExpress Presentation
Express Presentation
aaronheckmann
 

What's hot (20)

AngularJS Services
AngularJS ServicesAngularJS Services
AngularJS Services
 
Code igniter parameter passing techniques
Code igniter parameter passing techniquesCode igniter parameter passing techniques
Code igniter parameter passing techniques
 
Let's Build A Gutenberg Block | WordCamp Europe 2018
Let's Build A Gutenberg Block | WordCamp Europe 2018Let's Build A Gutenberg Block | WordCamp Europe 2018
Let's Build A Gutenberg Block | WordCamp Europe 2018
 
Div id
Div idDiv id
Div id
 
Backbone
BackboneBackbone
Backbone
 
Understanding backbonejs
Understanding backbonejsUnderstanding backbonejs
Understanding backbonejs
 
Hack tutorial
Hack tutorialHack tutorial
Hack tutorial
 
Flask and Angular: An approach to build robust platforms
Flask and Angular:  An approach to build robust platformsFlask and Angular:  An approach to build robust platforms
Flask and Angular: An approach to build robust platforms
 
JQuery introduction
JQuery introductionJQuery introduction
JQuery introduction
 
[Srijan Wednesday Webinars] Routing in Drupal 8: Decoupling hook_menu
[Srijan Wednesday Webinars] Routing in Drupal 8: Decoupling hook_menu[Srijan Wednesday Webinars] Routing in Drupal 8: Decoupling hook_menu
[Srijan Wednesday Webinars] Routing in Drupal 8: Decoupling hook_menu
 
Clojure Workshop: Web development
Clojure Workshop: Web developmentClojure Workshop: Web development
Clojure Workshop: Web development
 
The hitchhiker's guide to the Webpack - Sara Vieira - Codemotion Amsterdam 2017
The hitchhiker's guide to the Webpack - Sara Vieira - Codemotion Amsterdam 2017The hitchhiker's guide to the Webpack - Sara Vieira - Codemotion Amsterdam 2017
The hitchhiker's guide to the Webpack - Sara Vieira - Codemotion Amsterdam 2017
 
Introduction to AngularJS For WordPress Developers
Introduction to AngularJS For WordPress DevelopersIntroduction to AngularJS For WordPress Developers
Introduction to AngularJS For WordPress Developers
 
Angular Promises and Advanced Routing
Angular Promises and Advanced RoutingAngular Promises and Advanced Routing
Angular Promises and Advanced Routing
 
Creating effective ruby gems
Creating effective ruby gemsCreating effective ruby gems
Creating effective ruby gems
 
Backbone js in drupal core
Backbone js in drupal coreBackbone js in drupal core
Backbone js in drupal core
 
Javascript - Beyond-jQuery
Javascript - Beyond-jQueryJavascript - Beyond-jQuery
Javascript - Beyond-jQuery
 
Ruby on Rails Intro
Ruby on Rails IntroRuby on Rails Intro
Ruby on Rails Intro
 
Introducing AngularJS
Introducing AngularJSIntroducing AngularJS
Introducing AngularJS
 
Express Presentation
Express PresentationExpress Presentation
Express Presentation
 

Recently uploaded

Pushing the limits of ePRTC: 100ns holdover for 100 days
Pushing the limits of ePRTC: 100ns holdover for 100 daysPushing the limits of ePRTC: 100ns holdover for 100 days
Pushing the limits of ePRTC: 100ns holdover for 100 days
Adtran
 
RESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for studentsRESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for students
KAMESHS29
 
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Albert Hoitingh
 
Video Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the FutureVideo Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the Future
Alpen-Adria-Universität
 
Large Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial ApplicationsLarge Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial Applications
Rohit Gautam
 
20 Comprehensive Checklist of Designing and Developing a Website
20 Comprehensive Checklist of Designing and Developing a Website20 Comprehensive Checklist of Designing and Developing a Website
20 Comprehensive Checklist of Designing and Developing a Website
Pixlogix Infotech
 
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FMEEssentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
Safe Software
 
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practicesNational Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
Quotidiano Piemontese
 
Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !
KatiaHIMEUR1
 
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
James Anderson
 
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
Neo4j
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
Octavian Nadolu
 
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdfUni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems S.M.S.A.
 
Elizabeth Buie - Older adults: Are we really designing for our future selves?
Elizabeth Buie - Older adults: Are we really designing for our future selves?Elizabeth Buie - Older adults: Are we really designing for our future selves?
Elizabeth Buie - Older adults: Are we really designing for our future selves?
Nexer Digital
 
By Design, not by Accident - Agile Venture Bolzano 2024
By Design, not by Accident - Agile Venture Bolzano 2024By Design, not by Accident - Agile Venture Bolzano 2024
By Design, not by Accident - Agile Venture Bolzano 2024
Pierluigi Pugliese
 
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
James Anderson
 
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
Neo4j
 
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
Neo4j
 
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionGenerative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Aggregage
 
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex ProofszkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs
Alex Pruden
 

Recently uploaded (20)

Pushing the limits of ePRTC: 100ns holdover for 100 days
Pushing the limits of ePRTC: 100ns holdover for 100 daysPushing the limits of ePRTC: 100ns holdover for 100 days
Pushing the limits of ePRTC: 100ns holdover for 100 days
 
RESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for studentsRESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for students
 
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
 
Video Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the FutureVideo Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the Future
 
Large Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial ApplicationsLarge Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial Applications
 
20 Comprehensive Checklist of Designing and Developing a Website
20 Comprehensive Checklist of Designing and Developing a Website20 Comprehensive Checklist of Designing and Developing a Website
20 Comprehensive Checklist of Designing and Developing a Website
 
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FMEEssentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
 
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practicesNational Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
 
Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !
 
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
 
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
 
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdfUni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdf
 
Elizabeth Buie - Older adults: Are we really designing for our future selves?
Elizabeth Buie - Older adults: Are we really designing for our future selves?Elizabeth Buie - Older adults: Are we really designing for our future selves?
Elizabeth Buie - Older adults: Are we really designing for our future selves?
 
By Design, not by Accident - Agile Venture Bolzano 2024
By Design, not by Accident - Agile Venture Bolzano 2024By Design, not by Accident - Agile Venture Bolzano 2024
By Design, not by Accident - Agile Venture Bolzano 2024
 
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
 
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
 
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
 
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionGenerative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to Production
 
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex ProofszkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs
 

Introduce of the parallel distributed Crawler with scraping Dynamic HTML

  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17.
  • 18.
  • 19. require 'rubygems' require 'sinatra' post '/' do url = params[:url] data = params[:data] store(url, data) next_url = process(url) next_url end
  • 20. // ==UserScript== // @name greasi_scraper // @namespace http://libelabo.jp/ // @include http://images.google.co.jp/* // @require http://ajax.googleapis.com/ajax/libs/jquery/1.3.1/jquery.min.js // ==/UserScript== function postData(data) { var postData = $.param({url: location.href, data: JSON.stringify(data)}); GM_xmlhttpRequest({ method: "POST", url: "http://libelabo.jp/greasi/", headers: {'Content-type':'application/x-www-form-urlencoded'}, data: postData, onload: function(xhr){ location.href = xhr.responseText } }); }