Ruby Robots

RubyRobotsDaniel Cukier@danicuki http://www.ﬂickr.com/photos/ﬂysi/183272970

Relatives• spiders• crawlers• bots

Why robot?

http://www.ﬂickr.com/photos/nhankamer/5016628611

require anemoneAnemone.crawl(url) do |anemone| anemone.on_every_page do |page| puts page.url endend http://www.cantora.mus.br/ http://www.cantora.mus.br/fotos http://www.cantora.mus.br/?locale=en http://www.cantora.mus.br/?locale=pt-BR http://www.cantora.mus.br/musicas http://www.cantora.mus.br/videos http://www.cantora.mus.br/agenda http://www.cantora.mus.br/novidades http://www.cantora.mus.br/musicas/baixar http://www.cantora.mus.br/visitors/baixar http://www.cantora.mus.br/social http://www.cantora.mus.br/fotos?locale=pt-BR http://www.cantora.mus.br/musicas?locale=en http://www.cantora.mus.br/fotos?locale=en

XPath<html>...<div class="bla"> <a>legal</a></div>...</html>html_doc = Nokogiri::HTML(html)info = html_doc.xpath( "//div[@class=bla]/a")info.text=> legal

XPath<table id="super"> >> html_doc = Nokogiri::HTML(html) <tr> >> info = html_doc.xpath( <td>L1C1</td> "//table[@id=super]/tr") <td>L1C2</td> >> info.size => 3 </tr> <tr> >> info <td>L2C1</td> => legal <td>L2C2</td> </tr> >> info[0].xpath("td").size <tr> => 2 <td>L3C1</td> <td>L3C2</td> >> info[2].xpath("td")[1].text </tr> => "L3C2"</table>

rest-client

ETGhttp://www.ﬂickr.com/photos/amortize/766738216

http://www.flickr.com/photos/abbeychristine/223898960

Good bot /robots.txt User-agent: * Disallow:http://www.ﬂickr.com/photos/temily/5645585162

RubyRobotsDaniel Cukier@danicuki http://www.ﬂickr.com/photos/ﬂysi/183272970

http://www.ﬂickr.com/photos/nephelim/5632618462

maxRowsList=16

>> body = RestClient.get(url) >> json = JSON.parse(body) >> content = json["Content"] >> content.size => 16 AHA!!! http://.../artistas?maxRowsList=1600&ﬁlter=Recentes >> body = RestClient.get(url) >> json = JSON.parse(a) >> content = json["Content"] >> content.size => 1600http://.../artistas?maxRowsList=1600000&ﬁlter=Recentes >> content.size => 9154 Bingo!!!

>> b["Content"].map {|c| c["ProfileUrl"]}["caravella", "tomleite", "jeffersontavares", "rodrigoaraujo","jorgemendes", "bossapunk", "daviddepiro", "freetools", "ironia","tiagorosa", "outprofile", "lucianokoscky","bandateatraldecarona", "tlounge", "almanaque", "razzyoficial","cretinosecanalhas", "cincorios", "ninoantunes", "caiocorsalette","alinedelima", "thelio", "grupodomdesamba", "ladoz","alexandrepontes", "poeiradgua", "betimalu", "leonardobessa","kamaross", "marcusdocavaco", "atividadeinformal", "angelkeys","locojohn", "forcamusic", "tiaguinhoabreu", "marcelonegrao","jstonemghiphop", "uniaoglobal", "bandaefex", "severarock","manitu", "sasso", "kakka", "xsopretty", "belepoke", "caixaazul","wknd", "bandastarven", "bleiamusic", "3porcentoaocubo","lucianoterra", "hipnoia", "influencianegra", "bandaursamaior","mariafreitas", "jessejames", "vagnerrockxe", "stageo3","lemoneight", "innocence", "dinda", "marcelocapela","paulocamoeseoslusiadas", "magnussrock", "bandatheburk","mercantes", "bandaturnerock", "flaviasaolli", "tonysagga","thiagoponde", "centeio", "grupodeubranco", "bocadeleao","eusoueliascardan", "notoriaoficial", "planomasterrock", "rofgod","dreemonphc", "chicobrant", "osz", "bandalightspeed","cavernadenarnia", "sergiobenevenuto", "viniciusdeoliveira", ...]

email?phone?

>> html = RestClient.get("http://.../robomacaco")>> html_doc = Nokogiri::HTML(html)>> info = html_doc.xpath("//span[@class=name]")>> info.text=> "robo-macaco@hotmail.comRIO DE JANEIRO - RJ - Brasil21 9675-0199

cookiescookies = {}c = "s_nr=12954999; s_v19=12978609471; ... __utmc=206845458"cook = c.split(";").map {|i| i.strip.split("=")}cook.each {|u| cookies[u[0]] = u[1]}RestClient.get(url, :cookies => cookies)

Proxies

http://www.ip-adress.com/proxy_list

>> response = RestClient.get(url)>> html_doc = Nokogiri::HTML(response)>> table = html_doc.xpath("//table[@class=proxylist]")>> lines = table.children>> lines.shift # tira o cabeçalho Text IP>> lines[1].text=> "208.52.144.55 document.write(":"+i+r+i+r)anonymous proxy server-2 minutes ago United States"

JAVASCRIPT = RUBY http://www.flickr.com/photos/drics/4266471776/

>> lines[1].text=> "208.52.144.55 document.write(":"+i+r+i+r) anonymousproxy server-2 minutes ago United States">> server = lines[1].text.split[0]=> "208.52.144.55">> digits = lines[1].text.split(")")[0].split("+")=> ["208.52.144.55document.write(":"", "i", "r", "i", "r"]>> digits.shift>> digits=> ["i", "r", "i", "r"]>> port = digits.map {|c| eval(c)}.join("")=> "8080" VoilàRestClient.proxy = "http://#{server}:#{port}"

mechanizeagent = Mechanize.newsite = "http://www.cantora.mus.br"page = agent.get("#{site}/baixar")form = page.formform[visitor[name]] = danielform[visitor[email]] = "danicuki@gmail.com"page = agent.submit(form)tracks = page.links.select { |l| l.href =~ /track/ }tracks.each do |t| file = agent.get("#{site}#{t}) file.saveend

protection techniques javascript text as image captcha don’t be ingenuous

captchaprove you are not a robot YES you can!

3 steps1. Download Image2. filter image3. run OCR software

scalinghttp://www.ﬂickr.com/photos/liquene/3330714590

clouds$ knife ec2 server create

threads +queues

Nessa vida de programador malucoMe aparece cada situaçãoDe repente um cliente, uma proposta brutaPra pegar de um site informaçãoVocê tá louco, esse tipo de crime eu não façoSe quiser tenho uns amigos lá do sulFaz pra mim que eu te pago com essa jóia coolTe dou um rubyPra você roubarCom o seu robôQuer fazer robô?É só usar rubyÉ só usar rubyPra fazer robô http://www.ﬂickr.com/photos/jobafunky/5572503988

Thank youDaniel Cukier@danicuki

http://www.agileandart.com	823
http://eeruiz.blogspot.com	13
http://agileandart.com	7
http://www.hanrss.com	3

Ruby Robots

by Daniel Cukier on Aug 13, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

4 Embeds 846

Statistics

Ruby Robots — Presentation Transcript