The art of Node.js or how to do something cool using public data by Ilya Zaprutski
•
1 like•3,110 views
Today, economists believe that the most valuable resource is not oil, but data. The Internet, in turn, is the place where we can find any information, the main thing is to know where and what to look for. During the report, I will tell how, having an original idea and a limited budget, make an application that will be really useful, and on which you can make money.
Recommended
Recommended
More Related Content
What's hot
What's hot (6)
Similar to The art of Node.js or how to do something cool using public data by Ilya Zaprutski
Similar to The art of Node.js or how to do something cool using public data by Ilya Zaprutski (20)
More from OdessaJS Conf
More from OdessaJS Conf (20)
The art of Node.js or how to do something cool using public data by Ilya Zaprutski
- 1. The Art of Node.js or how to do something cool using public data Ilya Zaprutski Summer. Sea. JavaScript.
- 2. The Art of Node.js или как сделать что-то крутое, используя публичные данные
- 3. 3 Всем привет! Я - Илья Запруцкий Software Engineer в iTechArt Group Telegram: @ilya_zaprutski
- 7. 7 Его нет Даже за $
- 10. 10 Web scraping
- 11. 11
- 12. 12
- 14. 14 Этапы
- 17. request-promise – HTTP клиент, который позволяет делать простые и быстрые запросы 17
- 18. 18 import request from 'request-promise'; request('http://www.google.com') .then((htmlString) => { // Process html... }) .catch((err) => { // Crawling failed... });
- 19. Ограничения 19
- 22. 22 GraphQL
- 23. 23
- 24. 24 const options = { uri: 'https://www.instagram.com/ilya.zaprutski/', setViewport: { width: 1240, height: 680 }, }; const scrape = async (options) => { const browser = await puppeteer.launch(options); const page = await browser.newPage(); await page.setViewport(options.setViewport); await page.goto(options.uri); const content = await page.content(); await browser.close(); return content; }; 24
- 26. 26 Puppeteer
- 27. Scheduler 27
- 28. 28 node-schedule
- 29. 29 import schedule from 'node-schedule'; const job = schedule.scheduleJob('30 0 * * *', () => { generateAlerts(); });
- 31. 31
- 32. Загружаем и устанавливаем Tor 32
- 33. Запускаем 33
- 34. 34 const TorHttpAgent = require('socks5-http-client/lib/Agent'); const TorHttpsAgent = require('socks5-https-client/lib/Agent'); const scrape = (params) => { const options = { ...defaultOptions, ...params }; const parsedUrl = parse(options.url); const Agent = parsedUrl.protocol === 'https:' ? TorHttpsAgent : TorHttpAgent; options.headers = { ...options.headers, 'User-Agent': ua.getRandom() }; options.agentClass = Agent; options.agentOptions = { socksHost: 'localhost', socksPort: options.socksPort || 9050, }; return request(options); }; 34
- 36. 36 cheerio – jQuery для Node.js
- 37. 37 import cheerio from 'cheerio'; const htmlStr = '<h2 class="title">Hello</h2>’; const $ = cheerio.load(htmlStr); $('h2.title').text(‘Odessa js!!!'); $('h2').addClass('welcome’); $.html(); //= > <h2 class="title welcome">Odessa js!!!</h2> 37
- 38. 38 const defaultOptions = { uri: 'https://www.itechart.by', transform: html => cheerio.load(html), }; const scrape = (params) => { const options = { ...defaultOptions, ...params }; return request(options); }; 38
- 40. 40
- 42. 42
- 44. 44 Демо 44
- 45. 45 Демо
- 46. 4646
- 47. 47
- 49. 49 Озеро данных
- 50. 50 Данные с быстрым доступом Структурированные данные Архивные данные
- 53. 5353 Crawler
- 61. 6161
- 67. × Блокировка юзер-агента × Блокировка IP-адресов × Использование капчи 67
- 76. 76 Is Web Scraping Illegal? Web Scraping and Crawling Are Perfectly Legal, Right? Do not protect your website from scraping