Big Data Analytics with Google BigQuery. By Javier Ramirez. All your base Conference 2014
•Download as ODP, PDF•
3 likes•1,610 views
At teowaki we are using Google BigQuery to get analytics from our API usage. Learn how you can benefit from BigQuery to have Big Data analytics.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Big Data Analytics with Google BigQuery. By Javier Ramirez. All your base Conference 2014
Similar to Big Data Analytics with Google BigQuery. By Javier Ramirez. All your base Conference 2014 (20)
More from javier ramirez
More from javier ramirez (20)
Recently uploaded
Recently uploaded (20)
Big Data Analytics with Google BigQuery. By Javier Ramirez. All your base Conference 2014
- 1. Big Data Analytics with Google BigQuery javier ramirez @supercoco9
- 2. REST API + AngularJS web as an API client javier ramirez @supercoco9 https://teowaki.com nosqlmatters 2013
- 3. javier ramirez @supercoco9 https://teowaki.com
- 4. bigdata is doing a fullscan to 330MM rows, matching them against a regexp, and getting the result (223MM rows) in just 5 seconds javier ramirez @supercoco9 https://teowaki.com Javier Ramirez impresionable teowaki founder
- 5. Apache Hadoop Apache Cassandra Apache Spark Apache Storm Amazon Redshift javier ramirez @supercoco9 https://teowaki.com
- 6. bigdata is cool but... expensive cluster hard to set up and monitor not interactive enough
- 7. Our choice: Google BigQuery Data analysis as a service http://developers.google.com/bigquery javier ramirez @supercoco9 https://teowaki.com
- 8. Based on Dremel Specifically designed for interactive queries over petabytes of real-time data javier ramirez @supercoco9 https://teowaki.com
- 9. What Dremel is used for in Google • Analysis of crawled web documents. • Tracking install data for applications on Android Market. • Crash reporting for Google products. • OCR results from Google Books. • Spam analysis. • Debugging of map tiles on Google Maps. • Tablet migrations in managed Bigtable instances. • Results of tests run on Google’s distributed build system. • Disk I/O statistics for hundreds of thousands of disks. • Resource monitoring for jobs run in Google’s data centers. • Symbols and dependencies in Google’s codebase.
- 10. in BigQuery everything is a full-scan* *Over a ridiculously fast distributed filesystem. Dremel design goal: 1TB/sec. It was exceeded BigQuery delivers ~ 50Gb/Sec. javier ramirez @supercoco9 https://teowaki.com
- 11. Columnar storage javier ramirez @supercoco9 https://teowaki.com
- 12. highly distributed execution using a tree javier ramirez @supercoco9 https://teowaki.com rubyc kiev 14
- 13. loading data You can feed flat CSV-like files or nested JSON objects javier ramirez @supercoco9 https://teowaki.com
- 14. bq cli bq load --nosynchronous_mode --encoding UTF-8 --field_delimiter 'tab' --max_bad_records 100 --source_format CSV api.stats 20131014T11-42- 05Z.gz javier ramirez @supercoco9 https://teowaki.com
- 15. web console screenshot javier ramirez @supercoco9 https://teowaki.com
- 16. analytical SQL functions. correlations. window functions. views. JSON fields. timestamped tables. javier ramirez @supercoco9 https://teowaki.com
- 17. Things you always wanted to try but were too scared to select count(*) from publicdata:samples.wikipedia where REGEXP_MATCH(title, "[0-9]*") AND wp_namespace = 0; 223,163,387 Query complete (5.6s elapsed, 9.13 GB processed, Cost: 32¢) javier ramirez @supercoco9 https://teowaki.com
- 19. Global Database of Events, Language and Tone quarter billion rows 30 years updated daily http://gdeltproject.org/data.html#googlebigquery
- 20. SELECT Year, Actor1Name, Actor2Name, Count FROM ( SELECT Actor1Name, Actor2Name, Year, COUNT(*) Count, RANK() OVER(PARTITION BY YEAR ORDER BY Count DESC) rank FROM (SELECT Actor1Name, Actor2Name, Year FROM [gdelt-bq:full.events] WHERE Actor1Name < Actor2Name and Actor1CountryCode != '' and Actor2CountryCode != '' and Actor1CountryCode!=Actor2CountryCode), (SELECT Actor2Name Actor1Name, Actor1Name Actor2Name, Year FROM [gdelt-bq:full.events] WHERE Actor1Name > Actor2Name and Actor1CountryCode != '' and Actor2CountryCode != '' and Actor1CountryCode!=Actor2CountryCode), WHERE Actor1Name IS NOT null AND Actor2Name IS NOT null GROUP EACH BY 1, 2, 3 HAVING Count > 100 ) WHERE rank=1 ORDER BY Year
- 22. javier ramirez @supercoco9 https://teowaki.com
- 23. Automation with Apps Script Read from bigquery Create a spreadsheet on Drive E-mail it everyday as a PDF javier ramirez @supercoco9 https://teowaki.com
- 25. what is it being used for?
- 28. Analysing weather information Finding patterns in e-commerce Match online/offline behaviour Log analysys Analysing inventory/booking data ...
- 29. bigquery pricing $80 per stored TB 1000000 rows => $0.02288 / month $35 per processed TB 1 full scan ~ 240 MB 1 count = 0 MB 1 full scan over 1 column ~ 13 MB 10 GB => $0.35 / month *the 1st TB processed every month is free of charge javier ramirez @supercoco9 https://teowaki.com
- 30. Find related links at https://teowaki.com/teams/javier-community/link-categories/bigquery-talk Thanks Javier Ramírez @supercoco9
Editor's Notes
- nadie duda de que tu api sea técnicamente muy buena, pero...
- conclusión obvia esto va a ser un problema de big data el problema es que nosotros no sabíamos de big data. Nos sonaba map/reduce, hadoop, cassandra.. pero nos faltaban datos
- esto hace dos años era imposible. vivimos en el futuro para poder ejecutar consultas sobre nuestros datos, el primer punto es extraerlos de nuestro sistema, que en nuestro caso significa extraer la información de las peticiones del usuario conforme van pasando
- Apache Drill es el equivalente en open source. No funciona como servicio. bigquery es un recubrimiento REST encima de dremel. Usable desde cualquier plataforma que permita REST. Apis disponibles para diferentes lenguajes Solamente para inserciones! no borrados o updates.A menudo junto Map/reduce o hadoop. Análisis in place, sin carga previa, sin índices ni planificar las queries de antemano
- next: full scan regexp
- Column data is of uniform type; therefore, there are some opportunities for storage size optimizations available in column-oriented data that are not available in row-oriented data. also less I/O Además Dremel proporciona una estructura en árbol para lanzar las queries
- batch y tiempo real tanto en la entrada de datos (ficheros o stream) como en la salida (interactivo o batch) pagas por lo que usas
- batch y tiempo real tanto en la entrada de datos (ficheros o stream) como en la salida (interactivo o batch) pagas por lo que usas
- la carga puede ser de fichero plano (tsv para evitar problemas de comillas) o con json si necesitas estructura. Importar desde consola web, REST o command line se pueden importar ficheros comprimidos también se puede importar información para tiempo real en modo stream NEXT: CONCEPTOS DE BIGQUERY
- web console api rest command line Notice the validate button to avoid expenses
- next: full scan regexp
- total 313,797,035
- global database of events, language and tone quarter billion rows 30 years updated daily
- Column data is of uniform type; therefore, there are some opportunities for storage size optimizations available in column-oriented data that are not available in row-oriented data. also less I/O Además Dremel proporciona una estructura en árbol para lanzar las queries
- Column data is of uniform type; therefore, there are some opportunities for storage size optimizations available in column-oriented data that are not available in row-oriented data. also less I/O Además Dremel proporciona una estructura en árbol para lanzar las queries