Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

PykQuery.js

959 views

Published on

PykQuery is a conceptual framework that help organise complexity around data-driven interactive single page apps.

Published in: Data & Analytics
  • Be the first to comment

PykQuery.js

  1. 1. A conceptual framework for data-interactivity PykQuery.js Ritvvij Parrikh, Founder, pykih.com | pykcharts.com
  2. 2. Genesis Pykih is a data design company. We make data human i.e. we use algorithms and design to help humans consume data.
  3. 3. Genesis Built for FirstPost.com - As seen on History Channel Documentary - Vital Stats of India
  4. 4. Genesis GapMinder style design for 500 startups (VC) to analyse their portfolio
  5. 5. Genesis Real-time social media dashboard for Microsoft World Partner Conference
  6. 6. Genesis Nature of our business: • Data can be streaming live or static • Data could be big or small • User can be on mobile or laptop • Traffic can be in millions (media) or few hundreds (ERP) • Good Design and interactivity is a must 2, 10, 30, 50 projects later… we realised the need for PykQuery
  7. 7. Warning What it is NOT • NOT an Angular / React alternative • Does NOT help with data binding • NOT a monolithic framework • NOT an isomorphic framework (e.g. Meteor) • NOT a non-isomorphic framework • Does NOT choose the database for you (e.g. Meteor is built for MongoDB)
  8. 8. What PykQuery is a conceptual framework that help organise complexity around data-driven interactive single page apps.
  9. 9. What PykQuery is a conceptual framework that help organise complexity around data-driven interactive single page apps. Even MVC sites have 1 or 2 very powerful single page apps inside them Searching, Filtering, Analytics, Visualisation, Dashboards, etc. You can use our code or use the concepts to write your own code.
  10. 10. Why (Developer) PykQuery is a conceptual framework that help organise complexity around data-driven interactive single page apps. • Maintainable Code • Black box out data-related code • Black box out interactivity related code
  11. 11. Why (Business) PykQuery is a conceptual framework that help organise complexity around data-driven interactive single page apps. • Invest maximum time on design
  12. 12. Types of Interactivity Interactive visual and data driven analytics provides different ways to extract knowledge from very large data sets: Visual Interactivity —Visual interactivity allows user to change the styling/ visual appearance of elements by clicking on other elements. Data driven Interactivity — Data driven interactivity extends this to the capability of changing the data itself, allowing user to discover more things about data, on their own.
  13. 13. Is data interactivity only for data visualisation?
  14. 14. E-commerce Laptop Search Page
  15. 15. How we do it today? • Expose backend API that gives us list of all countries • Expose backend code to give us languages for selected country Backend Front-end • Ajax call to this API 1 • Render dropdown • User selects country • Ajax call to API 2 with selected country • Render dropdown
  16. 16. Extrapolate this 100x and see problems Spreadsheet for geo-spatial analysis
  17. 17. Problems with data interactivity #1 - Messy Backend Code • Front-end team dependent on backend team to write APIs • 5 min. to write drop down code; 15 min. to expose 1 API and deploy to prod. • Prod. code ends up with multiple APIs • Front-end can stop using the APIs without telling backend (dead code in prod.)
  18. 18. Step #1 - Run a backend server Download a piece of backend code - http://github.com/pykih/PykQuery.rb Connect to your database Switch on server
  19. 19. Step #2 - Write a query in JSON in browser { "dataformat":"json”, "scope":"local”, "impacts":[], "impactedby":["connector1”], "div_id":"map_local”, "alias":{}, "rumiparams":"pykih/icici-ha/untitled-grid-13-csv/“, "metrics":{}, "filters”:[ { "column_name":"city”, "condition_type":"values”, "next":"AND”, "in":["mumbai”], "not_in":[] } ], "limit":2000, "sort":[{"price_for_sorting":"desc"}], “dimensions”:["price_for_sorting","name","map_coordinate_lat", "map_coordinate_lon","city","address","computed_rating","slug”], "offset":0, "mode":”aggregation" }
  20. 20. Step #3 - Call the .call method c.call();
  21. 21. Step #4 - Render the DOM for the data received
  22. 22. Turns out most UI elements are SELECT queries
  23. 23. 1 scalable backend API for 70% of your front-end data needs Front-end developer with basic SQL knowledge writes a JSON object Chooses if he wants the API to be in GET or POST Chooses if he wants the API to be with or without tokens Chooses if he wants the output to be in CSV, JSON or XML or 2d Array Sends it to the API And he receives back response PostGreSQL MySQL Oracle MS SQL PykQuery Object SQL API Data
  24. 24. Problems with data interactivity #1 - Messy Backend Code #2 - No consistent way to propagate WHERE clauses onSelect of country —> Append “where country = selected_country” to Language query
  25. 25. Interactivity Management Disclaimer: This is a screenshot of BookMyShow and Pykih has no relation with BookMyShow.
  26. 26. Let’s abstract #cities #movies #cinemas #shows
  27. 27. How it works? #cities #movies #cinemas #shows
  28. 28. Theoretically speaking #cities Interactivity is a graph problem and hence theoretically there can be n*n-1 relationships for every n nodes. #movies #shows #cinemas
  29. 29. Problems with data interactivity #1 - Messy Backend Code #2 - No consistent way to propagate WHERE clauses #3 - Managing complex interactivity
  30. 30. Conceptual Abstraction - Connector Object Each PykQuery object has a scope: NODE or CONNECTOR Connector PykQuery Object holds: • where clauses • “impacted by” and “impacts” relationships Advantages • Number of relationships are down to 8 from 12 • You debug only the connector object and not every one #cities #shows #cinemas #moviesConnector
  31. 31. Step #5 - Define the interactivity #cities #shows #movies#cinemas Connector PykQuery Object
  32. 32. Problems with data interactivity #1 - Messy Backend Code #2 - No consistent way to propagate WHERE clauses #3 - Managing complex interactivity #4 - UI for the big data age is the browser (<5MB) • The problem is further aggravated for Mobile WEB where not only is the browser’s RAM less but the bandwidth is also poor.
  33. 33. Step #6 - Lazy load DB data. Query in-browser to render Pull slices from DB and query them in-browser Inspiration: Paging data between Cache, RAM and HDD DB Browser Memory / IndexedDB Rendered DOM API PykQuery In-browser Querying PykQuery DB Querying
  34. 34. Example use case
  35. 35. Problems with in-browser querying No good way of doing it • DB querying and in-browser querying APIs are not same. • In-browser querying is complex. Developers need to use individual functions, think and write logic to build. Native Underscore Crossfilter Angular In-browser querying Yes Yes Yes No In-browser data interactivity No No Yes No Grouping two columns No Complex Complex No Handles data across multiple API responses No Yes No No
  36. 36. In-Browser adapter on the PykQuery Object • Same interface to write query • Switch adapter attribute to “inbrowser" from “db”
  37. 37. Problems with data interactivity #1 - Messy Backend Code #2 - No consistent way to propagate WHERE clauses #3 - Managing complex interactivity #4 - UI for the big data age is the browser (<5MB) #5 - Can we save interactivity states? • You have a doctor or hospital discovery front-end. My friend’s father needs urgent surgery and I’m helping him out to find the right hospital. I filter on your UI to find the right hospitals. Next, I want to take the results found and share it with my friend. How are we doing this today?
  38. 38. Today • URL parameters http://www.flipkart.com/laptops/pr?p%5B%5D=facets.price_range%255B %255D%3DRs.%2B60001%2Band%2BAbove&sid=6bo %2Cb5g&q=laptops&ref=704d2c01-1701-4abf-b0c3-6276d7bd24cc
  39. 39. Step #7 - Save the PykQuery Object! • A built in function to list and delete filters • Save a filtered scenario • Load a saved scenario • Create an animation from a list of saved interaction states • Undo an interaction
  40. 40. Step #8 - Focus on design With data in your browser, data interactivity defined and taken care of, now your design team can focus maximum time on building the right UX.
  41. 41. Philosophical Construct Apps keep changing. Data is eternal. The river is your primary app. but around it you will build multiple canals that will pull data from the river for specific purposes. E.g. Why can’t a travel company build a micro-site that optimises on only Kumbh Mela related travel. It internally reuses the same infrastructure.
  42. 42. PykQuery v/s JS Alternatives Native Underscore Crossfilter Angular PykQuery In-browser querying Yes Yes Yes No Yes In-browser data interactivity No No Yes No Yes Grouping two columns No Complex Complex No Yes Handles data across multiple API responses No Yes No No In-browser No but DB yes DB querying No No No No Yes DB data interactivity No No No No Yes Data Binding (DOM) No No No Yes No
  43. 43. Releasing https://github.com/pykih/PykQuery.js — JS Library https://github.com/pykih/PykQuery.rb — Backend code in Ruby for PostGreSQL
  44. 44. We are @pykih http://PykCharts.com: 26 d3.js charts without the complexity of d3.js.
  45. 45. Countries Drop down Code A JSON Object that encapsulates all types of SELECT queries (except joins). <select id="c"></select> <script type="text/javascript"> window.c = new PykQuery.init("aggregation", "node", “c”, "db"); c.dimensions = [“country”]; c.sort = [{“country”: "asc"}]; c.call(); var data = c.flushToGet(); for(var i = 0; i < data.length; i++){ $("#c").append('<option value='+data[i].country +'>'+data[i].country+'</option>'); } </script>
  46. 46. Languages Dropdown code <select id="c"></select> <select id="l"></select> <script type="text/javascript"> window.l = new PykQuery.init("aggregation", "node", "l", "db"); l.dimensions = ["language"]; l.sort = [{"language": "asc"}]; l.executeOnFilter = function() { var data = l.flushToGet(); for(var i = 0; i < data.length; i++){ $("#c").append('<option value='+data[i].language+'>'+data[i].language+'</ option>'); } }; l.call(); c.addImpacts(["l"], false); $("#c").change(function() { c.addFilter([[{"column_name": "country", "condition_type": "values", "in": ["India"]}]], false, true); }); </script>
  47. 47. addImpacts function <select id="c"></select> <select id="l"></select> <script type="text/javascript"> window.l = new PykQuery.init("aggregation", "node", "l", "db"); l.dimensions = ["language"]; l.sort = [{"language": "asc"}]; l.executeOnFilter = function() { var data = l.flushToGet(); for(var i = 0; i < data.length; i++){ $("#c").append('<option value='+data[i].language+'>'+data[i].language+'</ option>'); } }; l.call(); c.addImpacts(["l"], false); $("#c").change(function() { c.addFilter([[{"column_name": "country", "condition_type": "values", "in": ["India"]}]], false, true); }); </script>
  48. 48. addFilter function <select id="c"></select> <select id="l"></select> <script type="text/javascript"> window.l = new PykQuery.init("aggregation", "node", "l", "db"); l.dimensions = ["language"]; l.sort = [{"language": "asc"}]; l.executeOnFilter = function() { var data = l.flushToGet(); for(var i = 0; i < data.length; i++){ $("#c").append('<option value='+data[i].language+'>'+data[i].language+'</ option>'); } }; l.call(); c.addImpacts(["l"], false); $("#c").change(function() { c.addFilter([[{"column_name": "country", "condition_type": "values", "in": ["India"]}]], false, true); }); </script>
  49. 49. executeOnFilter <select id="c"></select> <select id="l"></select> <script type="text/javascript"> window.l = new PykQuery.init("aggregation", "node", "l", "db"); l.dimensions = ["language"]; l.sort = [{"language": "asc"}]; l.executeOnFilter = function() { var data = l.flushToGet(); for(var i = 0; i < data.length; i++){ $("#c").append('<option value='+data[i].language+'>'+data[i].language+'</ option>'); } }; l.call(); c.addImpacts(["l"], false); $("#c").change(function() { c.addFilter([[{"column_name": "country", "condition_type": "values", "in": ["India"]}]], false, true); }); </script>
  50. 50. The whole code <select id="c"></select> <select id="l"></select> <script type="text/javascript"> window.l = new PykQuery.init("aggregation", "node", "l", "db"); l.dimensions = ["language"]; l.sort = [{"language": "asc"}]; l.executeOnFilter = function() { var data = l.flushToGet(); for(var i = 0; i < data.length; i++){ $("#c").append('<option value='+data[i].language+'>'+data[i].language+'</ option>'); } }; l.call(); c.addImpacts(["l"], false); $("#c").change(function() { c.addFilter([[{"column_name": "country", "condition_type": "values", "in": ["India"]}]], false, true); }); </script>
  51. 51. In-browser querying DB #raw data Node DB PykQuery Object 1. Data: country, language US, English US, Spanish India, English … window.rawdata = new PykQuery.init("aggregation", "node", "rawdata", "db"); rawdata.dimensions = ["country","language"]; rawdata.call();
  52. 52. In-browser querying… Connector Obj. Flush Raw data#raw data Node DB PykQuery Object 1. 2. window.connector_pykquery = new PykQuery.init("connector", "connector", "connector_pykquery", "inbrowser"); connector_pykquery.rawdata = JSON.parse(rawdata.flushToGet());
  53. 53. In-browser querying… #countries Node InBrowser PykQuery Object Connector Obj. InBrowser Query Data for Countries 2. 3. #raw data Node DB PykQuery Object 1. window.c = new PykQuery.init("aggregation", "node", "c", "inbrowser"); c.addImpacts(["connector_pykquery"], false); c.dimensions = ["country"]; c.call(); var data = c.flushToGet(); //LOOP on variable data to add OPTIONS to dropdown countries (#c)
  54. 54. In-browser querying… #countries Node InBrowser PykQuery Object #languages Node InBrowser PykQuery Object onSelect 3. 4. #raw data Node DB PykQuery Object 1. 2. Connector Obj. window.l = new PykQuery.init("aggregation", "node", "l", "inbrowser"); connector_pykquery.addImpacts(["l"], false); l.dimensions = ["language"]; l.executeOnFilter = function() { var data = l.flushToGet(); //LOOP on variable data to add OPTIONS to dropdown languages (#l) }; l.call();
  55. 55. In-browser querying… #languages Node InBrowser PykQuery ObjectInBrowser Query Data for Languages 4. 5. #raw data Node DB PykQuery Object 1. 2. Connector Obj. #countries Node InBrowser PykQuery Object 3. $("#c").change(function() { c.addFilter([[{"column_name": "country", "condition_type": "values", "in": ["India"]}]], false, true); });

×