Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Exploring JSON With jq

123 views

Published on

In this talk I show you how to examine JSON data using the jq command-line tool.
Sample files and more discussion can be found on blog at http://aijaz.net/2016/10/25/jq/

Published in: Software
  • Be the first to comment

  • Be the first to like this

Exploring JSON With jq

  1. 1. Slide the First 1 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  2. 2. Mythical Creatures 2 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  3. 3. http://weknowyourdreams.com/images/unicorn/unicorn-04.jpg 3 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  4. 4. https://shellback0608.files.wordpress.com/2016/07/bird-phoenix-flight-art-drawing.jpg 4 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  5. 5. https://i.ytimg.com/vi/kKJQESF0qqk/maxresdefault.jpg 5 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  6. 6. API Docs6 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  7. 7. Exploring JSON With jq Aijaz Ansari @_aijaz_ 7 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  8. 8. What is JSON? JavaScript Object Notation [ { "name": "Alice", "age": 20}, { "name": "Bob", "age": 30} ] 8 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  9. 9. Reading JSON returned by remote servers @interface AAAJSONResponseSerializer : AFJSONResponseSerializer @end @implementation AAAJSONResponseSerializer - (id)responseObjectForResponse:(NSURLResponse *)response data:(NSData *)data error:(NSError *__autoreleasing *)error { #ifdef DEBUG NSHTTPURLResponse * httpResponse = (NSHTTPURLResponse *) response; NSString *body = [[NSString alloc] initWithData:data encoding:NSUTF8StringEncoding]; DDLogInfo(@"RESPONSE BODY: %@", response.URL, body); DDLogInfo(@"RESPONSE: (%ld) %@ ", httpResponse.statusCode, response.URL); #endif // ... } @end 9 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  10. 10. JSON Looks Like This [{"name":"Aang","sex":"M","born":-12,"died":153,"bending":["Air","Water","Earth","Fire","Energy"],"identity":{"nationality":"Southern Air Temple", "ethnicity":"Air Nomad"},"spouse":"Katara","children":[{"sex":"M","name":"Bumi"},{"sex":"F","name":"Kya"},{"sex":"M","name":"Tenzin"}]},{"name":"Katara","sex": "F","born":85,"died":null,"bending":["Water","Blood"],"identity":{"nationality":"Southern Water Tribe","ethnicity":"Water Tribe"},"spouse":"Aang","children":[{" sex":"M","name":"Bumi"},{"sex":"F","name":"Kya"},{"sex":"M","name":"Tenzin"}]},{"name":"Sokka","sex":"M","born":84,"died":164,"bending":[],"identity": {"nationality":"Southern Water Tribe","ethnicity":"Water Tribe"},"spouse":null,"children":[]},{"name":"Toph Beifong","sex":"F","born":88,"died":null,"bending": ["Earth","Metal"],"identity":{"nationality":"Gaoling, Earth Kingdom","ethnicity":"Earth Kingdom"},"spouse":null,"children":[{"sex":"F","name": "Lin Beifong"},{"sex":"F","name":"Suyin Beifong"}]},{"name":"Iroh","sex":"M","born":null,"died":null,"bending":["Fire","Energy"],"identity":{"nationality": "Fire Nation Capital, Fire Nation","ethnicity":"Fire Nation"},"spouse":null,"children":[{"sex":"M","name":"Lu Ten"}]},{"name":"Zuko","sex":"M","born":83, "died":null,"bending":["Fire","Energy"],"identity":{"nationality":"Fire Nation Capital, Fire Nation","ethnicity":"Fire Nation"},"spouse":null,"children": [{"sex":"F","name":"Izumi"}]},{"name":"Kya","sex":"F","born":null,"died":null,"bending":["Water"],"identity":{"nationality":"Southern Water Tribe","ethnicity": "Water Tribe, Air Nomad"},"spouse":null,"children":[]},{"name":"Bumi","sex":"M","born":null,"died":null,"bending":["Air"],"identity":{"nationality": "United Republic","ethnicity":"Water Tribe, Air Nomad"},"spouse":null,"children":[]},{"name":"Tenzin","sex":"M","born":null,"died":null,"bending":["Air"], "identity":{"nationality":"Republic City, United Republic","ethnicity":"Water Tribe, Air Nomad"},"spouse":null,"children":[{"sex":"F","name":"Jinora"},{"sex": "F","name":"Ikki"},{"sex":"M","name":"Meelo"},{"sex":"M","name":"Rohan"}]},{"name":"Lin Beifong","sex":"F","born":120,"died":null,"bending":["Earth","Metal"], "identity":{"nationality":"Republic City, United Republic","ethnicity":"Earth Kingdom"},"spouse":null,"children":[]},{"name":"Suyin Beifong","sex":"F","born": 126,"died":null,"bending":["Earth","Metal"],"identity":{"nationality":"Republic City, United Republic","ethnicity":"Earth Kingdom"},"spouse":null,"children": [{"sex":"M","name":"Bataar Jr."},{"sex":"F","name":"Opal"},{"sex":"M","name":"Wei"},{"sex":"M","name":"Wing"},{"sex":"M","name":"Huan"}]}] 10 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  11. 11. jq Lightweight & Flexible Command-Line JSON Processor. https://stedolan.github.io/jq/ 11 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  12. 12. Pretty Printing $ echo '[{"a": 1},{"b": 2}]' | jq . [ { "a": 1 }, { "b": 2 } ] $ If you remember only one thing about jq, let it be this, the '.' filter. 12 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  13. 13. Filters → All jq programs are filters → A filter takes in input & produces output → 1 is the simplest filter: return 1 → . is the simplest useful filter: pretty print 13 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  14. 14. Object Value Filter $ echo '{"a": 1, "b": 2, "c":3}' | jq .c 3 $ 14 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  15. 15. Arrays $ echo '["a", "b", "c", "d", "e"]' | jq '.[2]' "c" $ echo '["a", "b", "c", "d", "e"]' | jq '.[1:4]' [ "b", "c", "d" ] $ 15 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  16. 16. Iterators Read [] as foreach $ echo '["a", "b", "c", "d", "e"]' | jq '.[]' "a" "b" "c" "d" "e" $ Read .[] as iterating with an identity filter 16 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  17. 17. Addition $ echo '["a", "b", "c", "d", "e"]' | jq '.[]+" hello"' "a hello" "b hello" "c hello" "d hello" "e hello" $ 17 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  18. 18. Addition $ echo '["a", "b", "c", "d", "e"]' | jq '"hey "+ .[]+" hello"' "hey a hello" "hey b hello" "hey c hello" "hey d hello" "hey e hello" $ 18 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  19. 19. Addition $ echo '[1,2,3,4,5]' | jq '.[]+10' 11 12 13 14 15 $ 19 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  20. 20. Collecting Results Into an Array $ echo '[1,2]' | jq '.[]+10' 11 12 $ echo '[1,2]' | jq '[.[]+10]' [ 11, 12 ] 20 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  21. 21. Filters can be piped with | 21 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  22. 22. Creating Objects $ echo '[{"a": 1, "b": 2, "c": 3}, {"a": 10, "b": 20, "c": 30}]' | jq '.[]' { "a": 1, "b": 2, "c": 3 } { "a": 10, "b": 20, "c": 30 } $ 22 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  23. 23. Creating Objects $ echo '[{"a": 1, "b": 2, "c": 3}, {"a": 10, "b": 20, "c": 30}]' | jq '.[]|{a,c}' { "a": 1, "c": 3 } { "a": 10, "c": 30 } $ If you remember only 2 things about jq, let this be the second: extracting just the keys you care about. 23 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  24. 24. Creating Objects $ echo '[{"fn": "Ann", "ln": "Smith"},{"fn": "Ted", "ln": "Jones"}]' | jq '.[]|{"first": .fn, "last": .ln}' { "first": "Ann", "last": "Smith" } { "first": "Ted", "last": "Jones" } $ 24 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  25. 25. Creating Objects $ echo '[{"fn": "Ann", "ln": "Smith"},{"fn": "Ted", "ln": "Jones"}]' | jq '.[]|{(.fn): .ln}' { "Ann": "Smith" } { "Ted": "Jones" } $ 25 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  26. 26. Functions keys $ echo '[{"a": 1, "b": 2, "c": 3}, {"a": 10, "b": 20, "c": 30}]' | jq '.[]|keys' [ "a", "b", "c" ] [ "a", "b", "c" ] $ If you remember only 3 things about jq, let this be the third: examining keys. 26 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  27. 27. Functions length $ echo '[{"a": 1, "b": 2, "c": 3}, {"a": 10, "b": 20, "c": 30}]' | jq '.[]|length' 3 3 $ 27 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  28. 28. Functions map $ echo '[1,2,3,4,5]' | jq 'map(.+10)' [ 11, 12, 13, 14, 15 ] map(x) is equivalent to [.[] | x] 28 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  29. 29. Addition $ echo '[1,2,3,4,5]' | jq '.[]+10' 11 12 13 14 15 $ 29 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  30. 30. Addition $ echo '[1,2,3,4,5]' | jq '[.[]+10]' [ 11, 12, 13, 14, 15 ] $ 30 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  31. 31. Functions map $ echo '[1,2,3,4,5]' | jq 'map(.+10)' [ 11, 12, 13, 14, 15 ] 31 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  32. 32. Functions reduce $ echo '[1,2,3,4,5]' | jq 'reduce .[] as $i (0 ; . + $i)' 15 $ Somewhat equivalent to: $i = 0 for var in .[] { $i = $var + $i } 32 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  33. 33. More Functions → has("foo") → in({"a": "b"}) → echo '[1,2,3]' | jq 'map(.+2)' # [3,4,5] → select(func) - like grep where func is true → ... and a whole lot more 33 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  34. 34. Detailed Examples With Real Data 34 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  35. 35. Sample JSON Body (sample.json) [{"name":"Aang","sex":"M","born":-12,"died":153,"bending":["Air","Water","Earth","Fire","Energy"],"identity":{"nationality":"Southern Air Temple", "ethnicity":"Air Nomad"},"spouse":"Katara","children":[{"sex":"M","name":"Bumi"},{"sex":"F","name":"Kya"},{"sex":"M","name":"Tenzin"}]},{"name":"Katara","sex": "F","born":85,"died":null,"bending":["Water","Blood"],"identity":{"nationality":"Southern Water Tribe","ethnicity":"Water Tribe"},"spouse":"Aang","children":[{" sex":"M","name":"Bumi"},{"sex":"F","name":"Kya"},{"sex":"M","name":"Tenzin"}]},{"name":"Sokka","sex":"M","born":84,"died":164,"bending":[],"identity": {"nationality":"Southern Water Tribe","ethnicity":"Water Tribe"},"spouse":null,"children":[]},{"name":"Toph Beifong","sex":"F","born":88,"died":null,"bending": ["Earth","Metal"],"identity":{"nationality":"Gaoling, Earth Kingdom","ethnicity":"Earth Kingdom"},"spouse":null,"children":[{"sex":"F","name": "Lin Beifong"},{"sex":"F","name":"Suyin Beifong"}]},{"name":"Iroh","sex":"M","born":null,"died":null,"bending":["Fire","Energy"],"identity":{"nationality": "Fire Nation Capital, Fire Nation","ethnicity":"Fire Nation"},"spouse":null,"children":[{"sex":"M","name":"Lu Ten"}]},{"name":"Zuko","sex":"M","born":83, "died":null,"bending":["Fire","Energy"],"identity":{"nationality":"Fire Nation Capital, Fire Nation","ethnicity":"Fire Nation"},"spouse":null,"children": [{"sex":"F","name":"Izumi"}]},{"name":"Kya","sex":"F","born":null,"died":null,"bending":["Water"],"identity":{"nationality":"Southern Water Tribe","ethnicity": "Water Tribe, Air Nomad"},"spouse":null,"children":[]},{"name":"Bumi","sex":"M","born":null,"died":null,"bending":["Air"],"identity":{"nationality": "United Republic","ethnicity":"Water Tribe, Air Nomad"},"spouse":null,"children":[]},{"name":"Tenzin","sex":"M","born":null,"died":null,"bending":["Air"], "identity":{"nationality":"Republic City, United Republic","ethnicity":"Water Tribe, Air Nomad"},"spouse":null,"children":[{"sex":"F","name":"Jinora"},{"sex": "F","name":"Ikki"},{"sex":"M","name":"Meelo"},{"sex":"M","name":"Rohan"}]},{"name":"Lin Beifong","sex":"F","born":120,"died":null,"bending":["Earth","Metal"], "identity":{"nationality":"Republic City, United Republic","ethnicity":"Earth Kingdom"},"spouse":null,"children":[]},{"name":"Suyin Beifong","sex":"F","born": 126,"died":null,"bending":["Earth","Metal"],"identity":{"nationality":"Republic City, United Republic","ethnicity":"Earth Kingdom"},"spouse":null,"children": [{"sex":"M","name":"Bataar Jr."},{"sex":"F","name":"Opal"},{"sex":"M","name":"Wei"},{"sex":"M","name":"Wing"},{"sex":"M","name":"Huan"}]}] http://aijaz.net/2016/10/25/jq/ 35 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  36. 36. http://feelgrafix.com/data_images/out/28/973590-avatar-the-last-airbender.jpg Example data of characters from the 'Avatar' series. 36 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  37. 37. # Pretty print the jq $ cat sample.json | jq '.' ... { "sex": "M", "name": "Huan" } ] } ] $ # The ']' on the last line tells me this is an array # The '}' on the 2nd-last line tells me this is an array of objects # From now on, for space reasons, # 'cat sample.json ' will be abbreviated to 'cj' 37 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  38. 38. # Just show me the first object $ cj | jq '.[0]' 38 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  39. 39. # Show me an object with the name and # sex of the first object $ cj | jq '.[0]|{name, sex}' { "name": "Aang", "sex": "M" } $ 39 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  40. 40. # Show me all the keys of each object $ cj | jq '.[]|keys' ... [ "bending", "born", "children", "died", "identity", "name", "sex", "spouse" ] $ 40 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  41. 41. # Show me all the number of keys of each object $ cj | jq '.[]|length' # same as .[]|keys|length 8 8 8 8 8 8 8 8 8 8 8 $ 41 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  42. 42. # The name of each character $ cj | jq '.[].name' "Aang" "Katara" "Sokka" "Toph Beifong" "Iroh" "Zuko" "Kya" "Bumi" "Tenzin" "Lin Beifong" 42 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  43. 43. # The number of records $ cj | jq '[.[]]|length' 11 $ 43 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  44. 44. # All female characters $ cj | jq '.[]|select(.sex == "F")' 44 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  45. 45. # The names of all female characters # Take the resulting objects, put thmm in an array # Then iterate over that array, printing the name $ cj | jq '[.[]|select(.sex == "F")]|.[].name' "Katara" "Toph Beifong" "Kya" "Lin Beifong" "Suyin Beifong" $ 45 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  46. 46. # The name of all members of the Fire Nation # You can drill down objects with multiple .s # e.g.: .identity.ethnicity $ cj | jq '[.[]|select(.identity.ethnicity == "Fire Nation")]|.[].name' "Iroh" "Zuko" 46 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  47. 47. # The name and bending list of all characters who can bend Fire # Similar to previous example: iterate, select, construct a list, iterate on that list, # and extract the name and bending fields $ cj | jq -c '[.[]|select(.bending[]|select(contains("Fire")))]|.[]|{name, bending}' {"name":"Aang","bending":["Air","Water","Earth","Fire","Energy"]} {"name":"Iroh","bending":["Fire","Energy"]} {"name":"Zuko","bending":["Fire","Energy"]} $ 47 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  48. 48. # Similar to previous, but make the name be the key of the object $ cj | jq -c '[.[]|select(.bending[]|select(contains("Fire")))]|.[]|{(.name): .bending}' {"Aang":["Air","Water","Earth","Fire","Energy"]} {"Iroh":["Fire","Energy"]} {"Zuko":["Fire","Energy"]} 48 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  49. 49. # Names and firstborns of each character, as hashes in an array $ cj | jq '[.[]|{name, firstBorn: .children[0].name}]' 49 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  50. 50. # How many children does each person have? $ cj | jq '[.[]|.name, (.children|length)]' [ { "Aang": 3 }, { "Katara": 3 }, ... { "Suyin Beifong": 5 } ] $ 50 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  51. 51. # List of each person, along with the number of their children # Essentially, merging the array of objects from the previous example into 1 big object $ cj | jq '[.[]|[.name, (.children|length)]] | reduce .[] as $i ({}; . + { ($i[0]): $i[1]} )' { "Aang": 3, "Katara": 3, "Sokka": 0, "Toph Beifong": 2, "Iroh": 1, "Zuko": 1, "Kya": 0, "Bumi": 0, "Tenzin": 4, "Lin Beifong": 0, "Suyin Beifong": 5 } $ 51 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  52. 52. # Take the object matrix from two slides ago, and transpose it $ cj | jq '[.[]|[.name, (.children|length)]] | transpose' [ [ "Aang", "Katara", "Sokka", "Toph Beifong", ... "Suyin Beifong" ], [ 3, 3, 0, 2, ... 5 ] ] $ 52 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  53. 53. Three Most Useful Filters '.' # Pretty print '.[]|{a, b}' # Extract just the a and b keys from each object '.[]|keys' # Display the keys of each object in the array Honorable Mention '[.[]|select(C)]' # Return an array of only those objects where condition C is true 53 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  54. 54. Resources → jq: https://stedolan.github.io/jq/ → jq manual: https://stedolan.github.io/jq/manual/ → Sample JSON file: http://aijaz.net/2016/10/25/jq/ 54 — Aijaz Ansari at NSCoderChicago, 10/25/2016
  55. 55. Thanks! 55 — Aijaz Ansari at NSCoderChicago, 10/25/2016

×