In Data Engineer's Lunch #19, we will introduce jq and how we can use it for data engineering. jq is a command-line tool like sed for JSON data and can be used to slice, filter, map, and transform structured data. GitHub Repo: https://github.com/Anant/example-introduction-to-jq Accompanying Blog: https://blog.anant.us/data-engineers-lunch-19-introduction-to-jq-for-data-engineering/ Accompanying YouTube: https://youtu.be/92jUGtk_ABg Sign Up For Our Newsletter: http://eepurl.com/grdMkn Join Data Engineer’s Lunch Weekly at 12 PM EST Every Monday: https://www.meetup.com/Data-Wranglers-DC/events/ Cassandra.Link: https://cassandra.link/ Follow Us and Reach Us At: Anant: https://www.anant.us/ Awesome Cassandra: https://github.com/Anant/awesome-cassandra Email: solutions@anant.us LinkedIn: https://www.linkedin.com/company/anant/ Twitter: https://twitter.com/anantcorp Eventbrite: https://www.eventbrite.com/o/anant-1072927283 Facebook: https://www.facebook.com/AnantCorp/

1. Version 1.0
Introduction to jq
An Anant Corporation Story.
jq for Data Engineering

2. jq
● jq is a tool that can be used to slice, filter, map, and transform
structured data with the same ease as sed and awk
● Written in C
● Typical Uses
○ Parsing and manipulating JSON data
○ Extracting a particular field of an object, converting a
number to a string, or various other standard tasks
● How it works
○ The input to jq is parsed as a sequence of whitespace-separated
JSON values which are passed through the provided filter one
at a time.
○ The output(s) of the filter are written to standard out, again as a
sequence of whitespace-separated JSON data.
● Basic syntax
○ jq -options 'program' input-file
○ Documentation
○ Tons of built-in operators and functions as well

3. Demo
● https://github.com/Anant/example-introduction-to-jq
● Basic Printing
● Selections
● Value Changes
● Filtering
● Deletions
● Real World Application (more complex programs / operations)

4. Resources
● https://stedolan.github.io/jq/manual/
● https://stackoverflow.com/questions/32960857/how-to-convert-arbitrary-si
mple-json-to-csv-using-jq

5. Strategy: Scalable Fast Data
Architecture: Cassandra, Spark, Kafka
Engineering: Node, Python, JVM,CLR
Operations: Cloud, Container
Rescue: Downtime!! I need help.
www.anant.us | solutions@anant.us | (855) 262-6826
3 Washington Circle, NW | Suite 301 | Washington, DC 20037