In Data Engineer’s Lunch #6: Common Data Formats Used in Data Engineering, we discuss common data formats used in data engineering including text/file and binary formats. Accompanying Blog: https://blog.anant.us/data-engineers-lunch-6-common-data-formats-used-in-data-engineering/ Accompanying Recording: https://youtu.be/m30xBgThrJo Join Data Engineer’s Lunch Weekly at 12 PM EST Every Monday: https://www.meetup.com/Data-Wranglers-DC/events/ Cassandra.Link: https://cassandra.link/ Follow Us and Reach Us At: Anant: https://www.anant.us/ Awesome Cassandra: https://github.com/Anant/awesome-cassandra Email: solutions@anant.us LinkedIn: https://www.linkedin.com/company/anant/ Twitter: https://twitter.com/anantcorp Eventbrite: https://www.eventbrite.com/o/anant-1072927283 Facebook: https://www.facebook.com/AnantCorp/

1. Version 1.0
Common Data Formats Used in Data
Engineering
An Anant Corporation Story.

2. Formats Discussed
● Text/File
● Binary

3. Text/File
● CSV - Comma Separated Value - text file
○ open it in vi / notepad / sublimetext
○ fieldnames at the top <- the metadata
○ every major record is one line
○ CSV is a mediocre data format for software (due to issues with escaping quotations and commas), but is a
great interchange format for interchanging data between people and organizations.
○ open it in Excel, Google Spreadsheets, Open Office
● TSV - Tab Separated Value - text file
○ open it in vi / notepad / sublimetext
○ fieldnames at the top <- the metadata
○ every major record is one line
○ humans cans scan it / read it like a "report"
○ TSV is also faster for some parsing operations.

4. Text/File Cont.
● XML
○ EBXML
○ Web Services - SOAP - Simple Object Access Protocol
○ XML-RPC
○ SGML -> HTML -> XML
○ Self Describing
■ <book>
1. <authors>
a. <author name="Rahul Singh">
b. <author name="Rahul Singh">
2. </authors>
3. <title name="Enterprise Consciousness">
4. <isbn number="">
■ </book>
■ <book>
1. <author name="William Angel">
2. <title name="Virtual Power...">
■ <book>
○ XSD
○ XSLT -> Reformat the data
○ Hierarchy
○ Heavy

5. Text/File Cont.
● JSON
○ Web Services - REST / JSON
○ Hierarchy
○ Readable
○ Metadata in the JSON
○ Not as heavy
○ Self Describing
○ Avro Spec
○ GeoSON
● SQL
● Text Files (What They Are & How to Open One)

6. Binary
● Serialization/Deserialization formats
○ PKL
■ A PKL file is a file created by pickle, a Python module that enables objects to be serialized to files on
disk and deserialized back into the program at runtime. It contains a byte stream that represents the
objects.
○ Kryo
● Compressed
○ * Text Files
○ tar
○ gz
○ zip
○ 7z
● Parquet
● JSON
○ BSON
○ Avro Compressed

7. Resources
● Comparison of data-serialization formats - Wikipedia
● Data Serialization Comparison: JSON, YAML, BSON, MessagePack - SitePoint
● Apache Arrow and Distributed Compute with Kubernetes - XenonStack

8. Strategy: Scalable Fast Data
Architecture: Cassandra, Spark, Kafka
Engineering: Node, Python, JVM,CLR
Operations: Cloud, Container
Rescue: Downtime!! I need help.
www.anant.us | solutions@anant.us | (855) 262-6826
3 Washington Circle, NW | Suite 301 | Washington, DC 20037