This describes a Python workflow script for preparing log files for analysis. It works with any log format (you need to provide a regex), and compresses files to the extremely efficient parquet format. https://advertools.readthedocs.io/en/master/advertools.logs.html

1. Log File Analysis
With advertools
@EliasDabbas
Feb 17, 2022

2. Raw Data

3. Parse with logs_to_df
• Supports any log format, you just need a regex. Some default formats
available already

4. Get hostnames with
reverse_dns_lookup

5. Split request URLs with url_to_df
• Analyze based on directories, query parameters,
fi
le types, and any combination

6. Split referer URLs with url_to_df

7. Parse User-agents with ua_parser*
* Separate package

8. Combine into one dataset
• Separate analysis work
fl
ow to be developed, these are simple quick
reports

9. Happy analyzing!