SQLite forensics - Free Lists, unallocated space, carving

http://belkasoft.com
SQLite Forensics
Yuri Gubanov, Belkasoft

Introducing myself
Yuri Gubanov
• CEO and Founder of Belkasoft
• Belkasoft – digital forensics software manufacturer
• Frequent speaker on industry-known conferences
• Senior lecturer in St-Petersburg State University, Russia
• Software developer by education, MSc

Today’s agenda
Digging deep about SQLite
• What is SQLite?
• Why is it important?
• Deleting data from SQLite: Freelists
• Unallocated: not only a hard drive
• SQLite WAL and journal files
• Some real stories
• A little practice

Before we start: my Skype profile

What is SQLite?
Popular database engine
• Relational SQL-based database
• Lightweight
• Does not require installation
• Just a file on disk
• Well, sometimes more than one
• Does not require system services

Why SQLite is important?
SQLite is massively used in modern apps
• Database of choice for 90+% of new apps
• Especially mobile ones
• Most of apps in Android/iOS
• E.g.: messengers Skype, WhatsApp, Viber
• E.g.: Major browsers such as Firefox, Chrome, Safari
• E.g. iPhone SMS database

Frequent approach to SQLite forensics
Often SQLite analysis is like following:
• Select free or open source tool
• E.g. DB Browse for SQLite (formerly SQLite Database Browser)
• E.g. SQLite viewer addon to Mozilla Firefox
• The price is reasonable! 
• Open database in the viewer
• Trust the results
• End of story

Drawbacks of using a non-forensic tool
“Free” does not equal “trustworthy”
• Does not show deleted records
• Does not even indicate presence of deleted records
• Does not show uncommitted records
• Does not show remnants in “unallocated”
• Does not carve database files
• Does not find SQLite files in RAM memory/dumps/hibernation/pagefile
Even single feature above can change investigation conclusions
dramatically (and we had real cases for that!)

Tool used for demos today
To demonstrate SQLite forensics today we will use
Belkasoft Evidence Center 2015 (Ultimate edition):
• Shows deleted records (“freelists”)
• Shows uncommitted records (WAL/journal files)
• Shows remnants in “unallocated”
• Carves SQLite files
• Find SQLite in RAM memory/dumps/hibernation/pagefile
• Has built-in SQLite Viewer
• Allows to review SQLite database in binary mode in built-in HexViewer

SQLite database structure
SQLite is a set of data pages of a fixed size

Freelist
Freelist: a list of free pages in a SQLite database

What is freelist
SQLite.org
“A database file might contain one or more pages that are not in active use.
Unused pages can come about, for example, when information is deleted from the
database. Unused pages are stored on the freelist and are reused when additional
pages are required.”

Is it true for every deleted record?
Freelists are only there if “auto_vacuum” option is not set
• Default: 0 (no autovacuum)
• 1 (Full) requires database rebuilding and implies
performance penalty
• Autovacuum makes database fragmentation worse

www.belkasoft.com
Do you see any records?

www.belkasoft.com
But in fact there are 38!

www.belkasoft.com
But why?!?
Why standard viewer does not show freelist information?
Because freelist is a technical trick to improve performance. It is not supposed
that regular user can see what’s inside.
What are you going to miss:
• Deleted SMSes on iPhone
• Deleted Chrome/Firefox browsing history
• Deleted Skype, WhatsApp, Kik chats
• and hundreds of other apps’ data stored in SQLite

Unallocated space: SQLite’s, not a hard drive’s
• Data pages: “leaf table B-tree”, filled from end to beginning
• Unallocated page space: Free space inside a SQLite page with
unstructured data

Unallocated space: SQLite’s, not a hard drive’s
• Unlike freelists, unallocated space is not formally referenced from
anywhere in the SQLite database
• Can’t determine, which page they originally belonged to
• Carving inside unallocated can give good results

SQLite Write-Ahead-Log file
Initial stage
Data changing
Commit
Since version 3.7.0, SQLite employs “Write Ahead Log”
Process of writing data to a SQLite database:

WAL journal
• Most actual data, which did not yet have a chance to merge to the
main database file
• “Checkpoint” event does actual commit
• Triggered by reaching a certain size
• By default, 1000 pages
• Awful lot for chat or browsing sessions
• What is the difference with Rollback Journal?

So, where data can hide in SQLite?
Deleted SQLite data can reside in one of the following:
• Freelist
• Unallocated space
Uncommitted data can reside in
• WAL journal (or, old records can reside in rollback journal)

Carving SQLite databases
SQLite has excellent header signature, what’s the difficulty?
• Indeed: “SQLite format 3”
• Easy to find, no false positives
• But… where’s the footer?
• There is no footer for SQLite database!
• There is also no data on DB size in DB header

Our solution
The only thing we know is a page size
• Read database header
• Find first page
• Read page size
• Read whole page and write to file
• Read next page header and validate it
• If valid, read page size and continue the process
Drawbacks
• Time-consuming
• Don’t work well with fragmentation

• A big enterprise suspected an employee to disclose an important
know-how
• Employee’s computer was imaged and investigated
• Particularly, an Android backup and Skype account found
• Nothing interesting there, but
• WhatsApp history encrypted
• Skype history empty
The plot

• Skype stores its history in a SQLite database
• Belkasoft Evidence Center has a built-in SQLite Viewer
• Why do you need it?
• Damaged carved SQLite databases
• Freelist analysis
• Unnalocated space analysis
• Unnoticed WAL/journal files
Dealing with empty Skype database

• More than 100 deleted Skype chats recovered by Belkasoft
• Recovered Skype logs along with decrypted WhatApp history solved
the case
Result

• My Skype: journal
• Sample db+journal and SQLite Database Browser
• HexViewer and Skype
• Carving unallocated
• Carving SQLite in Live RAM
Practice

Before we say “goodbye”
Belkasoft article on SQLite:
http://belkasoft.com/sqlite-analysis
Would like to get this presentation?
• Contact me at yug@belkasoft.com!
• Leave me your card
• Add me in LinkedIn (search for Yuri Gubanov)
Free demo version
• Downloadable full 1-month trial for all conference attendees:
belkasoft.com/trial
Request today!

SQLite forensics - Free Lists, unallocated space, carving

More Related Content

Similar to SQLite forensics - Free Lists, unallocated space, carving

Recently uploaded

SQLite forensics - Free Lists, unallocated space, carving