The document discusses SQLite, a widely used lightweight database format. It notes that SQLite databases are commonly used in smartphones and applications to store structured data. The document outlines challenges in recovering deleted data from SQLite databases and introduces an advanced SQLite recovery tool being developed by viaForensics. It provides information on SQLite database structure, including pages, B-trees, records, and data types. It also discusses viewing and analyzing SQLite databases using command line and graphical tools.
SQLite is a widely popular database format that is used extensively pretty much everywhere. Both iOS and Android employ SQLite as a storage format of choice, with built-in and third-party applications relying on SQLite to keep their data. A wide range of desktop and mobile Web browsers (Chrome, Firefox) and instant messaging applications use SQLite, which includes newer versions of Skype (the older versions don’t work anyway without a forced upgrade), WhatsApp, iMessages, and many other messengers.
Forensic analysis of SQLite databases is often concluded by simply opening a database file in one or another database viewer. One common drawback of using a free or commercially available database viewer for examining SQLite databases is the inherent inability of such viewers to access and display recently deleted (erased) as well as recently added (but not yet committed) records. Here we examine the forensic implications of three features of the SQLite database engine: Free Lists, Write Ahead Log and Unallocated Space.
More information: http://belkasoft.com/sqlite-analysis
In this session we will discuss, various methods to analyse possible criminal actions/accidents and pin point it to a specific person/group of persons and time/time frame.
We will discuss the goals of a forensic investigation, define breaches, types of breaches and how to verify them. We will also learn about various database file formats, methodology of forensic investigation, collection and analysis of artifacts. We will take a look at native SQL methods.
We will also cover what artifacts to collect and why.
We will also cover a couple of third party tools available in the market. Understand why it is not always easy to use these tools.
Can we retrace the DML/DDL statements and possibly undo the harm?
We will also learn how to preserve the evidence, how to setup HoneyPots.
We will also look at the Initial and Advanced Response Toolkit. How to use SQL Binaries to determine hack.
SQLite is a widely popular database format that is used extensively pretty much everywhere. Both iOS and Android employ SQLite as a storage format of choice, with built-in and third-party applications relying on SQLite to keep their data. A wide range of desktop and mobile Web browsers (Chrome, Firefox) and instant messaging applications use SQLite, which includes newer versions of Skype (the older versions don’t work anyway without a forced upgrade), WhatsApp, iMessages, and many other messengers.
Forensic analysis of SQLite databases is often concluded by simply opening a database file in one or another database viewer. One common drawback of using a free or commercially available database viewer for examining SQLite databases is the inherent inability of such viewers to access and display recently deleted (erased) as well as recently added (but not yet committed) records. Here we examine the forensic implications of three features of the SQLite database engine: Free Lists, Write Ahead Log and Unallocated Space.
More information: http://belkasoft.com/sqlite-analysis
In this session we will discuss, various methods to analyse possible criminal actions/accidents and pin point it to a specific person/group of persons and time/time frame.
We will discuss the goals of a forensic investigation, define breaches, types of breaches and how to verify them. We will also learn about various database file formats, methodology of forensic investigation, collection and analysis of artifacts. We will take a look at native SQL methods.
We will also cover what artifacts to collect and why.
We will also cover a couple of third party tools available in the market. Understand why it is not always easy to use these tools.
Can we retrace the DML/DDL statements and possibly undo the harm?
We will also learn how to preserve the evidence, how to setup HoneyPots.
We will also look at the Initial and Advanced Response Toolkit. How to use SQL Binaries to determine hack.
UNIT : -(6)
CONNECTING DATABASE WITH ADO.NET
Content:
•ADO.NET Architecture
•Data provider and its core object
•DataSet class
•Data Binding
•SQL Data Source
SQLite Database in Android used to store persistent data. If you want to store some data into local storage then SQLite Database is the most common storage option. It is lightweight database that comes with Android OS.
SQL Training Institute in Ambala ! Batra Computer Centrejatin batra
Batra Computer Centre is An ISO certified 9001:2008 training Centre in Ambala.
We Provide SQL Training in Ambala. BATRA COMPUTER CENTRE provides best training in C, C++, S.E.O, Web Designing, Web Development and So many other courses are available.
A presentation by Max and I for Aotearoa Instiutional Repositories Community Day (AKA NZIRC) 30 September 2021.
Discusses challenges of moving off a very old version of DSpace to a mixed-model with a modern DSpace and figshare. The focus is on DSpace and metadata
Slides for a talk at NDF 2017 by Stuart Yeates and Max Sullivan. See https://web.archive.org/web/20180213055412/http://www.ndf.org.nz/2017-workshops/#mets METS is Metadata for Encoding and Transmission Standard, see https://www.loc.gov/standards/mets/
Materials (sample METS files) are at https://figshare.com/articles/METS_metadata_for_complete_beginners_workshop_samples_/5606917
UNIT : -(6)
CONNECTING DATABASE WITH ADO.NET
Content:
•ADO.NET Architecture
•Data provider and its core object
•DataSet class
•Data Binding
•SQL Data Source
SQLite Database in Android used to store persistent data. If you want to store some data into local storage then SQLite Database is the most common storage option. It is lightweight database that comes with Android OS.
SQL Training Institute in Ambala ! Batra Computer Centrejatin batra
Batra Computer Centre is An ISO certified 9001:2008 training Centre in Ambala.
We Provide SQL Training in Ambala. BATRA COMPUTER CENTRE provides best training in C, C++, S.E.O, Web Designing, Web Development and So many other courses are available.
A presentation by Max and I for Aotearoa Instiutional Repositories Community Day (AKA NZIRC) 30 September 2021.
Discusses challenges of moving off a very old version of DSpace to a mixed-model with a modern DSpace and figshare. The focus is on DSpace and metadata
Slides for a talk at NDF 2017 by Stuart Yeates and Max Sullivan. See https://web.archive.org/web/20180213055412/http://www.ndf.org.nz/2017-workshops/#mets METS is Metadata for Encoding and Transmission Standard, see https://www.loc.gov/standards/mets/
Materials (sample METS files) are at https://figshare.com/articles/METS_metadata_for_complete_beginners_workshop_samples_/5606917
SQL Server 2014 Memory Optimised Tables - AdvancedTony Rogerson
Hekaton is large piece of kit, this session will focus on the internals of how in-memory tables and native stored procedures work and interact – Database structure: use of File Stream, backup/restore considerations in HA and DR as well as Database Durability, in-memory table make up: hash and range indexes, row chains, Multi-Version Concurrency Control (MVCC). Design considerations and gottcha’s to watch out for.
The session will be demo led.
Note: the session will assume the basics of Hekaton are known, so it is recommended you attend the Basics session.
Best Practices and Performance Tuning of U-SQL in Azure Data Lake (SQL Konfer...Michael Rys
When processing TB and PB of data, running your Big Data queries at scale and having them perform at peak is essential. In this session, we show you some state-of-the art tools on how to analyze U-SQL job performances and we discuss in-depth best practices on designing your data layout both for files and tables and writing performing and scalable queries using U-SQL. You will learn how to analyze performance and scale bottlenecks and will learn several tips on how to make your big data processing scripts both faster and scale better.
SQL Server 2014 Extreme Transaction Processing (Hekaton) - BasicsTony Rogerson
Far from Hekaton being an extension of DBCC PINTABLE, it’s a huge new piece of functionality that can significantly improve the scalability of various data based scenarios – not just OLTP but also ETL and real-time BI.
This session will introduce Hekaton features, how and when to use it; it will be demo led giving Hekaton end-to-end: enabling it, create tables, index design, query considerations, native stored procedures, durability [or not], introduce methods of identifying what to put in memory or not.
by Dhanraj Pondicherry, Sr. Solutions Architecture Manager, AWS
Analyzing big data quickly and efficiently requires a data warehouse optimized to handle and scale for large datasets. Amazon Redshift is a fast, petabyte-scale data warehouse that makes it simple and cost-effective to analyze big data for a fraction of the cost of traditional data warehouses. In this session, we take an in-depth look at data warehousing with Amazon Redshift for big data analytics. We cover best practices to take advantage of Amazon Redshift's columnar technology and parallel processing capabilities to deliver high throughput and query performance. We also discuss how to design optimal schemas, load data efficiently, and use work load management. Level: 300
Waiting too long for Excel's VLOOKUP? Use SQLite for simple data analysis!Amanda Lam
** This workshop was conducted in the Hong Kong Open Source Conference 2017 **
Excel formulas can be quite slow when you're processing data files with thousands of rows. It's also especially difficult to maintain the files when you have some messy mixture of VLOOKUPs, Pivot Tables, Macros and VBAs.
In this interactive workshop targeted for non-coders, we will make use of SQLite, a very lightweight and portable open source database library, to perform some simple and repeatable data analysis on large datasets that are publicly available. We will also explore what you can further do with the data by using some powerful extensions of SQLite.
While SQLite may not totally replace Excel in many ways, after the workshop you will find that it can improve your work efficiency and make your life much easier in so many use cases!
Who should attend this workshop?
- If you're frustrated with the slow performance of Excel formulas when dealing with large datasets in your daily work
- No coding experience is required
Oracle 12c New Features For Better PerformanceZohar Elkayam
Oracle 12cR1 and 12cR2 came with some great features for better performance and scaling. In this session we will talk about some of the new features that might improve performance greatly: Optimizer changes, adaptive plans improvements, changes to statistics gathering and we'll get to know Oracle 12cR2 new sharding option
On the agenda:
- Oracle Database In Memory (Column Store)
- Oracle Sharding (12.2.0.1)
- Optimizer changes in 12c
- Statistics changes in 12c.
Presented first at ilOUG - Israel Oracle User Group meetup in February 2017.
[including promised hidden slide.. :) ]
Adding real time reporting to your database oracle db in memoryZohar Elkayam
This is a presentation I gave in the UKOUG Scotland user conference in June 2015. This is presentation describe a proof of concept we did for Clarizen on the Oracle 12c Database In Memory Option.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfMalak Abu Hammad
Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers:
* What is Vector Search?
* Importance and benefits of vector search
* Practical use cases across various industries
* Step-by-step implementation guide
* Live demos with code snippets
* Enhancing LLM capabilities with vector search
* Best practices and optimization strategies
Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications.
#MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology
How to Get CNIC Information System with Paksim Ga.pptxdanishmna97
Pakdata Cf is a groundbreaking system designed to streamline and facilitate access to CNIC information. This innovative platform leverages advanced technology to provide users with efficient and secure access to their CNIC details.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Dr. Sean Tan, Head of Data Science, Changi Airport Group
Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Maruthi Prithivirajan, Head of ASEAN & IN Solution Architecture, Neo4j
Get an inside look at the latest Neo4j innovations that enable relationship-driven intelligence at scale. Learn more about the newest cloud integrations and product enhancements that make Neo4j an essential choice for developers building apps with interconnected data and generative AI.
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
Removing Uninteresting Bytes in Software FuzzingAftab Hussain
Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process.
In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds.
- These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.
Building RAG with self-deployed Milvus vector database and Snowpark Container...Zilliz
This talk will give hands-on advice on building RAG applications with an open-source Milvus database deployed as a docker container. We will also introduce the integration of Milvus with Snowpark Container Services.
Pushing the limits of ePRTC: 100ns holdover for 100 daysAdtran
At WSTS 2024, Alon Stern explored the topic of parametric holdover and explained how recent research findings can be implemented in real-world PNT networks to achieve 100 nanoseconds of accuracy for up to 100 days.
Climate Impact of Software Testing at Nordic Testing DaysKari Kakkonen
My slides at Nordic Testing Days 6.6.2024
Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
2. About viaForensics
viaForensics is an innovative digital forensics
and security company providing expert services
to:
• Law Enforcement
• Government Agencies
• Corporations
• Attorneys/Individuals
3. What’s the problem?
• We want to recover as much data from devices
as possible
• People delete data, mostly the data we want!
• SQLite is a very popular data storage format
• Currently no advanced SQLite recovery tool on
the market (but stay tuned)
4. What is SQLite?
• SQLite is a widely used, lightweight database
contained in a single cross-platform file used by
developers for structured data storage
• Used in most smart phones (iPhone, Android,
Symbian, webOS)
• Used in major operating systems and
applications (Apple OS X, Google Chrome and
Chrome OS, Firefox)
5. Why do developers need structured data storage?
• Applications need to store and retrieve data
• In past and today, developers created their own
file formats
• But why reinvent the wheel for basic data
storage?
• SQLite is free, open, high quality and takes care
of the messy details
6. Core SQLite characteristics (from their FAQ)
• Transactions are atomic, consistent, isolated, and durable (ACID)
even after system crashes and power failures.
• Zero-configuration - no setup or administration needed.
• A complete database is stored in a single cross-platform disk file.
• Small code footprint: 190KiB - 325KiB fully configured
• Cross-platform and easy to port to unsupported systems.
• Sources are in the public domain. Use for any purpose.
• Standalone command-line interface (CLI) client
7. SQL = Structured Query Language
• SQL is the language used to interact with many
databases, including SQLite
• Basic functions: Create, Read, Update and
Deleted (CRUD)
• Transactions: Start a change and it either
completes in entirety (commit) or not at all
(rollback)
• Very powerful, many variations
8. SQL – basic commands
• SELECT – queries data from tables or tables
– SELECT rowid, address, date, text FROM message;
• INSERT INTO – adds data row to table
– INSERT INTO message VALUES (NULL, ‘3128781100’, 1282844546, ‘text message’);
• UPDATE – updates data rows in tables
– UPDATE message SET date=1282846291 WHERE rowid=4;
• DELETE – deletes data rows in tables
– DELETE FROM message WHERE rowid=4;
• Many good tutorials online
9. Viewing a SQLite database – command line
• Command line apps
– sqlite3 for full SQLite functions
– sqlite_analyzer for db metadata
• Linux/Mac/Windows versions
• Represents latest version
• Full source code and documentation
• http://www.sqlite.org/download.html
10. Example sqlite3 session
Run sqlite3 on database file
ahoog@linux-wks-001:~/sqlite$ ./sqlite3 iPhone-3G-313-sms.db
SQLite version 3.7.4
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite>
List tables in database
sqlite> .tables
_SqliteDatabaseProperties msg_group
group_member msg_pieces
message
Examine schema (structure) of message database
sqlite> .schema message
CREATE TABLE message (ROWID INTEGER PRIMARY KEY AUTOINCREMENT, address TEXT, date
INTEGER, text TEXT, flags INTEGER, replace INTEGER, svc_center TEXT, group_id INTEGER,
association_id INTEGER, height INTEGER, UIFlags INTEGER, version INTEGER, subject TEXT,
country TEXT, headers BLOB, recipients BLOB, read INTEGER);
11. Example sqlite3 session - continued
View record “4” in 2 formats
sqlite> .headers on
sqlite> SELECT * FROM message WHERE ROWID = 4;
ROWID|address|date|text|flags|replace|svc_center|group_id|association_id|height|UIFlags
|version|subject|country|headers|recipients|read
4|(312) 898-4070|1282844546|Sure is a nice day out |3|0||3|1282844546|0|4|0||us|||1
sqlite> .mode line
sqlite> SELECT * FROM message WHERE ROWID = 4;
ROWID = 4
address = (312) 898-4070
date = 1282844546
text = Sure is a nice day out
flags = 3
replace = 0
svc_center =
group_id = 3
association_id = 1282844546
height = 0
UIFlags = 4
version = 0
subject =
country = us
headers =
recipients =
read = 1
12. sqlite3_analyzer – very useful in forensic analysis
ahoog@linux-wks-001:~/sqlite$ ./sqlite3_analyzer iPhone-3G-313-sms.db
/** Disk-Space Utilization Report For iPhone-3G-313-sms.db
Page size in bytes.................... 2048
Pages in the whole file (measured).... 14
Pages in the whole file (calculated).. 14
Pages that store data................. 13 92.9%
Pages on the freelist (per header).... 0 0.0%
Pages on the freelist (calculated).... 0 0.0%
Pages of auto-vacuum overhead......... 1 7.1%
Number of tables in the database...... 7
Number of indices..................... 4
Number of named indices............... 3
Automatically generated indices....... 1
Size of the file in bytes............. 28672
Bytes of user payload stored.......... 1833 6.4%
*** Page counts for all tables with their indices ********************
MESSAGE............................... 3 21.4%
SQLITE_MASTER......................... 3 21.4%
_SQLITEDATABASEPROPERTIES............. 2 14.3%
MSG_PIECES............................ 2 14.3%
<snip>
13. Viewing a SQLite database – SQLite Database Browser
• Freeware, public domain, open source visual tool used
to create, design and edit database files compatible with
SQLite
• Windows/Linux/Mac
• Support SQLite 3.x
• Last updated 12/2009
• http://sqlitebrowser.sourceforge.net/
• Many other (free) options listed at:
http://www.sqlite.org/cvstrac/wiki?p=ManagementTools
16. SQLite – database header format
• The first 100 bytes of the database file comprise the
database file header.
• First 5 of 22 fields
Offset Size Description
0 16 The header string: "SQLite format 3000"
16 2 The database page size in bytes. Must be a power of two
between 512 and 32768 inclusive, or the value 1 representing a
page size of 65536.
18 1 File format write version. 1 for legacy; 2 for WAL.
19 1 File format read version. 1 for legacy; 2 for WAL.
20 1 Bytes of unused "reserved" space at the end of each page.
Usually 0.
17. SQLite – Organized in Pages
• Database consists of one or more pages, logical units which store
data
• Pages are numbered beginning with 1
• A page is one of the following:
Page type Description
B-Tree page B-Tree pages are part of the tree structures used to store
database tables and indexes.
Overflow page Overflow pages are used by particularly large database
records that do not fit on a single B-Tree page.
Free page Free pages are pages within the database file that are not
being used to store meaningful data. (or so they think!)
Pointer-map page Part of auto-vacuum system
Locking page Tracks when database rows are locked for updating
18. B+tree and B-Tree formats – on-disk data structure
• Data structure which represents sorted data in a way
that allows for efficient insertion, retrieval and removal of
records
• Optimized for storage devices (vs. in memory) by
minimizing the number of disk accesses.
• In a B+tree, all data is stored in the leaves of the tree
instead of in both the leaves and the intermediate branch
nodes.
• A single B-Tree structure is stored using one or more
database pages. Each page contains a single B-Tree
node.
20. SQLite storage classes and data types
• Only 5 storage classes/data types :
1. NULL: The value is a NULL value.
2. INTEGER: The value is a signed integer, stored in 1, 2, 3, 4, 6, or 8 bytes
depending on the magnitude of the value.
3. REAL: The value is a floating point value, stored as an 8-byte IEEE floating
point number.
4. TEXT: The value is a text string, stored using the database encoding (UTF-
8, UTF-16BE or UTF-16LE).
5. BLOB: The value is a blob of data, stored exactly as it was input. Often
used to store binary data
21. SQLite storage classes – on disk example
• 5 storage classes in hex on disk:
• NULL: 0x00
• INTEGER (4-byte): 0x4c76a782 = 1282844546
• REAL: 0x41B1EC2EC004D9D7 = 300691136.018949
– http://babbage.cs.qc.edu/IEEE-754/64bit.html
• TEXT (ASCII): 0x53757265206973 = Sure is
• BLOB: hard to represent binary here…see Text
22. Variable Integers – saving space, adding confusion
• A variable-length integer or "varint" uses less space for small positive
values.
• Used in SQLite metadata (row headers, b-tree indexes, etc.)
• A varint is between 1 and 9 bytes in length.
• The varint consists of either zero or more byte which have the high-order bit
set followed by a single byte with the high-order bit clear, or nine bytes,
whichever is shorter. The lower seven bits of each of the first eight bytes
and all 8 bits of the ninth byte are used to reconstruct the 64-bit twos-
complement integer.
• Varints are big-endian: bits taken from the earlier byte of the varint are the
more significant and bits taken from the later bytes.
• http://www.sqlite.org/fileformat.html#varint_format
• Clear? How about an example ->
23. Variable Integers – example
• Let’s say you find the following hex varint: 0x8CA06F
– Examine each bit, if > 0x80 then not the last byte
– So, we have 3 bytes: 0x8C 0xA0 0x6F (since 0x6F < 0x80 it’s
the last byte). Here’s how to convert:
* MSB: Most significant bit (left most bit)
Original Bytes 0x8C 0xA0 0x6F
Convert to binary 1000 1100 1010 0000 0110 1111
Remove MSB* 000 1100 010 0000 110 1111
Concatenate 000110001000001101111
In hex/decimal Hex: 0x03106F Decimal: 200,815
24. Freelist / Free page list
• When information is deleted from the database,
pages containing that data are not in active use.
• Unused pages are stored on the freelist and are
reused when additional pages are required.
• Forensic value: “Freelist leaf pages contain no
information. SQLite avoids reading or writing
freelist leaf pages in order to reduce disk I/O.”
25. Rollback journal
• Created when a database is going to be updated
• The original unmodified content of that page is written
into the rollback journal.
• The rollback journal is always located in the same
directory as the database file and has the same name as
the database file but with the string "-journal" appended
• Excellent source of forensic data if recoverable
• Recoverable on many systems though some are now
writing to tmpfs/RAM disks
26. Write Ahead Log (WAL)
• New technique just introduced in 3.7.0
• Generally faster and disk I/O is more sequential (which helps us in
advanced recovery)
• All changes to the database are recorded by writing frames into the WAL.
• Transactions commit when a frame is written that contains a commit marker.
• A single WAL can and usually does record multiple transactions.
• Periodically, the content of the WAL is transferred back into the database file
in an operation called a "checkpoint".
• Forensic value: recovery of WAL files
27. Record Format
• A record contains a header and a body, in that order. The
header:
– begins with a single varint which determines the total number of
bytes in the header. The varint value is the size of the header in
bytes including the size varint itself.
– Following the size varint are one or more additional varints, one
per column. These additional varints are called "serial type"
numbers and determine the datatype of each column
– After the final header varint, the record data immediately follows
– The 2-bytes prior to the start of the header correspond to the
auto-increment integer assigned by the system (also a varint)
28. Record Format – visual representation
• http://www.sqlite.org/fileformat.html#record_format
29. Record Format
Header Value Data type and size
0 An SQL NULL value (type SQLITE_NULL). This value consumes zero bytes of space in the record's data area.
1 An SQL integer value (type SQLITE_INTEGER), stored as a big-endian 1-byte signed integer.
2 An SQL integer value (type SQLITE_INTEGER), stored as a big-endian 2-byte signed integer.
3 An SQL integer value (type SQLITE_INTEGER), stored as a big-endian 3-byte signed integer.
4 An SQL integer value (type SQLITE_INTEGER), stored as a big-endian 4-byte signed integer.
5 An SQL integer value (type SQLITE_INTEGER), stored as a big-endian 6-byte signed integer.
6 An SQL integer value (type SQLITE_INTEGER), stored as an big-endian 8-byte signed integer.
7 An SQL real value (type SQLITE_FLOAT), stored as an 8-byte IEEE floating point value.
8 The literal SQL integer 0 (type SQLITE_INTEGER). The value consumes zero bytes of space in the record's data
area. Values of this type are only present in databases with a schema file format (the 32-bit integer at byte
offset 44 of the database header) value of 4 or greater. (iOS4 uses this)
9 The literal SQL integer 1 (type SQLITE_INTEGER). The value consumes zero bytes of space in the record's data
area. Values of this type are only present in databases with a schema file format (the 32-bit integer at byte
offset 44 of the database header) value of 4 or greater. (iOS4 uses this)
10,11 Not used. Reserved for expansion.
bytes * 2 + 12 Even values greater than or equal to 12 are used to signify a blob of data (type SQLITE_BLOB) (n-12)/2 bytes
in length, where n is the integer value stored in the record header.
bytes * 2 + 13 Odd values greater than 12 are used to signify a string (type SQLITE_TEXT) (n-13)/2 bytes in length, where n
is the integer value stored in the record header.
30. Recovery from allocated SQLite with strings
ahoog@linux-wks-001:~/sqlite$ strings iPhone-3G-313-sms.db | less
<snip>
msg_group
(314) 267-6611us
(920) 277-1869us
(312) 898-4070us
(312) 401-1679us
(414) 331-5030us
Piece of cake! Can't wait to try em out on Sunday
text/plain
2text_0002.txt
image/jpeg
1IMG_6807.jpg?
Check out mccalister
text/plain
2text_0002.txt
image/jpeg
1IMG_6807.jpg
<snip>
32. Carving SQLite files – OS specific findings
• iOS
– Good recovery of both allocated and “latent”
SQLite files
• Android
– Excellent recovery but high repetition due to
log-structured file system repeating SQLite
header
• Other common file systems
– Good recovery form typical magnetic media
device running FAT, FAT32, NTFS, HFS, etc.
33. SQLite in Hex (really the only way to look at it)
0002270: 0000 0000 0000 0000 004d 0d12 0029 0445 .........M...).E
0002280: 0101 0001 0401 0101 0011 0000 0128 3331 .............(31
0002290: 3229 2038 3938 2d34 3037 304c 77d8 a257 2) 898-4070Lw..W
00022a0: 696c 6c20 796f 7520 676f 2067 6574 206d ill you go get m
00022b0: 6520 6120 636f 6666 6565 3f03 0003 4c77 e a coffee?...Lw
00022c0: d8a2 0000 0075 7301 3f0c 1200 2904 2901 .....us.?...).).
Name Type Header Converted Body Value / notes
Rowid – actual Varint 0x0d 13 So rowid = 13
Header Size Varint 0x12 18 Length of header is 18 bytes (header size + 17 rows)
Rowid NULL 0x00 0 NULL tells SQLite on insert to determine next auto increment
Address Text 0x29 (41 -13)/2 = 14 (312) 898-4070 [14 chars - covert 0x29 to decimal, calc size]
Date Integer 0x04 4-byte integer 0x4c77d8a2 in decimal is 1282922658 [recognize number format?]
Text Text 0x45 (69 -13)/2 = 28 Will you go get me a coffee?
Flags Integer 0x01 1-byte integer 0x03 = 3
Replace Integer 0x01 1-byte integer 0x00 = 0
Svc_center Text 0x00 NULL No value, not represented in data at all
Group_id Integer 0x01 1-byte integer 0x03 = 3
Association_id Integer 0x04 4-byte integer 0x4c77d8a2 in decimal is 1282922658 [recognize number format?]
Height Integer 0x01 1-byte integer 0x00 = 0
UIFlags Integer 0x01 1-byte integer 0x00 = 0
Version Integer 0x01 1-byte integer 0x00 = 0
Subject Text 0x00 NULL No value, not represented in data at all
Country Text 0x11 (17 – 13)/2 = 2 us
Headers Blob 0x00 NULL No value, not represented in data at all
Recipients Blog 0x00 NULL No value, not represented in data at all
Read Integer 0x01 1-byte integer 1 [Last data byte]
34. Advanced Technique
• Use well defined SQLite structure to develop a program
to recover SQLite rows
• Row header and data values “decay” over time due to
– Being (partially) re-allocated
– Fragmentation
– Compensated for this with simple probability engine which
determined likelihood sequence of bytes represented header row
we are interested in
• Underlying file system can have great impact, from FAT,
HFSplus (iPhone) and YAFFS2 (Android)
• Look for journal files and WAL data too
35. Contact Us
Andrew Hoog, CIO
ahoog@viaforensics.com
http://viaforensics.com
1000 Lake St, Suite 203
Oak Park, IL 60301
Tel: 312-878-1100 | Fax: 312-268-7281