Embedded Files in Documents: Risks, Challenges and Options

Embedded Files: Risks, Challenges
and Options
Tim Allison, Ph.D.
Data Scientist/Relevance Engineer
Artificial Intelligence, Analytics and Innovative
Development Organization (1740)
ITSD
The research was carried out at the NASA (National Aeronautics
and Space Administration) Jet Propulsion Laboratory, California
Institute of Technology under a contract with the Defense
Advanced Research Projects Agency (DARPA) SafeDocs
program. © 2022 California Institute of Technology. Government
sponsorship acknowledged.
Reference herein to any specific commercial product, process,
or service by trade name, trademark, manufacturer, or
otherwise, does not constitute or imply its endorsement by the
United States Government or the Jet Propulsion Laboratory,
California Institute of Technology.

jpl.nasa.gov
About me
• Data scientist (files and search) NASA’s Jet
Propulsion Laboratory, California Institute of
Technology
• Chair/V.P. Apache Tika
• Committer Apache PDFBox, POI, Lucene/Solr,
OpenNLP
• Member Apache Software Foundation
2
© 2022 California Institute of Technology. Government sponsorship acknowledged.
The research was carried out at the NASA (National Aeronautics and Space Administration)
Jet Propulsion Laboratory, California Institute of Technology under a contract with the
Defense Advanced Research Projects Agency (DARPA) SafeDocs program.

jpl.nasa.gov
Overview
• Intended audience – technical, with larger
implications
• This is a work in progress, please help!
9/22/22 3

jpl.nasa.gov
Takeaways
• Think: every file may have embedded files
• Develop budgets, risk assessments and workflows
accordingly
9/22/22 4
I don’t have all the answers!

jpl.nasa.gov
Why should you care?
9/22/22 5
https://twitter.com/WeirdMedieval/status/1532319439684874240

jpl.nasa.gov
Why should you care?
• Sensitive data, accidental data disclosure, not just in
the attachments but in the metadata about the
attachments hosted in the parent document
• Accessibility – how do we make these searchable,
discoverable and available to users
• Every other digital preservation concern…literally
every other digital preservation concern
9/22/22 6

jpl.nasa.gov
Diligence Spectrum
• Where are you? Where do you need to be?
• What do the digipres vendors support? What do
they need to support?
• Cost/benefit, complexity and budgets
9/22/22 7

jpl.nasa.gov
“Hidden data” not covered in this talk
• Encrypted files (without password)
• Hidden sheets/columns/data in spreadsheets
• Track changes/edits and incremental updates
(including deletes!)
• Text out of viewing area in PDFs
• Text in notes components of PPT(x)s
9/22/22 8

jpl.nasa.gov
“Hidden Data” not covered in this talk
• “ActualText” (in PDF)
• Alternative content
• Font too small/too big or same color as background
• Corrupt/missing Unicode mappings/fonts (in PDF)
• Steganography
• Files embedded in cavities whether within the
“parsable” parts or outside of the file-specific
“parsable” parts.
9/22/22 9

jpl.nasa.gov
Intentionally malicious files – not covered in this talk
• Malware – Denial of Service, remote code
execution, etc.
• Crafted parser differentials
9/22/22 10

jpl.nasa.gov
Challenging files also not covered in this talk
• Polyglot and schizophrenic files – files that may be
parsed as more than one file type (e.g. a PDF that is
also a zip file)
• Quines – zip or gz or other package format that
when unpackaged is byte for byte exactly the same
file as the original
9/22/22 11
Refs: Ange Albertini
https://blog.trailofbits.com/2019/11/01/two-new-tools-
that-tame-the-treachery-of-files/

jpl.nasa.gov
Categories of Embedded Files (1 of 2)
• Attachments – something a human or process
added to the file as supplementary information that
is intended to stand on its own/be easily exported
• Images – intended to be rendered as part of the file
• Thumbnail images
• Macros/code – executable code that is intended to
help the functionality (macros in MSOffice and/or
javascript in PDF/HTML, etc.)
9/22/22 12

jpl.nasa.gov
Categories of Embedded Files (2 of 2)
• Metadata files – XMP, anything else?
• Standalone files that help with the rendering/user
experience of a file, e.g. font files, International
Color Consortium (ICC) profiles, subtitle streams
• Files that normally don’t exist outside of files – EMF,
WMF, MSGraph
9/22/22 13

jpl.nasa.gov
Why grep is not sufficient
“Hello World” as stored in a PDF
9/22/22 14
“Brief Overview of the Portable Document Format (PDF) and Some Challenges for Text Extraction”
https://irsg.bcs.org/informer/wp-content/uploads/OverviewOfTextExtractionFromPDFs.pdf
Compressed object as stored in
the file
Uncompressed
! -> H
“ -> e
# -> l
$ -> o

jpl.nasa.gov
Notes on Some Specific Formats
9/22/22 15

jpl.nasa.gov
Hodge podge
• Email can have alternate content (text/html)
• Zips can have free text comments!
• Apple resource forks can have all sorts of things:
See Tyler Thorsted’s #iPres2022 talk if you haven’t!
9/22/22 16

jpl.nasa.gov
HTML
Even HTML can have embedded files!
9/22/22 17

jpl.nasa.gov
MSOffice: OLE2 (doc/ppt/xls)
• Directory + file based format like zip
• Embedded files need to be parsed out of streams;
they may not be stored as separate files even within
the zip-like structure
• Old .doc files may contain a “save history” – full file
paths for where the file has been saved
9/22/22 18

jpl.nasa.gov
MSOffice: OOXML (.docx/pptx/xlsx)
• Zip files
• Embedded files may be stored as standalone within the
zip structure
• Extra “bonus” embedded files not part of the
docx/pptx/xlsx may be stored
• Files may be wrapped in an OLE2 stream
• Full file paths for the source locations for embedded files
may be stored in xml within the zip file
• Excel may store full “last saved” path:
9/22/22 19

jpl.nasa.gov
PDF
• Incremental Updates
• Attachments
9/22/22 20

jpl.nasa.gov
PDF Incremental Updates
9/22/22 21
https://developers.foxit.com/developer-hub/document/incremental-updates/

jpl.nasa.gov
Simply truncate to earlier %%EOFs to get earlier file(s)!
See also tool: pdfresurrect
9/22/22 22
Fun file (starting around p 51):
https://www.usitc.gov/publications/337/pub1859.pdf

jpl.nasa.gov
PDFResurrect – Incremental Updates, 1 million sample
from Common Crawl CC-MAIN-2021-31
9/22/22 23
Updates Percentage
0 77%
1 21.75%*
2 1.02%
3 0.30%
4 0.14%
5 0.07%
6 0.04%
7 0.03%
8 0.02%
9 0.01%
* Many files
created by
MSWord with a
single incremental
update are not
significantly
different!
Max in 1 million
sample:
3,441 incremental
updates

jpl.nasa.gov
PDF Attachments
• Overheard: “We only have to worry about attachments in Portfolio PDFs”
9/22/22 24
No!!! Nearly all* PDFs may contain attachments!

jpl.nasa.gov
Nearly all*
• Files that conform to PDF/A-1 are not allowed to
contain attachments.
9/22/22 25
For all practical purposes, I don’t understand why an ingest
pipeline wouldn’t look for attachments whether or not the PDF
alleges that it is PDF/A-1 or whether or not the PDF actually
passes a conformance check for PDF/A-1.
More simply: assume everything has attachments until proven
otherwise.

jpl.nasa.gov
All PDFs may contain attachments!
• The dataset in the following consists of 8 million
PDFs from one month of Common Crawl CC-MAIN-
2021-31
• ~50k had at least one attachment
• Only 670 files were “Portfolio PDFs”
9/22/22 26

jpl.nasa.gov
Apache Tika – Attached Files, Embedded Depth = 1
9/22/22 27
Mime Count
text/plain; charset=ISO-8859-1 49,288
application/pdf 12,090
text/plain; charset=windows-1252 5,045
audio/mpeg 4,840
application/x-shockwave-flash 4,740
application/xml 4,727
text/html; charset=UTF-8 3,390
image/png 2,099
image/gif 1,656
image/svg+xml 1,476

jpl.nasa.gov
Apache Tika – Attached Files, Embedded Depth > 0
9/22/22 28
Mime Count
text/plain; charset=ISO-8859-1 49,397
image/wmf 14,419
application/pdf 12,564
application/vnd.ms-equation 12,387
image/png 7,126
text/plain; charset=windows-1252 5,127
application/xml 4,959
audio/mpeg 4,886
application/x-shockwave-flash 4,753
text/html; charset=UTF-8 3,391

jpl.nasa.gov
Apache Tika – Maximum Number of Embedded Files
9/22/22 29
Embedded File Counts PDF Count
1 42,054
2 1,416
3 884
4 563
6 423
5 303
8 193
7 176
9 171
16 145
One file has 3,852
embedded files!
Only 670 PDFs are
“Portfolio PDFs”

jpl.nasa.gov
Apache Tika – Embedded File Depths
9/22/22 30
Embedded Depth Count
0 7,931,327
1 98,268
2 37,547
3 3,137
4 177

jpl.nasa.gov
PDFs may include full file paths for various reasons
9/22/22 31

jpl.nasa.gov
TIFF
ExifTool on ~5300 TIFFs
9/22/22 32
• ~2400 have a binary data field (~800 of these are
thumbnails)
• They may contain OCR’d text
• They may contain full file paths
Source: https://corpora.tika.apache.org/base/share/tiffs-out.txt.gz

jpl.nasa.gov
Files that mostly only exist as
embedded files
9/22/22 33

jpl.nasa.gov
9/22/22 34
Container file
format
EMF/WMF Counts
ppt 57,869
doc 36,464
docx 8,246
image/emf 6,798
xls 5,007
pptx 2,970
rtf 2,163
xlsx 1,913
xls (macro) 988
pptx (variant) 381
WMF and EMF – Top 10 containers
• Windows Metafile
(WMF)
• Enhanced Windows
Metafile (EMF)
Habitat within
Apache Tika’s 1
million file regression
test sample

jpl.nasa.gov
WMF – Windows Metafile
“WMF specifies structures for defining
a graphical image. A WMF metafile
contains drawing commands, property
definitions, and graphics objects in a
series of WMF records.”
WMF 17.0 Specification
9/22/22 35
Like PDF, WMF may contain extractable text!

jpl.nasa.gov
EMF – Enhanced Windows Metafile
• “Enhanced metafile format (EMF) is a file format that is
used to store portable representations of graphical
images. EMF metafiles contain sequential records that
are parsed and processed to render the stored image on
any output device.”
EMF 17.0 Specification
9/22/22 36

jpl.nasa.gov
EMF, EMF+, EMFSpool
9/22/22 37
EMF 17.0 Specification
Like PDF, these file types
may contain extractable
text AND attached files!
AND “EMF metafiles define a
mechanism for the encapsulation
of arbitrary vendor-defined data.
The EMR_COMMENT record
(section 2.3.3.1) can contain
arbitrary private data that is
unknown to EMF.”

jpl.nasa.gov
EMF attachments in Apache Tika’s 1 million file
regression sample
9/22/22 38
Attachments in EMFs Counts
image/wmf 86,810
application/pdf 662

jpl.nasa.gov
XMP – eXtensible Metadata Platform
• Habitat: PDF, JPEGs, Photoshop, PNG, TIFF, video,
and more
• Embedded files
• SVG
• HTML
• Thumbnails (e.g. JPEGs)
9/22/22 39

jpl.nasa.gov
XMP – some potential issues
• History – what software packaged modified the file
when, nature of modification
• User information – creator, modified by, file path
links to embedded files or external
resources/references
• OriginalDocumentId, documentId, InstanceId
• Other metadata: title, keywords, subject
• Embedded binary images!
• Custom metadata!
9/22/22 40

jpl.nasa.gov
XFA – XML Forms Architecture
• Habitat: PDF
• XML representation of forms, questions and
answers
• NOTE: If the text extractor/parser is only processing
the PDF content, it will miss content from these
XMLs
9/22/22 41

jpl.nasa.gov
Why file-format specific tools (alone) are not sufficient
• Maximum embedded depth in the 8 million PDF
corpus from Common Crawl is 4
• Maximum embedded depth in the ~1 million Tika
regression corpus is 7
• For full workflow, file-format specific tools must be
able to call each other arbitrarily
9/22/22 42

jpl.nasa.gov
Conclusion
• Treat every file as if it has attachments until proven
otherwise
• Develop budgets, risk assessments and workflows
accordingly
9/22/22 43

jpl.nasa.gov
Extras
9/22/22 44

jpl.nasa.gov
• The Portfolio PDF and the PPT with attachment are
in the iPres2022 bakeoff corpus:
https://drive.google.com/drive/folders/1ACktqBv_Yo
oW9DLInBM5ad_I0yHJRoLU
• Step by step commandlines with example data for
this talk:
https://cwiki.apache.org/confluence/display/TIKA/Op
en+Preservation+Foundation+Talk+--
+21+September+2022
9/22/22 45

jpl.nasa.gov
Some tools
• Apache Tika
• ExifTool
• pdfresurrect
• Poppler: pdfinfo, pdfimages, pdfdetach
• Didier Stevens (forensics): oledump.py, pdfid.py, pdftool.py
• Philippe Lagadec (forensics): oletools
• Great blog from @bitsgalore on opensource PDF tools:
https://www.bitsgalore.org/2021/09/06/pdf-processing-and-
analysis-with-open-source-tools
9/22/22 46

jpl.nasa.gov
Commandlines
• JSON embedded file output
• java –jar tika-app-2.y.z.jar –J -t
digitally_signed_3D_Portfolio(1).pdf
• JSON embedded file output batch
• java –jar tika-app-2.y.z.jar –J –t –i
<input_dir> -o <output_dir>
• Dump first level attachments
• java –jar tika-app-2.y.z.jar –z
digitally_signed_3D_Portfolio(1).pdf
9/22/22 47
NOTE: -z only extracts first level. If you want full recursive, please
open an issue: https://issues.apache.org/jira/projects/TIKA

jpl.nasa.gov
XMP
• Extract the literal XMP from file.pdf into
xmp.xmp
• exiftool -a -o xmp.xmp file.pdf
9/22/22 48

jpl.nasa.gov
Commandlines
• Dump contents of OLE2 to local directory
• java -cp tika-app-2.Y.Z.jar
org.apache.poi.poifs.dev.POIFSDump
testWORD_1img.doc
• List contents of OLE2 files
• java -cp ~/Intellij/tika-main/tika-
app/target/tika-app-2.Y.Z.jar
org.apache.poi.poifs.dev.POIFSViewer
261779.ppt
9/22/22 49
See also Didier Stevens’ oledump.py

Embedded Files in Documents: Risks, Challenges and Options

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Embedded Files in Documents: Risks, Challenges and Options

Similar to Embedded Files in Documents: Risks, Challenges and Options (20)

Recently uploaded

Recently uploaded (20)

Embedded Files in Documents: Risks, Challenges and Options