This presentation is to provide a technical understanding of the Microsoft Extended
File Allocation Table (exFAT) for fore...
One of the is to provide the background and history of the file system, and the
relationship to the other FAT versions in ...
3
D4CS stands for Digital Forensics & Cyber Security
FCM 760 Fall 2009
4
This presentation was given 5 times within the 14 months since the SANS paper
was published.
It will now be given twice in...
Both books were published in 2012, other books have been published, and may
mention exFAT, but these two send the reader t...
Brian Carrier’s book is considered by some as the gold standard and bible on
explaining file systems.
It has become a litt...
This is the Sleuth Kit Wiki, on this page it is noted that exFAT support was added to
TSK, and one of the docs used as a g...
Microsoft published a patent that included the exFAT 1.00 specification.
This presentation and the paper attempt to stick ...
10
http://www.webopedia.com/TERM/O/OSR_2.html
OEM Service Release 2 – Windows 95B
According to Wikipedia, there was a 8 bit F...
exFAT is specifically designed for Removable media, but can be used for fixed
media as well.
NTFS is not recommended for r...
SD = Standard Definition
HD = High Definition
Quad HD is 4 times Full HD (Double wide, Double Long)
Amount of data being r...
Source: https://www.sdcard.org/consumers/cards
14
15
•You need to be able to locate the evidence, just in general
•This includes re-assembly when a file is fragmented
•Also in...
Don’t be a money pressing a button, need to know what is under the covers
17
If the OS can’t recognize the file system, then it thinks the media is not formatted.
When this slide was built, it was 20...
Any evidence with exFAT would probably be pushed aside with the lack of tools,
documentation and expertise to process it.
...
Linux and Open Source is used a lot for examinations
Commercial tools are lacking, but picking up
Little documentation or ...
http://www.cftt.nist.gov/presentations/AAFS-2013-Lyle-DFR.pptx
AAFS = American Academy of Forensic Sciences
http://www.cft...
Encase: http://www.cyberfetch.org/sites/default/files/EnCase Forensic v6.18.0.59
Test Report.pdf
FTK: http://www.cyberfetc...
23
24
When we use the term “Mega”, is it 1,000,000 (1006) 0r do we mean “Mega”
1,048,576 (220) ?
25
A quick note on exponents, since we will get our hands dirty with math
Some simple numbers should be like learning the tim...
In some cases you might see ZB or ZIB, technically they are really different, but are
close.
So when we say 1 kb of disk, ...
Just another slide
It is suggested that in English, the first syllable of the name of the binary-multiple
prefix should be...
Being off by 15% when talking about an "exabyte" means being off by about bytes,
or 150 petabytes.
29
exFAT uses 16 bit Unicode strings
This is the terminology as used in the specifications leaked in the patent
When reading ...
It is important to note that Pentium processers use the little-endian format, so
numbers stored in the file system are sto...
This is how Microsoft does Math, and then everyone uses these numbers not
knowing the full context
232 sectors * 29 bytes ...
http://support.microsoft.com/kb/955704
Volume size of 64ZB is architecturally incorrect. Currently it cannot exceed 128PiB...
http://en.wikipedia.org/wiki/Windows_CE
Microsoft Windows CE (now officially known as Windows Embedded Compact
and previou...
35
You never really see another sector size other than 512 bytes, but everyone just
assumes that it is only 512 (in earlier v...
This new format, called Advanced format, provides via hardware 4K physical sector
size.
An OS/FS can either use emulation ...
This command (help format) was issued on a Windows 7 32-bit system.
This snippet is for the allocation unit size.
Most “su...
Microsoft in the KB for Windows XP support indicated a capacity to 64ZiB and a file
size maximum to 64ZiB.
In reality, the...
With TexFAT there will be 2 FATS and 2 BITMAPS, with exFAT 1.0 – which does not
have TexFAT (Transactional FAT) support, t...
Any FS is limited, even FAT32 and NTFS.
This is Windows only, we are not talking GUID Partition Table (GPT)
Although a MBR...
Windows would not format FAT32 beyond 32GB, it required using a FAT32 format
on a different OS
Some Windows utilities did ...
3rd party file utilities may provide conversion to and from exFAT, but no Convert
command, and current convert command doe...
There are discussions of creation of exFAT on a Vista or Windows 7 machine that
can’t be seen on Vista. This is usually a ...
45
The SDXC media will not be backward compatiblee-solutuions/volkswagon for in
vehicle entertainment systems
DCF 2.0 – Desig...
This is just a selection of some dates, not exhaustive.
Volkswagon and Audi licensed Tuxera drivers to get exFAT support.
...
Tuxera
http://www.tuxera.com/products/tuxera-exfat-embedded/
Provides exFAT & NTFS to be integrated in embedded systems.
T...
The Linux community is very hostile to Microsoft, complaining that a Microsoft
standard was forced on them and Microsoft e...
50
51
52
I use a CPAP machine, and it has a SD card to record my sleep patterns
These are the uses, many of these devices take USB,...
SANDISK ships high capacity CF cards pre-formatted as exFAT
PNY 128GB Turbo USB flash
DigiStore SSD
Some Kingston Memory
E...
New Devices may accept SDXC, but older devices might not.
Mentioning memory cards such as camera cards is important becaus...
With Sony adopting the XC memory stick to exFAT, plus the SD market, is almost
90% of the market today.
http://anythingbut...
July 2012
57
The 137GiB comes from LBA-28 addressing.
228 = 268,435,456
228 * 29 = 128GiB = 137GB = 137,438,953,472
144PB comes from LB...
SD – up to 2GB – FAT
SDHC – 2GB to 32GB FAT32
SDXC – 32GB to 2TB exFAT
These are marketing hype and represent maximums
htt...
Nand gate wear and tear, less write, longer the memory may last
Although the SD association states that the official, stan...
Write Endurance (Program Erase Cycles)
Limited, maybe up to a million, writes
Writes require the storage area to be erased...
Why a 2TB limit when the CF has a 144PB limit?
Looks like SDXC uses a MBR partition to separate the protected area from th...
63
Currently use exFAT 1.00, but if a later version of exFAT is in use, it will check the
version # and not mount the FS unle...
65
4 Regions defined on the volume
The FAT tables reside outside the cluster heap
Everything except the data region is measur...
FAT and Cluster heap have their own offsets, which allow alignment, if needed to
force these region on a designated bounda...
Details follow in the next slides
A mirror of the VBR follows, and is a backup VBR
In case the first gets corrupted
FAT32 ...
When you take the volume length (64-bit) * 4K sector, that is 64+12 = 276.
270 = 1 ZiB, 26 = 64, thus based on this value,...
If there was no restriction, then the size of a cluster could be 4255
70
If the sector size is > 512 bytes, all space on the first sector of the VBR (Main Boot
Sector) is not used (Only the first...
Unlike the first sector, the other 8 boot sectors can use the entire sector and the
signature marker is moved to the last ...
If a virus modified the boot record, and doesn’t fix the checksum, there should be a
mount failure
73
Repeats over and over again, 4 bytes = 32 bit checksum
Can be used to determine if the VBR was modified
3 bytes in the VBR...
The BITMAP is used to track cluster allocation, and the FAT is only required for re-
assembling the original file. If the ...
Because there is no floppy support, there is only one possible media descriptor
value
Cluster 0 and 1 are not defined, so ...
77
The 3 main critical records: Allocation Bitmap, UP-Case Table, and Root Directory
will use FAT chains.
The Root Directory ...
If there are 2 FATs in a TexFAT Transactional Safe exFAT environment, then each
FAT is paired with a allocation bitmap
The...
When files on legacy are fragmented, and deleted, the deletion wipes out the FAT
chain because each FAT cell has to be zer...
If the files are made larger, then code can be hidden into those files
The SD Specification indicates that the Allocation ...
This is an eye chart, but the idea is to show how to get to the bitmap.
You start at the VBR (BPB), go to the root directo...
We will see details of the directory entry construction later, including what we mean
by an entry type.
If there are issue...
Benign directory records
Fake secondary records
Zero length/zero cluster files
Phantom / orphan Files
84
The first byte of every directory entry is the “entry type” and describes the directory
entry.
85
When a file set is not in use, it is usually (but not always) a deleted file
When a volume label is not in use, it means n...
Primary and Critical
In legacy FAT, the Volume label is in the Root Directory, and has an attribute
87
Since we use 16 bit unicode without string termination, we need the length of the
volume label – in unicode characters.
88
Primary and Critical. If the FS can’t find the BITMAP table, it can’t mount the FS
Since there is no flag, this file will ...
This was a small volume. 63 bytes can support maximum of 63x8 = 504 clusters.
90
Filenames are stored case insensitive, so when a search is done, the filenames are
converted to upper case (folded). The U...
The UP-Case table is less than 6K – imagine if it was in a 32K cluster, now imagine
if it was in a 32MB cluster, the amoun...
File Entry Set would have a File, Stream Extensions, and up to 17 File Name
Extension for a total of 19.
Later, when a new...
Attributes and Timestamps in later slides
Checksum is across the Primary and all secondaries in the set.
94
Modified, Access, and Create.
Timestamps are NOT stored in this order, but MAC is a common acronym in the
literature.
Time...
The standard DOS Date/Time, also used in the previous FAT versions, does not
count to the second, but double seconds.
To g...
97
FAT and exFAT timestamp behavior varies, but is just not reliable as far as last
accessed.
TSK research shows some differe...
These are pretty much the same as previous FAT versions.
Since we have a separate volume label entry, there is no attribut...
The update behavior on the 10ms Modified is also not predictable, sometimes it is
just set to zero.
Note that the create t...
In order to validate the analysis in reverse engineering the FS, I had to write a C
program to format the directory entrie...
There is 2 file lengths, one is supposed to be the physical file length and the other
the amount of data actually written ...
103
Since these values can vary based on the format parameters, for reference this is
what the samples in this presentation is...
Another output from the C program.
Allocation possible indicates that the directory entry specifies a cluster address fiel...
Allocation not possible indicates that there is no cluster address in the entry.
FAT Invalid has no meaning
106
Filename is 40 characters (80 bytes) and takes 3 entries to store it.
Notice that in Uni-Code the file name is stored in m...
When the entries are not in use, some may be overwritten, and some may not. This
means that a complete set may not exist.
...
109
There are discussions of creation of exFAT on a Vista or Windows 7 machine that
can’t be seen on Vista. This is usually a ...
New, but 8 years old, misunderstood
More forensics tools need exFAT support
Implementations across vendors are inconsisten...
112
I need followers
113
Since NTFS has a smaller maximum cluster size (64K) 216 (29 * 27 = 216) while
exFAT maximum is set to 225. Then the questi...
http://www.snia.org/sites/default/files2/SDC2012/presentations/File_Systems/JRTipt
on_Next_Generaltion-3.pdf
115
My paper on exFAT and the Microsoft Patent that exposes the specification
116
117
I encountered these other sites that have information on exFAT.
I include them here to provide more information for the re...
119
120
121
Upcoming SlideShare
Loading in …5
×

NYC4SEC - An Introduction to the Microsoft exFAT File System (Draft 2.01)

1,521 views
1,377 views

Published on

As investigators and information security professionals, we have to constantly be aware of changing file systems to track data changes and accurately apply attribution to system changes.

In 2006 Microsoft released a successor to the FAT32 file system named the Extended FAT file system - labeled exFAT for short. exFAT was initially released for the Windows CE handheld device and in 2008 a version of exFAT was released for Microsoft Desktop and Server operating systems. Today exFAT is licensed and supported on many devices and systems, including Unix/Linux systems. The SD card association, with the release of the Secured Digital Extended Capacity (SDXC) memory card, has adopted exFAT as the standard file system for SDXC media which is used in cameras, cell phones and other consumer electronics.

exFAT is implemented in a different file system organization than the legacy predecessor FAT family file systems such as FAT12/16/32, and the forensics investigator will be required to know and understand this new format as forensics examinations are conducted using this new file system.

Robert Shullich, Enterprise Security Architect at Tower Group Companies, will give a great overview of the exFAT file system and the implications for investigators.

exFAT topics to be covered in the session:

• History

• Features

• File System Limits

• Advantages/Disadvantages

• Relevance to forensics computing and digital investigation

• Hiding places to look out for – where criminals can hide things

File System Layout and Internals

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,521
On SlideShare
0
From Embeds
0
Number of Embeds
19
Actions
Shares
0
Downloads
15
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

NYC4SEC - An Introduction to the Microsoft exFAT File System (Draft 2.01)

  1. 1. This presentation is to provide a technical understanding of the Microsoft Extended File Allocation Table (exFAT) for forensics examiners. It will also provide general knowledge of exFAT. 1
  2. 2. One of the is to provide the background and history of the file system, and the relationship to the other FAT versions in the family. 2
  3. 3. 3
  4. 4. D4CS stands for Digital Forensics & Cyber Security FCM 760 Fall 2009 4
  5. 5. This presentation was given 5 times within the 14 months since the SANS paper was published. It will now be given twice in 2014, with some updates At the 4/19 Computer Forensics Show, the session was recorded and can be found on the AT&T Tech Channel 5
  6. 6. Both books were published in 2012, other books have been published, and may mention exFAT, but these two send the reader to the SANS paper for more detailed and in-depth information. 6
  7. 7. Brian Carrier’s book is considered by some as the gold standard and bible on explaining file systems. It has become a little outdated since it is about 9 years old, no 2nd edition, and other file systems have surfaced since the book’s release. Earlier in 2014 exFAT support was planned/announced for TSK 7
  8. 8. This is the Sleuth Kit Wiki, on this page it is noted that exFAT support was added to TSK, and one of the docs used as a guide during development was the SANS paper. 8
  9. 9. Microsoft published a patent that included the exFAT 1.00 specification. This presentation and the paper attempt to stick to the terminology used in the patent/specification as close as possible. Links to the patent and my paper will be on a later slide, and references to the paper will also be on my blog. The presentation will focus on desktop/server version of exFAT which in 2014 is still Version 1 00 as is still released on Windows 8Version 1.00, as is still released on Windows 8 There are differences between implementations of other vendors, including incompatibilities, and the Windows phone has advances features and also uses compression. The technology behind this SD card feature is called Content Protection for Recordable Media (CPRM)Recordable Media (CPRM). Content Protection for Pre-recorded Media (CPPM), 9
  10. 10. 10
  11. 11. http://www.webopedia.com/TERM/O/OSR_2.html OEM Service Release 2 – Windows 95B According to Wikipedia, there was a 8 bit FAT originally Ray Duncan (1988). The MS-DOS Encyclopedia - version 1.0 through 3.2. Microsoft Press. ISBN 1-55615-049-0. Although we talk about FAT12/16/32, there were many flavors as the FAT family evolved. 11
  12. 12. exFAT is specifically designed for Removable media, but can be used for fixed media as well. NTFS is not recommended for removable media, especially because of the lazy write problem. Faster I/O through less file system overhead Limitation on how many times you can write to a single electronic gate (e g nand)Limitation on how many times you can write to a single electronic gate (e.g. nand) FAT in general is simpler, so in the case of embedded systems, stick with a variation of FAT instead of implementing NTFS 12
  13. 13. SD = Standard Definition HD = High Definition Quad HD is 4 times Full HD (Double wide, Double Long) Amount of data being recorded depends on many factors, including frames per second, size, color size, resolution, and compression rations. Definitely will exceed the 4GB file size limitation. 13
  14. 14. Source: https://www.sdcard.org/consumers/cards 14
  15. 15. 15
  16. 16. •You need to be able to locate the evidence, just in general •This includes re-assembly when a file is fragmented •Also includes recovery of deleted files •You also need to know the hiding places where it can be hidden •For example, unallocated space •You need to validate what you found is correct, in order (proper assembly), and complete (no missing pieces) •CP (Child Porn) when created uses cameras, and as camera memory gets cheaper, and moves to exFAT, relevant evidence is going to be on exFAT 16
  17. 17. Don’t be a money pressing a button, need to know what is under the covers 17
  18. 18. If the OS can’t recognize the file system, then it thinks the media is not formatted. When this slide was built, it was 2010, on a Windows XP machine, that did not have the hotfix. This example is Microsoft specific, and with XP being retired, and later systems getting the support, this situation should not occur often anymore on Microsoft Systems. Back when this happened this message would lead one to believe that the mediaBack when this happened, this message would lead one to believe that the media was blank and unformatted. 18
  19. 19. Any evidence with exFAT would probably be pushed aside with the lack of tools, documentation and expertise to process it. Tools that were available were raw acquisitions and then data carving Data carving easier and automatic when the file is contiguous 19
  20. 20. Linux and Open Source is used a lot for examinations Commercial tools are lacking, but picking up Little documentation or publications on exFAT internals, 4 years later and SNAS paper appears the authoritative resounse. exFAT Computer Compatibility http://gopro.com/support/articles/exfat-computer-compatibility We are not there yet https://www.cyberfetch.org/sites/default/files/EnCase%20Forensic%20v6.18.0.59%2 0Test%20Report pdf0Test%20Report.pdf Test Results for Deleted File Recovery and Active File Listing: EnCase Forensic Version 6.18.0.59 This report was prepared for the Department of Homeland Security Science and 20 This report was prepared for the Department of Homeland Security Science and Technology Directorate Cyber Security Division by the Office of Law Enforcement Standards of the National Institute of Standards and Technology. For additional information about the Cyber Security Division and ongoing projects, please visit www.cyber.st.dhs.gov.
  21. 21. http://www.cftt.nist.gov/presentations/AAFS-2013-Lyle-DFR.pptx AAFS = American Academy of Forensic Sciences http://www.cftt.nist.gov/presentations/AAFS-2013-Lyle-DFR.pdf 6 Vendors, not named in the presentation 21
  22. 22. Encase: http://www.cyberfetch.org/sites/default/files/EnCase Forensic v6.18.0.59 Test Report.pdf FTK: http://www.cyberfetch.org/sites/default/files/FTK v3 3 0 33124 Test Report.pdf Access to Test Images Layout of test ImagesLayout of test Images Each test case is repeated at least four times to characterize the tool’s behavior for different file system families. These include FAT, exFAT, NTFS and ext. The NTFS and exFAT images contain a single partition. The FAT and ext images each contain three partitions. Each partition has the same pattern of files created and deleted for a gi en test case The FAT and e t cases (three partitions) ha e three times asa given test case. The FAT and ext cases (three partitions) have three times as many files as the NTFS and exFAT cases (one partition). The FAT images contain a FAT-12, a FAT-16 and a FAT-32 partition. The FAT partitions were created on a Windows Vista system. Some partitions marked as FAT-12 in the partition table, appear to have a FAT table that is actually FAT 16 (thi did t i ifi tl ff t t t lt ) Th NTFS i lFAT-16 (this did not significantly affect test results). The NTFS images were also created on a Microsoft Windows Vista system. The ext partitions were created on a Fedora Linux system. The exFAT partition and HFS+ partitions were created on a Mac running Snow Leopard, OSX Version 10.6. 22
  23. 23. 23
  24. 24. 24
  25. 25. When we use the term “Mega”, is it 1,000,000 (1006) 0r do we mean “Mega” 1,048,576 (220) ? 25
  26. 26. A quick note on exponents, since we will get our hands dirty with math Some simple numbers should be like learning the times table in school 26
  27. 27. In some cases you might see ZB or ZIB, technically they are really different, but are close. So when we say 1 kb of disk, they mean 1,000 bytes, but when we say 1 kb of memory, they mean 1024 bytes. IEC 60027-2 A.2 and ISO/IEC 80000 http://physics nist gov/cuu/Units/binary htmlhttp://physics.nist.gov/cuu/Units/binary.html 27
  28. 28. Just another slide It is suggested that in English, the first syllable of the name of the binary-multiple prefix should be pronounced in the same way as the first syllable of the name of the corresponding SI prefix, and that the second syllable should be pronounced as "bee." 28
  29. 29. Being off by 15% when talking about an "exabyte" means being off by about bytes, or 150 petabytes. 29
  30. 30. exFAT uses 16 bit Unicode strings This is the terminology as used in the specifications leaked in the patent When reading the paper, and as we discuss here, these are the ground rules in terminology 30
  31. 31. It is important to note that Pentium processers use the little-endian format, so numbers stored in the file system are stored in little-endian. This can be significant because you need to change the order of the bytes in order to read the values from a hex dump. This could have issues with support of exFAT in other architectures, and could affect acquisitions. The exFAT specification requires little endian. 31
  32. 32. This is how Microsoft does Math, and then everyone uses these numbers not knowing the full context 232 sectors * 29 bytes per sector (512B) = 241 = 2,199,023,255,552 (2TB) http://support.microsoft.com/kb/184006 32
  33. 33. http://support.microsoft.com/kb/955704 Volume size of 64ZB is architecturally incorrect. Currently it cannot exceed 128PiB because: 1) With 232 clusters (32 bit fat indices) tracking clusters with a maximum of 225 in size = 257 = 128 PiB [32+25] 2) With LBA 48 as the maximum addressable block on the FS with a 512B physical2) With LBA-48 as the maximum addressable block on the FS, with a 512B physical sector, the file system supported would be 257 = 128 PiB [48+9] Note that with #2, we could go further with native AF 4K sectors, however it is interesting how the numbers add up. For file size, the current architecture uses 64 bit numbers for the length of file, based on that the maximum (theoretical) file size is really 264-1 = 16EiB Now, since the architecture limits the filoe system to less than 128 PiB, and PIB is smaller than EiB, the maximum file size is almost the volume size minus overhead and metadata.and metadata. 33
  34. 34. http://en.wikipedia.org/wiki/Windows_CE Microsoft Windows CE (now officially known as Windows Embedded Compact and previously also known as Windows Embedded CE Small footprint, limited API Windows XPE, XP Embedded – Different, uses desktop code but not all features WinCE code is used to derive code for other embedded systems including the phone http://support.microsoft.com/default.aspx?scid=kb;EN-US;Q166915p pp p ; ;Q 34
  35. 35. 35
  36. 36. You never really see another sector size other than 512 bytes, but everyone just assumes that it is only 512 (in earlier versions of Windows NT, there were weird sector sizes) The 4096 size is special to support a device that is used for paging and supports 4K pages. But with the standard format, you can’t adjust sector size http://en.wikipedia.org/wiki/Advanced_Format – Advanced format is for any sector size > 520 bytes. Clusters (or blocks) are 64KiB max in FAT32 However, for Windows 95 OSR2, 98 and ME, the FAT32 clusters had a maximum of 32KiB, so for backward compatibility it was recommended to not exceed 32KiB ExFAT and FAT32 Root Directories not restricted in size, other than space available on the volume. Max files on FAT32 volume ≈ 228 while exFAT ≈ 232 This is based on a maximum of one file per allocation unit (each cell in the FAT is one allocation unit) 36 one file per allocation unit (each cell in the FAT is one allocation unit) Since exFAT supports empty files that don’t take up space in the cluster heap (length = 0 first cluster = 0) Max Files theoretically could be more Ref: http://support.microsoft.com/kb/955704
  37. 37. This new format, called Advanced format, provides via hardware 4K physical sector size. An OS/FS can either use emulation (512e) or native (4Kn) exFAT will support 4K sized sectors. 37
  38. 38. This command (help format) was issued on a Windows 7 32-bit system. This snippet is for the allocation unit size. Most “supports” largest clusters for 512b sectors at 64K, including NTFS, FAT & FAT32. FAT & FAT32 appear to support a larger allocation units of 128K and 256K when sectors are > 512b (probably AF sectors). Although in some cases 64K allocation units are supported, not all OS support it, and in some cases 64K+ not supported and must be a power of 2 thus 32Kand in some cases 64K not supported, and must be a power of 2, thus 32K. Differences between Windows 95/98/ME and Windows NT4/2000/XP, 7 & 8 Even though a FAT32 could lead to 8TB (for 32K) and 16TB for 64K cluster sizes, when putting into a MBR, LBA is 32 bits, a block is 512b, so all file systems in a MBR is restricted to 2TB Might get 8TB if AF (4K sectors) are used. 38
  39. 39. Microsoft in the KB for Windows XP support indicated a capacity to 64ZiB and a file size maximum to 64ZiB. In reality, the file system can only support up to 128PiB, and the file size up to 16EiB. Microsoft documentation indicates a maximum file system size as 512TiB The recommended maximum volume size is 512 TBThe recommended maximum volume size is 512 TB. http://support.microsoft.com/?kbid=955704 The volume size is limited by a 32-bit FAT and a 25-bit cluster size giving a 57-bit addressable volume size The file size is limited by the 8-byte (64-bit) number that holds the file size.The file size is limited by the 8 byte (64 bit) number that holds the file size. The volume label and file names are all 16 bit unicode Filenames to a maximum of 255 characters Subdirectory is max at 256MiB, Directory records are 32 bytes, and the smallest fileset is 3x32 = 96 bytes and assumes no ACL and a filename < 16 characters in length. exFAT better optimized, reduce the “write” actions For media that use OEM parameters this may be a method to convey device 39
  40. 40. With TexFAT there will be 2 FATS and 2 BITMAPS, with exFAT 1.0 – which does not have TexFAT (Transactional FAT) support, there is ony 1 FAT and 1 BITMAP, where previous FAT versions had 2 FATs. To be released later, but it is 6 years and we are still at VV.MM 01.00 TexFAT and ACL already exist in Windows CETexFAT and ACL already exist in Windows CE 40
  41. 41. Any FS is limited, even FAT32 and NTFS. This is Windows only, we are not talking GUID Partition Table (GPT) Although a MBR uses a 4 byte sector count, remember that the FS can be larger if you make the sectors larger (512 vs. 4096) and this causes a lot of confusion on how big a FS fits. A FAT32 filesystem could reach 8 TiB in size (2**28 x 32K), but with 512 sector sizes a MBR can only support 2TB (with 4K size a MBR can support 16TiB)sizes, a MBR can only support 2TB. (with 4K size, a MBR can support 16TiB) http://support.microsoft.com/kb/314463 http://techcosupport.com/press/maximum-size-of-a-fat-32-partition/ GUID Limits: http://support.microsoft.com/kb/302873http://support.microsoft.com/kb/302873 http://msdn.microsoft.com/en-us/library/windows/hardware/dn640535(v=vs.85).aspx A GPT GUID Partition uses a 64 bit number for the number of logical blocks In theory, a GPT disk can be up to 2^64 logical blocks in length. Logical blocks are commonly 512 bytes in size. 41 This would be 264 * 29 = 273 which is 270 = 1 ZiB and 23 = 8 for ≈ 8Zib maximum partition size of 264−1 sectors. For disks with 512-byte sectors, that would be 9.4 ZB (9.4 × 1021 bytes) or 8 ZiB−512 bytes (9,444,732,965,739,290,426,880 bytes or 18,446,744,073,709,551,615 (264−1) sectors × 512 (29) bytes per sector)
  42. 42. Windows would not format FAT32 beyond 32GB, it required using a FAT32 format on a different OS Some Windows utilities did not work properly with volume spaces GT 32GB, but you can mount a device that was GT 32GB Limitations of FAT32 File System: http://support.microsoft.com/kb/184006 SDXC predecessor (SDHC) had a max spec of 32GB. SDXC picks up from 32GB. (But starts around 48GB 32GB will till be SDHC for a while)(But starts around 48GB, 32GB will till be SDHC for a while) 4GB maximum file size barrier existed in both FAT and FAT32. SD 4.0 Specification – 300MB/s I/O speeds http://www.flashmemorysummit.com/English/Collaterals/Proceedings/2009/2009081 3_S204_Lin_Yee.pdf Starting at 104 mega bytes per second, and later to 300 mega bytes per second http://www.letsgodigital.org/en/20985/sdxc-cards/ Microsoft set limits on FAT32 volume size In one argument, older utilities could not format the volume correctly or could not determine the proper size 42 In another argument, since the larger volumes had a much larger FAT, massive reads of the FAT would be required to find free space. For example, with a 32K cluster size and a 32GB media, the FAT would be about 4MB, and for a heavily used (low free space) volume there could be a lot of I/O to find free clusters. FAT32 limited this overhead by adding a hint of freespace using the FAT32 File System Information sector, but it was a hint and not always to be relied upon, just to point the software to where free cluster might be
  43. 43. 3rd party file utilities may provide conversion to and from exFAT, but no Convert command, and current convert command doesn’t work even to change exFAT to NTFS or even FAT32. Mostly a Microsoft Desktop and Server World – there is Linux, and MAC, Microsoft dominates 43
  44. 44. There are discussions of creation of exFAT on a Vista or Windows 7 machine that can’t be seen on Vista. This is usually a case of creating the media on a machine with exFAT support and then trying to read the media on a different machine without exFAT support. The common mistake is creation of the file system on removable media with a Vista SP1 (or higher machine) and trying to read it on a machine with Vista RTM. 44
  45. 45. 45
  46. 46. The SDXC media will not be backward compatiblee-solutuions/volkswagon for in vehicle entertainment systems DCF 2.0 – Design Rule for Camera File System Camera and Imaging Products Association (CIPA) – DC-009-2010 Japan Electronics and Information Technology Industries Association (JEITA) CP3461BCP3461B Exchangeable image file format (officially Exif, not EXIF according to JEIDA/JEITA/CIPA specifications) is a standard that specifies the formats for images, sound, and ancillary tags used by digital cameras (including smartphones), scanners and other systems handling image and sound files recorded by digital cameras The specification ses the follo ing e isting file formats ith the additioncameras. The specification uses the following existing file formats with the addition of specific metadata tags: JPEG Discrete cosine transform (DCT) [1] for compressed image files, TIFF Rev. 6.0 (RGB or YCbCr) for uncompressed image files, and RIFF WAV for audio files (Linear PCM or ITU-T G.711 μ-Law PCM for uncompressed audio data, and IMA-ADPCM for compressed audio data). [2] It is not supported in JPEG 2000, PNG, or GIF. (Source Wikipedia) 46
  47. 47. This is just a selection of some dates, not exhaustive. Volkswagon and Audi licensed Tuxera drivers to get exFAT support. http://www.thesixthaxis.com/2014/04/30/the-hidden-features-and-changes-of- playstation-4-firmware-1-70/ The 1.70 PS4 firmware update is rumored to have exFAT support 47
  48. 48. Tuxera http://www.tuxera.com/products/tuxera-exfat-embedded/ Provides exFAT & NTFS to be integrated in embedded systems. They do software development and integrate the support They have a working relationship with MS, and have licensed exFAT. Their development then gets integrated for other companies. However, there are developers working on their own to build and generate drives for Linux. 48
  49. 49. The Linux community is very hostile to Microsoft, complaining that a Microsoft standard was forced on them and Microsoft expects to get paid. However, that is what they are stuck with. Meanwhile the community is trying to build exFAT drivers for the various Linux distributions in order to support exFAT. IN 2013 someone leaked the source code to Samsung’s exFAT kernal basedIN 2013 someone leaked the source code to Samsung s exFAT kernal based drivers. Eventually Samsung put those drivers under a GPL license and released them as open source. Even through there is now a GPL license for Samsung’s exFAT implementation, I would expect that Microsoft will want their royalties and there may be legal issue ahead. E er since the Tom Tom settlement companies are afraid of iolating Microsoft’sEver since the Tom Tom settlement, companies are afraid of violating Microsoft’s patent rights. 49
  50. 50. 50
  51. 51. 51
  52. 52. 52
  53. 53. I use a CPAP machine, and it has a SD card to record my sleep patterns These are the uses, many of these devices take USB, SD, CF, and even memory stick Most of these are embedded systems, but produce or consume media that can/or was processed on desktop systems 53
  54. 54. SANDISK ships high capacity CF cards pre-formatted as exFAT PNY 128GB Turbo USB flash DigiStore SSD Some Kingston Memory Even some magnet disk media, example Western Digital Pre-formatted file system is not usually specified in the product specs, so it is hard to determine unless you see discussions or go out and buy & test everything In 2010 a 64GB SANDISK SDXC card was selling on Amazon for $350, in May 2014 I bought 2 of these at $49.50 each (with free tax & free shipping) 54
  55. 55. New Devices may accept SDXC, but older devices might not. Mentioning memory cards such as camera cards is important because the target market for exFAT is removable storage. Back in 2009 I believed that this was going to be a big driver towards exFAT adoption but flash memory in any type of memory card or even SSD drives mayadoption, but flash memory in any type of memory card, or even SSD drives may have contributed to the adoption rate and prevalence. 55
  56. 56. With Sony adopting the XC memory stick to exFAT, plus the SD market, is almost 90% of the market today. http://anythingbutipod.com/2009/01/next-generation-sdxc-details/ Jan 8, 2009 56
  57. 57. July 2012 57
  58. 58. The 137GiB comes from LBA-28 addressing. 228 = 268,435,456 228 * 29 = 128GiB = 137GB = 137,438,953,472 144PB comes from LBA-48 addressing 248 = 281,474,976,710,656 248 * 29 = 128PiB = 144PB = 144,115,188,075,855,872 58
  59. 59. SD – up to 2GB – FAT SDHC – 2GB to 32GB FAT32 SDXC – 32GB to 2TB exFAT These are marketing hype and represent maximums http://www.dpreview.com/news/2009/1/8/sdxc http://www.computerworld.com/s/article/9125622/Memory_card_standard_could_pr ovide_up_to_2TB_on_an_SD_card The 64GB SDXC card, for instance, can store a full 16 hours of 1080p High- Definition footage (1920x1080 9Mbps H.264 AVC compression) or over 4000 RAW images (based on 14MB file size), ideal for situations that demand continuous burst-images (based on 14MB file size), ideal for situations that demand continuous burst mode shooting and non-stop video recording. Source: http://dk.transcend-info.com/About/press/10044 The size of a photo will vary based on the camera resolution and the effectiveness of compression. Transcend (from the statement above) indicates 4,000 Raw images based on 14MB file size. This was for a 64GB card, while the SD press releases were saying 4,000 images per 2TB card, something is off. And in the case of just taking the Nikon D7100 DX, RAW images are really almost 59
  60. 60. Nand gate wear and tear, less write, longer the memory may last Although the SD association states that the official, standard and only file system for SDXC is exFAT, users will format the card using other file systems. Some cameras may allow a SDXC card to be formatted as FAT32, but others will indicate that the SD Card is not formatted properly and ask to format it. Yet, a user may format the SD Card in another file system, and use it in a non- compliant device such as a slot on the laptop where only the OS will read and writecompliant device, such as a slot on the laptop where only the OS will read and write the card. 60
  61. 61. Write Endurance (Program Erase Cycles) Limited, maybe up to a million, writes Writes require the storage area to be erased first, almost like an EPROM Flash memory, nand and nor gates, should not be full formatted unless needed – do a quick format. Good for forensics because data is not erased Degrags should not be done either, flash memory doesn’t have moving parts so and the extra writes wear down the solid state chipsthe extra writes wear down the solid state chips. Writes need to be limited, the less writing the longer the memory will last and the better the performancebetter the performance Write Cliff Blocks are rotated and pre-erased, but if you run out of empty blocks, then you have to wait for a block to be erased before you can reuse it for a different set of values. Wear Leveling Methods of providing a block of gates, and rotating through the gates. 61
  62. 62. Why a 2TB limit when the CF has a 144PB limit? Looks like SDXC uses a MBR partition to separate the protected area from the user data area, and that limits volume to 2TB The follow-on the SDXC will need to use a GPT, a partition would probably still be required with a separate protected area. CD does not have DRM/copy protection, so extra partition probably not required. Then issue is superfloppy or GPTThen issue is superfloppy or GPT Format of a SDXC card puts standard boot code in the boot sector, while cards from the factory ha all those fields filled up with F4 Need to format card using SD Formatter utility and inside the camera – both cases to see what is going into the sector. AU Sizes vary based on size of total volume. 62
  63. 63. 63
  64. 64. Currently use exFAT 1.00, but if a later version of exFAT is in use, it will check the version # and not mount the FS unless it can suppoort it Checksums protect against corruption and viruses If there is a problem with critical directory entries, the FS should not mount. The dirty flag used to be in the 2nd FAT index in FAT32 64
  65. 65. 65
  66. 66. 4 Regions defined on the volume The FAT tables reside outside the cluster heap Everything except the data region is measured in and addressed as sectors. Data region is measured and addressed as blocks, blocks are called clusters 66
  67. 67. FAT and Cluster heap have their own offsets, which allow alignment, if needed to force these region on a designated boundary line Might be needed in SD and other flash memory The specification for exFAT says the # of FATs is either 1 or 2. For legacy FAT it is recommended to be 2, could be 1, but could even be more although rarely seenalthough rarely seen. Since there are offsets, I could build a 3rd, 4th or more FAT – just stick space between the 2nd FAT and the cluster heap start and have an area of slack space. 67
  68. 68. Details follow in the next slides A mirror of the VBR follows, and is a backup VBR In case the first gets corrupted FAT32 had a mirror also, the mirror was at sector 6 68
  69. 69. When you take the volume length (64-bit) * 4K sector, that is 64+12 = 276. 270 = 1 ZiB, 26 = 64, thus based on this value, a file system of 64 ZiB. However, the current architecture specification cannot produce a file that big unless some parameter somewhere gets changed. 69
  70. 70. If there was no restriction, then the size of a cluster could be 4255 70
  71. 71. If the sector size is > 512 bytes, all space on the first sector of the VBR (Main Boot Sector) is not used (Only the first 512 bytes) 71
  72. 72. Unlike the first sector, the other 8 boot sectors can use the entire sector and the signature marker is moved to the last 8 bytes of the sector 72
  73. 73. If a virus modified the boot record, and doesn’t fix the checksum, there should be a mount failure 73
  74. 74. Repeats over and over again, 4 bytes = 32 bit checksum Can be used to determine if the VBR was modified 3 bytes in the VBR are not calculated in the checksum This sector does not have a signture 74
  75. 75. The BITMAP is used to track cluster allocation, and the FAT is only required for re- assembling the original file. If the original file is contiguous, then the FAT isn’t needed for THAT file. We will see later that a flag in the directory record is used to tell the FS whether the FAT should be used or ignored. For Contiguous Cluster Allocation see Patent: US8606830B2 (Contiguous File Allocation in an Extensible File System) For TexFAT see Patent: US7613738B2 (FAT Directory Structure for use in Transaction Safe File System) For Extensible File System see Patent: US8583708B2 (Extensible File System) 75
  76. 76. Because there is no floppy support, there is only one possible media descriptor value Cluster 0 and 1 are not defined, so 0 & 1 are not significant (Same as legacy FAT) Since the FAT is no longer used for cluster allocation, 0 (zero) is no longer significant (used to mean the cluster was unused/free/unallocated) FF..F9 thru FF..FE were also EOC (end of cluster chain) markers in FAT32, but are unused in exFATunused in exFAT. Values 2 thru FF..F6 are cluster addresses. The Cell’s content is called an index or indices 1st indices contains FFFFFFF8 (Media Descriptor) 2nd indices contains FFFFFFFF and is not used to hold dirty volume flags 76
  77. 77. 77
  78. 78. The 3 main critical records: Allocation Bitmap, UP-Case Table, and Root Directory will use FAT chains. The Root Directory can grow and since it is dynamic in its growth, most likely will fragment. The UP-CASE Table and Allocation bitmap should be static and not grow or change, although theoretically they could probably be relocated and moved somewhere else on the volume. However, in the SD standard, the allocation bitmap must be within the first 4MB of the cluster heap. The locations (cluster addresses) of the 3 special metadata files may change, this is based on one formatting and in reality these files could eventually end up in any cluster. These 3 special files (4 if TexFAT) will use chains even if unfragmented becauseThese 3 special files (4 if TexFAT) will use chains even if unfragmented because even though they appear in the cluster heap, they are not true user files, and are defined by special directory records and not file records to point to these special system files. These special files don’t have the INVALID FAT flag. 78
  79. 79. If there are 2 FATs in a TexFAT Transactional Safe exFAT environment, then each FAT is paired with a allocation bitmap The allocation BITMAP is pointed to by a 0x81 entry. To locate an empty cluster, each 512 byte sector would hold 4096 allocation bits (512 bytes x 8 bits) More efficient but still require reading many sectors if the FS is large and the earlyMore efficient, but still require reading many sectors if the FS is large and the early part of the FS is allocated. 79
  80. 80. When files on legacy are fragmented, and deleted, the deletion wipes out the FAT chain because each FAT cell has to be zeroed out to indicate that the cell is no longer allocated. Since allocation is moved from the FAT to the Allocation Bitmap, the FAT cells for the chain remain intact. There is still the possibility of cells being overlaid by other file fragments after the delete. This is not in the spec, buit this behavior has been observed. 80
  81. 81. If the files are made larger, then code can be hidden into those files The SD Specification indicates that the Allocation Bitmap must be within the first 4MB of the file system. 81
  82. 82. This is an eye chart, but the idea is to show how to get to the bitmap. You start at the VBR (BPB), go to the root directory, look up the 0x81 entry to get the cluster address, and then go into the BITMAP table. The first byte of a directory record is the entry type, here we see x’83’, X’82’ and X’81’ 82
  83. 83. We will see details of the directory entry construction later, including what we mean by an entry type. If there are issues with the critical entries, then the file system should not mount. In FAT the largest directory size is 221 Which equals 2,097,152 http://read.pudn.com/downloads77/ebook/294884/FAT32%20Spec%20(SDA%20Co ntribution) pdfntribution).pdf In FAT, with a directory size of 221 and a director record size of 25 (32 bytes) the number of 32 character entries in the directory is 216 = 65,536 Keep in mind that the maximum number of files will be related to the FAT itself since not more than a single file can occupy a cluster. In exFAT the largest directory size is 228 = 256MiB exFAT directory size limit is 27 (128) times the size of the FAT limitation 83
  84. 84. Benign directory records Fake secondary records Zero length/zero cluster files Phantom / orphan Files 84
  85. 85. The first byte of every directory entry is the “entry type” and describes the directory entry. 85
  86. 86. When a file set is not in use, it is usually (but not always) a deleted file When a volume label is not in use, it means no volume label In a file set, it could be caused by renaming a file with a longer file name. Only files have secondary entries so far Missing Benign entries usually won’t prevent the file system from being mounted. 0x80 is not defined. In FAT32, file deletion is done by overlaying the first byte with 0x’E5’, or X’05” if the first byte of the filename is already an 0x’E5’. For KANJI character set based names, the value 0x05 is stored in DIR_Name[0] - if required - to represent 0xE5. 86
  87. 87. Primary and Critical In legacy FAT, the Volume label is in the Root Directory, and has an attribute 87
  88. 88. Since we use 16 bit unicode without string termination, we need the length of the volume label – in unicode characters. 88
  89. 89. Primary and Critical. If the FS can’t find the BITMAP table, it can’t mount the FS Since there is no flag, this file will always have a FAT chain, even if it is one cluster, will always have a cluster chain ending with EOC 89
  90. 90. This was a small volume. 63 bytes can support maximum of 63x8 = 504 clusters. 90
  91. 91. Filenames are stored case insensitive, so when a search is done, the filenames are converted to upper case (folded). The UP-CASE table is used to convert the filename to all uppercase. 91
  92. 92. The UP-Case table is less than 6K – imagine if it was in a 32K cluster, now imagine if it was in a 32MB cluster, the amount of available slack space. 92
  93. 93. File Entry Set would have a File, Stream Extensions, and up to 17 File Name Extension for a total of 19. Later, when a new exFAT version comes out, the ACL will be another secondary entry bringing this up to 20. As more file secondary entries are added, let’s say one for encryption, this increases to a max of 255 secondaries. 93
  94. 94. Attributes and Timestamps in later slides Checksum is across the Primary and all secondaries in the set. 94
  95. 95. Modified, Access, and Create. Timestamps are NOT stored in this order, but MAC is a common acronym in the literature. Timestamps are not one single field like NTFS which uses a 64 bit value. exFAT combines pieces to make a UTC value. TZ offset is absent in Vista SP1, and does not appear in the exFAT 1.00 spec. Note: By default, the creation time is tunneled if a file is deleted, and a file with the same name is created within 15 seconds. (See KB172190 http://support.microsoft.com/kb/172190) 95
  96. 96. The standard DOS Date/Time, also used in the previous FAT versions, does not count to the second, but double seconds. To get seconds, a 33 bit number would have been needed. The OS doesn’t always update last access. And even NTFS last access is disabled in some versions, can modify behavior with” fsutil behavior set disablelastaccess 0fsutil behavior set disablelastaccess 0 Not sure if FAT32/exFAT is relaiable 96
  97. 97. 97
  98. 98. FAT and exFAT timestamp behavior varies, but is just not reliable as far as last accessed. TSK research shows some differences between OS, so timestamp analysis could be very inconsistent Even in later Windows releases, NTFS doesn’t even update the Last Accessed on READ for performance reasons but this behavior can be restored via a registry keyREAD for performance reasons, but this behavior can be restored via a registry key. 98
  99. 99. These are pretty much the same as previous FAT versions. Since we have a separate volume label entry, there is no attribute for it, and since we don’t have 8.3 support, there is no LFN (Long File Name) attribute either because everything is LFN. Reserved1, which is mask 0x08 was ATTR_VOLUME_ID (0x08) in legacy FAT 99
  100. 100. The update behavior on the 10ms Modified is also not predictable, sometimes it is just set to zero. Note that the create time is really 3B866244 (reversed because of little-endian) 100
  101. 101. In order to validate the analysis in reverse engineering the FS, I had to write a C program to format the directory entries. This is an example of the output. All the timestamps are even because of the double seconds. But since the create is 168, this means that the create time was really 12:18:09.68 Secondary count is 4, meaning that this file set is 5 entries, 1 File, 1 Stream, and 3 filenamefilename. 101
  102. 102. There is 2 file lengths, one is supposed to be the physical file length and the other the amount of data actually written into the file so far (Valid Data Length - VDL) These are two 64 bit length and are similar to the two lengths in NTFS in the $FILE_NAME 0x30 attribute. Length of name is needed because there is no string termination, but the file name (max 255) may require multiple directory entries (we will see later). This is where the FS indicates whether the FAT is used if the FAT Invalid flag is setThis is where the FS indicates whether the FAT is used, if the FAT Invalid flag is set, then the FAT is ignored. In legacy FAT The cluster number of the first cluster of the file is recorded in the directory entry associated with the file. For zero-length files, the first cluster number in the associated directory entry is set to 0. exFAT also supports a first cluster of zero if the length is zero. One of the Lengths is called “DataLength” Field 102 One of the Lengths is called DataLength Field The specification states: If the corresponding file directory entry describes a directory, then the valid value for this field is the entire size of the associated allocation, in bytes, which may be zero. Further, for directories, the maximum value for this field is 256MB. The other length field is called “ValidDataLength” field says that if this is for a directory, then this value must match the DataLength field.
  103. 103. 103
  104. 104. Since these values can vary based on the format parameters, for reference this is what the samples in this presentation is using. 104
  105. 105. Another output from the C program. Allocation possible indicates that the directory entry specifies a cluster address field FAT invalid indicates that this file does not use the FAT This file is 18MB and required 143 clusters to store the file. As we said before, there are 3 filename entries (each holds 15 characters of the filename), and as we see above, the filename is 40 characters in length. 105
  106. 106. Allocation not possible indicates that there is no cluster address in the entry. FAT Invalid has no meaning 106
  107. 107. Filename is 40 characters (80 bytes) and takes 3 entries to store it. Notice that in Uni-Code the file name is stored in mixed case 107
  108. 108. When the entries are not in use, some may be overwritten, and some may not. This means that a complete set may not exist. 108
  109. 109. 109
  110. 110. There are discussions of creation of exFAT on a Vista or Windows 7 machine that can’t be seen on Vista. This is usually a case of creating the media on a machine with exFAT support and then trying to read the media on a different machine without exFAT support. The common mistake is creation of the file system on removable media with a Vista SP1 (or higher machine) and trying to read it on a machine with Vista RTM. Microsoft distributes a specification, each vendor writes their own drivers, so variations between vendors, causing compatibility issues are occurring Users try for format drive on Windows system, drive is >32GB, the only options are: exFAT & NTFS, they format in exFAT and then find out their device doesn’t work in other places due to lack of exFAT support. The drive of a user to get away from a device in FAT32 is the 4GB barrier. 110
  111. 111. New, but 8 years old, misunderstood More forensics tools need exFAT support Implementations across vendors are inconsistent and might not implement all features Needs to be fixed before it gets worse Even utilities for Disk Partition, Defragmentation, File Recovery, and commands like CHKDSK need exFAT supportCHKDSK need exFAT support More evidence is going to show up in exFAT format, need to acquire the right tools and get experience 111
  112. 112. 112
  113. 113. I need followers 113
  114. 114. Since NTFS has a smaller maximum cluster size (64K) 216 (29 * 27 = 216) while exFAT maximum is set to 225. Then the question is: What happends to NTFS? 114
  115. 115. http://www.snia.org/sites/default/files2/SDC2012/presentations/File_Systems/JRTipt on_Next_Generaltion-3.pdf 115
  116. 116. My paper on exFAT and the Microsoft Patent that exposes the specification 116
  117. 117. 117
  118. 118. I encountered these other sites that have information on exFAT. I include them here to provide more information for the reader 118
  119. 119. 119
  120. 120. 120
  121. 121. 121

×