Reverse-Engineering a Proprietary Sound Sample Format

Slide 1: Reverse-engineering a proprietary sound sample format: A detective story Andrew Bulhak http://dev.null.org/acb/

Slide 2: The Problem •You make electronic music with softsynth plugins •Your drum machine plugin uses a proprietary format for its samples •You want to move your samples to an open format, like AIFF

Slide 3: Why? •To use your samples (or presets) with other software •To use samples with hardware devices •Because open formats are always better than proprietary ones •For the challenge

Slide 4: The Drum Plugin Linplug RMIV

Slide 5: The Drum Plugin (2) Linplug RMIV •A VST/AudioUnit plugin that works with sequencer software •Can play both sample-based and synthesised sounds •Stores samples in a custom format named .D4T •Can import sounds in AIFF and WAV formats, converting them to .D4T

Slide 6: Examining the D4T format Looking at the same sounds in both AIFF and D4T format: % ls -l -rw-r--r-- 1 acb staff 20586 26 May 2000 606bd.wav -rw-r--r-- 1 acb staff 40904 11 Apr 14:06 606bd.D4T -rw-r--r-- 1 acb staff 15182 26 May 2000 606ch.wav -rw-r--r-- 1 acb staff 30096 11 Apr 14:06 606ch.D4T -rw-r--r-- 1 acb staff 33900 26 May 2000 606ht.wav -rw-r--r-- 1 acb staff 67536 11 Apr 14:06 606ht.D4T -rw-r--r-- 1 acb staff 31426 26 May 2000 606lt.wav -rw-r--r-- 1 acb staff 62588 11 Apr 14:06 606lt.D4T Do you notice a pattern?

Slide 7: The D4T Format (2) •Each D4T file is roughly twice the size of its corresponding WAV •The WAVs use 16-bit samples •Therefore, D4T uses 32-bit samples Hypothesis: D4T uses 32-bit float samples, between -1.0 and 1.0 (as that makes more sense than 32-bit ints).

Slide 8: Testing the Hypothesis Using hexdump(1), examine the first nonzero samples of a WAV and a D4T. Then see if they're equivalent. I.e., if W = the WAV sample and D the D4T one, if W = int(D*0x8000)

Slide 9: Testing the Hypothesis (2) % hexdump -C 606bd.wav |head 00000000 52 49 46 46 62 50 00 00 57 41 56 45 66 6d 74 20 |RIFFbP..WAVEfmt 00000010 10 00 00 00 01 00 01 00 44 ac 00 00 88 58 01 00 | ........D?...X.. 00000020 02 00 10 00 64 61 74 61 d0 4f 00 00 77 03 96 0b | ....data?O..w... % hexdump 606bd.D4T|head 0000000 00 00 00 00 00 00 00 00 00 04 08 40 01 04 29 00 0000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0000020 00 00 00 00 00 00 00 00 00 c0 dd 3c 00 60 b9 3d Thus our first samples are 0x0377 and the float represented by 00 c0 dd 3c.

Slide 10: Testing the Hypothesis (3) We can use Python's struct module to unpack binary floats. >>> struct.unpack('<f', 'x00xc0xddx3c') (0.027069091796875,) Aha, a float between -1.0 and 1.0. >>> int(struct.unpack('<f', 'x00xc0xddx3c')[0] * 0x8000) 887 >>> '%x' % 887 '377'

Slide 11: A First Attempt We can now write a simple Python script for converting sound files. Our script will: •Open a .D4T file •Skip 40 bytes •Read the remainder of the file, treating it as floats •Write all that to an AIFF file (using Python's aifc module). •You could just as easily write WAV

Slide 12: A First Attempt (2) Our script has a few limitations: •It assumes a default sample rate (44100, in this case) •It assumes samples have one channel •If a D4T has two channels, the resulting AIFF will be mono, with one after the other. As such, we need to extract more information from our D4T files.

Slide 13: The D4T Header The secrets must be encoded in the nonzero bytes at the top of the file. % hexdump 606bd.D4T|head 0000000 00 00 00 00 00 00 00 00 00 04 08 40 01 04 29 00 0000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0000020 00 00 00 00 00 00 00 00 00 c0 dd 3c 00 60 b9 3d But what do they mean?

Slide 14: Number of Channels We examine two D4T files, one mono and one stereo: % hexdump 606bd.D4T|head 0000000 00 00 00 00 00 00 00 00 00 04 08 40 01 04 29 00 0000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0000020 00 00 00 00 00 00 00 00 00 c0 dd 3c 00 60 b9 3d % hexdump BgBeatSnare.D4T|head -2 0000000 00 00 00 00 00 00 00 00 00 1d 2c 00 02 04 29 00 0000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Aha! We've found our channel count.

Slide 15: File Size The group of bytes immediately before the channel count is proportional to the file size, though doesn't translate into a sensible number of either bytes or samples. % ls -l -rw-r--r-- 1 acb wheel 40904 11 Apr 14:06 606bd.D4T -rwxr-xr-x 1 acb wheel 697976 8 Jul 00:42 808Kick.D4T -rwxr-xr-x 1 acb wheel 8152 8 Jul 00:42 GenericSynBass.D4T % hexdump 606bd.D4T | head -1 0000000 00 00 00 00 00 00 00 00 00 04 08 40 01 04 29 00 % hexdump 808Kick.D4T | head -1 0000000 00 00 00 00 00 00 00 00 00 45 4f 24 02 04 29 00 % hexdump GenericSynBass.D4T | head -1 0000000 00 00 00 00 00 00 00 00 00 00 51 0c 01 01 63 4d

Slide 16: File Size (2) We create a few AIFFs of specific sizes, get RMIV to convert them into .D4Ts, and examine those: % ls -l -rw-r--r-- 1 acb admin 132 8 Jul 14:18 test_23x1.D4T -rw-r--r-- 1 acb admin 136 8 Jul 14:18 test_24x1.D4T -rw-r--r-- 1 acb admin 140 8 Jul 14:18 test_25x1.D4T % hexdump test_23x1.D4T |head -1 0000000 00 00 00 00 00 00 00 00 00 00 00 5c 01 04 29 00 % hexdump test_24x1.D4T |head -1 0000000 00 00 00 00 00 00 00 00 00 00 00 60 01 04 29 00 % hexdump test_25x1.D4T |head -1 0000000 00 00 00 00 00 00 00 00 00 00 01 00 01 04 29 00 At small sizes, this looks like a byte count, though after 96 bytes, there's a discontinuity.

Slide 17: File Size - The Answer It appears that each byte in the file size can only contain a value under 100 (0x64). From this, we can conclude that the size is encoded in binary-coded centimal. •Each byte contains a base-100 digit, or two decimal digits, as a number from 0 to 99. •I.e., 10632 would be 0x01 0x06 0x20 •Why? I have no idea.

Slide 18: Sample Rate Now that we know about binary-coded centimal, the rest of the puzzle falls into place. The remaining 3 bytes after the channel count are the sample rate in BCC, i.e.: % hexdump 606bd.D4T|head 0000000 00 00 00 00 00 00 00 00 00 04 08 40 01 04 29 00 0x04 0x29 0x00 = 04 41 00 = 44100

Slide 19: Summary: The D4T Format We now know the structure of a .D4T: •8 zero bytes •4 byte total length (L) of sample data, in bytes, DCC-encoded •1 byte channel count C •3-byte sample rate, DCC-encoded •24 zero bytes •for i in 1..C: •(L/C)/4 samples, comprising that channel

Slide 20: Conclusion Armed with this knowledge, it is possible to write a Python script to convert D4T files to AIFF. That script lives at http://dev.null.org/code/dermiv/