Beginning Kindle Hackery

5,729
-1

Published on

I gave a talk on Kindle hacking at Shibuya Perl Mongers in April 2009. It documents my first couple months messing around with the Kindle.

Published in: Technology
1 Comment
9 Likes
Statistics
Notes
  • So what I've learned from this is that before getting to the base system and actually coding I should try the simple path and just convert! What an amazing victory for you!
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
5,729
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
108
Comments
1
Likes
9
Embeds 0
No embeds

No notes for slide
  • Beginning Kindle Hackery

    1. 1. You can take the Hacker out of Perl...
    2. 2. ...but you can’t take the Perl out of the hacker.
    3. 3. Laziness, Impatience and Hubris on the Kindle
    4. 4. This talk contains almost no Perl.
    5. 5. ...and almost no Japanese.
    6. 6. I’m very sorry.
    7. 7. And no x86 ASM
    8. 8. I’m not sorry.
    9. 9. The Amazon Kindle
    10. 10. eBook Reader
    11. 11. eInk Screen
    12. 12. 800x600; 16 levels of grey
    13. 13. No backlight
    14. 14. Looks great outside!
    15. 15. Good things about the Kindle
    16. 16. 3G for 1-click shopping
    17. 17. 250,000+ books to buy
    18. 18. (Most < USD10)
    19. 19. Newspaper, Magazine, Blog subscriptions
    20. 20. Delivered every morning
    21. 21. $1-$10 each month
    22. 22. Email → eBook conversion
    23. 23. $0.10 for autodelivery
    24. 24. $FREE if you copy the result by USB
    25. 25. Free Web Browser
    26. 26. Download .mobi, .prc, .azw, .txt for free
    27. 27. “Experimental”
    28. 28. Means: We might start charging money for this
    29. 29. NetFront 3.5
    30. 30. Basic CSS
    31. 31. Javascript
    32. 32. XmlHttpRequest
    33. 33. Free Wikipedia
    34. 34. Not “experimental”
    35. 35. Text to speech.
    36. 36. Minesweeper!
    37. 37. Bad things about the Kindle
    38. 38. DRM
    39. 39. When you buy books, they are locked to the Kindle
    40. 40. Amazon lawyers went after a tool to let you read non- Amazon DRMed ebooks
    41. 41. (The same tool can help you remove the DRM from books Amazon sells you)
    42. 42. Limited eBook formats
    43. 43. I want to read PDFs
    44. 44. I want to read ePubs
    45. 45. I want to read Manga
    46. 46. Actually, I don’t read manga
    47. 47. But lots of my friends do
    48. 48. Surveillance
    49. 49. In the US, 3G includes GPS
    50. 50. The Kindle has a GPS
    51. 51. Device logs are sent to Amazon
    52. 52. Including many user actions
    53. 53. ...like the websites you visit
    54. 54. ...and what books you read
    55. 55. Appears to include GPS info
    56. 56. Amazon knows where you are
    57. 57. Lock-in
    58. 58. Designed to work only with Amazon
    59. 59. 3G use is free, but only while Amazon likes you
    60. 60. Summary: The Kindle is an “appliance”
    61. 61. You wouldn’t hack a book, would you?
    62. 62. I got a Kindle to read books
    63. 63. I didn’t plan to hack it
    64. 64. I really wanted to read books in other formats
    65. 65. I’m a sucker for sexy platforms
    66. 66. And it was begging me
    67. 67. It’s a new toy.
    68. 68. I am going to hack it.
    69. 69. Laziness
    70. 70. http://igorsk.blogspot.com
    71. 71. Hidden debug commands
    72. 72. ;debugOn
    73. 73. ‘help
    74. 74. Hubris
    75. 75. “I can’t possibly brick my Kindle with the keyboard, right?”
    76. 76. So, I started typing commands.
    77. 77. “‘usbNetwork” sounds good.
    78. 78. ...nothing happened
    79. 79. How about “‘usbQa’”?
    80. 80. It turned off the WIFI
    81. 81. ..and turned off USB Disk mode.
    82. 82. Impatience
    83. 83. I gave up
    84. 84. Sometimes laziness wins
    85. 85. An hour later, I rebooted my Macbook Air
    86. 86. So I set up a DHCP server
    87. 87. Nothing...
    88. 88. sh-3.2# tcpdump -i en1 listening on en1, link-type EN10MB (Ethernet), capture size 96 bytes [ ...] 12:36:15.238229 arp who-has 192.168.15.200 tell 192.168.15.244
    89. 89. Now my Kindle can tether through my Macbook
    90. 90. I want to read other eBook formats - attempt #1
    91. 91. The Kindle has a browser
    92. 92. I have a web server
    93. 93. Web based proxy
    94. 94. Small perl app
    95. 95. mobiperl
    96. 96. Perl 4
    97. 97. no strict;
    98. 98. no warnings;
    99. 99. global variables
    100. 100. hmm. no ePub support
    101. 101. Spent a weekend learning how ePub format works
    102. 102. github.com/obra/ unsavory-epub-hacks
    103. 103. Slow. Annoying. Requires a Server
    104. 104. Servers are evil
    105. 105. Hm.
    106. 106. Back to the drawing board
    107. 107. I want to read other eBook formats - attempt #2
    108. 108. Let’s review what we know:
    109. 109. What’s inside the Kindle?
    110. 110. 800x600 eInk screen
    111. 111. You know about the screen
    112. 112. Freescale iMX31
    113. 113. ARM1136JF-S
    114. 114. (Includes FPU)
    115. 115. + Multimedia stuff
    116. 116. 2 GB Flash
    117. 117. 128 MB RAM
    118. 118. USB OTG + MicroUSB slot
    119. 119. Audio hardware
    120. 120. Keyboard
    121. 121. The Kindle sounds like a computer, not a book
    122. 122. It MUST use some GPL code...
    123. 123. https://www.amazon.com/gp/help/ customer/display.html?inodeId=200203720
    124. 124. gplrelease.tar.gz
    125. 125. alsa-lib-1.0.13 gcc-4.1.2 module-init-tools-3.2.2_patch alsa-lib-1.0.13_patch glib-2.12.9 monit-4.9 alsa-utils-1.0.13 glibc-2.5 mtd-utils-1.0.0 alsa-utils-1.0.13_patch gst-plugins-base-0.10.17 picocom-1.4 base-files-3.0.14.ipk gst-plugins-base-0.10.6 powertop-1.10 base-passwd_3.5.9 gstreamer-0.10.17 procps-3.2.7 binutils-2.17.50.0.5 hotplug-2004_09_20 procps-3.2.7_patch bonnie++-1.03c ifupdown_0.6.8 readline-4.3 bootchart-0.9 iptables-1.3.3 syslog-ng-1.6.11 busybox-1.7.2 klibc-1.5 sysvinit-2.86 dosfstools-2.11 libol-0.3.18 taglib-1.5 e2fsprogs-1.38 linux-2.6.22-lab126 uboot-1.3.0-rc3 e2fsprogs-1.38_patch lrzsz-0.12.20 udev-112 fuse-2.7.1 lzo-1.08 util-linux-2.12r fuse-2.7.1_link module-init-tools-3.2.2
    126. 126. Linux
    127. 127. I can work with this
    128. 128. But how do I get code onto it?
    129. 129. My friend nmap tells me...
    130. 130. The Kindle listens on a few ports.
    131. 131. None of them love me at all
    132. 132. I guess I’ll need to take matters into my own hands.
    133. 133. Hey, the Kindle has busybox
    134. 134. busybox has telnetd
    135. 135. Maybe I just need to install /etc/rc5.d/S99telnetd
    136. 136. More research from http://igorsk.blogspot.com
    137. 137. Kindle 1 update extractor
    138. 138. (Python script)
    139. 139. But I want to make new updates...
    140. 140. I reverse engineered the reverse engineering tool
    141. 141. Updates are a short header, an MD5 and a tarball
    142. 142. ...run through a trivial cipher
    143. 143. they’re not encrypted
    144. 144. The Kindle2 is a little different than the Kindle1
    145. 145. It has different magic #s in the update header
    146. 146. I waited for the first Kindle 2 “update.bin”
    147. 147. I grabbed its header
    148. 148. I built a new “update”
    149. 149. It installed one file
    150. 150. /etc/rc5.d/S99telnetd
    151. 151. /bin/busybox telnetd -p 2323
    152. 152. ...nope.
    153. 153. I ran strings on the Kindle’s busybox
    154. 154. No telnetd!
    155. 155. Where can I get a busybox for the Kindle?
    156. 156. What else has a a similar CPU?
    157. 157. My gPhone!
    158. 158. Lots of people built static busybox for the gPhone
    159. 159. telnetd: take 2
    160. 160. login:
    161. 161. login: root Password: Login incorrect
    162. 162. /bin/sh makes a better /bin/login
    163. 163. 125-6-81-160:ß jesse$ telnet kindle 2323 Trying 192.168.15.244... Connected to kindle. Escape character is '^Ü'. / # cat /etc/motd ############################################### ## # NOTICE * NOTICE * NOTICE # ############################################### ## Rootfs is mounted read-only. Invoke mntroot rw to switch back to a writable rootfs. ############################################### ## /#
    164. 164. What I found:
    165. 165. Most of /sbin is written in sh
    166. 166. Fun stuff in /proc
    167. 167. /proc/config.gz
    168. 168. # # Automatically generated make config: don't edit # Linux kernel version: 2.6.22.19 # Mon Mar 2 12:13:07 2009 # CONFIG_ARM=y CONFIG_SYS_SUPPORTS_APM_EMULATION=y # CONFIG_GENERIC_GPIO is not set CONFIG_GENERIC_TIME=y CONFIG_GENERIC_CLOCKEVENTS=y CONFIG_MMU=y # CONFIG_NO_IOPORT is not set CONFIG_GENERIC_HARDIRQS=y CONFIG_STACKTRACE_SUPPORT=y CONFIG_LOCKDEP_SUPPORT=y ...
    169. 169. ...I could rebuild the Kernel
    170. 170. /proc/filesystems
    171. 171. nodev sysfs nodev rootfs nodev bdev nodev proc msdos nodev sockfs vfat nodev pipefs nodev nfs nodev nodev rpc_pipefs anon_inodefs nodev fuse nodev futexfs fuseblk nodev tmpfs nodev fusectl nodev inotifyfs nodev devpts ext3 nodev ramfs
    172. 172. nodev sysfs nodev rootfs nodev bdev nodev proc msdos nodev sockfs vfat nodev pipefs nodev nfs nodev nodev rpc_pipefs anon_inodefs nodev fuse nodev futexfs fuseblk nodev tmpfs nodev fusectl nodev inotifyfs nodev devpts ext3 nodev ramfs
    173. 173. I’m not restricted to 2GB
    174. 174. It’s a Linux box
    175. 175. I can cross-compile!
    176. 176. http://www.codesourcery.com/ Prebuilt ARM toolchain
    177. 177. It generates generic ARM machine code
    178. 178. That’s ok, but not great
    179. 179. I’ll cross compile Perl
    180. 180. 1 day of frustration passes
    181. 181. I won’t cross-compile Perl
    182. 182. Crosscompiling Perl == Bad Joke
    183. 183. I’ll cross-compile Python
    184. 184. Same bad joke
    185. 185. Maybe I need a native compiler for ARM
    186. 186. Where do I get an ARM build farm?
    187. 187. I have a gPhone
    188. 188. It’s not a great build host
    189. 189. I have an N810
    190. 190. It’s not a great build host...
    191. 191. ...but it has an important advantage
    192. 192. apt-get install gcc
    193. 193. N810: Linux 2.6; glibc 2.5
    194. 194. N810 binaries run unmodified on the Kindle
    195. 195. I built perl in an hour
    196. 196. Sadly, I realized that Python is a better choice
    197. 197. I also realized that building on the Kindle works better than on the N810.
    198. 198. (Version skew in extra libraries makes things hard)
    199. 199. I tried building gcc on the N810...
    200. 200. Found Pengutronix / OSELAS.de
    201. 201. It’s a compiler toolchain builder.
    202. 202. I built my own crosscompilers for ARM1136JF-S - Linux 2.6 - glibc 2.5
    203. 203. I used the cross compiler to compile gcc, glibc (for proper headers), binutils, shellutils, dropbear & screen
    204. 204. I cross-compiled nfsmount
    205. 205. I nfs-mounted a disk image with the compiler
    206. 206. Then I started building more stuff
    207. 207. I am a Perl Hacker
    208. 208. I believe in the three virtues
    209. 209. Lazyness
    210. 210. Impatience
    211. 211. Hubris
    212. 212. Sometimes, Perl isn’t the Right Tool
    213. 213. Calibre is the Killer App for ebook conversion and management
    214. 214. Lazyness
    215. 215. I like the best tools
    216. 216. The best tools already exist
    217. 217. Calibre has dozens of eBook format converters.
    218. 218. Why reimplement them?
    219. 219. Hubris
    220. 220. “I can learn enough Python in a weekend to port this application to the Kindle”
    221. 221. The downside
    222. 222. Dependencies
    223. 223. Who’s dealt with Python app dependencies?
    224. 224. No CPAN.
    225. 225. Everything you need is in the Standard Library.
    226. 226. If it’s not in the Standard Library, it’s not worth using.
    227. 227. Except when you need it.
    228. 228. They have....
    229. 229. “easy_install”
    230. 230. It’s not so easy
    231. 231. It is very perlish
    232. 232. It does recursive web scraping to find tarballs on developers’ web sites.
    233. 233. Most of the deps actually installed ok.
    234. 234. I just ran the app over and over until it stopped erroring.
    235. 235. God I miss Perl.
    236. 236. And then we get to the big problem.
    237. 237. Qt
    238. 238. Calibre’s UI is in Qt
    239. 239. ...so its backend uses Qt because it’s easy
    240. 240. PyQt binds Qt to Python
    241. 241. For Qt for Windows
    242. 242. for Qt for Mac
    243. 243. for Qt for X11
    244. 244. No X11 on Kindle
    245. 245. (Just a Framebuffer)
    246. 246. QtEmbedded
    247. 247. No problem!
    248. 248. No PyQtEmbedded
    249. 249. Finally got Calibre running...
    250. 250. by hacking out components I don’t need.
    251. 251. It was good enough to try to convert a trivial ebook.
    252. 252. It took 12 hours...
    253. 253. ...after I built swaputils and gave it 256MB of swap
    254. 254. So what was it doing?
    255. 255. HTML → Mobipocket converter
    256. 256. With a full CSS engine
    257. 257. It visits every DOM element...
    258. 258. and computes CSS styles to convert them to trivial HTML 3.2...
    259. 259. ...twice.
    260. 260. Lazyness, Impatience, Hubris can all help here.
    261. 261. Help me Larry-wan.
    262. 262. Very few CSS rules really matter.
    263. 263. The Kindle supports very little HTML.
    264. 264. It mostly supports HTML 3.2...just no <pre>
    265. 265. ...that’s the only thing the 12 hour CSS engine got us
    266. 266. You can emulate <pre> with <tt> and &nbsp;
    267. 267. if tag == 'pre': self.inside_pre = 1 tag = 'tt' if prefixname(elem.tag, nsrmap) == 'pre': buffer.write('<br/>n') self.inside_pre = 0 if self.inside_pre: text=text.replace(' ','&nbsp;') text=re.sub(r'(rn|r|n)', '<br/>n', text)
    268. 268. Now it runs in 60 megs and about 10 minutes
    269. 269. So, now I can run code.
    270. 270. Still no UI access.
    271. 271. I don’t really want to hack Java GUI code.
    272. 272. And where could I plug my custom UI into the Kindle’s?
    273. 273. I don’t want to break Amazon’s UI.
    274. 274. Oh hey.
    275. 275. There is an application I could replace with something custom...
    276. 276. But really, I don’t want to.
    277. 277. Sure, I could decompile.
    278. 278. It’s obfuscated.
    279. 279. It’d be annoying.
    280. 280. If I built UI, I’d have to maintain a UI.
    281. 281. And users can break a UI.
    282. 282. No buttons = Less to screw up
    283. 283. But I have this ebook converter.
    284. 284. I do want to let users convert books.
    285. 285. What to do?
    286. 286. nodev sysfs nodev rootfs nodev bdev nodev proc msdos nodev sockfs vfat nodev pipefs nodev nfs nodev nodev rpc_pipefs anon_inodefs nodev fuse nodev futexfs fuseblk nodev tmpfs nodev fusectl nodev inotifyfs nodev devpts ext3 nodev ramfs
    287. 287. nodev sysfs nodev rootfs nodev bdev nodev proc msdos nodev sockfs vfat nodev pipefs nodev nfs nodev nodev rpc_pipefs anon_inodefs nodev fuse nodev futexfs fuseblk nodev tmpfs nodev fusectl nodev inotifyfs nodev devpts ext3 nodev ramfs
    288. 288. Inotify blocks on filesystem events.
    289. 289. pyInotify lets me get at fs events easily.
    290. 290. class InotifyListener (threading.Thread): global cv def run ( self ): global conversionQueue wm = WatchManager() # Watch Manager mask = IN_MOVED_TO | IN_CREATE # watched events p = PTmp() notifier = Notifier(wm, p) wdd = wm.add_watch('/mnt/us/documents', mask, rec=True) notifier.loop()
    291. 291. It works great for downloads
    292. 292. Copies over USB didn’t trigger inotify events.
    293. 293. It’s probably something fuse-related.
    294. 294. I went for the cheap hack.
    295. 295. When you eject the Kindle, it generates a DBus event.
    296. 296. class DbusWatcher (threading.Thread): global cv def run ( self ): global conversionQueue cmd='/usr/bin/dbus-monitor --system' pipe = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE).stdout while 1: line = pipe.readline() if any(line.find(i) != -1 for i in ['usbPlugOut', 'resuming']): for f in os.listdir('/mnt/us/documents'): maybe_enqueue_file('/mnt/us/documents/'+f)
    297. 297. What’s next?
    298. 298. Remember config.gz?
    299. 299. I can build a new kernel
    300. 300. ...and add back missing drivers
    301. 301. USB Mass Storage Host
    302. 302. USB WIFI?
    303. 303. What isn’t next?
    304. 304. Reverse engineering Java to extend the Kindle’s UI
    305. 305. Python, and Shell, I’m happy to hack for a good cause.
    306. 306. Java is another matter entirely.
    307. 307. Thanks!
    308. 308. I had a big finish planned.
    309. 309. I was going to build and show off a manga converter.
    310. 310. (for .cbz format books)
    311. 311. So I downloaded a .cbz.
    312. 312. ...and copied it to the Kindle...
    313. 313. ...and I saw this...
    314. 314. The best hacking is no hacking.
    315. 315. Thanks!
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×