Innovate – Tweak, Hack, and Bend Technology

6,522 views

Published on

We will take a small device (available for $100 at Dallas Semiconductor), about the size of an iPod Shuffle, but capable of executing Java-Byte-Code, and connect it to a digital toy camera, (available for about $10 on eBay).
Then we are going to write some Java code that we load on to the device and execute and even run some JUnit tests remotely. But be warned, we are targeting an 8-bit processor running at only 40 MHz and its 1MB SRAM serves as file-system, program memory, and runtime heap.
Along the way we may add some more hardware and learn a few things about how CMOS sensors capture image data and how a Bayer-pattern can be used to decode the sensor data.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
6,522
On SlideShare
0
From Embeds
0
Number of Embeds
3,368
Actions
Shares
0
Downloads
9
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Innovate – Tweak, Hack, and Bend Technology

  1. 1. Innovate: Tweak, Hack, and Bend Technology Wolf Paulus . com Wolf Paulus
  2. 2. Ren • ais • sance Renaissance The period of European history at the close of the Middle Ages and the rise of the modern world; a cultural rebirth from the 14th through the middle of the 17th centuries.© 2006 Wolf Paulus 2
  3. 3. © 2006 Wolf Paulus 3
  4. 4. Glass Gave Renaissance Paintings Their Glow By looking beyond the limits of their usual practice and transforming materials from other trades, the great artists of the Renaissance created a palette that gave them an immediate and lasting reputation as brilliant colorists.© 2006 Wolf Paulus 4
  5. 5. Innovate - Tweak, Hack, and Bend Technology. • To tweak, hack, and bend technology you need to know some of the Technology behind the Technology • Look beyond the limits of your usual practice and transform materials and ideas from others.© 2006 Wolf Paulus 5
  6. 6. Hacking Nick Click, a digital toy camera.© 2006 Wolf Paulus 6
  7. 7. Power: 9V Battery Shutter Serial Power Port : TX Ground RX© 2006 Wolf Paulus 7
  8. 8. 9V Power On/ Off Shutter Serial Port : TX Ground RX© 2006 Wolf Paulus 8
  9. 9. Microcontroller vs. Microprocessor • Microcontroller chips and most embedded devices embody the Harvard architecture, which has separate memories for programs and data. • Microprocessor based machines usually have a von Neumann architecture, with a single memory for both.© 2006 Wolf Paulus 9
  10. 10. Image Sensor, Key Features: • Effective Resolution 160 x 120 • Color filters form a Bayer Color Pattern • “Software is required to perform color processing.” [Source: 50 pages ST VV5301 & VV6301 Mono and Color QSIF Digital Video CMOS Image Sensors]© 2006 Wolf Paulus 10
  11. 11. Filter Array Sensor • Only one color can be acquired per pixel. • Most commonly used is the Bayer Pattern because of the eye’s higher sensitivity to green, 50% of the Bayer pattern consists of green information.© 2006 Wolf Paulus 11
  12. 12. Decoding the Bayer Pattern© 2006 Wolf Paulus 12
  13. 13. Shuffled Pixel Readout .. Crosstalk: Parasitic capacitance between tracks can cause the fast edges of high- speed logic to propagate from one signal line to another.© 2006 Wolf Paulus 13
  14. 14. .. leads to shorter Tracks LiNbO3 Silica-on-Silicon waveguides A B C D 1 1 1 1 A 1 B 1 E1 F1 G1 H1 C D 1 1 E 1 F 1 G A2 B2 C2 D2 1 H 1 A 2 E2 F2 G2 H2 B C 2 2 D 2 E 2 F 2 G Inputs 2 A3 B3 C3 D H 2 3 A 3 B E3 F3 G1 H3 3 C 3 D 3 E 3 F 3 G A4 B4 C4 D4 1 H 3 A 4 E4 F4 G4 H4 B 4 C 4 D 4 E 4 F 4 G 4 H 4 A1 B2 C1 D3 A 1 E1 F1 G2 H4 B C D 2 1 3 E 1 F 1 G A2 B1 C2 D2 2 H 4 A 2 B E2 F2 G1 H3 1 C 2 D 2 E 2 Outputs A4 B3 C3 D4 F G H 2 1 3 A 4 E3 F3 G3 H2 B C D 3 3 4 E 3 F 3 G A3 B4 C4 D1 1 H 2 A 3 E4 F4 G4 H1 B C D 4 4 1 E 4 F 4 G 4 H 1 8- 1x8 64- 1x4 Folded 128 x 128 WDM switch array Shuffle circuit© 2006 Wolf Paulus 14
  15. 15. Artifacts due to color interpolation© 2006 Wolf Paulus 15
  16. 16. Layered Image Sensor The red, green, and blue photodetectors are located at different depths within the sensor.© 2006 Wolf Paulus 16
  17. 17. Layered Image Sensor The red, green, and blue photodetectors are located at different depths within the sensor. For every pixel, three photodiodes are stacked like the three layers in color film. This arrangement utilizes the wavelength-dependent light absorption property of silicon to produce natural filters that use the incoming light to greatest advantage.© 2006 Wolf Paulus 17
  18. 18. Filter Array vs. Layered Sensor© 2006 Wolf Paulus 18
  19. 19. Communication Protocol Data STX Command Data 1 ... ETX n Data ACK STX Response Data 1 ... ETX n • STX = ASCII character (02h) • Command = defined data byte • Response = defined response byte • Data = command specific parameter data • ETX = ASCII character (03h) • ACK = Acknowledge (06h) • NAK = (15h)© 2006 Wolf Paulus 19
  20. 20. Example 1: Request Firmware Version Send Request via RS232 port at 57.6 kbps / 8bits / No Parity 02h V 0 03h Expected Response: 06h 02h v 1 . 6 9 03h© 2006 Wolf Paulus 20
  21. 21. Example 2: Take a Picture • Reset Image Counter to 0 ❷,A,0,❸ ⇒ ⇐ ❻,❷,a,0,❸ • Grab Image (Parameter defines delay and retries) ❷,G,127,❸ ⇒ ⇐ ❻,❷,g,0,❸ • Upload Image (Parameter defines image index) ❷,U,0,❸ ⇒ ⇐ ❻,❷,u,N1,..,N4, D1,..,Dn,❸ 02h Cmd param 03h 06h 02h cmd error code response .. 03h© 2006 Wolf Paulus 21
  22. 22. Java Code to drive the DigiCam public byte[] perform(byte[] cmd, int size); • cmd - array containing the command code, e.g. STX, CommandByte, ParameterByte(s), ETX • size - number of data bytes expected on inputstream, excluding 4 bytes for ACK STX cmd ETX • return - response bytes or null, if unsuccessful, e.g. ACK, STX, commandByte, DataBytes, ETX© 2006 Wolf Paulus 22
  23. 23. private byte[] perform(final byte[] cmd, final int size) { boolean ok = false; boolean busy = false; int counter = 0; byte[] buffer = new byte[size + 4]; // data + ACK,STX,RESPONSE,ETX try { // Flush the input stream while (in!=null && in.available() > 0) in.read(); } catch (IOException e) { // intented empty } do { try { out.write(cmd); // Send Command out.flush(); busy = false; int off = 0; // the number of bytes already read. int len = buffer.length; // the number of bytes still to read.© 2006 Wolf Paulus
  24. 24. while (len > 0) { // Reads up to len-off bytes from the input stream into buffer. int k = in.read(buffer, off, len); if (k <= 0) // k is the number of bytes actually read. throw new IOException("Couldnt finish reading input file."); off += k; len -= k; } if (buffer[0] != VV6301.ACK) // Expected: ACK,STX,cmd,..,ETX return null; ok = buffer[2] == (byte) (cmd[1] + RESPONSE_OFFEST); if (!ok) busy = buffer[2] == VV6301.RB_BUSY; } catch (IOException ioe) { System.err.println(ioe.getMessage()); } } while (!ok && busy && (++counter < VV6301.RETRIES)); return ok ? buffer : null; }© 2006 Wolf Paulus
  25. 25. Serial Port, still a “must have” • Keyspan High Speed USB-Serial Adapter http://www.keyspan.com/products/usb/USA19HS/ • Java communications libraries for OS X http://prdownloads.sourceforge.net/jmri/ JavaCommInstaller2.hqx • Installation instructions for Mac OS X http://jmri.sourceforge.net/install /MacOSX.html© 2006 Wolf Paulus 25
  26. 26. Dallas Semiconductor: TINI board 390© 2006 Wolf Paulus 26
  27. 27. TINI- Hardware 384K Java Runtime 512K 64K Shell 36 MHz Flash ROM CPU File Sys. Java Prog. Res.Files Ethernet Two 512K Serial-IO Controller SRAM Controller© 2006 Wolf Paulus 27
  28. 28. TINI’s Java Limitations • 8 Processes (incl. GC and Shell) • 32 Threads per Process • 24 Socket Connections • A compiled class file cannot be larger than 32k • 64k is the maximum size of any array© 2006 Wolf Paulus 28
  29. 29. more Java Limitations on the TINI • A class is limited to 255 static and 255 instance member variables. (includes inherited members) • A class method is limited to 63 local variables. • Only UTF8 and ISO 8859_1 are supported with ByteToChar and CharToByte • Only the default Locale is supported. • There are a few more limitations that concern Classloading and Reflection.© 2006 Wolf Paulus 29
  30. 30. TINIConverter TINIConverter is a Java Program Command line tool which outputs a *.tini file StringBuffer sb = new StringBuffer(); Java sb.append(“this is a test”); Byte Code Create new object “java.lang.StringBuffer” Pseudo Push reference to “this is a test” on the stack Invoke method “java.lang.StringBuffer.append(String)” Create new object with class number 0x54 TINI File Push reference to “this is a test” on the stack Invoke method 0x12 of class 0x54 Reduces the constant pool and speeds up class loading© 2006 Wolf Paulus 30
  31. 31. Socket for the TINI board 390 Null Modem Serial Port R 330 Ω LED (+) (-) 10-Base-T 5V Power Ethernet in on nex t slide zoom© 2006 Wolf Paulus 31
  32. 32. 330 Ω GREEN LED 330 Ω RED LED VCC = 5VDC t 5 t 5 Bi rt Bi rt 3 0 Po Po© 2006 Wolf Paulus 32
  33. 33. Tweaking Processor Port Pins • Driving an LED needs around 10 mA to make it clearly visible. • A Port Pin can supply about 50 μA but can sink current up to 15 mA. • LEDs have a voltage drop of about 2V. Therefore, the voltage across the Resistor is about 3V. Ohm’s law: U/R=I 3V / 330Ω = 9 mA BitPort led = new BitPort(BitPort.Port5Bit0); led.clear(); // turns it on led.set(); // turns it off© 2006 Wolf Paulus 33
  34. 34. TINI in action ... (typical setup) Nick Click CAT5 Cable Router TINI 390 Serial-Cable (3-wire)© 2006 Wolf Paulus 34
  35. 35. TINI in action ... (development setup) PowerBook 5 VD C Po we r CAT5 Cross Nick Click Over Cable TINI-390 Serial Cable© 2006 Wolf Paulus 35
  36. 36. Demo© 2006 Wolf Paulus
  37. 37. Demo Compact Wireless-G Broadband Router - WRT54GC Gateway 192.168.200.1 / 255.255.255.0 DS80C390: 192.168.200.6 DS80C390: 192.168.200.5 DS80C400: 192.168.200.4© 2006 Wolf Paulus 37
  38. 38. TiniNick -- Camera Driver & Image Server Serial Port /Camera Driver Module Access Properties Files accepts client requests, adds new client sockets into Vector Grabs Photos from Nick Click streams new images to all Sockets in Vector© 2006 Wolf Paulus 38
  39. 39. NickViewer -- Image Viewer Client Converts 20KB bayer-encoded byte array in 60KB RGB-Bitmap. Builds Viewer GUI and observes SocketConnector for new images. Opens TCP Socket to TickNick Image Server. Updates Observer when new image arrives.© 2006 Wolf Paulus 39
  40. 40. Shuffled Pixel Readout© 2006 Wolf Paulus 40
  41. 41. Before Bayer Decoding© 2006 Wolf Paulus 41
  42. 42. After Bayer Decoding© 2006 Wolf Paulus 42
  43. 43. Image Resolutions Compared ... Nick Click 160 x 120 1 Motorola V551 640 x 480 16 Canon 2048 x PowerShot 164 SD200 1536 Canon EOS 3456 x 350D / Digital 414 Rebel XT 2304© 2006 Wolf Paulus 43
  44. 44. DS TINIm400 (DS80C400 MC) DS TINIs400 Socket Board© 2006 Wolf Paulus
  45. 45. DS TINIm400 (DS80C400 MC)© 2006 Wolf Paulus
  46. 46. DS TINIm400 (DS80C400 MC)© 2006 Wolf Paulus
  47. 47. DS80C400 MC with 1 MByte integrated Ethernet Flash Mem DS TINIm400 (DS80C400 MC) Controller© 2006 Wolf Paulus
  48. 48. 2 x 512 DS TINIm400 (DS80C400 MC) KByte SRAM© 2006 Wolf Paulus
  49. 49. Hack, Tweak, and Bend Technology • Hacking a $10 Digital Toy Camera. • Tweaking Micro-Controller Ports© 2006 Wolf Paulus 49
  50. 50. On April 26, 1336, Francesco Petrarca climbed to the top of Mont Ventoux just to see the view, which at the time was rather unusual. This event is often cited as the start of the Renaissance. Mont Ventoux (1,909 m; 6,263 ft) is by far the biggest mountain in the Provence, a region of southern France.© 2006 Wolf Paulus 50
  51. 51. Resources • Textbooks • Don Loomis - TINI Specification and Developers Guide - ISBN: 0201722186 PDF version online: http://www.maxim-ic.com/products/tini/devguide.cfm • Ken Arnold - Embedded Controller Hardware Design - ISBN: 1878707523 • DeMuth, Eisenreich, Designing Embedded Internet Devices, ISBN: 1-878707-98-1 • Image Processing / Bayer Pattern • Tommy Olsen, Jo Steinar Strand, Agder University College Faculty of Engineering: "An improved image processing chain for mobile terminals", May 2002 http://student.grm.hia.no/master/ikt02/ikt6400/g23/Report.htm • Diagnostic Instruments, Inc.(http://www.diaginc.com) Elements of Sensor Resolution http://www.diaginc.com/downloads/public/pdf/3ShotColorMosaic.pdf • Data sheets and Drivers • CPiA webcam driver for Linux (http://webcam.sourceforge.net) RS232 command protocol specs (scicomms.doc, 10 pages, 82Kb). http://webcam.sourceforge.net/barbie/scicomms.doc • VV6301 Mono and Colour QSIF Digital Video CMOS Image Sensors http://www.datasheetarchive.com/search.php?search=vv6301&sType=part • Tini DS80C390 and Tini DS80C400 http://www.maxim-ic.com/pl_list.cfm/filter/tini • TINI, DS80C400 Evaluation Kit • http://www.maxim-ic.com/getds.cfm?qv_pk=4983 • http://www.maxim-ic.com/TINIplatform.cfm© 2006 Wolf Paulus 51
  52. 52. The End© 2006 Wolf Paulus
  53. 53. One more thing ...© 2003-2006 Carlsbad Cubes© 2006 wolfpaulus.com
  54. 54. TiffanyScreens
  55. 55. Tiffany Screens The term "democracy" was coined in ancient Athens in the 5th century BC. All Athenian citizens were free to speak in the Assembly.© 2003-2006 Carlsbad Cubes© 2006 wolfpaulus.com
  56. 56. “PowerPoint-ation”• Lecture Style Presentation • One guy presents everybody else is asleep• Changing Presenters severely interrupts a meeting, requiring change of location, unplugging, etc. • Presenter need to adjust Laptop’s screen resolution and refresh-rate.
  57. 57. 100 75 50 25 2004 0 2005 2006 2007• Since a picture is worth a 1000 words, it’s hard to win an argument against the presenter, even if you have your facts right.
  58. 58. TiffanyScreens• Detects who is running the same application. aka your Peer Group• One peer at a time broadcasts her screen onto other watching peers’ screens• Seamlessly changing broadcasters
  59. 59. Group Dictionary
  60. 60. Toolbar• Show your screen to your peers - exit [ESC]• Watch the presenter’s screen - exit [ESC]• Pick your Conference room• Look who’s currently where, doing what• Change your network settings etc. - if you really think you have to -
  61. 61. Statusbar• Status LED • blue = idle • green = ready to receive • orange = receiving• heartbeat - sharing your apps attributes• Confidentiality features• Number of peers, running this app.• Preferred watching window resolution
  62. 62. TiffanyScreens.com© 2003-2006 Carlsbad Cubes© 2006 wolfpaulus.com
  63. 63. Thanks for coming© 2003-2006 Carlsbad Cubes© 2006 wolfpaulus.com

×