Using Python3 to Build a Cloud Computing Service for my Superboard II

5,610
-1

Published on

The OSI Superboard II was the computer on which I first learned to program back in 1979. Python is why programming remains fun today. In this tale of old meets new, I describe how I have used Python 3 to create a cloud computing service for my still-working Superboard--a problem complicated by it only having 8Kb of RAM and 300-baud cassette tape audio ports for I/O.

Published in: Technology
1 Comment
7 Likes
Statistics
Notes
No Downloads
Views
Total Views
5,610
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
180
Comments
1
Likes
7
Embeds 0
No embeds

No notes for slide

Using Python3 to Build a Cloud Computing Service for my Superboard II

  1. 1. Using Python 3 to Build aCloud Computing Service for my Superboard II David Beazley (http://www.dabeaz.com) Presented at PyCon 2011 Atlanta
  2. 2. Note: This talk involves a number of livedemonstrations. You will probably enjoy it more by watching the PyCon 2011 video presentation http://pycon.blip.tv/file/4878868/
  3. 3. It Begins (Drumroll...)
  4. 4. Byte Magazine, January, 1979
  5. 5. Personal History• Superboard II was my familys first computer• What I first learned to program (~1979)• It had everything that a kid could want
  6. 6. Easy Setup!
  7. 7. Peripherals!
  8. 8. Potentially Lethal Suggestions!
  9. 9. Schematics!
  10. 10. You could get an electric shock You couldlook inside!
  11. 11. Pure Awesome! You could program
  12. 12. Pure Awesome! You could hack! Note the encouragement
  13. 13. Backstory• 1982-2010. Superboard sits in moms basement• July 2010. Eric Floehr mentions SB at Scipy• August 2010. Brother brings SB to Chicago• It still works! (to my amazement)
  14. 14. Question• What do you do with an old Superboard II • 1 Mhz 6502 CPU • 8K RAM • 8K Microsoft Basic (vers. 1.0) • 300 Baud Cassette Audio Interface • 1K Video Ram (24x24 visible characters)• Not a modern powerhouse
  15. 15. TO THE CLOUD!
  16. 16. HOW?(with Python 3 awesomeness of course) (plus some ØMQ and Redis) (and some 6502 assembler)
  17. 17. Getting to the Cloud This is the only I/O A pair of RCA audio jacks Used for cassette tape storage Not too promising
  18. 18. A Possible Solution
  19. 19. Cassette Interface • Behind the RCA jacks, sits a real serial "port" • Motorola 6850 ACIA • Ports are a 300 baud audio stream encoded via Kansas City Standard • Might be able to hack it
  20. 20. Byte, Feb. 1976
  21. 21. Python Audio Processing• pyaudio• http://http://people.csail.mit.edu/hubert/pyaudio/• A Python interface to the portaudio C library• Allows real-time access to audio line in/out• I ported it to Python 3
  22. 22. Reading Audio Inputimport pyaudiop = pyaudio.PyAudio()stream = p.open(format=pyaudio.paInt8, channels=1, rate=9600, chunksize=1024, input=True)while True: sample = stream.read(1024) process_sample(sample)
  23. 23. Building a Python Modem writer thread pyaudio byte writer encoder TCP client socket pyaudio byte reader decoder reader thread• A network socket bridged to two real-time audio encoding/decoding threads
  24. 24. KCS Audio Encoding 0= 1= (4 cycles @ 1200 Hz) (8 cycles @ 2400 Hz) A = 0x41 (01000001) 0 0 1 0 0 0 0 0 1 1 1start data stop
  25. 25. KCS Audio Decoding samples (8-bit)89, 103, 117, 151, 194, 141, 99, 64, 89, 112, 141, 203, 152, 107, 88, ... sign bits (0 = neg, 1= pos)000111000111000111000111000111111000000111111000000111111000000111111... sign changes (XOR adjacent bits)0001001001001001001001001001000001000001000001000001000001000001000001...Idea: Each 1 bit represents a 0-crossing
  26. 26. Example Codedef generate_sign_change_bits(stream): previous = 0 while True: samples = stream.read(CHUNKSIZE) # Emit a stream of sign-change bits for byte in samples: signbit = byte & 0x80 yield 1 if (signbit^previous) else 0 previous = signbit
  27. 27. KCS Bit Decoding idle (constant 1) 1 0 11 0 0 0 0 start data byte stopSign Change Bits push1001001001001001001001001001 1001001001001001 discard Sample buffer (size = audio samples for 1 bit) ~ 8 sign changes ~ 16 sign changes 0 1
  28. 28. Example Codefrom collections import deque# Sample buffersample = deque(maxlen=SAMPLES_PER_BIT)for b in generate_sign_change_bits(stream): sample.append(b) nchanges = sum(sample) if nchanges < 10: # Detected a start bit # Sample next 8 data bits ... (The actual code is more optimized)
  29. 29. Demo My Mac (linked via audio)
  30. 30. Interlude• Its Alive!• Basic communication is just the start!
  31. 31. This is uploading machine code~500 lines of 6502 assembler ... getvar_copy_string: ! ;; Move the variable address to INDIRECT where we c ! LDA![0x95, Y]! ; Length ! STA!%MEM_LENGTH ! INY ! LDA![0x95, Y]! ; address (low) ! STA!%INDIRECT ! INY ! LDA![0x95, Y]! ; address (high) ! STA!%INDIRECT+1 ! LDY!#0x00 ... Wrote a 6502 assembler ~500 lines of Python 3
  32. 32. Under the coversThis is uploading machine code msgdrv.asm~500 lines of 6502 .1C00/20 assembler asm6502.py 05 ... AE getvar_copy_string: EE ! ;; Move the variable address to INDIRECT where we c 1E ! msgdrv.hex Y]! ; Length LDA![0x95, 1E ! STA!%MEM_LENGTH AD ! INY 1E ! LDA![0x95, Y]! ; address (low) 1E ! STA!%INDIRECT 8D ! INY ! pymodem Y]! ; address (high) LDA![0x95, ! STA!%INDIRECT+1 ! LDY!#0x00 ... Wrote a 6502 assembler ~500 lines of Python 3
  33. 33. Messaging Driver code Under the coversThis is uploading machine request msgdrv.asm~500 lines of 6502 .1C00/20 Client Superboard II control assembler asm6502.py 05 ... AE• Superboard issues requests getvar_copy_string: ! ! msgdrv.hex Y]! ; Length LDA![0x95, EE ;; Move the variable address to INDIRECT where we c 1E 1E• Client responds and gets control ! ! ! STA!%MEM_LENGTH INY AD 1E LDA![0x95, Y]! ; address (low) 1E• Driver coexists with BASIC ! STA!%INDIRECT 8D ! INY ! pymodem Y]! ; address (high) LDA![0x95, ! STA!%INDIRECT+1 1024LDY!#0x00 message driver ! bytes ... Wrote a 6502 Workspace 7168 bytes BASIC assembler ~500 lines of Python 3
  34. 34. Example of makingcovers Messaging Driver code Under the a requestThis is uploading machine request 10 A = 96 msgdrv.asm~500 lines Superboard II of 6502 assembler control Client asm6502.py 20 B = 42 .1C00/20 05 ... AE• SuperboardY]!issues1Erequests getvar_copy_string: = "FOO" ! ! 30 the variable address to INDIRECT where we c ;; Move msgdrv.hex C$ 40 S =; Length LDA![0x95, EE USR(37) 1E• Client responds and gets control ! Client ! ! STA!%MEM_LENGTH INY control 1E AD LDA![0x95, Y]! ; address (low) 1E• Driver coexists- Get a BASIC region ! STA!%INDIRECT 8D ! INY PEEK with memory ! pymodem Y]! ; address (high) LDA![0x95, ! ! POKE - Set a memory region STA!%INDIRECT+1 1024LDY!#0x00 message driver bytes ... GET - Get a BASIC variable SET - Set a BASIC variable Wrote a 6502 -Workspace to BASIC RETURN assembler 7168 bytes BASIC Return ~500 lines of shared memory! Distributed Python 3
  35. 35. Messaging Architecture• There are two parts (driver and a client) superboard message client message driver audio socket pymodem BASIC Python 3• Uses a binary message protocol USR(37) : x20 x06 x01 x25 x00 x02 Command Size Seq Data LRC • Protocol details not so interesting
  36. 36. Message Driver• Interacts directly with Microsoft BASIC • Uses "Undocumented" ROM routines • Accesses BASIC interpreter memory• For this, some resources online • http://osiweb.org
  37. 37. Client Architecture• Client bridges Superboard II to the outside world• Uses ØMQ (http://www.zeromq.com)• And pyzmq (http://github.com/zeromq/pyzmq) Message client Servicesaudio socket pymodem ØMQ Python 3
  38. 38. Request Publishing • Requests are broadcast on a ØMQ PUB socket • Retransmitted every few seconds if no response Message clientUSR(37) ØMQ PUB "37 1" "37 1" "37 1" ... To the "Cloud" Python 3 • Requests include the USR code and a sequence #
  39. 39. Service Subscription• Services simply subscribe to the request feed import zmq context = zmq.Context() requests = context.socket(zmq.SUB) requests.connect("tcp://msgclient:21001") requests.setsockopt(zmq.SUBSCRIBE,b"37 ")• Now wait for the requests to arrive while True: request = requests.recv()• Clients are separate programs--live anywhere
  40. 40. Request Response • Message client has a separate ØMQ REP socket • Used by services to respond Message clientUSR(37) ØMQ PUB "37 1" driver ØMQ REP Service commands (subscribed to 37) Python 3 • Service initially acks by echoing request back • On success, can issue more commands
  41. 41. Command Connection• Setting up the command socket commands = context.socket(zmq.REQ) commands.connect("tcp://msgclient:21002")• Complete request/response cycle while True: request = requests.recv() commands.send(request) # Echo back resp = commands.recv() if resp[:2] == bOK: # In control of Superboard # Do evil stuff ...
  42. 42. Commands• Commands are just simple byte strings b"RETURN VALUE" b"PEEK ADDR SIZE" b"POKE ADDR DATA" b"GET VARNAME" b"SET VARNAME VALUE"• Response codes b"OK DATA" b"FAIL MSG" b"IGNORE" b"BUSY"
  43. 43. Interactive Demo>>> request_sock.recv()b37 1>>> command_sock.send(b37 1)>>> command_sock.recv()bOK>>> command_sock.send(bGET A)>>> command_sock.recv()bOK 96.0>>> command_sock.send(bGET B)>>> command_sock.recv()bOK 42.0>>> command_sock.send(bSET Q 2)>>> command_sock.recv()bOK>>> command_sock.send(bSET R 12)>>> command_sock.recv()bOK>>> command_sock.send(bRET 0)>>> command_sock.recv()bOK>>>
  44. 44. Big Picture Message client Up to 65536 Service IDs (N) ØMQ PUB USR(N) ØMQ REP Big Iron• Services can live anywhere• Written in any language• No real limit except those imposed by ØMQ
  45. 45. Superboard Emulation• I dumped the BASIC and system ROMS• Loaded them into Py65• Py65 : A 6502 Emulator Written in Python https://github.com/mnaberez/py65• Work of Mike Naberezny• I ported it to Python 3
  46. 46. Emulation in 60 Seconds You start with the Superboard II memory map (provided)
  47. 47. Emulation in 60 Seconds You identify hardware devices
  48. 48. Emulation in 60 Seconds You read (about keyboards)
  49. 49. Emulation in 60 Seconds You read (about video ram)
  50. 50. Emulation in 60 Seconds You read (about ACIA chips)
  51. 51. Emulation in 60 SecondsThen you just plug it into py65 (sic) def map_hardware(self,m): # Video RAM at 0xd000-xd400 m.subscribe_to_write(range(0xd000,0xd400), self.video_output) # Monitor the polled keyboard port m.subscribe_to_read([0xdf00], self.keyboard_read) m.subscribe_to_write([0xdf00], self.keyboard_write) # ACIA Interface m.subscribe_to_read([0xf000], self.acia_status) m.subscribe_to_read([0xf001], self.acia_read) m.subscribe_to_write([0xf001], self.acia_write) # Bad memory address to force end to memory check m.subscribe_to_read([0x2000], lambda x: 0)
  52. 52. Interactive Demo
  53. 53. Question• What do you do with an emulated Superboard?
  54. 54. A Superboard Cloud 10 PRINT "I WILL THINK OF A" 15 PRINT "NUMBER BETWEEN 1 AND 100" 20 PRINT "TRY TO GUESS WHAT IT IS" 25 N = 0 30 X = INT(RND(56)*99+1) 35 PRINT 40 PRINT "WHATS YOUR GUESS "; Program Storage 50 INPUT G 10 PRINT "HELLO WORLD" 20 END 10 FOR I = 1 TO 1000 20 PRINT I 30 NEXT I 20 END Cloud Datastore Service Stored Instances (images of running Virtualized machines) Superboard CPUs
  55. 55. Building The Cloud• I built it using Redis (http://redis.io)• Ported py-redis to Python 3• Redis is cool • Can use it as a key-value store • Has other data structures (sets, hashes, etc.) • Queuing • Atomic operations
  56. 56. Redis Exampleimport redisdb = redis.Redis()# Key-value storedb.set(foo,data)data = db.get(foo)# Queuingdb.lpush(queue,work)work = db.brpop(queue)
  57. 57. Superboard Cloud Features • Remote program store • Load/save programs • Instance creation • Creation • Run with input • Distributed shared memory • It must be usable from the Superboard II
  58. 58. Program Load/StoreBASICstrings msg cloud set driver service getprogram redissettings• BASIC program & workspace memory directly manipulated by the message driver• Stored in Python object and pickled to Redis
  59. 59. Instances• Instances are a running Superboard • 8K Program Memory • 1K Video RAM • Stored CPU context (registers, etc.)• Stored in a Python object• Pickled to Redis when inactive
  60. 60. Instance Execution• "Runner" programs watch a Redis queue import redis r = redis.Redis() ... while True: work = r.brpop("queue") # Wait for work ... inst = load_instance() # Get instance run_emulation(work) # Run emulation save_instance(inst) # Save instance• Based on supplying keyboard input to SB• Instance runs until no more input available
  61. 61. Instance Concurrency• Can have arbitrary number of runners Redis Runners• Asynchronous execution (w/ Superboard)• Uses Redis setnx() for locking
  62. 62. import superboard as skynet pymodem Up to 65536 Service IDs (N) ØMQ PUB ØMQ REP Big Iron Cloud Service programs 10 PRINT "I WILL THINK OF A" 15 PRINT "NUMBER BETWEEN 1 AND Virtualized 100" 20 PRINT "TRY TO GUESS WHAT IT IS" 25 N = 0 Superboard CPUs 30 X = INT(RND(56)*99+1) 35 PRINT 40 PRINT "WHATS YOUR GUESS "; redis 50 INPUT G 10 PRINT "HELLO WORLD" 20 END Stored 10 20 30 FOR I = 1 TO 1000 PRINT I NEXT I Instances 20 END
  63. 63. WHY?!
  64. 64. Non-Answer• I dont actually want to use my Superboard II• Its awful!• It was painful even in 1980
  65. 65. A Better Answer• For fun• Also to create a glorious mess!• Everything a systems hacker could want! • Hardware, device drivers, signal processing, protocols, networks, message passing, threads, synchronization, debugging, software design, testing, deployment, etc.• Awesome!
  66. 66. Real Answer : Python 3• Can Python 3 be used for anything real?• I needed to check it out myself• On a non-trivial project with many moving parts• And with a variety of library dependencies
  67. 67. Lessons Learned• You can use Python 3 to do real work• However, it’s bleeding edge• You really have to know what you’re doing• Especially with respect to bytes/unicode• Must be prepared to port and/or patch
  68. 68. Porting Experience• py65 (easy, some bytes issues)• pyredis (easy, bytes/unicode issues)• pypng (hard, really messy I/O)• pyaudio (hard, C extensions)• pyzmq (worked, Cython patch for Py3.2)
  69. 69. Finding Python 3 Code• Every single package I used was downloaded from development repositories (github, subversion, etc, etc)• You can often find Python 3 compatible libraries in project forks or issue trackers if you look for it
  70. 70. My Thoughts• Python 3 is cool• It keeps getting better >>> import numpy >>>• It’s different and fun• It might be a great language for distributed computing, messaging, and other applications
  71. 71. That’s All Folks!• Thanks for listening!• Look for the “Python Cookbook, 3rd” edition in late 2011!• More info on my blog• http://www.dabeaz.com
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×