c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 6 ( 2 0 0 9 ) 226–233 227image (e.g., edges, edge transitions, grayscale information, image streams before they enter the visual prosthesis (Fig. 3).and spatial frequencies) than an impaired retina in which the Moreover, such image processing systems must provide thephotoreceptor layer is dysfunctional due to diseases such as ﬂexibility of repeated application of image manipulation andretinitis pigmentosa and age-related macular degeneration. processing modules in a user-deﬁned order. Thus, current A more realistic approximation is achieved by employing a and future artiﬁcial vision implant carriers can customizesubject-independent mobile robotic platform that uses a pix- the individual visual perception generated by their visualelated view as its sole visual input for navigation purposes. prostheses, by actively manipulating parameters of individ-Such a mobile robotic platform, described in the follow- ual image processing ﬁlters or altering the sequence of theseing, represents not only a constantly available testbed for ﬁlters.real-time image processing systems, but even more so pro-vides a subject-independent means for testing and validatingthe efﬁciency and utility of real-time image processing and 2. Hardware descriptionautonomous navigation algorithms for enhanced visual per-ception and independent mobility for the blind and visually For the purpose of creating a subject-independent mobileimpaired using artiﬁcial vision prostheses. testbed for image processing and autonomous navigation The current state-of-the-art and near future artiﬁcial algorithms for artiﬁcial vision prostheses, we have cre-vision implants, such as epi-retinal and sub-retinal implants ated CYCLOPS, an All-Wheel Drive (AWD) remote-controllable[1–9] (Fig. 1), provide only tens of stimulating electrodes, robotic platform testbed with wireless Internet connectivitythereby allowing only for limited visual perception (pixela- and a fully articulated digital camera with wireless video linktion). Usually these implants are driven by extraocular [8,9] (Fig. 4) . For the basic robotic hardware we utilized a WiFi-or intraocular  high-resolution digital cameras that ulti- BoT . The WiFiBoT has a 4G Access Cube, which serves asmately result in orders of magnitude smaller numbers of the central onboard processor, controlling four electric motors.pixels that are relayed to the respective implant in use. Hence, We custom-built CYCLOPS by equipping it with:real-time image processing and enhancement will afford acritical opportunity to improve on the limited vision afforded • Bi-level metal chassis and sensor trays.by these implants for the beneﬁt of blind subjects. • General-purpose, high-performance mini Unix workstation Since tens of pixels/electrodes allow only for a very crude (i.e., Mac mini).approximation of the roughly 10,000 times higher optical reso- • Rechargeable battery for the Unix workstation.lution of the external camera image feed, the preservation and • Two rechargeable batteries for the wheel motors.enhancement of contrast differences and transitions, such as • Gimbaled IP camera that is user-controllable.edges, become very important as opposed to picture details • IEEE 1394 navigation camera with wide-angle ﬁeld of view.such as object texture. Image processing systems (Fig. 2), such • Two forward-looking IR proximity sensors.as the Artiﬁcial Vision Simulator (AVS) [11,12], perform real-time • Real-time voice synthesizer interface.(i.e., 30 fps) image processing and enhancement of camera • Wireless Internet capability.Fig. 1 – One instantiation of an artiﬁcial vision prosthesis. An intraocular retinal prosthesis using an externalmicroelectronic system to capture and process image data and transmit the information to an implanted microelectronicsystem. The implanted system decodes the data and stimulates via an electrode array the retina with a pattern of electricalimpulses to generate a visual perception.
228 c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 6 ( 2 0 0 9 ) 226–233Fig. 2 – Schematic diagram of a real-time image processing system for artiﬁcial vision prostheses (e.g., AVS [11,12]) that aredriven by extraocular [8,9] or intraocular  camera systems.
c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 6 ( 2 0 0 9 ) 226–233 229Fig. 3 – Typical palette of image processing modules (e.g., employed by AVS [11,12]) that can be applied in real time to avideo camera stream driving an artiﬁcial visual prosthesis. connects to one or more known “Com Servers” (Fig. 5) . The3. Testbed implementation Com Servers are known, established Internet entities to which both the mobile platform and the controlling computer sys-CYCLOPS supports both interactive tele-commanding via joy- tem connect, acting as a go-between and buffer. In this way,stick, and autonomous self-commanding. Due to its onboard neither end need know the actual IP address of the other, yetcomputing capabilities and battery life (i.e., 4.5 h for the an Internet connection is still established between them, withonboard mini Unix workstation and 2 h for the electric auto-reconnect in case of connection dropouts.motors), CYCLOPS can perform complex and numerically The cloud computing approach affords a great deal of archi-intensive calculations, such as: tectural ﬂexibility; many operational modes are supported, from fully synchronous to fully autonomous. In the case of• Testing and validation of image processing systems, such as the joystick operation of CYCLOPS, the mobile platform is in AVS [11,12], to further the experience of visual prostheses synchronous connection to the Com Server. However, this is users. not strictly required for other modes of operation. Minimally,• Testing of navigation algorithms and strategies to improve constant connectivity is not required, as a temporary connec- the degree of unaided mobility. tion would be sufﬁcient to upload from the mobile platform• Testing of additional sensors (e.g., infrared) to further the video and sensor data resulting from prior command sets, utility of visual prostheses. and to download to the mobile platform further operational commands (e.g., navigation commands), which would be exe-3.1. Cloud computing cuted autonomously. A fully “ofﬂine” autonomous operational capability, i.e., independent real-time onboard processing, isTo enable testing of real-time image processing modules, indi- currently in development for CYCLOPS.vidually or in sequence, and to enable the transmission ofthe resulting remote control navigation sequences, the mobileplatform must establish a connection between itself and the 3.2. Interconnectivitycomputer hosting the control software. A standard, direct one-to-one connection could be established, but this is fragile, as An Internet TCP/IP connection is established between the CPUeither system may not be reachable or known at the moment aboard the mobile platform testbed (via its wireless LAN) andthe connection attempt is made. Instead, a “cloud comput- the computer hosting the image processing and the front-ing” concept is utilized [15,16], wherein the mobile platform end control software via a Com Server. For the purpose of
230 c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 6 ( 2 0 0 9 ) 226–233 reliability, this connection is instantiated by the creation of a temporary AF INET stream socket at the transport layer, uti- lizing a three-way handshake. Once this full-duplex transport layer is accomplished, the mobile platform is able to trans- mit video frames and other sensor data in a packetized and compressed format over the layer. The mobile platform also transmits its housekeeping data (battery level, hardware sen- sor data, etc.) and awaits sensor and drive commands from the front-end software that are triggered by the results of the real-time image processing of the video frames (e.g., via AVS). 3.3. Video and sensor data processing The video and sensor data are treated similarly, however, the video data are ﬁrst preprocessed into a suitable data format. This is accomplished by packetizing the video. Each non- interlaced stream frame of video data is compressed and inserted into the payload portion of a header packet, tagging the data as to type, length, timestamp, and sequence. This has the advantage over time-division multiplexing of allowing for real-time synchronization to occur on the receiving end withFig. 4 – CYCLOPS, an AWD remote-controllable robotic minimal reconstruction processing. The network connectionplatform testbed with wireless Internet connectivity and a is thus used as a virtual N-receiver broadcast channel, eachfully articulated (user-controllable) digital camera with channel being a Q-ary data channel, providing the same mech-wireless video link, as well as an IEEE 1394 navigation anism for video, sensor, and hardware housekeeping data (e.g.,camera with wide-angle ﬁeld of view. It is equipped with a battery charge levels).general-purpose mini Unix workstation. CYCLOPS ispowered by rechargeable batteries. Furthermore, CYCLOPS 3.4. Navigation commandingsupports a sensor bay for additional sensors (e.g., infrared). Navigation and camera command transmittal to the mobile platform testbed is accomplished as follows: The platformFig. 5 – Principle of “cloud computing” [15,16]. A mobile platform connects to one or more known “Com Servers” on theInternet in lieu of direct end-to-end connection with the control system. In this way, neither end need know the actual IPaddress of the other.
c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 6 ( 2 0 0 9 ) 226–233 231utilizes two independent pairs of CPU-controlled electric dependent command sets in the pipeline to be invalidated demotors, each using a 3-gear reduction module, providing a 4:1 facto, thus returning the mobile platform to a “known good”reduction ratio for the increase of applied torque to the wheels state.and reduced power consumption . Each pair of wheelmotors is individually addressable via a command packet that 3.5. User interactivityspeciﬁes motor speed, direction, duration, and speed limiter.Transmittal of the command packet to the mobile platform For the purpose of commanding CYCLOPS interactively (lateris analogous to the preceding, as the existing transport layer autonomously), the front-end software has an integratedis utilized in a full-duplex mode, albeit strictly sequenced video panel (Fig. 6) for displaying the transmitted video framesfor simplicity of processing and performing the commands from the mobile platform’s on-board camera (Fig. 4); it is alsoin order. Commanding the onboard camera follows a similar outﬁtted with a USB-based joystick device. The user, con-procedure. trolling the joystick, is in the loop for the purpose of the It should be pointed out that the current input com- development of automated software and algorithms to con-mand set (e.g., set of navigation commands) for the mobile trol CYCLOPS. Once developed, such automated software canplatform differs in form and function from the video and be “plugged in” in lieu of the user for automatic control, withsensor data, which are received. Input commands require manual user override always available. The user’s movementscomparatively few bytes for expression, thus a simpler, more of the joystick are translated into camera orientation andexpedient architecture is utilized. The input command set wheel rotation commands, and are sent to the mobile plat-for the mobile platform forms a Strictly Ordered Command form. As the mobile platform begins to move, it also sends backPipeline (SOCP) set. Such sets form conditional pipeline branch video, sensor, and housekeeping data, which are displayed onmaps, with sequencing precluding the need for individual the front-end. With this feedback information, a user (or auto-command prioritization. For example, the mobile platform mated control software for self-commanding) is able to controlmay be instructed to perform a certain overall movement, e.g., CYCLOPS interactively (or automatically) from anywhere inmove to the other side of the room. This is translated into the world, in near real-time.a SOCP set resembling a binary tree; it comprises individual CYCLOPS uses only the pixelated camera images to moverobotic movements (turns, motor commands, etc.) to accom- about an environment (e.g., room/corridor with obstacles),plish the overall goal of moving to the other side of the room. If thus more realistically emulating the visual perception of aany individual command in the SOCP set cannot be executed, blind subject. It processes and enhances the pixelated imagerythat particular SOCP set is invalidated at that point, causing to result in new motion and navigation commands, such asFig. 6 – CYCLOPS Commanding Interface, controlling the AWD remote robotic platform testbed in near real-time. Theinterface displays the current status of CYCLOPS including battery charge levels, heading, velocity, obstacle proximity, andthe high-resolution gimbaled camera view.
232 c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 6 ( 2 0 0 9 ) 226–233 igation systems (Figs. 2 and 3) with respect to the utility and efﬁciency of supporting and enhancing visual prosthe- ses, while potentially reducing to a necessary minimum the need for valuable testing time with actual visual prosthesis carriers. It is difﬁcult to predict exactly what a blind subject with a camera-driven visual prosthesis may be able to perceive. Therefore, it is advantageous to offer a wide variety of image processing modules and the capability and ﬂexibility for repeated application of these modules in a user-deﬁned order. AVS [11,12], in particular, comprises numerous efﬁcient image processing modules, such as pixelation, contrast/brightness enhancement, grayscale equalization for luminance control under severe contrast/brightness conditions, grayscale levels for reduction of data volume transmitted to the visual pros- thesis, blur algorithms, and edge detection (e.g., [17,18]). With the development of CYCLOPS it is now possible to empirically determine, in the absence of a blind subject, which particular sequences of image processing modules may work best for the blind subject in real world scenarios (e.g., Fig. 7). One of the goals is to get CYCLOPS to “behave” simi- lar to a blind subject (especially motionwise) by developing, implementing, testing, and ﬁne-tuning/customizing onboard algorithms for image processing and analysis as well as auto- navigation. Once a certain degree of similarity in a behavioral pattern is achieved, such as navigating safely through a cor- ridor with obstacles or guideline following (e.g., Fig. 7), the underlying image processing and analysis algorithms as well as the sequences of image processing modules that enabled this successful behavior may be used to establish a practical initial conﬁguration for blind subjects when implemented in their respective visual prosthesis systems. Furthermore, test- ing with CYCLOPS may contribute to improving the design of environments that provide suitable access for the blind (e.g., rooms, corridors, entrances) by choosing those that CYCLOPS performed best in. Its Internet connectivity renders CYCLOPS a worldwide accessible testbed for researchers in the ﬁeld of artiﬁcial vision systems and machine vision. We have provided a commanding interface that allows the research community to easily interface their respective image processing and autonomous navigation software packages to CYCLOPS by merely using high-level commands, such as “turn right byFig. 7 – Navigation camera view of CYCLOPS at different 25 degrees” or “move forward one meter”. Additionally, wevisual resolutions (i.e., degrees of pixelation), mimicking have provided numerous interfaces for onboard cameras (Eth-the view afforded by artiﬁcial vision prostheses. Each ernet, IEEE 1394, USB). The direction and orientation of thecolumn from top to bottom: 64 × 64, 32 × 32, 16 × 16, 8 × 8. gimbaled camera can be user-controlled, allowing for theLeft column shows navigating a corridor while avoiding an emulation of head/eye-motion of a blind subject wearing anobstacle (i.e., a chair). Right column shows the following of artiﬁcial vision prosthesis. The onboard real-time voice syn-a high-contrast guideline on the ﬂoor of a corridor. thesizer can be used as a means to communicate audio cues (e.g., “Door 2 meters ahead.”) resulting from autonomous navigation and obstacle recognition/avoidance systems (e.g.,navigating a corridor while avoiding obstacles, and guideline ).following (Fig. 7). Researchers can interface their software packages either by remotely issuing high-level commands over the Internet, or by integrating and running their software packages locally on the4. Conclusion onboard Unix workstation, thereby bypassing the Internet for command transmittal. Regardless, researchers will be able toCYCLOPS enables subject-independent testing, evaluation, monitor remotely the actions and camera views of CYCLOPSand validation of image processing and autonomous nav- via its commanding interface (Fig. 6).
c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 6 ( 2 0 0 9 ) 226–233 233 CYCLOPS is directly and immediately applicable to any  S.C. DeMarco, The architecture, design, and electromagnetic(artiﬁcial) vision-providing system that is based on an imag- and thermal modeling of a retinal prosthesis to beneﬁt theing modality (e.g., video cameras, infrared sensors, sound, visually impaired, PhD Thesis, North Carolina State University, 2001.ultrasound, microwave, radar, etc.) as the ﬁrst step in the gen-  P.R. Singh, W. Liu, M. Sivaprakasam, M.S. Humayun, J.D.eration of visual perception. Weiland, A matched biphasic microstimulator for an implantable retinal prosthetic device, in: Proceedings of the IEEE International Symposium on Circuits and Systems, vol.Conﬂict of interest statement 4, 2004.  J.D. Weiland, W. Fink, M. Humayun, W. Liu, D.C. Rodger, Y.C.Fink and Tarbell may have proprietary interest in the technol- Tai, M. Tarbell, Progress towards a high-resolution retinalogy presented here as a provisional patent has been ﬁled on prosthesis, Conf. Proc. IEEE Eng. Med. Biol. Soc. 7 (2005)behalf of the California Institute of Technology. 7373–7375.  J.D. Weiland, W. Fink, M.S. Humayun, W. Liu, W. Li, M. Sivaprakasam, Y.C. Tai, M.A. Tarbell, System design of a highAcknowledgment resolution retinal prosthesis, Conf. Proc. IEEE IEDM (2008), doi:10.1109/IEDM.2008.4796682.The work described in this publication was carried out at  C.-Q. Zhou, X.-Y. Chai, K.-J. Wu, C. Tao, Q. Ren, In vivo evaluation of implantable micro-camera for visualthe California Institute of Technology under support of the prosthesis, Invest. Ophthalmol. Vis. Sci. 48 (2007) 668National Science Foundation Grant EEC-0310723. (E-Abstract).  W. Fink, M. Tarbell, Artiﬁcial vision simulator (AVS) forreferences enhancing and optimizing visual perception of retinal implant carriers, Invest. Ophthalmol. Vis. Sci. 46 (2005) 1145 (E-Abstract).  W. Liu, W. Fink, M. Tarbell, M. Sivaprakasam, Image  W. Liu, M.S. Humayun, Retinal prosthesis, in: IEEE processing and interface for retinal visual prostheses, in: International Solid-State Circuits Conference Digest of ISCAS 2005 Conference Proceedings, vol. 3, 2005, pp. Technical Papers, 2004, pp. 218–219. 2927–2930.  E. Zrenner, K.-D. Miliczek, V.P. Gabel, H.G. Graf, E. Guenther,  M.A. Tarbell, W. Fink, CYCLOPS: A mobile robotic platform H. Haemmerle, B. Hoefﬂinger, K. Kohler, W. Nisch, M. for testing and validating image processing algorithms in Schubert, A. Stett, S. Weiss, The development of subretinal support of visual prostheses, Invest. Ophthalmol. Vis. Sci. 50 microphotodiodes for replacement of degenerated (2009) 4218 (E-Abstract). photoreceptors, Ophthalmic Res. 29 (1997) 269–328.  Robosoft, http://www.robosoft.fr/eng/.  J.F. Rizzo, J.L. Wyatt, Prospects for a visual prosthesis,  R. Chellappa, Cloud computing—emerging paradigm for Neuroscientist 3 (1997) 251–262. computing, INFORMS, 1997.  E. Zrenner, Will retinal implants restore vision? Science 295  B. Hayes, Cloud computing, Commun. ACM 51 (2008). (2002) 1022–1025.  J.C. Russ, The Image Processing Handbook, CRC Press, 2002.  M.S. Humayun, J. Weiland, G. Fujii, R.J. Greenberg, R.  H.R. Myler, A.R. Weeks, The Pocket Handbook of Image Williamson, J. Little, B. Mech, V. Cimmarusti, G. van Boemel, Processing Algorithms in C, Prentice Hall PTR, 1993. G. Dagnelie, E. de Juan Jr., Visual perception in a blind  W. Fink, M. Tarbell, J. Weiland, M. Humayun, DORA: digital subject with a chronic microelectronic retinal prosthesis, object recognition audio-assistant for the visually impaired, Vision Res. 43 (2003) 2573–2581. Invest. Ophthalmol. Vis. Sci. 45 (2004) 4201 (E-Abstract).