Zhao cvs2006


Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Zhao cvs2006

  1. 1. Real-Time Median Filtering for Embedded Smart Cameras Yong Zhao Gabriel Taubin Brown University Brown University Yong Zhao@brown.edu taubin@brown.edu Figure 1: System architecture of our proof-of-concept Visual Sensor Network (VSN) built with off-the-shelf IP cameras and embedded single-board-computers (SBCs) as image processors which communicate over common ethernet switch fabric amongst themselves and with the server. Each SBC serves as image processor for one IP camera. The algorithm described in this paper runs in the SBCs as part of an indoors person detection and tracking application. Results presented include the overhead incurred by each SBC decoding JPEG frames sent by a corresponding IP camera. Abstract 1 Introduction This paper describes a new median filter algorithm opti- The two dimensional median filter has been extensively mized for real-time performance in smart cameras with em- used for smoothing operations in image processing since bedded processors. As in the JPEG and MPEG compres- its introduction by Tukey [11]. The result of applying the sion algorithms, each frame of the video stream is first median filter to an N × N image I using a W × W win- partitioned into a regular array of non-overlapping square dow, where W = 2w+1 is an odd number, is a new image blocks. The median value for each block is then computed M of the same size[9]. The output pixel value M [i, j] is and compared with corresponding values of neighboring computed as the median of all the input values within the blocks. If the magnitude of the difference does not exceed W × W window centered at the pixel I[i, j]: a threshold, the output value for all the pixels in the block is set to the median value. Otherwise, the output value for M [i, j] = median{I[a, b] : |a − i| ≤ w ∧ |b − j| ≤ w} . each pixel in the block is computed as the median value Computing this value requires sorting the input pixel inten- within a window of the same size centered at this pixel. We sity values within the window (the filter kernel), and this describe variations for binary and grayscale images. The al- has to be done for each pixel in the image, resulting in high gorithm has been implemented and tested in an embedded computational cost. Compared with a linear averaging fil- single-board-computer (SBC) with no hardware accelera- ter, which evenly diffuses impulsive noise to neighboring tion, as a component of a Visual Sensor Network (VSN) pixels, the median filter removes impulsive noise by ignor- system for real-time indoor person detection and tracking. ing it. Consequently, median filtering is hardly affected by In this system, where the SBCs have the additional over- impulsive noise smaller than the filter kernel. head of decoding JPEG frames from IP cameras, our new The algorithm described in this paper was developed and algorithm is 5 to 20 times faster than the traditional algo- implemented as a basic low-level operation for the proof-of- rithms for typical window sizes. We expect further speed- concept Visual Sensor Network (VSN) described in Figure ups to frame-rate performance on smart cameras with em- 1. This platform was used to implement a real-time indoor bedded image sensors and reconfigurable hardware. person detection and tracking application. Our long term goal is to build large scale VSNs for real-time operations with very large number of cameras (1000s). For this kind of scalability, bandwidth constraints requires the cameras toProceedings of the Fourth IEEE International Conference on Computer Vision Systems (ICVS 2006)0-7695-2506-7/06 $20.00 © 2006 IEEE
  2. 2. be smart, i.e., to host the most high data rate intensive im- age processing operations. Figure 7 shows the architecture and current state of our first smart camera design, which is not yet operational. The experimental results reported in this paper are based on the same SBC platform, but the image capture is simulated using off-the-shelf IP cameras, which imposes on the SBCs the overhead of communicat- ing with their corresponding camera over the same network a b used to communicate with the server, and decoding com- pressed JPEG frames before applying the new median filter algorithm. All of this overhead is included in the results. The paper is organized as follows. In Section 2 we de- scribe the new fast median filter algorithm in detail, first for binary images, and then for graylevel images. In Section 3 we provide brief descriptions of prior related work, and comparisons with our new approach. In Section 4 we de- c d scribe the platform where we performed our experiments. In Section 5 we describe our experimental results. Finally, in section 6 we end the paper with conclusive remarks and summary. 2 The Algorithm We first explain the new algorithm for binary images, and e f then show how it has to be modified for graylevel images. 2.1 Fast Binary Median Filter Binary image smoothing is used widely in many applica- tions. For example, in most object tracking applications foreground objects are detected by comparing each frame with a background model. Each pixel in the image is clas- g h sified as foreground or background. Various kinds of back- ground statistical models have been proposed because plain differencing is not robust enough. However, noise can not Figure 2: Fast binary median filter. (a) Original image captured by the sensor; (b) Noisy foreground segmentation obtained by be avoided mainly due to variations in illumination or back- comparing the input image with the background model; (c) Binary ground/foreground characteristics, etc. Therefore, for most image smoothed with traditional median filter. (d) Noisy image approaches the resulting images need to be smoothed prior partitioned into non-overlapping blocks (big blocks shown for il- to further processing. lustration purposes; actual blocks are smaller); (e) Low resolution Figure 2 shows an example of using our new fast me- foreground image generated by block approximation; (f) Detected dian filter algorithm to smooth a binary image in a person edge blocks; (g) Binary image smoothed with the algorithm de- tracking application. Image (a) is a frame captured from a scribed in this paper; (h) Object of interest segmented out from video camera, showing that a man is walking in the room. input image. Image (b) is the binary image obtained by comparing (a) with the background model. The value of each pixel (black or white) indicates whether it belongs to the background or the kernel window of median filter. Image (d) shows the the foreground. As can be seen, there is too much noise divided image (for illustration purposes the blocks shown in and too many fuzzy edges in this image. In order to remove image (d) are much bigger than the ones actually used). The this ”salt and pepper” noise and smooth the edges of the median value of each block is then computed, and assigned foreground region, we apply median filtering on this image. to all the pixels in the block. The result of this step is shown Image (c) shows the output of the traditional median filter. in image (e). That is, the algorithm evaluates the traditional Our algorithm first divides the binary image (b) into a median filtering only at the center points of each block and regular grid of non-overlapping square blocks the size of these values are replicated to the rest of the pixels in theProceedings of the Fourth IEEE International Conference on Computer Vision Systems (ICVS 2006)0-7695-2506-7/06 $20.00 © 2006 IEEE
  3. 3. W(pixels) 3 5 7 9 11 13 15 σ (%) 0.4 1.8 2.7 3.2 3.6 4.5 6.3 θ (%) 11.5 5.8 4.7 4.4 4.4 5.1 6.7 Table 1: Computational cost comparison of two median filters. needed to remove the noise features of same dimension. In other words, with the same filter kernel size, the the new Figure 3: Different kernel size is needed to remove the same size approach has stronger image smoothing power than the tra- noise by traditional median filter and new approach. ditional algorithm. Our primary concern for real-time embedded applica- tions is to reduce the computational cost. The computa- block. As can be seen in image (e), the noise is successfully tional cost in the traditional median filter algorithm is de- removed by this step, but details on the region edges are termined by the image size and kernel window size. For lost. In the following step the algorithm recovers the lost example, applying median filtering with a W × W window details. to a N × N digital image involves N 2 median value calcu- To decide which blocks require additional boundary de- lations; each median value calculation involves data sorting tails, the median value assigned to each block in image (e) for W 2 values. The temporal complexity of the traditional is compared with the corresponding values assigned to the median filtering: TTraditional = O(N 2 W log(W )). eight-connected neighboring blocks. If not all of these vales The computational cost of the new median filter algo- coincide, the central block is labeled as an edge block. Edge rithm with the same kernel window size on the same image blocks are blocks which contain both background and fore- is much lower because we only evaluate the median values ground pixels (one of them dominates the whole block af- on the center pixels of every block, and on all the pixels in ter the approximation). Image (f) shows the result of this edge blocks. The total temporal complexity of new median step. After all the edge blocks are found, the traditional filtering is: median filter algorithm is evaluated on each pixels of image (b) belonging to an edge block, and the corresponding out- N2 TNew = O + Aedge blocks W log(W ) put pixel is set to the computed median value. This means W2 that overall, the median value is computed on all the block centers, and on all the edge block pixels. Image (g) shows Where Aedge blocks denotes the area of all the edge blocks in the final result of fast noise removal and edge smoothing, pixels. and image (h) shows the result of segmenting image (a) ac- In order to show how fast the new approach is, we use cording to the foreground classification of image (g). θ to denote the computational cost ratio of the two median filters: 2.2 Algorithm Analysis TNew 1 Aedge blocks 1 σ θ= = 2+ = 2+ TTraditional W N2 W 100 Comparing images (c) and (g) in Figure 2, we find that the noise removal and edge smoothing quality of the new ap- Where σ denotes the percentage of edge blocks area in the proach are better than those obtained using the traditional whole image area. The value σ depends on the noise level, medial filter algorithm with the same filter kernel size. This size and shape of foreground region, and the size of kernel phenomenon can be easily explained by the following ex- window. ample: We should notice that for binary image, the data sort- As illustrated in Figure 3-(a), in order to entirely re- ing computation is much cheaper than that of intensity move a round-shaped noise disk of radius R, the minimum image case. And the temporal complexity is no longer size of kernel window needed is LTraditional × LTraditional , where O(n log(n)). But the computational cost ratio θ of two me- LTraditional = (2πR2 )1/2 ≈ 2.506 R, in the traditional me- dian filters is the same. For the image showed in Figure dian filter algorithm. While for the new approach, the min- 2, the value σ and the corresponding θ for different S are imum filter kernel size depends on the position of the noise displayed in Table 1. disk. The best case is when the center of the noise disk is We see that when W = 5, the area of ”edge blocks” at the vertex of blocks. In this case Lnew = (0.25πR2 )1/2 ≈ is just about 1.8% of the whole image area for the case 1.253 R. The worse case is when the center of the noise showed in Figure 3. Therefore the computational cost of disk is at the center of block. In this case Lnew = LTraditional . new approach is expected to be only about 5.8% of that of In general, in new approach a smaller filter kernel size is traditional one.Proceedings of the Fourth IEEE International Conference on Computer Vision Systems (ICVS 2006)0-7695-2506-7/06 $20.00 © 2006 IEEE
  4. 4. image (g), and our new algorithm outputs image (h). The example presented in Figure 5 shows that, in addi- tion to the significant speedup, the new median filter al- gorithm produces very good quality output images in the case of intensity image. With the same size of filter ker- nel, the new approach delivers better noise removal quality a b than the traditional median filter algorithm, especially when the noise is dense. Additionally, in the above example, the Figure 4: Example of block approximation. (a). Simple approxi- computational cost ratio θ (defined in Section 3.2) is about mation of four adjacent 5×5 blocks. In each block, median values 15%, which means that the new approach is expected to be of only the central pixels (marked by the circle) are computed and at least 6 times computationally more efficient that the tra- shared by all non-central pixels. (b). Smoother approximation dition algorithm. using bilinear interpolation algorithm with the computed median value for central pixels. 3 Related Work 2.3 Fast Intensity Median Filter Since its introduction, median filtering has been extensively used to remove impulsive salt and pepper noise from im- The basic idea of the new median filter described for the ages. It is a more robust method than the traditional linear binary case could be easily extended to the graylevel image filter because it preserves sharp edges. case. The only difference is at block approximation step. One of the major flaws of the median filter is that it In the binary case we simply approximate the median fil- is very computationally expensive when the filter kernel ter algorithm results of non-center pixels within each block, is big. The temporal complexity of a straightforward se- with the values assigned to the center pixel of the same rial computation of applying median filtering on a N × N block. digital image with W × W kernel is O(N 2 W 2 ). This In the graylevel image case this simple approximation can reduced to O(N 2 W log(W )) by Quicksort [1]. A fast does not work well because it produces block boundary ar- two-dimensional median filtering algorithm developed by tifacts. This problem is illustrated in Figure 4-(a). In order Huang, Yang and Tang [4] provides a O(N 2 W ) perfor- to get a smoother approximation, we smoothly interpolate mance. A separable median filter introduced by Naren- values from median values of adjacent blocks, as shown in dra [8] and latter improved by Basu and Brown [2] takes Figure 4-(b). We have tried some commonly-used continu- only O(N 2 log W ) time. Various kinds of hardware solu- ous interpolation approaches, such as nearest neighbor, bi- tions using configurable logic or a pyramid computer have linear, spline, cubic and sinc interpolation. The smoothness been introduced to achieve real-time performance [10, 3, 7]. performances produced by these approaches do not differ Dedicated hardware does provide high speed but it is too too much, but their computational costs vary dramatically. expensive and rarely available in most application systems. Because what we need is a fast approximation to avoid com- Note that all these previous works have concentrated on puting the median value on every pixel, we have chosen bi- how to accelerate the computing of the median number for linear interpolation as our block approximation step. every pixel. The focus of the approach presented in this We then use a scheme similar to the one used in the bi- paper is on how to decrease the number of median number nary case. Median values of adjacent blocks are compared, computations and still achieve equally good smoothing re- and the magnitudes of the differences are thresholded to de- sults. As mentioned in section 2.2, for most images only tect the edge blocks. For edge blocks, the median value is 0.4 6.3% the median number computations performed by computed for every pixel in the block. For the other blocks, other algorithms are evaluated by our algorithm. Our ap- the bilinear interpolation is used to set the output pixels. proach is complementary those listed above, and we plan The results from two steps are combined and then we have to incorporate some of these ideas in future versions of our the final result of well-smoothed image. algorithm to achieve further speedups. Figure 5 illustrates the steps of our fast median filter for graylevel images. 25% salt and pepper noise is added to 4 Platform input image (a), resulting in image (b). Image (c) shows the output from the traditional median filter with a 3 × 3 As mentioned in Section 1, and illustrated in Figure 1, we window. Image (d) shows the edge blocks detected in (b). constructed a VSN for real-time indoor people detection Image (e) is the output of our fast median filter with a 3 × 3 and tracking. In this first system We used CerfCube405EP window. When the noise level is increased to 40% in image SBCs powered by IBM PowerPC System-on-Chip (SoC) (f), the traditional algorithm with a 5 × 5 window outputs processors [6] as image processors, and D-Link DCS-900Proceedings of the Fourth IEEE International Conference on Computer Vision Systems (ICVS 2006)0-7695-2506-7/06 $20.00 © 2006 IEEE
  5. 5. a b a b c d Figure 6: (a). Architecture of embedded smart camera; (b). Pic- ture of our current camera hardware. Each smart camera consists of 3 functional PCB boards: a sensor board, a reconfigurable hard- ware image processor board and a CPU board. with this architecture, both in terms of computation and communications. With this structure only small amounts of data generated by the embedded image processors need e f be transferred over the network to the server. It is then pos- sible to construct a real-time, large-scale visual sensor net- work with the compact, power-saving embedded cameras, and one regular PC working as the server. Figure 6 illustrates the smart camera we will use in our next VSN system. The high level system architecture is showed in (a), and pictures of the three circuit which con- stitute this smart camera are shown in (b). This embedded camera has been designed and is currently under assembly g h and debugging. In order to develop and test the image pro- cessing software, such as the algorithm presented in this Figure 5: Example of fast intensity median filtering. (a). Original paper, we implemented the image capture device driver for grayscale image. (b). Image imposed with 25% impulsive noise. the SBC emulating the hardware platform that we are build- (c). Smoothed image by traditional image. (d) ”Edge blocks” de- ing. A mass-market IP camera was used to capture images, tected in new approach; (e) Smoothed image by new approach; (f) Image imposed with 40% impulsive noise. (g). Smoothed im- which were transmitted to the SBC over the Ethernet in- age of (f) by traditional image; (h) Smoothed image of (f) by new terface. Once a frame is received, decoded, and in local approach. memory, the SBC runs the image processing operations at the same speed it will on our future smart camera. The only difference is that the raw data in JPEG format is transferred IP cameras for image capture [5]. The common 100BaseT from the sensors to the image processors by Ethernet in- Ethernet switch fabric existing in the building was used for stead of inter-PCBs connections, and therefore the image communication between each camera and its correspond- processors has the overhead of communication and decom- ing image processing SBC, and between each SBC and the pressing the JPEG data data before processing it. server. This approach reduces the bandwidth requirements, distributes the computation, and allows the server to con- 5 Experimental Results centrate on other jobs, such as calibration, synchronization, information integration and visualization. Performing all The main purpose of our new median filter algorithm is to the computation at the server would have been impossible overcome the high computational cost of the traditional ap-Proceedings of the Fourth IEEE International Conference on Computer Vision Systems (ICVS 2006)0-7695-2506-7/06 $20.00 © 2006 IEEE
  6. 6. 6 Conclusion and Discussion In this paper we propose a new high-speed median filtering for real-time embedded applications. The main feature of this new approach is its much lower computational cost. In addition, it provides better smoothing quality than the tra- ditional approach. We discussed how to apply the new approach both to bi- nary and graylevel images, and we get very good experi- mental results. Further investigation could be done in how to apply the same idea on color images, which is a vector median filtering problem. However, the higher discrimina- Figure 7: Time consumption of two approaches in binary case on tion of color information might generate too high δ value embedded system. (edge block area ratio) to achieve satisfactory speed perfor- mance. Kernel Size 3 5 7 9 11 New Approach 35 47 72 149 189 7 Acknowledgements Traditional Approach 323 862 1614 3426 4195 To be written after the review period. Table 2: Time consumption of two approaches in intensity case References (unit: ms). [1] A. V. Aho, J. E. Hopcroft, and J. D. Ullman. The Design and Analysis of Computer Algorithms. Addison-Wesley, Read- ing, MA, 1974. [2] A. Basu and M. Brown. Algorithms and hardware for ef- proach. In previous sections we demonstrated that the qual- ficient image smoothing. Computer Vision, Graphics and ity of the images produced with the new approach is equal Image Processing, 40:131–146, February 1987. or better than with of traditional approach, especially when [3] J. G. R. Delva, A. M. Reza, and R. D. Turney. Fpga im- plementation of a nonlinear two dimensional fuzzy filter. the noise level is high. The algorithm analysis in provides In Proceedings of the International Conference on ASSP, reasons for the improvement of smoothing quality and re- Phoenix, Arizona, March 1999. duction of computational cost. In this section, we present [4] T. S. Huang, G.Y. Yang, and G. Y. Tang. A fast two dimen- speed test results for the new algorithm running in the VSN sional median filtering algorithm. In IEEE Transaction on ASSP, volume ASSP-27, pages 13–18, February 1979. system described above. We first tested the speed perfor- [5] D-Link Inc. Dcs-900 ip camera. http://www.dlink. mance of traditional and new median filters for binary im- com. ages. The median filtering works as foreground smoother [6] Intrinsyc Inc. Cerfcube 405ep embedded single board com- in this application. The time costs of the traditional median puter. http://www.intrinsyc.com. filter vs. the new algorithm are displayed in Figure 7. [7] G. Louverdis, I. Andreadis, and A. Gasterato. A new content based median filter. In EUSIPCO 2004, Vienna, Austria, On our current emulating embedded cameras, the objects September 2004. detection and tracking application runs at 8 to 10 frames per [8] P.M. Narendra. A separable median filter for image noise smoothing. IEEE Transactions on Pattern Analysis and Ma- second in QVGA mode, which is an encouraging result. We chine Intelligence, 3(1):20–29, 1981. expect a much higher frame rate, probably 30 fps, when we [9] S. Ranka and S. Sahni. Efficient serial and parallel algo- remove the overhead associated with interfacing with the IP rithms for median filtering. In Proceedings of the Inter- cameras over the common network, and run the algorithm national Conference on Parallel Processing, pages 56–62, 1989. in our new embedded cameras after the assembly and de- [10] S. L. Tanimoto. Algorithms for median filtering of images bugging work is finished. This is so because in that case, on a pyramid machine. In M.J.B. Duff, editor, Computing the video data will be transferred from the image sensor to Structures for Image Processing. Academic Press, London, the processor core through a private dedicated inter-PCB 1983. connections instead of the Ethernet, and no decompression [11] J.W. Tukey. Exploratory Data Analysis (preliminary ed.). Reading, MA: Addison-Wesley, 1971. by the image processor will be required. We also tested the fast intensity median filtering on some 1600 × 1200 resolution images on a regular PC running at 3.2 GHz. The average comparison results are listed in Table 2.Proceedings of the Fourth IEEE International Conference on Computer Vision Systems (ICVS 2006)0-7695-2506-7/06 $20.00 © 2006 IEEE