With the development of human social life, the city is undertaking an increasing population. Smart city is aimed at collecting, analyzing, integrating the key indicators of city operation through information technology, so as to make intelligent response to various needs including the people's livelihood, environmental protection, public security, city services, industrial and commercial activities. Recently the development of cloud computing, big data, internet of things technology, smart city is gradually evolving from the concept to a technology which can thoroughly change people's lives.
We have built a Streaming processing system (OCSP) based on Spark streaming, which has been used in two key fields (Location operation system and real time Marketing) in China Mobile corporation. The system used Spark streaming, Kafka, Flume, Redis technologies etc, and processed 30 million data records (type of location) per minute and 40 million (type of operation) data records per minute. The data comes from the real-time use of mobile phone from 60 million end users. After processing , the processed data are outputted to Kafka for other applications to use. OCSP provides APIs for developer to develop the applications for different use. Developers don't need to know the detail of RDD, streaming concept or other concepts in Spark. They just focus the logic implementation of the business.
We will describe the key technologies of OCSP system, then introduces large quantities of real- time data analysis and processing technology in smart tourism, and real-time processing data modeling methods. The technology has the characteristics of high real-time, high reliability, high accuracy of data processing, and has strong applicability, which be extended to other large data real-time processing scenarios.