Project Report on
Storage, retreival and process of continuous streaming data in a Wide
Area Frequency Measurement System
under the guidance of
Prof. A.M. Kulkarni
Department of Electrical Engineering
Indian Institute of Technology, Bombay
0.1 Problem Statement
Storage, retreival and process of continuous streaming data in a Wide Area
Frequency Measurement System and further analysis to generate the ﬂags
Power System Network is continuously subjected to disturbances in the form
of sudden load and generation changes. These disturbances give rise to os-
cillations in the rotor angle which are also seen in the frequency. To study
these oscillations wide area frequency measurement setup is implemented in
this project. The setup are strategically placed in ﬁve diﬀerent places in
India and are time synchronized via Network Time Protocol (NTP). The
frequency is measured every 20 ms and is time stamped and send through
internet to a server in IITB.
The server program which is continuously receiving the packets, checking
the correctness of packet information, doing the insertion in database ac-
cording their time and displaying it on the web.
0.2 Current Setup
In ‘Wide area frequency measurement system’ frequencies measured at dif-
ferent places in India are sent to a IITB Server. Frequencies are measured
by sensors at every 20 ms. Local frequency measured is stamped with the
time and place and sent to IITB Server.
The IITB server is continuously listening for the packets on UDP port 6000.
When a packet is received, it extracts the data and writes it in a ﬁle. A sep-
arate ﬁle is created for each sensor and when any ﬁle reaches its maximum a
new ﬁle is created for each sensor with incremented serial number. Plotting
of this data is done oﬄine.
0.2.1 Packet Format
Source Port = 6000
Destination Port = 6000
Checksum = Default
Data = “( Sensor name, year-month-day hours minutes seconds.milliseconds,
Frequency value )”
0.2.2 Need of Project
In existing scenario, data from each packet is stored in ﬁles at IITB sever.
The analysis of data in these ﬁles is a diﬃcult task and is done later oﬄine.
This project deals with online display of the incoming frequency data. Also
it does constraint checking of each packet such as authenticity of location
of sensor, validity of frequency range etc. If a packet passes such validation
checks it is stored in the database for further analysis.
1) Intel P4 processor
2) RAM (512 MB)
1) C (gcc compiler)
2) Perl (libperl-dev, libdbd-mysql-perl)
3) MySql (mysql-server, mysql-client, libmysqlclient15-dev)
4) PHP (php5, php5-gd, php5-mysql, php-cli)
5) Apache (apache2)
6) Bzip2 (with perl libraries)
0.3 Data Reception
Data from multiple sensors coming continuously at a very small time inter-
val need to be stored in an organized manner for further analysis.
0.3.2 Current Approach
Data is dumped into ﬂat ﬁles as it coming, with a separate ﬁle for each
sensor. When any ﬁle reaches its maximum limit new ﬁles are created for
each sensor with incremented serial number attached as a part of ﬁle name.
There is no way for soft real time display of such data. It is also very diﬃcult
to gather data from all the sensors in a particular time range and compare
them. Also current approach does not handle constraint checking.
0.3.3 Our Approach
A UDP Server implemented in C language. It is continuously receiving the
packets from multiple sensors.
Server program comprises of three functions
Continuously senses for the packets on UDP port 6000 and on reception
creates a thread by calling a fuction check constraint has calculate(). To
this function the message received is passed as an argument.
2. check constraint hash calculate():
Splits the received, message checks validity of each ﬁeld and performs hash
calculation. After that it calls insert packets().
Logic for ‘check constraint hash calculate()’
I. Each packet has Sensor location Name, Date Time, Frequency"
for ex. "surat, 2010-06-10 12 39 23.545, 49.654". We split
the message by comma and place the values in columns array,
for ex. columns=’surat’ columns=’2010-06-10 12 39 23.545’
II.We then perform some validity checks like
a) Check if columns contains the valid sensor name, if not
then entry is made toLog_Error table and no further
processing is done for the packet.
b) Check if columns contains a non null value, if not then
entry is made to Log_Error table and no further processing
is done for the packet.
c) Check if columns contains frequency value that lies between
47 and 52, if not then entry is made to Log_Error table and no
further processing is done for the packet.
d) If the packet passes all the validation checks then we further
process the packet. Splits the received, message checks validity
of each field and performs
d.i) Split Date Time value in packet by space ’ ’
e.g. columns=2009-09-07 10 12 322.743 will be split as
datetime=2009-09- 07,datetime=10 ,datetime=12,
d.ii) Split Date by ’-’
e.g. datetime=2009-09-07 will be split as date=2009,
d.iii) Split millisecond time(ms) by ’.’
e.g.datetime=322.743 as time=322,time=743
III. Now we calculate the hash value of each packet.
It is the number of millisiseconds for the date time value in
the packet. i.e. 2009-09-07 10 12 322.743 =>(x millisec)
IV. On performing hash calculation we call a function insert_packets()
and pass the calculated hash (i.e. x from above ) and
columns as arguments.
0.4 Data Storage
Storage and retrieval of data should be eﬃcient.
0.4.2 Current Approaches
Data is stored in ﬁles, but to retrieve this data and do analysis is very diﬃ-
0.4.3 Possible Approaches
Two ways of data storage
a) Store as ﬂat ﬁles as is done in the previous approach
b) Use database.
0.4.4 Our Approach
We have used Mysql database for data storage. Logic for insertion of packets
into the database table is a part of main server program i.e. insert packets()
called from check constraint hash calculate() contains the logic to insert
newly arrived packet values in the database table.
0.5 Data Display
Graphical display of frequency Values on web.
0.5.2 Current Approach
Oﬄine plotting of frequency stored in ﬁles is done in Matlab, below graph
is generated using Matlab and has to be done oﬄine.
0.5.3 Possible Approaches
1. Java Applets
0.5.4 Our Approach
We have chosen PHP and GD library to plot graphs of frequency data on
web as it is an open source allows to build dynamic graphs. It also has less
overhead compared to other web designing techniques.
Server Requirements a) Apache server
b) PHP (with GD)
Steps involved in Display:
1. In the ﬁrst step get the ﬁxed size graph (image) on the web browser
by using the following functions,
ImageCreate ( int $width , int $height ).
It returns an image identiﬁer representing a blank image of speciﬁed size. In
addition to ImageCreate() that created a graph, there are other functions
used to ﬁll the information in the graph such as drawing X-axis and Y-axis
lines, plotting appropriate axis names, deﬁning the color values etc.
ImageLine (resource $image,int $x1,int $y1,int $x2,int $y2,int $color)
Imageline function draws line between given points. This function will draw
a line between the points ($x1,$y1) & ($x2,$y2) and assign the color speci-
ﬁed by $color.
ImageString (resource $image,int $font,int $x,int $y,string $string,int
ImageString function is used to display a text on the given coordinates of
Here ($x,$y) is a coordinate point on the graph where $string will be written.
ImageColorAllocate (resource $image,int $red,int $green,int $blue)
ImageColorAllocate returns a color identiﬁer representing the color com-
posed of the given RGB components. To plot a line or write a string on the
graph we require colors. Imagecolorallocate() is used to associate color with
2. After the ﬁrst step a blank graph is created and axis are drawn php-
mysql database connection is established.
We use $con=mysql connect ( ’localhost’, ’username’, ’password’ ); mysql select db
(‘database name’, $con);
3. In third step we run mysql queries to fetch the records from table
sensor data. A view named display is created that will fetch last 10 sec-
onds records from the table sensor data. Also, there are other queries to
obtain minimum and maximum frequency values for the records obtained
from view. These are required to decide the limits of values on Y-axis on
each page refresh.
4. Fourth step is actual plotting of the frequency values. Time in seconds
on X-axis and frequency on Y-axis. On each page refresh, which happened
every second a new record set is fetched from the sensor data table. This
fetched data is plotted dynamically by adjusting X axis and Y axis. There
are two ﬁles, WAFMES.html and graph.php. WAFMES.html page calls
graph.php every second on page refresh. Every new second value is plotted
from right to left side. If the frequencies change suddenly it will be seen on
the display as spikes.
5. In the ﬁfth step we handle exceptions and errors. If any of data source
failed and the data is not coming at server we raise a ﬂag. This is done by
displaying a message “X Currently Oﬀ” where X is the name of the data
source. Null values in between the rows are also handled at display time.
These Null values in the database table may be due to UDP packet lost or
transmission delay or sensor side packet generation. These null values are
displayed as sudden spikes on graphs. To overcome this problem we have
taken previous frequency value in place of null only at display time and
plot. The actual entry in the database table is left intact. However if there
are continuous null values for a period of 10 seconds then we consider that
sensor is down and raise the ﬂag message.
0.6 Handling database issues
Since approximately 4.5 millions of rows are inserted to ‘sensor data’ table
every day and its maximum limit is 5.0 billion records we need to constantly
move the records from the table to make the space for the new data. For
this ‘Backup.pl’ script is run on daily basis. The job is scheduled in cron-
job. This will run at 1 am every day and move the last day entries from
table to a ﬁle compress the ﬁle with bzip2 and store it in the folder sen-
sor data backup. The system date is part of ﬁlename. After taking the
backup of the day those rows are deleted from the table sensor data.
0.6.1 Running the program
A shell script is written, so that when need to add a new sensor name,
add only his name in text ﬁle (but it must be in sequence as per in php ﬁle
and also need to update php ﬁle according to changes)
ﬁrst run ./setup.sh to create database
then ./run.sh to run the server
Making the database ready
Need to be done only ﬁrst time.
CREATE DATABASE WAFMES;
CREATE TABLE sensor data (
hash decimal(14,0) NOT NULL,
surat_value float ,
CREATE TABLE Log Error (
CREATE VIEW display AS
SELECT * FROM sensor_data where
hash>=((SELECT max(hash) FROM sensor_data)-12000) && hash<=((SELECT
max(hash) FROM sensor_data)-2000)ORDER BY hash DESC;
Make an entry in cronjob for the ﬁle Backup.pl to be run on daily basis.
It will get the last one day entries (approx 4.5 million) from database ta-
ble (sensor data) and write in a ﬁle and compress, take the date as ﬁle name.
When a new sensor is added we need to alter the table and create columns
for the new sensor. For ex. if x is a sensor location that has been newly
established, then make the following changes to the table.
ALTER TABLE sensor data add x time varchar(25), x value ﬂoat);
Changes also need to be reﬂected in the server.c and graph.php.
Compile the server program as ./compile.sh
Run the server program as ./run.sh
0.7 Literature Survey
The MySQL software delivers a very fast, multi-threaded, multi-user, and
robust SQL (Structured Query Language) database server. The MySQL
software is Dual Licensed. Users can choose to use the MySQL software as
an Open Source product under the terms of the GNU General Public License
(http://www.fsf.org/licenses/) or can purchase a standard commercial
license from Oracle. See http://www.mysql.com/company/legal/licensing/
for more information on our licensing policies.
It is the one of the most scalable (upto 5 Billion entries) and fast database
to meet our requirement.
PHP (Hypertext Processor) is the language to create dynamic web devel-
opment. PHP is an interpreted language and is executed on server side
(just like CGI or ASP scripts) contrary to scripts executed on client side (a
ciated with Apache and Mysql database. PHP with GD library to create
images, graphs on the web browser.
0.8 Future Scope
0.8.1 Interpolation of Frequencies
We are currently storing the frequency values from multiple sensors in a
20 millisecond time interval. For eg. if a packet that has arrived has the
following data ’surat,2010-07-11 10 12 32.320,47.8’. Its calculated hash is
1278843152320. It is stored in the database table against the hash(which is
a primary key) 1278843152330. Instead of storing the frequency against the
hash 1278843152330(2010-07-11 10 12 32.340) we can interpolate between
the current frequency value and the previous frequency of the sensor and
store it against 1278843152330, for that we need to store last one frequency
value for each sensor.
0.8.2 Daily triggers
Alarms and Triggers need to be genarated for the variations in the frequen-
cies between the sensors over a particular time interval.
Current program running sequencialy so packet loss may occure when the
number of sensor increased, in order to avodie such problems can be use
threading to achive the parallelism with every packet. We can create a
thread for each incoming packet and perform the constraint check and
1. For mysql http://dev.mysql.com/doc/refman/5.1/en/index.html
2. For PHP http://php.net/manual/en/book.image.php
3. For threads https://computing.llnl.gov/tutorials/pthreads