Wide area frequency easurement system iitb

Project Report on Storage, retreival and process of continuous streaming data in a Wide Area Frequency Measurement System by Nitesh Pandit Kedar Khandeparkar under the guidance of Prof. A.M. Kulkarni Department of Electrical Engineering Indian Institute of Technology, Bombay Mumbai-400076 July 2010

Contents 0.1 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . 2 0.1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . 2 0.2 Current Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 0.2.1 Packet Format . . . . . . . . . . . . . . . . . . . . . . 3 0.2.2 Need of Project . . . . . . . . . . . . . . . . . . . . . . 4 0.3 Data Reception . . . . . . . . . . . . . . . . . . . . . . . . . . 5 0.3.1 Requirement . . . . . . . . . . . . . . . . . . . . . . . 5 0.3.2 Current Approach . . . . . . . . . . . . . . . . . . . . 5 0.3.3 Our Approach . . . . . . . . . . . . . . . . . . . . . . 5 0.4 Data Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 0.4.1 Requirement . . . . . . . . . . . . . . . . . . . . . . . 7 0.4.2 Current Approaches . . . . . . . . . . . . . . . . . . . 7 0.4.3 Possible Approaches . . . . . . . . . . . . . . . . . . . 7 0.4.4 Our Approach . . . . . . . . . . . . . . . . . . . . . . 7 0.4.5 Pseudocode for Insertion & Updation in Database . . 8 0.5 Data Display . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 0.5.1 Requirement . . . . . . . . . . . . . . . . . . . . . . . 9 0.5.2 Current Approach . . . . . . . . . . . . . . . . . . . . 9 0.5.3 Possible Approaches . . . . . . . . . . . . . . . . . . . 9 0.5.4 Our Approach . . . . . . . . . . . . . . . . . . . . . . 9 0.6 Handling database issues . . . . . . . . . . . . . . . . . . . . . 11 0.6.1 Running the program . . . . . . . . . . . . . . . . . . 12 0.7 Literature Survey . . . . . . . . . . . . . . . . . . . . . . . . . 14 0.7.1 MYSQL . . . . . . . . . . . . . . . . . . . . . . . . . 14 0.7.2 PHP . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 0.8 Future Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 0.8.1 Interpolation of Frequencies . . . . . . . . . . . . . . . 15 0.8.2 Daily triggers . . . . . . . . . . . . . . . . . . . . . . . 15 0.8.3 Threading . . . . . . . . . . . . . . . . . . . . . . . . . 15 0.9 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 1

0.1 Problem Statement Storage, retreival and process of continuous streaming data in a Wide Area Frequency Measurement System and further analysis to generate the flags and alerts. 0.1.1 Introduction Power System Network is continuously subjected to disturbances in the form of sudden load and generation changes. These disturbances give rise to oscillations in the rotor angle which are also seen in the frequency. To study these oscillations wide area frequency measurement setup is implemented in this project. The setup are strategically placed in five different places in India and are time synchronized via Network Time Protocol (NTP). The frequency is measured every 20 ms and is time stamped and send through internet to a server in IITB. The server program which is continuously receiving the packets, checking the correctness of packet information, doing the insertion in database according their time and displaying it on the web. 2

0.2 Current Setup In ‘Wide area frequency measurement system’ frequencies measured at different places in India are sent to a IITB Server. Frequencies are measured by sensors at every 20 ms. Local frequency measured is stamped with the time and place and sent to IITB Server. The IITB server is continuously listening for the packets on UDP port 6000. When a packet is received, it extracts the data and writes it in a file. A separate file is created for each sensor and when any file reaches its maximum a new file is created for each sensor with incremented serial number. Plotting of this data is done offline. 0.2.1 Packet Format Where, Source Port = 6000 Destination Port = 6000 Checksum = Default Data = “( Sensor name, year-month-day hours minutes seconds.milliseconds, Frequency value )” 3

0.2.2 Need of Project In existing scenario, data from each packet is stored in files at IITB sever. The analysis of data in these files is a difficult task and is done later offline. This project deals with online display of the incoming frequency data. Also it does constraint checking of each packet such as authenticity of location of sensor, validity of frequency range etc. If a packet passes such validation checks it is stored in the database for further analysis. Hardware Requirements 1) Intel P4 processor 2) RAM (512 MB) Software Requirements 1) C (gcc compiler) 2) Perl (libperl-dev, libdbd-mysql-perl) 3) MySql (mysql-server, mysql-client, libmysqlclient15-dev) 4) PHP (php5, php5-gd, php5-mysql, php-cli) 5) Apache (apache2) 6) Bzip2 (with perl libraries) 4

0.3 Data Reception 0.3.1 Requirement Data from multiple sensors coming continuously at a very small time interval need to be stored in an organized manner for further analysis. 0.3.2 Current Approach Data is dumped into flat files as it coming, with a separate file for each sensor. When any file reaches its maximum limit new files are created for each sensor with incremented serial number attached as a part of file name. There is no way for soft real time display of such data. It is also very difficult to gather data from all the sensors in a particular time range and compare them. Also current approach does not handle constraint checking. 0.3.3 Our Approach A UDP Server implemented in C language. It is continuously receiving the packets from multiple sensors. Server program comprises of three functions 1. main(): Continuously senses for the packets on UDP port 6000 and on reception creates a thread by calling a fuction check constraint has calculate(). To this function the message received is passed as an argument. 2. check constraint hash calculate(): Splits the received, message checks validity of each field and performs hash calculation. After that it calls insert packets(). 5

Logic for ‘check constraint hash calculate()’ I. Each packet has Sensor location Name, Date Time, Frequency" for ex. "surat, 2010-06-10 12 39 23.545, 49.654". We split the message by comma and place the values in columns array, for ex. columns[0]=’surat’ columns[1]=’2010-06-10 12 39 23.545’ columns[2]=49.654. II.We then perform some validity checks like a) Check if columns[0] contains the valid sensor name, if not then entry is made toLog_Error table and no further processing is done for the packet. b) Check if columns[1] contains a non null value, if not then entry is made to Log_Error table and no further processing is done for the packet. c) Check if columns[2] contains frequency value that lies between 47 and 52, if not then entry is made to Log_Error table and no further processing is done for the packet. d) If the packet passes all the validation checks then we further process the packet. Splits the received, message checks validity of each field and performs d.i) Split Date Time value in packet by space ’ ’ e.g. columns[1]=2009-09-07 10 12 322.743 will be split as datetime[0]=2009-09- 07,datetime[1]=10 ,datetime[2]=12, datetime[3]=322.743 d.ii) Split Date by ’-’ e.g. datetime[0]=2009-09-07 will be split as date[0]=2009, date[1]=09,date[2]=07 d.iii) Split millisecond time(ms) by ’.’ e.g.datetime[3]=322.743 as time[0]=322,time[1]=743 III. Now we calculate the hash value of each packet. It is the number of millisiseconds for the date time value in the packet. i.e. 2009-09-07 10 12 322.743 =>(x millisec) IV. On performing hash calculation we call a function insert_packets() and pass the calculated hash (i.e. x from above ) and columns[] as arguments. 6

0.4 Data Storage 0.4.1 Requirement Storage and retrieval of data should be efficient. 0.4.2 Current Approaches Data is stored in files, but to retrieve this data and do analysis is very difficult. 0.4.3 Possible Approaches Two ways of data storage a) Store as flat files as is done in the previous approach b) Use database. 0.4.4 Our Approach We have used Mysql database for data storage. Logic for insertion of packets into the database table is a part of main server program i.e. insert packets() called from check constraint hash calculate() contains the logic to insert newly arrived packet values in the database table. 7

0.4.5 Pseudocode for Insertion & Updation in Database Algorithm 0.4.1: InsertPackets(newHash, datetime, f requency) comment: Insert the packet with $datetime$ and $frequency$ at correct position in the table global currentHash, minHash, nextHash local tdif f, tmod, k, i, t2, t3, tempHash if currentHash! = 0  if newHash >= currentHash       tdif f ← newHash − currentHash    if tdif f <= 10         then {UPDATE TABLE(currentHash,$datetime$,$frequecy$)                tmod ← tdif f %20;      if tmod <= 10        {              then t2 ← tdif f − tmod;         t3 ← (integer)(t2)       {      20     else t2 ← tdif f − tmod + 20;  then         t3 ← (integer)(t2)     else for k ← 0 to t3 − 1 20                  $currentHash$ ← $nextHash$            do $nextHash$ ← $currentHash$ + 20               INSERT INTO TABLE(currentHash,NULL,NULL)            $currentHash$ ← $nextHash$           $nextHash$ ← $currentHash$ + 20             INSERT INTO TABLE(currentHash,$datetime$,$frequency$)     tdif f ← currentHash − newHash    tmod ← tdif f %20   then  if newHash >= minHash              if tmod  10 <=             t2 ← tdif f − tmod;            then t3 ← currentHash − t2;          then        UPDATE TABLE(t3,$datetime$,$frequecy$)       t2 ← tdif f − tmod + 20;            else t3 ← currentHash − t2;             UPDATE TABLE(t3,$datetime$,$frequecy$)       else  if tmod <= 10     {             then t2 ← tdif f − tmod;         t3 ← (integer)(t2)       {       20        else t2 ← tdif f − tmod;         t3 ← (integer)(t2)     else 20     tempHash ← currentHash          for k ← 1 to t3 − 1        {              do tempHash ← tempHash − 20;   tempHash ←       INSERT INTO TABLE(tempHash,NULL,NULL)           INSERT INTO TABLE( currentHash,$datetime$,$frequency$)        8  minHash ← tempHash currentHash ← minHash ← newHash else newHash ← currentHash + 20  INSERT INTO TABLE(currentHash,$datetime$,$frequency$)

0.5 Data Display 0.5.1 Requirement Graphical display of frequency Values on web. 0.5.2 Current Approach Offline plotting of frequency stored in files is done in Matlab, below graph is generated using Matlab and has to be done offline. 0.5.3 Possible Approaches 1. Java Applets 2. JPGraph 3. PHP 4. Asp.net 0.5.4 Our Approach We have chosen PHP and GD library to plot graphs of frequency data on web as it is an open source allows to build dynamic graphs. It also has less overhead compared to other web designing techniques. Server Requirements a) Apache server b) PHP (with GD) c) MySql 9

Steps involved in Display: 1. In the first step get the fixed size graph (image) on the web browser by using the following functions, ImageCreate ( int $width , int $height ). It returns an image identifier representing a blank image of specified size. In addition to ImageCreate() that created a graph, there are other functions used to fill the information in the graph such as drawing X-axis and Y-axis lines, plotting appropriate axis names, defining the color values etc. ImageLine (resource $image,int $x1,int $y1,int $x2,int $y2,int $color) Imageline function draws line between given points. This function will draw a line between the points ($x1,$y1) & ($x2,$y2) and assign the color specified by $color. ImageString (resource $image,int $font,int $x,int $y,string $string,int $color) ImageString function is used to display a text on the given coordinates of graph. Here ($x,$y) is a coordinate point on the graph where $string will be written. ImageColorAllocate (resource $image,int $red,int $green,int $blue) ImageColorAllocate returns a color identifier representing the color com- posed of the given RGB components. To plot a line or write a string on the graph we require colors. Imagecolorallocate() is used to associate color with objects . 2. After the first step a blank graph is created and axis are drawn php- mysql database connection is established. We use $con=mysql connect ( ’localhost’, ’username’, ’password’ ); mysql select db (‘database name’, $con); 3. In third step we run mysql queries to fetch the records from table sensor data. A view named display is created that will fetch last 10 seconds records from the table sensor data. Also, there are other queries to obtain minimum and maximum frequency values for the records obtained from view. These are required to decide the limits of values on Y-axis on each page refresh. 10

4. Fourth step is actual plotting of the frequency values. Time in seconds on X-axis and frequency on Y-axis. On each page refresh, which happened every second a new record set is fetched from the sensor data table. This fetched data is plotted dynamically by adjusting X axis and Y axis. There are two files, WAFMES.html and graph.php. WAFMES.html page calls graph.php every second on page refresh. Every new second value is plotted from right to left side. If the frequencies change suddenly it will be seen on the display as spikes. 5. In the fifth step we handle exceptions and errors. If any of data source failed and the data is not coming at server we raise a flag. This is done by displaying a message “X Currently Off” where X is the name of the data source. Null values in between the rows are also handled at display time. These Null values in the database table may be due to UDP packet lost or transmission delay or sensor side packet generation. These null values are displayed as sudden spikes on graphs. To overcome this problem we have taken previous frequency value in place of null only at display time and plot. The actual entry in the database table is left intact. However if there are continuous null values for a period of 10 seconds then we consider that sensor is down and raise the flag message. 0.6 Handling database issues Since approximately 4.5 millions of rows are inserted to ‘sensor data’ table every day and its maximum limit is 5.0 billion records we need to constantly move the records from the table to make the space for the new data. For this ‘Backup.pl’ script is run on daily basis. The job is scheduled in cronjob. This will run at 1 am every day and move the last day entries from table to a file compress the file with bzip2 and store it in the folder sensor data backup. The system date is part of filename. After taking the backup of the day those rows are deleted from the table sensor data. 11

0.6.1 Running the program Step 1 A shell script is written, so that when need to add a new sensor name, add only his name in text file (but it must be in sequence as per in php file and also need to update php file according to changes) first run ./setup.sh to create database then ./run.sh to run the server Otherwise Making the database ready Need to be done only first time. CREATE DATABASE WAFMES; USE WAFMES; CREATE TABLE sensor data ( hash decimal(14,0) NOT NULL, surat_time varchar(25), surat_value float , sangli_time varchar(25), sangli_value float, ahm_time varchar(25), ahm_value float, mum_time varchar(25), mum_value float, pune_time varchar(25), pune_value float, bhu_time varchar(25), bhu_value float, PRIMARY KEY(hash) ); CREATE TABLE Log Error ( sensor_name varchar(10), time varchar(30), freq varchar(10)); CREATE VIEW display AS SELECT * FROM sensor_data where hash>=((SELECT max(hash) FROM sensor_data)-12000) && hash<=((SELECT max(hash) FROM sensor_data)-2000)ORDER BY hash DESC; 12

Make an entry in cronjob for the file Backup.pl to be run on daily basis. It will get the last one day entries (approx 4.5 million) from database table (sensor data) and write in a file and compress, take the date as file name. Step 2 When a new sensor is added we need to alter the table and create columns for the new sensor. For ex. if x is a sensor location that has been newly established, then make the following changes to the table. ALTER TABLE sensor data add x time varchar(25), x value float); Changes also need to be reflected in the server.c and graph.php. Step-3 Compile the server program as ./compile.sh Step-4 Run the server program as ./run.sh 13

0.7 Literature Survey 0.7.1 MYSQL The MySQL software delivers a very fast, multi-threaded, multi-user, and robust SQL (Structured Query Language) database server. The MySQL software is Dual Licensed. Users can choose to use the MySQL software as an Open Source product under the terms of the GNU General Public License (http://www.fsf.org/licenses/) or can purchase a standard commercial license from Oracle. See http://www.mysql.com/company/legal/licensing/ for more information on our licensing policies. It is the one of the most scalable (upto 5 Billion entries) and fast database to meet our requirement. 0.7.2 PHP PHP (Hypertext Processor) is the language to create dynamic web devel- opment. PHP is an interpreted language and is executed on server side (just like CGI or ASP scripts) contrary to scripts executed on client side (a Javascript or a Java applet executes on your computer). It is usually asso- ciated with Apache and Mysql database. PHP with GD library to create images, graphs on the web browser. 14

0.8 Future Scope 0.8.1 Interpolation of Frequencies We are currently storing the frequency values from multiple sensors in a 20 millisecond time interval. For eg. if a packet that has arrived has the following data ’surat,2010-07-11 10 12 32.320,47.8’. Its calculated hash is 1278843152320. It is stored in the database table against the hash(which is a primary key) 1278843152330. Instead of storing the frequency against the hash 1278843152330(2010-07-11 10 12 32.340) we can interpolate between the current frequency value and the previous frequency of the sensor and store it against 1278843152330, for that we need to store last one frequency value for each sensor. 0.8.2 Daily triggers Alarms and Triggers need to be genarated for the variations in the frequencies between the sensors over a particular time interval. 0.8.3 Threading Current program running sequencialy so packet loss may occure when the number of sensor increased, in order to avodie such problems can be use threading to achive the parallelism with every packet. We can create a thread for each incoming packet and perform the constraint check and database insertion. 15

0.9 References 1. For mysql http://dev.mysql.com/doc/refman/5.1/en/index.html 2. For PHP http://php.net/manual/en/book.image.php 3. For threads https://computing.llnl.gov/tutorials/pthreads 16

Wide area frequency easurement system iitb

by PanditNitesh

Accessibility

Upload Details

Usage Rights

1 Embed 1

Statistics

Wide area frequency easurement system iitb Document Transcript