NoSQL - "simple" web monitoring

811 views

Published on

Talk given in the CERN School of Computing 2012, Uppsala, Sweden

Total time : 8 minutes for presentation, 2 for questions

Published in: Technology, Design
1 Comment
1 Like
Statistics
Notes
No Downloads
Views
Total views
811
On SlideShare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
4
Comments
1
Likes
1
Embeds 0
No embeds

No notes for slide

NoSQL - "simple" web monitoring

  1. 1. NoSQL“Easy” web monitoring
  2. 2. HEP Computing environment• Big datacenters• Existent monitoring o Nagios/Cacti/similars monitoring o Site Availability Monitoring/Hammercloud o High level tests (Grid services) o General Monitoring of grid services• Missing/incomplete monitoring o Specific/detailed information about specific Grid services o Transfer Management tool (PhEDEx for CMS) o Batch system (Condor,PBS,LSF,Torque, etc) o Quota report based on Experiments characteristic organization (Physics groups, T2 federation, etc)
  3. 3. Options• SGDBs & SQL - Might be too much o More costs for deployment o Less flexibility o Increase applications complexity/time spent o Black sheep - SQLite - however, scaling problems• Scripting - Might be too bad o Database is too much, query the info from CLI, format and print (text file, HTML) o Parse logs, format and print (text file, HTML)• Bottom line is evaluate – Effort x Result x Quality you need
  4. 4. NoSQLIt doesnt mean SQL is a bad idea =) CouchDB,MongoDB, etc...
  5. 5. JSON  My apologies to the ones that know it...  Format that represents associative arrays (ke value pairs){ "Name": "Smith", "age": 25, "address": { "streetAddress": "21 2nd Street", "city": "New York", "state": "NY", "postalCode": "10021" },}
  6. 6. For the curious... MapReduce Trivial parallelization  First Map (filter, get documents subset)  Then reduce (count, sum, etc)
  7. 7. Overview Server Databases  Documents  Unique ID, revision  All the rest is free  Key/Value – datatype for many languages  Views  Static sets of what you want to show  View code scans all documents in the database  Caching  For Your Information – “CouchApps”
  8. 8. { Documents "_id": "osgce@osgce.hepgrid.uerj.br#1146503.0#1335286041", "_rev": "10-12a26ecf9bc0fecf5ed9073c374a620c", "data": { "NumJobStarts": { "i": "1" }, "EnteredCurrentStatus": { "i": "1335415262" }, "CurrentHosts": { http://<MYSERVER>/<MYDATABASE>/<DOCUMENT_ID> "i": "0" } }}
  9. 9. Views Extract from the documents, only what you want to see Only documents you want to seefunction(doc) { if (doc.Type == "customer") { emit(doc._id, {LastName: doc.LastName, FirstName: doc.FirstName, Address: doc.Address}); }} http://<MYSERVER>/<MYDATABASE>/_design/<MYDESIGN>/ _view/<MYVIEW>
  10. 10. CouchApps Webserver  CouchDB hosts your HTML(5...)+Javascript application Upload to the given “DB”, has access to its views, and docs. DB is replicated, everything goes together http://<MYSERVER>/<MYDATABASE>/_design/<MYCOUCHAPPAME>/index.html
  11. 11. GlobalMonitor

×