Łukasz Jagiełło – Gratka Technologie – MongoDB dla administratora

2. MongoDB dla administratora
●
●
Łukasz Jagiełło – l.jagiello@gratka-technologie.pl
http://geekandpoke.typepad.com/geekandpoke/2011/01/nosql.html

3. Agenda
● Kto ja i co ja
● Jak to się robi dzisiaj
● Jak to można zrobić jutro
● Pomysły...

4. Kto ja i co ja ?
Ukradzione z wyszukiwania w Grafika google.com

5. Jak to się robi dzisiaj

6. Jak to się robi dzisiaj
filter f_kern { facility(kern); };
filter f_cron { facility(cron); };
destination d_kern { file("/var/log/$YEAR$MONTH$DAY/kern"); };
destination d_cron { file("/var/log/$YEAR$MONTH$DAY/cron"); };
log { source(src); filter(f_kern); destination(d_kern); };
log { source(src); filter(f_cron); destination(d_cron); };

7. Jak to się robi dzisiaj

8. Jak to się robi dzisiaj
filter f_kern { facility(kern); };
filter f_cron { facility(cron); };
destination d_collector { udp("10.X.Y.Z" port(514)); };
log { source(src); destination(d_collector); };

9. Syslog-ng 3.4.0alpha1
Moduły:
● File ● JSON
● MongoDB ● Stream
● SQL ● CSV
● Socket
● Network
● Console

10. afmongodb
Przykładowy log z HAProxy:
2012­04­02T21:10:25+02:00 10.x.y.z haproxy[2913]:
Connect from 80.55.238.74:3935 to 195.8.99.31:80
(dom.gratka.pl/HTTP)
HAProxy -> syslog-ng(local) -> syslog-ng(remote) -> mongodb

11. afmongodb
destination d_mongodb {
mongodb(
value­pairs(
scope("selected­macros" "nv­pairs")
exclude("DATE")
exclude("HOST")
exclude("LEGACY_MSGHDR")
[...]};
rewrite r_rewrite_subst{subst("^Connect from (.*):
[[:digit:]]{1,5} to.*", "${1}", value("MESSAGE"));};

12. afmongodb
log {
#source(s_local);
source(s_network);
#destination(d_local);
rewrite(r_rewrite_subst);
destination(d_mongodb);
};

13. MongoDB
„Capped collections are fixed sized collections that have a very high
performance auto-FIFO age-out feature (age out is based on insertion
order). They are a bit like the "RRD" concept if you are familiar with that.”
„In addition, capped collections automatically, with high performance,
maintain insertion order for the documents in the collection; this is very
powerful for certain use cases such as logging.”
> db.createCollection("mycoll", {capped:true, size:100000})
> db.createCollection("messages", {capped: true,
size:10000000, autoIndexId:true})

14. MongoDB
> db.messages.find()
{ "MESSAGE" : "67.162.55.226", "_id" : ObjectId("4f7a0072c90d34ea7d2495db") }
{ "MESSAGE" : "193.151.114.11", "_id" : ObjectId("4f7a0072c90d34ea7d2495dc") }
{ "MESSAGE" : "89.167.68.120", "_id" : ObjectId("4f7a0072c90d34ea7d2495dd") }
{ "MESSAGE" : "83.18.64.235", "_id" : ObjectId("4f7a0072c90d34ea7d2495de") }
{ "MESSAGE" : "79.185.22.118", "_id" : ObjectId("4f7a0072c90d34ea7d2495df") }
{ "MESSAGE" : "85.89.184.6", "_id" : ObjectId("4f7a0072c90d34ea7d2495e0") }
{ "MESSAGE" : "85.89.184.6", "_id" : ObjectId("4f7a0072c90d34ea7d2495e1") }
{ "MESSAGE" : "85.89.184.6", "_id" : ObjectId("4f7a0072c90d34ea7d2495e2") }
{ "MESSAGE" : "66.249.68.51", "_id" : ObjectId("4f7a0072c90d34ea7d2495e3") }
{ "MESSAGE" : "37.47.121.77", "_id" : ObjectId("4f7a0072c90d34ea7d2495e4") }
{ "MESSAGE" : "89.167.68.120", "_id" : ObjectId("4f7a0072c90d34ea7d2495e5") }
has more

15. MongoDB
Puchnące mongo ?
17772 root 15 0 1463m 474m 276m S 0.0 8.0
2011 147:00 ./mongod ­­fork ­­logpath /d0/db/mongod.log
● ponad 5 miesięcy działania
● średnio co 30min wszystkie dane są wymienione
● 10mln rekordów w bazie – zawsze !
● 4GB na dysku
!! Wesje 2.x działają DUŻO lepiej niż wcześniejsze !!

16. mojology
http://mojology.madhouse-project.org/

17. mojology
http://mojology.madhouse-project.org/

18. mojology
http://mojology.madhouse-project.org/

19. Pomysły – Top10 IP
m=function () {
emit(this.MESSAGE, {count:1});
}
r=function (key, values) {
var total = 0;
for (var i = 0; i < values.length; i++) {
total += values[i].count;
}
return {count:total};
}
db.messages.mapReduce(m, r, { out : "myoutput" } );
db.myoutput.find().sort({value:­1});

20. Pomysły – Top10 IP
> db.messages.mapReduce(m, r, { out : "myoutput" } );
db.myoutput.find().sort({value:­1});
{ "_id" : "195.8.99.115", "value" : { "count" : 20771 } }
{ "_id" : "86.111.244.239", "value" : { "count" : 6816 } }
{ "_id" : "94.75.116.206", "value" : { "count" : 6716 } }
{ "_id" : "195.137.162.149", "value" : { "count" : 5638 } }
{ "_id" : "178.63.0.2", "value" : { "count" : 4902 } }
{ "_id" : "31.187.3.80", "value" : { "count" : 3573 } }
{ "_id" : "176.9.73.105", "value" : { "count" : 2734 } }
{ "_id" : "62.129.228.159", "value" : { "count" : 2092 } }
{ "_id" : "209.123.8.176", "value" : { "count" : 2016 } }
{ "_id" : "107.20.40.240", "value" : { "count" : 1927 } }
[...]

21. Pomysły cd.
http://www.slideshare.net/octplane/mongofr-mongodb-as-a-log-collector

22. Pomysły cd.
http://www.slideshare.net/octplane/mongofr-mongodb-as-a-log-collector

23. Pomysły cd.
● CAPTCHA
● Lokalizowanie błędów
● Kontrola
● Liczniki (youporn.com 8GB-15GB/godzinę)
● Itd.
Do statystyk mamy ciekawsze narzędzia