BillRun - Billing on top of MongoDB | MUG IL, Feb 2014

Like this? Share it with your network

Share

BillRun - Billing on top of MongoDB | MUG IL, Feb 2014

  • 50,126 views
Uploaded on

BillRun is open-source billing solution for enterprise, which built on top of MongoDB. ...

BillRun is open-source billing solution for enterprise, which built on top of MongoDB.
This presentation took place on Israeli MongoDB User Group meeting, February 20, 2014, at John Bryce college, Tel Aviv.

Find out more: http://billrun.net

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
50,126
On Slideshare
3,562
From Embeds
46,564
Number of Embeds
83

Actions

Shares
Downloads
71
Comments
0
Likes
8

Embeds 46,564

http://java.dzone.com 44,181
http://enzine.tistory.com 969
http://www.billrun.net 336
http://top-performance.blogspot.co.il 231
http://top-performance.blogspot.com 193
http://mongodbwise.wordpress.com 91
http://www.google.com 81
http://architects.dzone.com 34
http://top-performance.blogspot.it 29
http://top-performance.blogspot.kr 28
http://translate.googleusercontent.com 27
http://top-performance.blogspot.in 24
http://top-performance.blogspot.co.uk 22
http://top-performance.blogspot.com.es 20
https://twitter.com 18
http://top-performance.blogspot.ru 17
http://top-performance.blogspot.nl 15
http://top-performance.blogspot.no 14
http://top-performance.blogspot.fr 14
http://www.google.co.in 14
http://top-performance.blogspot.tw 13
http://top-performance.blogspot.com.br 12
http://www.google.co.il 11
http://top-performance.blogspot.hk 11
http://top-performance.blogspot.de 11
http://www.google.co.uk 9
http://webcache.googleusercontent.com 8
https://translate.googleusercontent.com 8
http://top-performance.blogspot.ca 6
http://www.google.com.mx 5
http://top-performance.blogspot.com.au 5
http://www.google.it 5
http://top-performance.blogspot.jp 5
http://www.google.ca 5
http://top-performance.blogspot.cz 5
http://www.google.se 4
https://mongodbwise.wordpress.com 4
http://www.google.com.br 4
http://www.slideee.com 4
http://www.google.es 4
http://www.google.com.hk 3
http://www.google.com.vn 3
http://www.google.nl 3
http://www.google.ae 3
http://top-performance.blogspot.se 3
http://top-performance.blogspot.ch 3
https://www.google.com 3
http://top-performance.blogspot.ro 3
http://www.google.by 2
http://www.google.com.my 2

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Billing on top of MongoDB Ofer Cohen, S.D.O.C. Ltd. MUG IL, February 2014
  • 2. Agenda 1. 2. 3. 4. 5. 6. 7. 8. Who am I How it all began Small Billing Main Billing Infrastructure Transactions Maintainence Q&A
  • 3. Agenda 1. 2. 3. 4. 5. 6. 7. 8. Who am I How it all began Small Billing Main Billing Infrastructure Transactions Maintainence Q&A
  • 4. Who am I ● ● ● ● ● Open Source Evangelist Board member Started from DevOp Shifted to Biz Co-Founder at S.D.O.C. Ltd. ○ open source solutions for enterprises ● Mongo user since 2012
  • 5. Agenda 1. 2. 3. 4. 5. 6. 7. 8. Who am I How it all began Small Billing Main Billing Infrastructure Transactions Maintainence Q&A
  • 6. How it all began ● Partnering a startup Golan Telecom (GT) ● Telecom CDR first use - Anti-Fraud ○ CDR = Call/Charge detail record ● CDR Types: MOC, MTC & Data ○ 3 Different data structure ○ MOC/MTC duration=0 means SMS
  • 7. SQL Implementation $base_query = "SELECT imsi FROM moc WHERE callEventStartTimeStamp >=" . $start . " UNION SELECT imsi FROM mtc WHERE callEventStartTimeStamp >=" . $start $base_query = "SELECT imsi FROM (" . $base_query . ") AS qry "; if (isset($args['imsi'])) $base_query .= "WHERE imsi = '" . $this->_connection->real_escape_string($args['imsi']) . "'"; $base_query .= "GROUP BY imsi "; $mtc_join_query = "SELECT 'mtc' AS type, imsi, SUM(callEventDuration) AS duration, SUM(CEILING(callEventDuration/60)*60) AS duration_round " . ", SUM(chargeAmount) charge " . ", SUM(IF(SUBSTRING(callingNumber, 1, 3)='972', callEventDuration, IF(CHAR_LENGTH(callingNumber)<=10, callEventDuration, 0))) AS israel_duration " . ", SUM(IF(SUBSTRING(callingNumber, 1, 3)='972', CEILING(callEventDuration/60)*60, IF(CHAR_LENGTH(callingNumber)<=10, CEILING (callEventDuration/60)*60, 0))) AS israel_duration_round " . ", SUM(IF(SUBSTRING(callingNumber, 1, 3)!='972', IF(CHAR_LENGTH(callingNumber)>10, callEventDuration, 0), 0)) AS non_israel_duration " . ", SUM(IF(SUBSTRING(callingNumber, 1, 3)!='972', IF(CHAR_LENGTH(callingNumber)>10, CEILING(callEventDuration/60)*60, 0), 0)) AS non_israel_duration_round " . ", SUM(IF(callEventDuration = 0, 1, 0)) AS sms_count " . "FROM mtc " . "WHERE callEventStartTimeStamp >=" . $start . " " . "GROUP BY type, imsi";
  • 8. Implementation cont. $moc_join_query = "SELECT 'moc' AS type, imsi, SUM(callEventDuration) AS duration, SUM(CEILING(callEventDuration/60)*60) AS duration_round " . ", SUM(chargeAmount) charge " . ", SUM(IF(SUBSTRING(connectedNumber, 1, 3)='972', callEventDuration, IF(CHAR_LENGTH(connectedNumber)<=10, callEventDuration, 0))) AS israel_duration " . ", SUM(IF(SUBSTRING(connectedNumber, 1, 3)='972', CEILING(callEventDuration/60)*60, IF(CHAR_LENGTH(connectedNumber)<=10, CEILING (callEventDuration/60)*60, 0))) AS israel_duration_round " . ", SUM(IF(SUBSTRING(connectedNumber, 1, 3)!='972', IF(CHAR_LENGTH(connectedNumber)>10, callEventDuration, 0), 0)) AS non_israel_duration " . ", SUM(IF(SUBSTRING(connectedNumber, 1, 3)!='972', IF(CHAR_LENGTH(connectedNumber)>10, CEILING(callEventDuration/60)*60, 0), 0)) AS non_israel_duration_round " . ", SUM(IF(callEventDuration = 0, 1, 0)) AS sms_count " . "FROM moc " . "WHERE callEventStartTimeStamp >=" . $start . " " . "GROUP BY type, imsi"; // $gprs_join_query = $this->build_query($args, "gprs");
  • 9. It still continues $group_query = "SELECT base.imsi, moc.duration AS moc_duration, moc.charge AS moc_charge, " . "mtc.duration AS mtc_duration, mtc.charge AS mtc_charge, " . "mtc.duration_round AS mtc_duration_round, moc.duration_round AS moc_duration_round, " . "moc.israel_duration AS moc_israel_duration, moc.non_israel_duration AS moc_non_israel_duration, " . "moc.israel_duration_round AS moc_israel_duration_round, moc.non_israel_duration_round AS moc_non_israel_duration_round, " . "mtc.israel_duration AS mtc_israel_duration, mtc.non_israel_duration AS mtc_non_israel_duration, " . "mtc.israel_duration_round AS mtc_israel_duration_round, mtc.non_israel_duration_round AS mtc_non_israel_duration_round, " . "mtc.sms_count AS mtc_sms_count, moc.sms_count AS moc_sms_count " . "FROM " . "( " . $base_query . " ) AS base " . " LEFT JOIN (" . $mtc_join_query . " ) AS mtc ON base.imsi = mtc.imsi " . " LEFT JOIN (" . $moc_join_query . " ) AS moc ON base.imsi = moc.imsi " ; if (isset($args['limit'])) { $limit = (int) $args['limit']; } else { $limit = 100000; }
  • 10. How it works with Mongo? $base_match = array( '$match' => array( 'source' => 'nrtrde', 'unified_record_time' => array('$gte' => new MongoDate($charge_time)), ) ); $where = array( '$match' => array( 'record_type' => 'MOC', 'connectedNumber' => array('$regex' => '^972'), 'event_stamp' => array('$exists' => false), 'deposit_stamp' => array('$exists' => false), 'callEventDurationRound' => array('$gt' => 0), // not sms ), ); $group = array( '$group' => array( "_id" => '$imsi', "moc_israel" => array('$sum' => '$callEventDurationRound'), 'lines_stamps' => array('$addToSet' => '$stamp'), ), ); $project = array( '$project' => array( 'imsi' => '$_id', '_id' => 0, 'moc_israel' => 1, 'lines_stamps' => 1, ), );
  • 11. How it works with Mongo? $having = array( '$match' => array( 'moc_israel' => array('$gte' => Billrun_Factory::config()->getConfigValue('nrtrde.thresholds.moc.israel')) ), ); $moc_israel = $lines->aggregate($base_match, $where, $group, $project, $having); //sms out to all numbers $where['$match']['record_type'] = 'MOC'; $where['$match']['callEventDurationRound'] = 0; $group['$group']['sms_out'] = $group['$group']['mtc_all']; unset($group['$group']['mtc_all']); unset($having['$match']['mtc_all']); $group['$group']['sms_out'] = array('$sum' => 1); $having['$match']['sms_out'] = array('$gte' => Billrun_Factory::config()->getConfigValue('nrtrde.thresholds.smsout')); $project['$project']['sms_out'] = 1; unset($project['$project']['mtc_all']); $sms_out = $lines->aggregate($base_match, $where, $group, $project, $having);
  • 12. Agenda 1. 2. 3. 4. 5. 6. 7. 8. Who am I How it all began Small Billing Main Billing Infrastructure Transactions Maintainence Q&A
  • 13. Next - Small (ILDs) Billing ● Billing only for ILDs (A.K.A. MABAL) ○ Received 6 types of CDRs *almost* the same ○ Each MABAL do what is under his 7th letter ● We decide from day 1 to go with Mongo
  • 14. Small Billing - TCO ● 2 weeks of dev ● 2 days of deployment and first run ● 10,000 invoices ● Low maintenance=>Avg of 2 hours per month ● Base architecture for the main billing
  • 15. Agenda 1. 2. 3. 4. 5. 6. 7. 8. Who am I How it all began Small Billing Main Billing Infrastructure Transactions Maintainence Q&A
  • 16. Main Billing storage requirements ● 10 types of CDRs ○ Some binary hierarchical - Nokia Siemens specification ● 500,000,000 CDRs per month ● Requirement to save each CDR in DB storage. ● Fields can be added on the fly
  • 17. Main Billing storage size ● 500,000,000/28 days/24 hr/60 min/60 sec ○ 232 CDRs per second
  • 18. Main Billing storage size ● 500,000,000/5 days/24 hr/60 min/60 sec ○ 1152 CDRs per second
  • 19. Main Billing storage size ● Today GT have 400,000 subscribers ● Might be 800,000 in the near future ● Data usage (CDRs) increasing (40% of the storage) ● Zero downtime ● 2 data centers redundancy
  • 20. Agenda 1. 2. 3. 4. 5. 6. 7. 8. Who am I How it all began Small Billing Main Billing Infrastructure Transactions Maintainence Q&A
  • 21. Infrastructure Thanks to Eran Uzan
  • 22. Sharding ● Shard key ○ Requirement for good balancing ● Updates are really fast with shard key ○ No collection lock yet ○ But lock only part of the DB (1 shard)
  • 23. The size [GB] > db.lines.stats(1024*1024*1024) { "sharded" : true, "ns" : "billing.lines", "count" : 1436122538, "numExtents" : 771, "size" : 1275, "storageSize" : 1302, "totalIndexSize" : 355, "indexSizes" : { "_id_" : 36, "aid_1" : 36, "aid_1_urt_1" : 48, "sid_1" : 36, "stamp_1" : 126, "type_1" : 30, "urt_1" : 30 }, "avgObjSize" : 8.878072492167796e-7, //954.7864099064851 Bytes "nindexes" : 7, "nchunks" : 21516,
  • 24. The size [GB] - each shard "rs0" : { "rs1" : { "ns" : "billing.lines", "ns" : "billing.lines", "count" : 238827456, "count" : 239694511, "size" : 212, "size" : 213, "avgObjSize" : 8.876701345426549e-7, "avgObjSize" : 8.886311126248528e-7, "storageSize" : 217, "storageSize" : 217, "numExtents" : 130, "numExtents" : 130, "nindexes" : 7, "nindexes" : 7, "lastExtentSize" : 1, "lastExtentSize" : 1, "paddingFactor" : 1.0000000000335132, "paddingFactor" : 1.0000000000337341, "systemFlags" : 1, "systemFlags" : 1, "userFlags" : 0, "userFlags" : 0, "totalIndexSize" : 59, "totalIndexSize" : 59, "indexSizes" : { "indexSizes" : { "_id_" : 6, "_id_" : 6, "stamp_1" : 21, "stamp_1" : 21, "urt_1" : 5, "urt_1" : 5, "aid_1" : 6, "aid_1" : 6, "sid_1" : 6, "sid_1" : 6, "type_1" : 5, "type_1" : 5, "aid_1_urt_1" : 8 "aid_1_urt_1" : 8 }, }, "ok" : 1 }, "ok" : 1 },
  • 25. Pay attention ● With SSD you get x20 more performance ○ 5000 lines/second inserted during peak ● Mongo loves RAM ○ All used indexes must to be in RAM ○ Means, each shard have 64 GB in our env
  • 26. Agenda 1. 2. 3. 4. 5. 6. 7. 8. Who am I How it all began Small Billing Main Billing Infrastructure Transactions Maintainence Q&A
  • 27. Billing - Transactions ● ● ● ● ● ● ● write concern = 1 (by default) Unique key findAndModify (A.K.A FAM) value=oldValue 2 phase commit on app side Auto increment (invoice id) Transaction lock by design
  • 28. Billing - Transactions ● More in our blog post, written by Shani Dalal http://www.billrun.net/mongo-acid
  • 29. Agenda 1. 2. 3. 4. 5. 6. 7. 8. Who am I How it all began Small Billing Main Billing Infrastructure Transactions Maintenance, BI & TCO Q&A
  • 30. Billing - Maintenance ● MongoDB subscription ○ Leverage your professionalism ○ Special for enterprise or start-up ○ Low TCO ● Monitoring ○ Built in commands for kick-off ○ MMS the best tool for this purpose
  • 31. Billing - Maintenance ● Backup - hot or cold ○ With replica it’s pretty easy (without downtime) ○ Can be done with MMS or simple script
  • 32. Billing - BI ● MongoDB to SQL ○ Script convert hierarchy to one dimensional object ○ Script to CSV to MySQL ● Pentaho BI tool
  • 33. Billing - TCO ● 3 months dev ● 1 month without SSD ● 2 months QA ● Solution can be extends easily ○ Without app change
  • 34. Q&A MUG IL, February 2014
  • 35. Thank you Ofer Cohen, S.D.O.C. Ltd. ofer@billrun.net @oc666 MUG IL, February 2014