BillRun - Billing on top of MongoDB | MUG IL, Feb 2014

137,742 views

Published on

BillRun is open-source billing solution for enterprise, which built on top of MongoDB.
This presentation took place on Israeli MongoDB User Group meeting, February 20, 2014, at John Bryce college, Tel Aviv.

Find out more: http://billrun.net

Published in: Technology

BillRun - Billing on top of MongoDB | MUG IL, Feb 2014

  1. 1. Billing on top of MongoDB Ofer Cohen, S.D.O.C. Ltd. MUG IL, February 2014
  2. 2. Agenda 1. 2. 3. 4. 5. 6. 7. 8. Who am I How it all began Small Billing Main Billing Infrastructure Transactions Maintainence Q&A
  3. 3. Agenda 1. 2. 3. 4. 5. 6. 7. 8. Who am I How it all began Small Billing Main Billing Infrastructure Transactions Maintainence Q&A
  4. 4. Who am I ● ● ● ● ● Open Source Evangelist Board member Started from DevOp Shifted to Biz Co-Founder at S.D.O.C. Ltd. ○ open source solutions for enterprises ● Mongo user since 2012
  5. 5. Agenda 1. 2. 3. 4. 5. 6. 7. 8. Who am I How it all began Small Billing Main Billing Infrastructure Transactions Maintainence Q&A
  6. 6. How it all began ● Partnering a startup Golan Telecom (GT) ● Telecom CDR first use - Anti-Fraud ○ CDR = Call/Charge detail record ● CDR Types: MOC, MTC & Data ○ 3 Different data structure ○ MOC/MTC duration=0 means SMS
  7. 7. SQL Implementation $base_query = "SELECT imsi FROM moc WHERE callEventStartTimeStamp >=" . $start . " UNION SELECT imsi FROM mtc WHERE callEventStartTimeStamp >=" . $start $base_query = "SELECT imsi FROM (" . $base_query . ") AS qry "; if (isset($args['imsi'])) $base_query .= "WHERE imsi = '" . $this->_connection->real_escape_string($args['imsi']) . "'"; $base_query .= "GROUP BY imsi "; $mtc_join_query = "SELECT 'mtc' AS type, imsi, SUM(callEventDuration) AS duration, SUM(CEILING(callEventDuration/60)*60) AS duration_round " . ", SUM(chargeAmount) charge " . ", SUM(IF(SUBSTRING(callingNumber, 1, 3)='972', callEventDuration, IF(CHAR_LENGTH(callingNumber)<=10, callEventDuration, 0))) AS israel_duration " . ", SUM(IF(SUBSTRING(callingNumber, 1, 3)='972', CEILING(callEventDuration/60)*60, IF(CHAR_LENGTH(callingNumber)<=10, CEILING (callEventDuration/60)*60, 0))) AS israel_duration_round " . ", SUM(IF(SUBSTRING(callingNumber, 1, 3)!='972', IF(CHAR_LENGTH(callingNumber)>10, callEventDuration, 0), 0)) AS non_israel_duration " . ", SUM(IF(SUBSTRING(callingNumber, 1, 3)!='972', IF(CHAR_LENGTH(callingNumber)>10, CEILING(callEventDuration/60)*60, 0), 0)) AS non_israel_duration_round " . ", SUM(IF(callEventDuration = 0, 1, 0)) AS sms_count " . "FROM mtc " . "WHERE callEventStartTimeStamp >=" . $start . " " . "GROUP BY type, imsi";
  8. 8. Implementation cont. $moc_join_query = "SELECT 'moc' AS type, imsi, SUM(callEventDuration) AS duration, SUM(CEILING(callEventDuration/60)*60) AS duration_round " . ", SUM(chargeAmount) charge " . ", SUM(IF(SUBSTRING(connectedNumber, 1, 3)='972', callEventDuration, IF(CHAR_LENGTH(connectedNumber)<=10, callEventDuration, 0))) AS israel_duration " . ", SUM(IF(SUBSTRING(connectedNumber, 1, 3)='972', CEILING(callEventDuration/60)*60, IF(CHAR_LENGTH(connectedNumber)<=10, CEILING (callEventDuration/60)*60, 0))) AS israel_duration_round " . ", SUM(IF(SUBSTRING(connectedNumber, 1, 3)!='972', IF(CHAR_LENGTH(connectedNumber)>10, callEventDuration, 0), 0)) AS non_israel_duration " . ", SUM(IF(SUBSTRING(connectedNumber, 1, 3)!='972', IF(CHAR_LENGTH(connectedNumber)>10, CEILING(callEventDuration/60)*60, 0), 0)) AS non_israel_duration_round " . ", SUM(IF(callEventDuration = 0, 1, 0)) AS sms_count " . "FROM moc " . "WHERE callEventStartTimeStamp >=" . $start . " " . "GROUP BY type, imsi"; // $gprs_join_query = $this->build_query($args, "gprs");
  9. 9. It still continues $group_query = "SELECT base.imsi, moc.duration AS moc_duration, moc.charge AS moc_charge, " . "mtc.duration AS mtc_duration, mtc.charge AS mtc_charge, " . "mtc.duration_round AS mtc_duration_round, moc.duration_round AS moc_duration_round, " . "moc.israel_duration AS moc_israel_duration, moc.non_israel_duration AS moc_non_israel_duration, " . "moc.israel_duration_round AS moc_israel_duration_round, moc.non_israel_duration_round AS moc_non_israel_duration_round, " . "mtc.israel_duration AS mtc_israel_duration, mtc.non_israel_duration AS mtc_non_israel_duration, " . "mtc.israel_duration_round AS mtc_israel_duration_round, mtc.non_israel_duration_round AS mtc_non_israel_duration_round, " . "mtc.sms_count AS mtc_sms_count, moc.sms_count AS moc_sms_count " . "FROM " . "( " . $base_query . " ) AS base " . " LEFT JOIN (" . $mtc_join_query . " ) AS mtc ON base.imsi = mtc.imsi " . " LEFT JOIN (" . $moc_join_query . " ) AS moc ON base.imsi = moc.imsi " ; if (isset($args['limit'])) { $limit = (int) $args['limit']; } else { $limit = 100000; }
  10. 10. How it works with Mongo? $base_match = array( '$match' => array( 'source' => 'nrtrde', 'unified_record_time' => array('$gte' => new MongoDate($charge_time)), ) ); $where = array( '$match' => array( 'record_type' => 'MOC', 'connectedNumber' => array('$regex' => '^972'), 'event_stamp' => array('$exists' => false), 'deposit_stamp' => array('$exists' => false), 'callEventDurationRound' => array('$gt' => 0), // not sms ), ); $group = array( '$group' => array( "_id" => '$imsi', "moc_israel" => array('$sum' => '$callEventDurationRound'), 'lines_stamps' => array('$addToSet' => '$stamp'), ), ); $project = array( '$project' => array( 'imsi' => '$_id', '_id' => 0, 'moc_israel' => 1, 'lines_stamps' => 1, ), );
  11. 11. How it works with Mongo? $having = array( '$match' => array( 'moc_israel' => array('$gte' => Billrun_Factory::config()->getConfigValue('nrtrde.thresholds.moc.israel')) ), ); $moc_israel = $lines->aggregate($base_match, $where, $group, $project, $having); //sms out to all numbers $where['$match']['record_type'] = 'MOC'; $where['$match']['callEventDurationRound'] = 0; $group['$group']['sms_out'] = $group['$group']['mtc_all']; unset($group['$group']['mtc_all']); unset($having['$match']['mtc_all']); $group['$group']['sms_out'] = array('$sum' => 1); $having['$match']['sms_out'] = array('$gte' => Billrun_Factory::config()->getConfigValue('nrtrde.thresholds.smsout')); $project['$project']['sms_out'] = 1; unset($project['$project']['mtc_all']); $sms_out = $lines->aggregate($base_match, $where, $group, $project, $having);
  12. 12. Agenda 1. 2. 3. 4. 5. 6. 7. 8. Who am I How it all began Small Billing Main Billing Infrastructure Transactions Maintainence Q&A
  13. 13. Next - Small (ILDs) Billing ● Billing only for ILDs (A.K.A. MABAL) ○ Received 6 types of CDRs *almost* the same ○ Each MABAL do what is under his 7th letter ● We decide from day 1 to go with Mongo
  14. 14. Small Billing - TCO ● 2 weeks of dev ● 2 days of deployment and first run ● 10,000 invoices ● Low maintenance=>Avg of 2 hours per month ● Base architecture for the main billing
  15. 15. Agenda 1. 2. 3. 4. 5. 6. 7. 8. Who am I How it all began Small Billing Main Billing Infrastructure Transactions Maintainence Q&A
  16. 16. Main Billing storage requirements ● 10 types of CDRs ○ Some binary hierarchical - Nokia Siemens specification ● 500,000,000 CDRs per month ● Requirement to save each CDR in DB storage. ● Fields can be added on the fly
  17. 17. Main Billing storage size ● 500,000,000/28 days/24 hr/60 min/60 sec ○ 232 CDRs per second
  18. 18. Main Billing storage size ● 500,000,000/5 days/24 hr/60 min/60 sec ○ 1152 CDRs per second
  19. 19. Main Billing storage size ● Today GT have 400,000 subscribers ● Might be 800,000 in the near future ● Data usage (CDRs) increasing (40% of the storage) ● Zero downtime ● 2 data centers redundancy
  20. 20. Agenda 1. 2. 3. 4. 5. 6. 7. 8. Who am I How it all began Small Billing Main Billing Infrastructure Transactions Maintainence Q&A
  21. 21. Infrastructure Thanks to Eran Uzan
  22. 22. Sharding ● Shard key ○ Requirement for good balancing ● Updates are really fast with shard key ○ No collection lock yet ○ But lock only part of the DB (1 shard)
  23. 23. The size [GB] > db.lines.stats(1024*1024*1024) { "sharded" : true, "ns" : "billing.lines", "count" : 1436122538, "numExtents" : 771, "size" : 1275, "storageSize" : 1302, "totalIndexSize" : 355, "indexSizes" : { "_id_" : 36, "aid_1" : 36, "aid_1_urt_1" : 48, "sid_1" : 36, "stamp_1" : 126, "type_1" : 30, "urt_1" : 30 }, "avgObjSize" : 8.878072492167796e-7, //954.7864099064851 Bytes "nindexes" : 7, "nchunks" : 21516,
  24. 24. The size [GB] - each shard "rs0" : { "rs1" : { "ns" : "billing.lines", "ns" : "billing.lines", "count" : 238827456, "count" : 239694511, "size" : 212, "size" : 213, "avgObjSize" : 8.876701345426549e-7, "avgObjSize" : 8.886311126248528e-7, "storageSize" : 217, "storageSize" : 217, "numExtents" : 130, "numExtents" : 130, "nindexes" : 7, "nindexes" : 7, "lastExtentSize" : 1, "lastExtentSize" : 1, "paddingFactor" : 1.0000000000335132, "paddingFactor" : 1.0000000000337341, "systemFlags" : 1, "systemFlags" : 1, "userFlags" : 0, "userFlags" : 0, "totalIndexSize" : 59, "totalIndexSize" : 59, "indexSizes" : { "indexSizes" : { "_id_" : 6, "_id_" : 6, "stamp_1" : 21, "stamp_1" : 21, "urt_1" : 5, "urt_1" : 5, "aid_1" : 6, "aid_1" : 6, "sid_1" : 6, "sid_1" : 6, "type_1" : 5, "type_1" : 5, "aid_1_urt_1" : 8 "aid_1_urt_1" : 8 }, }, "ok" : 1 }, "ok" : 1 },
  25. 25. Pay attention ● With SSD you get x20 more performance ○ 5000 lines/second inserted during peak ● Mongo loves RAM ○ All used indexes must to be in RAM ○ Means, each shard have 64 GB in our env
  26. 26. Agenda 1. 2. 3. 4. 5. 6. 7. 8. Who am I How it all began Small Billing Main Billing Infrastructure Transactions Maintainence Q&A
  27. 27. Billing - Transactions ● ● ● ● ● ● ● write concern = 1 (by default) Unique key findAndModify (A.K.A FAM) value=oldValue 2 phase commit on app side Auto increment (invoice id) Transaction lock by design
  28. 28. Billing - Transactions ● More in our blog post, written by Shani Dalal http://www.billrun.net/mongo-acid
  29. 29. Agenda 1. 2. 3. 4. 5. 6. 7. 8. Who am I How it all began Small Billing Main Billing Infrastructure Transactions Maintenance, BI & TCO Q&A
  30. 30. Billing - Maintenance ● MongoDB subscription ○ Leverage your professionalism ○ Special for enterprise or start-up ○ Low TCO ● Monitoring ○ Built in commands for kick-off ○ MMS the best tool for this purpose
  31. 31. Billing - Maintenance ● Backup - hot or cold ○ With replica it’s pretty easy (without downtime) ○ Can be done with MMS or simple script
  32. 32. Billing - BI ● MongoDB to SQL ○ Script convert hierarchy to one dimensional object ○ Script to CSV to MySQL ● Pentaho BI tool
  33. 33. Billing - TCO ● 3 months dev ● 1 month without SSD ● 2 months QA ● Solution can be extends easily ○ Without app change
  34. 34. Q&A MUG IL, February 2014
  35. 35. Thank you Ofer Cohen, S.D.O.C. Ltd. ofer@billrun.net @oc666 MUG IL, February 2014

×