Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

User-generated data in NetFortris – Connect Silicon Valley 2017


Published on

Speaker: John Kong, NetFortris/Fonality

NetFortris provides communication solutions for small and large businesses in various business sectors. Our users generate huge volumes of data while using our systems, and as our user base grew, we experienced constant production issues due to demand. Manipulating the data has posed a challenge in building scalable, highly available systems. After testing with several candidates, we chose Couchbase as the data store for storing user-generated data and cache. With several iterations of views, design doc, and N1QL, we optimized our production deployment substantially.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

User-generated data in NetFortris – Connect Silicon Valley 2017

  1. 1. John Kong NetFortris Inc.
  2. 2. • Netfortris provides an integrated phone, chat and text messaging system • Chat, Team/Group chat, Call, Conference, Call center. • HUD has the “remote control” – a fairly complex web/mobile app
  3. 3. • Users generate huge volumes of chat and call data  Chat and call data (include chat/call event)  Peak during work hours • Need to support different queries for user generated data  Per users, per conversation, per chat/call • A user’s configuration “state” is shared with all users of that customer User, conference, extension, device … Support all many different queries.
  4. 4. Controllers Services Dispatch Event Daos Message Queue RDMS
  5. 5. • Bad user experience  Slow response time - over 60 seconds to load page • User generated data is too big to manipulate efficiently in a single RDBMS instance  Weekly, sometime daily production issues due to data overload  Difficult (if not impossible) to partition RDMS • Difficult to scale as user base grows  Customer Support Issues for larger customers
  6. 6. Controllers Services Dispatch Event Daos CouchbaseMessage Queue RDMS
  7. 7. • Move user generated data from RDBMs to Couchbase with Java SDK • Get auto data partition, HA, Cache • Flexible in data definition, no need schema change when data model change. • Create two buckets for chat and calls • Create views to query user generated data
  8. 8. function (doc, meta) { if(doc.doc_type =='CL'){ emit([doc.owner_user_d, doc.iver], null) } }
  9. 9. • Store user generated data and as much customer/user management data in Couchbase as possible with Go SDK (Go with Couchbase gets superior performance over Java) • Use N1QL to manipulate data in Couchbase • CREATE INDEX `index_session_id` ON `chat_messages`(`session_id`) WHERE (`doc_type` = `CM`) USING GSI • Select `chat_messages`.* from `chat_messages` where sesssion_id = 111
  10. 10. SDK/N1QL Sample • Define N1QL query statement: • querySettings = gocb.NewN1qlQuery("select `fon_user`.* from 'fon_user` where owner = ? and ownerType = ? and customerId = ? and clientType = ? and docType = ?") • Run go to query: • bucket := this.CbPool.Borrow() qParams := []interface{}{userId, "USER", customerId, clientType, "settings"} • vResults, err := bucket.ExecuteN1qlQuery(querySettings, qParams)
  11. 11. Couchbase RDMS Daos query/cache Query all objects of the same type of a customer
  12. 12. o Response time – from over 60 seconds to sub-1 second o User generated data – from too big to under control o Weekly production issues – no more! o Scaling issues – no more! o Releases resources to new projects
  13. 13. • Eventually consistency issue o Define the view in separate design doc to reduce indexing duration. • Return key only by view query instead of full doc. • Use N1QL for new projects.
  14. 14. • Bulk operation for data structure o Insert the whole map/list o Retrieve range of elements of map or list.