Successfully reported this slideshow.
Your SlideShare is downloading. ×

Mobile Synchronization Patterns for Large Volumes of Data

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Loading in …3
×

Check these out next

1 of 43 Ad

Mobile Synchronization Patterns for Large Volumes of Data

Download to read offline

Do your mobile business apps require large amounts of data? Is the complexity of "offline" causing you to lose sleep? Come and learn "4 Best Practices," and their supporting patterns, for dealing with the synchronization of large data volumes in mobile apps built with OutSystems. In addition, we will discuss how to avoid the problems that can pop up whenever dealing with these kinds of applications.

Do your mobile business apps require large amounts of data? Is the complexity of "offline" causing you to lose sleep? Come and learn "4 Best Practices," and their supporting patterns, for dealing with the synchronization of large data volumes in mobile apps built with OutSystems. In addition, we will discuss how to avoid the problems that can pop up whenever dealing with these kinds of applications.

Advertisement
Advertisement

More Related Content

Slideshows for you (20)

Similar to Mobile Synchronization Patterns for Large Volumes of Data (20)

Advertisement

More from OutSystems (20)

Recently uploaded (20)

Advertisement

Mobile Synchronization Patterns for Large Volumes of Data

  1. 1. | Mobile Synchronization Patterns for Large Volumes of Data
  2. 2. || Mobile Synchronization Patterns for Large Volumes of Data João Pereira Developer | Do iT Lean @ in joao.pereira@doilean.com /joão-pedro-martins-pereira-8a115b120
  3. 3. || Mobile Synchronization Patterns for Large Volumes of Data Agenda
  4. 4. | Mobile Synchronization Patterns for Large Volumes of Data ● The Mobile Challenge ● Let’s Recap OutSystems Best Practices ● When We Can’t Follow Best Practices . . . ● Solving the Challenge! ● Patterns in Action
  5. 5. || Mobile Synchronization Patterns for Large Volumes of Data The Mobile Challenge
  6. 6. | Mobile Synchronization Patterns for Large Volumes of Data ● Mobile Devices are more powerful than ever ● Mobile apps are faster and more intuitive than ever ● Users expect your app to perform like all other apps they use ● How can we do it with OutSystems?
  7. 7. || Mobile Synchronization Patterns for Large Volumes of Data Let’s Recap OutSystems Best Practices
  8. 8. | Mobile Synchronization Patterns for Large Volumes of Data 1. Lightweight Local Database ● Design the local storage data model using only the attributes that you need ● Transform the local data model to avoid complex queries like having multiple JOINS ● Identify the best data synchronization patterns for your use case and define exactly when to run them
  9. 9. | Mobile Synchronization Patterns for Large Volumes of Data 1.1. Lightweight Local Database Example Server DB Local DB
  10. 10. | Mobile Synchronization Patterns for Large Volumes of Data 2. Control the Amount of Synchronized Data ● We can control the data fetched from the server using: ■ Sync Unit ■ Local Sync Properties ■ Only get the latest data from the server ■ Only get the data that we really need
  11. 11. | Mobile Synchronization Patterns for Large Volumes of Data 3. SyncUnit - Defining Synchronization Moments
  12. 12. | Mobile Synchronization Patterns for Large Volumes of Data Makes a sync when the add button is Clicked 4. Use the TriggerOfflineDataSync Synchronize Data in the Background Without Freezing the UI
  13. 13. | Mobile Synchronization Patterns for Large Volumes of Data 5. Avoid Multiple Server Requests On Client Actions Consolidate Server Requests into one Action Client Server
  14. 14. | Mobile Synchronization Patterns for Large Volumes of Data ● Evaluate network requirements use cases ■ Define Local Storage strategy ■ Optimize data sync moments ● Evaluate network status in runtime ■ Adapt behavior accordingly ■ Let the user know what’s happening 6. Control the Network Status
  15. 15. | Mobile Synchronization Patterns for Large Volumes of Data ● Corporate vs Public App ● Multiple types of devices ● Different operating systems (Android, iOS) 7. Fined Tune Data Transfers
  16. 16. || Mobile Synchronization Patterns for Large Volumes of Data When We Can’t Follow Best Practices . . .
  17. 17. | Mobile Synchronization Patterns for Large Volumes of Data 1. Mobile App = Web Version ● Client Demands: ■ Must have same functionality in web and mobile ■ The Mobile App is not going to be built based on mobile specific use cases but on the web application ● Developer Challenge ■ Must adapt the best practices to support this situation
  18. 18. | Mobile Synchronization Patterns for Large Volumes of Data
  19. 19. | Mobile Synchronization Patterns for Large Volumes of Data ● Cannot Create a Lightweight Device Database ■ Requiring JOIN’s will influence performance of the app ■ Must find ways to overcome this limitation in terms of Sync Moments ■ Analyze the use-cases the application needs ■ Minimize sync of tables that have a high volume of data ■ Minimize using data that have multiple joins on Mobile App Screens 2. All Data Must Be on the Mobile App
  20. 20. || Mobile Synchronization Patterns for Large Volumes of Data Solving the Challenge!
  21. 21. | Mobile Synchronization Patterns for Large Volumes of Data Controlling Time & Matter
  22. 22. | Mobile Synchronization Patterns for Large Volumes of Data ● Case A - Infrequent Change ■ Treat like static data ■ Synchronize all in the first use of the application ■ Sync only when the data changes ■ Inside of the Mobile App only sync the tables & data that is really needed 1. Understanding Your Data
  23. 23. | Mobile Synchronization Patterns for Large Volumes of Data ● Case B - Frequent Change ■ Like in the previous case - sync all data in first use ■ On Local to Server, only sync the modified data ■ Syncing Modified Data - Depending on the use- case, choose one of these scenarios ■ Last Write Wins ■ Server Always Win ■ Conflict Mechanism ■ Discard mechanism to prevent inconsistent data if sync processes went wrong 1. Understanding Your Data
  24. 24. | Mobile Synchronization Patterns for Large Volumes of Data ● SyncUnit ■ Create a static entity with the tables we need to sync ● Create a Sync Unit History ■ Keep track of tables that need sync ■ Last date the data was changed 2. Build Your Sync Mechanism
  25. 25. || Mobile Synchronization Patterns for Large Volumes of Data Patterns in Action
  26. 26. | Mobile Synchronization Patterns for Large Volumes of Data ● Use case – Need to have 3000 records available at the start of the application: ■ First Use Sync Strategy – Sync all the data - must guaranty the network is 3G or superior ■ When the tables get changed just synchronize the ones that changed ■ Sync only the data that is needed to execute the functionality that is needed on the Mobile App Real World Example
  27. 27. | Mobile Synchronization Patterns for Large Volumes of Data ● Create the SyncUnitHistory Record ■ Will be created on the CRUD – Create or Update Actions of the tables we want to sync ■ This table will have SyncUnitId and the Last Updated Date ■ Used to know the date of the last changed record of a table that we will need to sync How to know when and what needs to be Synced?
  28. 28. | Mobile Synchronization Patterns for Large Volumes of Data ● Get SyncUnits To Sync: ■ Action to check which SyncUnits need to be refreshed on your mobile app ■ Receives the Device Date and returns the list of ids of the tables that need to be sync ■ The action is used in the mobile synchronization action How to know what tables we need to Sync ?
  29. 29. | Mobile Synchronization Patterns for Large Volumes of Data ● Create a Control of Sync in Local Storage: ■ Sync controller in the local storage ■ This will kept track of the tables that already have been synced and when it was done Client-Side LastSync Controller
  30. 30. | Mobile Synchronization Patterns for Large Volumes of Data Example - MaterialType table Table with large volume of data
  31. 31. | Mobile Synchronization Patterns for Large Volumes of Data ● Sync of Material Type : ■ Get the Material-Types that are present on the server ■ In the first time will get all ■ Next time is filter by date from, date to and only the active ones ■ LocaleId is passed because app is multilingual ■ As output we get the Server Material Types and the total records 1. Create Sync Material-Type action
  32. 32. | Mobile Synchronization Patterns for Large Volumes of Data ● Sync of Material Type client side : ■ Create our Material-Type Local Storage; ■ Last Sync table to get the interval of time the date will be get ■ Updates the controller for the sync unit Material-Type 2. Action on client side to fill Material-Type local table
  33. 33. | Mobile Synchronization Patterns for Large Volumes of Data ● OfflineDataSync Action : ■ Sync the Local Data into the Server ■ If no errors occur, we start the sync of the large volume of data tables ■ Sync the tables with less volume of data 3. Mobile synchronization process
  34. 34. | Mobile Synchronization Patterns for Large Volumes of Data ● Sync_LC_Check Action : ■ Check’s our connection because we need to have network ■ Get SyncUnits to update list to check which tables need to be sync ■ Update the sync controller Last Sync and run the action Sync_LC_Update 4. Trigger and know which tables we going to Sync
  35. 35. | Mobile Synchronization Patterns for Large Volumes of Data ● Sync_LC_Update Action : ■ This action will receive as input the SyncUnit and only update the respective table 5. Action that Syncs the large volume of data tables
  36. 36. | Mobile Synchronization Patterns for Large Volumes of Data Dealing with Conflict
  37. 37. | Mobile Synchronization Patterns for Large Volumes of Data ● Use Case: ■ When multiple users can change the same data and the mobile app can be used offline ■ You need a mechanism to determine which data is conflicting ■ You need a screen so the user can choose the correct data How to Create a Conflict Screen on Your Mobile App
  38. 38. | Mobile Synchronization Patterns for Large Volumes of Data ● Solution: ■ Create a Static Table with the Entities name that we want to make conflict detection and resolution ■ Then use that static entity and create one local entity that will keep the table name and a flag if exist conflict ■ When we try to sync or update the local record into the server we need to check if a conflict exists ■ If a conflict exists create an entry on the table of conflicts and add a record to the list How to Create a Conflict Screen on Your Mobile App
  39. 39. | Mobile Synchronization Patterns for Large Volumes of Data 1. Creating the list of Conflicts
  40. 40. | Mobile Synchronization Patterns for Large Volumes of Data ● Conflict Resolution: ■ After click on the list the Conflict Element’s will appear ■ Local & Server record are serialize & deserialize into a JSON ■ Screen less heavy and with better performance ■ Can be optimized to choose attribute by attribute instead of record 2. Conflict Screen on Mobile App to Compare Data
  41. 41. | Mobile Synchronization Patterns for Large Volumes of Data
  42. 42. || Mobile Synchronization Patterns for Large Volumes of Data Q & A
  43. 43. | Thank You! @ injoao.pereira@doilean.com / joão-pedro-martins- pereira-8a115b120

Editor's Notes

  • My name is João Pereira
    Last year I was in your place
    So im so glad to be making this presentation
  • The Mobile Challenge
    Let’s Recap OutSystems Best Practices
    When We Can’t Follow Best Practices . . .
    Solving the Challenge!
    Patterns in Action
  • Mobile Devices are more powerful than ever
    Mobile apps are faster and more intuitive than ever
    Users expect your app to perform like all other apps they use
    How can we do it with OutSystems
  • Design the local storage data model using only the attributes that you need, instead of all attributes of the corresponding server entity. Consider denormalizing the local data model to avoid complex queries like having multiple JOINS.
    Identify the best data synchronization patterns for your use case and define exactly when to run them.
  • To explain better the OutSystems Best Practice I created a small DB and a concept of an App that got the objective of having a platform where clients find car dealers which have cars in stock;
    The car dealers can add cars to his stock
    The clients can search by car dealer or model they want to buy
    Process of denormalization
    Simplify the connection between tables and minimize the amount of information
  • We can control the data fetched from the server using;
    Sync Unit –several flows of synchornization
    Local Sync Properties – Create a local entity to keep track of the last synchronization moment
    Only get the latest data from the server – Compare the LastSycnOn from that sync unit with the CreatedOn, UpdatedOn and DeletedOn attributes from the server DB to know which records to fetch
  • This image represents the third best practice using the sync unit to define the moments we want to sync our DB between local and server
    As you can see we define 3 moments
    1. Get New Stock – On the Menu of the application the user clicks on a buttons or slides and refresh the list of car with the more recent
    2. Remove sold cars , when a Car dealer changes the status of the car to available to sold
    3. When the Owner of car dealers add new Change the image to more general information ;
  • TriggerOfflineDataSync – explanation
    Devemos antecipar qual a informação que o utilizador precisa antes da mesma ser utilizada para assim obter uma fluidez dentro da aplicação
  • As you know each server action called inside cliente actions executes a request to the server so if we call several server actions inside cliente action that will leave to open and close multiple requests in the same action that is going to be less performance that use like the exemple above one server action inside a cliente actions.
    Is visible that in that server action know it’s no problem use more than one
    This need to be applied to normal client actions of the app as well to our synchronization actions that will get and sync all the information we need
  • Evaluate network requirements and use cases - If the application needs to be online or offline
    To evaluate the data sync moments
    Adapt behavior according to network strength, including offline
    Check how long we got to do the sync actions
    The moments we must sync all the information of the app.
    #############################################################
    Evaluate network requirements use cases
    Define Local Storage strategy
    Optimize data sync moments
    Evaluate network status in runtime
    Adapt behavior accordingly
    Let the user know what’s happening

  • Find the Optimal Value of Records to BULK Create According to the Different Processing Capabilities of Each Device
    Talk about 2 diferente scenarios – 1 . Corporate Use , 2. Target is the world
    B – B
    B – E
  • After I recap you some of the OutSystems Best Pratices I will show you that they don’t work for every situation and that sometimes we cannot use them
  • Must have same functionalities in web and mobile;
    The Data model cannot suffer the changes we want to have a lightweight database on the mobile app;
    The Mobile App is not going to be built base on the specific use cases but on the Web App
    We need to adapt the best practices the best we can on this situation

    When you ever
    Put the boss with the tie photo
  • Have JOIN’s that will influence the performance of the application;
    Must think ways to overcome this limitation in terms of Sync Moments and how to do it;
    Analyze with are the use-cases the application needs
    Try to minimize sync of tables that got a high volume of data . Find strategies to sync the information the minimum times possible
    In the Mobile App Screens minimize the data that have multiple joins . So if we need the data that have multiple joins try to use only that information on the mobile screen
  • How did you solve it ? In this part of the presentation we are going to show how to overcome the difficulties when we can’t use the best practices and have a huge database to manage in the local storage and need to sync a huge flow of data between Web and Mobile
  • In the case of not changes frequently;
    Take care of this tables like lookup;
    Just need be synchronize all in the first use of the application;
    When this tables get changes just synchronize the ones that real change and not all
    Sync only the data that is neded to execute all the functionalities that are really need on the mobile AppCase A - Infrequent Change
    Treat like static data
    Synchronize all in the first use of the application
    Sync only when the data changes
    Inside of the Mobile App only sync the tables & data that is really needed
  • Like in the previous case in the first use we need to sync all the tables;
    When modified the data only sync the modified back to the server;
    Choose well what information we want to override choosing one of this scenarios (Last Write Wins, Server always Win or Conflict Mechanism Detection)
    If we don’t have a good network, we must have a mechanism that discard a sync that was not fully completed in order to not have inconsistent data in our application

  • Create a Sync Unit History
    Keep track of tables that need sync
    Last date the data was changed
    SyncUnit
    Create a static entity with the tables we need to sync
  • Patterns in Action – I’m going to present you the real cases scenarios we use some strategies like the sync unit as table selector and how we do the sync of a big volume database
  • In the case of not changes frequently;
    Take care of this tables like lookup;
    Just need be synchronize all in the first use of the application;
    When this tables get changes just synchronize the ones that real change and not all
    Sync only the data that is needed to execute all the functionalities that are really need on the mobile App

    The data doesn’t change a lot
  • Inside the Core of the tables we want to sync was created:
    Static table called Sync Unit that contain all the tables we want to sync to the mobile app (SyncUnit.Label = MaterialType)
    A new table called SyncUnitHistory that will have the static Id from the previous table and the Last Updated Date
    The records of the SycnUnitHistory are going to be created on the actions of Created or Updated a entity we want to sync to mobile
  • Also in the Core module was created a action to see what are the sync units that needs to sync data
    This action receives the Device Date and returns the list of id’s of the tables that needs to be sync
    Is going to be used on the mobile synchronization action
  • As we done in the previous step, we create a control sync server table
    We need to create a LastSync table at will got the SyncUnit Id and the date from and date to of the Last Sync made on the device
  • Client actions and server actions created to do the sync:
    Was created a new module in the Mobile App to manage the Local Storage and Sync actions to use in the APP
    All Local Tables are in this module
    Going to present the example to one table Material Type that got a dependency that is Material Type with a List of values, that is called Material Type LOV Dependency
  • This action will get the Material Types that are present on the server
    In the first time will get all but on the next time is filter by date from, date to and only the active ones
    Because the application is multilingual, we need to pass the LocaleId we want to get the data
    As output we got the Server Material Types and the total records just to control how many data was sync at that time
  • This an action will create our Material Type that is going to be at the Local Storage
    Used the Last Sync table to get the interval of time we are going to get the data
    In the end updates the controller for the sync unit Material Type
  • This action will first check our connection because we need to make sure we got a good connection to sync large volumes of data and we going to use server-side actions
    The next step is using our Get SyncUnits to update list to check which tables need to be sync
    After get the list we update the sync controller Last Sync and run the action Sync_LC_Update
  • This action will receive as input the SyncUnit and only update that respective table
    Then will run the Sync of Material Type on this case that we talk on the previous slide
  • When multiple users can change the same data and the mobile app can be used offline
    How to create the data of conflict ?
    Where we choose the correct data ?
  • Create a Static Table with the Entities name that we want to make conflict detection and resolution
    Then use that static entity and create one local that will keep the table name and a flag if exist conflict
    When we try to sync the local record into the server we need to check if a conflict exists
    If a conflict exists create an entry on the table of conflicts and add a record to a list
  • Situation when we can check the conflict:
    When we try to save the record if we got internet connection, we can compare with the information that are present in the server at the moment
    When we try to sync the local data and put them into the server will check if that records was change
    In any of this cases is created a list with the table name and the number of conflicts that exist at that moment
  • When we click in element of the previous list will appear a detail screen so the user can choose the server data or local data
    This data will get the Local record and the Server record and serialize & deserialize then into a JSON
    That process will make the screen less heavy and with better performance
    Can be optimized to choose attribute by attribute instead of record

    #####################################################################################################
    I hope you enjoyed this presentation and next time you face this kind of challenge you remember what we talk about today !


  • I hope you enjoyed this presentation and next time you face this kind of challenge you remember what we talk about today !

×