Mobile Synchronization Patterns for Large Volumes of Data
•Download as PPTX, PDF•
2 likes•1,176 views
Do your mobile business apps require large amounts of data? Is the complexity of "offline" causing you to lose sleep? Come and learn "4 Best Practices," and their supporting patterns, for dealing with the synchronization of large data volumes in mobile apps built with OutSystems. In addition, we will discuss how to avoid the problems that can pop up whenever dealing with these kinds of applications.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Mobile Synchronization Patterns for Large Volumes of Data
Similar to Mobile Synchronization Patterns for Large Volumes of Data (20)
More from OutSystems
More from OutSystems (20)
Recently uploaded
Recently uploaded (20)
Mobile Synchronization Patterns for Large Volumes of Data
- 1. | Mobile Synchronization Patterns for Large Volumes of Data
- 2. || Mobile Synchronization Patterns for Large Volumes of Data João Pereira Developer | Do iT Lean @ in joao.pereira@doilean.com /joão-pedro-martins-pereira-8a115b120
- 3. || Mobile Synchronization Patterns for Large Volumes of Data Agenda
- 4. | Mobile Synchronization Patterns for Large Volumes of Data ● The Mobile Challenge ● Let’s Recap OutSystems Best Practices ● When We Can’t Follow Best Practices . . . ● Solving the Challenge! ● Patterns in Action
- 5. || Mobile Synchronization Patterns for Large Volumes of Data The Mobile Challenge
- 6. | Mobile Synchronization Patterns for Large Volumes of Data ● Mobile Devices are more powerful than ever ● Mobile apps are faster and more intuitive than ever ● Users expect your app to perform like all other apps they use ● How can we do it with OutSystems?
- 7. || Mobile Synchronization Patterns for Large Volumes of Data Let’s Recap OutSystems Best Practices
- 8. | Mobile Synchronization Patterns for Large Volumes of Data 1. Lightweight Local Database ● Design the local storage data model using only the attributes that you need ● Transform the local data model to avoid complex queries like having multiple JOINS ● Identify the best data synchronization patterns for your use case and define exactly when to run them
- 9. | Mobile Synchronization Patterns for Large Volumes of Data 1.1. Lightweight Local Database Example Server DB Local DB
- 10. | Mobile Synchronization Patterns for Large Volumes of Data 2. Control the Amount of Synchronized Data ● We can control the data fetched from the server using: ■ Sync Unit ■ Local Sync Properties ■ Only get the latest data from the server ■ Only get the data that we really need
- 11. | Mobile Synchronization Patterns for Large Volumes of Data 3. SyncUnit - Defining Synchronization Moments
- 12. | Mobile Synchronization Patterns for Large Volumes of Data Makes a sync when the add button is Clicked 4. Use the TriggerOfflineDataSync Synchronize Data in the Background Without Freezing the UI
- 13. | Mobile Synchronization Patterns for Large Volumes of Data 5. Avoid Multiple Server Requests On Client Actions Consolidate Server Requests into one Action Client Server
- 14. | Mobile Synchronization Patterns for Large Volumes of Data ● Evaluate network requirements use cases ■ Define Local Storage strategy ■ Optimize data sync moments ● Evaluate network status in runtime ■ Adapt behavior accordingly ■ Let the user know what’s happening 6. Control the Network Status
- 15. | Mobile Synchronization Patterns for Large Volumes of Data ● Corporate vs Public App ● Multiple types of devices ● Different operating systems (Android, iOS) 7. Fined Tune Data Transfers
- 16. || Mobile Synchronization Patterns for Large Volumes of Data When We Can’t Follow Best Practices . . .
- 17. | Mobile Synchronization Patterns for Large Volumes of Data 1. Mobile App = Web Version ● Client Demands: ■ Must have same functionality in web and mobile ■ The Mobile App is not going to be built based on mobile specific use cases but on the web application ● Developer Challenge ■ Must adapt the best practices to support this situation
- 18. | Mobile Synchronization Patterns for Large Volumes of Data
- 19. | Mobile Synchronization Patterns for Large Volumes of Data ● Cannot Create a Lightweight Device Database ■ Requiring JOIN’s will influence performance of the app ■ Must find ways to overcome this limitation in terms of Sync Moments ■ Analyze the use-cases the application needs ■ Minimize sync of tables that have a high volume of data ■ Minimize using data that have multiple joins on Mobile App Screens 2. All Data Must Be on the Mobile App
- 20. || Mobile Synchronization Patterns for Large Volumes of Data Solving the Challenge!
- 21. | Mobile Synchronization Patterns for Large Volumes of Data Controlling Time & Matter
- 22. | Mobile Synchronization Patterns for Large Volumes of Data ● Case A - Infrequent Change ■ Treat like static data ■ Synchronize all in the first use of the application ■ Sync only when the data changes ■ Inside of the Mobile App only sync the tables & data that is really needed 1. Understanding Your Data
- 23. | Mobile Synchronization Patterns for Large Volumes of Data ● Case B - Frequent Change ■ Like in the previous case - sync all data in first use ■ On Local to Server, only sync the modified data ■ Syncing Modified Data - Depending on the use- case, choose one of these scenarios ■ Last Write Wins ■ Server Always Win ■ Conflict Mechanism ■ Discard mechanism to prevent inconsistent data if sync processes went wrong 1. Understanding Your Data
- 24. | Mobile Synchronization Patterns for Large Volumes of Data ● SyncUnit ■ Create a static entity with the tables we need to sync ● Create a Sync Unit History ■ Keep track of tables that need sync ■ Last date the data was changed 2. Build Your Sync Mechanism
- 25. || Mobile Synchronization Patterns for Large Volumes of Data Patterns in Action
- 26. | Mobile Synchronization Patterns for Large Volumes of Data ● Use case – Need to have 3000 records available at the start of the application: ■ First Use Sync Strategy – Sync all the data - must guaranty the network is 3G or superior ■ When the tables get changed just synchronize the ones that changed ■ Sync only the data that is needed to execute the functionality that is needed on the Mobile App Real World Example
- 27. | Mobile Synchronization Patterns for Large Volumes of Data ● Create the SyncUnitHistory Record ■ Will be created on the CRUD – Create or Update Actions of the tables we want to sync ■ This table will have SyncUnitId and the Last Updated Date ■ Used to know the date of the last changed record of a table that we will need to sync How to know when and what needs to be Synced?
- 28. | Mobile Synchronization Patterns for Large Volumes of Data ● Get SyncUnits To Sync: ■ Action to check which SyncUnits need to be refreshed on your mobile app ■ Receives the Device Date and returns the list of ids of the tables that need to be sync ■ The action is used in the mobile synchronization action How to know what tables we need to Sync ?
- 29. | Mobile Synchronization Patterns for Large Volumes of Data ● Create a Control of Sync in Local Storage: ■ Sync controller in the local storage ■ This will kept track of the tables that already have been synced and when it was done Client-Side LastSync Controller
- 30. | Mobile Synchronization Patterns for Large Volumes of Data Example - MaterialType table Table with large volume of data
- 31. | Mobile Synchronization Patterns for Large Volumes of Data ● Sync of Material Type : ■ Get the Material-Types that are present on the server ■ In the first time will get all ■ Next time is filter by date from, date to and only the active ones ■ LocaleId is passed because app is multilingual ■ As output we get the Server Material Types and the total records 1. Create Sync Material-Type action
- 32. | Mobile Synchronization Patterns for Large Volumes of Data ● Sync of Material Type client side : ■ Create our Material-Type Local Storage; ■ Last Sync table to get the interval of time the date will be get ■ Updates the controller for the sync unit Material-Type 2. Action on client side to fill Material-Type local table
- 33. | Mobile Synchronization Patterns for Large Volumes of Data ● OfflineDataSync Action : ■ Sync the Local Data into the Server ■ If no errors occur, we start the sync of the large volume of data tables ■ Sync the tables with less volume of data 3. Mobile synchronization process
- 34. | Mobile Synchronization Patterns for Large Volumes of Data ● Sync_LC_Check Action : ■ Check’s our connection because we need to have network ■ Get SyncUnits to update list to check which tables need to be sync ■ Update the sync controller Last Sync and run the action Sync_LC_Update 4. Trigger and know which tables we going to Sync
- 35. | Mobile Synchronization Patterns for Large Volumes of Data ● Sync_LC_Update Action : ■ This action will receive as input the SyncUnit and only update the respective table 5. Action that Syncs the large volume of data tables
- 36. | Mobile Synchronization Patterns for Large Volumes of Data Dealing with Conflict
- 37. | Mobile Synchronization Patterns for Large Volumes of Data ● Use Case: ■ When multiple users can change the same data and the mobile app can be used offline ■ You need a mechanism to determine which data is conflicting ■ You need a screen so the user can choose the correct data How to Create a Conflict Screen on Your Mobile App
- 38. | Mobile Synchronization Patterns for Large Volumes of Data ● Solution: ■ Create a Static Table with the Entities name that we want to make conflict detection and resolution ■ Then use that static entity and create one local entity that will keep the table name and a flag if exist conflict ■ When we try to sync or update the local record into the server we need to check if a conflict exists ■ If a conflict exists create an entry on the table of conflicts and add a record to the list How to Create a Conflict Screen on Your Mobile App
- 39. | Mobile Synchronization Patterns for Large Volumes of Data 1. Creating the list of Conflicts
- 40. | Mobile Synchronization Patterns for Large Volumes of Data ● Conflict Resolution: ■ After click on the list the Conflict Element’s will appear ■ Local & Server record are serialize & deserialize into a JSON ■ Screen less heavy and with better performance ■ Can be optimized to choose attribute by attribute instead of record 2. Conflict Screen on Mobile App to Compare Data
- 41. | Mobile Synchronization Patterns for Large Volumes of Data
- 42. || Mobile Synchronization Patterns for Large Volumes of Data Q & A
- 43. | Thank You! @ injoao.pereira@doilean.com / joão-pedro-martins- pereira-8a115b120
Editor's Notes
- My name is João Pereira Last year I was in your place So im so glad to be making this presentation
- The Mobile Challenge Let’s Recap OutSystems Best Practices When We Can’t Follow Best Practices . . . Solving the Challenge! Patterns in Action
- Mobile Devices are more powerful than ever Mobile apps are faster and more intuitive than ever Users expect your app to perform like all other apps they use How can we do it with OutSystems
- Design the local storage data model using only the attributes that you need, instead of all attributes of the corresponding server entity. Consider denormalizing the local data model to avoid complex queries like having multiple JOINS. Identify the best data synchronization patterns for your use case and define exactly when to run them.
- To explain better the OutSystems Best Practice I created a small DB and a concept of an App that got the objective of having a platform where clients find car dealers which have cars in stock; The car dealers can add cars to his stock The clients can search by car dealer or model they want to buy Process of denormalization Simplify the connection between tables and minimize the amount of information
- We can control the data fetched from the server using; Sync Unit –several flows of synchornization Local Sync Properties – Create a local entity to keep track of the last synchronization moment Only get the latest data from the server – Compare the LastSycnOn from that sync unit with the CreatedOn, UpdatedOn and DeletedOn attributes from the server DB to know which records to fetch
- This image represents the third best practice using the sync unit to define the moments we want to sync our DB between local and server As you can see we define 3 moments 1. Get New Stock – On the Menu of the application the user clicks on a buttons or slides and refresh the list of car with the more recent 2. Remove sold cars , when a Car dealer changes the status of the car to available to sold 3. When the Owner of car dealers add new Change the image to more general information ;
- TriggerOfflineDataSync – explanation Devemos antecipar qual a informação que o utilizador precisa antes da mesma ser utilizada para assim obter uma fluidez dentro da aplicação
- As you know each server action called inside cliente actions executes a request to the server so if we call several server actions inside cliente action that will leave to open and close multiple requests in the same action that is going to be less performance that use like the exemple above one server action inside a cliente actions. Is visible that in that server action know it’s no problem use more than one This need to be applied to normal client actions of the app as well to our synchronization actions that will get and sync all the information we need
- Evaluate network requirements and use cases - If the application needs to be online or offline To evaluate the data sync moments Adapt behavior according to network strength, including offline Check how long we got to do the sync actions The moments we must sync all the information of the app. ############################################################# Evaluate network requirements use cases Define Local Storage strategy Optimize data sync moments Evaluate network status in runtime Adapt behavior accordingly Let the user know what’s happening
- Find the Optimal Value of Records to BULK Create According to the Different Processing Capabilities of Each Device Talk about 2 diferente scenarios – 1 . Corporate Use , 2. Target is the world B – B B – E
- After I recap you some of the OutSystems Best Pratices I will show you that they don’t work for every situation and that sometimes we cannot use them
- Must have same functionalities in web and mobile; The Data model cannot suffer the changes we want to have a lightweight database on the mobile app; The Mobile App is not going to be built base on the specific use cases but on the Web App We need to adapt the best practices the best we can on this situation When you ever Put the boss with the tie photo
- Have JOIN’s that will influence the performance of the application; Must think ways to overcome this limitation in terms of Sync Moments and how to do it; Analyze with are the use-cases the application needs Try to minimize sync of tables that got a high volume of data . Find strategies to sync the information the minimum times possible In the Mobile App Screens minimize the data that have multiple joins . So if we need the data that have multiple joins try to use only that information on the mobile screen
- How did you solve it ? In this part of the presentation we are going to show how to overcome the difficulties when we can’t use the best practices and have a huge database to manage in the local storage and need to sync a huge flow of data between Web and Mobile
- In the case of not changes frequently; Take care of this tables like lookup; Just need be synchronize all in the first use of the application; When this tables get changes just synchronize the ones that real change and not all Sync only the data that is neded to execute all the functionalities that are really need on the mobile AppCase A - Infrequent Change Treat like static data Synchronize all in the first use of the application Sync only when the data changes Inside of the Mobile App only sync the tables & data that is really needed
- Like in the previous case in the first use we need to sync all the tables; When modified the data only sync the modified back to the server; Choose well what information we want to override choosing one of this scenarios (Last Write Wins, Server always Win or Conflict Mechanism Detection) If we don’t have a good network, we must have a mechanism that discard a sync that was not fully completed in order to not have inconsistent data in our application
- Create a Sync Unit History Keep track of tables that need sync Last date the data was changed SyncUnit Create a static entity with the tables we need to sync
- Patterns in Action – I’m going to present you the real cases scenarios we use some strategies like the sync unit as table selector and how we do the sync of a big volume database
- In the case of not changes frequently; Take care of this tables like lookup; Just need be synchronize all in the first use of the application; When this tables get changes just synchronize the ones that real change and not all Sync only the data that is needed to execute all the functionalities that are really need on the mobile App The data doesn’t change a lot
- Inside the Core of the tables we want to sync was created: Static table called Sync Unit that contain all the tables we want to sync to the mobile app (SyncUnit.Label = MaterialType) A new table called SyncUnitHistory that will have the static Id from the previous table and the Last Updated Date The records of the SycnUnitHistory are going to be created on the actions of Created or Updated a entity we want to sync to mobile
- Also in the Core module was created a action to see what are the sync units that needs to sync data This action receives the Device Date and returns the list of id’s of the tables that needs to be sync Is going to be used on the mobile synchronization action
- As we done in the previous step, we create a control sync server table We need to create a LastSync table at will got the SyncUnit Id and the date from and date to of the Last Sync made on the device
- Client actions and server actions created to do the sync: Was created a new module in the Mobile App to manage the Local Storage and Sync actions to use in the APP All Local Tables are in this module Going to present the example to one table Material Type that got a dependency that is Material Type with a List of values, that is called Material Type LOV Dependency
- This action will get the Material Types that are present on the server In the first time will get all but on the next time is filter by date from, date to and only the active ones Because the application is multilingual, we need to pass the LocaleId we want to get the data As output we got the Server Material Types and the total records just to control how many data was sync at that time
- This an action will create our Material Type that is going to be at the Local Storage Used the Last Sync table to get the interval of time we are going to get the data In the end updates the controller for the sync unit Material Type
- This action will first check our connection because we need to make sure we got a good connection to sync large volumes of data and we going to use server-side actions The next step is using our Get SyncUnits to update list to check which tables need to be sync After get the list we update the sync controller Last Sync and run the action Sync_LC_Update
- This action will receive as input the SyncUnit and only update that respective table Then will run the Sync of Material Type on this case that we talk on the previous slide
- When multiple users can change the same data and the mobile app can be used offline How to create the data of conflict ? Where we choose the correct data ?
- Create a Static Table with the Entities name that we want to make conflict detection and resolution Then use that static entity and create one local that will keep the table name and a flag if exist conflict When we try to sync the local record into the server we need to check if a conflict exists If a conflict exists create an entry on the table of conflicts and add a record to a list
- Situation when we can check the conflict: When we try to save the record if we got internet connection, we can compare with the information that are present in the server at the moment When we try to sync the local data and put them into the server will check if that records was change In any of this cases is created a list with the table name and the number of conflicts that exist at that moment
- When we click in element of the previous list will appear a detail screen so the user can choose the server data or local data This data will get the Local record and the Server record and serialize & deserialize then into a JSON That process will make the screen less heavy and with better performance Can be optimized to choose attribute by attribute instead of record ##################################################################################################### I hope you enjoyed this presentation and next time you face this kind of challenge you remember what we talk about today !
- I hope you enjoyed this presentation and next time you face this kind of challenge you remember what we talk about today !