Rethinking Syncing at AltConf 2019

RETHINKING
SYNCING
AltConf 2019

Syncing Variables
• Amount of Data
• When does user want to see or use data
• How much / how often does data change
• Offline access needed

REST API
https://sync.server.com/green/GET
GET
PUT
POST
DELETE
https://sync.server.com/green/123

To-Device Syncing Approaches
• The Everything Approach
• The Net Change Approach
• The Kinda Sorta Not Really Net Change Approach
• The As Needed Approach
• and more…

The Net Change Approach
https://sync.server.com/allmystuff?since=20180211

• Updates data from known point in time
• More efficient implementation, more complex logic required
• Uses less network bandwidth
• Requires handling of adds, changes, deletes on server and mobile
• Requires one source of truth for “since” parameter
https://sync.server.com/allmystuff?since=20180211

• Process:
• Make API calls - receive lists of “red,” “green,” and “blue” objects with actions identified
• Iterate each list - perform add / change / delete as needed to local storage
• Update UI

• Prepare API Call:
static func urlRequest(for url:URL, parameters:[String:String]?) -> URLRequest {
var requestUrl = url
if let passedParameters = parameters {
var queryItems:[URLQueryItem] = []
for (parameterName, parameter) in passedParameters {
queryItems.append(URLQueryItem(name: parameterName, value: parameter))
}
var requestComponents = URLComponents(url: requestUrl, resolvingAgainstBaseURL: false)
if let currentQueryItems = requestComponents?.queryItems {
queryItems.append(contentsOf: currentQueryItems)
}
requestComponents?.queryItems = queryItems
if let queryRequestUrl = requestComponents?.url {
requestUrl = queryRequestUrl
}
}
let request = URLRequest(url: requestUrl)
return request
}

• Make API Call:
static func fetchObjects<T:RemoteObject>(of type:T.Type,
completionHandler: @escaping (_ results:[T]) -> Void,
errorHandler: @escaping (_ error:Error) -> Void) -> Void {
let fetchURL = type.urlForList(from: AppEnvironment.APIBaseURL)
let fetchTask = APIManager.authSession?.dataTask(with: type.urlRequest(for: fetchURL, parameters: nil),
completionHandler: { (data, response, error) in
if let actualError = error {
DispatchQueue.main.async {
errorHandler(actualError)
}
return
}
if let httpResponse = response as? HTTPURLResponse, httpResponse.statusCode != 200 {
errorHandler(NetworkingError.unexpectedHTTPStatusCode)
}
return
}
…
})
fetchTask?.resume()
}

• Ingest Data:
do {
guard let payloadData = data,
let payload = try JSONSerialization.jsonObject(with: payloadData, options: []) as? [String:Any],
let resultsPayload = payload["results"] as? [[String:Any]] else {
errorHandler(NetworkingError.invalidJSONPayload)
}
return
}
let results = T.ingest(json: resultsPayload)
completionHandler(results)
}
} catch {
}
return
}

do {
}
return
}
}
} catch {
}
return
}
}
return
}
}
return
}
…
})
fetchTask?.resume()
}
static func urlRequest(for url:URL, parameters:[String:String]?) -> URLRequest {
var requestUrl = url
if let passedParameters = parameters {
var queryItems:[URLQueryItem] = []
for (parameterName, parameter) in passedParameters {
queryItems.append(URLQueryItem(name: parameterName, value: parameter))
}
var requestComponents = URLComponents(url: requestUrl, resolvingAgainstBaseURL: false)
if let currentQueryItems = requestComponents?.queryItems {
queryItems.append(contentsOf: currentQueryItems)
}
requestComponents?.queryItems = queryItems
if let queryRequestUrl = requestComponents?.url {
requestUrl = queryRequestUrl
}
}
let request = URLRequest(url: requestUrl)
return request
}
do {
}
return
}
}
} catch {
}
return
}
}
return
}
}
return
}
…
})
fetchTask?.resume()
}
do {
}
return
}
}
} catch {
}
return
}
let fetchTask = APIManager.authSession?.dataTask(with: type.urlRequest(for: fetchURL, param
}
return
}
}
return
}
…
})
fetchTask?.resume()
}

From-Device Syncing Approaches
• The Online Only Approach
• The Persist-The-Network-Call Approach
• The Offline First Queue Approach
• and more…

Complications to REST API Syncing
• Offline mode
• Queueing uploads
• Dependencies
• Must fetch one entity before fetching another
• Must create one entity before creating another
• Model changes
• Sync API changes with releases
• Support older versions

We didn’t
need to write
syncing
code…
any more?!

Eventually Consistent DB
• Server Database (can be NoSQL)
• Sync Server
• Mobile DB

Eventually Consistent DBs
• Examples:
• Couchbase
• Realm
• Google Firebase
• AWS AppSync
• IBM Cloudant
• others?

The idea:
Couchbase Server
Sync Server
Legacy DB
Services
Couchbase Lite

The idea:
Couchbase Server
Couchbase Lite
Sync Server
Legacy DB
Services

What just happened:
Couchbase Server
Sync Server
Legacy DB
Services
Couchbase Lite

How do we implement?
• Model Objects
• Data Controller
• Fetching Data for Display
• Handling Refresh on Change
• Adding Data
• Deleting Data

Model Objects
struct BlueBar:Codable {
var userId: String
var blueIntDetail: Int
var type: String
var addedBy: String
}
struct RedBar:Codable {
var userId: String
var redIntDetail: Int
var type: String
var addedBy: String
}
struct GreenBar:Codable {
var userId: String
var greenIntDetail: Int
var type: String
var addedBy: String
}

Data Controller
init() {
do {
database = try Database(name: "rethinking")
} catch {
print("Unable to initialize Couchbase database 'rethinking'")
}
}

Data Controller (cont)
func setUpSyncing(at url:URL) {
guard let database = database else {
print("No Couchbase database, cannot start syncing")
return
}
let endpoint = URLEndpoint(url: url)
let config = ReplicatorConfiguration(database: database, target: endpoint)
config.replicatorType = .pushAndPull
config.continuous = true
config.authenticator = BasicAuthenticator(username: "demo_user", password: "password")
config.channels = ["demo_user"]
self.replicator = Replicator(config: config)
//Set up replication change listeners
//Database.setLogLevel(.verbose, domain: .replicator)
//Database.setLogLevel(.verbose, domain: .network)
self.replicator?.start()
}

Fetching Data for Display
func fetchGreenBars() throws -> [GreenBar] {
let documentType = "greenbars"
var fetchedGreenBars: [GreenBar] = []
guard let database = self.database else {
throw DataControllerModelError.noDatabaseError
}
let greenBarQuery = QueryBuilder.select(SelectResult.all())
.from(DataSource.database(database))
.where(Expression.property("type").equalTo(Expression.string(documentType)))
if greenBarQueryListenerToken == nil {
greenBarQueryListenerToken = greenBarQuery.addChangeListener({ [weak self] (change) in
self?.greenBarRefreshHandler?()
})
}
do {
for result in try greenBarQuery.execute() {
let resultDict = result.toDictionary()
if let dataDict = resultDict["rethinking"] {
let data = try JSONSerialization.data(withJSONObject: dataDict, options: .prettyPrinted)
let decodedObject = try self.jsonDecoder.decode(GreenBar.self, from: data)
fetchedGreenBars.append(decodedObject)
}
}
} catch {
print(error)
}
return fetchedGreenBars
}

Handling Refresh on Change
dataController?.greenBarRefreshHandler = { [weak self] in
do {
self?.greenBars = try self?.dataController?.fetchGreenBars() ?? []
} catch {
print("Encounted error attempting to fetch bar records")
}
self?.reloadSection(for: .green)
}

Adding Data
func add(greenBar: GreenBar) throws {
}
let doc = MutableDocument(id: greenBar.userId)
let data = try jsonEncoder.encode(greenBar)
guard let json = try JSONSerialization.jsonObject(with: data) as? [String:Any] else {
throw DataControllerModelError.cannotSerializeJSONError
}
doc.setData(json)
try database.saveDocument(doc)
print("Added green bar: (greenBar.greenIntDetail)")
}

Deleting Data
func delete(greenBar: GreenBar) throws {
}
guard let documentToDelete = database.document(withID: greenBar.userId) else {
throw DataControllerModelError.notFoundError
}
try database.deleteDocument(documentToDelete)
}

struct BlueBar:Codable {
var userId: String
var blueIntDetail: Int
var type: String
var addedBy: String
}
struct RedBar:Codable {
var userId: String
var redIntDetail: Int
var type: String
var addedBy: String
}
struct GreenBar:Codable {
var userId: String
var greenIntDetail: Int
var type: String
var addedBy: String
}
init() {
do {
database = try Database(name: "rethinking")
} catch {
print("Unable to initialize Couchbase database 'rethinking'")
}
}
func fetchGreenBars() throws -> [GreenBar] {
let documentType = "greenbars"
var fetchedGreenBars: [GreenBar] = []
}
let greenBarQuery = QueryBuilder.select(SelectResult.all())
.from(DataSource.database(database))
.where(Expression.property("type").equalTo(Expression.string(documentType)))
if greenBarQueryListenerToken == nil {
greenBarQueryListenerToken = greenBarQuery.addChangeListener({ [weak self] (change) in
self?.greenBarRefreshHandler?()
})
}
do {
for result in try greenBarQuery.execute() {
let resultDict = result.toDictionary()
if let dataDict = resultDict["rethinking"] {
let data = try JSONSerialization.data(withJSONObject: dataDict, options: .prettyPrinted)
let decodedObject = try self.jsonDecoder.decode(GreenBar.self, from: data)
fetchedGreenBars.append(decodedObject)
}
}
} catch {
print(error)
}
return fetchedGreenBars
}
dataController?.greenBarRefreshHandler = { [weak self] in
do {
self?.greenBars = try self?.dataController?.fetchGreenBars() ?? []
} catch {
print("Encounted error attempting to fetch bar records")
}
self?.reloadSection(for: .green)
}
func add(greenBar: GreenBar) throws {
}
let doc = MutableDocument(id: greenBar.userId)
let data = try jsonEncoder.encode(greenBar)
guard let json = try JSONSerialization.jsonObject(with: data) as? [String:Any] else {
throw DataControllerModelError.cannotSerializeJSONError
}
doc.setData(json)
try database.saveDocument(doc)
print("Added green bar: (greenBar.greenIntDetail)")
}
func delete(greenBar: GreenBar) throws {
}
guard let documentToDelete = database.document(withID: greenBar.userId) else {
throw DataControllerModelError.notFoundError
}
try database.deleteDocument(documentToDelete)
}

Eventually Consistent DB Advantages
• Significant simplification of syncing code
• Set up DB, set up syncing call. ~10-20 lines of code
• No API calls needed*
• Dependencies much easier to handle
• Can use nested objects
• Thread safety - can ingest model objects in background and
*safely* use them in main queue. Try doing that in Core Data!
* You might need to do authentication / login with an API call prior to setting up your sync

Eventually Consistent DB Advantages (con’t)
• Significant simplification of model & ingestion code
• No Core Data model needed
• No mogenerated ManagedObjectSubclasses
• Models can be classes or structs using Codable - very simple
to code
• Simplification of services
• Fewer, less complex endpoints

Eventually Consistent DB Disadvantages
• Lack of familiarity
• Example: client refreshed source DB for staging - crushed our
CB instance
• Architecture requires different thinking for common use cases
• Cannot control priority of syncing - “eventual consistency”
• No NSFetchedResultsController-type code (at least in
Couchbase Mobile)

Eventually Consistent DB Best Use Cases
• Handle slowly changing “master” data
• For example, organizing data or lists of choices
• User created data, especially offline
• * Not necessarily images…
• Not ideal for large amounts of quickly changing data

THANK YOU
martiancraft.com
AltConf 2019

Rethinking Syncing at AltConf 2019

More Related Content

What's hot

Similar to Rethinking Syncing at AltConf 2019

Rethinking Syncing at AltConf 2019