Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Data Loading Made Easy with Mike Nakhimovich DroidCon Italy 2017

684 views

Published on

https://github.com/NYTimes/Store

Published in: Technology

Data Loading Made Easy with Mike Nakhimovich DroidCon Italy 2017

  1. 1. StoreDataLoadingMadeEasy-ish Mike Nakhimovich - NY Times
  2. 2. We ❤Making Apps
  3. 3. We ❤Open Source
  4. 4. Why?Open Source Makes Life Easier
  5. 5. Networking is easier: HTTPURLCONNECTION Volley Retrofit/ Okhttp
  6. 6. Storage is Easier SharedPreferences Firebase Realm SqlBrite/SqlDelight
  7. 7. Parsing is Easier JSONObject Jackson Gson Moshi
  8. 8. Fetching, Persisting & Parsing datahas become easy
  9. 9. What's NOT Easy? DATALOADINGEveryone does itdifferently
  10. 10. Whatis DataLoading? TheActofgetting datafroman external systemtoauser'sscreen
  11. 11. Loading is complicated Rotation Handling isa special snowflake
  12. 12. NewYorkTimes builtStoreto simplifydataloading github.com/NYTimes/store
  13. 13. OUR GOALS
  14. 14. Datashould survive configuration changes agnostic ofwhere itcomes from or howitis needed
  15. 15. Activitiesand presenters should stopretaining MBs ofdata
  16. 16. Offline as configuration Cachingas standard, notan exception
  17. 17. API should be simple enough for an internto use,yet robust enough for everydataload.
  18. 18. Howdowework with DataatNY Times?
  19. 19. 80% case: Need data, don'tcare iffresh or cached
  20. 20. Need fresh data background updates & pull to refresh
  21. 21. Requests needto be async & reactive
  22. 22. Datais dependenton each other, DataLoading should betoo
  23. 23. Performance is importantNewcalls should hook into in flight responses
  24. 24. Parsing should be done efficientlyand minimallyParse onceand cachethe resultfor future calls
  25. 25. Let's Use Repository PatternBy creating reactiveand persistent DataStores
  26. 26. Repository Pattern Separatethe logicthatretrievesthe dataand maps ittothe [view] model fromthe [view] logicthatacts onthe model. The repositorymediates betweenthe datalayerandthe [view] layer ofthe application.
  27. 27. WhyRepository? Maximizetheamountof codethatcan betested withautomation by isolatingthe datalayer
  28. 28. Why Repository? Makes iteasierto have multiple devsworking on same feature
  29. 29. Why Repository? Datasource from many locationswillbe centrally managedwith consistent access rulesand logic
  30. 30. Our Implementation https://github.com/NYTimes/Store
  31. 31. WhatisaStore? Aclassthatmanagesthe fetching, parsing,and storage ofaspecific data model
  32. 32. Tell a Store: Whatto fetch
  33. 33. Tell a Store: Whatto fetch Whereto cache
  34. 34. Tell a Store: Whatto fetch Whereto cache Howto parse
  35. 35. Tell a Store: Whatto fetch Whereto cache Howto parse The Store handles flow
  36. 36. Storesare Observable Observable<T> get( V key); Observable<T> fetch(V key) Observable<T> stream() void clear(V key)
  37. 37. Let's see howstores helped usachieve our goalsthrougha PizzaExample
  38. 38. Getis for Handling our 80% Use Case public final class PizzaActivity { Store<Pizza, String> pizzaStore; void onCreate() { store.get("cheese") .subscribe(pizza ->.show(pizza)); } }
  39. 39. Howdoes itflow?
  40. 40. On Configuration Change You onlyneedto Save/Restoreyour !to get your cached " public final class PizzaActivity { void onRecreate(String topping) { store.get(topping) .subscribe(pizza ->.show(pizza)); } } Please do notserializeaPizza Itwould beverymessy
  41. 41. Whatwe gain?:Fragments/Presenters don'tneedto be retained NoTransactionTooLargeExceptions Onlykeys needto be Parcelable
  42. 42. Efficiencyis Important: for(i=0;i<20;i++){ store.get(topping) .subscribe(pizza -> getView() .setData(pizza)); } ManyconcurrentGetrequestswillstillonly hitnetwork once
  43. 43. Fetching NewDatapublic class PizzaPresenter { Store<Pizza, String> store; void onPTR() { store.fetch("cheese") .subscribe(pizza -> getView().setData(pizza)); } }
  44. 44. Howdoes Store routethe request? Fetchalsothrottles multiple requestsand broadcastresult
  45. 45. Stream listens for events public class PizzaBar { Store<Pizza, String> store; void showPizzaDone() { store.stream() .subscribe(pizza -> getView().showSnackBar(pizza)); } }
  46. 46. How do We build a Store? By implementing interfaces
  47. 47. Interfaces Fetcher<Raw, Key>{ Observable<Raw> fetch(Key key); } Persister<Raw, Key>{} Observable<Raw> read(Key key); Observable<Boolean> write(Key key, Raw raw); } Parser<Raw, Parsed> extends Func1<Raw, Parsed>{ Parsed call(Raw raw); }
  48. 48. Fetcher defines how a Store will get new data Fetcher<Pizza,String> pizzaFetcher = new Fetcher<Pizza, String>() { public Observable<Pizza> fetch(String topping) { return pizzaSource.fetch(topping); };
  49. 49. Becoming Observable Fetcher<Pizza,String> pizzaFetcher = topping -> Observable.fromCallable(() -> client.fetch(topping));
  50. 50. Parsers helpwith Fetchersthatdon't returnViewModels
  51. 51. Some ParsersTransform Parser<Pizza, PizzaBox> boxParser = new Parser<>() { public PizzaBox call(Pizza pizza) { return new PizzaBox(pizza); } };
  52. 52. Others read Streams Parser<BufferedSource, Pizza> parser = source -> { InputStreamReader reader = new InputStreamReader(source.inputStream()); return gson.fromJson(reader, Pizza.class); }
  53. 53. MiddleWare is forthe commonJSON cases Parser<BufferedSource, Pizza> parser = GsonParserFactory.createSourceParser(gson,Pizza.class) Parser<Reader, Pizza> parser = GsonParserFactory.createReaderParser(gson,Pizza.class) Parser<String, Pizza> parser = GsonParserFactory.createStringParser(gson,Pizza.class) 'com.nytimes.android:middleware:CurrentVersion' 'com.nytimes.android:middleware:jackson:CurrentVersion' 'com.nytimes.android:middleware-moshi:CurrentVersion'
  54. 54. Data FlowwithaParser
  55. 55. Adding Offline with Persisters
  56. 56. File System Record Persister FileSystemRecordPersister.create( fileSystem,key -> "pizza"+key, 1, TimeUnit.DAYS);
  57. 57. File System is KISS storage interface FileSystem { BufferedSource read(String var) throws FileNotFoundException; void write(String path, BufferedSource source) throws IOException; void delete(String path) throws IOException; } compile 'com.nytimes.android:filesystem:CurrentVersion'
  58. 58. Don'tlike our persisters? No Problem! You can implement your own
  59. 59. Persister interfaces Persister<Raw, Key> { Observable<Raw> read(Key key); Observable<Boolean> write(Key key, Raw raw); } Clearable<Key> { void clear(Key key); } RecordProvider<Key> { RecordState getRecordState( Key key);
  60. 60. DataFlowwithaPersister
  61. 61. We have our components! Let's buildand openastore
  62. 62. MultiParser List<Parser> parsers=new ArrayList<>(); parsers.add(boxParser); parsers.add(GsonParserFactory.createSourceParser(gson, Pizza.class));
  63. 63. Store Builder List<Parser> parsers=new ArrayList<>(); parsers.add(boxParser); parsers.add(GsonParserFactory.createSourceParser(gson, Pizza.class)); StoreBuilder<String,BufferedSource,Pizza> parsedWithKey()
  64. 64. Add Our Parser List<Parser> parsers=new ArrayList<>(); parsers.add(boxParser); parsers.add(GsonParserFactory.createSourceParser(gson, Pizza.class)); StoreBuilder<String,BufferedSource,Pizza> parsedWithKey() .fetcher(topping -> pizzaSource.fetch(topping))
  65. 65. NowOur Persister List<Parser> parsers=new ArrayList<>(); parsers.add(boxParser); parsers.add(GsonParserFactory.createSourceParser(gson, Pizza.class)); StoreBuilder<String,BufferedSource,Pizza> parsedWithKey() .fetcher(topping -> pizzaSource.fetch(topping)) .persister( FileSystemRecordPersister.create( fileSystem, key -> "prefix"+key, 1, TimeUnit.DAYS))
  66. 66. And Our Parsers List<Parser> parsers=new ArrayList<>(); parsers.add(boxParser); parsers.add(GsonParserFactory.createSourceParser(gson, Pizza.class)); StoreBuilder<String,BufferedSource,Pizza> parsedWithKey() .fetcher(topping -> pizzaSource.fetch(topping)) .persister( FileSystemRecordPersister.create( fileSystem, key -> "prefix"+key, 1, TimeUnit.DAYS)) .parsers(parsers)
  67. 67. Timeto OpenaStore List<Parser> parsers=new ArrayList<>(); parsers.add(boxParser); parsers.add(GsonParserFactory.createSourceParser(gson, Pizza.class)); Store<Pizza, String> pizzaStore = StoreBuilder<String,BufferedSource,Pizza> parsedWithKey() .fetcher(topping -> pizzaSource.fetch(topping)) .persister( FileSystemRecordPersister.create( fileSystem, key -> "prefix"+key, 1, TimeUnit.DAYS)) .parsers(parsers) .open();
  68. 68. Configuring caches
  69. 69. Configuring MemoryPolicy StoreBuilder .<String, BufferedSource, Pizza>parsedWithKey() .fetcher(topping -> pizzaSource.fetch(topping)) .memoryPolicy(MemoryPolicy .builder() .setMemorySize(10) .setExpireAfter(24) .setExpireAfterTimeUnit(HOURS) .build()) .open()
  70. 70. Refresh On Stale - BackFillingthe Cache Store<Pizza, String> pizzaStore = StoreBuilder .<String, BufferedSource, Pizza>parsedWithKey() .fetcher(topping -> pizzaSource.fetch(topping)) .parsers(parsers) .persister(persister) .refreshOnStale() .open();
  71. 71. Network Before Stales Policy Store<Pizza, String> pizzaStore = StoreBuilder .<String, BufferedSource, Pizza>parsedWithKey() .fetcher(topping -> pizzaSource.fetch(topping)) .parsers(parsers) .persister(persister) .networkBeforeStale() .open();
  72. 72. Stores inthewild
  73. 73. Intern Project: BestSellers List public interface Api { @GET Observable<BufferedSource> getBooks(@Path("category") String category);
  74. 74. @Provides @Singleton Store<Books, BarCode> provideBooks(FileSystem fileSystem, Gson gson, Api api) { return StoreBuilder.<BarCode, BufferedSource, Books> parsedWithKey() .fetcher(category -> api.getBooks(category)) .persister(SourcePersisterFactory.create(fileSystem)) .parser(GsonParserFactory.createSourceParser(gson, Books.class)) .open(); }
  75. 75. public class BookActivity{ ... onCreate(...){ bookStore .get(category) .subscribe(); }
  76. 76. Howdo books getupdated? public class BackgroundUpdater{ ... bookStore .fresh(category) .subscribe(); }
  77. 77. DataAvailablewhen screen needs it UI calls get, background services call fresh
  78. 78. DependentCalls
  79. 79. Simple case Using RxJavaOperators MapStore resulttoanother Store Result public Observable<SectionFront> getSectionFront(@NonNull final String name) { return feedStore.get() .map(latestFeed -> latestFeed.getSectionOrBlog(name)) .flatMap(section -> sfStore.get(SectionFrontId.of(section))); }
  80. 80. More Complicated Example Video Store Returns singlevideos, PlaylistStore returns playlist, Howdowe combine?
  81. 81. Relationships by overriding Get/Fetch
  82. 82. Step1: CreateVideo Store Store<Video, Long> provideVideoStore(VideoFetcher fetcher, Gson gson) { return StoreBuilder.<Long, BufferedSource, Video>parsedWithKey() .fetcher(fetcher) .parser(GsonParserFactory .createSourceParser(gson, Video.class)) .open(); }
  83. 83. Step 2: Create PlaylistStore public class VideoPlaylistStore extends RealStore<Playlist, Long> { final Store<CherryVideoEntity, Long> videoStore; public VideoPlaylistStore(@NonNull PlaylistFetcher fetcher, @NonNull Store<CherryVideoEntity, Long> videoStore, @NonNull Gson gson) { super(fetcher, NoopPersister.create(), GsonParserFactory.createSourceParser(gson, Playlist.class)); this.videoStore = videoStore; }
  84. 84. Step 3: Override store.get() public class VideoPlaylistStore extends RealStore<Playlist, Long> { final Store<CherryVideoEntity, Long> videoStore; @Override public Observable<Playlist> get(Long playlistId) { return super.get(playlistId) .flatMap(playlist -> from(playlist.videos()) .concatMap(video -> videoStore.get(video.id())) .toList() .map(videos -> builder().from(playlist) .videos(videos) .build())); }
  85. 85. Listening for changes
  86. 86. Step1: Subscribeto Store, Filterwhatyou need sectionFrontStore.subscribeUpdates() .observeOn(scheduler) .subscribeOn(Schedulers.io()) .filter(this::sectionIsInView) .subscribe(this::handleSectionChange); Step2: Kick offafresh requesttothatstore public Observable<SectionFront> refreshAll(final String sectionName) { return feedStore.get() .flatMapIterable(this::updatedSections) .map(section->sectionFrontStore.fetch(section)) latestFeed -> fetch(latestFeed.changedSections)); }
  87. 87. Bonus: Howwe made Store
  88. 88. RxJavaforAPIand FunctionalNiceties public Observable<Parsed> get(@Nonnull final Key key) { return Observable.concat( cache(key), fetch(key) ).take(1); } public Observable<Parsed> getRefreshing(@Nonnull final Key key) { return get(key) .compose(repeatOnClear(refreshSubject, key)); }
  89. 89. GuavaCaches For MemoryCache memCache = CacheBuilder .newBuilder() .maximumSize(maxSize()) .expireAfterWrite(expireAfter(),timeUnit()) .build();
  90. 90. In FlightThrottlerToo inflighter.get(key, new Callable<Observable<Parsed>>() { public Observable<Parsed> call() { return response(key); } }) compile 'com.nytimes.android:cache:CurrentVersion' Why GuavaCaches? Theywillblockacrossthreads saving precious MBs ofdownloads when making same requests in parallel
  91. 91. FileSystem: Builtusing OKIOtoallowstreaming from Networkto disk & diskto parser On Fresh InstallNYTimesappsuccessfullydownloadsand caches dozens ofmbs ofdata
  92. 92. Would love contributions & feedback!thanks for listening

×