Google App Engine Developer - Day3

1,093 views
936 views

Published on

GAE Java - Day3
About memcache, task queue concept and operation

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,093
On SlideShare
0
From Embeds
0
Number of Embeds
8
Actions
Shares
0
Downloads
15
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Google App Engine Developer - Day3

  1. 1. GAE Developer - Day3 Simon @ MiCloud 2014Q1
  2. 2. Last Class ● Datastore basic ● Datastore operation ● Restrictions ● Transaction
  3. 3. Create an Entity with Key Entity job = new Entity("User", "simonsu@mitac.com.tw"); job.setProperty("name", "Simon Su"); job.setProperty("start", "20140103"); job.setProperty("create", new Date()); try { Key k = ds.put(job); out.println("Done..." + k.getId()); } catch (Exception e) { e.printStackTrace(); }
  4. 4. Create an Entity with Ancestor Key userKey = KeyFactory.createKey("User", "simonsu@mitac.com.tw"); Entity job = new Entity("Jobs", userKey); job.setProperty("name", "engineer"); job.setProperty("start", new Date()); try { Key k = ds.put(job); out.println("Done..." + k.getId()); } catch (Exception e) { e.printStackTrace(); }
  5. 5. Get data with Key DatastoreService ds = DatastoreServiceFactory.getDatastoreService(); Key key = KeyFactory.createKey("User", "simonsu@mitac.com.tw"); try { Entity ent = ds.get(key); out.print(ent); Map m = ent.getProperties(); out.println(m.toString()); } catch (EntityNotFoundException e) { e.printStackTrace(); }
  6. 6. Get data with Ancestor Query q = new Query("Jobs"); Key ancestor = KeyFactory.createKey("User", "simonsu@mitac.com.tw"); q.setAncestor(ancestor); PreparedQuery results = ds.prepare(q); Iterator iter = results.asIterable().iterator(); while(iter.hasNext()) { Entity ent = (Entity) iter.next(); out.println(ent); }
  7. 7. Today ● Memcache basic ● Task Queue concept ● Push Queue ● Pull Queue
  8. 8. Memcache
  9. 9. Why memcache ● Improve Application Performance ● Reduce Application Cost ● Caching for Read heavy operations ● Caching In Front of Datastore ● Semi-durable Shared state Across App Instances
  10. 10. Memcache - operation MemcacheService cache = MemcacheServiceFactory.getMemcacheService(); cache.setErrorHandler( ErrorHandlers.getConsistentLogAndContinue(Level.INFO)); // read from cache value = (byte[]) cache.get(key); // save to cache if (value == null) { // ........ cache.put(key, value); }
  11. 11. Async memcache - get // Using the asynchronous cache AsyncMemcacheService asyncCache = MemcacheServiceFactory.getAsyncMemcacheService(); asyncCache.setErrorHandler( ErrorHandlers.getConsistentLogAndContinue(Level.INFO)); // read from cache Future<Object> futureValue = asyncCache.get(key); // ... do other work in parallel to cache retrieval value = (byte[]) futureValue.get();
  12. 12. Async memcache - put if (value == null) { // get value from other source // ........ // asynchronously populate the cache // Returns a Future<Void> which can be used to block until completion asyncCache.put(key, value); }
  13. 13. Using jcache - init import java.util.Collections; import net.sf.jsr107cache.*; Cache cache; try { CacheFactory cacheFactory = CacheManager.getInstance().getCacheFactory(); cache = cacheFactory.createCache(Collections.emptyMap()); } catch (CacheException e) { // ... }
  14. 14. Using jcache - get & put String key; // ... byte[] value; // ... // Put the value into the cache. cache.put(key, value); // Get the value from the cache. value = (byte[]) cache.get(key);
  15. 15. Testing with ab benchmark ab -c 50 -n 50 http://[app-address]/[servlet_path] or ab -c 50 -t 50 http://[app-address]/[servlet_path]
  16. 16. Other Operation ● getAll(), putAll(), deleteAll() A single read or write operation for multiple memcache entries ● increment(key, delta), incrementAll(...) Provide atomic increment of numeric value(s) ● getIdentifiable(), putIfUntouched() A mechanism to update a value consistently by concurrent requests
  17. 17. Memcache Caveat ● Memcache Is Volatile ● Performance suggest: Batch size < 32 MB ● Dedicated memcache (pay): 10K OPS/s/GB, range 1-20 GB
  18. 18. Task Queue
  19. 19. Why Task Queue ● Frontend restriction of 60 second limit ● Need longer-running tasks ● Asynchronously outside front end request ● Best practice: ○ Email ○ Write datastore ○ Web crawler ○ Backend jobs
  20. 20. Task Queue Types ● By name ○ Default Queue (Push Queue) ○ Named Queue ● By type ○ Push queues ■ tasks execute automatically ■ only for App Engine app ○ Pull queues ■ tasks wait to be leased ■ available to workers outside the app ■ tasks can be batched
  21. 21. queue.xml <queue-entries> <queue> <name>Queue-Name</name> <rate>1/s</rate> ... </queue> <queue> <name>Queue-Name2</name> <mode>pull</mode> ... </queue> </queue-entries> Push Queue Pull Queue
  22. 22. Task Queue Manage Console
  23. 23. Task Queue - Default Push Queue Queue queue = QueueFactory.getDefaultQueue(); //named queue: //QueueFactory.getQueue("Qname"); TaskOptions task = TaskOptions.Builder .withUrl("/path-to-my-worker") .param(key, value); queue.add(task); Reachable Servlet
  24. 24. Detail Configure ● Task queues use token bucket algorithm [ref]
  25. 25. ● rate - usual rate ● bucket-size - cap on peak demand ● max concurrent requests ● Retry options ○ task-retry-limit - min retries ○ task-age-limit - min elapsed time to keep retrying (Both task-retry-limit and task-age-limit are reached, task is deleted) ○ min-backoff-seconds -- min delay between retries ○ max-backoff-seconds -- max delay between retries ○ max-doublings -- max times to double delay ● more parameters see DOC queue.xml parameters
  26. 26. Task Queue - With Transaction DatastoreService ds = DatastoreServiceFactory.getDatastoreService(); Queue queue = QueueFactory.getDefaultQueue(); try { Transaction txn = ds.beginTransaction(); // … other operations queue.add(TaskOptions.Builder.withUrl("/path-to-my-worker")); // … other operations txn.commit(); } catch (DatastoreFailureException e) { // … exception handle }
  27. 27. Pull Queue - Create Queue Queue queue = QueueFactory.getQueue("queue_name"); TaskOptions task = TaskOptions.Builder .withMethod(TaskOptions.Method.PULL) .payload(payload); queue.add(task);
  28. 28. Pull Queue - Lease Queue queue = QueueFactory.getQueue("queue_name"); tasks = q.leaseTasks( duration, TimeUnit.SECONDS, how-many);
  29. 29. Pull Queue - Delete Task //delete named task queue.deleteTask("task_ name"); //purge all tasks from queue QueueFactory.getQueue("queue_name").purge();
  30. 30. Task Queue ACL - queue.xml <queue> <name>pull-queue1</name> <mode>pull</mode> <acl> <user-email>me@gmail.com</user-email> <writer-email>you@gmail.com</writer-email> <writer-email>she@gmail.com</writer-email> </acl> </queue>
  31. 31. Cron
  32. 32. Cron <cronentries> <cron> <url>/recache</url> <description> Put your description here </description> <schedule>every 5 minutes</schedule> </cron> </cronentries>
  33. 33. ● every 12 hours ● every 5 minutes from 10:00 to 14:00 ● 2nd,third mon,wed,thu of march 17:00 ● every monday 09:00 ● 1st monday of sep,oct,nov 17:00 ● every day 00:00 ● every N (hours|mins|minutes) ["from" (time) "to" (time)] ● every 2 hours synchronized Cron Schedule Format
  34. 34. Cron Manage Console
  35. 35. Restrictions ● 20 cron jobs for free apps ● 100 for paid apps
  36. 36. Task Queue / Cron Reference ● Task Queues Developer Guide (Java) (Python) ● Task Queue Configuration (Java) (Python) ● Task Queue API Ref (JavaDoc) (Python) ● Task Queue REST API Ref ● Scheduled Tasks with Cron (Java) (Python) ● Task queue quota and limits ● Get started with Task API article (Python) ● Google I/O 2011: Putting Task Queues to Work (video)

×