Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
GAE Developer - Day3
Simon @ MiCloud
2014Q1
Last Class
● Datastore basic
● Datastore operation
● Restrictions
● Transaction
Create an Entity with Key
Entity job = new Entity("User", "simonsu@mitac.com.tw");
job.setProperty("name", "Simon Su");
jo...
Create an Entity with Ancestor
Key userKey = KeyFactory.createKey("User", "simonsu@mitac.com.tw");
Entity job = new Entity...
Get data with Key
DatastoreService ds = DatastoreServiceFactory.getDatastoreService();
Key key = KeyFactory.createKey("Use...
Get data with Ancestor
Query q = new Query("Jobs");
Key ancestor = KeyFactory.createKey("User", "simonsu@mitac.com.tw");
q...
Today
● Memcache basic
● Task Queue concept
● Push Queue
● Pull Queue
Memcache
Why memcache
● Improve Application Performance
● Reduce Application Cost
● Caching for Read heavy operations
● Caching In ...
Memcache - operation
MemcacheService cache = MemcacheServiceFactory.getMemcacheService();
cache.setErrorHandler(
ErrorHand...
Async memcache - get
// Using the asynchronous cache
AsyncMemcacheService asyncCache =
MemcacheServiceFactory.getAsyncMemc...
Async memcache - put
if (value == null) {
// get value from other source
// ........
// asynchronously populate the cache
...
Using jcache - init
import java.util.Collections;
import net.sf.jsr107cache.*;
Cache cache;
try {
CacheFactory cacheFactor...
Using jcache - get & put
String key; // ...
byte[] value; // ...
// Put the value into the cache.
cache.put(key, value);
/...
Testing with ab benchmark
ab -c 50 -n 50 http://[app-address]/[servlet_path]
or
ab -c 50 -t 50 http://[app-address]/[servl...
Other Operation
● getAll(), putAll(), deleteAll()
A single read or write operation for multiple memcache
entries
● increme...
Memcache Caveat
● Memcache Is Volatile
● Performance suggest: Batch size < 32 MB
● Dedicated memcache (pay): 10K OPS/s/GB,...
Task Queue
Why Task Queue
● Frontend restriction of 60 second limit
● Need longer-running tasks
● Asynchronously outside front end re...
Task Queue Types
● By name
○ Default Queue (Push Queue)
○ Named Queue
● By type
○ Push queues
■ tasks execute automaticall...
queue.xml
<queue-entries>
<queue>
<name>Queue-Name</name>
<rate>1/s</rate>
...
</queue>
<queue>
<name>Queue-Name2</name>
<...
Task Queue Manage Console
Task Queue - Default Push Queue
Queue queue = QueueFactory.getDefaultQueue();
//named queue:
//QueueFactory.getQueue("Qnam...
Detail Configure
● Task queues use token bucket algorithm [ref]
● rate - usual rate
● bucket-size - cap on peak demand
● max concurrent requests
● Retry options
○ task-retry-limit - min ...
Task Queue - With Transaction
DatastoreService ds = DatastoreServiceFactory.getDatastoreService();
Queue queue = QueueFact...
Pull Queue - Create Queue
Queue queue =
QueueFactory.getQueue("queue_name");
TaskOptions task =
TaskOptions.Builder
.withM...
Pull Queue - Lease
Queue queue =
QueueFactory.getQueue("queue_name");
tasks = q.leaseTasks(
duration,
TimeUnit.SECONDS,
ho...
Pull Queue - Delete Task
//delete named task
queue.deleteTask("task_ name");
//purge all tasks from queue
QueueFactory.get...
Task Queue ACL - queue.xml
<queue>
<name>pull-queue1</name>
<mode>pull</mode>
<acl>
<user-email>me@gmail.com</user-email>
...
Cron
Cron
<cronentries>
<cron>
<url>/recache</url>
<description>
Put your description here
</description>
<schedule>every 5 min...
● every 12 hours
● every 5 minutes from 10:00 to 14:00
● 2nd,third mon,wed,thu of march 17:00
● every monday 09:00
● 1st m...
Cron Manage Console
Restrictions
● 20 cron jobs for free apps
● 100 for paid apps
Task Queue / Cron Reference
● Task Queues Developer Guide (Java) (Python)
● Task Queue Configuration (Java) (Python)
● Tas...
Upcoming SlideShare
Loading in …5
×

Google App Engine Developer - Day3

1,396 views

Published on

GAE Java - Day3
About memcache, task queue concept and operation

Published in: Technology
  • Be the first to comment

Google App Engine Developer - Day3

  1. 1. GAE Developer - Day3 Simon @ MiCloud 2014Q1
  2. 2. Last Class ● Datastore basic ● Datastore operation ● Restrictions ● Transaction
  3. 3. Create an Entity with Key Entity job = new Entity("User", "simonsu@mitac.com.tw"); job.setProperty("name", "Simon Su"); job.setProperty("start", "20140103"); job.setProperty("create", new Date()); try { Key k = ds.put(job); out.println("Done..." + k.getId()); } catch (Exception e) { e.printStackTrace(); }
  4. 4. Create an Entity with Ancestor Key userKey = KeyFactory.createKey("User", "simonsu@mitac.com.tw"); Entity job = new Entity("Jobs", userKey); job.setProperty("name", "engineer"); job.setProperty("start", new Date()); try { Key k = ds.put(job); out.println("Done..." + k.getId()); } catch (Exception e) { e.printStackTrace(); }
  5. 5. Get data with Key DatastoreService ds = DatastoreServiceFactory.getDatastoreService(); Key key = KeyFactory.createKey("User", "simonsu@mitac.com.tw"); try { Entity ent = ds.get(key); out.print(ent); Map m = ent.getProperties(); out.println(m.toString()); } catch (EntityNotFoundException e) { e.printStackTrace(); }
  6. 6. Get data with Ancestor Query q = new Query("Jobs"); Key ancestor = KeyFactory.createKey("User", "simonsu@mitac.com.tw"); q.setAncestor(ancestor); PreparedQuery results = ds.prepare(q); Iterator iter = results.asIterable().iterator(); while(iter.hasNext()) { Entity ent = (Entity) iter.next(); out.println(ent); }
  7. 7. Today ● Memcache basic ● Task Queue concept ● Push Queue ● Pull Queue
  8. 8. Memcache
  9. 9. Why memcache ● Improve Application Performance ● Reduce Application Cost ● Caching for Read heavy operations ● Caching In Front of Datastore ● Semi-durable Shared state Across App Instances
  10. 10. Memcache - operation MemcacheService cache = MemcacheServiceFactory.getMemcacheService(); cache.setErrorHandler( ErrorHandlers.getConsistentLogAndContinue(Level.INFO)); // read from cache value = (byte[]) cache.get(key); // save to cache if (value == null) { // ........ cache.put(key, value); }
  11. 11. Async memcache - get // Using the asynchronous cache AsyncMemcacheService asyncCache = MemcacheServiceFactory.getAsyncMemcacheService(); asyncCache.setErrorHandler( ErrorHandlers.getConsistentLogAndContinue(Level.INFO)); // read from cache Future<Object> futureValue = asyncCache.get(key); // ... do other work in parallel to cache retrieval value = (byte[]) futureValue.get();
  12. 12. Async memcache - put if (value == null) { // get value from other source // ........ // asynchronously populate the cache // Returns a Future<Void> which can be used to block until completion asyncCache.put(key, value); }
  13. 13. Using jcache - init import java.util.Collections; import net.sf.jsr107cache.*; Cache cache; try { CacheFactory cacheFactory = CacheManager.getInstance().getCacheFactory(); cache = cacheFactory.createCache(Collections.emptyMap()); } catch (CacheException e) { // ... }
  14. 14. Using jcache - get & put String key; // ... byte[] value; // ... // Put the value into the cache. cache.put(key, value); // Get the value from the cache. value = (byte[]) cache.get(key);
  15. 15. Testing with ab benchmark ab -c 50 -n 50 http://[app-address]/[servlet_path] or ab -c 50 -t 50 http://[app-address]/[servlet_path]
  16. 16. Other Operation ● getAll(), putAll(), deleteAll() A single read or write operation for multiple memcache entries ● increment(key, delta), incrementAll(...) Provide atomic increment of numeric value(s) ● getIdentifiable(), putIfUntouched() A mechanism to update a value consistently by concurrent requests
  17. 17. Memcache Caveat ● Memcache Is Volatile ● Performance suggest: Batch size < 32 MB ● Dedicated memcache (pay): 10K OPS/s/GB, range 1-20 GB
  18. 18. Task Queue
  19. 19. Why Task Queue ● Frontend restriction of 60 second limit ● Need longer-running tasks ● Asynchronously outside front end request ● Best practice: ○ Email ○ Write datastore ○ Web crawler ○ Backend jobs
  20. 20. Task Queue Types ● By name ○ Default Queue (Push Queue) ○ Named Queue ● By type ○ Push queues ■ tasks execute automatically ■ only for App Engine app ○ Pull queues ■ tasks wait to be leased ■ available to workers outside the app ■ tasks can be batched
  21. 21. queue.xml <queue-entries> <queue> <name>Queue-Name</name> <rate>1/s</rate> ... </queue> <queue> <name>Queue-Name2</name> <mode>pull</mode> ... </queue> </queue-entries> Push Queue Pull Queue
  22. 22. Task Queue Manage Console
  23. 23. Task Queue - Default Push Queue Queue queue = QueueFactory.getDefaultQueue(); //named queue: //QueueFactory.getQueue("Qname"); TaskOptions task = TaskOptions.Builder .withUrl("/path-to-my-worker") .param(key, value); queue.add(task); Reachable Servlet
  24. 24. Detail Configure ● Task queues use token bucket algorithm [ref]
  25. 25. ● rate - usual rate ● bucket-size - cap on peak demand ● max concurrent requests ● Retry options ○ task-retry-limit - min retries ○ task-age-limit - min elapsed time to keep retrying (Both task-retry-limit and task-age-limit are reached, task is deleted) ○ min-backoff-seconds -- min delay between retries ○ max-backoff-seconds -- max delay between retries ○ max-doublings -- max times to double delay ● more parameters see DOC queue.xml parameters
  26. 26. Task Queue - With Transaction DatastoreService ds = DatastoreServiceFactory.getDatastoreService(); Queue queue = QueueFactory.getDefaultQueue(); try { Transaction txn = ds.beginTransaction(); // … other operations queue.add(TaskOptions.Builder.withUrl("/path-to-my-worker")); // … other operations txn.commit(); } catch (DatastoreFailureException e) { // … exception handle }
  27. 27. Pull Queue - Create Queue Queue queue = QueueFactory.getQueue("queue_name"); TaskOptions task = TaskOptions.Builder .withMethod(TaskOptions.Method.PULL) .payload(payload); queue.add(task);
  28. 28. Pull Queue - Lease Queue queue = QueueFactory.getQueue("queue_name"); tasks = q.leaseTasks( duration, TimeUnit.SECONDS, how-many);
  29. 29. Pull Queue - Delete Task //delete named task queue.deleteTask("task_ name"); //purge all tasks from queue QueueFactory.getQueue("queue_name").purge();
  30. 30. Task Queue ACL - queue.xml <queue> <name>pull-queue1</name> <mode>pull</mode> <acl> <user-email>me@gmail.com</user-email> <writer-email>you@gmail.com</writer-email> <writer-email>she@gmail.com</writer-email> </acl> </queue>
  31. 31. Cron
  32. 32. Cron <cronentries> <cron> <url>/recache</url> <description> Put your description here </description> <schedule>every 5 minutes</schedule> </cron> </cronentries>
  33. 33. ● every 12 hours ● every 5 minutes from 10:00 to 14:00 ● 2nd,third mon,wed,thu of march 17:00 ● every monday 09:00 ● 1st monday of sep,oct,nov 17:00 ● every day 00:00 ● every N (hours|mins|minutes) ["from" (time) "to" (time)] ● every 2 hours synchronized Cron Schedule Format
  34. 34. Cron Manage Console
  35. 35. Restrictions ● 20 cron jobs for free apps ● 100 for paid apps
  36. 36. Task Queue / Cron Reference ● Task Queues Developer Guide (Java) (Python) ● Task Queue Configuration (Java) (Python) ● Task Queue API Ref (JavaDoc) (Python) ● Task Queue REST API Ref ● Scheduled Tasks with Cron (Java) (Python) ● Task queue quota and limits ● Get started with Task API article (Python) ● Google I/O 2011: Putting Task Queues to Work (video)

×