Reliable and Efficient       Facebook data         processingAndres BuriticaSocal Piggies2013-01-17
http://thelinuxkid.comPython developerFacebook Graph API experienceUbernearFounderDating
MetaDraws on lessons from UbernearBased on traversal of public nodes in Graph
Use casePages or users (owners) with public eventsEvent discovery
Flow                     Update owners   Owner IDs         Update owner events Expire events Event details          Partia...
Why?Separation of concernsParallel processing with separateuser/apps/serversLess data load on batch requestsWant to store ...
Update ownersCheck for migrated owners  ○ (#21) Page ID <old_id> was migrated to page ID    <new_id>. Please update your A...
Update owner eventsEvents for owners not checked since datetimeStop at last event previously collected
Expire eventsend_time has passed move to another tableFalse  ○ Data might existShould have returned False?  ○ [100] Unsupp...
Expire eventsAlias not found  ○ (#803) Some of the aliases you requested do not    exist...
Event detailsSkip completed (no refreshing)Transient errors (retry)   ○   None   ○   OAuthException...Error validating app...
DatetimesAll in ISO-8601Events  ○   date_format modifier has no effect  ○   timezones after "Events Timezone Migration"  ○...
Batch requestsPOST50 requests in oneLarge or complex can time outNested calls count towards rate limitingOne top level acc...
Batch request exampleUsers profile and 50 friendsbatch=[{  "method":"GET",  "relative_url":"me"  },  {  "method":"GET",  "...
Batch dependenciesReference results of a previous operationJSONPathChild operation executed after parentParent returns Non...
Batch dependencies exampleGet details of 5 friendsbatch=[{  "method":"GET",  "name":"get-friends",  "relative_url":"me/fri...
Field expansionGET"join" multiple graph queries into a single callReplacement for FQLfields, connections, modifiers and id...
Field expansion exampleUsers name and birthday plus id and picture ofthe last 10 photos/me? fields=   name,   birthday,   ...
Batch request with field expansionsUsers profile and picture link of 10 photosbatch=[{  "method":"GET",  "relative_url":"m...
ProcessFacebook is constantly improvingTesting crucialBeta tier
Not coveredRefreshing dataPaginationThrottlingInsights
ThanksSample code at http://ubernear.comQuestionshttp://thelinuxkid.com
Upcoming SlideShare
Loading in...5
×

Reliable and Efficient Facebook data processing

850

Published on

Talk for the socal piggies meetup 2013-01-17 at Telesign's office.

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
850
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Reliable and Efficient Facebook data processing

  1. 1. Reliable and Efficient Facebook data processingAndres BuriticaSocal Piggies2013-01-17
  2. 2. http://thelinuxkid.comPython developerFacebook Graph API experienceUbernearFounderDating
  3. 3. MetaDraws on lessons from UbernearBased on traversal of public nodes in Graph
  4. 4. Use casePages or users (owners) with public eventsEvent discovery
  5. 5. Flow Update owners Owner IDs Update owner events Expire events Event details Partial events Complete events
  6. 6. Why?Separation of concernsParallel processing with separateuser/apps/serversLess data load on batch requestsWant to store all data
  7. 7. Update ownersCheck for migrated owners ○ (#21) Page ID <old_id> was migrated to page ID <new_id>. Please update your API calls to the new IDMove migrated owners to separate tableAdd new owners
  8. 8. Update owner eventsEvents for owners not checked since datetimeStop at last event previously collected
  9. 9. Expire eventsend_time has passed move to another tableFalse ○ Data might existShould have returned False? ○ [100] Unsupported get request ○ Data might exist
  10. 10. Expire eventsAlias not found ○ (#803) Some of the aliases you requested do not exist...
  11. 11. Event detailsSkip completed (no refreshing)Transient errors (retry) ○ None ○ OAuthException...Error validating application ○ (#1) "Unknown error occured" ○ "(#2) Service temporarily unavailable" ○ "(#4) User request limit reached" (throttle) ○ "(#4) Application request limit reached" (throttle) ○ "(#17) User request limit reached" (throttle)
  12. 12. DatetimesAll in ISO-8601Events ○ date_format modifier has no effect ○ timezones after "Events Timezone Migration" ○ is_date_only ○ legacy without timezone
  13. 13. Batch requestsPOST50 requests in oneLarge or complex can time outNested calls count towards rate limitingOne top level access token, many nested
  14. 14. Batch request exampleUsers profile and 50 friendsbatch=[{ "method":"GET", "relative_url":"me" }, { "method":"GET", "relative_url":"me/friends?limit=50"}]
  15. 15. Batch dependenciesReference results of a previous operationJSONPathChild operation executed after parentParent returns None unless forced
  16. 16. Batch dependencies exampleGet details of 5 friendsbatch=[{ "method":"GET", "name":"get-friends", "relative_url":"me/friends?limit=5", }, { "method":"GET", "relative_url":"?ids={result=get-friends:$.data.*.id}"}]
  17. 17. Field expansionGET"join" multiple graph queries into a single callReplacement for FQLfields, connections, modifiers and identifiersNo JSONPath
  18. 18. Field expansion exampleUsers name and birthday plus id and picture ofthe last 10 photos/me? fields= name, birthday, photos.limit(10).fields(id,picture)
  19. 19. Batch request with field expansionsUsers profile and picture link of 10 photosbatch=[{ "method":"GET", "relative_url":"me" }, { "method":"GET", "relative_url":"me?fields=photos.limit(10).fields(picture)"}]
  20. 20. ProcessFacebook is constantly improvingTesting crucialBeta tier
  21. 21. Not coveredRefreshing dataPaginationThrottlingInsights
  22. 22. ThanksSample code at http://ubernear.comQuestionshttp://thelinuxkid.com
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×