Your SlideShare is downloading. ×
  • Like
Reliable and Efficient Facebook data processing
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Reliable and Efficient Facebook data processing

  • 685 views
Published

Talk for the socal piggies meetup 2013-01-17 at Telesign's office.

Talk for the socal piggies meetup 2013-01-17 at Telesign's office.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
685
On SlideShare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
3
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Reliable and Efficient Facebook data processingAndres BuriticaSocal Piggies2013-01-17
  • 2. http://thelinuxkid.comPython developerFacebook Graph API experienceUbernearFounderDating
  • 3. MetaDraws on lessons from UbernearBased on traversal of public nodes in Graph
  • 4. Use casePages or users (owners) with public eventsEvent discovery
  • 5. Flow Update owners Owner IDs Update owner events Expire events Event details Partial events Complete events
  • 6. Why?Separation of concernsParallel processing with separateuser/apps/serversLess data load on batch requestsWant to store all data
  • 7. Update ownersCheck for migrated owners ○ (#21) Page ID <old_id> was migrated to page ID <new_id>. Please update your API calls to the new IDMove migrated owners to separate tableAdd new owners
  • 8. Update owner eventsEvents for owners not checked since datetimeStop at last event previously collected
  • 9. Expire eventsend_time has passed move to another tableFalse ○ Data might existShould have returned False? ○ [100] Unsupported get request ○ Data might exist
  • 10. Expire eventsAlias not found ○ (#803) Some of the aliases you requested do not exist...
  • 11. Event detailsSkip completed (no refreshing)Transient errors (retry) ○ None ○ OAuthException...Error validating application ○ (#1) "Unknown error occured" ○ "(#2) Service temporarily unavailable" ○ "(#4) User request limit reached" (throttle) ○ "(#4) Application request limit reached" (throttle) ○ "(#17) User request limit reached" (throttle)
  • 12. DatetimesAll in ISO-8601Events ○ date_format modifier has no effect ○ timezones after "Events Timezone Migration" ○ is_date_only ○ legacy without timezone
  • 13. Batch requestsPOST50 requests in oneLarge or complex can time outNested calls count towards rate limitingOne top level access token, many nested
  • 14. Batch request exampleUsers profile and 50 friendsbatch=[{ "method":"GET", "relative_url":"me" }, { "method":"GET", "relative_url":"me/friends?limit=50"}]
  • 15. Batch dependenciesReference results of a previous operationJSONPathChild operation executed after parentParent returns None unless forced
  • 16. Batch dependencies exampleGet details of 5 friendsbatch=[{ "method":"GET", "name":"get-friends", "relative_url":"me/friends?limit=5", }, { "method":"GET", "relative_url":"?ids={result=get-friends:$.data.*.id}"}]
  • 17. Field expansionGET"join" multiple graph queries into a single callReplacement for FQLfields, connections, modifiers and identifiersNo JSONPath
  • 18. Field expansion exampleUsers name and birthday plus id and picture ofthe last 10 photos/me? fields= name, birthday, photos.limit(10).fields(id,picture)
  • 19. Batch request with field expansionsUsers profile and picture link of 10 photosbatch=[{ "method":"GET", "relative_url":"me" }, { "method":"GET", "relative_url":"me?fields=photos.limit(10).fields(picture)"}]
  • 20. ProcessFacebook is constantly improvingTesting crucialBeta tier
  • 21. Not coveredRefreshing dataPaginationThrottlingInsights
  • 22. ThanksSample code at http://ubernear.comQuestionshttp://thelinuxkid.com