Loading…

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

Like this presentation? Why not share!

Ungooglable

on

  • 1,317 views

This talk covers a basic methodology for finding and fixing problems in a live system. It covers general techniques for finding the source of issues quickly, workarounds, patching, digging into code, ...

This talk covers a basic methodology for finding and fixing problems in a live system. It covers general techniques for finding the source of issues quickly, workarounds, patching, digging into code, when and how to get help.

Statistics

Views

Total Views
1,317
Views on SlideShare
1,306
Embed Views
11

Actions

Likes
1
Downloads
19
Comments
0

2 Embeds 11

http://www.linkedin.com 10
https://www.linkedin.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Ungooglable Ungooglable Presentation Transcript

  • Un able managing “disasters” without loosing your cool @eleddy
  • This talk is for the Develadminisystemators who have to constantly deal with UNKNOWNS
  • Three Commands ‣ Know thy system ‣ Know thy tools ‣ Know thy neighbors
  • Stairway to Document Freedom Fix Clean Patch Horizon of Intervention Diagnose Damage Control Isolate Prepare
  • Communicate coworkers bossman Mayday! High Priority Dear Magic Makers - As some of you may already know, customers are experiencing troubles retrieving their historical records because our archive server is not responding. I am investigating the issue now and will send an update in 20 minutes. Please fence calls in the meanwhile. If someone can please get me a redbull and some nacho cheese corn nuts in the meanwhile, that would be stellar. Thanks! Prepare Isolate Control Diagnose Patch Clean Fix Document
  • Prepare for the Worst ‣ Backups ‣ Local Data.fs ‣ Set a time limit Prepare Isolate Control Diagnose Patch Clean Fix Document
  • Disable Interference Disabled all backups and packing Opened up port 8080 to outside network Moved logs to temporary disk Prepare Isolate Control Diagnose Patch Clean Fix Document
  • Isolation by Elimination Network Hardware Software Data obvious, not works for everything sporadic recreatable me else crazy shit locally Prepare Isolate Control Diagnose Patch Clean Fix Document
  • Isolation by Elimination Network Hardware Software Data obvious, not works for everything sporadic recreatable me else crazy shit locally Prepare Isolate Control Diagnose Patch Clean Fix Document
  • Isolation by Elimination Network Hardware Software Data obvious, not works for everything sporadic recreatable me else crazy shit locally Prepare Isolate Control Diagnose Patch Clean Fix Document
  • Isolation by Elimination Network Hardware Software Data obvious, not works for everything sporadic recreatable me else crazy shit locally Prepare Isolate Control Diagnose Patch Clean Fix Document
  • Isolation by Elimination Network Hardware Software Data obvious, not works for everything sporadic recreatable me else crazy shit locally Prepare Isolate Control Diagnose Patch Clean Fix Document
  • Zopesplosion 3000 Architecture Apache Zope ZEO 1-4 Varnish Zope ZEO 5-8 HAProxy Zope ZEO 9-12 Zope WTF mate CDN Zope MySQL APIs Zope MongoDB Zope SPARQL Prepare Isolate Control Diagnose Patch Clean Fix Document
  • Zopesplosion 3000 Architecture ? Apache Zope ZEO 1-4 Varnish Zope ZEO 5-8 HAProxy Zope ZEO 9-12 Zope CDN Zope MySQL APIs Zope MongoDB Zope SPARQL Prepare Isolate Control Diagnose Patch Clean Fix Document
  • Zopesplosion 3000 Architecture Apache Zope ZEO 1-4 Varnish Zope ZEO 5-8 HAProxy Zope ZEO 9-12 Zope ? CDN APIs Zope Zope Zope ? MySQL MongoDB SPARQL Prepare Isolate Control Diagnose Patch Clean Fix Document
  • Zopesplosion 3000 Architecture ? Apache Zope ZEO 1-4 Varnish Zope ZEO 5-8 HAProxy Zope ZEO 9-12 Zope CDN Zope MySQL APIs Zope MongoDB Zope SPARQL Prepare Isolate Control Diagnose Patch Clean Fix Document
  • How Zeo Cache Works Mem. Zope Zeo Cache Machine A Machine B
  • How Zeo Cache Works I Want X Mem. Zope Zeo Cache Machine A Machine B
  • How Zeo Cache Works I Want X I Need X Mem. Zope Zeo Cache Machine A Machine B
  • How Zeo Cache Works I Want X I Need X X Mem. Zope Zeo Cache Machine A Machine B
  • How Zeo Cache Works I Want X I Need X X Mem. Zope Zeo Cache X Machine A Machine B
  • How Zeo Cache Works I Want X I Need X X X Mem. Zope Zeo Cache X Machine A Machine B
  • How Zeo Cache Works I Want X I Need X X X Mem. Zope Zeo Modified X Cache X Machine A Machine B
  • How Zeo Cache Works I Want X I Need X X X Mem. Zope Zeo Modified X Cache X‘ Machine A Machine B
  • How Zeo Cache Works I Want X I Need X X X Mem. Zope Zeo Modified X Cache Modified X X‘ Machine A Machine B
  • How Zeo Cache Works I Want X I Need X X X Mem. Zope Zeo Modified X Cache Modified X X‘ Machine A Machine B I Want X Inconsistent State! X Disk Zope Zeo Modified X Cache X‘ RESTART Machine A Machine B
  • Zopesplosion 3000 Architecture Apache Zope ZEO 1-4 Varnish Zope ZEO 5-8 HAProxy Zope ZEO 9-12 Hot Zope damn! CDN Zope MySQL APIs Zope MongoDB Zope SPARQL Prepare Isolate Control Diagnose Patch Clean Patch Document
  • Take time to make time ‣ Minimize customer angst ‣ Hang out in custom ‣ Acquisition is your friend ‣ Remember request and response Prepare Isolate Control Diagnose Patch Clean Fix Document
  • Prepare Isolate Control Diagnose Patch Clean Fix Document
  • Unique or Just Not Obvious? ‣ Zope, zeo, system logs ‣ System stats/monitoring Prepare Isolate Control Diagnose Patch Clean Fix Document
  • Probably Test Case not... Sarcoidosis! + Estimate Fix Time Prepare Isolate Control Diagnose Patch Clean Fix Document
  • Horizon of Intervention Can Can I I do it in Yes Yes handle this a timely problem? manner? NO NO IRC Plone-users Friends Colleagues Prepare Isolate Control Diagnose Patch Clean Fix Document
  • Front End Errors Take the performance hit Disable the malfunctioning piece Prepare Isolate Control Diagnose Patch Clean Fix Document
  • temporary patch full patch Prepare Isolate Control Diagnose Patch Clean Fix Document
  • Have I mentioned the importance of working with BACKUPS yet? Especially when unfucking data... Prepare Isolate Control Diagnose Patch Clean Fix Document
  • Clean up Disabled all backups and packing Opened up port 8080 to outside network Moved logs to temporary disk Disabled zopes 5-10 Prepare Isolate Control Diagnose Patch Clean Fix Document
  • Clean up Disabled all backups and packing Opened up port 8080 to outside network Moved logs to temporary disk Disabled zopes 5-10 Prepare Isolate Control Diagnose Patch Clean Fix Document
  • Clean up Delete extra/bad files Scripts in version control Communicate Prepare Isolate Control Diagnose Patch Clean Fix Document
  • I’ve got a fever, and the only solution... is MORE PATCH! Prepare Isolate Control Diagnose Patch Clean Fix Document
  • ‣ Update/Close Tickets ‣ Integrate Test Cases ‣ Document Processes Prepare Isolate Control Diagnose Patch Clean Fix Document
  • Handling Data Errors Network Hardware Software Data obvious, not works for everything sporadic recreatable me else crazy shit locally Prepare Isolate Control Diagnose Patch Clean Fix Document
  • Handling Data Errors Network Hardware Software Data obvious, not works for everything sporadic recreatable me else crazy shit locally Prepare Isolate Control Diagnose Patch Clean Fix Document
  • Handling Data Errors Network Hardware Software Data obvious, not works for everything sporadic recreatable me else crazy shit locally Prepare Isolate Control Diagnose Patch Clean Fix Document
  • Handling Data Errors Network Hardware Software Data obvious, not works for everything sporadic recreatable me else crazy shit locally Prepare Isolate Control Diagnose Patch Clean Fix Document
  • Handling Data Errors Network Hardware Software Data obvious, not works for everything sporadic recreatable me else crazy shit locally Prepare Isolate Control Diagnose Patch Clean Fix Document
  • Prepare Isolate Control Diagnose Patch Clean Fix Document
  • How Data is Stored news.2010.09.08 news.2010.06.13 users roles Members News Events acl_users users roles acl_users Plone temp_folder root (app) Prepare Isolate Control Diagnose Patch Clean Fix Document
  • The Basics ‣ ./bin/instance debug ‣ app ‣ dir, __dict__ Prepare Isolate Control Diagnose Patch Clean Fix Document
  • Direct Connect >>> from ZODB.FileStorage import FileStorage >>> from ZODB.DB import DB >>> storage = FileStorage('var/filestorage/Data.fs') >>> db = DB(storage) >>> connection = db.open() >>> root = connection.root() >>> from ZEO import ClientStorage >>> from ZODB import DB >>> address = '10.0.1.5', 8001 >>> db = DB(storage) >>> connection = db.open() >>> root = connection.root() >>> root[‘app’] = PloneSite() >>> root[‘status’] = ‘Running’ Prepare Isolate Control Diagnose Patch Clean Fix Document
  • >>> import transaction >>> del app.Plone.news[‘news-item-id’] >>> transaction.commit() Prepare Isolate Control Diagnose Patch Clean Fix Document
  • _p_changed Prepare Isolate Control Diagnose Patch Clean Fix Document
  • When in doubt... ‣ PDB is your friend ‣ The source is your friend ‣ Throw a party for your friends Prepare Isolate Control Diagnose Patch Clean Fix Document
  • ‣ Know your System ‣ Understand the Tools ‣ Be Nice to your Neighbors