Un ablemanaging “disasters” without loosing your cool
@eleddy
Develadminisystemators
This talk is for the
who have to
constantly deal with
UNKNOWNS
‣ Know thy system
‣ Know thy tools
‣ Know thy neighbors
Three
Commands
Stairway to
Freedom
Prepare
Isolate
Damage Control
Diagnose
Patch
Clean
Fix
Document
Horizon of Intervention
Communicate
Prepare Isolate Control Diagnose Patch Clean Fix Document
Dear Magic Makers -
As some of you may already know,...
Prepare for the Worst
‣ Backups
‣ Local Data.fs
‣ Set a time limit
Prepare Isolate Control Diagnose Patch Clean Fix Docume...
Disable Interference
Disabled all backups and
packing
Opened up port 8080 to
outside network
Moved logs to temporary
disk
...
Isolation by Elimination
Prepare Isolate Control Diagnose Patch Clean Fix Document
Network Hardware Software Data
works fo...
Isolation by Elimination
Prepare Isolate Control Diagnose Patch Clean Fix Document
Network Hardware Software Data
works fo...
Isolation by Elimination
Prepare Isolate Control Diagnose Patch Clean Fix Document
Network Hardware Software Data
works fo...
Isolation by Elimination
Prepare Isolate Control Diagnose Patch Clean Fix Document
Network Hardware Software Data
works fo...
Isolation by Elimination
Prepare Isolate Control Diagnose Patch Clean Fix Document
Network Hardware Software Data
works fo...
Zopesplosion 3000 Architecture
Apache
Varnish
HAProxy
CDN
APIs
Zope
Zope
Zope
Zope
Zope
Zope
Zope MySQL
MongoDB
SPARQL
WTF...
Zopesplosion 3000 Architecture
Apache
Varnish
HAProxy
CDN
APIs
Zope
Zope
Zope
Zope
Zope
Zope
Zope MySQL
MongoDB
SPARQL
ZEO...
Zopesplosion 3000 Architecture
Apache
Varnish
HAProxy
CDN
APIs
Zope
Zope
Zope
Zope
Zope
Zope
Zope MySQL
MongoDB
SPARQL
ZEO...
Zopesplosion 3000 Architecture
Apache
Varnish
HAProxy
CDN
APIs
Zope
Zope
Zope
Zope
Zope
Zope
Zope MySQL
MongoDB
SPARQL
ZEO...
Machine BMachine A
How Zeo Cache Works
Zope
Mem.
Cache
Zeo
Machine BMachine A
How Zeo Cache Works
Zope
Mem.
Cache
Zeo
I Want X
Machine BMachine A
How Zeo Cache Works
Zope
Mem.
Cache
Zeo
I Want X
I Need X
Machine BMachine A
How Zeo Cache Works
Zope
Mem.
Cache
Zeo
I Want X
I Need X
X
Machine BMachine A
How Zeo Cache Works
Zope
Mem.
Cache
Zeo
I Want X
I Need X
X
X
Machine BMachine A
How Zeo Cache Works
Zope
Mem.
Cache
Zeo
I Want X
I Need X
X
X
X
Machine BMachine A
How Zeo Cache Works
Zope
Mem.
Cache
Zeo
I Want X
I Need X
X
X
X
Modified X
Machine BMachine A
How Zeo Cache Works
Zope
Mem.
Cache
Zeo
I Want X
I Need X
X
X
X
Modified X
‘
Machine BMachine A
How Zeo Cache Works
Zope
Mem.
Cache
Zeo
I Want X
I Need X
X
X
X
Modified X
‘ Modified X
Machine BMachine A
Machine BMachine A
How Zeo Cache Works
Zope
Mem.
Cache
Zeo
I Want X
I Need X
X
X
X
Modified X
‘ Modified ...
Zopesplosion 3000 Architecture
Apache
Varnish
HAProxy
CDN
APIs
Zope
Zope
Zope
Zope
Zope
Zope
Zope MySQL
MongoDB
SPARQL
ZEO...
Take time to make time
‣ Minimize customer angst
‣ Hang out in custom
‣ Acquisition is your friend
‣ Remember request and
...
Prepare Control Diagnose Patch Clean Fix DocumentIsolate
Unique or Just Not Obvious?
‣ Zope, zeo, system logs
‣ System stats/monitoring
Prepare Isolate Control Diagnose Patch Clea...
Test Case
Prepare Isolate Control Diagnose Patch Clean Fix Document
Sarcoidosis!
Probably
not...
Estimate
Fix Time
+
Horizon of Intervention
Prepare Isolate Control Diagnose Patch Clean Fix Document
Can I
handle this
problem?
Can
I do it i...
Front End
Errors
Take the performance hit
Disable the malfunctioning piece
Prepare Isolate Control Diagnose Patch Clean Fi...
temporary patch
Prepare Isolate Control Diagnose Patch Clean Fix Document
full patch
Have I mentioned the
importance of
Prepare Isolate Control Diagnose Patch Clean Fix Document
BACKUPS
working with
yet?
Esp...
Clean up
Prepare Isolate Control Diagnose Patch Clean Fix Document
Disabled all backups and
packing
Opened up port 8080 to...
Clean up
Prepare Isolate Control Diagnose Patch Clean Fix Document
Disabled all backups and
packing
Opened up port 8080 to...
Prepare Isolate Control Diagnose Patch Clean Fix Document
Delete extra/bad files
Scripts in version control
Communicate
Cl...
Prepare Isolate Control Diagnose Patch Clean Fix Document
I’ve got a fever, and
the only solution... is
MORE PATCH!
‣ Update/Close Tickets
‣ Integrate Test Cases
‣ Document Processes
Prepare Isolate Control Diagnose Patch Clean Fix Docume...
Handling Data Errors
Prepare Isolate Control Diagnose Patch Clean Fix Document
Network Hardware Software Data
works for
me...
Handling Data Errors
Prepare Isolate Control Diagnose Patch Clean Fix Document
Network Hardware Software Data
works for
me...
Handling Data Errors
Prepare Isolate Control Diagnose Patch Clean Fix Document
Network Hardware Software Data
works for
me...
Handling Data Errors
Prepare Isolate Control Diagnose Patch Clean Fix Document
Network Hardware Software Data
works for
me...
Handling Data Errors
Prepare Isolate Control Diagnose Patch Clean Fix Document
Network Hardware Software Data
works for
me...
Prepare Isolate Control Diagnose Patch Clean Fix Document
How Data is Stored
Plone
root (app)
NewsMembers Events
acl_users
acl_users
users roles
users roles
news.2010.09.08 news.20...
The Basics
Prepare Isolate Control Diagnose Patch Clean Fix Document
‣ ./bin/instance
debug
‣ app
‣ dir, __dict__
Direct Connect
>>> from ZODB.FileStorage import FileStorage
>>> from ZODB.DB import DB
>>> storage = FileStorage('var/file...
Prepare Isolate Control Diagnose Patch Clean Fix Document
>>> import transaction
>>> del app.Plone.news[‘news-item-id’]
>>...
_p_changed
Prepare Isolate Control Diagnose Patch Clean Fix Document
When in doubt...
‣ PDB is your friend
‣ The source is your friend
‣ Throw a party for your friends
Prepare Isolate Control...
‣ Know your System
‣ Understand the Tools
‣ Be Nice to your
Neighbors
Ungooglable
Ungooglable
Upcoming SlideShare
Loading in...5
×

Ungooglable

1,204

Published on

This talk covers a basic methodology for finding and fixing problems in a live system. It covers general techniques for finding the source of issues quickly, workarounds, patching, digging into code, when and how to get help.

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,204
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
20
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Ungooglable

  1. 1. Un ablemanaging “disasters” without loosing your cool @eleddy
  2. 2. Develadminisystemators This talk is for the who have to constantly deal with UNKNOWNS
  3. 3. ‣ Know thy system ‣ Know thy tools ‣ Know thy neighbors Three Commands
  4. 4. Stairway to Freedom Prepare Isolate Damage Control Diagnose Patch Clean Fix Document Horizon of Intervention
  5. 5. Communicate Prepare Isolate Control Diagnose Patch Clean Fix Document Dear Magic Makers - As some of you may already know, customers are experiencing troubles retrieving their historical records because our archive server is not responding. I am investigating the issue now and will send an update in 20 minutes. Please fence calls in the meanwhile. If someone can please get me a redbull and some nacho cheese corn nuts in the meanwhile, that would be stellar. Thanks! coworkers Mayday! High Priority bossman
  6. 6. Prepare for the Worst ‣ Backups ‣ Local Data.fs ‣ Set a time limit Prepare Isolate Control Diagnose Patch Clean Fix Document
  7. 7. Disable Interference Disabled all backups and packing Opened up port 8080 to outside network Moved logs to temporary disk Prepare Isolate Control Diagnose Patch Clean Fix Document
  8. 8. Isolation by Elimination Prepare Isolate Control Diagnose Patch Clean Fix Document Network Hardware Software Data works for me obvious, sporadic crazy shit everything else not recreatable locally
  9. 9. Isolation by Elimination Prepare Isolate Control Diagnose Patch Clean Fix Document Network Hardware Software Data works for me obvious, sporadic crazy shit everything else not recreatable locally
  10. 10. Isolation by Elimination Prepare Isolate Control Diagnose Patch Clean Fix Document Network Hardware Software Data works for me obvious, sporadic crazy shit everything else not recreatable locally
  11. 11. Isolation by Elimination Prepare Isolate Control Diagnose Patch Clean Fix Document Network Hardware Software Data works for me obvious, sporadic crazy shit everything else not recreatable locally
  12. 12. Isolation by Elimination Prepare Isolate Control Diagnose Patch Clean Fix Document Network Hardware Software Data works for me obvious, sporadic crazy shit everything else not recreatable locally
  13. 13. Zopesplosion 3000 Architecture Apache Varnish HAProxy CDN APIs Zope Zope Zope Zope Zope Zope Zope MySQL MongoDB SPARQL WTF mate ZEO 1-4 ZEO 5-8 ZEO 9-12 Prepare Control Diagnose Patch Clean Fix DocumentIsolate
  14. 14. Zopesplosion 3000 Architecture Apache Varnish HAProxy CDN APIs Zope Zope Zope Zope Zope Zope Zope MySQL MongoDB SPARQL ZEO 1-4 ZEO 5-8 ZEO 9-12 Prepare Control Diagnose Patch Clean Fix DocumentIsolate ?
  15. 15. Zopesplosion 3000 Architecture Apache Varnish HAProxy CDN APIs Zope Zope Zope Zope Zope Zope Zope MySQL MongoDB SPARQL ZEO 1-4 ZEO 5-8 ZEO 9-12 Prepare Control Diagnose Patch Clean Fix DocumentIsolate ? ?
  16. 16. Zopesplosion 3000 Architecture Apache Varnish HAProxy CDN APIs Zope Zope Zope Zope Zope Zope Zope MySQL MongoDB SPARQL ZEO 1-4 ZEO 5-8 ZEO 9-12 Prepare Control Diagnose Patch Clean Fix DocumentIsolate ?
  17. 17. Machine BMachine A How Zeo Cache Works Zope Mem. Cache Zeo
  18. 18. Machine BMachine A How Zeo Cache Works Zope Mem. Cache Zeo I Want X
  19. 19. Machine BMachine A How Zeo Cache Works Zope Mem. Cache Zeo I Want X I Need X
  20. 20. Machine BMachine A How Zeo Cache Works Zope Mem. Cache Zeo I Want X I Need X X
  21. 21. Machine BMachine A How Zeo Cache Works Zope Mem. Cache Zeo I Want X I Need X X X
  22. 22. Machine BMachine A How Zeo Cache Works Zope Mem. Cache Zeo I Want X I Need X X X X
  23. 23. Machine BMachine A How Zeo Cache Works Zope Mem. Cache Zeo I Want X I Need X X X X Modified X
  24. 24. Machine BMachine A How Zeo Cache Works Zope Mem. Cache Zeo I Want X I Need X X X X Modified X ‘
  25. 25. Machine BMachine A How Zeo Cache Works Zope Mem. Cache Zeo I Want X I Need X X X X Modified X ‘ Modified X
  26. 26. Machine BMachine A Machine BMachine A How Zeo Cache Works Zope Mem. Cache Zeo I Want X I Need X X X X Modified X ‘ Modified X Zope Disk Cache Zeo I Want X X X Modified X ‘ RESTART Inconsistent State!
  27. 27. Zopesplosion 3000 Architecture Apache Varnish HAProxy CDN APIs Zope Zope Zope Zope Zope Zope Zope MySQL MongoDB SPARQL ZEO 1-4 ZEO 5-8 ZEO 9-12 Prepare Control Diagnose Patch Clean Patch DocumentIsolate Hot damn!
  28. 28. Take time to make time ‣ Minimize customer angst ‣ Hang out in custom ‣ Acquisition is your friend ‣ Remember request and response Prepare Control Diagnose Patch Clean Fix DocumentIsolate
  29. 29. Prepare Control Diagnose Patch Clean Fix DocumentIsolate
  30. 30. Unique or Just Not Obvious? ‣ Zope, zeo, system logs ‣ System stats/monitoring Prepare Isolate Control Diagnose Patch Clean Fix Document
  31. 31. Test Case Prepare Isolate Control Diagnose Patch Clean Fix Document Sarcoidosis! Probably not... Estimate Fix Time +
  32. 32. Horizon of Intervention Prepare Isolate Control Diagnose Patch Clean Fix Document Can I handle this problem? Can I do it in a timely manner? Yes IRC Plone-users Yes NONO Friends Colleagues
  33. 33. Front End Errors Take the performance hit Disable the malfunctioning piece Prepare Isolate Control Diagnose Patch Clean Fix Document
  34. 34. temporary patch Prepare Isolate Control Diagnose Patch Clean Fix Document full patch
  35. 35. Have I mentioned the importance of Prepare Isolate Control Diagnose Patch Clean Fix Document BACKUPS working with yet? Especially when unfucking data...
  36. 36. Clean up Prepare Isolate Control Diagnose Patch Clean Fix Document Disabled all backups and packing Opened up port 8080 to outside network Moved logs to temporary disk Disabled zopes 5-10
  37. 37. Clean up Prepare Isolate Control Diagnose Patch Clean Fix Document Disabled all backups and packing Opened up port 8080 to outside network Moved logs to temporary disk Disabled zopes 5-10
  38. 38. Prepare Isolate Control Diagnose Patch Clean Fix Document Delete extra/bad files Scripts in version control Communicate Clean up
  39. 39. Prepare Isolate Control Diagnose Patch Clean Fix Document I’ve got a fever, and the only solution... is MORE PATCH!
  40. 40. ‣ Update/Close Tickets ‣ Integrate Test Cases ‣ Document Processes Prepare Isolate Control Diagnose Patch Clean Fix Document
  41. 41. Handling Data Errors Prepare Isolate Control Diagnose Patch Clean Fix Document Network Hardware Software Data works for me obvious, sporadic crazy shit everything else not recreatable locally
  42. 42. Handling Data Errors Prepare Isolate Control Diagnose Patch Clean Fix Document Network Hardware Software Data works for me obvious, sporadic crazy shit everything else not recreatable locally
  43. 43. Handling Data Errors Prepare Isolate Control Diagnose Patch Clean Fix Document Network Hardware Software Data works for me obvious, sporadic crazy shit everything else not recreatable locally
  44. 44. Handling Data Errors Prepare Isolate Control Diagnose Patch Clean Fix Document Network Hardware Software Data works for me obvious, sporadic crazy shit everything else not recreatable locally
  45. 45. Handling Data Errors Prepare Isolate Control Diagnose Patch Clean Fix Document Network Hardware Software Data works for me obvious, sporadic crazy shit everything else not recreatable locally
  46. 46. Prepare Isolate Control Diagnose Patch Clean Fix Document
  47. 47. How Data is Stored Plone root (app) NewsMembers Events acl_users acl_users users roles users roles news.2010.09.08 news.2010.06.13 Prepare Isolate Control Diagnose Patch Clean Fix Document temp_folder
  48. 48. The Basics Prepare Isolate Control Diagnose Patch Clean Fix Document ‣ ./bin/instance debug ‣ app ‣ dir, __dict__
  49. 49. Direct Connect >>> from ZODB.FileStorage import FileStorage >>> from ZODB.DB import DB >>> storage = FileStorage('var/filestorage/Data.fs') >>> db = DB(storage) >>> connection = db.open() >>> root = connection.root() Prepare Isolate Control Diagnose Patch Clean Fix Document >>> from ZEO import ClientStorage >>> from ZODB import DB >>> address = '10.0.1.5', 8001 >>> db = DB(storage) >>> connection = db.open() >>> root = connection.root() >>> root[‘app’] = PloneSite() >>> root[‘status’] = ‘Running’
  50. 50. Prepare Isolate Control Diagnose Patch Clean Fix Document >>> import transaction >>> del app.Plone.news[‘news-item-id’] >>> transaction.commit()
  51. 51. _p_changed Prepare Isolate Control Diagnose Patch Clean Fix Document
  52. 52. When in doubt... ‣ PDB is your friend ‣ The source is your friend ‣ Throw a party for your friends Prepare Isolate Control Diagnose Patch Clean Fix Document
  53. 53. ‣ Know your System ‣ Understand the Tools ‣ Be Nice to your Neighbors
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×