Troubleshooting Plone

Troubleshooting Plone Ricardo Alves [email_address] Plone Conference 2009 - Budapest

What can go wrong?

Main Issues

Site errors and broken objects
Instance not responding
Server IO errors
Unauthorized / security errors

Site Errors

Analyzing the traceback at error_log:
http://mysite/plonesite/error_log/manage_main

Site Errors

Analyzing Zope's event log:
INSTANCE_HOME/logs/event.log

Module zope.tal.talinterpreter, line 346, in interpret Module zope.tal.talinterpreter, line 745, in do_insertStructure_tal Module zope.tales.tales, line 696, in evaluate - URL: file:/Users/rsa/eurotux/udp/iporto/bundle/plone/CMFPlone/skins/plone_content/newsitem_view.pt - Line 41, Column 15 - Expression: <PythonExpr here.tag(scale='mini', css_class='newsImage')> - Names: {'container': <PloneSite at /plone>, 'context': <ATNewsItem at /plone/new-website-soon>, 'default': <object object at 0x395528>, 'here': <ATNewsItem at /plone/new-website-soon>, 'loop': {}, 'nothing': None, 'options': {'args': ()}, 'repeat': <Products.PageTemplates.Expressions.SafeMapping object at 0x6f4c0d0>, 'request': <HTTPRequest, URL=http://localhost:9095/plone/new-website-soon/newsitem_view>, 'root': <Application at >, 'template': <FSPageTemplate at /plone/newsitem_view used for /plone/new-website-soon>, 'traverse_subpath': [], 'user': <PropertiedUser 'admin'>} Module Products.PageTemplates.ZRPythonExpr, line 49, in __call__ - __traceback_info__: here.tag(scale='mini', css_class='newsImage') Module PythonExpr, line 1, in <expression> Module Products.ATContentTypes.content.newsitem, line 130, in tag AttributeError: 'NoneType' object has no attribute 'tag'

2009-10-28T16:43:46 WARNING LinguaPlone The thread number -1337921536 doesn't have an associated request object. ------ 2009-10-28T16:45:00 INFO SignalHandler Caught signal SIGINT ------ 2009-10-28T16:45:00 INFO Z2 Shutting down ------ 2009-10-28T16:45:21 INFO ZServer HTTP server started at Wed Oct 28 16:45:21 2009 Hostname: 0.0.0.0 Port: 9095 ------ 2009-10-28T16:45:23 INFO Marshall libxml2-python not available. Unable to register libxml2 based marshallers. ------ 2009-10-28T16:45:27 INFO ZEO.ClientStorage (4478) ClientStorage (pid=4478) created RW/normal for storage: '1' ------ 2009-10-28T16:45:27 INFO ZEO.cache created temporary cache file '<fdopen>' ------ 2009-10-28T16:45:27 INFO ZEO.ClientStorage (4478) Testing connection <ManagedClientConnection ('127.0.0.1', 9096)> ------ 2009-10-28T16:45:27 INFO ZEO.zrpc.Connection(C) (127.0.0.1:9096) received handshake 'Z303'

Logging level

Analyzing Zope's event log:
INSTANCE_HOME/etc/zope.conf

<eventlog> level info <logfile> path $INSTANCE/log/event.log level info # "CRITICAL", 'ERROR", WARN", "INFO", # "DEBUG", and "ALL" </logfile> </eventlog> <logger access> level WARN <logfile> path $INSTANCE/log/access.log format %(message)s </logfile> </logger>

Logging yourself

Logging to event log from product code:

>>> import logging >>> logger = logging.getLogger('My package') >>> logger.log(logging.INFO, 'My message') >>> context.plone_log('My message')

From skin layers (as Plone):

System logs $ tail -f /var/log/system.log

Specially if connecting with other applications (mail server, etc) $ tail -f /var/log/messages $ tail -f /var/log/system.log

Make sure that Zope is failing:

$ telnet localhost 80 Trying ::1... telnet: connect to address ::1: Connection refused Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. GET /plone HTTP/1.1 HTTP/1.1 200 OK Server: Zope/(Zope 2.10.4-final, python 2.4.4, darwin) ZServer/1.1 Plone/3.1.7 Date: Wed, 28 Oct 2009 19:13:49 GMT Content-Length: 20541 Expires: Sat, 1 Jan 2000 00:00:00 GMT Content-Type: text/html;charset=utf-8 Content-Language: en <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" " http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd "> ...

Instance not responding $ telnet localhost 80 Trying ::1... telnet: connect to address ::1: Connection refused Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. GET /plone HTTP/1.1 HTTP/1.1 404 Not Found Server: Zope/(Zope 2.10.4-final, python 2.4.4, darwin) ZServer/1.1 Plone/3.1.7 Date: Wed, 28 Oct 2009 19:16:57 GMT Bobo-Exception-Line: 100 Content-Length: 812 Bobo-Exception-Value: See the server error log for details Content-Type: text/html; charset=iso-8859-15 Bobo-Exception-Type: NotFound Bobo-Exception-File: BaseRequest.py <html> <head><title>Zope</title></head> ...

Make sure that ZEO is responding:

$ telnet localhost 81 Trying ::1... telnet: connect to address ::1: Connection refused Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. Z303

Start Zope in foreground:

$ INSTANCE_HOME/bin/zopectl fg

Start Zope in debug mode:

INSTANCE_HOME/etc/zope.conf:

# Directive: debug-mode ... # # Setting this to 'off' when Zope is in a production environment is # encouraged, as it speeds execution (sometimes dramatically). # # Default: off # # Example: # debug-mode on

About Zope debug mode:

Errors in product initialization prevent startup
Elements from skin layers (templates, scripts, etc) can be edited while the server is running

Start Zope in debug mode:

buildou.cfg

[instance]
recipe = plone.recipe.zope2instance
debug-mode = on

Zope Python console

Access database directly:

$ INSTANCE_HOME/bin/zopectl debug Starting debugger (the name "app" is bound to the top-levelZope object) >>> app <Application at > >>> app.plone <PloneSite at /plone> >>> app.plone.portal_catalog(portal_type='News Item') [<Products.ZCatalog.Catalog.mybrains object at 0x669e270>] >>>

Zope Python console

Tab completion:

>>> import rlcompleter >>> import readline >>> readline.parse_and_bind("tab: complete") >>> app.plone.portal_catalog.search<TAB PRESSED>

Authentication:

>>> from AccessControl.SecurityManagement import newSecurityManager >>> app.acl_users.getUserById('admin').__of__(app.acl_users) >>> newSecurityManager(None, user)

Zope Python console

Synchronization:

>>> app._p_jar.sync()

Transactions:

>>> import transaction >>> transaction.commit() # Explicit commit >>> transaction.abort() # Transaction aborted >>> transaction.savepoint() # Savepoint >>> app._p_jar.sync() >>> app._p_jar.sync()

Python debugger

Set a breakpoint:

Post-mortem:

try: return self.getField('photo').tag(self, **kwargs) except: import pdb, sys e, m, tb = sys.exc_info() pdb.post_mortem(tb) >>> import pdb; pdb.set_trace() INSTANCE_HOMT/plone/ATContentTypes/content/newsitem.py(133)tag()->None -> import pdb; pdb.set_trace( (Pdb) self <ATNewsItem at /plone/new-website-soon> (Pdb) self.getField('image') <Field image(image:rw)> (Pdb) self.getField('photo') >>> import pdb; pdb.set_trace()

Server IO errors

“ No space left on device”

Check disc space available for database
Check disc space at /tmp dir

Other errors:

“ No such file or directory”
“ Too many open files”
“ Read-only file system”
...

Debugging security issues

Default traceback gives poor information:

... Module Products.CMFPlone.browser.ploneview, line 111, in _initializeData Module plone.memoize.view, line 55, in memogetter Module plone.app.layout.globals.context, line 123, in object_title Module Products.CMFPlone.utils, line 158, in pretty_title_or_id Module Products.CMFPlone.utils, line 463, in base_hasattr Module Products.CMFPlone.utils, line 474, in safe_hasattr Module Shared.DC.Scripts.Bindings, line 184, in __getattr__ Unauthorized: You are not allowed to access 'Title' in this context

Enable verbose security

INSTANCE_HOME/etc/zope.conf:
security-policy-implementation python
verbose-security on
buildout.cfg:
[instance]
recipe = plone.recipe.zope2instance
verbose-security = on

More informative traceback:

Module plone.app.layout.globals.context, line 123, in object_title Module Products.CMFPlone.utils, line 158, in pretty_title_or_id Module Products.CMFPlone.utils, line 463, in base_hasattr Module Products.CMFPlone.utils, line 474, in safe_hasattr Module Shared.DC.Scripts.Bindings, line 184, in __getattr__ Module AccessControl.ImplPython, line 563, in validate Module AccessControl.ImplPython, line 461, in validate Module AccessControl.ImplPython, line 808, in raiseVerbose Unauthorized: Your user account does not have the required permission. Access to 'Title' of (ATNewsItem at /plone/new-website-soon) denied. Your user account, usertest, exists at /plone/acl_users. Access requires one of the following roles: ['Contributor', 'Editor', 'Manager', 'Owner', 'Reader']. Your roles in this context are ['Authenticated', 'Member'].

Reminder about Zope security:

Permissions -> Roles
Roles -> Principals
Principals (Groups and Users)
Groups -> Users

Use the Python console:

>>> app.plone.restrictedTraverse(' /path/to/object ') >>> from AccessControl.SecurityManagement import newSecurityManager >>> app.acl_users.getUserById( USERNAME ).__of__(app.acl_users) >>> newSecurityManager(None, user)

Debugging Tools