Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Scale Audit & Reporting with a NoSQL Architecture

2,150 views

Published on

Lucas Patingre presents Scale Audit & Reporting with a NoSQL Architecture at the Alfresco Summit 2014 in London.

Audit is a key functionality within Alfresco especially when using Records Management.

Alfresco out-of-the-box audit functionality is great, but for those scenarios where lots of audit records are required, it can be a bottleneck. This presentation shows a NoSQL Audit implementation that allows you to maximise the performance and scale the solution in an easy way. This audit solution eases the implementation of Reports and Analytics plugins and this presentation shows both the architecture and technology stack utilised behind it.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Scale Audit & Reporting with a NoSQL Architecture

  1. 1. Audit & Reporting with Alfresco & NoSQL architecture Lucas Patingre Alfresco consultant and technical lead at Zaizi
  2. 2. Summary  Some context  Alfresco audit  Scaling the audit  Benefits  Expanding on the architecture
  3. 3. Why auditing? A bit of context
  4. 4. Zaizi for Alfresco: • Platinum Partner and Best System Integrator Partner of Year 2012 & 2013 • Specialist Alfresco ECM partner. • Implemented the biggest and most complex Alfresco projects in the UK. • Help global enterprises manage their growing information volumes by leveraging Alfresco’s outstanding performance and scalability. Alfresco Partner of the Year 2012 & 2013
  5. 5. Audit data from Alfresco  Who  When  What  Action  Target  Details /alfresco-access /transaction /action=<actionNamegt /sub-actions=<sub action listgt /path=<prefixPathgt /type=<prefixTypegt /node=<nodeRefgt /user=<usergt /copy /from /node=<nodeRefgt /path=<prefixPathgt /type=<prefixTypegt /move /from /node=<nodeRefgt /path=<prefixPathgt /type=<prefixTypegt /properties /from=<mapOfValuesgt /<propertyNamegt=<propertyValuegt /to=<mapOfValuesgt /<propertyNamegt=<propertyValuegt /add=<mapOfValuesgt /<propertyNamegt=<propertyValuegt /delete=<mapOfValuesgt /<propertyNamegt=<propertyValuegt /aspects /add=<mapOfNamesgt /<aspectNamegt=null /delete=<mapOfNamesgt /<aspectNamegt=null /version-properties=<mapOfValuesgt /sub-action/<sequencegt /action=<actionNamegt /copy /move /properties /aspects
  6. 6. Existing Alfresco audit Sharing Alfresco database
  7. 7. Components overview
  8. 8. Alfresco's audit dashlet
  9. 9. Alfresco's audit storage
  10. 10. SQL to retrieve audit entries SELECT entry.id 'Id', entry.audit_time 'Time', user_string.string_value 'User', act_string.string_value 'Application', sv.string_value 'Value' FROM alf_audit_entry entry INNER JOIN alf_prop_value user ON (entry.audit_user_id = user.id) INNER JOIN alf_prop_string_value user_string ON ((user.persisted_type = 3 OR user.persisted_type = 5) AND user.long_value = user_string.id) INNER JOIN alf_audit_app app ON (entry.audit_app_id = app.id) INNER JOIN alf_prop_value act ON (app.app_name_id = act.id) INNER JOIN alf_prop_string_value act_string ON ((act.persisted_type = 3 OR act.persisted_type = 5) AND act.long_value = act_string.id) INNER JOIN alf_prop_link pl on (pl.root_prop_id = entry.audit_values_id) INNER JOIN alf_prop_value pv on (pl.value_prop_id = pv.id) LEFT JOIN alf_prop_string_value sv on (sv.id = pv.long_value and (pv.persisted_type = 3 OR pv.persisted_type = 5))
  11. 11. Alfresco's RM audit
  12. 12. Scaling the audit Presentation of the components
  13. 13. The challenges  Make the audit scale without hindering Alfresco  Keep the audit queries fast  Not delay too much time until stored  Backward compatible with Alfresco's default  Not break existing RM view audit  Have a similar look-and-feel
  14. 14. The Alfresco search approach  Move from Lucene  Embedded in Alfresco  Limited inspection tools  To SOLR  Externalised  Can be clustered  Comes with an administration console
  15. 15. Components overview
  16. 16. Syslog  Standard, efficient and well integrated in Java  Easy to implement a file rotation  Possibility to re-compute all the audit data from file  Lighten the weight on the database that is no longer a bottleneck
  17. 17. Logstash  Open Source  Works well with log files  Able to handle our audit.log  But potentially others too (OSSEC)  Already has an ElasticSearch connector OOTB
  18. 18. Elastic search  Open source  Powerful indexing capabilities  Easily scalable  Can be queried from Alfresco
  19. 19. Kibana  Open source  Web UI for easy access
  20. 20. Scaling the audit (2) Overview of the implementation
  21. 21. Override the audit component <bean id="auditComponent" class="org.alfresco.repo.audit.AuditComponentImpl"> <property name="transactionService" ref="transactionService"/> <property name="auditDAO" ref="auditDAO"/> <property name="auditModelRegistry" ref="Audit"/> <property name="propertyValueDAO" ref="propertyValueDAO"/> <property name="userAuditFilter" ref="userAuditFilter"/> <property name="auditFilter"> <bean class="org.alfresco.repo.audit.PropertyAuditFilter"> <property name="properties" ref="global-properties" /> </bean> </property> </bean> <bean id="auditComponent" class="com.zaizi.elasticsearch.audit.CustomAuditComponentImpl"> <property name="transactionService" ref="transactionService" /> <property name="auditDAO" ref="auditDAO" /> <property name="auditModelRegistry" ref="Audit" /> <property name="propertyValueDAO" ref="propertyValueDAO" /> <property name="auditFilter"> <bean class="org.alfresco.repo.audit.PropertyAuditFilter"> <property name="properties" ref="global-properties" /> </bean> </property> <property name="logAudit" ref="logAudit" /> </bean>
  22. 22. Specialise the audit component auditDAO.createAuditEntry(applicatio nId, time, username, auditData); auditDAO.createAuditEntry(applicatio nId, time, username, auditData);  Create JSON for action  Add non-action-related parameters to JSON  Configure Syslog  Log the resulting JSON to audit.log
  23. 23. Quality-of-life improvements  Poll Syslog availability  If not available, switch system to readonly  When back available, re-enable  Toggle logging system  Only file/only database/both  Availability through JMX
  24. 24. Retrieve audit data
  25. 25. Replace audit webscript audit.get.js var nodeRefAuditURI = '/api/node/' model.nodeRef +'/rmauditlog'; var auditURI = "/api/rma/admin/rmauditlog"; var nodeRefAuditURI = '/api/node/'+ model.nodeRef +'/esauditlog?appname=RM'; var auditURI = "/api/es/admin/esauditlog";  Switch at the share level  Enables us to handle a richer result returned  Modifying the Alfresco webscript would be a viable approach too
  26. 26. Reap the benefit What this whole work was for
  27. 27. Performance / stability  Asynchronous  Processing after audit.log non-blocking  Independent  Audit falling doesn't bring Alfresco down  Alfresco falling doesn't prevent to consult ES  No stress on Alfresco database when querying audit data
  28. 28. Extend use of “view audit log”
  29. 29. Draw real time statistics out of it
  30. 30. Expanding on the architecture Non-alfresco-generated audit data
  31. 31. Auditing the logins <AuditPath key="login"> <RecordValue key="user" dataExtractor="simpleValue" dataSource="/alfresco-access/loginUser" dataTrigger="/alfresco-access/login" /> </AuditPath> <AuditPath key="loginFailure"> <RecordValue key="user" dataExtractor="simpleValue" dataSource="/alfresco-access/loginUser" dataTrigger="/alfresco-access/loginFailure" /> </AuditPath> /alfresco-access /login/user=<user> /loginFailure/user=<user> /logout/user=<user>
  32. 32. Introduction to OSSec  Open Source  Intrusion Detection System  log analysis  file integrity checking  rootkit detection  Grabs data from most of our systems  Software  OS  Inject in ElasticSearch via logstash
  33. 33. Auditing the security
  34. 34. Take-away Intellectual doggy bag
  35. 35. Conclusion  A lot  Of open source products  Of scaling potential  A reasonable amount  Of Alfresco customisation  A little  Changed from the Out Of The Box Alfresco UI

×