Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Improving Performance and Flexibility
of Content Listings Using Criteria API
Nils Breunese
Public Broadcaster since 1926 The Netherlands
Online since 1994 Open-source CMS released in 1997
Using Magnolia since 2010 Still migrating websites
Tens of thousands of pages Multiple sites like that
Overview pages Lots of them
Thanks for the warning… Even 10 seconds would be way too long
WARN info.magnolia.module.cache.filter.CacheFilter
-- The fo...
Overview models Standard Templating Kit
Tracking back from the template newsOverview.ftl
(...)
[#assign pager = model.pager]

[#assign newsList = cmsfn.asContentM...
Constructing the pager AbstractItemListModel
public STKPager getPager() throws RepositoryException {

(...)

return new ST...
Four step pipeline AbstractItemListModel
public Collection<Node> getItems() throws RepositoryException {

List<Node> items...
Step 1a: Constructing the query TemplateCategoryUtil
public static List<Node> getContentListByTemplateNames(...) {
(...)
S...
Step 1b: Executing the query TemplateCategoryUtil
public static List<Node> getContentListByTemplateNames(...) {
(...)

Nod...
Step 2: Filtering the item list STKDateContentUtil
public static void filterDateContentList(...) {

CollectionUtils.filter...
Step 3: Time to sort STKDateContentUtil
public static void sortDateContentList(...) {

Collections.sort(itemsList, new Com...
Step 4: Shrinking the list STKTemplatingFunctions
public List<Node> cutList(List<Node> itemsList, final int maxResults) {
...
Step 5: Get the items from the pager STKPager
public Collection getPageItems() {



Collection subList = items;

int offse...
When this becomes a problem We have multiple sites like this
select * from nt:base
where jcr:path like '/siteX/news/%' AND...
A query could do all steps at once JCR queries are pretty flexible
Everything in a single JCR query Only 20 nodes returned
SELECT * FROM nt:base
WHERE jcr:path LIKE '/siteX/news/%' AND
mgnl...
Criteria API For those familiar with Hibernate/JPA
Criteria criteria = JCRCriteriaFactory.createCriteria()

.setBasePath("...
Criteria API for Magnolia CMS Magnolia module by Openmind
jcr-criteria https://github.com/vpro/jcr-criteria
Custom pager Only a single page worth of items
public class VtkPager<T> extends STKPager {


private final List<? extends ...
Use it in your model classes VtkContentListModel (vpro)
public abstract class VtkContentListModel ... {



protected final...
Concrete Example VtkNewsOverviewModel (vpro)
@Override

protected VtkPager<Node> createPager() {
(...)

AdvancedResult res...
Still this. Was it all for nothing? :o(
WARN info.magnolia.module.cache.filter.CacheFilter
-- The following URL took longe...
Example VtkNewsOverviewModel (vpro)
@Override

protected VtkPager<Node> createPager() {
(...)

AdvancedResult result = JCR...
AdvancedResultImpl (jcr-criteria)
@Override

public int getTotalSize() {

if (totalResults == null) {

int queryTotalSize ...
jackrabbit-core 2.8.0
protected void getResults(long size) throws RepositoryException {

(...)

result = executeQuery(maxR...
It used to be fast! https://issues.apache.org/jira/browse/JCR-3858
jackrabbit-core 2.10.0+
protected void getResults(long size) throws RepositoryException {
(...)
if (sizeEstimate) {

numRe...
Enable Jackrabbit’s 'sizeEstimate' Jackrabbit 2.10+
<SearchIndex class="org.apache.jackrabbit.core.query.lucene.SearchInde...
Rendering times down to 1-2 seconds Bingo
Time for questions
Anyone?
Feel free to contact me
Nils Breunese

@breun

n.breunese@vpro.nl
Upcoming SlideShare
Loading in …5
×

Improving Performance and Flexibility of Content Listings Using Criteria API

356 views

Published on

This presentation was given at Magnolia Conference Basel, June 7-9 2016.

https://youtu.be/AVNY0bxQxK4?list=PLxHBbwVVoCoaJ49u_q7IUTvzlTdQ1Uzpt

Published in: Software
  • Be the first to comment

  • Be the first to like this

Improving Performance and Flexibility of Content Listings Using Criteria API

  1. 1. Improving Performance and Flexibility of Content Listings Using Criteria API Nils Breunese
  2. 2. Public Broadcaster since 1926 The Netherlands
  3. 3. Online since 1994 Open-source CMS released in 1997
  4. 4. Using Magnolia since 2010 Still migrating websites
  5. 5. Tens of thousands of pages Multiple sites like that
  6. 6. Overview pages Lots of them
  7. 7. Thanks for the warning… Even 10 seconds would be way too long WARN info.magnolia.module.cache.filter.CacheFilter -- The following URL took longer than 10 seconds (63969 ms) to render. This might cause timeout exceptions on other requests to the same URI.
  8. 8. Overview models Standard Templating Kit
  9. 9. Tracking back from the template newsOverview.ftl (...) [#assign pager = model.pager]
 [#assign newsList = cmsfn.asContentMapList(pager.pageItems)!] (...)
  10. 10. Constructing the pager AbstractItemListModel public STKPager getPager() throws RepositoryException {
 (...)
 return new STKPager(currentPageLink, getItems(), content);
 }
  11. 11. Four step pipeline AbstractItemListModel public Collection<Node> getItems() throws RepositoryException {
 List<Node> itemsList = search();
 
 this.filter(itemsList);
 this.sort(itemsList);
 itemsList = this.shrink(itemsList);
 
 return itemsList;
 } 1 2 3 4
  12. 12. Step 1a: Constructing the query TemplateCategoryUtil public static List<Node> getContentListByTemplateNames(...) { (...) StringBuffer sql = new StringBuffer( "select * from nt:base where jcr:path like '" + path + "/%'"); (...add 'mgnl:template=' clauses...) (...add 'ORDER BY' clauses...)
 return getWrappedNodesFromQuery(sql.toString(), repository, maxResultSize); } maxResultSize == Integer.MAX_VALUE
  13. 13. Step 1b: Executing the query TemplateCategoryUtil public static List<Node> getContentListByTemplateNames(...) { (...)
 NodeIterator items = QueryUtil.search( repository, sql.toString(), Query.SQL, NodeTypes.Content.NAME); }
  14. 14. Step 2: Filtering the item list STKDateContentUtil public static void filterDateContentList(...) {
 CollectionUtils.filter(itemsList, new Predicate() {
 @Override
 public boolean evaluate(Object object) {
 (...) return date.after(minDate) && date.before(maxDate);
 }
 });
 }
  15. 15. Step 3: Time to sort STKDateContentUtil public static void sortDateContentList(...) {
 Collections.sort(itemsList, new Comparator<Node>() {
 @Override
 public int compare(Node c1, Node c2) {
 (...)
 if (StringUtils.equals(sortDirection, ASCENDING)) {
 return date2.compareTo(date1);
 }
 return date1.compareTo(date2);
 }
 });
 }
  16. 16. Step 4: Shrinking the list STKTemplatingFunctions public List<Node> cutList(List<Node> itemsList, final int maxResults) {
 if (itemsList.size() > maxResults) {
 return itemsList.subList(0, maxResults);
 }
 return itemsList;
 } NewsOverviewModel passes Integer.MAX_VALUE, so shrink does effectively nothing in this case
  17. 17. Step 5: Get the items from the pager STKPager public Collection getPageItems() {
 
 Collection subList = items;
 int offset = getOffset();
 if (count > 0) {
 int limit = maxResultsPerPage + offset;
 if (items.size() < limit) {
 limit = count;
 }
 subList = ((List) items).subList(offset, limit);
 
 }
 return subList;
 } maxResultsPerPage is typically something like 20
  18. 18. When this becomes a problem We have multiple sites like this select * from nt:base where jcr:path like '/siteX/news/%' AND mgnl:template = 'standard-templating-kit:pages/stkNews' 20000 pages under website:/siteX/news Four step pipeline returns STKPager with 20000 items (page nodes) [#assign model.pager] [#assign newsList = cmsfn.asContentMapList(pager.pageItems)!] STKPager returns list with 20 page nodes 19980 Node objects created, but not rendered
  19. 19. A query could do all steps at once JCR queries are pretty flexible
  20. 20. Everything in a single JCR query Only 20 nodes returned SELECT * FROM nt:base WHERE jcr:path LIKE '/siteX/news/%' AND mgnl:template = 'standard-templating-kit:pages/stkNews' AND jcr:created < cast('2016-06-07T00:00:00.000Z' AS DATE) ORDER BY date ASCENDING LIMIT 20 OFFSET 20 Search Filter Sort Paging
  21. 21. Criteria API For those familiar with Hibernate/JPA Criteria criteria = JCRCriteriaFactory.createCriteria()
 .setBasePath("/siteX/news")
 .add(Restrictions.eq( "@mgnl:template", "standard-templating-kit:pages/stkNews"))
 .add(Restrictions.betweenDates("@jcr:created", minDate, maxDate))
 .addOrder(Order.asc("date"))
 .setPaging(20, 1); 
 ResultIterator<...> items = criteria.execute(session).getItems(); Sort Paging Filter Search
  22. 22. Criteria API for Magnolia CMS Magnolia module by Openmind
  23. 23. jcr-criteria https://github.com/vpro/jcr-criteria
  24. 24. Custom pager Only a single page worth of items public class VtkPager<T> extends STKPager { 
 private final List<? extends T> items; private final int pageSize;
 private final int count;
 
 (...)
 
 @Override
 public List<? extends T> getPageItems() {
 return items;
 } }
  25. 25. Use it in your model classes VtkContentListModel (vpro) public abstract class VtkContentListModel ... {
 
 protected final VtkPager<ContentMap> pager;
 
 @Override
 public String execute() {
 pager = createPager();
 return super.execute();
 }
 
 protected abstract VtkPager<T> createPager(); (...) }
  26. 26. Concrete Example VtkNewsOverviewModel (vpro) @Override
 protected VtkPager<Node> createPager() { (...)
 AdvancedResult result = JCRCriteriaFactory.createCriteria()
 .setBasePath(path)
 .add(Restrictions.in("@mgnl:template", templates))
 .add(Restrictions.betweenDates("@jcr:created", minDate, maxDate)) .addOrder(Order.asc("date")) .setPaging(itemsPerPage, pageNumberStartingFromOne) .execute(session);
 List<Node> items = new ArrayList<>();
 for (AdvancedResultItem item : result.getItems()) {
 items.add(item.getJCRNode());
 }
 
 int count = result.getTotalSize();
 
 return new VtkPager<>(link, items, content, itemsPerPage, count); }
  27. 27. Still this. Was it all for nothing? :o( WARN info.magnolia.module.cache.filter.CacheFilter -- The following URL took longer than 10 seconds (63969 ms) to render. This might cause timeout exceptions on other requests to the same URI.
  28. 28. Example VtkNewsOverviewModel (vpro) @Override
 protected VtkPager<Node> createPager() { (...)
 AdvancedResult result = JCRCriteriaFactory.createCriteria()
 .setBasePath(path)
 .add(Restrictions.in("@mgnl:template", templates))
 .add(Restrictions.betweenDates("@jcr:created", minDate, maxDate)) .addOrder(Order.asc("date")) .setPaging(itemsPerPage, pageNumberStartingFromOne) .execute(session);
 List<Node> items = new ArrayList<>();
 for (AdvancedResultItem item : result.getItems()) {
 items.add(item.getJCRNode());
 }
 
 int count = result.getTotalSize();
 
 return new VtkPager<>(link, items, content, itemsPerPage, count); } This call takes 10-60+ seconds!
  29. 29. AdvancedResultImpl (jcr-criteria) @Override
 public int getTotalSize() {
 if (totalResults == null) {
 int queryTotalSize = -1;
 try { // jcrQueryResult instanceof JackrabbitQueryResult) {
 Method m = jcrQueryResult.getClass().getMethod("getTotalSize");
 queryTotalSize = (int) m.invoke(jcrQueryResult);
 } catch (InvocationTargetException | IllegalAccessException e) {
 LOG.error(e.getMessage(), e);
 } catch (NoSuchMethodException e) {
 
 }
 if (queryTotalSize == -1 && (itemsPerPage == 0 || applyLocalPaging)) {
 try {
 totalResults = (int) jcrQueryResult.getNodes().getSize();
 } catch (RepositoryException e) {
 // ignore, the standard total size will be returned
 }
 }
 
 if (queryTotalSize == -1) {
 totalResults = queryCounter.getAsInt();
 } else {
 totalResults = queryTotalSize;
 
 }
 }
 return totalResults;
 } We end up here
  30. 30. jackrabbit-core 2.8.0 protected void getResults(long size) throws RepositoryException {
 (...)
 result = executeQuery(maxResultSize); // Lucene query (...) // Doesn’t use result.getSize(), call collectScoreNodes(...) } private void collectScoreNodes(...) {
 while (collector.size() < maxResults) {
 ScoreNode[] sn = hits.nextScoreNodes(); (...)
 // check access
 if (isAccessGranted(sn)) {
 collector.add(sn);
 } else {
 invalid++;
 }
 }
 } QueryResultImpl
  31. 31. It used to be fast! https://issues.apache.org/jira/browse/JCR-3858
  32. 32. jackrabbit-core 2.10.0+ protected void getResults(long size) throws RepositoryException { (...) if (sizeEstimate) {
 numResults = result.getSize(); // Use count from Lucene
 
 } else {
 // do things the Jackrabbit 2.8.0 way
 (...)
 } (...) } QueryResultImpl
  33. 33. Enable Jackrabbit’s 'sizeEstimate' Jackrabbit 2.10+ <SearchIndex class="org.apache.jackrabbit.core.query.lucene.SearchIndex"> (...) <param name="sizeEstimate" value="true"/> </SearchIndex>
  34. 34. Rendering times down to 1-2 seconds Bingo
  35. 35. Time for questions Anyone?
  36. 36. Feel free to contact me Nils Breunese @breun n.breunese@vpro.nl

×