Achievement and challenges in Calc Summary of my activities in 2009, and plans going forward. OpenOffice.org Developer Nov...
Agenda <ul><li>Highlight of 2009 </li><ul><li>External reference re-work
DataPilot improvements
Row limit increase to 1 million rows
Other on-going work worth mentioning </li></ul><li>Where do we go from here?
Working remotely from home </li></ul>
External Reference Re-Work
External range name support <ul><li>Feature inspired by Excel interop. (i3740)
Requires caching of referenced data. </li><ul><li>Range names.
Cell values used by range names. </li></ul><li>ScExternalRefManager was born. </li><ul><li>Caches referenced data only.
Originally for caching range names only. </li></ul></ul>
Anatomy of ScExternalRefManager
Old implementation <ul><li>Emulated cache tables with existing features </li><ul><li>Hidden sheets with special name.
Cached entire sheet including non-referenced data. </li></ul></ul>Reference a single cell... Whole sheet gets cached & sti...
External reference re-work Let's use ScExternalRefManager for both external range names and external references. <ul><li>P...
No more “is this table an external cache?” query. </li></ul><li>Saves precious table indexes (0 – 255).
Efficient storage.
Works better with .xlsx import. </li></ul></ul>
External reference re-work (cont'd) <ul><li>CONS </li><ul><li>Lots of changes necessary in many areas </li><ul><li>Formula...
ODF import/export – populating cache from, and dumping cache to, <table:table> structures.
Excel import/export – modifies the way cached data are loaded and written.  SUPBOOK, EXTERNNAME, XCT, and CRN records. </l...
Work started in mooxlsc CWS <ul><li>Developers involved </li><ul><li>Kohei Yoshida
Eike Rathke
Daniel Rentz </li></ul><li>Lots of hard work, lots of grounds to cover. (~2 months).
Integrated into DEV300_m38.  A big load off... </li></ul>
The Unexpected (!) <ul><li>External references in charts broke! </li><ul><li>Data sequence providers to be re-implemented.
External reference listeners added.
Large re-factoring after feature freeze (3000 LOC). </li></ul><li>Cell functions broke! </li><ul><li>More functions than a...
Had to be re-implemented.
Some are yet to be fixed (database functions). </li></ul></ul>
DataPilot Improvements
Field popup window Before After
Take 1 (Go-OO 3.1) <ul><li>To provide visual feedback on fields with hidden members.
Easy access to field members' visibility.
Pop-up window; not a dialog.
Bare minimum; just enough functionality. </li></ul>
Take 2 (Go-OO 3.1.1) <ul><li>To support sorting of field members via custom sort list.
Upcoming SlideShare
Loading in …5
×

Achievements

526 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
526
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Achievements

  1. 1. Achievement and challenges in Calc Summary of my activities in 2009, and plans going forward. OpenOffice.org Developer Novell, Inc. 9:00 AM – 9:45 AM November 5, 2009
  2. 2. Agenda <ul><li>Highlight of 2009 </li><ul><li>External reference re-work
  3. 3. DataPilot improvements
  4. 4. Row limit increase to 1 million rows
  5. 5. Other on-going work worth mentioning </li></ul><li>Where do we go from here?
  6. 6. Working remotely from home </li></ul>
  7. 7. External Reference Re-Work
  8. 8. External range name support <ul><li>Feature inspired by Excel interop. (i3740)
  9. 9. Requires caching of referenced data. </li><ul><li>Range names.
  10. 10. Cell values used by range names. </li></ul><li>ScExternalRefManager was born. </li><ul><li>Caches referenced data only.
  11. 11. Originally for caching range names only. </li></ul></ul>
  12. 12. Anatomy of ScExternalRefManager
  13. 13. Old implementation <ul><li>Emulated cache tables with existing features </li><ul><li>Hidden sheets with special name.
  14. 14. Cached entire sheet including non-referenced data. </li></ul></ul>Reference a single cell... Whole sheet gets cached & still a regular sheet.
  15. 15. External reference re-work Let's use ScExternalRefManager for both external range names and external references. <ul><li>PROS </li><ul><li>Cleaner implementation </li><ul><li>Single caching back-end instead of two.
  16. 16. No more “is this table an external cache?” query. </li></ul><li>Saves precious table indexes (0 – 255).
  17. 17. Efficient storage.
  18. 18. Works better with .xlsx import. </li></ul></ul>
  19. 19. External reference re-work (cont'd) <ul><li>CONS </li><ul><li>Lots of changes necessary in many areas </li><ul><li>Formula parser – add new token types to handle external data separately.
  20. 20. ODF import/export – populating cache from, and dumping cache to, <table:table> structures.
  21. 21. Excel import/export – modifies the way cached data are loaded and written. SUPBOOK, EXTERNNAME, XCT, and CRN records. </li></ul></ul></ul>
  22. 22. Work started in mooxlsc CWS <ul><li>Developers involved </li><ul><li>Kohei Yoshida
  23. 23. Eike Rathke
  24. 24. Daniel Rentz </li></ul><li>Lots of hard work, lots of grounds to cover. (~2 months).
  25. 25. Integrated into DEV300_m38. A big load off... </li></ul>
  26. 26. The Unexpected (!) <ul><li>External references in charts broke! </li><ul><li>Data sequence providers to be re-implemented.
  27. 27. External reference listeners added.
  28. 28. Large re-factoring after feature freeze (3000 LOC). </li></ul><li>Cell functions broke! </li><ul><li>More functions than anticipated.
  29. 29. Had to be re-implemented.
  30. 30. Some are yet to be fixed (database functions). </li></ul></ul>
  31. 31. DataPilot Improvements
  32. 32. Field popup window Before After
  33. 33. Take 1 (Go-OO 3.1) <ul><li>To provide visual feedback on fields with hidden members.
  34. 34. Easy access to field members' visibility.
  35. 35. Pop-up window; not a dialog.
  36. 36. Bare minimum; just enough functionality. </li></ul>
  37. 37. Take 2 (Go-OO 3.1.1) <ul><li>To support sorting of field members via custom sort list.
  38. 38. Larger font size.
  39. 39. Submenu-capable menu control, dynamically populated (to be used for autofilter later). </li></ul>
  40. 40. Take 3 (upstream build) <ul><li>Toggle All check box.
  41. 41. Single item buttons.
  42. 42. Keyboard access.
  43. 43. Accessibility support.
  44. 44. Scheduled for 3.3 inclusion.
  45. 45. In progress in koheidatapilot03 cws </li></ul>
  46. 46. Custom field names <ul><li>Assign names different from the original, simply by typing into the cell. </li><ul><li>Fields
  47. 47. Field members
  48. 48. Subtotal
  49. 49. Grand total </li></ul><li>Requires ODF file format change. ODF-Next ? </li></ul>
  50. 50. Row Limit Increase (1 million Rows)
  51. 51. Lots of users want this... <ul><li>Issue 30215, with 147 votes (!)
  52. 52. Natural progression of how people use spreadsheet app; quasi database.
  53. 53. Interop with Excel 2007. </li></ul>
  54. 54. So, what's so hard about this ? <ul><li>Performance, performance, performance! - lots of 0 to MAXROW loop all over the place.
  55. 55. Ordinary-sized documents should not be affected.
  56. 56. More rows, more data. </li></ul>Wait and see approach; let's increase the row limit, and see what happens.
  57. 57. Row flag storage <ul><li>Grouped bit-fields over row segments: hidden, filtered, page breaks (automatic and manual).
  58. 58. Separate attributes of different distribution characteristics. </li><ul><li>Hidden and filtered – range-oriented, top-skewed.
  59. 59. Automatic page breaks – single rows, distributed semi-evenly from 0 to MAXROW.
  60. 60. Manual page breaks – single rows, very few, random. </li></ul><li>::std::set for page breaks.
  61. 61. Flat segment tree for hidden and filtered attributes. </li></ul>
  62. 62. Flat segment tree
  63. 63. Flat segment tree (cont'd) <ul><li>Modified from segment tree.
  64. 64. Segments don't overlap.
  65. 65. Neighboring segments have different values.
  66. 66. A query for a key also provides range info. </li><ul><li>Shrinks the number of loops by skipping hidden or filtered rows.
  67. 67. Avoid querying for row flags for each and every row. </li></ul><li>Independent from OOo code base. </li></ul>
  68. 68. Pagenation performance <ul><li>Calculating page break positions took ~40 seconds. </li><ul><li>Problem: sub-optimal data structure for storing page break positions.
  69. 69. Solution: store page break positions in ::std::set separately.
  70. 70. Result: pagenation now ~0.1 second. </li></ul></ul>
  71. 71. Print preview performance <ul><li>Page preview was horrendously slow, with constant 100% CPU usage. </li><ul><li>Problem: pagenation was performed at every pixel movement of the mouse cursor.
  72. 72. Solution: Perform pagenation only when necessary.
  73. 73. Result: CPU usage down to ~idle during print preview. </li></ul></ul>
  74. 74. Range selection performance <ul><li>Calc became sluggish while selecting columns. </li></ul><ul><ul><li>Problem: the position of the auto-fill mark became expensive to calculate & it was done on every change of range selection mark.
  75. 75. Solution: Reduce the number of auto-fill position calculation by: </li><ul><li>calculating it only when it's visible, and
  76. 76. cache the calculated position to avoid redundant calculation. </li></ul><li>Still room for improvement. </li></ul></ul>
  77. 77. Other minor performance tweaks <ul><li>Sorting on large data. </li><ul><li>Remove redundant broadcasting.
  78. 78. Use ::std::vector instead of Table to store sorted data to avoid excessive array re-allocation. Don't forget to call reserve().
  79. 79. Sort time down from 64 seconds to 8 seconds. </li></ul><li>Selective use of (GTK's) progress bar. </li><ul><li>Set lower boundary on use of progress bar.
  80. 80. GTK's progress bar quite expensive to draw. </li></ul></ul>
  81. 81. Outstanding issues <ul><li>Other performance issues lurking somewhere (such as Issue 106249).
  82. 82. Note positioning at higher rows.
  83. 83. Maybe other stuff I haven't thought of. </li></ul>
  84. 84. Other On-going Work Worth Mentioning.
  85. 85. Colored tab (by Daniel Watson) <ul><li>Blessed by the UX.
  86. 86. Needs ODF file format change.
  87. 87. In progress in calctabcolor cws. </li></ul>
  88. 88. Text import filter enhancement
  89. 89. Copied range border <ul><li>Paste by ENTER (by Jon Pryor).
  90. 90. On-going in koheicopyborder cws. </li></ul>
  91. 91. PHONETIC function <ul><li>Original patch by Hiroyuki Ikezoe.
  92. 92. Important for Japanese users. Perhaps usable for Chinese users (Pinyin)?
  93. 93. In calcphonetic cws.
  94. 94. Long way to go? </li><ul><li>Editing UI
  95. 95. Display in cell
  96. 96. IM interaction </li></ul></ul>
  97. 97. Automatic decimal adjustment
  98. 98. Cursor positioning in selection <ul><li>Cursor stays at origin cell.
  99. 99. Improved usability for keyboard users. </li></ul>
  100. 100. Where do we go from here? <ul><li>Realistically (every day work) </li><ul><li>Feature parity with Excel.
  101. 101. Better round-trip with xls. </li></ul><li>Probably (still important) </li><ul><li>Performance </li><ul><li>Calculation – multi-threading?
  102. 102. DataPilot </li></ul></ul><li>Hopefully (personal pet peeve, ITO?) </li><ul><li>Eye candy
  103. 103. More visual feedback </li></ul></ul>
  104. 104. Working remotely from home <ul><li>European-centric development – hard on those of us in remote time zones.
  105. 105. Become your own expert – code analysis, class diagram, symbol database.
  106. 106. No face-to-face talk, limited to emails, IRC – occasional voice communication and periodic face-to-face meetings helpful. Build trust.
  107. 107. Unable to share hardware – get your own, invest wisely. </li></ul>

×