Open Source Automated Documentation in a Development Environment
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Open Source Automated Documentation in a Development Environment

on

  • 1,010 views

A number of case studies ranging from simple to complex solutions to problems of documentation within development groups.

A number of case studies ranging from simple to complex solutions to problems of documentation within development groups.

Statistics

Views

Total Views
1,010
Views on SlideShare
1,010
Embed Views
0

Actions

Likes
1
Downloads
4
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Open Source Automated Documentation in a Development Environment Presentation Transcript

  • 1. Using Open Source Toolsfor Document AutomationNeale MorisonMay 2012
  • 2. Who Am I? • Various jobs • Technical writer • Journalist • Editor • Web developer • Software developer • Electronic technician • Various employers • Startups – hardware and software • IBM, Cisco, Microsoft, Fujitsu… • Today, CSIRO: Commonwealth Scientific and Industrial Research OrganisationCSIRO. Using Open Source Tools for Document Automation 2
  • 3. What is ASKAP?• Australian SKA Precursor • CSIRO radio telescope 36 antennas in Western Australia • World’s fastest survey radio telescope at 0.7-1.8 GHz • The system will process 2Tb/s from each antenna • Processed data then • What will it do? shipped via fibre to a Perth • Discover millions of new galaxies supercomputer - exascale • Investigate dark matter, dark energy • Similar issues to LHC (Large Hadron Collider) CSIRO. Using Open Source Tools for Document Automation 3
  • 4. What is ASKAP?CSIRO. Using Open Source Tools for Document Automation 4
  • 5. Who Are You?• Technical Writers• Experience with: • Web development • Software development • Hardware development • System administration • HTML • XML • JavaScript • Python CSIRO. Using Open Source Tools for Document Automation 5
  • 6. What am I Talking About? • Automation Any kind of software or system that assists in technical communication • Open Source Tools Broadly defined: • Software: with an Open Source License • Standards: non-proprietary or open e.g. PDF, open since June 2008 • Languages: Python, JavaScript (ECMAscript), HTML, XML, • Apps: Apache HTTP server, MySQL • Free Tools Free (as in beer) apps – e.g. web browsersCSIRO. Using Open Source Tools for Document Automation 6
  • 7. Prologue - Fairlight 1985 • Fairlight CMI (Computer Musical Instrument) Series III • Manual manual literally cut and paste • Index created automatically • Simple algorithmsample index.txt: sample output:using code 32 a blog, writing 128making an index 42 an index, making 42writing a blog 128 blog, writing a 128 code, using 32 index, making an 42 making an index 42 using code 32 writing a blog 128 CSIRO. Using Open Source Tools for Document Automation 7
  • 8. CSIRO ASKAP Antenna IDs – Problem• Two different identification systems • Manufacturer: CETC54 ID • ASKAP pad ID • Tests refer to both• Almost insoluble • Which one to use? • Why not both? • Create a table showing the correspondence CSIRO. Using Open Source Tools for Document Automation 8
  • 9. CSIRO ASKAP Antenna IDs• Requirements • Show ID correspondence • Sort by either ID • Link to test directories • Show other useful information • Portable – all contained in a single small file • Easy to update – antenna data keeps changing CSIRO. Using Open Source Tools for Document Automation 9
  • 10. CSIRO ASKAP Antenna IDs - Solution• Dynamic Document • Fixed (static) HTML: more maintenance • Better to write JavaScript that writes the HTML for you • You have to maintain only the data:tableRows = [{cetc54:1, askap:29, testDir:ASKAP29_C01, FAT:Y, SAT:Y, MRO:Y,},{cetc54:2, askap:3, testDir:ASKAP03_C02, FAT:Y, SAT:Y, MRO:Y,},…] • Responds to user requirements • Smaller than static HTML • ASKAP Antenna Table • Filtered ASKAP Antenna TableCSIRO. Using Open Source Tools for Document Automation 10
  • 11. CSIRO ASKAP Antenna IDs - Solution• Dynamic Document • Fixed (static) HTML: more maintenance • Better to write JavaScript that writes the HTML for you • You have to maintain only the data:tableRows = [{cetc54:1, askap:29, testDir:ASKAP29_C01, FAT:Y, SAT:Y, MRO:Y,},{cetc54:2, askap:3, testDir:ASKAP03_C02, FAT:Y, SAT:Y, MRO:Y,},…] • Responds to user requirements • Smaller than static HTML • ASKAP Antenna Table • Filtered ASKAP Antenna TableCSIRO. Using Open Source Tools for Document Automation 11
  • 12. Principles of Open Information • Open Store information in an open format • Terse Provide only the necessary information • DRY Don’t Repeat Yourself - Single Source [3] • Accessible Easy to find, easy to use • Free As in speech, love, beer – technology, IP • Re-usable • Maintainable • SimpleCSIRO. Using Open Source Tools for Document Automation 12
  • 13. Cisco Radiata Group• WiFi on a chip• Hardware startup exploiting: • wireless invention by John O’Sullivan and CSIRO radiophysics team • analog IC design innovations• Sold to Cisco in 1999 for unheard of sum (around $400 million) CSIRO. Using Open Source Tools for Document Automation 13
  • 14. Cisco Radiata Register Docs – Problem• Not really a problem – an opportunity• Developers used a single source register description in a Python script: • generate XML register description • generate Verilog code from which IC is built• Required – a document generated from XMLCSIRO. Using Open Source Tools for Document Automation 14
  • 15. Cisco Radiata Register Docs - Solution• XML can be reformatted using XSLT and XSL/FO (Extensible Stylesheet Language / Formatting Objects)• Processed with Java tools, e.g. Saxon, Xalan• Various outputs: • HTML • Postscript -> PDF • LaTeX • FrameMaker - MIF• Result: a 100 page register reference manual in seconds• Register document example CSIRO. Using Open Source Tools for Document Automation 15
  • 16. Cisco Radiata Register Docs - Principles• Open XML• Terse Describes only the registers• DRY One XML file stores all information• Accessible one pdf• Free common, available technology: • Adobe Reader, Web browsers • XML / XSLT • HTML• Re-usable use for next chip• Maintainable change only if XML format changes• Simple for the users CSIRO. Using Open Source Tools for Document Automation 16
  • 17. Cisco Radiata Test Data – Problem• Tests run on chip simulation• Result: large quantity of numerical data• Difficult to determine how data fits specification• Provided visualisation facilities do not meet requirementCSIRO. Using Open Source Tools for Document Automation 17
  • 18. Cisco Radiata Test Data - Solution• Process the data to produce a customised visualisation• Tools used: • Perl • Lisp-like script • Gnuplot• Many tables, plots, in HTML document• Test data visualisation example CSIRO. Using Open Source Tools for Document Automation 18
  • 19. Cisco Radiata Test Data - Principles• Open Proprietary, but documented• Terse No repeated data• DRY Everything generated from single set of data• Accessible HTML presentation• Free common, available technology: • Web browsers • Perl, HTML • GnuPlot• Re-usable use for next tests• Maintainable• Simple for the users CSIRO. Using Open Source Tools for Document Automation 19
  • 20. VaST XML Reports• Software models of embedded systems, system-on-chip (SoC)• Used by developers of cell phones, automotive systems, handheld devices to model the chips they were designing• Low-level, cycle-accurate model that ran at speeds comparable to the chip, sometimes faster• Huge savings in time and money – chip developers could write the firmware and software for the chip before fabrication• Integrated Development Environment (IDE) ran on Windows, now also Linux.• Acquired by Synopsys in 2010 CSIRO. Using Open Source Tools for Document Automation 20
  • 21. VaST XML Reports – Problem• Models shown as a hierarchy obscures connections• Item details hidden multiple clicks, right clicks required• No way to print full details or to view as a wholeCSIRO. Using Open Source Tools for Document Automation
  • 22. VaST XML Reports - Solution• Data stored in XML• XML can be: • transformed using XSLT • output as HTML in a browser• With a single button click, a full model report can be instantly generated: PMX report example CSIRO. Using Open Source Tools for Document Automation
  • 23. VaST XML Reports – Good News,, Bad News• Tricky to get it to work - XLST• Co-operation required from GUI developers• You only have to do it once• Works when you’re not around, for the customer’s models• Maintenance: infrequent when model storage format changes• Value permanently added CSIRO. Using Open Source Tools for Document Automation 23
  • 24. VaST XML Reports - Principles• Open XML• Terse Describes only the model• DRY One XML file stores all information• Accessible one click• Free common, available technology: • Web browsers • XML / XSLT • HTML• Re-usable• Maintainable• Simple for the users; for the developer, not so much CSIRO. Using Open Source Tools for Document Automation
  • 25. CSIRO ASKAP Repositories – Problem• SharePoint • Corporate choice• Subversion • Developer choice• Other repositories in use• Don’t want to discourage or confuse• Don’t want to introduce yet another system CSIRO. Using Open Source Tools for Document Automation 25
  • 26. CSIRO ASKAP Repositories - Solution• Document Database • Python CGI SQL app • Scans repositories • Can enter manually • Captures metadata • Single central site • Searchable • Links to issue tracker • ASKAP Document Database CSIRO. Using Open Source Tools for Document Automation 26
  • 27. CSIRO ASKAP Repositories - Principles• Open SQL• Terse normal form db• DRY Single SQL db• Accessible standard web presentation, form fill• Free • Web browsers • Python • SQL• Re-usable – other databases use same code• Maintainable – some issues - complex to scan repositories. Code is documented.• Simple – for users; but can I pass this on to another maintainer? CSIRO. Using Open Source Tools for Document Automation 27
  • 28. CSIRO ASKAP Document Management – Problem• Want to track: ? • Review History Tracking • Approval ? ? • History Document Repositories Database• Tie in to repositories• Tie in to document database ?• Complex and costly to program into database - violates simplicity rule CSIRO. Using Open Source Tools for Document Automation 28
  • 29. CSIRO ASKAP Document Management – Solution• Use issue tracking system Redmine, Bugzilla… Redmine Issue & History Tracking• Perform review and approval in URL ID issue tracker• Capture history in issue Repositories Document Database• Link to repositories ID via document URL• Link from document database via document ID• Link back to document database via document ID• Requires procedures, training – document in wiki CSIRO. Using Open Source Tools for Document Automation 29
  • 30. CSIRO ADE Diagrams – Problem• How Engineers think:• Requirements • Hierarchy of clickable diagrams • Links from diagrams to sub-diagrams • Document for every block in diagram • Links from diagrams to documents CSIRO. Using Open Source Tools for Document Automation 30
  • 31. CSIRO ADE Diagrams – Problem• SVG No – IE, XP• HTML5 No – IE, XP• PDF Maybe, but awkward to add links clunky browser display• Graphviz Good range of outputs Gets busy quickly: A B C Not the style wanted Links only in SVG output CSIRO. Using Open Source Tools for Document Automation 31
  • 32. CSIRO ADE Diagrams – Solution A• Use hierarchy, not diagrams• Procedure: • Go through diagrams and manually transfer nodes, links to custom data format • Use Python to read data and output JSON • Use JavaScript to generate a hierarchy map from JSON • Add document names, responsibilities, metadata • Generate documents using PowerShell, COM objects  • ADE Hierarchy CSIRO. Using Open Source Tools for Document Automation 32
  • 33. CSIRO ADE Diagrams – Solution B• Do the diagrams HTML Image Map, JavaScript• Procedure: • Create diagrams as simple graphics – PPT • Save as individual images • Create image maps – free program • JavaScript code transforms simple image map • JavaScript links to the hierarchy page • ADE System Block Diagrams CSIRO. Using Open Source Tools for Document Automation 33
  • 34. CSIRO ADE Diagrams - Principles• Open Python, HTML, JavaScript Doc creation not open, but free: PowerShell• Terse Only data• DRY One source• Accessible standard web presentation• Free • Web browsers • Python • JavaScript • PowerShell• Re-usable• Maintainable• Simple CSIRO. Using Open Source Tools for Document Automation 34
  • 35. Conclusion• Coding is there to be learned • HTML, XML, JavaScript, C, Java, Perl, Python, SQL, LaTeX … • Excellent online resources• You have to learn something – either a proprietary system, or open tools• Advantages of open tools • Skill Transfer • Embody general principles • Expert and helpful community• You can solve otherwise intractable problems• Free The only cost is time CSIRO. Using Open Source Tools for Document Automation 35
  • 36. Glossary• ASKAP Australian SKA Pathfinder• CGI Common Gateway Interface• DRY Don’t Repeat Yourself• DRY Don’t Repeat Yourself• FAT Factory Acceptance Test• GNU GNU is Not Unix• HTML HyperText Markup Language• ID Identifier• Recursion see Recursion• SKA Square Kilometre Array• SAT Site Acceptance Test• SQL Structured Query Language• SVG Scaleable Vector Graphics• URL Uniform Resource Locator• XML eXtensible Markup Language• XSLT eXtensible Stylesheet Language Transformations• XSL/FO eXtensible Stylesheet Language /Formatting ObjectsCSIRO. Using Open Source Tools for Document Automation 36
  • 37. References• 1. Open Source Initiative http://www.opensource.org• 2. Fairlight Instruments Web site fairlightinstruments.com.au• 3. DRY – Don’t Repeat Yourself The Pragmatic Programmer: From Journeyman to Master by Andrew Hunt and David Thomas http://pragprog.com/the-pragmatic-programmer• 4. ASKAP• http://www.atnf.csiro.au/projects/askap/• 5. CSIRO http://www.csiro.au/• 6. CSIRO Wireless LAN patentCSIRO. Using Open Source Tools for Document Automation
  • 38. CSIRO Astronomy and Space ScienceNeale MorisonASKAP Documentation ManagerPhone: 02 9372 4726Email: neale.morison@csiro.auWeb: www.csiro.au/org/cassThank you Contact Us Phone: 1300 363 400 or +61 3 9545 2176 Email: enquiries@csiro.au Web: www.csiro.au