Converting Unstructured Docs to XML/DITA/ePub
Upcoming SlideShare
Loading in...5

Like this? Share it with your network


Converting Unstructured Docs to XML/DITA/ePub



DCL's Presentation for LavaCon 2011

DCL's Presentation for LavaCon 2011



Total Views
Views on SlideShare
Embed Views



0 Embeds 0

No embeds



Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

Converting Unstructured Docs to XML/DITA/ePub Presentation Transcript

  • 1. Converting Unstructured Docs to XML/DITA/ePubMark Gross Linda Morone
  • 2. Background of Data Conversion Laboratory 30 years of experience providing electronic document conversion services meeting the needs of technology…today & in the future • More than 1 billion pages converted to date • US Based project management team • Global capabilities • Transform legacy & future documents • From any format to any format • Specialize in complex projects • Identify redundant data for content reuse • Employ a proven automated process • Quality Assurance service is standard in all projects • Additional services include consulting, composition & transcription & translation (Confidential) 2
  • 3. Serving All Industries • Publishers • Government • Defense • Life sciences • Automotive • Aerospace • Heavy and Industrial Equipment • Financial Services • Manufacturing • Computing • Utilities • Semiconductors • Telecommunications (Confidential) 3
  • 4. Serving a Broad Client Base (Confidential) 4
  • 5. Converting Legacy Data … Is it Worth the Expense? • Comply with regulations • Match Industry standards • Meet customer expectations & needs • Support internal departments • Expand into new markets • Multi-purpose content (Confidential) 5
  • 6. Legacy Conversion: Fact or Fiction Client’s Perception Reality • Painful Process • Expertise & Planning • Complex • QC & Automation • Expensive • Guaranteed Results • Drain on Resources • Low Costs (Confidential) 6
  • 7. So … Which Format do you ChooseePUB and Rendering-Focused DTDs NLM and Publishing DTDs• Designed for e-readers & mobile • Support traditional publishing devices • Flexible open standard• Freely available • Freely available• Open standard • Human-readable format• Adaptable to – Books DITA and Module-Based DTDs – Documents • Designed for multi-purposing and – Manuals content reuse – User guides • Topic based & modular• Support for print publishing • Supports requirements is limited – Multiple variants – Multiple languages – Context independent content (Confidential) 7
  • 8. The Story with ePub and Rendering-Focused DTDs • ePub is an emerging standard used for most eReaders • Mobi is also a large player, proprietary to Amazon Kindle • ePub is an evolving standard • ePub is supported differently by different eReaders • There are no “Silver Bullets” • eBooks are publications and need care in their production • Not just novels; recent DCL survey shows 75% will be using eBooks for complex materials (Confidential) 8
  • 9. Things to Keep in Mind When Converting • Smaller screen size • Large tables may not fit • Not all Character Sets supported by all devices • MathML not currently supported (Confidential) 9
  • 10. OCR/Text Extraction Pitfalls of Text Extraction • Special Characters • Emphasis • Ligatures • Hyphens – Soft and Hard (Confidential) 10
  • 11. Handling of Objects Mid-Paragraph Converting exactly per source may lead to problems … (Confidential) 11
  • 12. Math as Images – Changing Font Size Doesn’t Change Images (Confidential) 12
  • 13. Unicode Symbols Will Adjust with the Font Size Change (Confidential) 13
  • 14. Large Tables Table as Text (searchable but cut off) Table as Image (Confidential) 14
  • 15. When Layout Matters Testing Materials Poetry (Confidential) 15
  • 16. When Layout Matters (cont’d) Letter Recipe (Confidential) 16
  • 17. Some Notes on the Kindle • Designed for reading long documents • Designed for simplicity • Has some features that others don’t • But also missing some features that others have • Therefore, need to design the conversion differently (Confidential) 17
  • 18. Glossary Definitions iPad screenshot Kindle screenshot (Confidential) 18
  • 19. Use of CSS “Float” Style iPad screenshot Kindle screenshot (Confidential) 19
  • 20. Use of Borders iPad screenshot Kindle screenshot (Confidential) 20
  • 21. Color/Spanning/Large Tables iPad screenshot Kindle screenshot (Confidential) 21
  • 22. The Story with NLM and Publishing DTDs • Well-documented public domain standard. • Well-tested on a wide variety of materials; designed for complex publishing. • Originally designed with NIH support for Scientific, Technical, and Medical (STM) publications. • Extended to be robust for many more uses; widely used in non-STM areas. • DocBook and PRISM are other standard DTD’s; each with its own strengths – all designed for “print” publications. (Confidential) 22
  • 23. Choosing the Content to Convert Which content will be auto-generated? • TOC • Index • Labels • Titles • List of Table, Figures, etc. (Confidential) 23
  • 24. Capturing Items as Multiple Formats  Math as images and MathML  Tables as images and XHTML <disp-formula id="FD1"> <mml:math id="M1" display=block> <mml:semantics> <mml:mrow> <mml:mi>L</mml:mi> <mml:mo>&#x0003D;</mml:mo> <mml:mo>&#x02211;</mml:mo> <mml:mrow> <mml:msub> <mml:mrow> <mml:mi>l</mml:mi></mml:mrow> <mml:mi>i</mml:mi></mml:msub> <mml:mo>&#x0002F;</mml:mo> <mml:mi>N</mml:mi></mml:mrow> <mml:mo>&#x0002E;</mml:mo></mml:mrow> </mml:semantics></mml:math> </disp-formula> (Confidential) 24
  • 25. Determining Data Elements Appearance Based: Content Based: • Alignment • <email> - @ • Placement • <uri> - www • Point size • <degrees> - PhD, MD, BA • Font • <fig> - Figure, Illustration, Chart, Scheme (Confidential) 25
  • 26. Granularity of Tagging: Front Matter (Confidential) 26
  • 27. Granularity of Tagging: Back Matter • Are the references Harvard or Numeric? • Is the author name last/first or first/last? • What is the placement of the year within the citation? • Is a comma or period used after the author names? (Confidential) 27
  • 28. The Story with DITA and Module-Based DTDs • Allows for modularization of your content with Topics, and easy re-use in multiple outputs • Pre-packaged & ready to use XML (almost) • Ready-to-go for techdocs (mostly) • Infrastructure included - taxonomy (DTD and schema); printing stylesheets; lots of tools • Printable with standard tools • Extensible with specializations • Further specializations for publishing, testing, and other specialized areas • Content-based • What do you when things don’t fit (Confidential) 28
  • 29. What Makes DITA Conversions Difficult “Getting there using DITA is like building with prefabricated modular components that can be quickly assembled into a suitable structure.” - Doug Henschen, • DITA is a conceptual departure from linear information – and is difficult for many to get used to • Turns the traditional book into a collection of Topics • Topics can be thought of as interchangeable parts – to be reassembled in multiple ways – to be repurposed for multiple outputs – to be reused across multiple products • …but your documents weren’t likely to have been designed to do this. (Confidential) 29
  • 30. Structuring a Book into Topics in DITA Book 3 Reference 4 Book A Book 1 Concept 1 Reference 1 Concept 4 Reference 1 DITA Content Reference 2 Task 3 Concept 3 Management System Concept 5 Reference 3 Reference 1 Task 2 Task 1 Concept 2 Concept 1 Reference 1 Task 2 Reference 5 Concept 2 Reference 2 Task 1 Book 4 Concept 3 Reference 3 Book B Concept 2 Concept 4 Reference 4 Concept 1 Task 1 Concept 5 Reference 5 Book 2 Reference 1 Concept 3 Task 1 Task 1 Concept 1 Task 2 Task 2 Concept 2 Reference 2 Reference 3 Reference 2 Concept 5 Task 3 Task 3 Task 3 Task 2 Task 1 “Getting there using DITA is like building with prefabricated modular Reference 5 components that can be quickly assembled into a suitable structure.” Concept 2 – Doug Henschen, (Confidential) 30
  • 31. Further Complications in DITA Conversions • There’s the usual conversion issues – Accuracy of the transferred text – Tables – Math – Special Characters • There’s also the structuring issues – Identifying topics – Identifying reusable content • And the people issues – Deciding what needs re-authoring – Getting used to a new “document” paradigm – Getting rugged individualists to collaborate more (Confidential) 31
  • 32. Overview of Typical DITA Technical Conversion Issues • Architectural constraints of DITA – the square pegs – Multiple steps within a single task topic – TaskProcedure authored as a table in the source – Presence of untitled tasks/topics in the source – References to page numbers (irrelevant cross-references) – Having more than two levels of steps • How your rendering system will handle XML – Figures – Steps • Other conversion considerations: – Hierarchy in Map Files – Metadata in Map Files and Topics – Index Terms – Conditional Text – Glossary Terms – Content Terms (Confidential) 32
  • 33. Square Peg 1 - Task / Procedure Authored As a Table Issue: Tasks are done as tables rather than numbered lists. If there’s no clear consistent pattern, then automated conversion keeps the tables as tables, and steps are not tagged as steps. 1 Overview In general, backup and recovery refers to the various strategies and procedures involved in protecting a system against data loss. 2 Backup strategy and A backup is a copy of key files. Files included frequency in the backup are: • A logical backup of the database 1. Key system files • Network files • Timezone 2. Configuration files … (Confidential) 33
  • 34. Square Peg 2 - Multiple Steps In A Single Task Issue: Only one set of steps is allowed in a single task topic. When a task has two sets of steps within a topic, such as for two different scenarios, only one of the scenarios can be tagged as <steps> as per the DTD. Example: Replacing an XYZ Module Use this procedure to replace an XYZ module Remove XYZ Module 1. Loosen the screws. 2. Disengage the ejectors 3. Pull the module straight out Insert Replacement XYZ Module 1. Align the module. 2. Insert the module, pressing in firmly 3. Engage the ejectors 4. Securely tighten the screws (Confidential) 34
  • 35. Square Peg 3 - Irrelevant Cross-References Issue: Conversion to DITA may make some source cross-references irrelevant. For example, assuming all empty chapter headings are dropped, a reference to a chapter is no longer valid. In these cases, a <required- cleanup> tag is inserted to flag these occurrences for clean-up. See Chapter 1, Introduction on page 2 Would be tagged as: See <required-cleanup><xref href=”chap1”> Chapter 1, Introduction</xref></required-cleanup> NOTE: Hard-keyed page numbers are typically dropped from the cross- reference string since they are no longer relevant in DITA. (Confidential) 35
  • 36. So … Maybe You Shouldn’t Bother Converting Your Content? • It seems like such a pain to go through all the old luggage in the attic. • There is always a need for some rewriting - few writers have the clairvoyance to author content with the intent that be converted in the future – might as well rewrite it all. • My writers aren’t very busy right now anyway. • It’s more fun and seems like less trouble to author anew. (Confidential) 36
  • 37. In Reality … Converting Your Content is Worth the Bother • Throwing it out and starting over is an expensive option – In DITA, rewriting at $25/page vs. converting at $3-$4/page – The hidden costs of redoing index entries, links and other features you’ve built in – The hidden cost of reviewing, reproofing, and recertifying it all • It’s usually easier to use what you have as a base, and convert over – Needs planning – Needs time • Planning for a good conversion experience – Which content will you need? – Which content is worth converting? – Which content is suitable for re-use in multiple places? – What tools are available? – How to specify the conversion to get it right? – When do you start all this planning? (Confidential) 37
  • 38. Conversion Scope Options Option 1: Convert nothing • No conversion costs 2 • Delayed ROI 1 Option 2: Convert everything cost • High conversion costs • Reduced ROI 3 Option 3: Convert ‘frequently used’ documents • Some conversion costs • Maximized ROI time (Confidential) 38
  • 39. What to Convert, and in What Order • Categorizing – Active documents in good shape – Active documents that need a lot of work – Somewhat inactive document that will likely be retired – Archival materials • Prioritizing – Documents that are most used – Documents that are customer favorites – Documents with longest product life – Start with most recent documents and go back • Identifying the process – Can be converted as is – Can be converted with some work – Needs to be rewritten – Don’t convert – just keep archival copies (Confidential) 39
  • 40. Closing Thoughts • Know the scope of what you want to accomplish – Are you trying to get eBooks quickly, or are you changing your publishing process – Are you moving everything, or will a phased approach work – Will your content work naturally with the selected DTD • Start the conversion process early – Shifts the critical path; speeds the process; reduces cleanup – Organizing early lets more of the work be done by the content owners – eases the training and change acceptance burdens – setting up collaborative teams sets the tone and allows one to “divide and conquer” • Converting legacy data is not trivial – …but faster, safer and less expensive than rewriting – Each DTD has special considerations to be taken into account – Much can be automated, but it needs planning (Confidential) 40
  • 41. Questions... & Answers Data Conversion Laboratory Mark Gross, President 61-18 190th St., 2nd Floor Fresh Meadows, NY 11365 718-307-5711 Telephone: (718) 357-8700 Linda Morone, Sr. VP of Sales & Marketing Fax: (718) 357-8776 Web: 718-307-5728 (Confidential) 41