The document describes the Content Transfer Services (CTS) used at the Library of Congress to reliably transfer and track large amounts of digital content. CTS uses the BagIt specification to "bag" digital content, adding metadata files that allow verification of complete and accurate file transfers. It provides automated workflows, inventory of content and events, and reporting capabilities to manage the complex transfers of collections like the National Digital Newspaper Program. CTS addresses the challenges of preserving content as it moves across systems, organizations, and over time.

1. CTS * at LC **
Daniel Chudnov - 2010-10-15 - dchud at loc gov
Access 2010 - Winnipeg
* Content Transfer Services follow along at
** Library of Congress slideshare.net / dchud

2. slideshare.net / dchud

3. work in progress

4. transfer
verification
inventory
reporting
workflow
notification
status
access

5. hard to show
but that won’t stop me

9. tinyurl.com/cts2010

10. when i’m done
this should
make sense

11. NDNP

12. publishing
breaking news*
online
* 100 years after it happens

14. chroniclingamerica
.loc.gov

15. 1,442,264
pages
last year at Access

16. 2,692,369
pages
this year at Access

17. went live
spring 2007

18. first two years
1.4M
last year
2.7M

19. 56 TB content
117 TB in copies

20. how?

21. 1.
built a
better
access system

22. faster ingest
from 1 month to 1 day

23. 2.
workflow

24. CTS
does
this

25. months
batches
page counts

26. press event
the gap
first batch
received
2005-10
went live spring 2007

27. 2010-09
1-2 month lag
2-3 month lag
3-4 month lag 2009-09
first CTS workflow

28. ingest rate
approaches
receipt rate

29. this makes us
smile

31. content
transfer
services

32. some requirements

33. LC started
digitizing
in the 1980s

34. we have a
lot
of stuff

35. in a
lot
of places

36. distributed
computing
environment

37. commercial
MFT *
license
* Managed File Transfer

38. buy or build?
why, both, thank you

40. 100s of collections

41. dozens of
curatorial organizations

42. lots more
stuff
coming every day

43. a
long term project

44. collecting
and
making available

45. services for
transfer of
content

46. any content

47. lots of transfers
“movage”

48. several services

49. transfer
verification
inventory
reporting
workflow
notification
status
access

50. transfer across
systems
organizations
time

51. content transfer
is
risky

52. copies fail

53. bits go bad

54. drives get lost

55. you forget
what you did

56. you forget
what you had

57. people retire

58. software breaks

59. hardware breaks

60. three blizzards
in
DC

61. CTS helps
make transfers
reliable and resilient

62. reliable
know when you’ve
succeeded

63. BagIt
packing slip
for data

64. .
|-- bag-info.txt
|-- bagit.txt
|-- data
| |-- batch.xml
| |-- batch_1.xml
| |-- batch_ne_dewitt_rework
| | |-- 00206538016_batch.xml
| | |-- 00206538028_batch.xml
| | `-- sn99021999
| `-- sn99021999
|
|
|
|-- 00206538016
|
|
|-- 0000.jp2
|-- 0000.pdf
data in a Bag
| | |-- 0000.tif
| | |-- 0000.xml
| | |-- 0001.jp2
| | |-- 0001.pdf
| | |-- 0001.tif
| | |-- 0001.xml

65. .
|--
|--
bag-info.txt
bagit.txt identifies
a bag
|-- data
| |-- batch.xml
| |-- batch_1.xml
| |-- batch_ne_dewitt_rework
| | |-- 00206538016_batch.xml
| | |-- 00206538028_batch.xml
| | `-- sn99021999
| `-- sn99021999
| |-- 00206538016
| | |-- 0000.jp2
| | |-- 0000.pdf
| | |-- 0000.tif
| | |-- 0000.xml
| | |-- 0001.jp2
| | |-- 0001.pdf
| | |-- 0001.tif
| | |-- 0001.xml

66. .
where the
|-- bag-info.txt
|-- bagit.txt
|-- data
|
|
|
|-- batch.xml
|-- batch_1.xml data starts
|-- batch_ne_dewitt_rework
| | |-- 00206538016_batch.xml
| | |-- 00206538028_batch.xml
| | `-- sn99021999
| `-- sn99021999
| |-- 00206538016
| | |-- 0000.jp2
| | |-- 0000.pdf
| | |-- 0000.tif
| | |-- 0000.xml
| | |-- 0001.jp2
| | |-- 0001.pdf
| | |-- 0001.tif
| | |-- 0001.xml

67. .
|-- bag-info.txt
|-- bagit.txt
|-- data
| |-- batch.xml
| |-- batch_1.xml
| |-- batch_ne_dewitt_rework
| | |-- 00206538016_batch.xml
| | |-- 00206538028_batch.xml
| | `-- sn99021999
| `-- sn99021999
| |-- 00206538016
| | |-- 0000.jp2
| | |-- 0000.pdf
| | |-- 0000.tif
| | |-- 0000.xml
|
|
|
|
|
|
|-- 0001.jp2
|-- 0001.pdf
| ...
packing
|--
`--
manifest-md5.txt
tagmanifest-md5.txt slip

68. 71607ad119be88c842268a76f0b6b9e9 data/sn99021999/00206538107/1884091301/0621.pdf
c602d2ac07508059ce5f5597e239b97f data/sn99021999/00206538120/1885100601/0831.xml
a59795bd1584532d5cbc0b1d82f75cf8 data/sn99021999/00206538016/1880061401/0593.pdf
3c64fac7e2d49671e0d93908ae42a779 data/sn99021999/00206539616/1888101801/0905.xml
03158a560baa7479b3805d2b45ee02cd data/sn99021999/00206538028/1880111501/0405.tif
fa56ea18580e1446939ed62709e5b2db data/sn99021999/00206538077/1883061901/1145.pdf
bf4fb83ff8305e8256970a3466c1a12d data/sn99021999/00206538120/1885061501/0043.pdf
8f3649fc812de74b9d9443ee90a8ac9c data/sn99021999/00206538120/1885111101/1109.tif
e0b83a7f9ca228271fdaecf6348e1cec data/sn99021999/00206538120/1885101201/0871.xml
1c2f84e12792c123ba0aabedd0c0bbad data/sn99021999/00206538107/1884071401/0197.xml
080e557fe9f68037605e5b80df4bc4ac data/sn99021999/0020653820A/1888050701/0543.tif
532efe32c156459d9d9589caf618f502 data/sn99021999/00206538120/1885071401/0250.tif
ce607af59a96f2656d9448f38ffda072 data/sn99021999/0020653820A/1888052801/0731.pdf
60b626d8fd40aca1b425e86a004bb055 data/sn99021999/00206539628/1888111801/0088.xml
a467cd62350334c7aa83cf1e9056c1c6 data/sn99021999/00206539616/1888091701/0629.jp2
1a434f7a4d843a2c8ffe8d0824fafc3f data/sn99021999/00206538028/1880120801/0482.jp2
22996d89b4a3334256afaddcaa0238d8 data/sn99021999/00206538016/1874102001/0259.jp2
36f550da273ad4c592fee1761c98322a data/sn99021999/00206538016/1880052201/0518.jp2
7f7ccec3f2afae896338498372fd476e data/sn99021999/00206539616/1888080101/0200.pdf
c247a5d74d0e7f857c534d935661adbe data/sn99021999/00206538107/1884072601/0286.jp2
4d497a18a154adcc8636239378ab340b data/sn99021999/00206539628/1889021101/0868.pdf
2e8ca2558b54b5c49b2f20a355a60895 data/sn99021999/00206538065/1882092001/0136.xml
fb71493048e5010100f18012f5060d42 data/sn99021999/00206538028/1880123001/0569.xml
40b100432890b055a5defbfbea815d57 data/sn99021999/00206538107/1884090901/0590.xml
46f6d61480dadc1c988b0baa4de8b6c4 data/sn99021999/00206539628/1888122801/0463.pdf
1cb8af0648e8c9df395b63226fe7371f data/sn99021999/00206538016/1874101501/0244.pdf
9257834023c683b02f354888b2740b8f data/sn99021999/00206539616/1888102301/0956.xml
0d52b3b2b1c5459b7e8d500a8566b0bf data/sn99021999/00206538120/1885080801/0425.tif

69. indicates two things

70. 1
what i think
i’m sending you

71. 2
whether you
received it

72. just like
a
packing slip

73. works across
space

74. works across
systems

75. works across
orgs

76. works across
time

77. easy to make

78. md5deep

79. BIL
BagIt
Library

80. Bagger
desktop GUI

82. BIL is free software
Bagger will be soon

83. sf.net/projects/loc-xferutils/

84. see also:
BagIt
in Wikipedia
edsu++

85. reliability
through
bagging

86. resilience
through
persistence

87. verify that
copies succeed

88. know when
copies fail

89. repeat until
copies succeed

90. debug
&
diagnose

91. record all of it

92. know what you have
know what you did

93. inventory

94. BagIt checksums
in a DB

95. content properties
project, process, type

96. event timeline

97. receipt
verification
QR
copies
accept/reject
ingest/release
comments

98. life cycle of
some set
of content

99. basic
facts
project
all the copies
details

100. event timeline

101. comments along the way

102. life cycle of
NDNP
batch

103. two key things

104. 1
automated workflow
using jBPM

105. this part

106. process definition
manages the steps
doesn’t let us forget

107. 2
when content partners
call
we can answer
their questions

108. reporting
answering our
own questions

109. annual reports
very important

110. file counts
overall size
etc.

111. used to be
very difficult
to determine

112. now
immediate
anytime

114. mostly
NDNP
newer
partners

115. also project
reporting /
planning

116. NDNP batches - one awardee

117. NDNP batches - all awardees
(same data, CSV export)

118. provides
5000’ view

119. workflow

122. working status
at a glance

123. a personalized view

124. overview of a whole project

125. overview of a system
overview of a person

126. not exactly
“Facebook for bags”
but kinda

127. but wait,
there’s more

128. browse live copies

130. go right to the content

131. many benefits

132. aaaand...
a RESTy web API

133. we can build
complex workflows
with
inventory
and reporting
in CTS

134. we can build
QR/workflow/auditing
outside of CTS
with inventory
and reporting
through CTS

135. CTS:
java, spring, mysql
hibernate, velocity, tiles
jquery, jBPM, jetty

136. NDNP:
python, django,
mysql, solr, apache

137. nice clean interfaces
nice separation

138. different coders,
different styles

139. same benefits
from using CTS

140. what’s next?

141. many more
content collections

142. now:
NDNP
Web Archives
NDIIPP
Copyright Cards

143. next:
P&P
G&M
WDL
AFC
Twitter
Copyright EDeposit

144. also coming:
more simple workflows

145. “Receive and Copy”

146. fits many use cases
receive
bag/verify
copy to archival
copy to access

147. works for recon
works for new stuff

148. and,
get past typical problems
permissions
insufficient storage
failed copies

149. connection
with
high expectation

150. and, finally
a UI redesign

151. thanks!

152. BagIt - wikipedia
sf.net/projects/loc-xferutils/
hooray for protovis
@dchud - dchud at loc gov