2. The USF-I Records Collection comprised of
many firsts which required thinking outside
the box
First time in military history that the majority of
records created were digital (95%+)
Largest single transfer of electronic data from a war
zone during an ongoing military operation
Largest single collection of electronic war records in
history (so far, Afghanistan will dwarf OIF in total
size)
3. Only a small percentage of Gulf War Records
ever made it to the National Archives
The change in the Operational Command
Structure of the U.S. military had the effect of
rendering records management to an
unimportant endeavor
The post-war investigation into the issue
reported that many units actively burned their
entire collection of operational records rather
than be bothered to haul them back to their
home stations
4. As early as 2006 USCENTCOM was working
on preservation projects designed to capture
out going USCENTCOM Commanders records
These two projects resulted in what is called
the Abizaid/Franks record collection
These 2 collections contains nearly 200,000 high
level operational records from the early days of
Operation ENDURING FREEDOM and
Operation IRAQI FREEDOM
This entire dual collection was pre-accessioned
to NARA in 2011 as additional preservation
precaution
5. NARA formed War Records Group, April 2009
Joint Staff visits USCENTCOM, June 2009
Informs USCENTCOM of War Records Group creation
Begins asking how much data has been transferred to
USCENTCOM
Records Policy stated that records be transferred back
to USCENTCOM as a recommended, not a required
procedure
USCENTCOM at the time had limited authority to
demand that records actually be transferred before
mission end
All efforts in the later part of 2009 was to get USF-I to
implement a fully functional RM program consisting of
training staff, highlighting the importance of records
preservation and working with USCENTCOM on a
future transfer
6. Coordination with CENTCOM began in late Fall 2009
5 day assessment conducted in April 2010 revealed the
following
Most data was unstructured, had little or no metadata and lay
un-managed
No RM draw-down plan in place
Volume, location and size of USF-I collection unknown
Not everything about the assessment findings was
accurate since the draft report released to the press was
never staffed out for comment
The success in the assessment was the elevation of the
problem with those in theater and it was this that
paved the way to enhanced cooperation and
coordination between USCENTCOM and USF-I J6
information and records management staffs at a very
crucial time
7. The Records Inventory was in the planning
stages while the assessment was underway
Mission was fact-finding
How much data is there
What data is there
Were their gaps of records
Was the data organized
Results of the Inventory
More than 50 TB located representing records as far
back as early 2004
Little organization below main folders
8. June 2010
RM requested a technical transfer team be assembled
by the technology team
MAJ Hart assigned
July 2010
Begin procurement of 80 TB SAN
Begin working on requirements for contractors who
will work on the collection
August 2010
Drafted Concept of Operations (CONOPS)
Begin to arrange processing team
9. On August 31, President Obama declared the
combat mission in Iraq was at an end and that
Contingency Operation NEW DAWN would
replace it the following day
The result of this declaration resulted in an
cutoff of unprecedented scale
All records created by USF-I under this Contingency
Operation immediately became the property of
USCENTCOM
Transfer team left for Iraq on 1 SEP to retrieve all
data held at numerous HQs in the Baghdad region
10. Sept 2010
MAJ Hart departs for Iraq
Oct 2010
Work is completed
SAN is moved to Kuwait
Nov 2010
SAN departs Kuwait for CONUS (Continental
United States)
SAN arrives in CONUS
Work begins on hooking up SAN to CENTCOM IT
infrastructure
11.
12. SAN hooked up to network in DEC 2010
Backup solution sought
2 Backups created
1 for USCENTCOM
1 for NARA
Focus on training processing staff
Working with NARA on pre-accessioning
agreement and MOU
13. Tampa is a Hurricane zone
Even a Cat 1 Hurricane would place a large portion
of MacDill AFB under water
Decision was made to transfer an exact copy of
original un-weeded and un-organized record
content for preservation in the event of a
mission/natural disaster
Records would be sent to NARA as part of a
pre-accession agreement once the identified
records sets were completed (several of these
sets have already been sent to NARA as
additional protection)
14. Limited Staff
40-50 million records
Millions of emails
Massive amount of transitory and non-record
material
Poor organization within the primary folders
Little Records Management Foundation
inherent in the collection (Share Drives were a
mess)
Large amount of redundancy
Large amount of PII
15. From CONOPS:
OIF Taxonomy: The initial OIF Taxonomy design will be
created prior to the records being made available on the
network. The design will establish base criteria for
ensuring that the original order of the records is preserved
to the greatest extent possible. The information
taxonomy shall be created using new Record Types whose
use shall be restricted to the OIF Document Collection.
The creation of specific meta types for this collection is
required to ensure the separation of the three differing
records collections within TRIM (Abizaid/Franks,
USCENTCOM RM, OIF). This new taxonomy will ensure
that once records are transferred to NARA as part of the
pre-accession, a more accurate metadata picture will be
provided for all the records.
16. After a meeting with NARA following the
April 2011 Joint Staff Records Conference in
Tampa, a decision was arrived at how to
handle the massive amount of emails within
the collection by both NARA and
USCENTCOM.
GO/FOs were declared Permanent
O6s and below for 10 years
This hierarchical system is now being adapted
for use for the entire Federal Government via
the NARA Capstone project
17. Total Records and Information Management
Selected by USCENTCOM in 2006 as the commands
Electronic Records Management Solution
Provides capability to instantly retrieve record
material
Material can be organized in a variety of ways
Records collections can be segregated
Ensures records are marked as FINAL so they
cannot be altered
Has exporting tool so records along with
metadata can be transferred to another location
(NARA)
18. It’s all a question of ROT (Redundant, Obsolete, Transitory)
Assists in organizing unstructured content
Things we can clean up quickly without a more detailed analysis
0 byte files
Unknown file extensions
Obvious policy violations
Executable(s) in data-only directories
Personal files: vacation photos, music, mp3s
Unused files
Temporary or redundant folders
Have not been accessed/modified in X number of years
Policy based on records management standards
PII Finding capabilities
In less than a week Active Navigation cleaned 177,282 files on the
F, I and M Drives on the OIF SAN totaling over 1.18 TB worth of
useless data.
19.
20. Non-existent
While others in the DoD Community are
currently under congressional inquiries for lost
records, not one DoD civilian or other
contributor to this mission has ever received a
word of acknowledgement from anyone higher
than an O6
Records Management does not function
without leadership and this includes
supporting and awarding those “Admin” staff
who keep them out of congressional hearings
Editor's Notes
Good morning, before I begin, I first have to file a few disclaimers with all of you.
I no longer work at USCENTCOM
I am here today at SAA as private citizen on my own dime and I do not represent the government of the United States today
The presentation you are going to hear represents 4 years of my and others efforts to preserve a records set of national importance, I am here only representing myself and no other agency, individual or company.
I also wish to point out that the collection of which I speak about today only relates to records created by the Headquarters units (MNF-I/USF-I) and its supporting units and Joint Task Forces in the Iraqi theater of which CENTCOM was the oversight and responsible agent for collecting and preserving those records. Therefore, when I speak of the OIF collection, I am referring only to operational records created by those Headquarters units.
The overall topic this morning’s seminar is how today’s military archives are thinking outside the box to solve their problems, while technically USCENTCOM is not an archive, it has been a Combatant Command at war since 11 SEP 2001 and has both created and collected a massive amount of record material on the campaigns in Iraq and Afghanistan. The records created there and in the field by its subordinate HQ commands represents over a decade of information that will one day end up being one of, if not thee, definitive primary source collection on the wars in Iraq and Afghanistan.
What are some of the significant aspects of this collection?
1. It is the first time in military history that the majority of records created were digital, well over 95%
2. The transfer of USF-I Records from that headquarters to CENTCOM HQ represented the single largest transfer of electronic data from a war zone during an ongoing military operation, and I think this will stand for some time as the methodology for gathering records from Afghanistan has been improved based on lessons learned from this operation.
3. The USF-I OIF collection represents the largest single cohesive collection of electronic war records in history (so far, Afghanistan will dwarf OIF in total size)
1.There has been a lot of talk about the “Records Management Disaster” following the end of the 1990-1991 Gulf War. This lessons learned from this conflict were brought up repeatedly by many of us in the DoD records management community to get leadership to listen.
2. What was our lessons learned for this mission? The reality is the only lesson we had was what not to do. A truism, if you will, of records management, is that if there are no records, there is nothing to manage. With that maxim in mind, from the onset, the primary goal was to preserve everything possible in whatever organization or lack thereof it had imposed on it in the field. That if we had to, we would put humpty-dumpty back together again back at CENTCOM HQ. The initial priorities all revolved around getting those at the HQs in Iraq to preserve their record material.
3. I believe we were very quite successful in learning from the past.
Stewart information paper reference 20 Feb 2009, also reference from CMH Archives Branch calls the 1990 Operation Desert Storm, “A Records Disaster”
Prior to my arrival at USCENTCOM the command had been working on historical preservation which resulted in an important collection which has since been pre-accessioned to the National Archives for additional preservation. This early archival work clearly shows that even early in the OIF era CENTCOM was cognizant of the importance of preserving the Commands Records.
However, the collection was never finished as priorities shifted elsewhere, however the records were still in the system and I will mention later why this collection became very important to the OIF collection
Interest in records collection in Iraq OUTSIDE of USCENTCOM first came from the National Archives with the creation of the War Records Working Group organized by the then Assistant Archivist, Dr. Michael Kurtz and led by Michael Carlson, NARA approached the Joint Staff Records Manager Dr. James Willson-Quayle in May of 2009 in an effort to gather information on the extent of records collected from OIF.
In June of 2009 during a visit to CENTCOM, Dr. Willson-Quayle sat down with myself and Mr. Robert Bennett (Div Chief, CCJ6-Resources and Analysis Division under which Records Management tasks fell). During this meeting the Joint Staff informed us of the creation of the War Records Group and what it was focused on. We were then asked, “what have you collected?”
We informed the Joint Staff of our RM policy that suggested that records be sent back to CONUS every quarter, but that this policy was routinely ignored by MNF-I/USF-I staff. CENTCOMs oversight authority on RM for the 4 star Command did not include being able to order them to turn over their records (this did not occur until 2010). Traditional RM policy states that the records creator is the owner and responsible party of its records and until OIF came to an end CENTCOM would not have been justified in ordering USF-I to turn over its records to higher command, barring some type of serious negligence. Previous Staff Assistance Visits to MNF-I/USF-I did not find any serious violations of RM policy so they were simply allowed to carry on. They were repeatedly told however in 2009 to ratchet up their RM program with additional policy, that the RM officer needed to be at least an O4 and that organization and preservation of record material should be stressed. When 2010 came around we realized that the RM message for organization never really took hold within the Command, fortunately though, USF-I got the message on preservation of its record material.
I think a couple of things need to be mentioned on the impact of other agencies on this mission. First, NARA’s role in invigorating the start of the project cannot be ignored. The formation of the War Records team in April 2009 assisted in elevating the discourse surrounding the capture of permanent records from Iraq within a large portion of DoD community.
The follow up by Joint Staff with CENTCOM shortly thereafter also cannot be similarly ignored. The constant communication and close relationship between the Joint Staff records program and the CENTCOM program was no doubt a key ingredient to the successful operation.
In April of 2009 the Joint Staff, along with NARA and some of the Service branches travelled to Iraq to elevate the Records Management concerns to leadership there.
The 5 day assessment resulted in a unfinished and unsigned white paper which was made public last November
The problem with the RM Assessment White Paper that was released to the press in 2011 and published by propublica.org was that this report was never vetted or coordinated and was in draft form. It was never signed by anyone within the Joint Staff. With that being said lets go over some of the issues brought up by the report.
The report revealed that “most data was unstructured.” This was not technically accurate, while a large percentage (perhaps 20%) was unstructured, most of it was structured to some degree. The majority of the records were found under the hierarchical taxonomy of the USF-I HQ, J1 records were in J1, J3 records were in J3 etc. Could there have been better organization, sure. However, it is always important to remember than when you are being shot at and having mortars lobbed at you on an almost daily basis its not exactly a high priority to file your records accurately.
The report also stated that there was no RM draw down in place. As early as October 2009, USCENTCOM RM staff had begun preparing USF-I RM staff on a host of issues regarding a drawdown and was drafting a FRAGO (Fragmentary Order) to order USF-I to transfer a copy of all records out of theater as soon as OIF ended.
As for volume and location, this one is accurate, despite requests for this information we had no idea the extent of the size of the collection until May 2010, which still gave CENTCOM more than enough time to procure the needed equipment to get the records out of theater soon after OIF ended.
The success in the assessment was the elevation of the problem with those in theater and it was this that paved the way to enhanced cooperation and coordination between CENTCOM and USF-I J6 information and records management staffs at a very crucial time.
The problem early on in the OIF Records Collection mission is that we really did not know what we were dealing with.
Following a recommendation from CENTCOM RM staff prior to departure of the Joint Staff Assessment team, a proposal was made to USF-I J6 from Mr. Robert Bennett to allow a team of RM experts to conduct a systems wide Records Inventory of all record domains owned by USF-I. In May I led a team of 5 CENTCOM records staff to Iraq and we spent a week cataloging everything on the USF-Is domains.
The inventory identified everything required to take the crucial next steps in gathering IT support for actually moving the records back to CONUS, as well as providing information to relate up to the Joint Staff and the National Archives.
This mission was highly successful in gathering information and this same methodology was recently carried out in Afghanistan in preparation of drawdowns there.
Following the completion of the Records Inventory, working in conjunction with my division chief, we began to assemble a 2 pronged team. 1 team would be purely IT while the other would focus on the Operational aspects of managing the records once brought back.
The IT teams initial focus would provide assistance in identifying the correct equipment needed to transfer 50+ TB of records, then head to Iraq with the equipment and begin to copy all the records. Once complete, ship the equipment home, assist in setup of the equipment within the CENTCOM IT infrastructure, and ensure that its backed up.
The other team would focus solely on processing the records, as well as provide assistance in migrating the records onto external HDs for shipment to the National Archives. This team was comprised solely of government contractors trained by government staff on how to identify record types and would begin to segregate the temporary record material from the Permanent. Once segregated the permanent records would be pre-accessioned to the National Archives.
In June MAJ David Hart was assigned as the lead of the Technical Transfer Team under RM oversight. We worked for a couple of weeks on identifying the right SAN and had numerous communications with the MAJ in charge of USF-I RM department
The work just prior to the end of OIF was focused on the drafting of the CONOPS surrounding how the records were to be processed, more on that later.
By the time the cut-off day arrived, everything was in place, the Tech transfer team was on the way there and IT staff in theater were ready to receive them, the CONOPs was completed, negotiations with NARA were underway regarding pre-accessioning, transfers and assistance was being negotiated between myself, Dr. James Willson-Quayle from the Joint Staff and Michael Carlson from NARA.
A notice was sent out to the USF-I to ensure that Operation NEW DAWN Record material should be kept separate from OIF.
The copying of the records went without issue over a period of 3 weeks.
In fact I departed for Iraq for the second time but things were going so well with the transfer team I diverted myself to Bahrain and NAVCENT headquarters to work on the collection of hundreds of gigabytes of NAVCENT records stemming both from OIF as well as Anti-piracy records which we copied and turned the originals over to the U.S. Navy with our complements.
MAJ Hart completed his mission during the first week of October. The SAN was then taken by him to Kuwait and turned over to a special shipment company for transferring classified records.
It was this phase of the operation that led me to many stressful weeks until the SAN actually arrived in Tampa. I had this recurring daymare of the SAN falling off the plane or being left out in the rain on the tarmac. When the SAN finally arrived and MAJ Hart fired it up for the first time I began to compare my inventory notes with records on the SAN and I can remember feeling this huge burden being lifted.
The slide you are seeing here represents the 52 TB collected in Iraq in the Fall of 2010. It does not represent ALL records from Iraq. For example, the records from Operation New Dawn, or the Top Secret/SCI level records were collected and preserved under a different mission and methodology.
Each of these different lines represents a different Server domain which the SAN was plugged into and the records copied. You can see here that a few of these collections have been completed and copies of the records sent to the National Archives.
Additionally, each domain was looked upon as different records set and each of the sets were considered an independent collection, but all under the OIF Collection umbrella.
If you have any questions about this slide I will be glad to answer them after the presentation.
Every records project of this magnitude has one of those “what if” or “I should have done” moments. As I look back now I always second guess myself on this one.
One of the few things I would have done differently during the mission occurred at this juncture and it had to do with the backup tapes. After we procured the SAN I simply assumed that J6 had backup tapes. I was not aware that the Command was moving to a tapeless backup system and no tapes were available when the SAN arrived. So we needed to procure hundreds of backup tapes which delayed the processing by at least 2 months. However, there was a very desirous side effect which resulted due to the delay.
The newly arrived OIF Staff arrived with varying degrees of background and all of them required advanced training in records management and more importantly records identification techniques. They also needed training on TRIM. The added training time I believe is what contributed to the 99%+ accuracy in records identification in the first month of processing.
Here is where the Abizaid/Franks collection, I previously mentioned, became extremely important. Since there was about 100,000+ records left unprocessed I used this collection as a training guide for the OIF Records staff. The intervening months from the time the OIF Processing staff arrived at CENTCOM till the OIF SAN was added to the network, went through IA, and was backed up, the processors were completing the unfinished 2006 project and learning how to archivly identify and process record material quickly and efficiently.
Looking back now, I think the only time I did not think about this mission on a daily basis was the few days directly following the Packers winning the Super Bowl in 2011, other than that it was pretty much my entire life for my 4 years at CENTCOM.
One of the issues that concerns any archivist or Records Manager is the preservation of their collection.
Due to the historical importance of this collection, combined with the location of CENTCOM in Tampa, FL the preservation issue was dealt with quickly and effectively. The SAN arrived in Tampa just after the end of the 2010 Hurricane season. By the time the next Hurricane season began in 2011 a full copy of the OIF Dataset was sent to NARA as an added precaution.
Again, the inter-agency cooperation between CENTCOM and NARA were really paying dividends for both parties.
This is where the real fun began. While the project up to this point had its minor issues they did not really rise to the challenges presented to us at this phase.
First, “how do we arrange and process over 40+ million records with a staff of 3?” I knew from the onset once we reached this phase staff limitations would prove the ultimate challenge. In an era of growing government austerity I actually was very pleased to get what I got, but that did not solve the problem of how I get 50 TB of records reviewed, organized and eventually sent to the National Archives.
Next, how do we deal with the millions of Emails in the collection? What about the terabytes of, well, how can you put it otherwise, junk. Those non-record and transitory records that exist on just about every Shared Drive ever stood-up? How can that quickly be weeded out so the processors only spend their time organizing what really matters.
And once organized how you can be sure that two processors, working independently, did not process the same electronic material?
And finally, what about material that should not have been there to begin with?
During the months between when the SAN was shipped back to CONUS and was setup on the CENTCOM network early the following year, many steps were taken to mitigate these challenges. The first was to establish a Taxonomy and utilize our Electronic Records Management Application TRIM, to its fullest extent.
As you can see here in this slide, which is taken directly from the CONOPS I drafted early on in the project was that original order must be preserved at all costs. By keeping the original order of the record material a great deal of time was saved in having to reinvent the taxonomy wheel. This decision has allowed USCENTCOM by June of this year to have processed over 9.5 million records or an average of 375,000 records a month and 125,000 records per staff member.
Along with the taxonomy came the training element. In the first months of 2011 NARA and CENTCOM signed an Memorandum of Agreement. The MOU I drafted with NARA included a clause that NARA would provide a SME to CENTCOM for a period of 1 year, with additional time to be added later if needed. This allowed CENTCOM to get the services of Mrs. Donna Read to assist me with training the contract staff and then assisting with the review processes to ensure accuracy.
How do you processes millions of emails?
In April 2011 during a conference between myself, John Powell, Bob Spangler and Donna Read of NARA we discussed at length how emails should be processed. The issue at hand was that with only a staff of 3 there was simply no way possible we could sift through millions of emails to determine which were permanent and which were not.
The arrived decision was to utilize a top to bottom approach based on the hierarchy of the staff. ALL General and Admiral emails would be considered permanent along with a large portion of O6 staff who held important positions, such as JOC Floor Leader, Deputy Assistant to the Chief of Staff, etc.
The other emails would be preserved for a period of time determined by rank and position from between 3-10 years. Because of this methodology ALL emails from the OIF collection have been completed and preserved.
Processing massive electronic records collections will be the new challenge for future archivists and records managers. Because of this, it is key that we begin to look at a host or combination of existing technologies.
My belief is that today’s ERMA’s, like TRIM, in combination with the developing Document Analytical tools, like Active Navigation, in the market place make the most effective strategy. By themselves they are fine tools, but together they can solve the problems of the massive digital collection.
I am not going to spend time talking about TRIM, since ERMAs are such a well known product type and have been used by records managers and even archives for years now.
I do want to spend just a few minutes on Document Analytical tools and Document Analytics in general. I believe the future is now here when it comes to these types of tools. For years we in the records business have heard of them being a game changer. I was not a believer and I was well-founded in that belief for a long time as the technology was simply not there. I have since changed my mind. And if appropriately combined can help save archives, government agencies, businesses, universities, and private companies both large and small millions of dollars in unneeded storage and back-up infrastructure.
We knew early on in the OIF project that the processing of the records was going to be the most time consuming so we needed to find tools that would help assist us in speeding up the weeding process.
One of these tools was Active Navigation
During an November 2010 ARMA Conference I had spoken with HP Staff there and they talked about a company called Active Navigation that could potentially help solve our 40+ million record problem. When Donna Read came on board the project in early February she also brought up Active Navigation and a pilot test underway at NARA. After further investigation we began to look into how this type of product could help us, in not only the OIF project, but could be used for the future Afghan project and improve records management as a whole throughout the Command.
In late April 2011 CENTCOM met with representatives from Active Navigation to learn about the product and over the course of several months determined that the tool would be required to sift though the massive collection.
Its capabilities on finding redundant or near redundant records helped us considerably once we moved away from the easily identifiable records. The same is true for the massive amount of empty folders and zero byte files.
Active Navigation also allowed us to find non-record material quickly and easily. The OIF collection held thousands of file extensions that simply could never be considered records. These were quickly removed or isolated from the rest of the collection.
While I am no longer at USCENTCOM I truly look forward to hearing about the ongoing work being done there utilizing Document Analytics. I believe it is truly ground-breaking in the DoD Community.
This slide represents the work conducted since the OIF Project began.
Lastly, I would like to make a final comment about acknowledgement of service.
CENTCOM is still actively working on the OIF Project and is now also focused on preserving 150 Tb coming from Afghanistan in the months and years ahead.
I cannot stress enough to the DoD civilian and military leadership to actively acknowledge their work, both past, present and future, on this endeavor.
Thank you very much.