1. Reproduced with permission from The United States Law Week, 84 U.S.L.W. 22, 7/7/15. Copyright ஽ 2015 by The
Bureau of National Affairs, Inc. (800-372-1033) http://www.bna.com
E v i d e n c e
O f fi c i a l D o c u m e n t s
What should litigators do when they need to delegate a case’s large-scale document
review? Hunton & Williams attorneys Dionne Rainey and Kevin E. Gaunt outline how the
supervising attorney can make the process run efficiently. Among other things, they sug-
gest creating review protocols, including making sure the reviewers know the codes to be
used and how to spot confidential documents. They also suggest implementing quality con-
trols.
Best Practices for Supervising a Large-Scale Electronic Document Review
BY DIONNE RAINEY AND KEVIN E. GAUNT
I
f you are a litigator, you have almost certainly spent
time reviewing documents for production in a case.
Attorneys working on a case typically have a handle
on the subject matter, the key players, and the central
documents, so the document review process is merely
an extension of the attorney’s involvement with the liti-
gation. When conducting a review yourself, there is
usually little need for a formal review protocol.
However, in some cases, it is necessary to delegate
document review to other attorneys, either because of
the sheer volume of documents to review, and the time
it will take to review them, or due to financial con-
straints. For those cases, an attorney will take on the
additional responsibility of supervising reviewers who
are not familiar with the case. There are ways to make
this process runs smoothly and efficiently, and to en-
sure that the reviewers have properly and consistently
coded the reviewed documents. The following best
practices will ensure that your review is not only defen-
sible if it is ever challenged in a discovery dispute, but
run in a cost effective manner.
Before Review
Before the document review begins, you need to do
some legwork to set yourself up for success. Here are a
few steps you should take before beginning the review.
Creation of Review Protocol
The review protocol is a comprehensive document
given to the reviewers that should provide clear and
thorough guidance up front regarding the document re-
view. If you do not provide proper instructions at the
outset, you will end up correcting or changing the in-
structions mid-stream, causing a substantial amount of
re-review and QC steps that can increase your time
Dionne Rainey is a partner with Hunton &
Williams LLP’s commercial litigation
team and the head of the firm’s information
governance & e-discovery group. Kevin Gaunt
is an associate in the commercial litigation
practice. They can be reached at dcrainey@
hunton.com and kgaunt@hunton.com.
This article presents the views of the authors
and does not necessarily reflect those of
Hunton & Williams or its clients. The informa-
tion presented is for general information and
education purposes. No legal advice is
intended to be conveyed; readers should con-
sult with legal counsel with respect to any
legal advice they require related to the subject
matter of the article.
COPYRIGHT ஽ 2015 BY THE BUREAU OF NATIONAL AFFAIRS, INC. ISSN 0148-8139
The United States
Law Week
Case Alert & Legal News™

2. frame and costs. A good review protocol typically in-
cludes the following items:
s Background – Provide the review team with a
thorough history of the case, facts and procedural
posture so that reviewers have context for the docu-
ments that they will be reviewing.
ࡗ It is helpful to give the reviewers some in-
sight into how the documents were collected and
from whom. For instance, if the parties agreed
upon the search terms that were used to collect
documents from the client, there may be a pre-
sumption that many documents are going to be re-
sponsive; on the other hand, if the collection con-
sisted of everything on a custodian’s hard drive,
then the relevance analysis may be more difficult.
ࡗ Similarly, provide the reviewers an explana-
tion of whether any sort of predictive coding or
other technology-assisted review techniques were
used to collect the review documents. The more
context you can give the reviewers early on about
the case and the documents (without overwhelm-
ing them), the better prepared they will be to
make judgment calls on coding specific docu-
ments during the review.
s Key Players – Include a detailed list of the key
players involved in the litigation. This generally
means including information about the litigants
themselves, other key parties, any employees or
document custodians whose names and e-mails the
reviewers will be seeing, and any attorneys or other
individuals who might be implicated in the privilege
review (e.g., doctors, accountants, etc.).
s Coding Instructions – Provide information re-
garding the actual coding procedures that the re-
viewers will use. The protocol should describe what
tagging fields to use for each document, as well as a
description or example of when each field or tag
should be used. In a typical review, reviewers usually
code for relevance, privilege, and confidentiality, at a
minimum. However, you should also consider coding
key documents, documents needing redactions, as
well as documents that fit under certain issues or cat-
egories.
ࡗ It is also helpful to clarify which coding
fields are mutually exclusive and which can co-
exist with other tags. For example, a document
can only be ‘‘Not Privileged’’ or ‘‘Privileged’’ – it
cannot be both. The same applies to confidential-
ity calls. Issue tags, on the other hand, will often
require the reviewers to select any and all of the
tags that apply.
s Responsive Criteria – Include guidance that will
allow the reviewers to determine the relevance of a
document. Giving the reviewers copies of each of the
opposing side’s document requests often proves un-
wieldy and impractical. It is best to summarize the
document requests or types of responsive documents
that the reviewers are looking for. (However, you
might consider including short excerpts of the lan-
guage from the document requests in the protocol to
make sure you do not inadvertently summarize the
requests more narrowly or more broadly than in-
tended.)
ࡗ If you choose to summarize the document
requests for the reviewers, it is wise to keep a
copy of the document requests on hand in the re-
view room for the supervising attorneys to con-
sult.
s Privilege Criteria – Provide guidelines for deter-
mining whether a document is privileged or confi-
dential. Relying on an attorney’s law school training
and understanding of privilege typically results in in-
accurate and inconsistent privilege calls.
ࡗ For the ‘‘Privilege’’ section, in addition to
providing guidance on the types of privilege calls
you expect to be made (attorney-client, work-
product, or others), you also need to provide guid-
ance on who the attorneys are, who the ‘‘client’’ is
(i.e., whether subsidiary or parent corporations
are included, how agents and representatives are
handled, etc.), and whether you are going to log
privileged documents that are created after the
litigation commenced.
s Confidentiality Criteria - For the ‘‘Confidential-
ity’’ section, clarify what types of documents should
be tagged ‘‘Not Confidential,’’ ‘‘Confidential,’’ and, if
applicable, ‘‘Attorneys’ Eyes Only’’ or the equivalent.
Be sure to consult the Protective Order entered into
in the case to provide guidance for these categories.
Again, if you can provide specific examples of the
types of documents you want coded for each cat-
egory, you will help ensure consistent coding over
the course of the review.
Finally, when putting together the review protocol,
spend some time thinking about how you want the re-
viewers to handle specific types of documents so they
are all handled consistently over the course of the re-
view. For example:
s You might specify that if the document con-
tains no data, image, or any type of content, or con-
tains only a logo, it should be tagged ‘‘NCB,’’ for
‘‘Non-Content Bearing.’’ Making this clear up front is
especially helpful for e-mail footers (e.g., the image
that says ‘‘Think before you print’’ or that contains
the client’s logo) and other recurring file types that
reviewers will encounter.1
s Clarify the standard for reviewing and coding
family members (‘‘family members’’ usually refers to
the scenario of a ‘‘parent’’ e-mail with its ‘‘children’’
attachments). Do you want the reviewers to code
each document on its face, or should the children
documents take the same coding as the parent? De-
ciding how you will treat family members that are
not responsive or privileged will need to be factored
into your decision.
Again, the reviewers will rely heavily on this review
protocol, so the more detailed and clear it is, the better
your work product will be.
Pre-review
Try to make time to conduct your own pre-review of
a small subset of the documents to get a feel for the
types of documents the reviewers will be facing, as well
as to ensure the coding fields make sense. This will give
you an idea of the reviewers’ work flow and help you
1
It’s important to have these coded consistently so that
when you are producing families, you know that all of those
e-mail logos either are or are not going to be produced with
their families.
2
7-7-15 COPYRIGHT ஽ 2015 BY THE BUREAU OF NATIONAL AFFAIRS, INC. LW ISSN 0148-8139

3. anticipate any issues that may arise in the first day or
two of the review.
s If there are large amounts of similar types of
documents (invoices, presentations, etc.), have ex-
amples to show reviewers during training and tell
them exactly how you want them coded.
Prepare in Advance With Vendor or Database
Manager
Regardless of whether your firm is using an outside
vendor or an in-house group to host the data and pro-
vide the review platform, you will have a database or
project manager (often referred to as a ‘‘DM’’ or ‘‘PM’’)
to work with who will set up the review platform. The
DM can create the issue tags and fields, create review
batches, help run searches, and assist with performing
quality control checks. It is wise to spend time with the
DM before the reviewers begin so that you can get ev-
erything configured properly. When working to set up
your review, consider the following:
s Ask to have a dedicated database manager for
the length of your review. In an ideal world, you
should be able to IM or e-mail the database manager
with quick questions, but at a minimum, you want a
direct line to get in touch with the DM at any time.
s Lock the reviewers’ platform views so that they
do not inadvertently turn off key user-interface fea-
tures of the platform (i.e., accidentally closing the
coding pane, mistakenly checking out multiple re-
view batches at once, etc.).
s Train all supervising attorneys on the use of the
tool and provide them with administrator creden-
tials. It is imperative that the supervising attorneys
stay involved in the review and guide the reviewers
throughout the process.
ࡗ As a practice tip, have the DM set up a ‘‘Sec-
ond Level Review Complete’’ issue tag in the re-
view pane for the supervising attorneys to use.
This way, every document that supervisors check
during the review will be marked for tracking pur-
poses. This will avoid doing duplicative work later
when conducting your second-level reviews be-
cause you can exclude those documents that the
supervisors have already reviewed.
s Train the review team on how and when to
choose between native, image, or text views in the
review platform. For instance, it is better to view
spreadsheets in native (so that all hidden rows and
worksheets are visible) whereas e-mails and word
documents are viewed better in text (where the for-
matting is clear and legible).
s Do not allow review batches to be marked com-
plete and checked back in unless all review criteria
are met.
s Make use of the highlighting feature that some
review platforms call ‘‘persistent highlighting.’’ The
highlighting tool allows you to define and use param-
eters to make certain defined words or terms always
appear highlighted during the document review,
such as attorney names or key terms. For example,
set all of your ‘‘potentially privileged’’ search terms
to be permanently highlighted. Seeing these terms
highlighted is an automatic cue to the review team to
be on alert that the document may be privileged.
s If the vendor or the review platform has the
ability to group similar document types, such as re-
curring monthly reports or accounts receivable
spreadsheets, consider batching those documents so
that the same reviewers are assigned batches con-
taining similar types of documents. This way, similar
documents are assigned to the same one or two re-
viewers, to ensure coding consistently.
ࡗ The same goes for e-mail threads—if the re-
view platform can group related e-mails, even if
not identical, you can ensure consistent coding by
having related e-mail threads all assigned to the
same review batch.
During Review
Assuming that your reviewers are all gathered at a
common location to conduct the review (whether at a
vendor or at your office), then you and other attorneys
at your firm should take turns supervising the docu-
ment review each day. It is important to have a case at-
torney supervisor on site to answer questions and pro-
vide guidance at all times.
Particularly when multiple supervising attorneys are
involved, it is important to establish a process to ensure
that the reviewers are being given consistent guidance
and answers to their questions each day. For instance,
a supervising attorney may see fit to announce to the re-
viewers that a certain type of invoice should always be
coded as ‘‘Relevant,’’ ‘‘Confidential,’’ and ‘‘Not Privi-
leged.’’ Each of the other supervising attorneys need to
be aware of that guidance to avoid giving conflicting
guidance later in the review. Below are a few tips to
help you ensure that everyone—reviewers and supervi-
sors alike—stays on the same page.
Daily Consistency
It is helpful to keep a running log of common ques-
tions and answers, major announcements, and guid-
ance on specific documents to pass along to supervising
attorneys. Whether it is a summary e-mail at the end of
the day or a shared Evernote page that all supervisors
can access, logging each of the days’ key announce-
ments and guidance will ensure consistency during the
course of the review.
In a similar vein, it is helpful to post key announce-
ments on a whiteboard in the review room or in a daily
handout to the reviewers. This will essentially serve as
a running update to the review protocol. Reviewers are
sure to have questions, and preserving the answers in
one place helps ensure consistency amongst the review-
ers’ coding (and will help cut down on the number of
repeat questions you have to answer).
Quality Control
As a supervisor, it is very helpful to spend time each
day in the review platform conducting a second-level
review/QC of documents that have been first-level re-
viewed, particularly early in the review. The earlier in
the process you can catch mistakes, the fewer issues
you will have at the end of the review. So, be sure to
keep these QC tips in mind.
s Take notes on what you find during your QC,
whether it is inconsistent coding amongst reviewers
on similar documents, specific documents you want
to highlight, incorrect usage of an issue tag, or the
like. Conducting a second-level review each day,
3
U.S. LAW WEEK ISSN 0148-8139 BNA 7-7-15

4. even if you can only look at 1 percent of the reviewed
documents, will help prevent surprises at the end of
the review. Each morning, the supervising attorney
should update the reviewers with clarifications, cor-
rections, or new guidance that stems from doing the
QC.
s Ask your DM to provide stats at the end of each
day to show individual reviewers’ progress. Review-
ers at either end of the spectrum—very fast or very
slow, in terms of documents reviewed per hour—
may warrant some additional scrutiny.
ࡗ For those reviewers needing additional scru-
tiny, consider doing a second-level review of a
small percentage of EACH of their documents.
This will let you determine whether the slowest
reviewers are having to slog through long, techni-
cal documents that slowed their pace, or whether
they are simply being slow and inefficient and
warrant a one-on-one conversation about their
pace.
s Also take note of any documents being tagged
with codes such as ‘‘Needs Further Review’’ or
‘‘Technical Issue.’’ You may be able to quickly clarify
early on how documents should be coded to avoid
having to do extra work at the end of the review.
Defensibility
As a final step and upon completion of the document
review, you should maintain records of the decisions
and processes used for executing the review. Keep a
copy of the review protocol and details concerning ad-
ditional guidance given to the reviewers, the quality
control performed, the collection criteria, the produc-
tion specifications, and any other pertinent material
throughout the pendency of the review. This record,
sometimes referred to as a ‘‘defensibility binder,’’ will
help you answer any questions about the review that
may later arise. It may also be useful in the event of a
discovery dispute to prove that the review and produc-
tion was completed with diligence and reasonable care.
Conclusion
Applying these techniques and suggestions will en-
sure that your next large-scale document review is run
efficiently, economically, and with minimal duplication
of efforts. This translates to success for you, your firm,
and your client.
4
7-7-15 COPYRIGHT ஽ 2015 BY THE BUREAU OF NATIONAL AFFAIRS, INC. LW ISSN 0148-8139