This document discusses predictive coding 2.0 as an improved method for e-discovery. Predictive coding 1.0 had limitations in dealing with incomplete document collections that are continuously updated and changing coding calls. Predictive coding 2.0 utilizes a flexible analytics framework based on bipartite graphs that can dynamically assess documents and adapt to new information as the collection and coding changes over time, allowing for continuous case assessment. The authors provide examples of how predictive coding 2.0 could enhance e-discovery in complex litigation matters.
Interaction Mining: the new frontier of Call Center AnalyticsVincenzo Pallotta
This document discusses using interaction mining to analyze call center data. It outlines how interaction mining can identify customer-oriented behaviors, root causes of problems, problematic customers, and best practices for dealing with customers. The document describes applying automatic argumentative analysis to call center conversations to label statements with discourse relations and argumentative labels. It evaluates this approach on meeting conversations, achieving 81% precision and 98% recall. Experiments applying these techniques to a corpus of 213 manually transcribed call center conversations are also discussed.
Residual Cash Blaster (RCB) is a multi-level marketing company that provides business marketing tools and a compensation plan to help people generate residual income. The compensation plan uses a 2x4 forced matrix to fill 30 positions, which can generate up to $43,000 monthly from four phases of commissions. There is no cost to join RCB as an independent marketing director and no requirement to purchase products.
OBS Built Melbourne Business School's (MBS) new SharePoint website. The website presents all program information and contact details and receives the majority of program inquiries, student registrations and donations, via payment gateway. OBS Managed Services manages the website.
This short song welcomes and invites the Lord into a holy place and house of praise, acknowledging him as righteous and glorious, and worthy of praise. The singer welcomes the Lord into this location for worship multiple times.
Dr. Andreas Oskar Kempf, Ute Sondergeld: Indicator-Based Monitoring of an Interdisciplinary Field of Science. The Example of Educational Research - Presentation at IASSIST 2013
The Employee Referral Program office is inviting employee referrals for various job openings requiring skills in GTS-SD drive, support skills, DBM, BAP, MW skills, ECS, U&I-Midrange, SMI, AOD, and EUS. Details of the job descriptions are provided in an attached Excel file. Candidates for agent fresher positions should be sent directly to the interview location.
Interaction Mining: the new frontier of Call Center AnalyticsVincenzo Pallotta
This document discusses using interaction mining to analyze call center data. It outlines how interaction mining can identify customer-oriented behaviors, root causes of problems, problematic customers, and best practices for dealing with customers. The document describes applying automatic argumentative analysis to call center conversations to label statements with discourse relations and argumentative labels. It evaluates this approach on meeting conversations, achieving 81% precision and 98% recall. Experiments applying these techniques to a corpus of 213 manually transcribed call center conversations are also discussed.
Residual Cash Blaster (RCB) is a multi-level marketing company that provides business marketing tools and a compensation plan to help people generate residual income. The compensation plan uses a 2x4 forced matrix to fill 30 positions, which can generate up to $43,000 monthly from four phases of commissions. There is no cost to join RCB as an independent marketing director and no requirement to purchase products.
OBS Built Melbourne Business School's (MBS) new SharePoint website. The website presents all program information and contact details and receives the majority of program inquiries, student registrations and donations, via payment gateway. OBS Managed Services manages the website.
This short song welcomes and invites the Lord into a holy place and house of praise, acknowledging him as righteous and glorious, and worthy of praise. The singer welcomes the Lord into this location for worship multiple times.
Dr. Andreas Oskar Kempf, Ute Sondergeld: Indicator-Based Monitoring of an Interdisciplinary Field of Science. The Example of Educational Research - Presentation at IASSIST 2013
The Employee Referral Program office is inviting employee referrals for various job openings requiring skills in GTS-SD drive, support skills, DBM, BAP, MW skills, ECS, U&I-Midrange, SMI, AOD, and EUS. Details of the job descriptions are provided in an attached Excel file. Candidates for agent fresher positions should be sent directly to the interview location.
Este documento presenta un programa de formación experiencial sobre nuevas formas de autoridad y liderazgo que se llevará a cabo del 6 al 10 de febrero de 2015. El programa explorará temas como el ejercicio de la autoridad, el liderazgo, la gestión del cambio y las dinámicas grupales a través de experiencias grupales. Contará con la participación de líderes, ejecutivos y consultores interesados en mejorar su comprensión de los procesos organizacionales.
The document discusses the design of quests for language learning in Second Life Chinese School. It proposes using problem-based quests that involve meaningful social interactions and embodied experiences to scaffold learners from novice to expert levels. The quests encourage learners to organize their own learning and assume abilities they wish to develop through repeated practice, feedback and negotiation during social interactions. This situated and improvisational approach aims to make language learning fun yet challenging.
A datastore is a central system that offers data in a consistent place and provides important context and metadata about the data. It allows data owners to publish their data through various methods like email submissions, file dropboxes, data proxies, or data replication. The document also discusses challenges around building and maintaining datastores, as well as opportunities for engaging communities around open data.
Both album covers feature pictures relating to their titles and the band's name prominently displayed. The inside covers of both albums include pictures of the band members, similar to Guns N' Roses' Appetite for Destruction. The back covers have plain, dark backgrounds to make the song titles and other text stand out clearly. The cartoon style of the covers, poster, and website helped convey the tongue-in-cheek brand image.
Alex Wood Presentation - CIRANO & l'École Polytechnique de Paris Workshop: Co...Sustainable Prosperity
This document discusses a workshop on corporate social responsibility (CSR) and firm performance. It presents a framework for understanding the relationship between CSR and financial performance. While research shows only a correlation, focusing on CSR may boost employee morale and productivity, and provide reputational benefits. In the long run, CSR focus helps firms address rising costs from resources, regulations, and other factors. However, CSR does not immediately boost cash flows. The role of policy is to provide certainty that helps CSR integration become more financially profitable over time as constraints increase.
Ventaforce is a comprehensive MLM software that allows companies to professionally manage all aspects of their multi-level marketing business through a single platform. It provides features such as distributor management, genealogy tracking, payment processing, reporting, and administration tools to help companies efficiently operate their network marketing operations.
Code quality is important to ensure code is easy to understand, maintain and extend. Key aspects of code quality include having high test coverage, following design principles like SOLID, and using metrics like maintainability index and technical practices such as pair programming, code reviews, refactoring and test-driven development. Tools like Visual Studio and Resharper can help analyze code quality metrics and identify areas for improvement.
The document discusses creating copy assignment operators that are stable and safe for self-assignment. It recommends including a check for self-assignment (if(this != &rhs)) to avoid unnecessary copying. While some argue this adds branches that hurt performance, the document argues the performance impact is negligible compared to other operations like memory allocation. Premature optimization should be avoided, and code clarity and correctness should come before micro-optimizations.
This document discusses patterns in distributed computing. It outlines some of the key hurdles in distributed systems like asynchrony, locality, failure, and Byzantine faults. It then summarizes common distributed algorithms like leader election, group membership, and consensus. It presents two Ruby libraries, TokenWorker and StaticQueueWorker, that implement leader election and work distribution patterns. It also briefly outlines the Paxos consensus algorithm but notes that a full implementation is very challenging. The document advocates for practical approaches over purely formal solutions and emphasizes starting with reasonable reliability requirements.
The document provides techniques for estimating the risk level and number of defects in code during testing. It discusses counting previous defects, investigating code coverage, assigning risk scores, and using metrics like cyclomatic complexity and bug level to estimate risk for functionality. It also outlines techniques for deciding when sufficient testing has been done, such as looking for a plateau in detected defects and ensuring test cases fully exercise the code. Methods for estimating the number of defects include using the number of requirements, lines of code, similar previous projects, and defects found in prior phases.
How much time it takes for my feature to arrive?Daniel Alencar
How much time it takes for a bug fix or a new feature be available to users? We did an empirical work to better understand what makes a new feature or bug fix to arrive faster to users
The study analyzed delays in integrating addressed issues into software releases. It found that 34-60% of addressed issues in traditional release cycles were delayed, while 98% were delayed in rapid release cycles. The study built prediction models using random forests to predict integration delays. Integration workload was the most influential variable in the models. The models performed better than random guessing with precisions of 0.59-0.88 and ROC areas above 0.74.
Technical debt is often characterized as design or code tradeoffs. In this talk I discuss how shortcuts in requirements analysis might lead to technical debt as well.
We'll discover the reasons why it is a risky bet to not *aim* to manage infrastructure and its configuration with idempotence and immutability at heart.
Sharing real world experience, we'll see why configurations should not be done by humans (it's like playing Djenga), and why what may work at the beginning does not work over a long period of time or scale (pet vs cattle problem).
Jon Powell - MECLABS - offer response optimization - IMS Boston 2012thepulsenetwork
This document summarizes an A/B test conducted to determine the most effective content focus for marketing a medical provider specializing in chronic back pain treatment. The test compared four content focuses in online ads: back pain resources, causes and solutions, symptoms, and treatment options. The goal was to see which focus generated the highest clickthrough rate from condition-based searchers. The results would inform the medical provider's content marketing strategy.
Industry - Relating Developers' Concepts and Artefact Vocabulary in a Financ...ICSM 2011
Paper: Relating Developers' Concepts and Artefact Vocabulary in a Financial
Software Module
Authors: Tezcan Dilshener and Michel Wermelinger
Session: Industry Track 2 - Reverse Engineering
Use SAS to identify what tell-tale signs in consumers’ credit history would best model the bad consumers, and, in turn, use this as a way to prevent potential future bad consumers from getting approved for lines of credit.
AWS Public Sector Symposium 2014 Canberra | Putting the "Crowd" to work in th...Amazon Web Services
"Cloud" computing provides significant advantages and enormous cost savings by allowing IT infrastructure to be provisioned as a ubiquitous, metered, unit priced and on demand service. However, the other major resourcing issue faced by CIO’s is the provision of skilled labour to develop, support and maintain a increasing wide range of IT applications.
This session will show attendees how the worldwide pool of freelance developers, the "Crowd", can be utilised as a ubiquitous, metered, unit priced and on demand resource pool to work in the "Cloud" to improve responsiveness to customer demands, reduce development timeframes and achieve significant cost savings.
Although the crowd can bring enormous benefits in terms of cost and agility, there are some technical and business barriers to adoption in large organisations. This presentation will discuss the barriers and, using some real examples, will explain how GoSource overcomes them.
Build systems orchestrate how human-readable source code is translated into executable programs. In a software project, source code changes can induce changes in the build system (aka. build co-changes). It is difficult for developers to identify when build co-changes are necessary due to the complexity of build systems. Prediction of build co-changes works well if there is a sufficient amount of training data to build a model. However, in practice, for new projects, there exists a limited number of changes. Using training data from other projects to predict the build co-changes in a new project can help improve the performance of the build co-change prediction. We refer to this problem as cross-project build co-change prediction.
In this paper, we propose CroBuild, a novel cross-project build co-change prediction approach that iteratively learns new classifiers. CroBuild constructs an ensemble of classifiers by iteratively building classifiers and assigning them weights according to its prediction error rate. Given that only a small proportion of code changes are build co-changing, we also propose an imbalance-aware approach that learns a threshold boundary between those code changes that are build co-changing and those that are not in order to construct classifiers in each iteration. To examine the benefits of CroBuild, we perform experiments on 4 large datasets including Mozilla, Eclipse-core, Lucene, and Jazz, comprising a total of 50,884 changes. On average, across the 4 datasets, CroBuild achieves a F1-score of up to 0.408. We also compare CroBuild with other approaches such as a basic model, AdaBoost proposed by Freund et al., and TrAdaBoost proposed by Dai et al.. On average, across the 4 datasets, the CroBuild approach yields an improvement in F1-scores of 41.54%, 36.63%, and 36.97% over the basic model, AdaBoost, and TrAdaBoost, respectively.
Building on quicksand microservices indicthreadsIndicThreads
The document discusses the evolution of distributed systems from single machines to replicated databases and services. It explains how eventual consistency allows for higher availability but reduces correctness by allowing stale data reads. The key is that different applications have different consistency needs based on their risk tolerance. Rather than strict consistency, eventual consistency with apologies is often sufficient and enables more flexible tradeoffs between correctness and availability for increased business value.
The document discusses finding a good development partner. It recommends looking for a partner that has clear coding and testing standards. Specifically, it suggests standards should require single trigger frameworks, 85% or higher test coverage, use of assertions in tests, and comments in the code. It also advises verifying code quality by ensuring tests generate their own data and contain assertions rather than relying on @seeAllData annotations or methods that simply increment variables to fake coverage.
Este documento presenta un programa de formación experiencial sobre nuevas formas de autoridad y liderazgo que se llevará a cabo del 6 al 10 de febrero de 2015. El programa explorará temas como el ejercicio de la autoridad, el liderazgo, la gestión del cambio y las dinámicas grupales a través de experiencias grupales. Contará con la participación de líderes, ejecutivos y consultores interesados en mejorar su comprensión de los procesos organizacionales.
The document discusses the design of quests for language learning in Second Life Chinese School. It proposes using problem-based quests that involve meaningful social interactions and embodied experiences to scaffold learners from novice to expert levels. The quests encourage learners to organize their own learning and assume abilities they wish to develop through repeated practice, feedback and negotiation during social interactions. This situated and improvisational approach aims to make language learning fun yet challenging.
A datastore is a central system that offers data in a consistent place and provides important context and metadata about the data. It allows data owners to publish their data through various methods like email submissions, file dropboxes, data proxies, or data replication. The document also discusses challenges around building and maintaining datastores, as well as opportunities for engaging communities around open data.
Both album covers feature pictures relating to their titles and the band's name prominently displayed. The inside covers of both albums include pictures of the band members, similar to Guns N' Roses' Appetite for Destruction. The back covers have plain, dark backgrounds to make the song titles and other text stand out clearly. The cartoon style of the covers, poster, and website helped convey the tongue-in-cheek brand image.
Alex Wood Presentation - CIRANO & l'École Polytechnique de Paris Workshop: Co...Sustainable Prosperity
This document discusses a workshop on corporate social responsibility (CSR) and firm performance. It presents a framework for understanding the relationship between CSR and financial performance. While research shows only a correlation, focusing on CSR may boost employee morale and productivity, and provide reputational benefits. In the long run, CSR focus helps firms address rising costs from resources, regulations, and other factors. However, CSR does not immediately boost cash flows. The role of policy is to provide certainty that helps CSR integration become more financially profitable over time as constraints increase.
Ventaforce is a comprehensive MLM software that allows companies to professionally manage all aspects of their multi-level marketing business through a single platform. It provides features such as distributor management, genealogy tracking, payment processing, reporting, and administration tools to help companies efficiently operate their network marketing operations.
Code quality is important to ensure code is easy to understand, maintain and extend. Key aspects of code quality include having high test coverage, following design principles like SOLID, and using metrics like maintainability index and technical practices such as pair programming, code reviews, refactoring and test-driven development. Tools like Visual Studio and Resharper can help analyze code quality metrics and identify areas for improvement.
The document discusses creating copy assignment operators that are stable and safe for self-assignment. It recommends including a check for self-assignment (if(this != &rhs)) to avoid unnecessary copying. While some argue this adds branches that hurt performance, the document argues the performance impact is negligible compared to other operations like memory allocation. Premature optimization should be avoided, and code clarity and correctness should come before micro-optimizations.
This document discusses patterns in distributed computing. It outlines some of the key hurdles in distributed systems like asynchrony, locality, failure, and Byzantine faults. It then summarizes common distributed algorithms like leader election, group membership, and consensus. It presents two Ruby libraries, TokenWorker and StaticQueueWorker, that implement leader election and work distribution patterns. It also briefly outlines the Paxos consensus algorithm but notes that a full implementation is very challenging. The document advocates for practical approaches over purely formal solutions and emphasizes starting with reasonable reliability requirements.
The document provides techniques for estimating the risk level and number of defects in code during testing. It discusses counting previous defects, investigating code coverage, assigning risk scores, and using metrics like cyclomatic complexity and bug level to estimate risk for functionality. It also outlines techniques for deciding when sufficient testing has been done, such as looking for a plateau in detected defects and ensuring test cases fully exercise the code. Methods for estimating the number of defects include using the number of requirements, lines of code, similar previous projects, and defects found in prior phases.
How much time it takes for my feature to arrive?Daniel Alencar
How much time it takes for a bug fix or a new feature be available to users? We did an empirical work to better understand what makes a new feature or bug fix to arrive faster to users
The study analyzed delays in integrating addressed issues into software releases. It found that 34-60% of addressed issues in traditional release cycles were delayed, while 98% were delayed in rapid release cycles. The study built prediction models using random forests to predict integration delays. Integration workload was the most influential variable in the models. The models performed better than random guessing with precisions of 0.59-0.88 and ROC areas above 0.74.
Technical debt is often characterized as design or code tradeoffs. In this talk I discuss how shortcuts in requirements analysis might lead to technical debt as well.
We'll discover the reasons why it is a risky bet to not *aim* to manage infrastructure and its configuration with idempotence and immutability at heart.
Sharing real world experience, we'll see why configurations should not be done by humans (it's like playing Djenga), and why what may work at the beginning does not work over a long period of time or scale (pet vs cattle problem).
Jon Powell - MECLABS - offer response optimization - IMS Boston 2012thepulsenetwork
This document summarizes an A/B test conducted to determine the most effective content focus for marketing a medical provider specializing in chronic back pain treatment. The test compared four content focuses in online ads: back pain resources, causes and solutions, symptoms, and treatment options. The goal was to see which focus generated the highest clickthrough rate from condition-based searchers. The results would inform the medical provider's content marketing strategy.
Industry - Relating Developers' Concepts and Artefact Vocabulary in a Financ...ICSM 2011
Paper: Relating Developers' Concepts and Artefact Vocabulary in a Financial
Software Module
Authors: Tezcan Dilshener and Michel Wermelinger
Session: Industry Track 2 - Reverse Engineering
Use SAS to identify what tell-tale signs in consumers’ credit history would best model the bad consumers, and, in turn, use this as a way to prevent potential future bad consumers from getting approved for lines of credit.
AWS Public Sector Symposium 2014 Canberra | Putting the "Crowd" to work in th...Amazon Web Services
"Cloud" computing provides significant advantages and enormous cost savings by allowing IT infrastructure to be provisioned as a ubiquitous, metered, unit priced and on demand service. However, the other major resourcing issue faced by CIO’s is the provision of skilled labour to develop, support and maintain a increasing wide range of IT applications.
This session will show attendees how the worldwide pool of freelance developers, the "Crowd", can be utilised as a ubiquitous, metered, unit priced and on demand resource pool to work in the "Cloud" to improve responsiveness to customer demands, reduce development timeframes and achieve significant cost savings.
Although the crowd can bring enormous benefits in terms of cost and agility, there are some technical and business barriers to adoption in large organisations. This presentation will discuss the barriers and, using some real examples, will explain how GoSource overcomes them.
Build systems orchestrate how human-readable source code is translated into executable programs. In a software project, source code changes can induce changes in the build system (aka. build co-changes). It is difficult for developers to identify when build co-changes are necessary due to the complexity of build systems. Prediction of build co-changes works well if there is a sufficient amount of training data to build a model. However, in practice, for new projects, there exists a limited number of changes. Using training data from other projects to predict the build co-changes in a new project can help improve the performance of the build co-change prediction. We refer to this problem as cross-project build co-change prediction.
In this paper, we propose CroBuild, a novel cross-project build co-change prediction approach that iteratively learns new classifiers. CroBuild constructs an ensemble of classifiers by iteratively building classifiers and assigning them weights according to its prediction error rate. Given that only a small proportion of code changes are build co-changing, we also propose an imbalance-aware approach that learns a threshold boundary between those code changes that are build co-changing and those that are not in order to construct classifiers in each iteration. To examine the benefits of CroBuild, we perform experiments on 4 large datasets including Mozilla, Eclipse-core, Lucene, and Jazz, comprising a total of 50,884 changes. On average, across the 4 datasets, CroBuild achieves a F1-score of up to 0.408. We also compare CroBuild with other approaches such as a basic model, AdaBoost proposed by Freund et al., and TrAdaBoost proposed by Dai et al.. On average, across the 4 datasets, the CroBuild approach yields an improvement in F1-scores of 41.54%, 36.63%, and 36.97% over the basic model, AdaBoost, and TrAdaBoost, respectively.
Building on quicksand microservices indicthreadsIndicThreads
The document discusses the evolution of distributed systems from single machines to replicated databases and services. It explains how eventual consistency allows for higher availability but reduces correctness by allowing stale data reads. The key is that different applications have different consistency needs based on their risk tolerance. Rather than strict consistency, eventual consistency with apologies is often sufficient and enables more flexible tradeoffs between correctness and availability for increased business value.
The document discusses finding a good development partner. It recommends looking for a partner that has clear coding and testing standards. Specifically, it suggests standards should require single trigger frameworks, 85% or higher test coverage, use of assertions in tests, and comments in the code. It also advises verifying code quality by ensuring tests generate their own data and contain assertions rather than relying on @seeAllData annotations or methods that simply increment variables to fake coverage.
Certification Study Group - Professional ML Engineer Session 3 (Machine Learn...gdgsurrey
Dive into the essentials of ML model development, processes, and techniques to combat underfitting and overfitting, explore distributed training approaches, and understand model explainability. Enhance your skills with practical insights from a seasoned expert.
DockerCon SF 2019 - Observability WorkshopKevin Crawley
This document contains the slides from a workshop on observability presented by Kevin Crawley of Instana and Single Music. The workshop covered distributed tracing using Jaeger and Prometheus, challenges with open source monitoring tools, and advanced use cases for distributed tracing demonstrated through Single Music's experience. The agenda included labs on setting up Kubernetes and applications, monitoring metrics with Grafana and Prometheus, distributed tracing with Jaeger, and analytics use cases.
ACCOUNTING INFORMATION SYSTEMSAccess and Data Analytics Test.docxSALU18
ACCOUNTING INFORMATION SYSTEMS
Access and Data Analytics Test
General Instructions.
This exam has four parts. Part 1 is in class. Parts 2, 3, and 4 are take-home. Submit all parts to the
designated dropbox folder. I expect your individual effort on all parts. Parts 2 to 4 are described in a
separate document.
Part 1 – Access (50 points).
To get full credit, you must set up appropriate relationships among the tables and enforce referential
integrity for each link. Your queries must produce the correct values, the fields must by labeled and
formatted appropriately, and query designs must not include extraneous tables. In other words, you
should follow the list of fundamental rules for Access posted on BeachBoard and included at the end of
this document for reference.
1. Download the Fall_2019 database posted in the Access and Data Analytics Test Module under
CONTENT on BeachBoard.
2. Ensure that primary keys are set and establish appropriate relationships among the tables:
Stores, Vendors, Purchases, and Purchase_Items. Stores and Vendors should be linked to
Purchases. Purchases should be linked to Purchase_Items.
3. Prepare the following queries, naming the queries qa, qb, qc, qd, corresponding to the
identifying letters below:
a. Use the purchase_items table to calculate the dollar amount of each item purchased in
an extension query; name your new calculated field purchase_item_amount and format it
appropriately.
b. Use qa and the purchases table to sum the purchase item amounts for each purchase in
an accumulation query; include all fields from the purchases table and the
purchase_item_amount field from qa; name your summed field purchase amount and
format it appropriately.
c. Use qb and the vendors table to sum the purchase amounts from each vendor in
another accumulation query; include vendor number, name, city, and state; name your
summed field vendor purchases and format it appropriately.
d. Use the qb query. Keeping all fields from qb, calculate the month of the purchase;
name that field purchase month.
BEFORE SUBMITTING, ask me to review your work. After I say that you are done, then submit your file
to the BeachBoard DROPBOX. Be sure to close Access before you upload your results.
1
Some Fundamental Rules for Access
1. Look at your tables and think about what information those tables provide before you start
linking tables and creating queries.
2. Make sure each table has a primary key designated.
3. Always establish relationships between tables first, before starting queries.
4. Always enforce referential integrity (or understand why you can’t).
5. No “expr1” field names.
6. Do not click on the big sigma to produce totals if the query doesn’t require totals (i.e., an
extension query).
7. Avoid “SumOf…” field names in accumulation queries.
8. Include identifying information in addition to the primary key in accumulation queries that
provide subtotals.
9. Always format new fields prope.
The Use of Development History in Software Refactoring Using a Multi-Objectiv...Ali Ouni
The document presents a multi-objective approach to automate software refactoring using evolutionary algorithms. It formulates refactoring as a multi-objective optimization problem to improve code quality, preserve semantics, and maximize reuse of past development history. An evaluation on two open source projects shows the approach corrects most defects while maintaining high refactoring precision compared to existing techniques. Future work includes leveraging refactoring histories from multiple systems and improving context-based similarity measures.
Fine-tuning Large Language Models by Dmitry BalabkaDevClub_lv
focusing on the hands-on process of preparing datasets and fine-tuning models for a specific business task. This session will cover dataset preparation, model fine-tuning, and cloud ML accelerators like TPUs and related libraries. It’s aimed at those seeking hands-on knowledge in applying ML techniques.
Fine-tuning Large Language Models by Dmitry Balabka
Predictive Coding Legaltech
1. Predictive Coding 2.0
Making E-Discovery
More Efficient and Cost Effective
John Tredennick
Jeremy Pickens
Jim Eidelman
2. How Many Do I Have to Check?
1. You have a bag with 1 million M&Ms
2. It contains mostly brown M&Ms
3. You cannot see into the bag
4. You have a scoop that will pull out 100
M&Ms at a time
5. Your hope is that there are no red
M&Ms in the bag
6. You pull out a scoop and they are all
brown
How many scoops do you need to review
to be confident there are no red M&Ms?
3. Let’s Take a Poll
How many scoops?
2
1 3
5 10 20
100? 500?
1,000?
4. How Confident Do You Need to Be?
Does 95% work? How about 99%
How many errors can you tolerate?
§ Five out of a hundred?
§ One out of a hundred?
§ One percent = 10,000
At a 95% confidence level and 5% percent margin of error: 384 M&Ms
At a 99% confidence level and 1% margin of error: 459 M&Ms
At a 100% confidence level and 0% margin of error: 1,000,000 M&Ms
8. What Have the Courts Said?
“Until there is a judicial opinion approving (or
even critiquing) the use of predictive coding,
counsel will just have to rely on this article as a
sign of judicial approval. In my opinion,
computer-assisted coding should be used in
those cases where it will help ‘secure the just,
speedy, and inexpensive’ (Fed. R. Civ. P. 1)
determination of cases in our e-discovery
world.”
Magistrate Judge Andrew Peck
9. Predictive Coding 1.0
1. Assemble your corpus.
2. Assemble a seed set of
documents.
3. Review the seed set.
4. Apply machine learning and
automatically tag the remainder
of the corpus.
10. Predictive Coding 1.0
§ Tremendous gains in review
effectiveness
§ Substantial cost savings
§ It works. Often quite well
….when the corpus is complete.
13. In which upload and on which day do your responsive
documents show up?
67 166
uploads days
Terms that do not appear early begin appearing later.
14. Machine-Assisted Decision Making
Upload timeline of 6 TB case.
When should machine-assisted
Is it here? decision making (e.g. early case
assessment) begin?
Or here?
15. Example: Responsive Early, Junk Later
To: bob@company.com, alice@company.com
From: charles@company.com
Subject: Company Picnic
Bob, would you coordinate with Alice and make sure we have
enough hamburger buns for the company picnic? Please try
and find them at a reasonable price.
Responsive Junk
16. Example: Junk Early, Responsive Later
To: bob@company.com, alice@privatemail.com
From: charles@company.com
Subject: Get Together
Let’s get together at 7pm at the Sports Bar to discuss pricing of
our components. The Broncos are playing and I really want to
watch Tebow.
Junk
Responsive
17. Problems With Predictive Coding 1.0
The corpus is almost never complete
§ Continuous collection and rolling uploads
§ When does “Early Case Assessment” begin?
Changing Issues
§ Responsiveness is “bursty”
Shifting Concept Relationships
§ Due both to increasing corpus and changing issues
§ Exploration is extremely limited
18. Our Approach
Predictive Coding 2.0 necessitates the ability to deal with
dynamic change and flux.
We have developed a flexible analytics framework based
on bipartite graphs
It is aware of changes in corpus and in coding so as to
enable smart review and adaptive related concept
suggestion as information pours in.
19. Our Approach
Avoid the lock-in that arises due to poor decision making that
occurs early in the matter when corpus (collection) and coding
information is incomplete.
Goal:
Continuous Case Assessment
20. What Is Underneath?
A full bipartite graph of the
documents and features (e.g.
words, phrases, dates) that
comprise those documents
22. Feedback: Immediate and Continuous
Continuous feedback aids better decision
making and predictive coding.
Adapts to both:
New arrival of coding information
New arrival of documents and terms
24. Predictive Coding 2.0
Feedback – and
improvement – is iterative,
continuous, amplified.
The more you review, the
less you have to review
% of Docs Examined Manually
25. Better Decisions As Understanding Improves
Term relationships change over time
Using continuous improvement,
decisions can be revised and refined
as the matter proceeds.
26. Terms Documents
Time
uncovers
new
relationships
27. Looking at Concepts Over Time
20%
65%
Start with the lube
fuels
key term piping
fob
battery
purityethane
“fuel” mounted
petrochemicals
redundant
fin
batteries
paraxylene
At 20% compartments
cif
mixture
phy
these are airflow
fwd
the related ansi
swopt
ventilation
brentpartials
terms chargers
brg
stainless
locswap
rotor
benzene
And at 65% bleed
diff
accessory
spd
plenum
liquids
detector
opt
30. Putting Related Concepts to Work
The whole corpus
Topic 203
…whether the Company had met,
or could, would, or might meet its
financial forecasts, models,
projections, or plans…
Topic 205
…analyses, evaluations,
TREC collection projections, plans, and reports on
with many topics the volume(s) or geographic
identified location(s) of energy loads.
31. Model In the Whole Collection
Term
Score
Look at the
keyword “model” modeling
1000
equation
864
Scope is the stochastic
706
whole collection variables
677
parameters
518
probability
365
simulation
337
assumption
325
returns
251
curves
211
32. Model In Topic 203
Term
Score
Look at the
keyword “model” flows
1000
assumptions
913
Scope: Topic 203 gains
872
shares
864
meeting liquidity
486
financial fluctuations
374
forecasts
analysts
285
cents
254
whitewing
237
handles
166
33. Model In Topic 205
Term
Score
Look at the
keyword “model” bids
1000
congestion
611
Scope: Topic 205 loads
455
constraints
354
analyzing clearing
292
energy zonal
194
volumes
signals
192
procure
190
dispatch
152
csc
120
34. Model In Comparison
Now, Whole Corpus
Topic 203
Topic
205
imagine this modeling
flows
bids
with batches equation
assumptions
congestion
and coding stochastic
gains
loads
changes variables
shares
constraints
over time! parameters
liquidity
clearing
Note: Our system probability
fluctuations
zonal
can accept any simulation
analysis
signal
combination of
coding and assumption
cents
procure
metadata filters
to dynamically
returns
whitewing
dispatch
assess your data curves
handles
csc
36. Predictive Coding 2.0
Problem: The corpus is almost never complete
Answer: Review Algorithms that are iterative and continuous
Problem: Changing Issues
Answer: Review Algorithms that are adaptive and continuous
Problem: Shifting Concept Relationships
Answer: Concept Relationships that are calculated dynamically, on-
the-fly, and coding-aware.
Continuous Case Assessment
37. Analytics Consulting
§ Analytics consulting and predictive ranking for nearly 4 years
§ How it started -- Before “Predictive Coding” became popular:
“Can’t you predict what documents are probably
relevant based on your review so far?”
– Judge, SDNY
§ Predictive Ranking: Iterative search techniques + algorithms
§ Then off-the-shelf Predictive Coding 1.0 technologies
§ Catalyst’s research is exciting! We apply the research to real-world
scenarios. Applying Bipartite Analytics…
38. Smart Review with the Bipartite Analytics
Technology Advantages:
§ Accurate
§ Dynamic
§ Flexible
§ “Just in Time” suggestions
39. Smart Review Scenarios
1. “What happened” – examples: FCPA investigation, conspiracy ECA
2. Typical large scale litigation with lots of ESI –
e.g., class action lawsuit
3. Highly complex litigation with multiple issues –
e.g. patent and unfair competition claims
40. Scenario 1 – What happened?
Goal: Rapidly determine facts and resolve matter if possible
Applying the Technology
Small number of knowledgeable attorneys drill into documents using the
fusion of advanced search features and flexible predictive coding.
41.
42.
43.
44. Scenario 1 – What happened?
Goal: Rapidly determine facts and resolve matter if possible
Applying the Technology
Small number of knowledgeable attorneys drill into documents using
the fusion of advanced search features and flexible predictive coding.
§ Faster location of valuable “veins” of information
due to search filters
§ Rapid learning and application of that learning
through flexible, “just in time” predictive coding 2.0.
§ “Choose your own adventure”
45. Scenario 2 – Large Scale Litigation
Goal: Minimize cost because of learning across large document set,
increase quality with focused review, and maximize protection of
privilege and trade secrets
Applying the Technology:
§ Prioritized review based on rapid, continuous learning
§ Large scale defensible culling
§ More accurate ranking of “potentially privileged” documents
46. Scenario 3– Highly Complex Litigation
Goal: Review and produce with multiple and changing issues
Applying the Technology
§ Rapid learning across multiple topics
§ Leverage ability to adjust for change in topics
§ Review quality improves because of focus
§ Explore otherwise hidden subjects with Concept Explorer
§ Leverage learning across narrow, focused lines of inquiry (e.g.,
emails between two people in a narrow time window)
§ Protect privileged documents
47. Predictive Coding 2.0
Making E-Discovery
More Efficient and Cost Effective
John Tredennick
Jeremy Pickens
Jim Eidelman