The document discusses a new three-year partnership between LexisNexis and NC State to create a lab exploring software engineering methods for big data applications. The lab will help engineers grow their skills in assessing different approaches to analyzing large datasets and validating results. It also explores new algorithms, ways to validate industrial findings, making better use of prior knowledge, and techniques for achieving results using fewer data examples.

1. LEXISNEXIS,
NCSTATE
OPPORTUNITIES
TIM MENZIES
COMPUTER SCIENCE,
JUNE 2015

2. SEBIG LAB : SE FOR BIG DATA
• Three year partnership
• New lab to explore SE methods for big data apps.
• Grow skill set of engineers:
• Assess different approaches to Big Data
• Validation of results
2

3. LAB PROCESSES VS
INDUSTRIAL PROCESSES
• Lab processes
• Make 10ml of
oxygen?
• Easy!
• Make 100,000 liters
per day?
• That’s another matter
3

4. INDUSTRIAL PROCESSES FOR
DATA MINING
4

5. INDUSTRIAL PROCESSES
FOR DATA MINING
5
1
23
4
5

6. EXPLORING NEW ALGORITHMS
• New ideas
• SVM
• Deep learning
• Ensembles
• etc
• Visualizations
• Parameter tuning
• Synonym discovery
• Incremental association
rule learning
6
1

7. VALIDATION STUDIES
• Independent checks of industrial results
• Optimizing validation:
• ? Mechanical Turk
• Better support tools for coding new
functionality
• Better test suites for
certifying new functionality
7
2

8. CAN WE MAKE BETTER
USE OF OLD KNOWLEDGE?
• Learning domain ontologies.
• Corpus definition.
• How to revise old knowledge?
• The privileged review problem.
• Transfer learning.
8
3

9. SUPPORT
Gather case study data
Synthetic studies
Annonymization of data
Training
• Papers
• Tutorials
• Learning information
seeking behavior
9
4

10. LESS IS MORE
• Reasoning via fewer, most representative
examples
• Active learning
• Early stopping
• Stack ranking (early stop)
10
5