1 / 15

When Good Data Goes Bad

Tim Menzies (tim@menzies.us)

Lane Department of Computer Science and Electrical Engineering

West Virginia University

http://menzies.us/pdf/06good2bad.pdf

August 13, 2007

Executive Summary

Executive Summary Background: AI- it works Eg1: text mining Eg2: effort estimation Eg3: severity prediction

2 / 15

Data mining NASA project data Five examples where data mining

found clear quality predictors

for effort for defects

In only \underline{one} of those cases is that data source still active.

All that dead data.

What to do?

“Don’t let it eat away at you. You ex wasn’t that smart. She said you’d rot in Hell. You, my friend, are not rotting.”

Background: AI- it works

Executive Summary Background: AI- it works Eg1: text mining Eg2: effort estimation Eg3: severity prediction

3 / 15

1980s: AI summer 1980s (late): bubble

bursts, AI winter 1990s: great success

with planning, scheduling, data mining

2000s: many successes of AI (data mining) for SE

This talk: AI really works (5 success stories with NASA data) Still, main problem is organizational, not technological

Despite clear success, \frac{4}{5} of those data sources have vanished What to do?

Eg #1: text mining at @ NASA

Executive Summary Background: AI- it works

Eg1: text mining What is data mining? Less is more Less is More (2) Less is more (3)

Eg2: effort estimation

Eg3: severity prediction

4 / 15

What is data mining?

Executive Summary Background: AI- it works Eg1: text mining What is data mining? Less is more Less is More (2) Less is more (3) Eg2: effort estimation Eg3: severity prediction

5 / 15

Diamonds in the dust Sumamrization: not 1000

records, but 3 rules Example #1:

text mining issue reports (Menzies, 2007)

901 NASA records, PITS issue tracker: {severity, free text}

severity  frequency

All unique words, sorted by

“magic” (see below) Rules leared from top N words Severity 2 predictions

10*\{train on \frac{9}{10} , test on \frac{1}{10} \}

N  a=recall  b=precision  F = \frac{2∗a∗b}{a+b}
100  0.81  0.93  0.87
50  0.80  0.90  0.85
25  0.79  0.03  0.85
12  0.74  0.92  0.82
6  0.71  0.94  0.81
3  0.74  0.82  0.78

Rules (from 3 words):

if (rvm ≤ 0) & (srs = 3) → sev=4
else if (srs ≥ 2)  → sev=2
else  → sev=3

Diamonds in the dust

Not 9414 words total or 1662 unique words but 3 most predictive.

1 (panic)  0 2  311 3  356 4  208 5 (yawn)  26

Less is more

Executive Summary Background: AI- it works Eg1: text mining What is data mining? Less is more Less is More (2) Less is more (3) Eg2: effort estimation Eg3: severity prediction

6 / 15

Feature subset selection (FSS)

y = β_{0} + β_{1}f_{1} + β_{2}f_{2} + β_{3}f_{3}... Variance in y reduced by pruning some x_{i} But don’t prune too much:

e.g. ∀x_{i}, y = β_{0}

Analagous argument holds for other representations.

Q: How to do FSS? A1:Apply domain knowledge

e.g., in text mining, TF*IDF:

term frequency inverse document frequency

Frequent terms, but only in a small number of documents

TF*IDF = F [i, j] ∗ log(IDF)

F [i, j] = frequency of word i in document j IDF = #documents / (#documents with i) Above study used top 100 T F ∗ IDF words.

Less is More (2)

Executive Summary Background: AI- it works Eg1: text mining What is data mining? Less is more Less is More (2) Less is more (3) Eg2: effort estimation Eg3: severity prediction

7 / 15

Q: How to do FSS? A2: Exponential time FSS

Try all 2^{F} subsets of F on a target learner Possible, with small feature sets, with some heuristic search

best first, STALE=5 ?.

A3: Linear time FSS (not as thorough as 2^{F}): Sort F, somehow; Try first 1 ≤ f ≤ F features Above study sorted top 100 TF*IDF terms on \underline{infogain}

Intially:

H(C) = − ∑_{c∈C} p(c)log_{2}p(c). After seeing feature f:

H(C|f) = − ∑_{x∈f} p(x) ∑_{cinC} p(c|x)log_{2}p(c|x) So InfoGain = H(C) − H(C|f)

Less is more (3)

Executive Summary Background: AI- it works Eg1: text mining What is data mining? Less is more Less is More (2) Less is more (3) Eg2: effort estimation Eg3: severity prediction

8 / 15

Overfitting avoidance After learning “it” ... ... Try throwing away bits of “it”

e.g. RIPPER ?

If you learn a conjunction, prune with greedy back select If you learn a set of rules, prune with greedy back select For the surviving rules, try replace it with..

a dumb alternative

or a carefully selected modification Very fast: O(m(log(m))^{2}) for m examples Often produces smaller theories than other methods

as above:

if (rvm ≤ 0) & (srs = 3) → sev=4
else if (srs ≥ 2)  → sev=2
else  → sev=3

Eg #2: effort estimation @ NASA

Executive Summary Background: AI- it works

Eg1: text mining

Eg2: effort estimation Effort estimation No “best” effort model

Eg3: severity prediction

9 / 15

Effort estimation

Executive Summary Background: AI- it works

Eg1: text mining

Eg2: effort estimation Effort estimation No “best” effort model

Eg3: severity prediction

10 / 15

NASA COCOMO data: results from Menzies et.al ? learners for continous classes A study of 160 effort estimation methods 20 * { pick any 10, train on remaining, test on 10 }

100 ∗ (pred − actual)/actual

50% percentile 65% percentile 75% percentile
mode= embedded  -9  26  60
project= X  -6  16  46
all  -4  12  31
year= 1975  -3  19  39
mode= semidetached  -3  10  22
ground systems  -3  11  29
center= 5  -3  20  50
missionplanning  -1  25  50
project= gro  -1  9  19
center= 2  0  11  21
year= 1980  4  29  58
avionics monitoring  6  32  56
median  -3  19  39

i.e. usually, very accurate estimates

No “best” effort model  ^{11 / 15} So the generation method is general, but the generated model is task-specific.

Eg #3: severity prediction @ NASA

Executive Summary Background: AI- it works

Eg1: text mining

Eg2: effort estimation

Eg3: severity prediction SILAP SILAP + RIPPER + FSS Maturing knowledge

12 / 15

SILAP: Early Lifecycle Severity Detection

Executive Summary Background: AI- it works

Eg1: text mining

Eg2: effort estimation

Eg3: severity prediction SILAP SILAP + RIPPER + FSS Maturing knowledge

13 / 15

NASA defect data: 5 projects, (Menzies et.al 2007) SILAP: predict error {potential, consequence} from project description

Derived  Raw features

co = Consequence

am =Artifact Maturity

dv = Development  as =Asset Safety

ep = Error Potential cl =

CMM Level

pr = Process  cx =Complexity

sc = Software Characteristic di =Degree of Innovation do =Development Organization dt =Use of Defect Tracking System

ex =Experience

fr =Use of Formal Reviews hs =Human Safety

pf =Performance

ra =Re-use Approach rm =Use of Risk Management System ss =Size of System uc =Use of Configuration Management us =Use of Standards

function CO( tmp)    { tmp=0.35*AS + 0.65 *PF; return (round((HS) < tmp) ? HS : tmp)
function EP()        { return round(0.579*DV() + 0.249*PR() + 0.172*SC())}
function SC()        { return 0.547*CX + 0.351*DI + 0.102*SS  }
function DV()        { return 0.828*EX + 0.172*DO }
function PR()        { return 0.226*RA + 0.242*AM + formality()  }
function formality() { return 0.0955*US+ 0.0962*UC+ 0.0764*CL + 0.1119*FR +0.0873*DT + 0.0647*RM}

2^{F} FSS + RIPPER + SILAP data (211 componenents on 5 projects)

Executive Summary Background: AI- it works

Eg1: text mining

Eg2: effort estimation

Eg3: severity prediction SILAP SILAP + RIPPER + FSS Maturing knowledge

14 / 15 severity predictions a=pd, b=prec x = 2ab/(a + b)

features  |F |  12 3 4 5  X = (∑ x) /4
A all - L1 - L2 - group(6)  8  0.97 0.95 0.97 0.99  0.97
B all - L1 - L2 - group(5 + 6)  7  0.95 0.94 0.97 0.96  0.96
C all - L1 - L2 - group(4 + 5 + 6)  6  0.93 0.95 0.98 0.93  0.95
D all - L1 - L2  16  0.94 0.94 0.93 0.96  0.94
E all - L1 - L2 - group(3 + 4 + 5 + 6) 4  0.93 0.97 0.90 0.87  0.92
F {co*ep, co, ep}  3  0.94 0.84 0.55 0.70  0.76
G L1  1  0.67 0.69 0.00 0.46  0.45
H just ”us”  1  0.64 0.60 0.00 0.00  0.31
I L2  1  0.57 0.00 0.32 0.00  0.22

Rules from “E”: rule 1  if uc ≥ 2 ∧ us = 1  then severity = 5 rule 2 else if am = 3  then severity = 5 rule 3 else if uc ≥ 2 ∧ am = 1 ∧ us ≤ 2 then severity = 5 rule 4 else if am = 1 ∧ us = 2  then severity = 4 rule 5 else if us = 3 ∧ ra ≥ 4  then severity = 4 rule 6 else if us = 1  then severity = 3 rule 7 else if ra = 3  then severity = 3 rule 8 else if true  then severiry = 1 or 2

FSS to mature business knowledge

Executive Summary Background: AI- it works

Eg1: text mining

Eg2: effort estimation

Eg3: severity prediction SILAP SILAP + RIPPER + FSS Maturing knowledge

15 / 15

2005: Delphi session results:  weight*  contribution goal  feature  weight  filter  filter  to goal consequence  hs  0.35  1  0.350  35% (co)  pf  0.65  1  0.650  65% error  ex  0.828  0.579  0.479  47% potential  cx  0.547  0.172  0.094  9% (ep)  do  0.172  0.579  0.100  9% di  0.351  0.172  0.060  6% am  0.242  0.249  0.060  6% ra  0.226  0.249  0.056  5% us  0.0955  0.249  0.024  2% uc  0.0962  0.249  0.024  2% fr  0.119  0.249  0.030  2% dt  0.0873  0.249  0.022  2% ss  0.102  0.172  0.018  1% cl  0.0764  0.249  0.019  1% rm  0.0647  0.249  0.016  1%

2007: Features seen in 10 FSS (90% samples): number of times group  feature  notes  selected 1  us  use of standards  10 2  uc  config management  9 ra  reuse approach  9 am  artifact maturity  9 3  fr  formal reviews  8 ex  experience  8 4  ss  size of system  7 5  rm  risk management  6 6  cl  CMM level  5 dt  defect tracking  5 do  development organization  4 di  degree of innovation  4 hs  human safety  3 as  asset safety  2 cx  complexity  2 pf  performance  1