{\bf 2.2 EXPECTED SIGNIFICANCE:}
Building high quality software and providing adequate assessment is
expensive. Developers typically skew their quality assurance (QA)
work towards artifacts where they believe it will be most useful.

This skewing process can introduce an inappropriate bias to quality
assurance (QA). If  the QA activities concentrate on  project
artifacts, say {\em A,B,C,D}, then that leaves {\em blind spots} in
{\em E,F,G,H,I,...}. Blind spots can compromise high assurance
software. Leveson remarks that in modern complex systems, unsafe
operations often result from an unstudied interaction between
components~\cite{leven95}. For example, Lutz and
Mikulski~\cite{lutz03} found a blind spot in NASA deep-space
missions: most of the mission critical {\em in-flight} anomalies
resulted from errors in {\em ground software} that fails to
correctly collect data from the flight systems.

To avoid blind spots, one option is to rigorously assess all aspects
of all  software modules or other artifacts. But this is
impractical. Software projects budgets are finite and QA
effectiveness increases with QA effort. A {\em linear} increase in
the confidence $C$ that we have found all faults can take {\em
exponentially} more effort. For example, to detect one-in-a-thousand
module faults, moving $C$ from 90\% to 94\% to 98\% takes 2301,
2812, and 3910 black box tests (respectively)\footnote{A randomly
selected input to a program will find a fault with probability $x$.
Voas observes~\cite{voas95} that after $N$ random black-box tests,
the chance of the inputs not revealing any fault is $(1-x)^N$.
Hence, the chance $C$ of seeing the fault is $1-(1-x)^N$ which can
be rearranged to
 $N(C,x)=\frac{log(1 -
C)}{log(1-x)}$. For example, $N(0.90,10^{-3})=2301$.}.
%\item {\bf Automatic formal methods:}
%The
%infamous state space explosion problem imposes
%strict limits on how much a system can be explored
%via automatic formal methods~\cite{me00q}.
Lowry et.al.~\cite{lowrey98} and Menzies and Cukic~\cite{me99q,me00y}
offer numerous other examples
where assessment effectiveness is exponential on effort.

Exponential cost increase quickly exhausts finite QA resources.
Hence, blind spots can't be avoided and must be managed. Standard
practice is to apply the best available assessment methods on the
sections of the program that the best available domain knowledge
declares is the most critical.  We endorse this approach.  Clearly,
the most critical sections require the best known assessment
methods, in hope of minimizing the risk of safety or mission
critical failure occurring post deployment. However, this focus on
certain sections can blind us to defects in other areas which,
through interactions, may cause similarly critical failures.
Therefore, the standard practice should be augmented with a {\em
lightweight sampling policy} that (a)~explores the rest of the
software and (b)~raises an alert on parts of the software that
appear problematic.  This sampling approach is incomplete by
definition. Nevertheless, it is the only option when 
resource limits block
complete assessment.

{\bf 2.3. BACKGROUND: } 
One such lightweight sampling policy is the
use of {\em fault predictors}. To learn such prediction models,
either from projects previously developed in the same environment or
from a continually expanding base of current project's artifacts,
tables of examples are formed where one column has a boolean value
for ``faults detected'' and the other columns describe software
features such as (i)~lines of code, (ii)~number of unique
symbols~\cite{halstead77}, or (iii)~max. number of possible
execution pathways~\cite{mccabe76}. Each row in the table holds data
from one ``module''; i.e. the  unit of functionality. Depending on
the language, modules may be called ``functions'', ``methods'',
``procedures'' or ``files''. The data mining task is to find
combinations of  features that predict for the value in the defects
column.  Once such combinations are found, managers can use them to
determine where to best focus their QA effort. Better yet, if they
have already focused their QA effort on the most critical portions
of the system, the detectors can ``nudge'' them towards areas that
are also in need of quality improvement.

The value of static code features as defect predictors has
been widely debated.
While some researchers vehemently oppose them~\cite{fenton00b,shepperd94},
many others endorse their use~\cite{%
polyspace,%
chapman02,%
hall00,%
halstead77,%
journals/ese/KhoshgoftaarS03,%
conf/icse/NagappanB05a,%
conf/ictai/TangK04,%
khoshgoftaar01,%
khoshgoftaar99,%
mccabe76,%
me02e,%
me03a,%
me03k,%
me03q,%
me04g,%
me07b,%
me07e,%
nagappan05,%
nikora03,%
porter90,%
srinivasan95,%
tiang95%
}. Standard verification and validation (V\&V) textbooks
\cite{rakitin01} advise using static code complexity attributes to
decide which modules are worthy of manual inspections. For several
years, the PIs have worked on-site at the NASA Independent software
Verification and Validation facility where large government software
contractors won't review software modules {\em unless} tools like
the McCabe static source code analyzer predicts that they exhibit
high code complexity measures.

One reason for the popularity of static code fault predictors is
that such features can be cheaply collected, even for very large
systems~\cite{nagappan05}. By contrast, other methods such as manual
code reviews are labor-intensive. Depending on the review method,
$8$ to $20$ lines of code (LOC) per minute can be inspected and this
effort repeats for all members of the review team, which can be as
large as four or six~\cite{me02f}.

Nevertheless, static code attributes are hardly a complete
characterization of the internals of a program module. Fenton offers
an insightful example where {\em the same} functionality is achieved
using {\em different} programming language constructs resulting in
{\em different} static measurements for that module~\cite{fenton97}.
Fenton uses this example to argue the uselessness of static code
attributes for fault prediction.

An {\em alternative interpretation} of Fenton's example is that
static attributes can never be a certain indicator of the presence
of a  fault.  Nevertheless, they are useful as probabilistic
statements that the frequency of faults tends to increase in code
modules that trigger the predictor.

Shepperd~\&~Ince and Fenton~\&~Pfleeger might reject the {\em
alternative interpretation}. They present empirical evidence  that
the McCabe static attributes offer nothing more than uninformative
attributes that are mostly the derivatives of the lines of code.
Fenton~\&~Pfleeger note that the main McCabe's attribute (cyclomatic
complexity, or $v(g)$) is highly correlated with lines of
code~\cite{fenton97}. Also, Shepperd~\&~Ince remark that ``for a
large class of software it (cyclomatic complexity) is no more than a
proxy for, and in many cases outperformed by, lines of
code''~\cite{shepperd94}.

If Shepperd \& Ince and Fenton \& Pfleeger are right, then
%\bi\item
%Learning from just lines of code should do as well as anything else;
%\item
the performance of  predictors learned by  data mining static code
features should be poor. However, this is not true, at least for the
NASA code we have
studied~\cite{me07e,me07b,me04g,me04c,me03q,me03k,me02l,me02g,me02e,jiang07}.
Using NASA data, our fault prediction models find defect
predictors~\cite{me07b} with a probability of detection ($pd$) and
probability of false alarm ($pf$) of $mean(pd,pf)=(71\%,25\%)$.
These values can be best understood via a comparison against known
industrial baselines: \bi
\item
Raffo  found that industrial reviews find \mbox{$pd=TR(35, 50,
65)\%$}\footnote{ $TR(a,b,c)$ is a triangular distribution with
min/mode/max of $a,b,c$.} of a systems errors' (for full Fagan
inspections~\cite{fagan76}) to \mbox{$pd=TR(13, 21, 30)\%$} for
less-structured inspections.
\item
Similar conclusions were made at a panel at IEEE Metrics 2002. That
panel declined to endorse claims by Fagan~\cite{fagan86} and
Schull~\cite{shull00a} regarding the efficacy of their inspection or
directed inspection methods. Rather, it concluded that manual
software reviews can find ${\approx}60\%$ of defects~\cite{shu02};
\ei All the above results were collected by  a variety of different
methods. Hence, it is problematic to directly compare our data
mining models with these industrial baselines. The least we can say
is that automatically generated defect predictors do not appear to
perform {\em worse} than industrial averages and might even perform
somewhat better. Also, while the manual methods require {\em manual}
effort, our defect predictors can be {\em automatically} and rapidly
developed, then applied to large libraries of code in a
cost-effective manner.
