The research work referenced above has produced useful results.
However, there is much room for improvement; e.g. probabilities of
detection reported in recent repeatable studies, should not only be
achievable on a regular basis but improved.

It has proved difficult to achieve those improvements. In our
January 2007 TSE study~\cite{me07b} we set out to define a
repeatable experiment in learning defect  predictors. The intent of
that work was to offer a benchmark in defect prediction that other
researchers could repeat/ improve/ refute. That experiment included:
\bi
\item
Public domain data sets (from the the PROMISE
repository\footnote{\url{http://promisedata.org/repository}});
\item
Open source data mining
tools (the WEKA toolkit~\cite{witten05});
\item
Repeated  randomization of the  order of training data (to avoid
order effects);
\item
10-way cross-validation (to assess the results via data {\em not} used in training);
\item
Learning via multiple types of machine learning algorithms (rule
learners, decision tree learners, Bayes classifiers);
\item
Assessment via multiple criteria such as probability of detection ($pd$),
probability of false alarm ($pf$), and $balance$ that combines $\{pd,pf\}$ (see \fig{perform});
\item
Statistical hypothesis tests over the assessment criteria;
\item
Novel visualization methods for the results;
\item
Feature subset selection to find the most important subset of the static code
features;
\ei

\begin{wrapfigure}{r}{4.5in}
\small
\begin{tabular}{|p{.95\linewidth}|}\hline
$\{A,B,C,D\}$ are the
true negatives,
false negatives,
false positives, and
true positives
found by a defect predictor.
{\footnotesize\begin{eqnarray}
pd=&recall=&D/(B+D)\\
pf=&&C/(A+C)\\\label{eq:bal}
bal=&balance=&1 - \sqrt{\left(0 - pf\right)^2 + \left(1 - pd\right)^2}/ \sqrt{2}
\end{eqnarray}}
All these values range zero to one.
Better and {\em larger} balances fall  {\em closer} to the desired  zone
of no false alarms ($pf=0$) and 100\% detection ($pd=1$).

We deprecate other measures such as $accuracy$  and $precision$
since, in the defect data sets we are exploring,
the percent of defective examples
is usually very small (median value around 8\%).
Accuracy and precision
are poor indicators
of performance for data were the target class is so rare
(for more on this issue, see~\cite{me07e,me07b}).
\\\hline
\end{tabular}
\caption{Performance measures}\label{fig:perform}
\end{wrapfigure}
That experiment resulted in the defect predictors with
$mean(pd,pf)=(71\%,25\%)$. Best results were obtained using a
combination of a Bayes classifier and a simple pre-processing of the
numeric features\footnote{The ``logging'' pre-processor replaces all
numerics $N$ with $log(N)$. This spreads out exponential curves more
evenly across the space from the minimum to maximum values. This
``spreading'' can significantly improve the effectiveness of data
mining~\cite{me07b}.}. Bayes classifiers use Bayes' Theorem that
offers a relationship between  fragments of evidence $E_i$, a prior
probability for a class $P(H)$, and a posteriori  probability
$P(H|E)$; i.e. $P(H|E)=P(H)/ P(E)\prod_iP(E_i|H)$. When building
defect predictors, the posterior probability of each class
(``defective'' or ``defect-free'') is calculated, given the features
extracted from a module. The module is assigned to the possibility
with the highest probability. For numeric features, a feature's mean
$\mu$ and standard deviation $\sigma$ is used in a Gaussian
probability function~\cite{witten05}: $f(x) =
1/(\sqrt{2\pi}\sigma)e^{-\frac{(x-\mu)^2}{2\sigma^2}}$.
%Apart from
%doing better prior work~\cite{me04g},
%the feature subset selection
%from this study were the most interesting result. Feature subset selection (FSS)
%checks if subsets of the available features perform as well as some superset~\cite{kohavi97,hall03}.
%Menzies' FSS experiments did
%not discard everything {\em except} lines of code. Often, the selected subset
%included combinations of multiple types of features such as
%Halstead's readability
%metrics~\cite{halstead77}.
%That is,
%contrary to the pessimism of
%Fenton \& Pfleeger or Shepperd \& Ince, there is some information content
%in these static code features
%above mere lines of code counts.

Since that study, we have tried to find  better data mining
algorithms for defect prediction. To date, we have failed. Our
recent (as yet, unpublished) experiments have found no additional
statistically significant improvement from the application of the
following data mining methods: logistic regression; average
one-dependence estimators~\cite{webb05}; under- or
over-sampling~\cite{drummond03} random forests~\cite{breiman01},
RIPPER~\cite{cohen95r}, J48~\cite{quinlan92}, OneR~\cite{holte93}
and bagging~\cite{brieman96}. Only boosting~\cite{FreSch97} on
discretized data offers a statistically better result than a Bayes
classifier. However, we cannot recommend boosting: the median
improvement is quite negligible and  boosting is orders of
magnitudes slower than a simple Bayes classifier.

Other researchers have also failed to improve our results. For the
past four years, we participated in the PROMISE workshop on
repeatable software engineering experiments. The rule of that
workshop reauires that if  a paper offers an empirical conclusion,
then it must also offer the data used to reach the conclusions. That
data is stored on-line in the PROMISE repository, which we
administer and maintain for the PROMISE community. Our work on that
web site, plus our interaction with the PROMISE community, gives us
a unique insight into the empirical analysis as well as access to,
as yet, unpublished results. Consequently, we are aware of studies
by other researchers (currently under review) that tried 25 other
data mining methods in fault prediction experiments. Those studies
investigated the statistical difference of the results from 25
learners on the same datasets using the Wilcoxon signed ranked
test~\cite{wilcoxon45}. The Bayesian method discussed above ties in
first place along with 15 other methods.

How can we  explain all these failed attempts to improve fault
prediction models in repeatable experiments using same (PROMISE)
data sets? One possibility is that all these experiments use
simplistic static code features such as lines of code, number of
unique symbols in the module, etc. We agree with Shepperd \& Ince
and Fenton \& Pfleeger that such simplistic static code features are
hardly a complete characterization of the internals of a module.
Static code features might be best described having {\em limited
information content}, with two properties. Firstly, they can be
quickly and completely discovered by even simple Bayesian
classifiers. Secondly, more sophisticated methods will discover no
further information.

\begin{wrapfigure}{r}{2.5in}
\noindent
\includegraphics[angle=90,width=2in]{incrementalPC5.pdf}\newline
\includegraphics[angle=90,width=2in]{incrementalMC1.pdf}
\caption{Training set size vs $balance$.
}\label{fig:tone}
\end{wrapfigure}
We  have made two tests of this {\em limited information}
conjecture. {\em Test one} checked how {\em little}
information was required to learn a defect predictor. 
{\em Test
two} augmented the static code features with other features.
In {\em test one}, fault predictors were learned from
\mbox{$N\in\{100, 200, 300,...\}$} instances  then $Test$ed on another 100
instances. For each $N$, 10 experiments were performed where
training was conducted on $Train=90\%*N$ instances. For all
experiments, the $N,Train,Test$ instances were selected at random.

Test one was conducted on twelve data sets, all of which yielded
results like those depicted in \fig{tone}~\cite{me07i}. In that
figure, the X-axis is the size of training set and the  Y-axis is
the $balance$ measure defined in \fig{perform}.
%All results come from a hold-out set of 100 instances.
Note that the performance foes not change much regardless of whether
the model is inferred from 100 instances or from up to several
thousand instances. In fact, learning from too many training
examples was actually detrimental (witness the widening variance as
the training set increases). A Mann Whitney U test~\cite{mann47}
(95\% confidence) confirms the visual pattern apparent in
\fig{tone}: static code features used as the basis for predicting
module's fault content reveal all that they can reveal after as
little as 100 instances.

To implement {\em test two}, we linked code modules to the
requirements that prompted their development. Features were
extracted from the requirements using a lightweight linear-time text
parser. Those features included {\em weak words} such as ``TBD'' and
``but not limited to''; {\em imperatives} such as ``must'',
``should''; and {\em options} such as ``can'', and ``may''. Training
instances for the fault prediction models were then created using
just static code features, just requirements features, or both
(using a database inner join operation to connect requirements with
modules in which they have been implemented).

The results are shown in \fig{ttwo}. This figure  plots $pd$ vs $pf$
seen in the two data sets after a 10-way cross validation.
% when modules were considered defective if $\frac{M}{N}$ of the 10-ways
%classified a module as ``defective''. The curves in those plots
%result from varying $M$ from one to ten.
The ideal spot on these ROC curves is top left; i.e. no false alarms
and perfect detection ($\{pd,pf\}=\{1,0\}$). The dashed lines on
those plots show $\{pd,pf\}$ results when fault prediction models
used requirements or code features in isolation. The solid lines
show the results of models which used these two kinds of features in
combination. Note the remarkable improvement: learning from {\em
multiple perspectives} such as code {\em and} requirements measures
in conjunction, lead to dramatically improved $pd$ and $pf$.

\begin{figure}[!t]
\begin{center}
\includegraphics[width=2.3in]{pc1r.png}
\includegraphics[width=2.3in]{cm1r.png}
\end{center}
\caption{$\{pd,pf\}$ curves seen when using
code and/or requirements features.
From~\cite{jiang07}.}\label{fig:ttwo}
\end{figure}
The knowledge of the values of multiple feature sets from different
sources along the project's development life cycle leads to
conclusion that the resulting models likely represent
domain-specific, if not project-specific solutions. \fig{ttwo}
strongly suggest in favor of for using multiple feature sets from
different software artifacts.  While an exciting result, it has some
drawbacks. Different projects use different tools and build software
using different processes. Hence, different projects have access to
different sets of features which may be useful for predicting where
faults hide; e.g. an agile process may have no access to the
detailed requirement documents used to build model in \fig{ttwo}.

Other experiments  also suggest that the {\em best} set of features
are domain specific. {\em Feature subset selection} (FSS) checks if
feature subsets
performs as well as  supersets.
In all the data sets studied
in~\cite{me07b}, 38 static code features could be pruned down to
three {\em without} reducing the efficacy of the defect predictor.
Significantly,
in each data sets, FSS selected
{\em different features}. That is, FSS found no universally {\em
best} features for defect prediction.  An analogous result has been
been reported by Shepperd and Ince~\cite{shepperd94}. They reviewed
18 publications in which no {\em best} predictive feature was found.
In those publications, an equal number of studies reported that 
cyclomatic complexity is the same, is better, or is worse
than LOC in predicting defects.

\fig{sorted} explains the FSS results and the results of Shepperd
and Ince. It also cautions that the best set of features will change
from domain to domain. This figure plots the {\em information gain}
associated with 38 static code features.  This {\em infogain}
measure is used in decision tree learners~\cite{quinlan92} to select
the feature that best divides up the examples\footnote{Infogain
compares the number of bits required to encode the distribution of
defective/non-defective classes before and after dividing the
examples amongst the range of feature F.}. The first 16 features
have very similar information content. In fact, given the standard
deviations shown in the error bars of that figure, the 16th-ranked
feature is statistically insignificantly different to the top-ranked
features. Hence, minor changes in the training sample
 change what features are found to be the {\em best} software  quality predictors.


\begin{wrapfigure}{r}{3.5in}
\begin{center}
\includegraphics[width=3.5in]{infogain.pdf}
\end{center}
\caption{Mean and standard deviation (t-bars)
of InfoGain in 10*90\% random samples of defect data.
}\label{fig:sorted}
\end{wrapfigure}
Based on the above, in contrast with the current trend in the
literature, we recommend against trying supposedly better AI methods
to learn defect predictors. We make this recommendation
for two reasons.

Firstly, different  data miners have failed to improve the state of the art
in learning defect predictors. The early plateau effect of
\fig{tone} suggests that static code features yield all they can
yield after as little as a few hundred examples. This information
can be found by relatively simple learners such as Naive Bayes. That
is, the added value of more sophisticated AI data miners must be
questioned.

Secondly, there is a limited generality in the predictors learned by standard
AI data miners. \fig{ttwo} showed that combining data sources can
yield fault predictors that out-perform predictors learned from
either source in isolation. Different domains and different projets
offer different data sources and, consequently, the {\em best} set
of features to use in a quality prediction will be domain-dependent.
Further evidence for the domain dependent nature of quality
predictors comes from \fig{sorted}. Given a set of features, many of
them compete to be the {\em ``best"} one and minor variations in the
sampling methods can yield different selection of {\em best}
predictors. 
%While {\em better algorithms} may not be useful, {\em better
%data} might be. \fig{ttwo} showed that
%{\em augmenting}
%static code features with other kinds of features
%resulted in a  remarkable
%increase in defect predictor effectiveness.
%Nevertheless, it is an open question how to best augment existing
%features.
%Based on
%\bi
%\item
%Our FSS studies~\cite{me07d};
%\item
%Older work
%by Shepperd \& Ince~\cite{shepperd97};
%\item and \fig{sorted}
%\ei
%we must caution that the {\em best} features for learning defect
%predictors may be domain-specific.
%

In summary, the research challenge is clear:
\bi
\item 
We should stop trying
out supposedly better AI data miners. Instead, we must
look to
other sources of information based on the software engineering
knowledge and project's business environment if we are going to
improve on our current generation of quality predictors. 
\item
We
must acknowledge that it is insufficient to merely apply one
machine learning method and then use the model on all future
problems. Rather, we need to find cost-effective methods to quickly
learn domain dependent quality predictors.
\ei
