

Ideally, software projects have all the resources required to ensure the quality
of their products. This is rarely true so managers are forced to spend their
QA budget according to where they believe it will be most effective. If the QA is
focused on $\{A,B,C,D\}$ then it is possible  that some defect in $\{E,F,G,...\}$ will
go unnoticed.

For five years we have been building quality predictors that can 
automatically check if  $\{E,F,G,...\}$ contain defects. 
Contrary to the pessimism of Fenton\&Plegger~\cite{fenton00b} and Shepperd\&Ince~\cite{shepperd94}, we find that that 
defect predictors learned from static code
features
can be surprisingly effective. 

Our current generation
of automatic quality predictors have encountered a ceiling effect. 
Despite
much research, the performance of these detectors has not moved from our
established baseline~\cite{me07b}.

We have hypothesized that standard
AI algorithms lack the business knowledge to recognize better
quality predictors.
To add that business knowledge,
we have proposed here to introduce more topology
into our search space. 
Using a combination of automatic rule generation and human-in-the-loop
rule maintenance, we believe we can learn better features and evaluation criteria that
will let our data miners 
break through this ceiling effect. 

If we are correct, then:
\bi
\item
Managers can safely focus of $\{A,B,C,D\}$, confident in the knowledge that
they quality predictors will raise an alert about issues in $\{E,F,G...\}$, if
such issues actually exist.
\item This proposal will create a new high water mark in automatic quality prediction.
\item This proposal will create a new paradigm in automatic quality
prediction. Instead of leaving quality prediction to some black-box AI algorithm,
we will make quality prediction a joint partnership between natural and artificially
intelligence entities.
\ei
The net result will be the delivery of higher quality software.

In terms of repeatable science, the strongest aspect of this proposal is the refutable nature
of our hypothesis.
{\bf Task5} (described in C-12)
clearly defined a set of observations that would falsify our claims.

%need methods for the rapid 
%
%\noindent
%One subtle feature of the \fig{tone} results is that, as discussed below,
%they enable an expert-in-the-loop feature discovery process.
%The {\em goal} of year one of this project is to conduct such expert-in-the-loop studies.
%The {\em outcome} of year 1 would be:
%\bi
%\item
%A set of features that experienced programmers use in their own defect prediction.
%Our design would allow defect prediction rules to access arbitrary features extracted from development artifacts
%(and not merely static code attributes).
%\item An assessment of the value of different subsets of those features
%for defect prediction.
%\ei
%
%Why use expert-in-the-loop studies?
%Is not the whole point of data mining is
% that we can dispense with live experts and, instead,
%build theories from real-world data?  Perhaps so, but
%
%
%
%We would build an  incremental knowledge capture (IKC) environment where the researchers on this project
%would monitor experienced programmers reviewing code and declaring the code ``defective'' or not.
%
%We have much experience in building such incremental rule capture environments. In the 1980s and 1990s,
%one of use (Menzies) was part of a team
%that built one of the world's fastest rule capture environments~\cite{me91a,me94,me94z,me95zb,menaco92}. At that time,
%there was much discussion about the {\em knowledge acquisition bottleneck}. That is,  a rule-based expert system  executes over rules
%and it can be a tedious task to collect such rules.
%
%
%Such a rule capture environment composes {\em feature extractors} a {\em co
%
%One reaction to the above experiments might be to
%take all possible features extractable from all possible software artifacts, give them to a data miner,
%and let the learners work out what features are best suited for defect prediction.
%
%That is {\em not} the plan of this proposal. Such a jumble of features could be difficult and expensive to collect.
%Also, different projects would support different features (e.g. an object-oriented project might generate a ``average depth of inheritance'' features
% and ``shot-gun'' approach has serious statistical drawbacks
%
%\noindent {\bf 2.1 State of the art in defect prediction:}
%
%\noindent {\bf 2.2 Summary on defect prediction:}
%
%\noindent {\bf 3. Research Thrusts }
%
%\noindent {\bf 3.1 A}
%
%\noindent {\bf 3.1.1 a2}
%
%\noindent {\bf A. Problem Description}
%
%\noindent {\bf B. Preliminary Results}
%
%\noindent {\bf C. Proposed work (Task lead: Menzies, Duration months 1 to 24)}
%
%\noindent {\bf 3.1.2 a2}
%
%\noindent {\bf A. Problem Description}
%
%\noindent {\bf B. Preliminary Results}
%
%\noindent {\bf C. Proposed work (Task lead: Cukic, Duration months 1 to 24)}
%
%\noindent {\bf 5. Schedule and Tasks}
%
%\noindent {\bf Year 1}
%
%{\em Thrust 1:}
%
%{\em Thrust 2:}
%
%\noindent {\bf Year 2}
%
%{\em Thrust 1:}
%
%{\em Thrust 2:}
%
%\noindent {\bf Year 3}
%
%{\em Thrust 1:}
%
%{\em Thrust 2:}
%
%\noindent {\bf 5. Broader Impacts}
%
%{\bf {\em Benefits to Society}}
%
%{\bf {\em Dissemination of Results}}
%
%{\bf {\em Impact on Edutation}}
%
%{\bf {\em Impact on Histocailly Underpresentated Groups in Science and Engineering:}}
%
%\noindent {\bf 7. Results from Prior NSF Support}
%
%
