
{\bf Task5- evaluation:}
A final evaluation $task5$ is required to assess the end results.
$Task5$ would gauge the success of this work by testing
the core claims of this work:
\bi
\item One core claim was that adding more topology to the search space would yield better
quality predictors. This claim is false if after task4, the newly
reinitialized quality detectors do not perform better than either our TSE 2007 baselines~\cite{me07b} or the initial set of rules generated by task2.
Note here that by {\em better} we do not necessarily mean
higher probabilities of detection or lower probabilities of false alarm,
Instead, it is possible that our new rules will have the {\em same}
(pd,pf) performance but will score better on the new business evaluation
criteria learned as part of this research.
\item Another core claim was that quality predictors are best
viewed as domain-specific constructs. Hence, elaborate rule customization
environments are required to tune rules to local business conditions.
This claim would be rejected if
we see that users {\em never} need to patch the rules learned
via treatment learning.
\ei
Other, more specific tests to be applied in {\b task5} are:
\bi
\item Test \#1
requires that  we can elicit from our user population
the $\alpha_i$ weights. This is false if, in task2, our business
criteria GUI can't predict if users like some 
randomly generated set of criteria.
\item
We claimed above that our process
is simple and fast to use.
This claim is false if either:
\bi
\item
 Within any month of operation,
users lose access to the ripple-down rules environment for more than half their work days while
programmers handle patch repairs. 
\item
When patch failure does {\em not} occur,
the rate of rule patches within the ripple-down rules environment is too slow; say,  more than
five minutes per rule.
\ei
\ei

{\bf Schedule:}
The project is envisioned as a 3-year collaborative effort lead by
the PI and the Co-PIs.  While some initial results indicate feasibility
and scientific promise, the planned activities are comprehensive.
Some of the tools required for the research are available in public
domain (machine learning and statistical analysis packages).  The
same holds for the software engineering datasets, with PROMISE
repository and open source project artifacts offering most of the
data we will need to conduct research.  At the same time, we will
strive to further nurture our collaboration with software development
companies and, to the extent possible, solicit additional information
primarily related to the business environment related metrics and
utility measures. 


Some of the proposed tasks require the development of the original
tools.  From our related projects, we have operational implementation
of the treatment learner (needed in Task 2). The other tasks will
require design and implementation of project specific tools, including
the business case elicitation graphical user interface (Task 1),
human-in-the-loop rule maintenance environment (Task 3) based on
ripple-down rules, support for global reorganization of these rules
(Task 4), and the tools support for the evaluation methodology (Task
5).  The preliminary schedule of the planned tasks and the division
of responsibilities between the PIs are indicated in \fig{sced}.

\begin{figure}[!t]
\includegraphics[width=6.5in]{bojan/schedule1.pdf}
\caption{Schedule}\label{fig:sced}
\end{figure}

