\subsection{{\Large {\W}2} vs SEESAW}

\input{figs/w2-vs-seesaw.tex}

Average median and spread results over the 30 trials are shown in ~\fig{w2-seesaw-comparison}. The fourth column in each group (labeled ``Median Reduc'') shows the relative change in effort, defect, months found by {\W}2 or SEESAW. A {\em negative} amount in this column denotes an optimization failure (increased defect, effort, months).
Note that such negative results occur only in a single result.

The ``Win'' column indicates any member of a pair that was both statistically and significantly different. Note that for most pairs, the results are not statistically significantly different (Mann-Whitney, 95\% confidence level).

\begin{figure}[!t]
\center
\includegraphics[width=3in]{figs/changes.pdf}
\caption{Range of changes in median and spread generated by applying
the recommendations of either {\W} or SEESAW. The median observed
changes were (34, 34)\% for (medians, spreads), respectively. For the sake
of brevity, this
graph ignores the  -94\% outlier value seen in OSP2 defects for SEESAW.
}\label{fig:changes}
\end{figure}

Before commenting on SEESAW vs {\W}2, we first note that our results should encourage more use
of quality optimization. Observe that, in the majority of cases, {\em quality optimization works} regardless of how it is implemented (e.g. CBR vs parametric models). In the 48 experiments of \fig{w2-seesaw-comparison}, positive quality improvements were seen for $46/48=96\%$ experiments (the 2 exceptions are in the effort reductions of SEESAW for COC81 OSP2 and defect reductions of SEESAW for COC81 OSP).

Another result that should encourage more use of quality optimizers is the reduction in the spreads. In all but one experiment the amount of uncertainty in the median estimates was reduced. As shown in \fig{changes}, the reduction in the spread was usually over 34\%. This is an advantage of quality optimizers since uncertainty is an serious issue that plagues the managers of software engineering projects. As shown in \fig{changes}, the expected median reduction in any quality estimate was  only 34\%. Note that if this were otherwise, then that would be a somewhat damning critique of current software engineering practices. To see this, consider the implications of  quality optimizers finding recommendations that resulted in an order of magnitude reduction in effort {\em and} defects {\em and} development time. That would suggest that the managers of software engineering projects are routinely missing changes that would significantly improve their projects.

Another feature to note is that the median month optimizations for both SEESAW and {\W}2 are below 20\% for $7/8=87.5\%$ of cases. That is, data sets can contain an an inherent set of constraints that cannot be changed, even by smart algorithms. Certainly, we can fine tune the structure of a project to obtain some improvements in effort, defects, and months but managers should not expect a magic silver bullet that offers orders of magnitude improvement in their software process.

We conducted statistical tests on each pair of {\W}2 vs SEESAW improvements in median/spread for each query.
A Mann Whitney U test (95\% confidence) was performed on the two sets of reduction distributions from each comparison. The statistical tests are summarized in \fig{win-loss-tie}. Note that, in the majority case ($\frac{18}{24}$), {\W}2's case-based reasoning performs as well as $SEESAW's$ parametric modeling.

\input{figs/win-loss-tie.tex}

Also, when the performance results were different, case-based reasoning did better than parametric modeling ($\frac{6}{24}$) sometimes spectacularly so, Observe the median reductions for NASA93 OSP (above the center divider of \fig{w2-seesaw-comparison}): {\W}2 offered median reductions of 72\%, 69\%, and 43\% compared to SEESAW's 22\%, 37\%, 13\% for defects, effort, and months respectively.

In summary, the simple case-based reasoning of {\W}2 performs just as well, or better,
than SEESAW's elaborate parametric modeling.

\subsection{{\Large {\W}} Performance Across Multiple Datasets}

Because {\W} makes no underlying model assumptions, we aren't limited to USC COCOMO for our performance evaluations. To demonstrate the effectiveness of {\W} in any data environment, we offer median reductions for effort reduction for five arbitrary datasets from \texttt{http://promisedata.org/data}. The model-agnostic simpilicity of {\W} made implementing these tests easy as one need only describe a query space and a target utility measure. In the case of these five datasets, software effort was the common target for reduction. 

Given that we did not have access to case studies as we did with NASA93 and COC81 (ground, flight, osp, and osp2) for these datasets, synthetic queries were developed. Three queries were generated for each of the five datasets. The first contained the entire space of possible project attribute values (All), representing complete freedom to recommend any change within the space. The other two queries were generated by randomly choosing 50\% of each attribute values from either the lower, middle, or upper ranges for each project attribute (Proj1, Proj2). These queries represent more common restrictions on possible changes for a given software project.

Effort reductions can be seen in ~\fig{other-reduce-values} and ~\ref{fig:other-reduce}. The chart in ~\fig{other-reduce-values} shows strong improvement in median effort for the Telecom and Miyazaki datasets, with strong performance in spread reduction across all datasets. While the Finnish, China, and Kemerer datasets show only marginal or no improvement in median effort, the certainty of their estimations is improved via a reduction in spread.

\begin{figure}[!h]
\center
\includegraphics[width=3in]{figs/noncocomo-reduce.pdf}
\caption{Plot showing the distribution of median and spread reductions in software effort for five unique datasets.
}\label{fig:other-reduce}
\end{figure}

\begin{figure}[!h]
\begin{center}
\begin{tabular}{|c|c|r|r|}\cline{3-4}
 \multicolumn{2}{c}{~}& \multicolumn{2}{|c|}{Improvement}\\\hline
dataset&query $q$&median&spread\\\hline
Telecom  & Proj1 & 96\% & 23\%\\
Telecom  & Proj2 & 91\% & 41\%\\
Telecom  & All   & 86\% & 28\%\\
Miyazaki & All   & 78\% & 33\%\\
Miyazaki & Proj2 & 69\% & 21\%\\
Miyazaki & Proj1 & 53\% & 67\%\\
Finnish  & All   & 22\% & 31\%\\
Finnish  & Proj2 & 11\% & 27\%\\
Finnish  & Proj1 & 4\% & 25\%\\
China    &  All  & 20\% & 55\%\\
China    & Proj2 & 14\% & 43\%\\
China    & Proj1 & \cellcolor[rgb]{0.8,0.8,0.8}0\% & 13\%\\
Kemerer  & Proj1 & 21\% & 61\%\\
Kemerer  & Proj2 & \cellcolor[rgb]{0.8,0.8,0.8}0\% & 49\%\\
Kemerer  & All   & \cellcolor[rgb]{0.8,0.8,0.8}-4\% & 53\%
\\\hline
\multicolumn{2}{r|}{median} & 21\% & 33\% \\\cline{3-4}
\end{tabular}
\end{center}
\caption{Effort estimation improvements ($100*\frac{initial-final}{intial}$)
for five unique datasets. Sorted by median improvement. Gray cells represent no improvement in effort estimates.}
\label{fig:other-reduce-values}
\end{figure}

\subsection{Comparing Drastic Changes to {\Large {\W}}}

Prior work considered explored impact of so-called {\em drastic} changes to software projects~\cite{me09a}. A drastic change occurs when the recommendation falls outside the defined projects ranges of a software project. In other words, when the recommended course of action is dramatic. For example, with the OSP NASA case study (~\fig{cases}), attempting to improve programmer language and tool experience (ltex) to 5 (very high, 6+ years experience), would be a drastic change as the maximum defined value for ltex is 4 (high). 

To test this, {\W}2's recommendations for the four NASA case studies (~\fig{cases}) in the NASA93 dataset were overridden with the drastic changes from ~\fig{drastic-overview}. For 3 distinct drastic changes, {\W}2 attempted to apply the drastic changes until no improvement was measured, then reported the median effort, defects, and months for that change. Note that not all changes from ~\cite{me09a} were applicable, due to a lack of extreme cases in the NASA93 dataset.

The results for these changes are reported in Figures ~\ref{fig:drastic-flight},~\ref{fig:drastic-ground},~\ref{fig:drastic-osp},~\ref{fig:drastic-osp2}. Of the 12 comparisons, in only one case does {\W}2 perform better statistically and significantly better than the three drastic changes. However, in terms of median reductions, {\W}always performs in the top 50\% of of cases. Most importantly, even when compared to extreme project recommendations, {\W} is not constrained by the limited project attribute ranges allowed for its recommendations.

\begin{figure}[!t]
\begin{center}
\begin{tabular}{c l l}
  & \textbf{Drastic Change}   & \textbf{Attribute Effects}\\\hline
1 & Improve Process Maturity  & pmat = 5\\\hline
2 & Improve Tools\&Techniques & time = 3; stor = 3;\\
  &                           & pvol = 2; tool = 5; site = 6\\\hline
3 & Reduce Functionality      & data = 2;\\\hline
\end{tabular}
\end{center}
\caption{Examples of drastic changes to software projects.}
\label{fig:drastic-overview}
\end{figure}

\input{figs/drastic.tex}

%\begin{itemize}
%\item perform in the upper ranges
%\item some problems
%\item figs: ~\ref{fig:drastic-ground} ~\ref{fig:drastic-flight} ~\ref{fig:drastic-osp} ~\ref{fig:drastic-osp2}
%\end{itemize}
