\section{Results}
\label{res}

\subsection{Results for Research Question 1}

\begin{figure}[!h]
  \centering
  \subfloat[Cocomo81]{\label{fig:bestKcocomo81}\includegraphics[width=0.28\textwidth]{plots/cocomo81FloatingK.png}}                
  \subfloat[Nasa93]{\label{fig:bestKNasa93}\includegraphics[width=0.28\textwidth]{plots/nasa93FloatingK.png}}
 \\
  \subfloat[Desharnais]{\label{fig:bestKDesharnais}\includegraphics[width=0.28\textwidth]{plots/desharnaisFloatingK.png}}    
\caption{Distribution of Optimum \textit{k} Values. Optimum \textit{k} value for each instance is found and these optimum values are shown in a histogram.}
  \label{FigureFloatingKHistograms}
\end{figure}

Our first research question was how we could better understand the underlying characteristics of dataset.
Actually a majority of research studies in software effort estimation tries to address this question.
However, as it was reported\cite{Kitchenham2009}\cite{Kitchenham2001} most of the methods in literature were tested on a single or a very limited number datasets, thereby reducing the credibility of the proposed method.
To avoid this pitfall, we included many datasets from different domains and different sources.
Although \textit{k}-based CBR methods have been reported to produce better results than traditional regression based methods\cite{Atkinson1994}\cite{Shepperd1997}\cite{Shepperd1996} and handle datasets with discontinuities better\cite{Kitchenham2009}, they still have limited predictive power due to the fixed number of analogies that they use.
In other words, a single best performing \textit{k} value that is yielding the lowest MRE values for the whole dataset does not necessarily produce lowest MRE values for individual projects.
We can better see this fact from Figure \ref{FigureFloatingKHistograms}.
To produce histograms given in Figure \ref{FigureFloatingKHistograms}, we calculated the optimum number of analogies, i.e. best-\textit{k} value that yielded lowest MRE value for each project of a dataset.
For a dataset of size $n$, optimum best-\textit{k} value can range from $1$ to $n-1$.
What we can see from Figure \ref{FigureFloatingKHistograms} is that for a considerable amount of projects a fixed-\textit{k} approach will be far from optimum.
Since 3 datasets were enough to illustrate our viewpoint, we did not include all ten projects into Figure \ref{FigureFloatingKHistograms}. 

In this research we propose a model called TEAK that makes use of GAC trees.
Unlike previous CBR calibration methods, TEAK does not use a fixed number of analogies for all test instances.
Instead TEAK prunes all unnecessary analogies on the basis of performance-variance and dynamically chooses the analogies to be used for each test instance.
In Figure \ref{FigureBoxplotGAC2Instances}, we can see the box plot for the number of instances that were send to GAC2 tree.
It is seen from Figure \ref{FigureBoxplotGAC2Instances} that TEAK is indeed capable of selecting various \textit{k} values dynamically to be used while building up GAC2.
As our results indicate dynamic instance selection that takes the characteristics of each dataset into account on the basis of performance-variance improves the prediction accuracies.

\begin{figure*}
\centering
\includegraphics[width=1.1\textwidth]{plots/instanceForGAC2.png}
\caption{Boxplot for the Number of Instances Send to GAC2 Tree}
\label{FigureBoxplotGAC2Instances}  
\end{figure*}

Furthermore, as we can see from MRE plots of 20 repeats on each dataset that is given in Figure \ref{fig:mreValues}, selecting small training sets on a per-instance basis making use of variance improves CBR effort estimation performance and outperforms fixed-\textit{k} approaches.

\begin{figure*}[!hp]
  \centering
  \subfloat[Cocomo81]{\label{fig:cocomo81}\includegraphics[width=0.4\textwidth]{plots/cocomo81.png}}                
  \subfloat[Cocomo81e]{\label{fig:cocomo81e}\includegraphics[width=0.4\textwidth]{plots/cocomo81e.png}}
  \\
  \subfloat[cocomo81o]{\label{fig:cocomo81o}\includegraphics[width=0.4\textwidth]{plots/cocomo81o.png}}                
  \subfloat[Nasa93]{\label{fig:nasa93}\includegraphics[width=0.4\textwidth]{plots/nasa93.png}}
  \\
  \subfloat[Nasa93c2]{\label{fig:nasa93c2}\includegraphics[width=0.4\textwidth]{plots/nasa93c2.png}}                
  \subfloat[Nasa93c5]{\label{fig:nasa93c5}\includegraphics[width=0.4\textwidth]{plots/nasa93c5.png}}
  \\
  \subfloat[Desharnais]{\label{fig:desharnais}\includegraphics[width=0.4\textwidth]{plots/desharnais.png}}                
  \subfloat[SDR]{\label{fig:sdr}\includegraphics[width=0.4\textwidth]{plots/sdr.png}}
  \\
  \subfloat[Albrecht]{\label{fig:albrecht}\includegraphics[width=0.4\textwidth]{plots/albrecht.png}}
  \subfloat[ISBSG]{\label{fig:isbsg}\includegraphics[width=0.4\textwidth]{plots/isbsg.png}}
    \caption{Log-Mre Values of Datasets for GAC2 and multiple \textit{k} Values. For each dataset, log of MRE values for 20 repeats are sorted.}
    \label{fig:mreValues}
\end{figure*}

\subsection{Results for Research Question 2}

The second research question was whether we could ease the procedure of selecting the best performing number of analogies and whether we could increase the predictive performance while doing that.
As a matter of fact, with the utilization of GAC trees in TEAK, we have saved the researcher from the task of choosing the best \textit{k} value.
Since TEAK itself builds up GAC1 and prunes irrelevancies while sending the project instances to GAC2 as well as while building up GAC2, we have left the entire best \textit{k} selection process to TEAK.
Furthermore, apart from being able to choose the number of analogies for each test instance on its own, TEAK outperforms all the other \textit{k}-based CBR methods.

When we look at the win, tie, loss values in Table \ref{table:winTieLossValues}, we see that in all ten data sets, TEAK has the highest $win-loss$ values. 
This suggests that TEAK has attained lower MRE values than all other methods.
Indeed, TEAK has a $loss$ value of $0$ for all datasets and this shows that TEAK has never been outperformed by any other method in all datasets for statistically significant cases.
Also the plots of MRE lines given in Figure \ref{fig:mreValues} supports the statistics in Table \ref{table:winTieLossValues}.
In other words, MRE values of TEAK are lower than all other methods and therefore the MRE line for TEAK lies below all the others.
Among ten datasets Albrecht seems to be the most challenging one.
From the MRE lines given in Figure \ref{fig:mreValues}, it is difficult to claim a best performing method for Albrecht dataset.

Furthermore, three datasets look like they challenge all methods: Albrecht, Cocomo81o, Nasa93c2.
Regarding the remaining seven cases:
\begin{itemize}
\item For three cases (Cocomo81, Nasa93 and SDR) we see that TEAK is the best method by an amount of more than 90\% improvement in accuracy; e.g. in SDR TEAK comes best over a  hundred times compared to the next best number of $(wins-losses)$ which is 8.
\item In Cocomo81, Cocomo81o, Cocomo81e, Nasa93, Desharnis TEAK comes best at least twice as much as the next contender.
\item Only in one case out of seven does another method come close to TEAK: In ISBSG-Banking, TEAK comes best 54 times compared to the 44 times of $k=16$.
\end{itemize}

\begin{table*}[ht]
  \centering
\caption{Win-Tie-Loss Values for TEAK and multiple \textit{k} values}
\label{table:winTieLossValues}
\subfloat[Cocomo81]{\label{tab:tableCocomo81}
  \begin{tabular}{ | l | l | l | l | l | }  
  \hline
  \textbf{Algorithm} & \textbf{Win} & \textbf{Tie} & \textbf{Loss} & \textbf{Win - Loss} \\ \hline			
TEAK&108&132&0&108\\\hline
k=bestK&52&180&8&44\\\hline
k=8&34&170&36&-2\\\hline
k=16&42&154&44&-2\\\hline
k=1&28&148&64&-36\\\hline
k=4&20&160&60&-40\\\hline
k=2&16&136&88&-72\\\hline
\end{tabular}
}
\subfloat[Cocomo81e]{\label{tab:tableCocomo81e}
  \begin{tabular}{ | l | l | l | l | l | }   
  \hline
  \textbf{Algorithm} & \textbf{Win} & \textbf{Tie} & \textbf{Loss} & \textbf{Win - Loss} \\ \hline				
TEAK&110&130&0&110\\\hline
k=bestK&72&162&6&66\\\hline
k=16&44&178&18&26\\\hline
k=8&34&168&38&-4\\\hline
k=4&26&166&48&-22\\\hline
k=1&20&134&86&-66\\\hline
k=2&6&118&116&-110\\\hline
\end{tabular}
}

\subfloat[Cocomo81o]{\label{tab:tableCocomo81o}
  \begin{tabular}{ | l | l | l | l | l | }   
  \hline
  \textbf{Algorithm} & \textbf{Win} & \textbf{Tie} & \textbf{Loss} & \textbf{Win - Loss} \\ \hline			
TEAK&22&218&0&22\\\hline
k=16&12&228&0&12\\\hline
k=8&8&232&0&8\\\hline
k=bestK&8&232&0&8\\\hline
k=4&8&228&4&4\\\hline
k=2&2&222&16&-14\\\hline
k=1&2&196&42&-40\\\hline
\end{tabular}
}
\subfloat[Nasa93]{\label{tab:tableNasa93}
  \begin{tabular}{ | l | l | l | l | l | }   
  \hline
  \textbf{Algorithm} & \textbf{Win} & \textbf{Tie} & \textbf{Loss} & \textbf{Win - Loss} \\ \hline			
TEAK&80&160&0&80\\\hline
k=bestK&44&192&4&40\\\hline
k=16&38&194&8&30\\\hline
k=8&40&182&18&22\\\hline
k=4&28&178&34&-6\\\hline
k=2&18&132&90&-72\\\hline
k=1&12&122&106&-94\\\hline
\end{tabular}
}

\subfloat[Nasa93c2]{\label{tab:tableNasa93c2}
  \begin{tabular}{ | l | l | l | l | l | }   
  \hline
  \textbf{Algorithm} & \textbf{Win} & \textbf{Tie} & \textbf{Loss} & \textbf{Win - Loss} \\ \hline			
TEAK&42&198&0&42\\\hline
k=8&28&208&4&24\\\hline
k=bestK&28&206&6&22\\\hline
k=16&26&206&8&18\\\hline
k=4&18&200&22&-4\\\hline
k=1&16&176&48&-32\\\hline
k=2&2&166&72&-70\\\hline
\end{tabular}
}
\subfloat[Nasa93c5]{\label{tab:tableNasa93c5}
  \begin{tabular}{ | l | l | l | l | l | }   
  \hline
  \textbf{Algorithm} & \textbf{Win} & \textbf{Tie} & \textbf{Loss} & \textbf{Win - Loss} \\ \hline				
TEAK&48&192&0&48\\\hline
k=16&32&208&0&32\\\hline
k=bestK&28&210&2&26\\\hline
k=8&26&208&6&20\\\hline
k=4&12&200&28&-16\\\hline
k=2&6&182&52&-46\\\hline
k=1&2&172&66&-64\\\hline
\end{tabular}
}

\subfloat[Desharnais]{\label{tab:tableDesharnais}
  \begin{tabular}{ | l | l | l | l | l | }   
  \hline
  \textbf{Algorithm} & \textbf{Win} & \textbf{Tie} & \textbf{Loss} & \textbf{Win - Loss} \\ \hline				
TEAK&64&176&0&64\\\hline
k=16&34&200&6&28\\\hline
k=8&30&202&8&22\\\hline
k=bestK&30&202&8&22\\\hline
k=4&32&192&16&16\\\hline
k=2&14&156&70&-56\\\hline
k=1&2&140&98&-96\\\hline
\end{tabular}
}
\subfloat[SDR]{\label{tab:tableSDR}
  \begin{tabular}{ | l | l | l | l | l | }    
  \hline
  \textbf{Algorithm} & \textbf{Win} & \textbf{Tie} & \textbf{Loss} & \textbf{Win - Loss} \\ \hline			
TEAK&102&138&0&102\\\hline
k=8&26&196&18&8\\\hline
k=bestK&22&198&20&2\\\hline
k=4&16&202&22&-6\\\hline
k=16&18&188&34&-16\\\hline
k=1&20&176&44&-24\\\hline
k=2&2&170&68&-66\\\hline
\end{tabular}
}

\subfloat[Albrecht]{\label{tab:tableAlbrecht}
  \begin{tabular}{ | l | l | l | l | l | }   
  \hline
  \textbf{Algorithm} & \textbf{Win} & \textbf{Tie} & \textbf{Loss} & \textbf{Win - Loss} \\ \hline				
TEAK&6&234&0&6\\\hline
k=16&6&234&0&6\\\hline
k=8&4&236&0&4\\\hline
k=4&0&240&0&0\\\hline
k=bestK&0&240&0&0\\\hline
k=2&0&234&6&-6\\\hline
k=1&0&230&10&-10\\\hline
\end{tabular}
}
\subfloat[ISBSG-Banking]{\label{tab:tableISBSG}
  \begin{tabular}{ | l | l | l | l | l | }   
  \hline
  \textbf{Algorithm} & \textbf{Win} & \textbf{Tie} & \textbf{Loss} & \textbf{Win - Loss} \\ \hline			
TEAK&54&186&0&54\\\hline
k=16&44&196&0&44\\\hline
k=8&36&196&8&28\\\hline
k=bestK&34&200&6&28\\\hline
k=4&18&194&28&-10\\\hline
k=2&8&154&78&-70\\\hline
k=1&6&154&80&-74\\\hline
\end{tabular}
}
\end{table*}
