


\section {Background}
\label{sect:background}


\subsection {Comparison of Multiple Estimating Methods}
\label{sect:motivation}

%As witnessed by recent international SE conferences (e.g. ESEM, PROMISE, ICSE),
%effort estimation is an active area of research.
%Further, as data mining research matures, an increasing number of estimation methods
%are being explored in the literature.



%The methodological implications of Shepperd \& Kododa's conclusions are 
%very disquieting. 


With the availability of different software cost estimating methods, it is becoming a more non-trivial task to select the appropriate modeling
methods for a particular software development situation.
Despite decades of research, there is still no consensus of which effort predictors are better or worse than any others. 
Such a comparison process is technically unavailable in experimental studies, researchers have expressed concerns and even doubt that such a ranking of appropriateness measure can ever be generated.
%
%The result of a classification/ranking procedure is a list of performance
%indicators, ranked according to their relevance to the target
%problem. Unlike dataset feature subset selection~\cite{kohavi97}
%or ensemble-based learning~\cite{Seni2010}, there is no consolidated
%theory exists in literature for estimator selection stability.
%Ranked estimator lists are highly unstable in the sense that different
%method combining with different preprocessors may yield very different
%rankings, and that a small change of the data set usually affects
%the obtained estimator list considerably. The estimator ranking
%stability issue has not been considered for its importance in the
%literature, but unfortunately, the issue has not grown into the
%main focus of research in the last few years, perhaps as a consequence
%of immediate benefits of individual development of estimators claimed
%to be more superior than the others, but limited to a very specific
%circumstance.
%
%Without being able to understand the ranking stability, it is unlikely to progress the research in the area of software cost estimation, as a consequence there is not convincing evidence to support the practical usage of the developed methods and tools available in the literature. 
%
%To derive  stable rankings about which estimator is ``best'',
%there have been attempts in trying to compare model prediction
%performance of different approaches. For example,

For example,
Shepperd and Kododa \cite{shepperd01b} compared regression, rule
induction, nearest neighbor and neural nets, in an attempt to
explore the relationship between accuracy, choice of prediction
system, and different dataset characteristics by using a simulation
study based on artificial datasets. A number of conflicting results were reported 
exist in the literature as to which method
provides superior prediction accuracy, and offered possible explanations
including the misuse of an evaluation criteria such as MMRE or a
malformed dataset being used, and all of these can
have a strong influence on the relative effectiveness of different
prediction models. 

%Their work as a {\em simulation study}
%that took a single dataset, then generated very large artificial datasets
%using the distributions seen on that data. 
%In that it is concluded the following based on a simulation study:
That it was 
generally {\em infeasible} to determine which prediction technique
is the ``best'': 

\bi
\item
{\em None}
of these existing predictors  were consistently ``best'';
\item
The accuracy of an estimate depends on the dataset characteristic
and a suitable prediction model for the dataset. 
\ei

\subsection {Ranking Instability}
More recent results suggest that it is appropriate to revisit the ranking instability hypothesis.
Menzies et al.~\cite{menzies11} applied 158 predictors to various subsets of two
COCOMO datasets. In a result consistent  with Shepperd and Kododa, 
they found the precise ranking of the 158 predictors 
{\em changed} according to 
\bi
\item the random number seeds used to generate train/test sets;
\item the performance evaluation criteria used; 
\item and which subset of the data was used.
\ei
In addition, there are 4 methods consistently outperformed the other 154
across all datasets, across 5 different random number seeds, and across
three different evaluation criteria, making the result {\em stable}. 

%There are now many datasets in public domain readily available for stability studies.
%Also,  there are now many more public domain datasets, readily available for stability studies.
The hypothesis in here is that if we are able to derive a stable conclusion using simulated data across 5 different random number seeds as source of changes, and across 3 different evaluation criteria, similar behavior should be observed when applying heterogenous datasets from public domains where they are from different sources with various differences in project characteristics.  
\fig{datasets} lists 20 real world software development project datasets which have become available at the PROMISE repository of reusable SE data\footnote{\url{http://promisedata.org/data}}.
This enables a more in-depth understanding of a more stable conclusion using real SE project data instead of simulated data (as reported by Shepperd and Kadoda~\cite{shepperd01b}) or to study merely two datasets (as reported by Menzies et al.~\cite{menzies11}). An more extensive empirical analysis can be carried out. 


\input{ekrem/data}

When previous studies and conclusions are considered, unless we address the instability issue, we cannot make conclusive remarks about neither the algorithms nor the datasets. 
Our fundamental motivations is to question the stability issue and we propose a methodology for evaluating the stability (see methodology of \fig{rank_changes}).
%Given that methodology, we will propose that if the mean-rank change among methods or datasets is $x$, then we will need a dataset or algorithm amount of more than $2$ times the value of this $x$ to be able to make stable conclusions, which allows us to select the correct predictor for software cost estimation.
%The rest of this paper explores conclusion stability over $90$ methods and $20$ datasets.

