%\documentclass[12pt,journal,draftcls,letterpaper,onecolumn]{IEEEtran}
\documentclass[9pt,journal,cite, compsoc,twocolumn]{IEEEtran}
%\documentclass[9.5pt,journal,cite,final,finalsubmission,twocolumn]{IEEEtran}
%\documentclass[journal,compsoc,twocolumn]{IEEEtran}
% *************************************************************************

%\makeatletter
%  \AddToShipoutPicture{%
%    \setlength{\@tempdimb}{.5\paperwidth}%
%    \setlength{\@tempdimc}{.5\paperheight}%
%    \setlength{\unitlength}{1pt}%
%      \makebox(600,900){\rotatebox{45}{\textcolor[gray]{0.80}{\fontsize{9cm}{9cm}\selectfont{Draft}}}}}
%\makeatother
%
% If the IEEEtran.cls has not been installed into the LaTeX system files,
% manually specify the path to it:
% \documentclass[journal,draftcls,onecolumn]{../sty/IEEEtran}
% Different font in captions
\newcommand{\captionfonts}{\small}

\makeatletter  % Allow the use of @ in command names
\long\def\@makecaption#1#2{%
  \vskip\abovecaptionskip
  \sbox\@tempboxa{{\captionfonts #1: #2}}%
  \ifdim \wd\@tempboxa >\hsize
    {\captionfonts #1: #2\par}
  \else
    \hbox to\hsize{\hfil\box\@tempboxa\hfil}%
  \fi
  \vskip\belowcaptionskip}
\makeatother   % Cancel the effect of \makeatletter

\usepackage{colortbl}
\newcommand{\shade}{\cellcolor[rgb]{0.8,0.8,0.8}}
% some very useful LaTeX packages include:
\usepackage{url}
\usepackage[all]{xy}
\usepackage{cite}      % Written by Donald Arseneau
                        % V1.6 and later of IEEEtran pre-defines the format
                        % of the cite.sty package \cite{} output to follow
                        % that of IEEE. Loading the cite package will
                        % result in citation numbers being automatically
                        % sorted and properly "ranged". i.e.,
                        % [1], [9], [2], [7], [5], [6]
                        % (without using cite.sty)
                        % will become:
                        % [1], [2], [5]--[7], [9] (using cite.sty)
                        % cite.sty's \cite will automatically add leading
                        % space, if needed. Use cite.sty's noadjust option
                        % (cite.sty V3.8 and later) if you want to turn this
                        % off. cite.sty is already installed on most LaTeX
                        % systems. The latest version can be obtained at:
                        % http://www.ctan.org/tex-archive/macros/latex/contrib/supported/cite/

\usepackage{graphicx}  % Written by David Carlisle and Sebastian Rahtz
                        % Required if you want graphics, photos, etc.
                        % graphicx.sty is already installed on most LaTeX
                        % systems. The latest version and documentation can
                        % be obtained at:
                        % http://www.ctan.org/tex-archive/macros/latex/required/graphics/
                        % Another good source of documentation is "Using
                        % Imported Graphics in LaTeX2e" by Keith Reckdahl
                        % which can be found as esplatex.ps and epslatex.pdf
                        % at: http://www.ctan.org/tex-archive/info/

%\usepackage{psfrag}    % Written by Craig Barratt, Michael C. Grant,
                        % and David Carlisle
                        % This package allows you to substitute LaTeX
                        % commands for text in imported EPS graphic files.
                        % In this way, LaTeX symbols can be placed into
                        % graphics that have been generated by other
                        % applications. You must use latex->dvips->ps2pdf
                        % workflow (not direct pdf output from pdflatex) if
                        % you wish to use this capability because it works
                        % via some PostScript tricks. Alternatively, the
                        % graphics could be processed as separate files via
                        % psfrag and dvips, then converted to PDF for
                        % inclusion in the main file which uses pdflatex.
                        % Docs are in "The PSfrag System" by Michael C. Grant
                        % and David Carlisle. There is also some information
                        % about using psfrag in "Using Imported Graphics in
                        % LaTeX2e" by Keith Reckdahl which documents the
                        % graphicx package (see above). The psfrag package
                        % and documentation can be obtained at:
                        % http://www.ctan.org/tex-archive/macros/latex/contrib/supported/psfrag/

%\usepackage{subfigure} % Written by Steven Douglas Cochran
                        % This package makes it easy to put subfigures
                        % in your figures. i.e. "figure 1a and 1b"
                        % Docs are in "Using Imported Graphics in LaTeX2e"
                        % by Keith Reckdahl which also documents the graphicx
                        % package (see above). subfigure.sty is already
                        % installed on most LaTeX systems. The latest version
                        % and documentation can be obtained at:
                        % http://www.ctan.org/tex-archive/macros/latex/contrib/supported/subfigure/

\usepackage{url}       % Written by Donald Arseneau
                        % Provides better support for handling and breaking
                        % URLs. url.sty is already installed on most LaTeX
                        % systems. The latest version can be obtained at:
                        % http://www.ctan.org/tex-archive/macros/latex/contrib/other/misc/
                        % Read the url.sty source comments for usage information.

\usepackage{stfloats}  % Written by Sigitas Tolusis
                        % Gives LaTeX2e the ability to do double column
                        % floats at the bottom of the page as well as the top.
                        % (e.g., "\begin{figure*}[!b]" is not normally
                        % possible in LaTeX2e). This is an invasive package
                        % which rewrites many portions of the LaTeX2e output
                        % routines. It may not work with other packages that
                        % modify the LaTeX2e output routine and/or with other
                        % versions of LaTeX. The latest version and
                        % documentation can be obtained at:
                        % http://www.ctan.org/tex-archive/macros/latex/contrib/supported/sttools/
                        % Documentation is contained in the stfloats.sty
                        % comments as well as in the presfull.pdf file.
                        % Do not use the stfloats baselinefloat ability as
                        % IEEE does not allow \baselineskip to stretch.
                        % Authors submitting work to the IEEE should note
                        % that IEEE rarely uses double column equations and
                        % that authors should try to avoid such use.
                        % Do not be tempted to use the cuted.sty or
                        % midfloat.sty package (by the same author) as IEEE
                        % does not format its papers in such ways.

\usepackage{amsmath}   % From the American Mathematical Society
                        % A popular package that provides many helpful commands
                        % for dealing with mathematics. Note that the AMSmath
                        % package sets \interdisplaylinepenalty to 10000 thus
                        % preventing page breaks from occurring within multiline
                        % equations. Use:
%\interdisplaylinepenalty=2500
                        % after loading amsmath to restore such page breaks
                        % as IEEEtran.cls normally does. amsmath.sty is already
                        % installed on most LaTeX systems. The latest version
                        % and documentation can be obtained at:
                        % http://www.ctan.org/tex-archive/macros/latex/required/amslatex/math/



% Other popular packages for formatting tables and equations include:

% Frank Mittelbach's and David Carlisle's array.sty which improves the
% LaTeX2e array and tabular environments to provide better appearances and
% additional user controls. Array.sty is already installed on most systems.
% The latest version and documentation can be obtained at:
% http://www.ctan.org/tex-archive/macros/latex/required/tools/

% Mark Wooding's extremely powerful MDW tools, especially mdwmath.sty and
% mdwtab.sty which are used to format equations and tables, respectively.
% The MDWtools set is already installed on most LaTeX systems. The lastest
% version and documentation is available at:
% http://www.ctan.org/tex-archive/macros/latex/contrib/supported/mdwtools/

% V1.6 of IEEEtran contains the IEEEeqnarray family of commands that can
% be used to generate multiline equations as well as matrices, tables, etc.


% Also of notable interest:

% Scott Pakin's eqparbox package for creating (automatically sized) equal
% width boxes. Available:
% http://www.ctan.org/tex-archive/macros/latex/contrib/supported/eqparbox/




% *** Do not adjust lengths that control margins, column widths, etc. ***
% *** Do not use packages that alter fonts (such as pslatex).         ***
% There should be no need to do such things with IEEEtran.cls V1.6 and later.


% correct bad hyphenation here
\hyphenation{op-tical net-works semi-conduc-tor}


\newenvironment{smallitem}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{itemize}
  \setlength{\leftmargin}{.2in}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{itemize}}

\newenvironment{smallenum}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{enumerate}
  \setlength{\leftmargin}{.2in}  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{enumerate}}


\newcommand{\sect}[1]{\S\ref{sec:#1}}
\newcommand{\bd}{\begin{description}}
\newcommand{\ed}{\end{description}}
\newcommand{\bi}{\begin{smallitem}}
\newcommand{\ei}{\end{smallitem}}
\newcommand{\be}{\begin{smallenum}}
\newcommand{\ee}{\end{smallenum}}
\newcommand{\fig}[1]{Figure~\ref{fig:#1}}
\newcommand{\eq}[1]{Equation~\ref{eq:#1}}
\newcommand{\tion}[1]{\S\ref{sec:#1}}
\newcommand{\aive}{{\"aive}\ }

%\newcommand{\boxplot}[5]{
%	\begin{picture}(100,7)
%		%\put(#1,4){\circle*{2}}
%		\put(#3,4){\circle*{4}}
%                %\put(#5,4){\circle*{1}}
%		\put(50,0){\line(0,1){7}}
%                \linethickness{0.25mm}
%                \put(#2,4){\line(1,0){#4}}		
%	\end{picture}
%} 
%
%------------------------------------------------------------------------- 
\begin{document}
\title{Recognizing Inappropriate Changes for Software Projects}
\author{Tim Menzies\thanks{
Part of the research described in this paper was carried out at the Jet Propulsion Laboratory, California Institute of Technology, under a contract with the National Aeronautics and Space Administration.
Reference herein to
any specific commercial product, process, or service by trade name,
trademark, manufacturer, or otherwise, does not constitute or imply
its endorsement by the United States 
Government.
}%
,  Steve Williams, Oussama El-Rawas. Barry Boehm, Jairus Hihn\thanks{
Dr. Menzies, Mr. Williams, Mr. El-Rawas,  are with the Lane Department
of Computer Science and Electrical Engineering, West Virginia University, USA:
\protect\url{tim@menzies.us}, \protect\url{swilli12@mix.wvu.edu},%
\protect\url{oelrawas@mix.wvu.edu}.}%
\thanks{Dr. Boehm is with the 
Computer Science Deptartment of Univesity of Southern California, USA,%
\protect\url{boehm@sunset.usc.edu}}%
\thanks{Dr. Hihn is with the Jet Propulsion Laboratory, %
California Institure of Technology, California, USA, \protect\url{jairus.hihn@jpl.nasa.gov}}}



\maketitle
\thispagestyle{empty}

%\IEEEaftertitletext{\vspace{-1\baselineskip}\noindent
\begin{abstract}
Before performing {\em drastic changes} to a project,  it is  prudent
explore alternatives within the current structure of a
project.  An alternative to drastic change are {\em internal changes} that
adjust current options within a software project.   In this paper, we  show  that the
effects of numerous internal changes can out-weigh the effects of  drastic
changes.  That is,  the benefits of drastic change can often be achieved
without disrupting a project.

The key to our technique is SEESAW, a novel stochastic stability tool that
(a)~considers a very large set of minor changes using stochastic sampling; and
(b)~carefully selects the right combination of effective minor changes.

Our results show, using SEESAW, project managers have more project improvement options than
they currently realize.  This result should be welcome news to  managers
struggling to maintain control and continuity over  their project in the face
of multiple demands for drastic change.
\end{abstract}



%------------------------------------------------------------------------- 




\section{Introduction}
There are many software engineering (SE) technologies that a manager
might apply in the hope of improving how a project develops software.
Some of these technologies are {\em paper-based} methods such as
the checklists proposed by orthogonal defect classification~\cite{chill92}.
Other techniques are
{\em tool-based} such as  using the new generation of functional
programming languages like
OCaml\footnote{\protect\url{http://caml.inria.fr/}}. Yet other
techniques are more {\em process-based} including  process improvement
initiatives; changing an organization's hiring practices; a continual
renegotaion of the requirements as part of a agile software
development~\cite{port08}; or applying value-based SE principles
to select the right balance of defect removal methods while rushing
a product to market~\cite{bi05}.


SE technologies are inappropriately applied to projects when  
project manager do not assess the {\em benefits} of proposed  SE technology options against their
associated {\em drawbacks}.
For example, to improve personnel via better hiring
practices, it may be necessary to fire consultants and hire an
entirely new development time.  Also reducing quality (by decreasing
defect removal methods) or  relaxing schedule (to deliver a product
later to market) may effect profitability if a product is late to
market or if it contains too many defects.

Assessing competing technologies is not an easy task.
In theory, the value of different technologies can be assess via
``what-if'' queries over  software process models.
Each such query would adjust model inputs to reflect one possible change and change ``X'' vs ``Y''
could then be assessed using the predictions for cost, quality, etc, made by the model outputs.
In practice, such what-if queries are complicated by the {\em prediction variance problem}.
Process models can exhibit a very large variance in their predictions.
For example, inside the
COCOMO~\cite{boehm00b} effort estimation model are two ``tuning'' parameters $(a,b)$ that model
linear and exponential effects (respectively) on the output prediction.
Baker~\cite{baker07}
studied the range of $(a,b)$ learned from thirty 90\% random samples of some NASA data
The ranges varied widely:
$\left( 2.2 \le a \le 9.18 \right) \wedge \left(0.88 \le b \le 1.09\right)$ 
Such large variations make it possible to misunderstand the effects of SE technologies.
For example, suppose some some SE technology doubles productivity but $a$ moves from 9 to 4.5.
The improvement resulting from that technology would be obscured by the prediction variance.

Before we can technology options, we need to tame prediction variance.
Accordingly, we explore its origins.
The predictions of a model about a software engineering project are altered by 
project variables $P$ and tuning variables $T$:
\[
prediction = model(P, T)
\]
For example, 
consider the following simplified COCOMO model, 
\begin{equation}\label{eq:simp}
effort = a \cdot LOC^{b +  pmat}\cdot acap  
\end{equation}
Here, $(a,b)$ are the variables mentioned above that control the linear and exponential
effects (respectively) on model estimates.
Also,
$pmat$ (process maturity) and $acap$ (analyst capability) are project choices
adjusted by managers. 
That is, the tuning options $T$ are the range of $(a,b)$ and the 
project 
options  $P$ are the range of $pmat$ and $acap$.

In much of our prior work
we have assumed that the
tuning options $T$ are the dominate influence on the prediction variance.
Under this assumption, we cannot use a process model to assess different SE technologies until {\em after} we have reduced
the tuning variance (which, as shown by our prior work, is a difficult task~\cite{baker07,chen05,jalali07,me06d}). 

This paper explores an alternative assumption; namely that the
project options $P$ are the dominate  influence on the predictions.
If this is true, then it should be possible to control prediction variance by controlling the project options $P$, without
requiring local data to constrain the tuning options $T$. 
This in, in turn, would enable the evaluation of competing SE technology options as follows:
\bi
\item Given sets of technology options $O=o_1,o_2,...$;
\item Given a range of project options $P$
\item Given a range of tuning options $T$;
\item Given a model that can convert $P$ and $T$ to an estimate we want to minimize (e.g. defects, or development effort/time);
\item Search for the smallest subset of $P$ which, when combined with $o_i$ most minimizes the estimates; 
\item Return the technogy option that yields minimum estimates; i.e.
\begin{equation}\label{eq:argmin}
\arg\min_i \left(\overbrace{p \subseteq \left(o_i \cap P\right)}^{AI search}, \underbrace{t \subseteq T, model(p,t)}_{Monte Carlo}\right) 
\end{equation}
\ei
In our method, an AI search algorithm constrains the set of project options. Each reduced set is scored by a Monte Carlo
simulation that selects model inputs at random from:
\bi
\item
The constraints found by the AI and 
\item The space of possible tuning options.
\ei
In the case where $P$ dominates the estimates, then the distribution of estimates generated by 
SE technology options $o_i$ should be  distinguishable from  those generated by $o_j$.
To demonstrate this method, consider the following case study.
In prior work we identified nine changes that are often proposed to fix problematic projects and/or software
development companies~\cite{boehm99}. These changes are shown in \fig{dchange}. While not an exhaustive list of {\em all} possible
changes, they serve to illustrate the range of SE technologies that might be applied to a problematic project:
\bi
\item Change \#2 could
use 
the latest generation of visual programming languages~\cite{me00v}; 
\item
Change \#4
could use be implemented using, say,
architectural
description languages~\cite{shaw96}; 
\item
Change \#6 could use technology develped by the software process community~\cite{paulk99};
\item
Change \#5, and \#9 could results from value-based sftware egnineering.
to decide where to make cost/quality trade-offs~\cite{bi05}.
\ei
\begin{figure}[!t]
\footnotesize
\begin{tabular}{c@{~}p{.4\linewidth}p{.5\linewidth}}
$o_i$&Change & {\bf Possible undesirable impact}\\\hline
$o_1$&Improve personnel & Firing and re-hiring personnel leading to wide-spread union unrest.\\\hline
$o_2$&Improve tools, techniques, or  
development platform & Changing operating systems, IDEs, coding languages\\\hline
$o_3$&Improve precedentness / \newline development flexibility &
Changing the goals
of the project and the development method.\\\hline
$o_4$&Increase architectural \newline analysis /  risk resolution &
Far more elaborate early life cycle analysis.\\\hline
$o_5$&Relax schedule  & Delivering the system later.\\\hline
$o_6$&Improve process maturity & May be expensive in the short term.\\\hline
$o_7$&Reduce 
functionality & Delivering less than expected.\\\hline
$o_8$&Improve the team & Requires effort on team building.\\\hline

$o_9$&Reduce quality 
& Less user approval, smaller market.
\end{tabular}
\caption{Options = $\{o_1, o_2,o_3,o_4,o_5,o_6,o_7,o_8,o_9\}$ from~\cite{boehm99}.}\label{fig:dchange}
\end{figure}
Consider a process model with a set of possible $inputs$. Any projects studied by that model
must be described in terms of a subset of those inputs; i.e. the project options $P$ are a 
subset the inputs  
($P \subseteq inputs$). 
The project changes of \fig{dchanges} can be mapped into changes to $P$.
For example, if we are using the model of \eq{one}, the applying   Change \#6 (improve process
maturity) would {\em increase} the $pmat$ value.
We distinuish two kinds of  changes to $P$: {\em internal} or {\em drastic} changes:
\bi
\item
{\em Internal changes} are some subset of the project options; i.e. $internal \subseteq P$. Internal
changes can be effected by the manager of a project.
\item
Drastic changes requiring moving the project outside its current description; i.e.  
$drastic \subseteq (inputs - P)$. As shown in the right-hand-side colum of \fig{dchanges}, such
drastic changes may be highly disruptive. Hence, they often require
the consent of upper management.
\ei
For example, 
for projects with  a range
of reliability options ``low'' to ``high'':
\bi
\item  A {\em drastic change} would be to suddenly
demand that a project becomes a 
``very high'' reliability project.
\item
An {\em internal change} would be setting reliability to a value within
``low'' to ``high'';
\ei
%\begin{figure}
%\begin{center}
%\includegraphics[width=2in]{comparesReport_lc.pdf}
%\end{center}
%\caption{Differences between predicted and actual seen in 30 experiments with NASA data 
%where
%COCOCOMO models were tuned using 90\% of the data, then tested on the remaining 10\%. 
%Tuning was performed
%using Boehms {\em local calibration} method~\cite[p526-529]{boehm81}. 
%Data was taken from 
%\protect\url{http://promisedata.org/repository/data/nasa93/nasa93.arff}. Results from \cite{baker07}.
%}\label{fig:ohcrap}
%\end{figure}

%Recently~\cite{me09a,me07f,me08c},
%we have observed project variance domination in 
%two software 
%process model: the
%COCOMO effort predictor~\cite[p29-57]{boehm00b}, and 
%the COQUALMO defect predictor~\cite[p254-268]{boehm00b}.
%This paper reports an experiment that checks if reducing project, but not tuning variance,
%is enough to
% unambigous rank change ``X''
%vs change `Y'' using COCOMO and COQUALMO.
%
%Our tool in this experiment is SEESAW.
%To use SEESAW,  a manager maps their proposed process 
%change down to their effects on COCOMO/COQUALMO input 
%parameters. 
%SEESAW is
%a 
%{\em stochastic search} tool that explores the space of project options to comparatively assess
%the value of technology ``X'' vs ``Y''.  
%When controlling mission-critical software, stochastic sampling is not desirable. However,
%when studying the uncertainty associated with unknowns about software processes, stochastic algorithms
%like SEESAW are very useful.
%SEESAW makes no attempt to constraint tuning variance. Rather, it randomly samples across
%the space of all tuning effects. Hence, the conclusions reached by SEESAW are stable {\em accoss the space
%of possible tunings}. 
%
%For example, your manager has returned from ICSE`09
%excited about the next new thing in software engineering.
%``We need to drastically alter our projects,'' she says, ``in order to support (say) automatic
%formal methods, agile process,  etc.''. 
%This change
%may entail switching from Windows to LINUX (since there are more 
%research prototypes for automated formal methods running on LINUX);
%stopping production
%while engineers retrain;  or, worse, firing the engineers and making new hires. 
%Alternatively,
%your manager knows that a budget cut is imminent and 
%wants to discuss drastic reduction in project scope.
%This change may
%alter projected revenue streams 
%(since
%the next release will have less features and hence less market
%appeal). Changing revenue projections can be very detrimental to
%the rate
%of investor capital received by a software development company.
%For a list of other drastic changes, see
%\fig{dchange}.
%
\newcommand{\boxplot}[5]{
\begin{picture}(100, 6)
\put(#1, 2){\line(0, 1){4}}
\put(#1, 4){\line(1, 0){#2}}
\put(#3, 4){\circle*{3}}
\put(#3, 4){\line(1, 0){#4}}
\put(#5, 2){\line(0, 1){4}}
\put(50, 3){\line(0, 1){4}}
\end{picture}
}
%We contrast the results of {\em drastic change} and  {\em AI search} with the current project structure
%as follows.
%After an AI search within the bounds of the current project,
%the resulting recommendations can be implemented by the manager running that project.
%For example, the client may be happy with reliability levels from moderate to high
%and the project manager is free to adjust the intended reliability to moderate.
%On the other hand, {\em drastic change} 
%requires the involvement of senior management since it
%falls outside of the current definition of a project.
%For example, a drastic change might require firing the entire current development
%team.
%
\begin{figure}[!t]
\noindent
{\scriptsize\begin{center}
\begin{tabular}{ rc @{} c @{} c @{}}
        & & Defects / KLOC\\
 Change & & (normalized 0..100, min..max)\\
\hline
 {\bf SEESAW}&  & \boxplot{2.2727273}{2.2727273}{4.5454545}{2.2727273}{6.818182} &  \\
 Improve pcap &  & \boxplot{11.363636}{2.2727273}{13.636364}{2.2727273}{15.909091} &  \\
 Improve tool/tech/plat &  & \boxplot{20.90909}{2.7272727}{23.636364}{5.0}{28.636364} &  \\
 Reduce functionality &  & \boxplot{26.363636}{3.6363637}{30.0}{3.1818182}{33.18182} &  \\
 Improve pmat &  & \boxplot{41.81818}{6.3636365}{48.18182}{16.363636}{64.545456} &  \\
 Improve prec/flex &  & \boxplot{47.272728}{7.2727275}{54.545456}{16.363636}{70.90909} & \\
 Improve team &  & \boxplot{49.545456}{7.7272725}{57.272728}{12.727273}{70.0} &  \\
 Relax schedule &  & \boxplot{57.727272}{5.909091}{63.636364}{10.0}{73.63636} &  \\
 Arch/risk resolution &  & \boxplot{52.272728}{13.181818}{65.454544}{10.454545}{75.90909} & \\
 Do nothing &  & \boxplot{81.818184}{4.090909}{85.90909}{2.7272727}{88.63636} &  \\
 Reduce quality &  & \boxplot{92.72727}{2.7272727}{95.454544}{2.2727273}{97.72727} & \\ \hline
 &  & 50\% & 
\end{tabular}
\end{center}}
\caption{
Defect predictions from changing
a project (25 to 75\% percentile range from 100 simulations;  median values shown as a dot).
Note that ``do nothing'' (second last line) means running the projects without imposing
drastic or internal changes.
This result come from
case studies with
four NASA case projects
and three variants of a standard commercial system.
}\label{fig:results}
\end{figure}

\fig{results} shows the results of applying \eq{argmin} to the 
The main result of this paper is that, using SEESAW, it is possible to show that
\begin{quote}
{\em Combinations of internal changes
usually perform as well or better than
drastic changes}.
\end{quote}
That is, using internal changes,
the benefits of drastic change can be achieved without
project disruption.
For example,
\fig{results} presents some of the results from later in this paper: 
projects treated with SEESAW's internal change recommendations
had fewer defects that a project altered by any of the drastic project chagnes
of \fig{dchange}.


 
%This result should be welcome news to  managers
%struggling to maintain control and continuity over  their project in the face
%of multiple demands for drastic change.
%
%XXX related work. bayes etc.
%
%More formally, we say that we if a model exhibits three properties, then
%we can control it without local tuning data.
%A model converts inputs to outputs using tuned internal parameters; i.e.,
%\[Out=model(In,Tunings)\]
%The output mean
%   $\mu$ and variance $\sigma$ comes from
%the mean and variance of $In$   ($\mu_{I},\sigma_{I}$) and
%the space of possible $Tunings$ ($\mu_T,\sigma_T$).
%Model tuning takes historical data to change $\mu_T$ and reduce $\sigma_T$
%and, clearly, untuned models have large $\sigma_T$ and hence large output $\sigma$.
%If the goal is to reduce model output (property1)
%and if model
%output variance is reduced by the mean
%(i.e.,  property2: $\mu \propto \sigma$)
%then (a)~minimizing 
%mean output values has the desirable side-effect of also reducing the variance; and
%(b)~the variance added by 
%sampling across $\{\mu_T,\sigma_T\}$ can be selecting 
%appropriate input values $selected \subseteq I$. 
%Also, 
%for models where the effects on the output of $I$ are greater than $T$ (property3), then 
%the output $\mu$ can be minimized by using the right input selections.
%
%The
%empirical results presented below (and in~\cite{me08c,me07f}) are consistent with the hypothesis
%that commonly used software process models (specifically, the USC COCOMO suite of models)
%have these three properties; i.e., they can be controlled by AI search within the space of current project options.




XXX The rest of this paper explains how \fig{results} was generated.
After some preliminaries,  we discuss the options
available within the current structure of four sample projects.
\fig{dchange}'s drastic changes  will then be implemented as 
operators on those projects.
Next we describe SEESAW and the models it operates on:
the COCOMO effort and time estimator~\cite[p29-57]{boehm00b}; 
and the COQUALMO defect predictor~\cite[p254-268]{boehm00b}.
In the results that follow we show that, for our case studies,
SEESAW's search through the space
of internal changes usually
out-performs the 
drastic changes of \fig{dchange}.

The contribution of this paper is a demonstration that 
managers have more options than they may
realize.
SEESAW is an  insightful 
method
for discovering useful project reconfigurations that improve a project while avoiding requiring drastic and
disruptive project change.   
This result should be welcome news to managers struggling to maintain 
control and continuity over their project in the face of multiple 
demands for drastic change. 

This paper extends a prior publication~\cite{me09a} as follows:
\bi
\item Previously, we only tested SEESAW on NASA case studies. This paper confirms that our results
from NASA projects hold for standard commercial applciations.
\item Previously, we did not compare the performance of SEESAW to other search engines.
Hence, it was possible that our results were sub-optimum compared to an alternate approach.
Here, show that  SEESAW significantly out-performs at least
half dozen standard AI search engines.
\item Previously, our cases studied  explored 
extreme definitions of the drastic changes; for example, decreasing functionality, we reduced the KLOC by 50\%.
Consequently, it was possible that our conclusions were based on a ``straw man'' comparison  with 
an over-zealous set of drastic policies. In the paper, we also explore {\em weaker} changes
such as reducing
functionality via a 20\% reduction in the size of the system.
\item
This paper also contains more tutorial material describing our technique.
\ei


\section{Related Work }\label{sec:variance}
\bi
\item
Feature subset selection~\cite{chen05} to remove noisy attributes; 
\item
Instance-based learning~\cite{baker07} to ingore outlier and train on data most relevant to the current test instance;
\item Increased data collection.
\ei
Our
results to date are not compelling.
Even after
applying instance-based learning, the variance in our predictions remains very large~\cite{jalali07}.
Feature subset selection has also been disappointing: while it reduces
the median performance variance somewhat (in our experiments, from
150\% to 53\%~\cite{me06d}) the residual error rantes are  still large enough to confuse a discussion about the relative merits of
different SE technologies.
Lastly, further data collection has not proven useful.
Certainly, there is an increase
in
the availability
of
historical data on prior projects\footnote{e.g. see \url{http://promisedata.org.data/data}}. However, it is not
clear if it is advisable to use data from other companies data to learn local effort models~\cite{kitch07}.
As to collecting more local data, this has proven remarkably difficuly:
\bi
\item
Even after 26 years of trying, we have only collected less
than 200 sample projects for the COCOMO database from our group of industrial affiliates.
Also,
even after two years of effort we were only able to
add 7 records to a NASA-wide software cost metrics repository~\cite{me07f}.
\item
Other research report similar problems. Norman Fenton spent years advocating data collection~\cite{fenton96} but,
recently, has 
despaired of that approach. At a keynote address in 2007\footnote{\url{http://promisedata.org/?cat=130}} he shocked
his audience by saying:
{\em ``....much of the current software metrics research is inherently irrelevant to the industrial mix ...
any software metrics program that depends on some extensive metrics collection is doomed to failure.''} 
\ei
\bi
\item
Feature subset selection~\cite{chen05} to remove noisy attributes; 
\item
Instance-based learning~\cite{baker07} to ingore outlier and train on data most relevant to the current test instance;
\item Increased data collection.
\ei
Our
results to date are not compelling.
Even after
applying instance-based learning, the variance in our predictions remains very large~\cite{jalali07}.
Feature subset selection has also been disappointing: while it reduces
the median performance variance somewhat (in our experiments, from
150\% to 53\%~\cite{me06d}) the residual error rantes are  still large enough to confuse a discussion about the relative merits of
different SE technologies.
Lastly, further data collection has not proven useful.
Certainly, there is an increase
in
the availability
of
historical data on prior projects\footnote{e.g. see \url{http://promisedata.org.data/data}}. However, it is not
clear if it is advisable to use data from other companies data to learn local effort models~\cite{kitch07}.
As to collecting more local data, this has proven remarkably difficuly:
\bi
\item
Even after 26 years of trying, we have only collected less
than 200 sample projects for the COCOMO database from our group of industrial affiliates.
Also,
even after two years of effort we were only able to
add 7 records to a NASA-wide software cost metrics repository~\cite{me07f}.
\item
Other research report similar problems. Norman Fenton spent years advocating data collection~\cite{fenton96} but,
recently, has 
despaired of that approach. At a keynote address in 2007\footnote{\url{http://promisedata.org/?cat=130}} he shocked
his audience by saying:
{\em ``....much of the current software metrics research is inherently irrelevant to the industrial mix ...
any software metrics program that depends on some extensive metrics collection is doomed to failure.''} 
\ei
\begin{figure}[!t]
\begin{center}
\includegraphics[width=2.5in]{sd.pdf}
\end{center}
\caption{ Standard deviations of $\frac{abs(actual-predicted)}{actual}$ seen in 30 runs
of 11 projects taken from the NASA93 data set (see \protect\url{http://promisedata.org/?p=35}).
In each run, 10 test instances were selected at random and an effort estimation model
was trained on the remaining data.
Effort models were learned
using Boehms's COCOMO-I ``local calibration''~\cite{boehm81} method.
Results from~\protect\cite{me06d}.
}\label{fig:sd}
\end{figure}
\input{cocomo}
The motivation of this paper is the {\em performance variance problem}; i.e.
the output of a process model may range so widely that it is hard to use that model
to assess change ``X'' vs change ``Y''.
Before proceeding with the paper, we pause to document that effect.
Note that the following results come only from an analysis of COCOMO. We speculate that
larger process models,
that contain
more than the two dozen variables of COCOMO/COQUALMO, would exhibit even larger performance variances.

Note that the results from this section come from experiments that reduce tuning, but not
project variance. The rest of this paper discusses experiments with the reverse approach;
i.e. reduce project, not tuning, variance.


\section{Options Within Projects}
SEESAW ranks project changes using the COCOMO and COQUALMO effort, time,
and defect predictors. The terminology of those models is reviewed
in \fig{cparems}.

Using that terminology, \fig{cases} summarizes seven NASA case studies:
\bi
\item TPS(10), TPS(100), TPS(1000) are a standard transaction processing 
system of the kind commonly seen in industrly.
While all these three case studies share the ranges and values of TPS(all),
their KLOC values are different and range around 10 to 100 to 1000K.
\item OSP is the GNC (guidance, navigation, and control)
component of NASA's 1990s {\em Orbital Space Plane}; 
\item OSP2 is a later version of OSP;
\item Flight and ground systems reflect typical ranges seen at NASA's Jet Propulsion Laboratory. 
\ei
Some of the features in \fig{cases}
are known precisely (see all the features with single {\em values}).
But  many of the features in \fig{cases} do not
have precise values
(see all the features that {\em range} from some {\em low} to {\em high}
value).  Sometimes the ranges
are very narrow (e.g., the process maturity of JPL ground software is between 2
and 3), and  sometimes the ranges are very broad.
\fig{cases} does not mention all the  features listed in \fig{cparems}
inputs.  For example, our defect predictor has inputs for
use of {\em automated analysis}, {\em peer
reviews}, and {\em execution-based testing tools}. For all inputs
not mentioned in \fig{cases}, values are picked at random from the full
range of \fig{cparems}.

\section{Imposing Drastic Changes}
To use our technique, proposed changes have to be mapped onto the input space of the COCOMO/COQUALMO models.
Elsewhere~\cite{boehm00a,boehm99}, we have published extensive tutorials on how to perform such mappings.
\begin{figure}
\begin{center}
\scriptsize
\begin{tabular}{|c@{~}p{.4\linewidth}p{.4\linewidth}|}\hline
&{\bf Drastic change} 
& {\bf Effects on \fig{cases}} \\\hline\hline

1&Improve personnel 
& 
acap = 5; pcap = 5;
                    pcon = 5 \newline 
                    apex  = 5 ;
                    plex = 5 ;
                    ltex = 5 \\\hline
1*&Improve personnel*
&
acap = 4; pcap = 4; pcon = 4; \newline aexp = 4; plex = 4; ltex = 4\\\hline


2&Improve tools, techniques, or  development platform 
& 
    time = 3; 
                    stor = 3\newline
                    pvol = 2;
                    tool = 5\newline
                    site = 6\\\hline
2*&Improve tools, techniques, or  development platform *
& 
    time = 3; 
                    stor = 3\newline
                    pvol = 2;
                    tool = 4\newline
                    site = 5\\\hline
3&Improve precedentness / development flexibility 
&
prec = 5;
flex = 5
\\\hline

4&Increase architectural analysis /  risk resolution &resl = 5 \\\hline
5&Relax schedule  
& 
    sced = 5      
\\\hline

6&Improve process maturity 
& pmat = 5
\\\hline

7&Reduce 
functionality 
& data = 2; kloc  * 0.5
\\\hline
7*&Reduce 
functionality* 
& data = 2; kloc  * 0.8
\\\hline
8&Improve the team 
& team = 5 
\\\hline
8*&Improve the team* 
& team = 4 
\\\hline

9&Reduce quality 
& rely = 1 ;
                    docu = 1\newline
                    time = 3 ;
                    cplx = 1\\\hline
9*&Reduce quality* 
& rely = 2 ;
                    docu = 2;\\\hline
\end{tabular}
\end{center}
\caption{Implementing drastic changes, from~\protect\cite{boehm99}.
 The enties 1*,2*,7*,8*,9* are less drastic versions of 1,2,7,8,9. 
}\label{fig:how}
\end{figure}
For example, 
\fig{how} shows several such mappings (and are taken from~\cite{boehm99}).
Each mapping defines a different drastic changes.
Most of the values in \fig{how} are self-explanatory with three exceptions:
\be
\item
$kloc*0.5$ in change \#7 ``reduce functionality'' means that,  when imposing this drastic change,
we only implement half the system.
\item
Most of the features fall in the range one to five.
However, some have minimum values of 2 or higher (e.g., $pvol$ in ``improve tools/tech/dev''),
and some have maximum values of 6 (e.g., $site$ in ``improve tools/tech/dev'').
This explains why some of the drastic changes result in values other than one or five.
\item 
A valid criticism of our previous work~\cite{me09a} was that our drastic changes were too
extreme (e.g. \#7, slicing a program in half).
In this study, we include weaker drastic changes.
For example, in \fig{how}, 
(1*,2*,7*,8*,9*). 
are weaker forms forms of changes (1,2,7,8,9).
\ee
To impose a drastic change on a case study,
if that change refers to feature $X$ (in the right-hand column of
\fig{how}),
then we
first
(a)~removed $X$ from the values and ranges of the case study (if it was present); then
(b)~added the changes of \fig{how} as fixed values for that case study.
For example, if we imposed change \#1 ``improve personnel'' onto JPL ground systems, then this would generate
\fig{changed}.
\begin{figure}[!t]
\renewcommand{\baselinestretch}{0.92}
\begin{center}
\scriptsize
\begin{tabular}{c|lrr|lr}
       &\multicolumn{3}{c|}{ranges}      &\multicolumn{2}{c}{values}\\\hline
project&feature&low&high&feature&setting\\\hline
   & prec   &          3  &      4 & resl & 4\\
TPS(all)& flex   &          3  &      4 & team & 4\\
   & data   &          3  &      4 & pmat & 4\\
   & cplx   &          3  &      4 & rely & 4\\
   & ruse   &          2  &      4 & docu & 3\\
   & pvol   &          2  &      3 & time & 3\\
   & acap   &          3  &      5 & stor & 3\\
   & pcap   &          3  &      4 &  apex&3\\
   & pcon   &          4  &      5 & plex&4\\
   & ltex   &          2  &      4 & tool & 4\\
   & site   &          2  &      4 &\\
   & sced   &          2  &      3 &\\\hline
   
TPS(10)  & KLOC & 7.5 & 12.5 &        &\\\hline
TPS(100)  & KLOC & 75 & 125 &        &\\\hline
TPS(1000)  & KLOC & 750 & 1250 &        &\\\hline

&prec&1&2&data&3\\
OSP:&flex&2&5&pvol&2\\
Orbital&resl&1&3&rely&5\\
space&team&2&3&pcap&3\\
plane&pmat&1&4&plex&3\\
&stor&3&5&site&3\\
&ruse&2&4&&\\
&docu&2&4&&\\
&acap&2&3&&\\
&pcon&2&3&&\\
&apex&2&3&&\\
&ltex&2&4&&\\
&tool&2&3&&\\
&sced&1&3&&\\
&cplx&5&6\\
&KSLOC&75&125\\\hline
&prec&3&5&flex&3\\
OSP2 &pmat&4&5&resl&4\\
&docu&3&4&team&3\\
&ltex&2&5&time&3\\
&sced&2&4&stor&3\\
&KSLOC&75&125&data&4\\
&&&&pvol&3\\
&&&&ruse&4\\
&&&&rely&5\\
&&&&acap&4\\
&&&&pcap&3\\
&&&&pcon&3\\
&&&&apex&4\\
&&&&plex&4\\
&&&&tool&5\\
&&&&cplx&4\\
&&&&site&6\\\hline
&rely&3&5&tool&2\\
JPL &data&2&3&sced&3\\
flight&cplx&3&6&&\\
software&time&3&4&&\\
&stor&3&4&&\\
&acap&3&5&&\\
&apex&2&5&&\\
&pcap&3&5&&\\
&plex&1&4&&\\
&ltex&1&4&&\\
&pmat&2&3&&\\
&KSLOC&7&418&&\\\hline
&rely&1&4&tool&2\\
JPL &data&2&3&sced&3\\
ground&cplx&1&4&&\\
software&time&3&4&&\\
&stor&3&4&&\\
&acap&3&5&&\\
&apex&2&5&&\\
&pcap&3&5&&\\
&plex&1&4&&\\
&ltex&1&4&&\\
&pmat&2&3&&\\
&KSLOC&11&392&&\\
\end{tabular}
\end{center}
\caption{Seven case studies: TSP(10), TSP(100), TSP(1000), OSP, OSP2, flight, ground.
Numeric values 
$\{1,2,3,4,5,6\}$ map to $\{very low, low, nominal,
high, very high, extra high\}$.}\label{fig:cases}
\end{figure}
\begin{figure}
\renewcommand{\baselinestretch}{0.92}
\begin{center}
\scriptsize
\begin{tabular}{c|lrr|lr}
       &\multicolumn{3}{c|}{ranges}      &\multicolumn{2}{c}{values}\\\hline
project&feature&low&high&feature&setting\\\hline
             & rely & 1 & 4 & tool & 2\\
JPL          & data & 2 & 3 & sced & 3\\
ground       & cplx & 1 & 4 & {\bf acap} & {\bf 5}\\
software     & time & 3 & 4 & {\bf pcap} & {\bf 5} \\
with         & stor & 3 & 4 & {\bf pcon} & {\bf 5} \\
improved     & pmat  & 2 &3 & {\bf apex} & {\bf 5} \\
personnel    & KSLCO&  11 &  392 & {\bf plex}  & {\bf 5} \\
             & & & & {\bf ltex} & {\bf 5} \\
\end{tabular}
\end{center}
\caption{Ground systems, after applying drastic change \#1 ``improve personnel'' (changes shown in bold in the
left hand column).
}\label{fig:changed}
\end{figure}
\input{model}

\section{Finding Alternatives to Drastic Change}
With the models defined, we can now describe methods for exploring them.
This section reviews a range of such methods, starting with simplistic approaches
like gradient descent and exhastive enumeration, then moving on to SEESAW and other search algorithms.
 
\subsection{Simpler Than SESSAW}\label{tion:stochast}
The reader may wonder why we use a stochastic method like SEESAW
to explore project options. Would not a simpler method suffice?
This section discusses two such alternatives: linear extrapolation and exhaustive enumeration.
\subsubsection{Gradient Descent}
In the case of {\em linear models} that have been {\em
precisely  tuned} using local data, it is a simple
matter to check if a combination of internal changes does better than drastic change.
Many of the relationships inside COCOMO
model are linear.
For such models,  ``what-if'' queries require just 
a simple
{\em gradient descent} to
assess the relative effectiveness of, say, reducing functionality versus
some combination of internal changes.

Unfortunately, as mentioned in the introduction,
not all tunings are {\em precise}. Recalling Baker's result from the introduction,
even after tuning,
the gradient of the relationships 
may not be known with certainty.

Another drawback with simplistic gradient descent is  that, when
optimizing for effort {\em and} time {\em and} defects, there may
be contradictory effects.  For example, we show below
one result
where effort {\em reduced}, but defects
{\em increased dramatically}. Hence, optimizing our models is not
a simple matter of moving fixed distances over some linear effect:
there are also some trade-offs to be considered (e.g. using a tool that considers combinations of effects,
like SEESAW).
\subsubsection{Exhasitve Enumeration}\label{sec:ee}
In exhastive enumeration, all possible inputs are presented to a model.
This method returns the input that generates the top scoring output. Note that exhaustive enumeration
proposes one setting to {\em every} input value.

We do not recommend exhaustive enumeration for the several reasons.
Our models accept one continuous input (KLSOC) and 24 discrete inputs, each of which usually takes one of five values (very low to very high).
If we divide the KSLOC into ten ranges, then this leads to an maximum input space of $10*5^{24}\approx10^{17}$ possibilities.
Becuase of the tuning variance problem, SEESAW explores the full space of tunings. Our models can tune
21 COCOMO paramters and 24 COQUALMO parameters.
If we divided the tunings into ten regions, this leads to a total
space of internal tunings of  $10^{21} * 10^{24}=10^{45}$ tunings. 
Decades of AI research has shown that seemingly large search spaces (such as the combined space of $10^{17+45}$), can be effectively
explored with a few simple heuristics. This paper tests several such heuristic search methods. 

Also, the best solution returned by exhaustive enumeration may be needlessly complicated. A repeated result
in data mining~\cite{kohavi97,hall03} and linear algebra~\cite{miller02} is that good predictors may only 
require a small subset of the available inputs. That is, if the project inputs are $\{A,B,C...,Z\}$
and exhaustive enumeration returns $\{A=1,B=2,C=0,...,Z=1\}$, it may well be that the same score may be obtained by setting some subset (say,
$\{A=1,C=0\}$) and let the other inputs roam at random.
if we want to minimize the impact of SEESAW's recommendations, we  
should seek the {\em smallest} the set of changes to the project  
that {\em most} improve model predictions. 

Further, if we could enumerate all $10^{17+45}$ options, it may not be informative. The top solution returned by exhaustive enumeration
would constrain every input variables {\em and} every tuning. This is inappropriate:
tunings are not controlable- they arise from local domain conditions.

Finally, proposing settings to all project options is inapproarite when managers have only partial control over those options. The authors have worked
with many government software projects staffed by teams of contractors and sub-contractors and sub-sub-contractors. In such a situation,
it is likely that any government-mandated requirement will be only partially  implemented by the contractors. Hence, we need not single solutions,
but solutions that are stable within a space of neighboring solutions.  
Harman~\cite{harman07} comments that understanding the soltuion neighborhood is an open and pressing issue in search-based SE:
\begin{quote}
``... solution 
{\em robustness} may be as important as solution functionality. For 
example, it may be better to locate an area of the search 
space that is rich in fit solutions, rather than identifying an 
even better solution that is surrounded by a set of far less fit 
solutions. ''
\end{quote}

In summary, our preferred search engine does not explore all options and returns the smallest set of contstraints to project inputs
that perform best across a space of possible tunings.
Further, that search should also comment on the stability  of the result in the neighborhood of the soltuion. As we shall see, SEESAW achieves
all those goals.

\subsection{SEESAW}
Our preferred method for finding changes to projects that avoid
the need for drastic change is the  SEESAW
algorithm. 
SEESAW searches within the ranges of \fig{cases} to find constraints
that most
reduce development effort, development time, and defects.
 \input{seesaw}
\subsection{Other Alernatives to SEESAW}

\begin{figure}
\begin{center}
\includegraphics[width=1.5in]{plots/scatter-tps100-all-isamp.pdf}%
\includegraphics[width=1.5in]{plots/scatter-tps100-all-maxwalksat.pdf}%       

\includegraphics[width=1.5in]{plots/scatter-tps100-all-sa.pdf}               
\includegraphics[width=1.5in]{plots/scatter-tps100-all-a-star.pdf}%           

\includegraphics[width=1.5in]{plots/scatter-tps100-all-beam.pdf}%             
\includegraphics[width=1.5in]{plots/scatter-tps100-all-seesaw.pdf}
\end{center}
\caption{aa}
\end{figure}
\input{engine-defects}
\input{engine-effort}
\input{engine-months}
\input{strategy-all/defects.tex}
\input{strategy-all/effort.tex}
\input{strategy-all/months.tex}
\section{Results}

SEESAW 21
BEAM 15
A* 9
SA 1

Our experiments compared predictions
after either
(a)~applying
the drastic changes of \fig{dchange};
or (b)~constraining some internal options selected
by 
SEESAW trying to minimize a linear combination of normalized defects, effort, and time;
or (c)~doing nothing at all and just running a Monte Carlo simulation
across the ranges of \fig{cases}.
Each experiment comprised 100 runs of \fig{MWF}. Each time a set
of project options were scored, $m$ values were selected at random
from 
Equations \ref{eq:mem},\ref{eq:linesf},\ref{eq:coq} and \ref{eq:linedefrem}.

In 
Figures \ref{fig:effort}, \ref{fig:time}, and \ref{fig:defect},
the rows are sorted by median scores.
The ``Do nothing'' row  comes from  Monte Carlo simulations of the current ranges, without any changes.
The {\em ranks} shown in the left-hand column are computed from a Mann-Whitney 
test at 95\% confidence 
test (this test was chosen since
(a) due to the random nature of
Monte Carlo simulations, the inputs to each run are not paired;
and (b) ranked tests make no, possibly inappropriate,
 assumption about normality of the results).
Two rows have the same rank 
if there is 
no statistical difference in their distributions. 

Each row shows
results from 100 
{\tt run}s (see \fig{MWF}): 
\bi
\item All results are normalized to run 0..100, min..max.
\item Each row shows the 25\% to 75\% quartile range of the normalized
scores 
collected during the simulation.
\item The median result is shown as a black dot. 
\ei
All the performance scores (effort, time, defects) get {\em better} when
the observed scores get {\em smaller}. 
Note that, usually, SEESAW obtains the better scores.

Figures \ref{fig:effort}, \ref{fig:time}, and \ref{fig:defect} show results from 12 experiments:
\[ 
\{effort, time, defects\}*\{flight, ground, OSP, OSP2\}\]
In $\frac{12}{12}$ (i.e., all) experiments, the median score from
SEESAW is better than at least half of the other changes.
Better yet, in $\frac{6}{12}$ (i.e., half) experiments, SEESAW
offers the best predictions,
and in two more experiments (the defect results for OSP and OPS2), 
SEESAW ties the top rank.
This result is the justification 
for our claim in the introduction that 
in  a majority of experiments ($\frac{8}{12}$), SEESAW's
search through the space of internal changes to current project options
does as well or better than
drastic change.

\input{freq.tex}
Another result of interest is that the range of SEESAW results is typically
smaller than most other changes
in
Figures \ref{fig:effort}, \ref{fig:time}, and \ref{fig:defect} (observe how the 75\% - 25\% band
in SEESAW's results are usually smaller than all the others).
That is, of all the changes proposed here, the most stable were those generated by the stochastic
search. This paradoxical result
is simple to explain: SEESAW does not just {\em sample} a space of options; it also hunts for
options that {\em reduces} the median score within those options.
For models where the variance is proportional to the median, then reducing the median score
also reduces the variance.

Another stable feature of SEESAW are the actual recommendations.
An issue with stochastic sampling is that the output recommendations can
vary dramatically from one run to another.
Such variance can occur when a stochastic methods staggers between
multiple solutions of near equal value.
\fig{freq} shows the recommendations made by 20 runs of SEESAW for the flight and ground projects.
Note that the majority of the recommendations occur in all 20 runs and
represent the most important
actions that a project manager could make.
The recommendations that occur at low frequencies are
still valid, if used in conjunction with the other recommendations of that particular run.

The worst results from SEESAW
are see in
\fig{effort} where the median effort found by SEESAW
is near the 
middle of all changes. 
This result is hardly surprising.
This result comes from OSP2  and this case study
is
the most constrained case explored by this
paper (observer in \fig{cases} that,  for the most part,
the features all have fixed values). SEESAW works by exploring subsets of the available options.
When that option space is limited, the benefits of SEESAW are also limited.
That is, we cannot find
successful options unless we are first given enough  options to explore.
Previously~\cite{me07g} we have expressed this as
``if you fix everything, there is nothing left to fix''.

In other cases where drastic changes wins,
those changes
are often unacceptable.  For example,
in the effort and time results, SEESAW is beaten by ``reduce
quality'' in $\frac{4}{12}$ experiments.  The benefits of these
effort and time reductions should be weighed against the
implications on defects. \fig{defect} shows that ``reduce quality''
results in the largest number of defects seen in any change. While
there may exist projects that permit delivering code with large
numbers of defects, we believe that usually this is unacceptable.

Another drastic change that defeats SEESAW is ``improve pcap'' (personnel capabilities).
Recalling \fig{how}, we see that improving personnel capability requires changing
two analyst features, three personnel features, as well as addressing some issues relating
to language and tool experience. Of all the drastic changes studied in this paper, this
change is the most complex. It might also be the hardest to implement since it requires
dramatic change to the current personnel working on the project (perhaps even firing old
staff in order to hire new personnel).
If an organization can handle such large-scale change, then in some
cases (e.g., 
in $\frac{4}{12}$ experiments from 
Figures \ref{fig:effort}, \ref{fig:time}, and \ref{fig:defect})
the ``improve pcap'' option might indeed
outperform SEESAW.
However, managers might want to avoid such complex organizational change, preferring
instead to fine tune their project within its current structure using just internal changes.

\section{Discussion}

\subsection{Value of Process Standaradization?}

This paper makes the case that, sometimes, development teams should reject 
major project changes proposed by, say, senior management.
For example, in \fig{results}, we see that
SEESAW's recommended internal changes out-performs the drastic change ``improve pmat'' (increase the CMM 
software process maturity level).
Lest we over-state our case, we offer the following notes on the benefits
of process standardardization.

Emmerich, Finkelstein et at.~\cite{emmerich98} argue convincingly for
software process standards across an organization. Such standards simplify:
\bi
\item {\em Certification:} Prior to certification, development teams can modify their practices
to ensure they are in conformance with accepted practices. During certification,
outside consultants need only check for a certain set of practices.
 \item
{\em Management:} Once a standard is in place, supervising managers can collect standard reports from projects and benchmark those reports against data collected from other projects.
\item
{\em Co-ordination between teams:} If two development teams have adopted the
same standards, it becomes simpler for one team to understand and use artificats from the other.
\item {\em Quality assurance:} While the use of
standards  does not necessarily ensure quality,
their application does 
ensure that a
development team has adopted at least
some generally accepted useful techniques.
\item {\em Project planning:} Rather than having to organize their work {\em tabula rasa}, 
managers
can use standards as templates to intiailize the first version of their project plans.
\item {\em Training:} Newcomers to a project can quickly find an answer to the question
``how do you do thing things around here?''.
\ei
Nevertheless, it is not always true that {\em exactly the same} software enginnering
techniques
should be imposed on {\em every} project in an organization.
Lumsden and Strigel debate the value of large project techniuqes
to small projects~\cite{lumsden07} 
(and by ``smaller projects'' they mean the example, 85\% of software projects developed by teams of 50 people or less~\cite{rich07}).
Strigel believes that small projects must use different techniques- a view that would
resonate
strongly with the agile community~\cite{beed01,boehm02}.
Even Lumsden, who takes takes the opposite view, agrees that there  are occasions when small/large projects
can usefully borrow (some) techniques that, usually,
are only seen in large/small projects (respectively). 

More generally,  Emmerich (personnel communciation) and Paulk~\cite{paulk99} remark
that   every software process can be tuned to local conditions.
For example, there are a large number of choices within 
the SEI capability maturity model (CMM):
\bi
\item
18 {\em key process areas}: cluster of related activities that, when performed collectively, 
achieve a set of goals; e.g. ``software project planning''.
\item
dozens
of {\em goals}: summaries of key practices and are used to determine if an organization of proect of a key process area; e.g. ``software estimates are documented for use in planning and tracking the software project''.
\item
 316 key practices; e.g. ``develop according to a documented procedure''. 
\ei
Paulk~\cite{paulk99} argues that organizations should strive to use
all key process areas and goals. However, regarding the key practices, organizations
have much freedom in selecting
the practices most relevant to their organization. 

While there is consensus that local managers should select the best local practices, there
is very little guidance on how to perform that task (counter-examples: ~\cite{paulk99,boehmturner03}).
If that discussion becomes adverserial, it may be necessarily for
managers to precisely explain why
they advocated a particular drastic change or set of internal changes. As mentioned in the introduction,
such a dialogue {\em should} be enabled by software process models. However, given the large variance observed in those models, some other technique is required to find stable conclusions within the space of uncertainty associated
with a project. Hence this paper.

\subsection{Other Issues}

Our thesis is that there exist some process models (e.g. COCOMO/COQUALMO)
that can still be used to rank proposed process changes, even without sufficient data to achieve
precise local tunings. This is not
to say that local tuning should not be attempted. Indeed, our strong view is that the preferred option is to use
precisely tuned models. However, in domains where such tunings
are unavailable, we recommend SEESAW.

We show that for COCOMO/COQUALMO, prediction uncertainty can be mitigated 
by selecting the right
project options {\em regardless} of uncertainty in the tunings.
This is {\em not} to say that prediction uncertainty of {\em all}
process models can be mitigated in the same manner. However, the observation that COCOMO/COQUALMO models
can be controlled in this manner is
one more reason to prefer these models.

Nor do we  claim that 
stochastic stability {\em always} beats {\em all} drastic change for all projects.
Indeed, we offer  examples where
extremely drastic change {\em defeats} the internal changes proposed by SEESAW.
However, in those cases, those changes were so drastic 
(reduce functionality, or make major change the personnel) that many managers may
prefer SEESAW's slightly-less-then-best recommendations.

\section{Related Work}
Much of the related work on uncertainty in software engineering uses a Bayesian analysis.
For example,
Pendharkar et.al.~\cite{pendharkar05} demonstrate the utility of Bayes networks in effort
estimation while Fenton and Neil explore Bayes nets and defect prediction~\cite{fenton99}
(but unlike this paper,
neither of these teams links defect models to effort models).
We elect to take a non-Bayesian approach since most of the industrial and government
contractors we work with use parametric models like
COCOMO. 

The process simulation community (e.g. Raffo~\cite{raffo05c}) studies
models far more elaborate than COCOMO or COQUALMO. While such models
offer more detailed insight into an organization, the effort required
to tune them is non-trivial. For example, Raffo spent two years
tuning one such model to one particular site~\cite{raffo95}.

Other related work is the search-based SE approach advocated by Harmon~\cite{harman04}. 
Search-Based Software Engineering (SBSE) uses optimization techniques
from
operations research and meta-heuristic search (e.g. simulated annealing and genetic algorithms)
to hunt for near
optimal solutions to complex and over-constrained
software engineering problems. 
The SBSE approach can and has been applied to many
problems in software engineering (e.g. requirements engineering~\cite{jalali08})
 but most often in the field of software testing~\cite{andrews07}.
Harmon's writing inspired us try simulated annealing to search the what-ifs in untuned COCOMO
models~\cite{me07f}. However, we found that SEESAW ran much faster and produced results with far less
variance than simulated annealing.

\section{Future Work}
The results of this paper assume that all the options found by SEESAW are {\em controllable};
i.e. a manager can implement all of SEESAW's recommendations. Future work should repeat this
analysis assuming that only certain subsets of the internal options are controllable.

Also,
this paper only explores the drastic changes of \fig{dchange} (these where defined in one
of our prior publications~\cite{boehm99}). There may be other drastic changes that defeat
the recommendations of SEESAW. Future work should explore a wider range of drastic changes.

%As to our own related work,
%for several decades, one of us (Boehm) has worked on a variety of software process models including
%the  COCOMO effort and time predictor~\cite[p29-57]{boehm00b}; 
%and
%the COQUALMO defect predictor~\cite[p254-268]{boehm00b}.
%One advantage of these models is that,
%unlike other models such as PRICE-S~\cite{park88}, 
%SLIM~\cite{putnam92}, or SEER-SEM~\cite{jensen83},
%our models are
%fully described in the literature. Also, at least
%for the effort model, there exist  baseline results~\cite{chulani99}. 
%\fig{cparems} lists the features of those  models.
%
%%XXX related work
%Most of the COCOMO literature
%concerns 
% {\em manual} use of these models for decision making or
%{\em automatic model tuning}.
%An alternative approach is the use of
%simulated annealing~\cite{kirkpatrick83} to explore {\em automatic} decision making
%{\em without local tuning}.
%In the COCOMO community, our work is somewhat unique. A standard approach in that community
%is to perform Monte Carlo analysis across the input space, sort the output predictions, then
%report the X\% percentile band as ``the'' prediction (usually 50\% to 75\%)~\cite{me08f}. 
%The work in
%this paper is different in that SEESAW does not just {\em
%report} the prediction variance, it tries
%to {\em reduce} it.
%
%
%At ASE`07~\cite{me07f}
%we used COCOMO and COQUALMO and
%simulated annealing to
%explore a very focused question: when is it
%useful to use automated software engineering tools? 
%While that question is of great interest to the automated software engineering
%community, here we explore a question 
%that is relevant to a much broader audience: what are the relative merits
%of combinations of internal changes versus drastic project changes?
%
%
%In his text {\em Empirical Methods in 
%Artificial Intelligence}, Cohen~\cite{cohen95}
%advises baselining a sophisticated
%better
%methods against simpler alternatives. 
%A simpler alternative to SEESAW is 
%the manual application of known best
%practices in improving  a project. 
%Before we endorse SEESAW or any other  algorithm,
%it is appropriate to first test
%if these engines do better than accepted best practices.
%Previously~\cite{boehm99}, we have reviewed  nine 
%such practices that can be manually applied---see the list   in \fig{dchange}.
%This paper performs Cohen's baseline test 
%by comparing the effects of applying accepted best practices versus the
%results of applying changes proposed by SEESAW.
%
%
\section{Conclusion}
Drastic change is highly disruptive to a project.
For example, your manager has returned from ICSE`09
excited about the next new thing in software engineering.
``We need to drastically alter our projects,'' she says, ``in order to support (say) automatic
formal methods, agile process,  etc.''. 
This change
may entail switching from Windows to LINUX (since there are more 
research prototypes for automated formal methods running on LINUX);
stopping production
while engineers retrain;  or, worse, firing the engineers and making new hires. 

An alternative to drastic change are combinations of internal changes within the current
bounds of a project. Given precise local tunings to linear software process models 
(e.g. the COCOMO effort predictor and the COQUALMO defect predictors), then simple
linear extrapolation may suffice for exploring different process options.
However, when the tunings are not precise, or competing goals have to be achieved, 
simple linear extrapolation is insufficient.

This paper has explored the hard case of
(a)~ranking different process changes using (b)~models with competing influences
that (c)~have not been precisely tuned using local data.
In this case, the core task is to find {\em stability} within a very large space of possibilities.
Given a project with some variance in its process options and a model with some variance
in its tunings, then the model prediction
uncertainty  is some combination of the uncertainty
 with a project's
options and uncertainty with the model's tunings.
Local data can be used to reduce 
the uncertainty in the
tunings. However, as shown in \eq{ab}, even after tuning it is possible tuning uncertainty remains.

Based on research dating back to 1981~\cite{boehm81}, we assert that the
space of {\em
possible tunings} for COCOMO/COQUALMO is well defined
(see Equations \ref{eq:mem},\ref{eq:linesf},\ref{eq:coq} and \ref{eq:linedefrem}).
We show here that
at least for the these models,
%$U_3$ dominates $U_2$ since:
%\begin{quote}
%{\em The prediction uncertainty $U_1$ associated with untuned models can be mitigated via the
%right choice of project options.
%}
%\end{quote}
%This result does not mean local tuning 
%is worthless. Ideally, process options are ranked via precise local tunings. However, in domains with
%to insufficient data for precise tunings, it is still possible to rank
%process options. 
%
it is possible to explore
a very large range of ``what-ifs'' (the space of all possible tunings and internal project changes
)  
to find project option settings that improve model predictions, despite uncertainty in the tunings.

%XXX controllable assumption

Specifically, in studies with 4 NASA case studies and 8 drastic
changes, we found that, usually, SEESAW's stochastic
sampling found internal options that worked as well or
out-performed drastic change. Essential to the above is {\em stochastic stability}. Stochastic search enables the rapid
sampling of a very wide range of options. More importantly, when studying the space of
unknowns associated with a particular project (e.g. \fig{cases}),
stochastic sampling
can return multiple results after multiple runs. These results can be studied to check
the
{\em stability} of the solutions within the space of project options.
Figures \ref{fig:effort}, \ref{fig:time}, and \ref{fig:defect} 
show that
the SEESAW stochastic stability tool typically generated the smallest range in
performance predictions.  That is, of all the changes studied here,
SEESAW's recommendations were typically the most stable.

Our results suggest that project managers
may have more project improvement options than they currently realize.
Hence, before performing drastic
changes it can be worthwhile to check
for stochastically stable effects amongst the internal changes, if only to understand
better 
the available 
options within the current structure of  a project.

\section*{Acknolwedgements}
Thanks to 
Phillip Green,
Topi Haapio and Andi Marcus and 
for their feedback on earlier drafts.
% local tuning still useful
\bibliographystyle{IEEEtran}
\bibliography{refs}
\appendix \input{pseudocode}

\end{document}
\underline{{\bf KEY:}}\\
The quartile charts, shown on left,
plot the defects predictions after either
AI search with  AI search (line one);
or  applying various drastic changes; or 
after 
doing nothing (second last line).
\bi
\item
Results are sorted on 
median results from 100 Monte Carlo simulations.
\item
The ``Baseline'' row (second from the bottom) shows the defects predicted from the projects,
without any additional constraints.
\item
Median results are  shown as black circles. These
medians are surrounded  left and right
by
black lines showing bottom/top quartiles (respectively).
\item
The ranks shown in the left-hand-column are computed from a Mann-Whitney 95\% confidence test:
two rows have the same rank 
if there is 
no statistical difference in their distributions. 
For example, the
four rows have the same rank since their distributions are essentially the same.
\ei\\\hline
\end{tabular}
\end{minipage}
\caption{AI search (see line 1) reduces defects
far more than 
drastic process changes.
As might be expected, ``reducing quality'' 
produces more defects than
``Baseline'' (see last last).

