\documentclass[twocolumn,global]{svjour}
\usepackage{url,alltt,graphicx,cite,url,fancyheadings,bar,fancyvrb}
\usepackage{rotating,stfloats}

\newenvironment{ccode}{\scriptsize\begin{alltt}}{\end{alltt}\normalsize}


   \newcommand{\code}[3]{%
     \begin{figure}[!t]%
        \fvset{  samepage=true,%
                 fontsize=\scriptsize,%
                 commandchars=\\\{\},%
                 framesep=2mm,%
                 label={\small{\textit{#1}}},%
                 labelposition=topline,%
                 %numbersep=1pt,%
                 %stepnumber=5%
                 %numbers=left,%
                 frame=single%
                 }%
            {\bf \VerbatimInput{#1}}
    \caption[#2]{#3}\label{fig:#1}%
    \end{figure}%
}

%\usepackage[para,symbol*]{footmisc}
%\setfnsymbol{lamport}
\usepackage{amssymb,times}
\DeclareFontShape{OT1}{cmtt}{bx}{n}{
  <5><6><7><8><9><10><10.95><12><14.4><17.28><20.74><24.88>cmttb9}{}

\newcommand{\mytitle}{
RST milestone 1.1.5.9:\\
Applying trade space analysis to recommend CEV/CLV 
options for\\ SW capabilities and development of 
processes and tools 
}
\footrulewidth 0.0pt\headrulewidth 0.1pt
\lhead{}\rhead{}\chead{}\rfoot{}\lfoot{}\cfoot{}
\rhead{{\scriptsize 
\date{\today}\\
WP ref: http://now.unbox.org/all/trunk/doc/06/xomo201.tex\\
Download this paper at
\url{http://menzies.us/pdf/06xomo202.pdf}.}}
\lhead{\includegraphics[width=8cm]{wvu.pdf}}

\newenvironment{smallitem}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{itemize}
   \setlength{\leftmargin}{.2in}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{itemize}}

\newenvironment{smallenum}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{enumerate}
  \setlength{\leftmargin}{.2in}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{enumerate}}


 \newcommand{\bd}{\begin{description}}
\newcommand{\ed}{\end{description}}
 \newcommand{\bi}{\begin{smallitem}}
\newcommand{\ei}{\end{smallitem}}
 \newcommand{\be}{\begin{smallenum}}
\newcommand{\ee}{\end{smallenum}}
\newcommand{\fig}[1]{Figure~\ref{fig:#1}}
\newcommand{\eq}[1]{Equation~\ref{eq:#1}}
\newcommand{\tion}[1]{\S\ref{sec:#1}}

%\topmargin -3cm

\begin{document}
\title{
\vspace{2cm}
~\\
\mytitle}
\author{Tim Menzies}
\institute{Lane Department of Computer Science and Electrical Engineering, West Virginia
University,
USA\newline
\url{tim@menzies.us}\newline
\url{http://menzies.us}~\\\vspace{4cm}}
\maketitle \thispagestyle{fancy}
\pagestyle{plain}

\begin{abstract}
We seek an AI agent that can negotiate the trade space within
the design of modern complex systems such as NASA's new CEV/CLV
rockets.
The conclusions of the agent must be comprehensible to busy
humans engaged in elaborate discussions regarding project options.
Hence, those conclusions should be clear to read and clearly useful.
As an on-going task in this project, we are assessing 
the
merits of the TAR3 {\em treatment learning} for such an agent. 

This paper presents several case studies where TAR3
successfully found succinct recommendations that clearly
effected the development effort, number of defects, and threats to
the project plan.

The technology described here integrates into the RST test bed via an interface
called {\em slot trees} (described in this document).
 \end{abstract}
\newpage

{\bf Acknowledgments } This research was conducted at
West Virginia University 
under NASA sub-contract project 100005549, task 5g, award 1002193r.
All software discussed here is available from
\url{http://unbox.org/wisp/xomo}
under the  GNU Public License (version 2:
see \protect\url{www.gnu.org/copyleft/gpl.html}).

~\\~\\~\\~\\

{\bf Disclaimer} 
Reference herein to
any specific commercial product, process, or service by trade name,
trademark, manufacturer, or otherwise, does not constitute or imply
its endorsement by the United States 
Government.

~\hrule~
\newpage
\tableofcontents\newpage
\listoffigures\clearpage
\section{Introduction}
The manager of any NASA project must answer three questions:

\be
\item What level of risk is acceptable for this project?
\item How much risk does a project have?
\item How to reduce \#2 to \#1?
\ee

This paper is about questions two and three.  
Once the appropriate level of risk has been determined, then a
project manager struggles to determine and reduce the project risk.

Technically, risk reduction is a {\em trade-space} exploration
exercise. Given a set of options generated by a set of models,
managers must ``walk'' (metaphorically) around that space looking
for decisions that most improve the current situation.
Our trade-spaces are defined by
three models:
\bi
\item
The  COCOMO effort estimation model~\cite[p29-57]{boehm00b}; 
\item
The COQUALMO defect model~\cite[p254-268]{boehm00b};
\item
The THREAT  schedule threat model~\cite[284-291]{boehm00b}\footnote{Historical note:
previously, we called ``THREAT'' the ``Madachy schedule risk model''~\protect\cite{me05e}.}.
\ei

The models use dozens of variables, not all of which are known with
certainty.  For example, until software is completed, the exact size of
a program may be unknown. Hence, exploring our effort, defect, and threat models
requires exploring a large trade-space of possible model inputs. 

To achieve this, our
recommended approach uses  Monte Carlo simulation and data mining in the cyclic manner
of \fig{cycle}:
\bi
\item
A Monte Carlo simulator repeatedly call some models,  drawing model inputs
from some pre-defined ranges. 
\item
The output from the runs are then studied by data
miners to find some ``fitter'' ranges to use in  subsequent calls to  the model.
\item
This cycle continues, defining narrower and narrower ranges; i.e.  
\mbox{$ranges_{i+1} \subseteq ranges_i$}.
\ei
The cycle ends when new ranges are not ``fitter'' than old ones.
Here,
``fitter'' is determined by some fitness function that scores model output:
the fitter the input ranges, the higher the scores of the model
output.
For example:
\bi
\item
When processing COCOMO, COQUALMO, and THREAT,
the fitness function would select for
lower effort, lower defects, and fewer threats. 
\item
For other models the tools et of \fig{cycle} remains mostly the same, but the fitness function
changes (see \fig{old}).
\ei

\begin{figure}[!t]
\begin{center}
\includegraphics[width=2in]{cycle.png}
\end{center}
\caption{Cyclic learning}\label{fig:cycle}
\end{figure}
\begin{figure}[!b]
\footnotesize
\begin{center}
\begin{tabular}{|p{3.2in}|}\hline
Within NASA, \fig{cycle} has been applied to:
\bi
\item Spacecraft design, where fitter means ``covers more requirements and reduces most risk, costs the least''~\cite{fea02a,me03i}.
\item Software process control using:
\bi
\item A Chung-Mypolopous soft-goal graph where fitter means ``cover more non-functional requirements''~\cite{me03j}.
\item The COCOMO and THREAT models where fitter means ``lower effort and fewer threats''~\cite{me00e};
\item The COCOMO and THREAT and COQUALMO
models where fitter means ``lower effort and fewer threats and lower defects''~\cite{me05e};
\item Qualitative inference diagrams where fitter means ``higher quality''~\cite{me06a}.
\item The IV\&V ``Silap'' model that selects work break down structures for V\&V where fitter means ``lower risks''~\cite{me06b};
\ei
\ei
\\\hline
Outside of NASA, \fig{cycle} has been previously applied to: \bi
\item Circuit design, with fitter being ``more bulbs shine''~\cite{me01g}. 
\item Finite state machines, where fitter means ``better chances of covering transitions''~\cite{me02h,owen02a}.
\item Economic policy where fitter means ``longer human life''~\cite{gel03a};
\item Whiskey production,  where fitter means ``more alcohol''~\cite{burk04};
\item Software process control using:
\bi
\item a CMM level 2 model, where fitter mean ``chance of reaching a lost cost project''~\cite{me01e};
\item Discrete event simulation where fitter was defined by a utility function combining quality, expense, and development time~\cite{me02f}.
\ei
\ei
\\\hline
\end{tabular}
\end{center}
\caption{Prior applications of \fig{cycle}.}\label{fig:old}
\end{figure}
The tool set that implements \fig{cycle}
using the COCOMO, COQUALMO, THREAT models and the
TAR3 treatment learner ~\cite{me03c}  is called
XOMO (pronounced ``x-o-mow'')~\cite{me05e}.
XOMO uses TAR3 
since
this learner
assumes
that
busy managers of software projects
don't need (or can't use) complex
models. Rather, busy people need to know the {\em least} they need to
do to achieve the {\em most} benefits. For example, when dealing with
complex situations with many unknowns (e.g. developing software),
it can be a wise tactic to focus your efforts on a small
number of key factors rather than expending great effort trying to
control all possibilities.
\begin{figure*}
\begin{center}
\includegraphics[width=5in]{metrics5.png}
\end{center}
\caption{Types of software measures. From~\cite{fenton00}. }\label{fig:metrics}
\end{figure*}




This paper applies  XOMO to a variety of 
models
taken from the
Reliable Software Test bench
under development at the AMES Research Center.
For example, 
one item in RST is ARES, a model of the guidance and navigation control (GNC) system of
\mbox{ARES1} (formerly known as the CEV). 
Our methods supported a fine-grained analysis of the merits
of various process options: e.g.
automated analysis vs execution-based testing.
When such execution-based testing is impractical, we could identify what could
be gained using just (e.g.) automatic analysis.
 

\section{Background}

\subsection{Related Work}

Fenton and Pfleeger~\cite{fenton97}
advise that measurements from software projects divide into
the three groups of \fig{metrics}:
\bi
\item
{\em Process measures}: describing {\em how} the software is built;
\item
{\em Product measures}: describing {\em what} is being built;
\item
{\em Resource measures}: describing  {\em who} is doing the building and {\em which}
tools are being used in the process.
\ei
Each of these measures have their own kind of risks:
\bi
\item
Process risks can arise when (say) the methods are applied to a a project;
\item
Product risks can arise when (say) a faulty piece of software is delivered;
\item
Resource risks can arise when (say) developers are given too little time to build
a complex device.
\ei
Risk reduction methods for all possible process, product, and resource risks
are beyond the scope of this paper. Previously~\cite{me00e} we
have followed
the lead of Madachy~\cite{madachy97},
and have focused  on the risks that can
be expressed using the COCOMO ontology \fig{emsf2}.
One advantage of COCOMO (and this is why we use it)
is that unlike  many other costing models such as 
PRICE-S~\cite{park88}, SEER-SEM~\cite{jensen83} or SLIM~\cite{putnam92},
COCOMO is an {\em
open model} so complete information is available on its
internal  structure~\cite{boehm81, boehm00b}.
Also, on NASA projects,
COCOMO
similar results to other cost models.
For example, in comparative
studies of SEER and COCOMO-II for CEV, similar cost estimates were generated.

\begin{figure}
\begin{center}
{\scriptsize
\begin{tabular}{l|r@{:~}l|}\cline{2-3}
scale   &prec & have we done this before?\\
factors &flex & development flexibility \\
        &resl & any risk resolution activities?\\
        &team &  team cohesion\\
        &pmat & process maturity\\\cline{2-3}

upper  &acap & analyst capability\\
       &pcap & programmer capability\\
       &pcon & programmer continuity\\
       &aexp &  analyst experience\\
       &pexp &  programmer experience\\
       &ltex &  language and tool experience\\
       &tool &  tool use\\
       &site &  multiple site development\\
       &sced & length of schedule   \\\cline{2-3}

lower &rely &    required reliability  \\
      &data &   secondary memory  storage requirements\\
      &cplx &  program complexity\\
      &ruse &  software reuse\\
      &docu &   documentation requirements\\
      &time &   runtime pressure\\
      &stor &   main memory requirements\\
     &pvol &    platform volatility  \\\cline{2-3}
\end{tabular}}
\end{center}
\caption[The COCOMO ontology]{
The COCOMO ontology.
Development effort is {\em exponentially}, 
{\em linearly}, and {\em negatively linearly}
proportional to the {\em scale factors}, {\em upper} terms, and {\em lower} terms (respectively).
}\label{fig:emsf2}
\end{figure}


To some degree, the COCOMO ontology addresses issues of process,product, and resource metrics.
For example, the COCOMO terms of  \fig{emsf2}  can be divided as follows:
\bi
\item 
{\em Resl, pmat, site} and {\em, sced} are process measures;
\item
{\em Prec, tool,  rely, data, cplx, ruse, docu, time, stor} 
and {\em pvol} are  product measures;
\item
{\em Flex, team, acap, pcap, pcon, aexp} and {\em ltex}  are  resource measures.
\ei
Nevertheless, the standard COCOMO does not address many other risk issues. The COCOMO team
recognizes this and new terms were added for the
COQUALMO defect prediction model\footnote{\url{http://csse.usc.edu/research/coqualmo/}}:
\bi
\item COQUALMO recognizes project phases: {\em requirements}, {\em design} and {\em coding};
\item COQUALMO also divides defect removal tools into {\em automated analysis, peer reviews}
and {\em execution testing  and tools}.
\ei
\subsection{ Are our Models ``Correct''?}
One drawback with our results is that they come from simulation and
not from empirical observations of real world development teams
applying the policy decisions made by the learners. If our models are
wrong (e.g. poorly calibrated) then our results are suspect.  

Our methodology partially addresses this concern. For example, the
COCOMO effort model contains two calibration parameters and the above
results hold for simulations across the space of possible
calibrations. That is, Monte Carlo plus data mining can find stable
conclusions within the space of possibilities (this is a conclusion we
have made elsewhere, many
times~\cite{me00e,me01g,me02c,me02f,me03c,me03j,me04a}).

Nevertheless, simulations across the space of options will never give
the right answers if that model is fundamentally flawed;
e.g. important domain factors are missing from the model.  This is a
problem with all model-based reasoning: if the model is wrong then the
reasoning is wrong as well. 

However, as George Box says, ``all models are wrong but some are useful''.
Certainly this
has been the recent experience in physics. Over the last 100 models,
numerous revisions to the atomic theory of matter have been proposed:
\begin{center}
\includegraphics[width=3.5in]{models.png}
\end{center}
Each new model was wrong since it was superseded by a newer model.
But each new model was useful in the sense that it explained more
effects than the previous model.

The lesson here is that committing to a model of the current best
understanding of a phenomenon is good practice, even if that model is
not ``correct'' in some absolute sense. And once that new model is
generated, it is right and proper that it be exercised, criticized,
and improved.  In the next few months, the XOMO models will be
used in panel sessions were experts will convene to debate cost and
risk models for autonomous NASA software. At those sessions, it is expected
that the XOMO models will be critiqued and extensively revised.

During those panels, it is important that the experts' time is put to best
use. NASA's top software and hardware
 experts are scarce and it will take a significant administrative effort
to collect them all together at the same place and at the same time. It is therefore
vital that no time be wasted in discussing irrelevancies. The XOMO toolkit
can be used to quickly prune debates about relatively unimportant issues. The panel
moderator could (gently) guide the discussion onto other matters if the matters
under debate have little effect on the model behaviors. Similar, the moderator
could ask XOMO for the next most important issue to discuss (that issue would
be the one that most changes the current model's behavior).

\section{Model Inputs: Slot Trees}

XOMO inputs a set of ranges, and some constraints on those ranges.
It then executes COCOMO, COQUALMO, and THREAT by selecting inputs drawn from
those constrained ranges. The resulting data set is then passed to our
learner.
After learning, XOMO outputs ``better'' constraints on those ranges; i.e. the subset
of the constraints that most select for (say) fewer defects, lower cost, fewer bugs.

XOMO's inputs are expressed the  {\em slot tree} format. A {\em slot tree}
is a tree of frames. Each frame contains slots. The root of the tree stores maximal ranges
for each variable and sub-frames 
stores restrictions on those values. In the current slot tree language, ranges can be defined
using one of four forms:

\bi
\item {X is int Min Max}
\item {X is real Min Max}
\item {X is normal Mean Sd}
\item {X is of item1 item2 item3 ....}
\ei

\noindent
For example:
{\scriptsize \begin{verbatim}
age    is int    0 120 
angle  is real   0 6.28
weight is normal 20 4 
city   is of     oxford london paris
\end{verbatim}}

\noindent
This root frame defines:
\bi
\item
  {\em age} to be an integer ranging from zero to 120;
\item
{\em angle} to be a real number ranging from 0 to 6.28; 
\item
{\em weight} to be a Gaussian with a mean of
20 and
standard deviation of 4; \item
and {\em city} to be one of {\em oxford}, {\em london}, or {\em paris}.
\ei
\begin{figure}
{\scriptsize\begin{alltt}
     1	# constants
     2	A     is real 2.25 3.25
     3	B     is real 0.9  1.1
     4	ksloc is int 2 10000
     5	
     6	automated_analysis          is int 1 6
     7	peer_reviews                is int 1 6
     8	execution_testing_and_tools is int 1 6
     9	
    10	# scale factors
    11	prec  is int 1 5
    12	flex  is int 1 5
    13	resl  is int 1 5
    14	team  is int 1 5
    15	pmat  is int 1 5
    16	
    17	# effort multipliers
    18	time  is int 3 6
    19	stor  is int 3 6
    20	data  is int 2 5
    21	pvol  is int 2 5
    22	ruse  is int 1 5
    23	rely  is int 1 5
    24	docu  is int 1 5
    25	acap  is int 1 5
    26	pcap  is int 1 5
    27	pcon  is int 1 5
    28	aexp  is int 1 5
    29	plex  is int 1 5
    30	ltex  is int 1 5
    31	tool  is int 1 5
    32	sced  is int 1 5
    33	cplx  is int 1 6
    34	site  is int 1 6
\end{alltt}}
\caption{The ranges in ``system''.}\label{fig:system}
\end{figure}

The root frame for XOMO is shown in \fig{system}
(note
that it offers default ranges from the COCOMO terms of
\fig{emsf2}).

In a slot tree, frames can have sub-frames and, recursively,
sub-frames can have sub-sub-frames. This tree of sub-frames can have two applications.
Firstly, it can be used to succinctly define a 
software project.  For example, \fig{ares} describes a NASA project
as a specialization of the general COCOMO ranges of \fig{system}. Note that \fig{ares}
does not need to mention all the COCOMO variables- variables that skipped in a leaf are inherited
from higher in the slot tree.

Leaf frames offer restrictions to root ranges (but can't define new ranges).
Leaf frames are connected to parent frames with the keyword $with$.
Restrictions
can be defined using one of two forms:
\bi
\item X just A B
\item X = Value
\ei
\noindent
For example, \fig{ares} shows some restrictions to \fig{system}. 
\begin{figure}
{\scriptsize\begin{alltt}
     1		system with ares
     2			ksloc just  75 125
     3			prec just 3 5
     4			flex = 3
     5			resl = 4
     6			team = 3      
     7			pmat just 4 5 
     8			rely = 5
     9			data = 4
    10			cplx = 4
    11			ruse = 4
    12			docu just 3 4
    13			time = 3
    14			stor = 3       
    15			pvol = 3
    16			acap = 4
    17			pcap = 3
    18			pcon = 3
    19			aexp = 4
    20			plex = 4
    21			ltex just 2 5 
    22			tool = 5
    23			site = 6
    24			sced just 2 4
\end{alltt}}
\caption[The ranges in ``ares''.]{The ranges in ``ares''.
ranges.
}\label{fig:ares}
\end{figure}

Unless some sub-tree restrains a range, the Monte Carlo simulator
will select randomly from some pre-defined range. For example, lines 2\&3
of \fig{system} show ranges for the two COCOMO effort model {\em calibration
parameters}. Standard practice is to tune COCOMO by tuning these
values using local data. When such local data is missing (e.g. in this
study), we use Monte Carlo simulation to explore a wide range of
possible calibration parameters.

Note also that ``ares'' restrains some, but not all,
of the ranges in ``system''. For example, ``ares'''s KSLOC guesstimates
(75 to 125) are much less than for arbitrary ``system'''s (2 to 10,000).
However,
there are 29 ranges in \fig{system} and only
23 in \fig{ares}; that is, all we know about ARES is not
enough to generate deterministic predictions about the effort,
defects, and threats to ARES software. 


The other role of slot trees is iterative learning.
XOMO's data miners run in cycles. In each cycle, a few hundred
Monte Carlo simulations are followed by some learning which, in turn, outputs some new  
restraints.  The outputs of each cycle can be a new slot tree leaf.

\begin{figure*}
{\footnotesize
\begin{center}
\begin{tabular}{|c|p{1.7in}|p{1.5in}|p{1.2in}|p{1.4in}|}\hline

 & Definition & Low-end
 &Medium &High-end\\\hline

\multicolumn{5}{l}{}\\
\multicolumn{5}{l}{Scale factors:}\\\hline
flex   &  development flexibility   & development process
rigorously defined & some guidelines, which can be relaxed & only
general goals defined\\\hline

pmat    & process maturity  &  CMM level 1 &   CMM level 3  &  CMM level 5 \\\hline

prec & precedentedness  &  we have never built this kind
of software before &    somewhat new &
thoroughly familiar \\\hline

resl &  architecture or risk resolution  &  few interfaces
defined or few risk eliminated  &  most interfaces defined or most
risks eliminated   & all interfaces defined or all risks
eliminated\\\hline

team  &   team cohesion  &  very difficult interactions &
basically co-operative  &  seamless interactions\\\hline

\multicolumn{5}{l}{}\\
\multicolumn{5}{l}{Effort multipliers}\\\hline
acap  &  analyst capability  &  worst 15\% &   55\%  &  best 10\% \\\hline

aexp   &  applications experience  &  2 months &   1 year  &  6 years\\\hline

cplx   &  product complexity   & e.g. simple read/write
statements & e.g. use of simple interface widgets  &  e.g.
performance-critical embedded systems\\\hline

data   &  database size 
(DB bytes/SLOC) &
10 & 100 &    1000 \\\hline

docu   &  documentation   & many life-cycle phases not
documented      & &  extensive reporting for each life-cycle phase\\\hline

ltex   &  language and tool-set experience   & 2 months  &  1
year & 6 years \\\hline

pcap   &  programmer capability  &  worst 15\%   & 55\%  &  best 10\% \\\hline


pcon   &  personnel continuity \newline
(\% turnover per year) &
    48\% &    12\%  & 3\% \\\hline

plex   &  platform experience  &  2 months  &  1 year  &  6 years\\\hline


pvol   &  platform volatility  ($\frac{frequency~of~major~changes}{frequency~of~minor~changes}$) &
$\frac{12~months}{1~month}$   & $\frac{6~months}{2~weeks}$ &
$\frac{2~weeks}{2~days}$\\\hline



rely   &  required
reliability &   errors mean slight inconvenience  &  errors are easily
recoverable   & errors can risk human life\\\hline




ruse   &  required
reuse &   none &    multiple program  & multiple product lines\\\hline

sced  &   dictated development\newline schedule &    deadlines moved closer to
75\% of the original estimate &  no change
&  deadlines moved back to  160\% of original estimate\\\hline

site   &  multi-site development   & some contact: phone, mail&
some email  &  interactive multi-media\\\hline

stor  &   main storage constraints  \newline (\% of available
RAM) & N/A
 &   50\% &  95\% \\\hline


time  &   execution time constraints \newline (\% of available CPU) &
N/A&     50\%
   &  95\% \\\hline


tool   &  use of software tools  &  edit,code,debug &&
integrated with life cycle\\\hline

\end{tabular}
\end{center}
} \caption[COCOMO: parameters]{Parameters of the COCOMO-II effort risk model; adapted
from~\protect\url{http://sunset.usc.edu/COCOMOII/expert_cocomo/drivers.html}.
``{\tt Stor}'' and ``{\tt time}'' score ``N/A"" for low-end values
since they have no low-end defined in COCOMO-II. }
\label{fig:cparems}
\end{figure*}





\section{Models}
At runtime, XOMO pulls values from slot trees and passes them to three models.

\subsection{The COCOMO Effort Model}

COCOMO measures effort in calendar months where one month is 152
hours (and includes development and management hours). 
COCOMO assumes that as systems grow
in size, the effort required to create them grows exponentially,
i.e. \mbox{$effort \varpropto KSLOC^x$}. More precisely, COCOMO-II uses the variable
of \fig{cparems} as follows:

{\small\begin{equation}\label{eq:coc2}
months=a*\left(KSLOC^{\left(b+0.01*\sum_{i=1}^{5}SF_i\right)}\right)*\left(\prod_{j=1}^{17}EM_j\right)
\end{equation}}
\begin{figure}
\begin{ccode}
function Effort() \{
  return A() * Ksloc() ^ E()   * Rely()* Data()* Cplx()* 
         Ruse()* Docu()* Time()* Stor()* Pvol()* Acap()* 
         Pcap()* Pcon()* Aexp()* Plex()* Ltex()* Tool()* 
         Site()* Sced() 
\}

function E() \{ 
  return B() + 0.01*(Prec() + Flex() 
             + Resl() + Team() + Pmat()) 
\}
\end{ccode}
\caption{COCOMO: computing effort.}\label{fig:em}
\end{figure}
where  $a$ and $b$ are  domain-specific parameter, and KSLOC is  estimated
directly or computed from a function point analysis.
 $SF_i$ are the scale factors (e.g. factors
such as ``have we built this kind
 of system before?'') and
$EM_j$ are the cost drivers
(e.g.
required level of reliability). Scale factors have an exponential impact on software cost while
effort multipliers have a linear impact.  

\fig{em} shows XOMO implementation of the COCOMO effort equation. 
Values such as (e.g.) {\tt flex=1}
get converted to numerics as follows. First, the integers
\{1, 2, 3, 4, 5, 6\} are converted to the symbols \{vl, l, n, h, vh, xh\} (respectively) representing
very low, low, nominal, high, very high, and extremely high. Next, these are mapped into
the look-up table
of \fig{inside}.

\begin{figure}
{\scriptsize
\begin{center}
\begin{tabular}{crrrrrrr}
    &  &  vl&    l&    n&    h&   vh&   xh\\\hline
\multicolumn{8}{l}{{\em Scale factors:}}\\
flex&    & 5.07& 4.05& 3.04& 2.03& 1.01&  \\
pmat&    & 7.80& 6.24& 4.68& 3.12& 1.56&\\
prec&    & 6.20& 4.96& 3.72& 2.48& 1.24&   \\
resl&    & 7.07& 5.65& 4.24& 2.83& 1.41& \\
team&    & 5.48& 4.38& 3.29& 2.19& 1.01&  \\\hline
\multicolumn{8}{l}{{\em Effort multipliers:}}\\
acap&    & 1.42& 1.19& 1.00& 0.85& 0.71 & \\   
aexp&    & 1.22& 1.10& 1.00& 0.88& 0.81&    \\ 
cplx&    & 0.73& 0.87& 1.00& 1.17& 1.34& 1.74\\
data&    &     & 0.90& 1.00& 1.14& 1.28&     \\
docu&    & 0.81& 0.91& 1.00& 1.11& 1.23&     \\
ltex&    & 1.20& 1.09& 1.00& 0.91& 0.84&     \\
pcap&    & 1.34& 1.15& 1.00& 0.88& 0.76&     \\
pcon&    & 1.29& 1.12& 1.00& 0.90& 0.81&     \\
plex&    & 1.19& 1.09& 1.00& 0.91& 0.85&     \\
pvol&    &     & 0.87& 1.00& 1.15& 1.30&     \\
rely&    & 0.82& 0.92& 1.00& 1.10& 1.26&     \\
ruse&    &     &0.95& 1.00& 1.07& 1.15& 1.24\\
sced&    & 1.43& 1.14& 1.00& 1.00& 1.00&\\
site&    & 1.22& 1.09& 1.00& 0.93& 0.86& 0.80\\
stor&    &     &     & 1.00& 1.05& 1.17& 1.46\\
time&    &     &     & 1.00& 1.11& 1.29& 1.63\\
tool&    & 1.17& 1.09& 1.00& 0.90& 0.78&    
\end{tabular}
\end{center}}
\caption{COCOMO: co-efficients}\label{fig:inside}
\end{figure}
\subsection{THREATS: a Heuristic Risk Model}

The Madachy Heuristic schedule THREAT model was an
experiment in explicating the heuristic nature of effort
estimation. It returns a heuristic estimate of the chances of a
schedule over run in the project. 
Values of 0-5 are considered to be
``low threat''; 5-15 ``medium threat''; 15-50 ``high threat''; and 50-100
``very high threat''. Studies with the COCOMO-I project database have shown
that the Madachy THREAT index correlates well with $\frac{months}{KDSI}$
(where KDSI is thousands of delivered source lines of
code)~\cite{madachy97}.


Internally, the model contains dozens of tables of the
form of \fig{madeg}. Each such table  adds some ``threatiness'' value
to the overall project threat.
These 
tables are read as follows. Consider the exceptional case of building
 high
reliability systems with very tight schedule pressure (i.e. {\tt
sced=vl} or and {\tt  rely=vh} or {\tt vh}).
Recalling \fig{inside}, the
 COCOMO co-efficients for these ranges are
1.43 (for
{\tt sced=vl}) and 1.26 (for
{\tt rely=vh}). These co-efficients also have a threat factor of
2 (see \fig{madeg}). Hence, a project with these two attribute ranges
would contribute 1.43*1.26*2=3.6036 to the schedule threat.

The details of the THREAT calculations  are shown in \fig{srm}. The
threat  tables of the current model are shown in \fig{madachy}. 

\begin{figure}
{\scriptsize
\begin{center}
\begin{tabular}{|p{0.8in}|c|c|c|c|c|}\cline{2-6}
\multicolumn{1}{c|}{} &rely=&rely=&rely=&rely=&rely=\\
\multicolumn{1}{c|}{}& very & low &nominal & high & very\\
\multicolumn{1}{c|}{}&  low&  & &  &  high\\\hline sced= very
low& 0 & 0 & 0  &1  &2\\\hline sced= low     & 0 &0  & 0 &
0&1\\\hline sced= nominal& 0 & 0 & 0  &0  &0\\\hline sced= high&
0& 0 & 0  & 0 &0\\\hline sced= very high& 0 & 0 & 0  &0 &0\\\hline
\end{tabular}
\end{center}
} \caption{THREAT: an example threat table}\label{fig:madeg}
\end{figure}
\begin{figure}
\begin{ccode}

Total_threat =
 (Schedule_threat  + Product_threat   + 
  Personnel_threat + Process_threat   + 
  Platform_threat  + Reuse_threat)/3.73 


Schedule_threat=
  Sced_Rely_threat + Sced_Time_threat + 
  Sced_Pvol_threat + Sced_Tool_threat + 
  Sced_Acap_threat + Sced_Aexp_threat + 
  Sced_Pcap_threat + Sced_Plex_threat + 
  Sced_Ltex_threat + 
  Sced_Pmat_threat


Product_threat =
  Rely_Acap_threat + Rely_Pcap_threat + 
  Cplx_Acap_threat + Cplx_Pcap_threat + 
  Cplx_Tool_threat + Rely_Pmat_threat +  
  Sced_Cplx_threat + Sced_Rely_threat + 
  Sced_Time_threat + Ruse_Aexp_threat + 
  Ruse_Ltex_threat


Personnel_threat =
  Pmat_Acap_threat + Stor_Acap_threat + 
  Time_Acap_threat + Tool_Acap_threat + 
  Tool_Pcap_threat + Ruse_Aexp_threat + 
  Ruse_Ltex_threat + Pmat_Pcap_threat + 
  Stor_Pcap_threat + Time_Pcap_threat + 
  Ltex_Pcap_threat + Pvol_Plex_threat + 
  Sced_Acap_threat + Sced_Aexp_threat + 
  Sced_Pcap_threat + Sced_Plex_threat + 
  Sced_Ltex_threat + Rely_Acap_threat + 
  Rely_Pcap_threat + Cplx_Acap_threat + 
  Cplx_Pcap_threat + Team_Aexp_threat


Process_threat =
  Tool_Pmat_threat + Time_Tool_threat + 
  Tool_Pmat_threat + Team_Aexp_threat + 
  Team_Sced_threat + Team_Site_threat + 
  Sced_Tool_threat + Sced_Pmat_threat + 
  Cplx_Tool_threat + Pmat_Acap_threat + 
  Tool_Acap_threat + Tool_Pcap_threat + 
  Pmat_Pcap_threat


Platform_threat =
  Sced_Time_threat + Sced_Pvol_threat + 
  Stor_Acap_threat + Time_Acap_threat + 
  Stor_Pcap_threat + Pvol_Plex_threat + 
  Time_Tool_threat


Reuse_threat =
  Ruse_Aexp_threat + Ruse_Ltex_threat



\end{ccode}
\caption{THREAT: the calculations.}\label{fig:srm}
\end{figure}



\begin{figure*}
\begin{center}
{\scriptsize
\begin{tabular}{l|rrrrrrr}
&& vl&  l&  n&  h& vh& xh\\\hline
&rely\\
sced&        vl&  &  &  &  1&  2&\\ 
     &    l&  &  &  &  &  1& \\\hline
&cplx\\
sced&          vl&  &  &  &  1&  2&  4\\
&         l&  &  &  &  &  1&  2\\
&         n&  &  &  &  &  &  1\\\hline
&time\\
sced&        vl& & &  &  1&  2&  4\\
 &        l& & &  &  &  1&  2\\
  &       n& & &  &  &  &  1\\\hline
&pvol\\
sced&        vl&  &  &  &  1&  2& \\
&         l&  &  &  &  &  1& \\\hline
&tool\\
sced&        vl&  2&  1&  &  &  & \\
 &        l&  1&  &  &  &  & \\\hline
&pexp\\
sced&        vl&  4&  2&  1&  &  & \\
&         l&  2&  1&  &  &  & \\
&         n&  1&  &  &  &  & \\\hline
&pcap\\
sced&        vl&  4&  2&  1&  &  & \\
  &       l&  2&  1&  &  &  & \\
   &      n&  1&  &  &  &  & \\\hline
&aexp\\
sced&        vl&  4&  2&  1&  &  & \\
     &    l&  2&  1&  &  &  & \\
      &   n&  1&  &  &  &  & \\\hline
&acap\\
sced&        vl&  4&  2&  1&  &  & \\
  &       l&  2&  1&  &  &  & \\
  &       n&  1&  &  &  &  & \\\hline
&ltex\\
sced&        vl&  2&  1&  &  &  & \\
     &    l&  1&  &  &  &  & \\\hline
&pmat\\
sced&        vl&  2&  1&  &  &  & \\
 &        l&  1&  &  &  &  & \\
\multicolumn{3}{c}{}\\
\multicolumn{3}{c}{}\\
\end{tabular}~~~~~~~~~~~~~~~~~~~~~~~~
\begin{tabular}{l|rrrrrrr}
          && vl&  l& n\\\hline
&acap\\
rely&         n&  1&  &   \\
&         h&  2&  1& \\
 &       vh&  4&  2&  1 \\\hline
&pcap\\
rely&         n&  1&  & \\
     &    h&  2&  1& \\
      &  vh&  4&  2&  1 \\\hline
&acap\\
cplx&         h&  1&  & \\
&        vh&  2&  1& \\
 &       xh&  4&  2&  1 \\\hline
&pcap\\
cplx&         h&  1&  & \\
     &   vh&  2&  1& \\
      &  xh&  4&  2&  1 \\\hline
&tool\\
cplx&         h&  1&  & \\
     &   vh&  2&  1& \\
      &  xh&  4&  2&  1 \\\hline
&pmat\\
rely&         n&  1&  &   \\
     &    h&  2&  1&    \\
      &  vh&  4&  2&  1  \\\hline
&acap\\
pmat&        vl&  2&  1& \\
     &    l&  1&  & \\\hline
&acap\\
stor&         h&  1&  & \\
   &     vh&  2&  1& \\
   &     xh&  4&  2&  1 \\\hline
&acap\\
time&         h&  1&  & \\
    &    vh&  2&  1& \\
    &    xh&  4&  2&  1 \\\hline
&acap\\
tool&        vl&  2&  1& \\
    &     l&  1&  & \\\hline
&pcap\\
tool&        vl&  2&  1& \\
     &    l&  1&  & \\
\end{tabular}~~~~~~~~~~~~~~~~~~~~~~~~
\begin{tabular}{l|rrrr}
&& vl&  l&  n \\\hline
&aexp\\
ruse&         h&  1&  & \\
     &   vh&  2&  1& \\
      &  xh&  4&  2&  1 \\\hline
&ltex\\
ruse&         h&  1&  & \\
     &   vh&  2&  1& \\
      &  xh&  4&  2&  1\\\hline
&pcap\\
pmat&        vl&  2&  1& \\
     &    l&  1&  & \\\hline
&pcap\\
stor&         h&  1&  & \\
 &       vh&  2&  1& \\
  &      xh&  4&  2&  1 \\\hline
&pcap\\
time&         h&  1&  & \\
     &   vh&  2&  1& \\
      &  xh&  4&  2&  1 \\\hline
&pcap\\
ltex&        vl&  4&  2&  1 \\
     &    l&  2&  1& \\
        & n&  1&  & \\\hline
&pexp\\
 pvol&        h&  1&  & \\
      &  vh&  2&  1& \\\hline
&pmat\\
 tool&       vl&  2&  1&    \\
      &   l&  1&  &    \\\hline
&tool\\
 time&       vh&  1&  & \\
      &  xh&  2&  1& \\\hline
&aexp\\
 team&       vl&  2&  1&   \\
      &   l&  1&  &    \\\hline
&sced\\
team&        vl&  2&  1&    \\
     &         l&  1&  &    \\\hline
&site\\
team&        vl&  2&  1&    \\
     &        l&  1&  &   
\end{tabular}}
\end{center}
\caption[THREAT: the details.]{THREAT: the details. For example, 
looking at the top-left matrix,
the {\tt Sced\_Rely\_risk} is highest when the reliability
is very high but the schedule pressure is very tight.}\label{fig:madachy}
\end{figure*}

\begin{figure*}
{\scriptsize
\begin{center}
\begin{tabular}{clllllllllllllllll}
    &rely&data&ruse&docu&cplx&time&stor&pvol&acap&pcap&pcon&aexp&plex&ltex&tool&site&sced\\\hline
\multicolumn{12}{l}{{\em requirements:}}\\
xh&&&1.05&&1.32&1.08&1.08&1.16&&&&&&&&0.83\\
vh&0.7&1.07&1.03&0.86&1.21&1.05&1.05&1.1&0.75&1&0.82&0.81&0.9&0.93&0.92&0.89&0.85\\
h &0.85&1.04&1.02&0.93&1.1&1.03&1.03&1.05&0.87&1&0.91&0.91&0.95&0.97&0.96&0.95&0.92\\
n&1&1&1&1&1&1&1&1&1&1&1&1&1&1&1&1&1\\
l&1.22&0.93&0.95&1.08&0.88&&&0.86&1.17&1&1.11&1.12&1.05&1.04&1.05&1.1&1.09\\
vl&1.43&&&1.16&0.76&&&&1.33&1&1.22&1.24&1.11&1.07&1.09&1.2&1.18\\\hline
\multicolumn{12}{l}{{\em design:}}\\
xh&&&1.02&&1.41&1.2&1.18&1.2&&&&&&&&0.83\\
vh&0.69&1.1&1.01&0.85&1.27&1.13&1.12&1.13&0.83&0.85&0.8&0.82&0.86&0.88&0.91&0.89&0.84\\
h&0.85&1.05&1&0.93&1.13&1.06&1.06&1.06&0.91&0.93&0.9&0.91&0.93&0.91&0.96&0.95&0.92\\
n&1&1&1&1&1&1&1&1&1&1&1&1&1&1&1&1&1\\
l&1.23&0.91&0.98&1.09&0.86&&&0.83&1.1&1.09&1.13&1.11&1.09&1.07&1.05&1.1&1.1\\
vl&1.45&&&1.18&0.71&&&&1.2&1.17&1.25&1.22&1.17&1.13&1.1&1.2&1.19\\\hline
\multicolumn{12}{l}{{\em coding:}}\\
xh&&&1.02&&1.41&1.2&1.15&1.22&&&&&&&&0.85\\
vh&0.69&1.1&1.01&0.85&1.27&1.13&1.1&1.15&0.9&0.76&0.77&0.88&0.86&0.82&0.8&0.9&0.84\\
h&0.85&1.05&1&0.92&1.13&1.06&1.05&1.08&0.95&0.88&0.88&0.94&0.94&0.91&0.9&0.95&0.92\\
n&1&1&1&1&1&1&1&1&1&1&1&1&1&1&1&1&1\\
l&1.23&0.91&0.98&1.09&0.86&&&0.82&1.05&1.16&1.15&1.07&1.08&1.11&1.13&1.09&1.1\\
vl&1.45&&&1.18&0.71&&&&1.11&1.32&1.3&1.13&1.16&1.22&1.25&1.18&1.19
\end{tabular}
\end{center}}
\caption{COQUALMO: effort multipliers  and defect introduction}\label{fig:COQUALMOemin}
\end{figure*}

\subsection{COQUALMO: defect introduction and removal}


COQUALMO models how process options {\em add} and {\em remove}
defects to software during {\em requirements}, {\em design}, and 
{\em coding}. 
For example, poor documentation leads to more
errors since developers lack the guidance required to code the right
system. So, \fig{COQUALMOemin} offers its large defect introduction
values when the effort multiplier {\tt docu=vl} is
very low.
See also  \fig{COQUALMOsfin} for the defects introduced by various
settings to the scale factors.

\begin{figure}
{\scriptsize
\begin{center}
\begin{tabular}{clllll}
   &prec&flex&resl&team&pmat\\\hline
\multicolumn{6}{l}{{\em requirements:}}\\
xh&0.7&1&0.76&0.75&0.73\\
vh&0.84&1&0.87&0.87&0.85\\
h&0.92&1&0.94&0.94&0.93\\
n&1&1&1&1&1\\
l&1.22&1&1.16&1.17&1.19\\
vl&1.43&1&1.32&1.34&1.38\\\hline
\multicolumn{6}{l}{{\em design:}}\\
xh&0.75&1&0.7&0.8&0.61\\
vh&0.87&1&0.84&0.9&0.78\\
h&0.94&1&0.92&0.95&0.89\\
n&1&1&1&1&1\\
l&1.17&1&1.22&1.13&1.33\\
vl&1.34&1&1.43&1.26&1.65\\\hline
\multicolumn{6}{l}{{\em coding:}}\\
xh&0.81&1&0.71&0.86&0.63\\
vh&0.9&1&0.84&0.92&0.79\\
h&0.95&1&0.92&0.96&0.9\\
n&1&1&1&1&1\\
l&1.12&1&1.21&1.09&1.3\\
vl&1.24&1&1.41&1.18&1.58
\end{tabular}
\end{center}}
\caption{COQUALMO: scale factors and defect introduction}\label{fig:COQUALMOsfin}
\end{figure}

As shown in \fig{din} the COQUALMO defect introduction factors
are effects-per-1000 lines of code. A small weighting factor (10,20,30)
is added to show an increasing number of defects as the life cycle
progresses.

\begin{figure}
\begin{ccode}
function defectsIntroduced() \{
 return 10*Ksloc()*defectsIntroduced1("requirements") + 
        20*Ksloc()*defectsIntroduced1("design") + 
        30*Ksloc()*defectsIntroduced1("coding") \}
                      
function defectsIntroduced1(table) \{
  # return the product of the \fig{COQUALMOemin} and 
  # and the \fig{COQUALMOsfin} figures  \}
\end{ccode}
\caption{COQUALMO: defects introduced.}\label{fig:din}
\end{figure}


\begin{figure}
\begin{center}
{\scriptsize
\begin{tabular}{lrrr}
&automated&peer&execution\_testing\\
&analysis&reviews&\_and\_tools\\\hline
\multicolumn{4}{l}{{\em requirements:}}\\
xh           &0.4               &0.7         &0.6 \\
vh           &0.34              &0.58        &0.57\\
h            &0.27              &0.5         &0.5\\
n            &0.1               &0.4         &0.4\\
l            &0                 &0.25        &0.23\\
vl           &0                 &0           &0\\\hline
\multicolumn{4}{l}{{\em design:}}\\
xh           &0.5               &0.78        &0.7\\
vh           &0.44              &0.7         &0.65\\
h            &0.28              &0.54        &0.54\\
n            &0.13              &0.4         &0.43\\
l            &0                 &0.28        &0.23\\
vl           &0                 &0           &0\\\hline
\multicolumn{4}{l}{{\em coding:}}\\
xh           &0.55              &0.83        &0.88\\
vh           &0.48              &0.73        &0.78\\
h            &0.3               &0.6         &0.69\\
n            &0.2               &0.48        &0.58\\
l            &0.1               &0.3         &0.38\\
vl           &0                 &0           &0
\end{tabular}}
\end{center}
\caption{COQUALMO: defect removal}\label{fig:drf}
\end{figure}

\begin{figure}
\begin{ccode}
function Total_defects() \{
  return defects("requirements",Coqualr) + 
         defects("design",      Coquald) + 
         defects("coding",      Coqualc)  
\} 

function defects(what,table) \{
  introduced     = defectsIntroduced1(what,table);
  percentRemoved = defectsRemovedRatio(what);
  return percentRemoved*introduced 
\}

\end{ccode}
\caption{COQUALMO: defects added and removed}\label{fig:addremove}
\end{figure}


\begin{figure}
\begin{ccode}
function defectsRemovedRatio(table, auto,review,tool) \{
  return (1 - drf(table,"automated_analysis")) *
         (1 - drf(table,"peer_reviews")) *
         (1 - drf(table,"execution_testing_and_tools")) 
\}

function drf(table,x ) \{    
  # return x's value in table from \fig{drf} 
\}

\end{ccode}
\caption{COQUALMO: ratio of defects removed}\label{fig:rdefects}
\end{figure}

The defects remaining in software is the product of the defects
introduced times the percentage removed (see \fig{drf} and \fig{addremove}).
The removal percentage is calculated in
\fig{rdefects} which shows how various actions ({\tt
automated analysis}, {\tt peer reviews}, and {\tt execution testing
and tools}) remove defects during {\em requirements}, {\em design} and
{\em coding}. These values are ratios per 1000 lines of code so their
complement represents the remaining defects (see \fig{rdefects}).

\section{Handling the Output: Learning}

The output from the above learners are passe to the BORE pre-processor,
then to the TAR3 treatment learner.
\subsection{Multi-Dimensional Optimization using ``BORE''}

Our goal is reducing development effort {\em and} the risk of schedule risk {\em and}
the defect density in our code. Optimizing for all these three goals can be difficult.
The last 3 columns of \fig{xomoouts} show some scores from COCOMO, the risk model, and 
COQUALMO. The rows are sorted by the COQUALMO scores; i.e. by
the estimated number of defects per 1000 lines of code. Interestingly, high
number of remaining defects are not correlated with high schedule risk or development effort:
\bi
\item
The second and last rows have {\em similar} efforts but very {\em different} defect densities.
\item
Row two has the {\em highest} schedule threat but one of the {\em lowest} defect densities.
\ei
\begin{figure}
\begin{center}{\em
{\footnotesize
\begin{tabular}{r@{~}r@{~}rr@{~}r@{~}r@{~}r@{~}|r@{~}|r@{~}|r@{~}}
\multicolumn{7}{c}{26 inputs}&\multicolumn{3}{|c}{3 outputs}\\\hline
&& && & &  &&schedule&\\
rely&  plex& ksloc&\dots&pcap& time&aa&effort&threats&defects\\\hline

5&1&118.80&\dots&5&3&5&2083&69&0.50\\
5&1&105.51&\dots&1&3&5&4441&326&0.86\\
5&4&89.26&\dots&3&5&3&1242&63&0.96\\
5&2&89.66&\dots&1&4&5&2118&133&2.30\\
5&1&105.45&\dots&2&4&5&6362&170&2.66\\
5&3&118.43&\dots&2&6&2&7813&112&4.85\\
5&4&110.84&\dots&4&4&4&4449&112&6.81\\ 
\ldots
\end{tabular}}
}\end{center}
\caption{Some sample XOMO output.}\label{fig:xomoouts}
\end{figure}

The reason for these non-correlations is simple: even though
the three models within XOMO using the {\em same} variables, they
predict
for {\em different} goals. This complicates optimization since 
 any gain achieved in one dimension may have detrimental effects on other dimensions.

To model this multi-dimensional optimization problem, XOMO uses a 
multi-dimensional classification scheme called BORE (short for ``best or rest'').
BORE maps simulator
outputs into a hypercube which has  one dimension for each utility; in our case,
one dimension for effort, remaining defects, and schedule risk,
These utilities are 
normalized to ``zero'' for ``worst'', and ``one'' for ``best''.  The corner
of the hypercube at 1,1,... is the {\em apex} of the cube and represents
the desired goal for the system.
All the examples are scored by their
normalized
Euclidean distance to the apex. 

For each run $i$ of the simulator, these three outputs where normalized to the range $0..1$ as follows:
\[\begin{array}{rcl}
X_i & = & \frac{cocomo_i - min(cocomo)}{max(cocomo) - min(cocomo)}\\
~\\
Y_i & = & \frac{coqualmo_i - min(coqualmo)}{max(coqualmo) - min(coqualmo)}\\
~\\
Z_i & = & \frac{threat_i - min(threat)}{max(treat) - min(threat)}
\end{array}
\]
The Euclidean distance of $\{X_i,Y_i,Z_i\}$ to the ideal position of zero effort ($X_i=0$),
zero defects ($Y_i=0$) and zero threats ($Z_i=0$) was then computed and normalized to the range $0..1$ as follows:

\[W_i= 1 - \frac{\sqrt{X_i^2 + Y_i^2 + Z_i^2}}{\sqrt{3}}\]

\noindent
$W_i$ has the following properties:
\bi
\item $0 \le W_i \le 1$.
\item
The {\em higher} $W_i$, the {\em better} the run. 
\item 
$W_i$ is reduced by increasing  {\em any} of the
COCOMO effort, COQUALMO defect, or THREAT index scores.
That is, improving $W_i$ can only be achieved by  decreasing
{\em all} the effort, defects and treat scores  from all the models.
\ei

To determine the ``best'' and ``rest'' values,
all the $W_i$ scores were sorted.  The top 33\% were then classified as ``best''
and the remainder as ``rest''. For example, BORE might divide \fig{xomoouts} into
\fig{bore0}. 

\begin{figure}
\begin{center}{\em
{\footnotesize
\begin{tabular}{r@{~}r@{~}rr@{~}r@{~}r@{~}r@{~}|r@{~}|r@{~}|r@{~}}
rely&  plex& ksloc&\dots&pcap& time&aa&effort&secdRisk&defects\\\hline
{\em best:}\\
5&4&89.26&\ldots&3&5&3&1242&63&0.96\\
5&1&118.80&\ldots&5&3&5&2083&69&0.50\\
5&2&89.66&\ldots&1&4&5&2118&133&2.30\\\hline
{\em  rest:}\\
5&1&105.51&\ldots&1&3&5&4441&326&0.86\\
5&4&110.84&\ldots&4&4&4&4449&112&6.81\\
5&3&118.43&\ldots&2&6&2&7813&112&4.85\\
\end{tabular}}
}\end{center}
\caption{BORE: classification of \protect\fig{xomoouts}.}\label{fig:bore0}
\end{figure}

\begin{figure}[!t]
 {\em
\begin{center}
\begin{tabular}{rcccl}
outlook & temp($^o$F) &  humidity & windy? & class
\\\hline
sunny&  85&     86&     false&  none\\
sunny&     80&     90&     true&   none\\
sunny&     72&     95&     false&  none\\
rain&  65&     70&     true&         none\\
rain&        71&     96&     true&   none\\\hline
      rain&         70&     96&     false&      some\\
      rain&      68&     80&     false&      some\\
      rain&      75&     80&     false&  some\\\hline
      sunny&    69&     70&     false&  lots \\
      sunny&    75&     70&     true&   lots\\
      overcast&  83&     88&     false&  lots\\
      overcast&  64&     65&     true&   lots\\
      overcast&  72&     90&     true&   lots\\
     overcast&  81&     75&     false&  lots\\\hline
\end{tabular}\end{center}}
\caption{TAR3: Playing golf.}\label{fig:golfdata}
\end{figure}



\begin{figure*}
\begin{center}
{\footnotesize
\begin{tabular}{l|p{2in}|l@{~}p{1.5in}|l@{~}p{1.8in}}
input:
 &{\footnotesize
\begin{alltt}
SELECT class
FROM golf
\end{alltt}} &&
{\footnotesize
\begin{alltt}
SELECT class
FROM golf
WHERE
outlook = 'overcast'
\end{alltt}} &&
{\footnotesize
\begin{alltt}
SELECT class
FROM golf
WHERE
humidity >= 90
\end{alltt}}\\\hline
%&&&\\
output: &{\footnotesize
\begin{alltt}
none none none none none 
some some some lots lots 
lots lots lots lots
\end{alltt}}&&
{\footnotesize
\begin{alltt}
lots lots lots lots\end{alltt}} && {\footnotesize
\begin{alltt}
none none none some lots
\end{alltt}}\\\hline
&&&&&\\
distributions:&~~~~~~\begin{barenv} \setstyle{\sffamily} \setwidth{8}
     \setstretch{5}
  \setyaxis{0}{6}{2}
\sethspace{1} \setnumberpos{down} \bar{5}{1}
 \bar{3}{4}
 \bar{6}{8}
\end{barenv}&&~~~~~~
\begin{barenv}
\setstyle{\sffamily} \setwidth{8}
 \setstretch{5}
  \setyaxis{0}{6}{2}
\sethspace{1} \setnumberpos{down} \bar{0}{1} \bar{0}{4} \bar{4}{8}
\end{barenv}&&~~~~~~
\begin{barenv}
\setstyle{\sffamily} \setwidth{8}
 \setstretch{5}
  \setyaxis{0}{6}{2}
\sethspace{1} \setnumberpos{down} \bar{3}{1} \bar{1}{4} \bar{1}{8}
\end{barenv}~\\\\
\end{tabular}}

legend: \protect\legend1{none} \protect\legend4{some}
\protect\legend8{lots}

\end{center}

~\\

 \caption[TAR3:
Class distributions]{TAR3: Class distributions selected by different conditions in \protect\fig{golfdata}.  }
\label{fig:golft}
\end{figure*}
 
\subsection{Treatment Learning with TAR3}


Once the above models run, and BORE classifies the output into $best$
and $rest$, a data miner is used to find input settings that select
for the better outputs. This study uses the TAR3 data miner since this
learning method return the {\em smallest} theories that {\em most}
effect the output. In terms of software process changes, such
minimal  theories are useful since they require the fewest
management actions to improve a project.


TAR3
inputs a set of training examples 
$E$. Each example maps a set of attribute
ranges  to some class symbol; i.e.  $\{R_i,R_j,... \rightarrow C\}$
The class symbols $C_1, C_2..$ are stamped with some 
 utility score that ranks the classes; i.e. $\{U_1 < U_2 < .. < U_C\}$.
With $E$, these classes occur at frequencies
$F_1\%,F_2\%, ..., F_C\%$.
A ``treatment'' 
$T$ of size $X$ is a conjunction of attribute ranges  $\{R_1 \wedge R_ 2 ... \wedge R_X\}$.
Some subset of $e \subseteq E$ are consistent with the treatment.
In that subset, the classes occur at  frequencies
$f_1\%,f_2\%, ... f_C\%$.
TAR3 seeks the
seek smallest $T$ which most changes the weighted sum of the utilities times frequencies
of the classes. Formally, this is called the $lift$ of a treatment:
\[lift=\frac{\sum_C U_Cf_C}{\sum_C U_CF_C}\]


For example, consider the log of golf
playing behavior seen in \fig{golfdata}. In that log, we only play
{\em lots} of golf in $\frac{6}{5+3+6}= 43\%$ of cases. To improve
our game, we might search for conditions that increases our
golfing frequency. Two such conditions are shown in the {\tt
WHERE} test of the select statements  in \fig{golft}. In the case
of {\tt outlook= overcast}, we play {\em lots} of golf all the
time. In the case of {\tt humidity $\le$ 90}, we only play {\em
lots} of golf in 20\% of cases.  So one way to play lots of golf
would be to select a vacation location where it was always
overcast. While on holidays, one thing to watch for is the
humidity: if it rises over 90\%, then our frequent golf games are
threatened.


The tests in the {\tt WHERE} clause of the select statements in
\fig{golft} is a treatment. Classes in treatment learning
get a score $U_C$
and the learner uses this to assess the class
frequencies resulting from {\em applying a treatment} (i.e. using
them in a {\tt WHERE} clause). In normal operation, a treatment learner
does {\em
controller learning} that finds a treatment which selects for
better classes and reject worse classes  By reversing the scoring
function, treatment learning can also select for the worse classes
and reject the better classes. This mode is called {\em monitor
learning} since it finds the thing we should most watch for. In
the golf example, {\em outlook = 'overcast'} was the controller
and $humidity \ge 90$ was the monitor.

Formally, treatment learning is a weighted-class minimal
contrast-set association rule learner.  The treatments are associations that occur
with preferred classes. These treatments serve to contrast undesirable situations with
desirable situation where more of the outcomes are favorable. Treatment learning
is different to other contrast set learners like
STUCCO~\cite{bay99} since those other learners don't focus on minimal theories.

Conceptually, a treatment learner explores all possible subsets of the attribute
ranges looking for good treatments. Such a search is impractical
in practice so the art of treatment learning is quickly pruning unpromising attribute
ranges. This study uses the
TAR3 treatment learner~\cite{ying02} that uses  stochastic search
to find its treatments. 

\subsection{Iterative TAR3}
Sometimes, one round of TAR3 is not enough.
{\em Iterative TAR3} runs by conducting multiple Monte Carlo
simulations over the ranges of any uncertain variables. For example,
there are 28 variables in the XOMO models:
\bi
\item
Ksloc;
\item
5 scale factors;
\item
17 effort multipliers;
\item
2 calibration parameters (``{\tt a,b}'');
\item
3 defect removal activities (automated analysis, peer reviews,
execution testing and  tools).  
\ei 
In the case studies below, only a partial description of some of these variables are available.
Hence, learning is a process of sampling from the known constraints, finding the best
treatment, then revising the constraints.
 After, say,
1000 Monte Carlo runs, 
BORE classifies the outputs as (say) either the
333 $best$ or 667 $rest$.  The treatment learner studies the
results and notes which input ranges select for $best$.
The ranges found by the learner then become restraints for 
future
simulations. The whole cycle looks like this:

{\footnotesize
\[\begin{array}{c}
restraints_i \rightarrow simulation_i \rightarrow learn \rightarrow \\
\rightarrow restraints_{i+i} \rightarrow simulation_{i+1}
\end{array}\]}
\begin{figure*}
\centering {\small
\begin{tabular}{c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}|c@{~}c@{~}c@{~}c@{~}c@{~}c}
\begin{sideways}=execution testing and tools\end{sideways}&\begin{sideways}flex\end{sideways}&\begin{sideways}stor\end{sideways}&\begin{sideways}aexp\end{sideways}&\begin{sideways}docu\end{sideways}&\begin{sideways}site\end{sideways}&\begin{sideways}plex\end{sideways}&\begin{sideways}cplx\end{sideways}&\begin{sideways}ltex\end{sideways}&\begin{sideways}peer reviews\end{sideways}&\begin{sideways}\$ksloc\end{sideways}&\begin{sideways}rely\end{sideways}&\begin{sideways}data\end{sideways}&\begin{sideways}ruse\end{sideways}&\begin{sideways}acap\end{sideways}&\begin{sideways}?\$A\end{sideways}&\begin{sideways}sced\end{sideways}&\begin{sideways}automated analysis\end{sideways}&\begin{sideways}pcon\end{sideways}&\begin{sideways}?\$B\end{sideways}&\begin{sideways}pcap\end{sideways}&\begin{sideways}tool\end{sideways}&\begin{sideways}time\end{sideways}&\begin{sideways}prec\end{sideways}&\begin{sideways}pvol\end{sideways}&\begin{sideways}resl\end{sideways}&\begin{sideways}team\end{sideways}&\begin{sideways}pmat\end{sideways}&\begin{sideways}$<$\$risk\end{sideways}&\begin{sideways}$<$\$effort\end{sideways}&\begin{sideways}$<$\$defects\end{sideways}\\\hline
1&3&3&4&4&6&4&4&5&6&116&5&4&4&4&3.10408&3&6&3&0.915776&3&5&3&4&3&4&3&5&0.68&345.87&1.67\\
1&3&3&4&4&6&4&4&5&6&78&5&4&4&4&2.80696&3&6&3&0.92094&3&5&3&3&3&4&3&4&0.68&238.43&2.00\\
1&3&3&4&4&6&4&4&5&6&118&5&4&4&4&2.51974&3&6&3&1.09107&3&5&3&5&3&4&3&5&0.68&621.76&1.55\\
1&3&3&4&4&6&4&4&5&6&118&5&4&4&4&2.808&3&6&3&1.09649&3&5&3&3&3&4&3&4&0.68&862.15&2.00\\
1&3&3&4&3&6&4&4&5&6&118&5&4&4&4&3.23396&3&6&3&1.04294&3&5&3&5&3&4&3&4&0.68&615.55&1.88\\
1&3&3&4&3&6&4&4&5&6&90&5&4&4&4&2.59488&3&6&3&1.01176&3&5&3&3&3&4&3&4&0.68&348.81&2.16\\
1&3&3&4&3&6&4&4&5&6&94&5&4&4&4&2.54054&3&6&3&0.9104&3&5&3&3&3&4&3&4&0.68&226.74&2.16\\
1&3&3&4&4&6&4&4&5&6&97&5&4&4&4&2.38838&3&6&3&1.02511&3&5&3&5&3&4&3&5&0.68&343.77&1.55\\
1&3&3&4&4&6&4&4&5&6&125&5&4&4&4&2.33558&3&6&3&1.02301&3&5&3&3&3&4&3&4&0.68&540.69&2.00\\
...
\end{tabular}
}
\caption{XOMO output.}\label{fig:xout}
\end{figure*}

\begin{figure*}
\centering {\small
\begin{tabular}{c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}|c@{~}}
\begin{sideways}=execution testing and tools\end{sideways}&\begin{sideways}flex\end{sideways}&\begin{sideways}stor\end{sideways}&\begin{sideways}aexp\end{sideways}&\begin{sideways}docu\end{sideways}&\begin{sideways}site\end{sideways}&\begin{sideways}plex\end{sideways}&\begin{sideways}cplx\end{sideways}&\begin{sideways}ltex\end{sideways}&\begin{sideways}peer reviews\end{sideways}&\begin{sideways}\$ksloc\end{sideways}&\begin{sideways}rely\end{sideways}&\begin{sideways}data\end{sideways}&\begin{sideways}ruse\end{sideways}&\begin{sideways}acap\end{sideways}&\begin{sideways}?\$A\end{sideways}&\begin{sideways}sced\end{sideways}&\begin{sideways}automated analysis\end{sideways}&\begin{sideways}pcon\end{sideways}&\begin{sideways}?\$B\end{sideways}&\begin{sideways}pcap\end{sideways}&\begin{sideways}tool\end{sideways}&\begin{sideways}time\end{sideways}&\begin{sideways}prec\end{sideways}&\begin{sideways}pvol\end{sideways}&\begin{sideways}resl\end{sideways}&\begin{sideways}team\end{sideways}&\begin{sideways}pmat\end{sideways}&\begin{sideways}best or rest?\end{sideways}\\\hline
1&3&3&4&4&6&4&4&5&6&77&5&4&4&4&3.16394&3&6&3&0.900243&3&5&3&5&3&4&3&5&\_1\\
1&3&3&4&3&6&4&4&5&6&75&5&4&4&4&2.98155&3&6&3&1.09603&3&5&3&3&3&4&3&4&\_0\\
1&3&3&4&3&6&4&4&5&6&84&5&4&4&4&2.4412&3&6&3&1.01243&3&5&3&3&3&4&3&4&\_0\\
1&3&3&4&3&6&4&4&5&6&102&5&4&4&4&3.1926&3&6&3&0.971668&3&5&3&3&3&4&3&4&\_0\\
1&3&3&4&3&6&4&4&5&6&117&5&4&4&4&2.33322&3&6&3&0.934947&3&5&3&4&3&4&3&4&\_0\\
1&3&3&4&4&6&4&4&5&6&115&5&4&4&4&2.95164&3&6&3&0.990819&3&5&3&5&3&4&3&5&\_1\\
1&3&3&4&3&6&4&4&5&6&122&5&4&4&4&2.51071&3&6&3&0.957017&3&5&3&5&3&4&3&5&\_1\\
1&3&3&4&4&6&4&4&5&6&117&5&4&4&4&3.01523&3&6&3&0.961085&3&5&3&4&3&4&3&4&\_0\\
1&3&3&4&3&6&4&4&5&6&112&5&4&4&4&2.48963&3&6&3&1.07945&3&5&3&4&3&4&3&4&\_0\\
1&3&3&4&3&6&4&4&5&6&117&5&4&4&4&2.77276&3&6&3&1.00916&3&5&3&4&3&4&3&5&\_0\\
...
\end{tabular}
}
\caption[BORE output]{BORE output. In the last column, \_0 denotes ``rest'' and \_1 denotes ``best''.}\label{fig:bore}
\end{figure*}

\section{Running the System}

To generate our results, XOMO was run as follows:

{\scriptsize
\begin{verbatim}
xomo         \
  -S $RANDOM  \
  -l           \
  -R 1000       \
  -d system.dat  \
  -p ares.dat     \
  -P are           \
  -s '<$defects,<$effort,<$threats,?$A,?$B,$ksloc' > xomo.out 
\end{verbatim}
}
The command can be read as follows:

{\tt -S \$RANDOM}: use a random seed for the simulations.

{\tt -R 1000}: perform 1000 Monte Carlo Simulations

{\tt -l}: minor detail- print a header on the output;

{\tt -d \$system.dat}: load default ranges from \fig{system};

{\tt -p \$ares.dat}: load the ARES ranges \fig{ares};

{\tt -P ares}: focus the simulation on ARES;

{\tt -s '{<}\$defects....'}: 
a string specifying the goal statement; i.e. minimize defect, effort, and treats.

Prior to calling TAR3, the results on the above command (stored in the $tmp$ file)
must be divided into ``best'' and ``rest''

{\scriptsize
\begin{verbatim}
bore N=0.33 Pass=1 xomo.out \
            Pass=2 xomo.out  \
            Pass=3 xomo.out > xomo.data
\end{verbatim}
}
For example, \fig{xout} shows a typical $xomo.out$ file and \fig{bore} shows the result
of BORE
replacing the last three columns with ``best'' or ``rest''.

Once the data has been BOREd, it is passed to TAR3 as follows:

{\footnotesize
\begin{verbatim}
tar3 xomo
\end{verbatim}
}
\noindent
With the above call, TAR3 expects to find three files: 
\be
\item $xomo.data$ : generated by BORE and shown in \fig{bore};
\item $xomo.names$ : a data dictionary for the data file, shown in \fig{names};
\item $xomo.cfg$ : some control settings, shown in \fig{cfg}.
\ee
$Xomo.names$ is almost self-explanatory. One non-obvious feature is on line one where the
order of the classes tells TAR3 what to seek and what to avoid. In a TAR3 $names$ file, the classes
are weighted left to right 2,4,8,16,etc. Hence, in \fig{names}, \_1 (i.e. ``best'')
is the preferred class.

\begin{figure}
{\scriptsize
\begin{alltt}
\_0,\_1

execution_testing_and_tools :  1,2,3,4,5,6.
flex 						:  1,2,3,4,5,6.
stor 						:  1,2,3,4,5,6.
aexp 						:  1,2,3,4,5,6.
docu 						:  1,2,3,4,5,6.
site 						:  1,2,3,4,5,6.
plex 						:  1,2,3,4,5,6.
cplx 						:  1,2,3,4,5,6.
ltex 						:  1,2,3,4,5,6.
peer_reviews 				:  1,2,3,4,5,6.
ksloc 						:  continuous.
rely 						:  1,2,3,4,5,6.
data 						:  1,2,3,4,5,6.
ruse 						:  1,2,3,4,5,6.
acap 						:  1,2,3,4,5,6.
A 						    :  continuous.
sced 						:  1,2,3,4,5,6.
automated_analysis 			:  1,2,3,4,5,6.
pcon 						:  1,2,3,4,5,6.
B 							:  continuous .
pcap 						:  1,2,3,4,5,6.
tool 						:  1,2,3,4,5,6.
time 						:  1,2,3,4,5,6.
prec 						:  1,2,3,4,5,6.
pvol 						:  1,2,3,4,5,6.
resl 						:  1,2,3,4,5,6.
team 						:  1,2,3,4,5,6.
pmat 						:  1,2,3,4,5,6.
\end{alltt}
}
\caption{xomo.names.}\label{fig:names}
\end{figure}
\begin{figure}
{\scriptsize
\begin{verbatim}
granularity:   2
maxNumber:    10
maxSize :     10
randomTrials: 100
futileTrials:  5
bestClass:    20\%
\end{verbatim}
}
\caption{xomo.cfg}\label{fig:cfg}
\end{figure}


\begin{figure*}[!b]
\begin{center}
\begin{minipage}{.25\linewidth}
{\bf round1:}\newline sced = 3 

~\\

{\bf round2:}\newline		execution\_testing\_and\_tools = 6

~\\


{\bf round3:}\newline		peer\_reviews = 6

~\\


{\bf round4:}\newline	    automated\_analysis = 6

~\\


{\bf round5:}\newline		ltex = 5
\end{minipage}
 \begin{tabular}{|c|c|c|}\cline{1-3}
\includegraphics[width=1.5in]{study1/defectsMeanSd.pdf} &
\includegraphics[width=1.5in]{study1/effortMeanSd.pdf} &
\includegraphics[width=1.5in]{study1/threatsMeanSd.pdf} \\
\includegraphics[width=1.5in]{study1/defectsMinMeanMax.pdf} &
\includegraphics[width=1.5in]{study1/effortMinMeanMax.pdf} &
\includegraphics[width=1.5in]{study1/threatsMinMeanMax.pdf} \\\cline{1-3}
\end{tabular}
\end{center}
\caption[Results: case study 1]{Results. In the top row, the green error bars denote $\pm$ one standard deviation
about the mean. }\label{fig:results}
\end{figure*}



$Xomo.cfg$ requires some explanation:
\bi
\item
$Granularity$ controls how the continuous ranges are divided into bins.
For reporting purposes, an odd number for $Granularity$ is best since (e.g.) 
a $Granularity$ of 5 can be reported as ``3 means no change, 4 and 2 means some changes up and down, and 5 and 1 mean larger changes up and down''.
$Granularities$ over 7 are rarely useful and for problematic data sets (like the XOMO data), this number can go as low as 2. 
\item
TAR3 only reports the top $MaxNumber$ of treatments (in this case, 10).
\item
When composing a treatment, TAR3 will build constraints of up to $MaxSize$ items. 
The upper bound on this number is the number of attributes but, in practice, a MaxSize of 5-10
often suffices. 
\item
$RandomTrils$ sets the number of trial we perform before we
$pause$ to look for new best treatments. 
\item
$FutileTrails$ is the number of allowed pauses (so the total number of trials is 100*5).
\item
TAR3 rejects any treatment that contains less than $BestClass$ percentage of the best class.
\ei
The settings
in \fig{cfg} could be improved but, as we shall see, they suffice for the XOMO data.



\section{Case Studies}
\subsection{Case Study 1}

\fig{results} shows the effects of applying \fig{cycle} to the GNC model
of ARES1 modeling in \fig{ares}.
After each round, TAR3 learns treatments and the best treatment is
selected to restrain the Monte Carlo simulation of the next round.
At each round, the ``best/rest'' discretization described above was repeated so ``best'' was
the top 33\% seen in each round:

The first round decision was $sced=3$; i.e. set the schedule pressure
to the mid-point of its ranges \{2,3,4\} (as shown on the last line
of \fig{ares}.  The second round decision was to apply maximum
effort to execution-based testing.  As shown right-hand-side plots
of \fig{cycle}, these failed to reduce mean defects or effort. Round
1 and 2 did achieve a small reduction in the threat index. However,
recall that the THREAT model classifies threat indexes less than 6
as ``low threat''.  Hence, the round 1\&2 reduction of THREAT from
2 to 0.5 is not particularly interesting.

At round 3, TAR3 proposed maximizing the effort on peer reviews
and this had dramatic impact on the defects: the mean value halved
and the maximum value dropped 75\%.

At round 4, TAR4 proposed maximizing the effort on automated analysis.
This round halved again the mean estimated number of defects (from
two to one) and reduced the maximum value by a further 80\%.

At round 5, the best treatment TAR3 found was to increase the
language and tools experience; i.e. $ltex=5$. Since the improvements seen
from round 4 to round 5 are negligible, cyclic learning was terminated.

One  prior XOMO-based analysis of the trade space within
the  space shuttle sub-system offered far more dramatic
results than \fig{results}~\cite{me05e}. That prior study found ways to:

\bi
\item 
Reduce the residual defects per KSLOC 
by 85\%; 
\item
Halve the threat of 
schedule over-run; 
\item
Decrease  the  development effort to nearly half of its original value. 
\ei

Why did that study produce more impressive results than this one? Our view is that the issue 
is in the problem domain and not in the analysis method. In this study, 24 attributes were set
and only a handful could be varied. In the former study, 
our data miner was given far more opportunity to improve the project.

\subsubsection{Discussion: Case Study 1}

\begin{figure}
\begin{center}
{\footnotesize
\begin{tabular}{r|rrrrr|rrrr}
\multicolumn{6}{c}{}&
     \multicolumn{4}{|c}{defects}\\\cline{7-10}
round&\begin{sideways}sced\end{sideways}&\begin{sideways}peer reviews\end{sideways}&\begin{sideways}ltex\end{sideways}&\begin{sideways}execution testing and tools\end{sideways}& \begin{sideways}automated analysis\end{sideways}&min& mean& max &sd\\\hline
6a   & 3  &         6  &   5&    1                         & 1                  &3.0&  3.6&  4.25& 3.6\\
6b   & 3  &         6  &   5&    1                         & 2                  &2.9&  3.5&  4.09& 3.5\\
6c   & 3  &         6  &   5&    1                         & 3                  &2.6&  3.1&  3.6& 3.1\\
6d   & 3  &         6  &   5&    1                         & 4                  &2.2&  2.6& 3.0 & 2.6\\
6e   & 3  &         6  &   5&    1                         & 5                  &1.7&  2.1&  2.42& 2\\
6f   & 3  &         6  &   5&    1                         & 6                  &1.6&  1.8&  2.2&1.8\\\hline
6    & 3  &         6  &   5&    6                         & 6                  &0.42&  0.5&  0.59& 0.5

\end{tabular}
}
\end{center}
\caption{Effects of increasing automated analysis.}\label{fig:aa}
\end{figure}

The results of \fig{results} shows interesting subtleties:

\bi
\item
Despite certain uncertainties in the domain (e.g. the exact value
of the calibration parameters $A$, $B$), note that the effort and threat mins/max/mean
variables are quite stable. 
That is, 
the  ARES system displays much stability in
its process properties.    
\item
The merits of execution-based testing, by itself, seem very small
for this software. Note that the addition of execution-based testing (in round 2)
barely changed the defect rate.
\item
On the other hand, peer reviews seem most powerful for this software. Note that
the greatest defect rate reduction occurred when peer reviews were added in round 3.
\ei

The ability to make fine-grained trade-offs between alternate
technologies is one of the main advantages of the XOMO technology.
For example, \fig{aa} and \fig{plotted} shows  an XOMO study with ARES
where:
\bi
\item
 peer
review usage was maximized;
\item
 execution-based testing was minimized;
\item
and automatic analysis was increased from very low to very high
levels.  
\ei
This represents a very common situation at NASA- given
limited access to hardware test beds, IV\&V analysts perform very
limited execution-based testing. In such a situation, peer reviews
are widely used (since they are relatively cheap to perform). 
In this context,
it might be necessary to make
a business case to buy some automatic analysis tool.
\fig{aa} and \fig{plotted} show  the effects of only changing the level of automatic analysis while
holding other defect removal methods steady.
Automatic
analysis can be cheap to perform (e.g. static code parsers) and
sometimes can be conducted very early in the life cycle (e.g.
lightweight formal models of requirements). 

\begin{figure}
\begin{center}
\includegraphics[width=2.5in]{whatif.pdf}
\end{center}
\caption[\protect\fig{aa}, plotted.]{\protect\fig{aa}, plotted. Green error bars denote $\pm$ one standard
deviation about the mean.}\label{fig:plotted}
\end{figure}

\fig{plotted} focuses on how the level of automatic analysis effects defect estimates. Note that automatic analysis by itself does not get rid of all the defects.  But, in the absence of execution-based testing tools,
it can half the mean predicted number of defects.


\subsection{Case Study 2}

\begin{figure}[!b]
{\scriptsize\begin{alltt}
     1	system with kc1
     2		    A = 2.94
     3	        B = 0.91
     4			acap just 2 3
     5			aexp just 2 3
     6			cplx  just 5 6
     7			data = 3
     8			docu just 2 4
     9			ksloc just  75 125
    10			ltex just 2 4 
    11			pcap = 3
    12			pcon just 2 3
    13			plex = 3
    14			pmat just 1 4 
    15			prec just 1 2
    16			pvol = 2
    17			rely = 5
    18			resl just 1 3 
    19			ruse  just 2 4
    20			sced just 1 3 
    21			site = 3
    22			stor just 3 5
    23			team just 2 3
    24			tool just 2 3
    25		    flex just 2 5
\end{alltt}}
\caption{The ranges of study 2.}
\label{fig:study2}
\end{figure}

\begin{figure*}
\begin{center}
\begin{minipage}{.25\linewidth}
{\bf round1:}\newline time just 3 4 and
pmat just 3 4
~\\

{\bf round2:}\newline	acap=3 and
sced=3
~\\


{\bf round3:}\newline		flex just 4 5 and
ltex = 4 \newline
~\\


{\bf round4:}\newline	    ksloc just 75 91 and
pmat = 4 and \newline
tool = 3 and
time just 3 4\newline
acap =3
~\\


{\bf round5:}\newline		stor just 3 4 and
prec = 2 and \newline
aexp = 3
\end{minipage}
 \begin{tabular}{|c|c|c|}\cline{1-3}
\includegraphics[width=1.5in]{study2/dataMMM.pdf} &
\includegraphics[width=1.5in]{study2/effortMMM.pdf} &
\includegraphics[width=1.5in]{study2/threatsMMM.pdf} \\
\includegraphics[width=1.5in]{study2/dataMSD.pdf} &
\includegraphics[width=1.5in]{study2/effortMSD.pdf} &
\includegraphics[width=1.5in]{study2/threatsMSD.pdf} \\
\cline{1-3}
\end{tabular}
\end{center}
\caption{Results of case study 2. 
}\label{fig:results2}
\end{figure*}

This second study explores a less constrained problem than the
first study.
\fig{study2} shows the input ranges for software from
``KC1''-  an experimental NASA space plane.
KC1 was the seed of the design for the Orbital Space
Plane which, in turn, was used for the initial design work for ARES.
Note that the slots can vary over a wider range of values.

\fig{results2} shows the effects of iterative treatment learning with TAR3
on KC1. 
In the first round of learning, changing the runtime pressure ($time$)
and process maturity ($pmat$)
results in moderate reductions in effort/defects/ and threats. Much larger
reductions were achieved by round 3 after constraining analyst capability, 
 ($acap$)
schedule pressure ($sced$), 
development flexibility 
($flex$)
and language and tool experience ($ltex$).

Further rounds offered some further improvements in defects/effort/ and threats
but required much more constraints. By round 3, management action would be required to
make 6 process changes. Rounds 4 and 5 required 8 more. Hence, in terms of cost/benefit,
there might be case to pause after round 3.

\subsubsection{Discussion: Case Study 2}
The most important feature of \fig{results2} is what is {\em missing}\footnote{
``Is there any point to which you would wish to draw my attention?'' \newline
``To the curious incident of the dog in the night time.''\newline
``The dog did nothing in the night time.'' \newline   ``That was the curious incident,''
remarked Sherlock Holmes. \newline
--Sir Arthur Conan Doyle. {\em Silver Blaze}}. Note that  all the recommendations
proposed by the treatment learner were {\em process}-based, not {\em product}-based.
Missing in the recommendations are any mention of (e.g.) defect removal tools such
as the automated analysis or the execution/testing tools discussed in the last case study.

One explanation for these missing features can be found in the different
{\em magnitude} of the
problems facing KC1 or ARES. The y-axis of the \fig{results2} graphs are much larger than the
\fig{results} plots; for example, the largest predicted
number of defects faced by ARES  and KC1
are 8/KSLOC and 300/KSLOC respectively.

One lesson from these two case studies is that process issues trump product issues.
Debating which formal method to apply is irrelevant if your analysts are all low quality.
When faced with large defect problems (e.g. KC1's 300/KSLOC), it may well be more important
to make process changes rather than hope that applying some tool will solve your project's problems.


\section{Conclusion}

The trade space for NASA systems can be very large. 
Software models like COCOMO, COQUALMO, and THREAT contain many
assumptions about their domain. The conclusions gained from this
models should be scrutinized  by domain experts. Early in the life cycle
of a software project, such scrutiny is complicated by all the
unknowns associated with a project.  Exploring all those unknowns can lead
to massive data overload as domain experts are buried beneath a mountain of data
coming from their simulators.

Here, we have discussed
tools that automate exploring that space. These tools integrate to other NASA-funded
tools via slot trees.
XOMO, BORE, and treatment learners like TAR3 can assist in
trade space exploration.  These tools can find automatically find software
process decisions that reduce defects and effort and risk of schedule
over run. These tools sample the space of options and report sample
conclusions within the space of possibilities.

To demonstrate that technique, this paper conducted case studies 
with software development for NASA systems.
The case studies proved to be very different: one required product patches (ARES)
while the other required massive changes to process structure (KC1).

While the particular case studies examined here is
quite specific, the analysis method is quite general.
There is nothing stopping an analyst from using XOMO
to study other kinds of software development. The only requirement is that
the essential
features of that software can be mapped onto COCOMO-like models.
\begin{itemize}
\item
All the models used here contain most of their knowledge
in easy-to-modify tables representing the particulars of different domains. 
\item
All the tools used here are portable and use simple command-line
switches that allow an analyst to quickly run through a similar study
for a different kind of project.
\end{itemize}


\bibliographystyle{abbrv}
\bibliography{refs}
\end{document}

 to implement The precise numeric values for the factors and multiplers
are shown in 

\begin{figure}
\begin{ccode}
function Effort() \{
  return A()* Ksloc()^E() * Rely()* Data() * 
    Cplx()* Ruse()* Docu()* Time()* Stor() * 
    Pvol()* Acap()* Pcap()* Pcon()* Aexp() *
    Plex()* Ltex()* Tool()* Site()* Sced() \}

function E() \{ 
  return B() + 0.01*(Prec() + Flex() 
             + Resl() + Team() + Pmat()) \}
\end{ccode}
\caption{Effort model.}\label{fig:em}
\end{figure}

When XOMO runs 

\begin{scriptsize}\begin{verbatim}
  while(n--) {
    reset();
    months=Effort(); 
    risk=Total_risk();
    defects=Total_defects()/(Ksloc());
    for(j in Cache)
      printf("%.2f,",Cache[j]);
    printf("%.0f,%.0f,%.2f",months,risk,defects); print ""
  }
\\section{Case Study}

XOMO picks model inputs using  \fig{xomoranges}, plus any additional
restraints supplied on the command line.
The command line 
can 
set exact values (using the ``{\tt
-=}'' flag) or can define a  range from some lower to upper value (using
the ``{\tt -l}'' and ``{\tt -u}'' flags).

Using that syntax, we define the inputs to our case study as follows:
\begin{description}
\item[{\tt -p ksloc -l 75 -u 125}]: We assume that 
some  developer
from a prior autonomy project has guess-timate that this new project
will require 75,000 to 125,000 lines of code.
\item[{\tt -p rely -=5}]: At NASA, everything must be have highest reliability.
In COCOMO, {\tt rely}'s maximum value is very high; i.e. 5.
\item[{\tt -p prec -= 1}]: 
Since
this team has never done this sort of thing before, the precedence (or {\tt prec}) is
set to the lowest value.
\item[{\tt -p acap -= 5}]: This team is skillful; i.e. has highest analyst capability.
\item[{\tt -p aexp -= 1}]: Their experience in this kind of software is non-existence.
\item[{\tt -p cplx -= 6}]: The software is very complex.
\item[{\tt -p ltex -= 1}]: The team has no experience with the languages and tools used for
autonomous systems.
\item[{\tt -p ruse -= 6}]: This team, in their enthusiasm, believe that
the tools they are building here will be reused by many developers
in the future.
\end{description}

\begin{figure}
{\scriptsize\begin{alltt}
runxomo() \{ 
  Scenario="-p ksloc -l 75 -u 125 
            -p rely -= 5           
            -p prec -= 1            
            -p acap -= 5             
            -p aexp -= 1              
            -p cplx -= 6               
            -p ltex -= 1                
            -p ruse -= 6"
  xomo \$Scenario \}         
\end{alltt}}
\caption{XOMO: specifying restraints.}\label{fig:xomoins}
\end{figure}

\fig{xomoins} summarizes the {\tt XOMO} command line used in this  study.
Each run of \fig{xomoins} generates one line of \fig{xomoouts}.
Between
each run, the $Cache$ is cleared so that the next run is free to select another set of inputs.
Observe in \fig{xomoouts} how the selected values satisfy {\em both} the ranges of \fig{xomoranges}
and the restraints of \fig{xomoins}. For example, the {\tt rely} values are all 5 (since the
command line included \mbox{{\tt -p rely -= 5}}) while the other values can range more widely.
  e 
\section{Case Studies}

\subsection{Case Stduy \#1}

Ideally, software effort-estimation models like COCOMO-II should
be tuned to
their  local domain. Off-the-shelf ``untuned'' models have been up
to 600\% inaccurate in their estimates, e.g. \cite[p165]{mukho92}
and~\cite{kemerer87}. However, tuned models can be far more
accurate.
For example,~\cite{chulani99}
reports a study with a Bayesian tuning algorithm using the COCOMO
project database. After Bayesian tuning, a cross-validation study
showed that
 COCOMO-II model produced estimates that are within 30\% of the
 actuals,
 69\% of the time.

Elsewhere, with Boehm, Chen, Port, Hihn, and Stukes~\cite{chen05,me04h,me05a,me05c} we have explored
calibration methods for COCOMO. Here, we take a new approach and ask ``what conclusions hold across
the space of possible tunings''?. Hence we treat the tuning parameters 
``{\tt a}'' and ``{\tt b}'' as random variables (see \fig{xomoranges}, last two lines).



\begin{figure*}
\begin{center}
\begin{minipage}{.25\linewidth}
\vspace{0pt}
\includegraphics[width=1.65in]{constraints.pdf}
\end{minipage}\hfill\begin{minipage}{.7\linewidth}
\vspace{0pt}
{\footnotesize\begin{tabular}{p{2.5cm}|p{1.9cm}|p{1.5cm}|p{2.2cm}|p{1.5cm}}
&\multicolumn{4}{c}{learned restraints}\\\cline{2-5}
baseline & 1000 & 2000 & 3000 & 4000\\\hline
75 $\le$ ksloc $\le$ 125\newline
rely = 5\newline
prec = 1\newline 
acap = 5\newline 
aexp = 1\newline
cplx = 6\newline               
ltex = 1\newline                
ruse = 6
&
sced=4\newline
peer\_reviews=5
&
pmat=5\newline
pcap=4
&
tool=4\newline
execution\_testing-
\_and\_tools=5
&
team=5\newline
resl=5\newline
automated-
analysis=5
\end{tabular}}
\end{minipage}
\end{center}
\caption{TAR3: learned restraints}\label{fig:restraints}
\end{figure*}
\begin{figure*}
\begin{center}
\includegraphics[width=1.65in]{effort.pdf}
\includegraphics[width=1.65in]{defectDens.pdf}
\includegraphics[width=1.65in]{scedRisk.pdf}
\end{center}
\caption{TAR3: impact of \protect\fig{restraints}'s restraints.}\label{fig:results}
\end{figure*}

\section{Results}

For this study
the initial baseline restraints were set according to our autonomy settings; 
i.e. \fig{xomoins}.
 XOMO was run 1000 times each iteration and
BORE returned the 100 $best$ examples and a random sample of 100 of the $rest$.
These $best$ and $rest$ examples were passed to TAR3 and the
best learned treatment was imposed as restraints on 
subsequent
iterations.

\fig{restraints} shows the restraints learned by four
iterations of iterative treatment learning.
\fig{results} shows the effects of these restraints on the output of the {\tt XOMO} models:
\be
\item
The mean development effort was nearly halved: 3257  to 1780 months;
\item
The mean THREAT halved:  77 to 36; 
\item
 The mean defects densities
were reduced by 85\% from 0.97 to 0.15.
\item
The variance on the above measures was significantly reduced: the COQUALMO
and THREAT standard deviations nearly reached zero.
\ee

Several of the \fig{results} curves flatten out after 2000 runs of
{\tt XOMO}. A parsimonious management strategy could be
formed from just the results of the first two rounds of learning.
Interestingly, in those first two rounds, process changes were more
important than the application of technology.  Technology-based
techniques such as {\tt tool} support or {\tt execution testing and
tools} did not arise till iteration three.  On the other hand,
the first two iterations
labeled ``1000,2000'' in \fig{restraints} want to decrease schedule
pressure ({\tt sced}), increase process maturity ({\tt pmat}), and
programmer capability ({\tt pcap}) and requested the user of peer
reviews.  

Another interesting feature of the results is that many of
the inputs were never restrained. The left-hand-side plot of
\fig{restraints} shows that even after four rounds of learning, only
17 of the 28 inputs were restrained.  That is, management commitments
to 11 of the 28 inputs would have been a waste of time. Further, if management
is content with the improvements gained from the first two iterations, then 
only 12 restraints are required and decisions about the remaining 16 inputs would
have been superfluous.

nd{verbatim}\end{scriptsize}
