183,266d182
< 
< \begin{figure*}[!t]
< {\footnotesize
< \begin{tabular}{|p{3in}|p{3in}|}\hline
< \begin{description}
< \item[{\bf Satellite}] {\em
< Description:} Analysts at the NASA Jet Propulsion
< Laboratory have built a semantic network describing the
< interactions between 88 features of a proposed dead space
< mission. Each edge was annotated with the numeric cost and
< benefits of taking some action. Some of these nodes
< represented base decisions within the project (e.g.
< selection of  a particular type of power supply). The net
< can be  executed using some random combination of the base
< decisions and the resulting costs and benefits  collected.
< 
< {\em Assessment:}
< The generated cost and benefit scores can be sorted
< and divided into $X$ percentile bands.
< An individual run can then be scored as a pair
< $<Cost_i,Benefit_j>$ where $i$ and $j$ select one of the
< percentile bands. In this application, the preferred band
< would be lowest cost and highest benefit.
< 
< {\em References:} The semantic net editor for this
< application was developed  by Feather et.al.~\cite{fea00}.
< No prior report explores machine learning in this domain.
< \item[{\bf CMM2:}]
< {\em Description:} The expressiveness of the semantic net
< processed  by  Feather et.al.~\cite{fea00} was extended by
< Menzies \& Kiper to create a more general rule-based
< language~\cite{me01e}. This language was used to encode
<  a model of best practices in software engineering
< (level 2 of the Software Engineering Institute's capability
< maturity model, also known as CMM2~\cite{paulk93}).
< 
< {\em Assessment:} Menzies & Kiper doubted the accuracy of
< the cost and benefit supplied by domain experts. Hence,
< they wrote a simulator that randomly perturbed these
< weights by $\pm50\%$. The goal of the l Same as with {\em
< satellite}.
< \end{description}
< &
< \begin{description}
< \item[{\bf COCOMO:}] {\em
< Description:} The goal of the COCOMO-II project is to build
< an open-source software cost estimation
< model~\cite{cocomoII}. Internally, the model contains a
< matrix of  parameters that should be tuned to a particular
< software organization. Using COCOMO-II, the Madachy risk
< model can assess the risk of a software cost
< over-run~\cite{madachy97}.
< 
< {\em Assessment:} For machine learning purposes, the goal
< of using the Madachy model is to find a change to a
< description of a software project that reduces the
< likelihood of a poor risk software
< project~\cite{me00e,me01f}.
< 
< \item[{\bf  Circuit:}] A qualitative description of a circuit
< of 47 wires connecting 9 light bulbs and 16 other
< components was coded in Prolog. The model was expressed as
< a set of constraints; e.g. the {\tt sum} of the voltages of
< components in series is the {\tt sum} of the voltage drop
< across each component. The definition of {\tt sum} honored
< the nondeterminism of qualitative arithmetic; e.g.  {\tt
< sum(-,+,Any)} notes that the sum of a negative and a
< positive value is unknown.
< 
< {\em Assessment:} The goal of the circuit was to illuminate
< a space using the 9 light bulbs. The problem with the
< circuit was out-of-control nondeterminism. On backtracking,
< this seemingly simple circuit generated 35,228 solutions to
< the constraints. Worse, over all those solutions, the
< circuit was mostly dark: only two bulbs glowing (on
< average). The goal of the machine learning was to find a
< minimal set of changes to the circuit to increase the
< illumination~\cite{me01g}.
< \end{description}
< \\\hline
< \end{tabular}
< } \caption{Some of the models used in  TAR2 validation
< studies.}
< \end{figure*}
273,279c189,190
< terms are defined below). The output of the learner can be assessed
< via a standard N-way cross-validation  study.
< However,  a more convincing demonstration is to
< imposing the proposed treatment (either controller or monitor) as a
< constraint on a model,  then  check if the model's future behavior changes
< in the manner predicted by TAR2. This demonstration has worked
< on numerous models, including the ones described in \fig{models}.
---
> terms are defined below). Validation studies
> suggest that the algorithm is effective and is general to many domains.
282,284c193,195
< it's repeated success suggests that~(i) many
< example sets have a very simple structure; so (ii)~very simple
< learners will be  adequate to  probe many domains.
---
> it's repeated success suggests that many
> example sets have a very simple structure. Such simple structures
> can be adequately probed with very simple learners.
720,721d630
< For example, in the {\em COCOMO2} application described in \fig{models},
< XXX
722a632,634
> Menzies and Kiper studied a model of best practices in software engineering
> (level 2 of the Software Engineering Institute's capability
> maturity model, als known as CMM2~\cite{paulk93}).
757,760c669,682
<  generated successful control rules.
< 
< 
<  The frequency of these criteria are shown as the
---
>  generated successful control rules. In the first study,
> analysts at the NASA Jet Propulsion Laboratory built a
> semantic network describing the interactions between 88
> features of a proposed dead space mission. Each edge was
> annotated with the numeric cost and benefits of taking some
> action. Some of these nodes represented base decisions
> within the project (e.g. selection of  a particular type of
> power supply). 42,000 times, the net was executed using
> some random combination of the base decisions. The
> resulting costs and benefits were collected, sorted, and
> divided into three 33.3\% percentile bands. This resulted
> in nine criteria scores: hH,hM,hL,  mH,mM,mL, lH,lM,lL
> where h,m,l=high,medium,low costs and H,M,L=high,medium,low
> benefits. The frequency of these criteria are shown as the
805,807c727,745
< circuit containing
< 
< 
---
> circuit containing 47 wires connecting 9 light bulbs and 16 other components.
> The circuit was coded in Prolog and expressed
> as a set of constraints; e.g.
> the sum of the voltages of components in series is the sum
> of the voltage drop across each component.
> The {\tt bulb} relation of \fig{bulb4} describes our qualitative knowledge of
> bulbs in the circuit. For example, {\tt bulb(blown,dark,Any,0)} says that
> a blown bulb is dark, has zero current across it, and can have any
> voltage at all.   {\tt Switch}
> describes our qualitative knowledge of
> electrical switches. For example {\tt switch(on,0,Any)} says
> that if a switch is on, there is zero voltage drop across it while
> any current can flow throw it. {\tt Sum} describes our knowledge of qualitative
> arithmetic, some of which is nondeterminate. For example, {\tt sum(-,+,Any)}
> notes that the sum of a negative and a positive value is unknown. The nondeterminacy
> of {\tt sum} means that the behavior of the whole circuit can vary wildly. On backtracking,
> this circuit generated 35,228 runs. Each run was scored according to how many bulbs glowed
> {\tt light}ly and, on average, only two bulbs glowed in each run (see the {\em raw}
> distribution of \fig{run2}.
837c775
<  {\em \fig{nudges}.i: housing}\newline {\tiny
---
>  {\em \fig{dels}.i: housing}\newline {\tiny
860c798
<  {\em \fig{nudges}.ii: page-blocks}\newline {\tiny
---
>  {\em \fig{dels}.ii: page-blocks}\newline {\tiny
880c818
<   {\em \fig{nudges}.iii: wine}\newline {\tiny
---
>   {\em \fig{dels}.iii: wine}\newline {\tiny
904c842
< {\em \fig{nudges}.iv: car}\newline {\tiny
---
> {\em \fig{dels}.iv: car}\newline {\tiny
913c851
<  {\em \fig{nudges}.v: satellite}\newline {\tiny
---
>  {\em \fig{dels}.v: satellite requirements}\newline {\tiny
931c869
<    {\em \fig{nudges}.vi: QR circuit}\newline{\tiny
---
>    {\em \fig{dels}.vi: QR circuit}\newline{\tiny
945c883
<   {\em \fig{nudges}.vii: cmm2}\newline {\tiny
---
>   {\em \fig{dels}.vii: cmm2}\newline {\tiny
963c901
<  {\em \fig{nudges}.viii: cocomo2}\newline  {\tiny
---
>  {\em \fig{dels}.viii: cocomo2}\newline  {\tiny
1058,1060c996,998
< cocomo     &30000  & 0          & 23       & 4       & 1     & 2\\\hline
< satellite
<  &30000  & 0          & 99       & 9       & 5     &86\\\hline
---
> cocomoII     &30000  & 0          & 23       & 4       & 1     & 2\\\hline
> satellite\newline
> requirements &30000  & 0          & 99       & 9       & 5     &86\\\hline