\documentclass[twocolumn,global]{svjour}
\usepackage{lastpage,url,alltt,graphicx,cite,url,fancyheadings,bar,fancyvrb}
\usepackage{rotating,wrapfig,stfloats}

\usepackage{eso-pic}
\usepackage{color}
\usepackage{type1cm}

%\makeatletter
%  \AddToShipoutPicture{%
%    \setlength{\@tempdimb}{.5\paperwidth}%
%    \setlength{\@tempdimc}{.5\paperheight}%
%    \setlength{\unitlength}{1pt}%
%      \makebox(600,900){\rotatebox{45}{\textcolor[gray]{0.80}{\fontsize{9cm}{9cm}\selectfont{Draft}}}}}
%\makeatother
%%    \put(\strip at pt\@tempdimb,\strip at pt\@tempdimc){%
 %     \makebox(500,700){\rotatebox{45}{\textcolor[gray]{0.75}{\fontsize{10cm}{10cm}\selectfont{Draft}}}}

\newenvironment{ccode}{\scriptsize\begin{alltt}}{\end{alltt}\normalsize}


   \newcommand{\code}[3]{%
     \begin{figure}[!t]%
        \fvset{  samepage=true,%
                 fontsize=\scriptsize,%
                 commandchars=\\\{\},%
                 framesep=2mm,%
                 label={\small{\textit{#1}}},%
                 labelposition=topline,%
                 %numbersep=1pt,%
                 %stepnumber=5%
                 %numbers=left,%
                 frame=single%
                 }%
            {\bf \VerbatimInput{#1}}
    \caption[#2]{#3}\label{fig:#1}%
    \end{figure}%
}

%\usepackage[para,symbol*]{footmisc}
%\setfnsymbol{lamport}
\usepackage{amssymb,times}
\DeclareFontShape{OT1}{cmtt}{bx}{n}{
  <5><6><7><8><9><10><10.95><12><14.4><17.28><20.74><24.88>cmttb9}{}

\newcommand{\mytitle}{
Improving IV\&V Techniques Through the Analysis of Project Anomalies:\\
Text Mining PITS issue reports - preliminary report
}
\footrulewidth 0.0pt\headrulewidth 0.1pt
\lhead{}\rhead{}\chead{}\rfoot{}\lfoot{}\cfoot{}
\rhead{ {\scriptsize 
\date{\today}\newline
source: \url{http://now.unbox.org/all/trunk/doc/06/tellingmore/report2.*}\newline
download: \url{http://menzies.us/pdf/06anomalies-pits0.pdf}
}}

\rhead{{\scriptsize 
\date{\today}\\
download: \url{http://menzies.us/pdf/06anomalies-pits0.pdf}\\
wp: \url{http://now.unbox.org/all/trunk/doc/06/tellingmore/report2.*}
}}
\lhead{\includegraphics[width=6cm]{wvulogo.pdf}}

\newenvironment{smallitem}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{itemize}
   \setlength{\leftmargin}{.2in}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{itemize}}

\newenvironment{smallenum}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{enumerate}
  \setlength{\leftmargin}{.2in}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{enumerate}}


 \newcommand{\bd}{\begin{description}}
\newcommand{\ed}{\end{description}}
 \newcommand{\bi}{\begin{smallitem}}
\newcommand{\ei}{\end{smallitem}}
 \newcommand{\be}{\begin{smallenum}}
\newcommand{\ee}{\end{smallenum}}
\newcommand{\fig}[1]{Figure~\ref{fig:#1}}
\newcommand{\eq}[1]{Equation~\ref{eq:#1}}
\newcommand{\tion}[1]{\S\ref{sec:#1}}

%\topmargin -3cm

\begin{document}
\title{
\mytitle
\author{Tim Menzies }
\institute{Lane Department of Computer Science and Electrical Engineering, West Virginia
University,
USA
\newline
\url{tim@menzies.us}\newline
\url{http://menzies.us}\newline
}}
\maketitle \thispagestyle{fancy}
\pagestyle{fancy}

\vspace{-1cm}\begin{abstract}
This project is in two parts.
The {\em second part} will try to combine two (or more) of the IV\&V data sources
 into an active monitoring
framework where data collected during an active IV\&V project will trigger an
alert
if a project becomes unusual'' (and defining ``unusual'' is one of the goals of this project).

But before we can generalize between sources, we need to study each source in isolation
to determine its strengths and weaknesses.
Hence, the {\em first part} of this project  aims to gain experience with the various IV\&V data sources
available to researchers like myself; i.e.
		 \bi
	 \item
	 SILAP, from
	 the IV\&V planning and scoping team; 
	 \item
	 James Dabney's Bayes networks that
	 describe the IV\&V business practices of the L3
 IV\&V contractor; 
	 \item
	 The PITS issue tracking data; 
	 \item
	 The LINKER database
	 project that intends to
	 join PITS to other data sources; 
	 \item
	 Balanced score card strategy maps from NASA Langley. 
	 \item
	 and the COCOMO data sets from JPL.
	 \ei

This is the first year of a  three year project that started in  June 2006.
The project is data-rich project and much progress has already been achieved.
\bi
\item
At SAS'06, a preliminary report described what had been learned from the
SILAP data. 
A ranking was offered on the most common IV\&V work-breakdown structure (WBS) activities.
This ranking can be used for (e.g.) identifying what WBS tasks would benefit most from optimization.
\item
In early October, a preliminary report
was delivered on the Bayes network. On a limited case study, it was shown that Bayes nets
and treatment learning could generate parsimonious explanations 
for project events.
\ei
This report presents a preliminary report on our use of the
PITS issues tracking database. It will be shown that, using text mining,
PITS can be used to generate an expert system
that audits  a test engineer's proposed severity level for an issue.
\end{abstract}

\noindent
{\bf Credits:}
This work was made possible due to the heroic efforts
of Ken Costello (chief engineering at NASA IV\&V) who provided the PITS defect reports.
The text mining  technology used here was inspired by the trace-ability work of
Jane Hayes and Alex 
Dekhtyar. Alex was particularly helpful is mapping out the ABCs of text mining.
This research was conducted at
West Virginia University 
under NASA sub-contract project 100005549, task 5e, award 1002193r.

~\\

\noindent
{\bf Cautions}
Reference herein to
any specific commercial product, process, or service by trade name,
trademark, manufacturer, or otherwise, does not constitute or imply
its endorsement by the United States 
Government.

~\\

\noindent
{\bf Code:}All software discussed here is available from
\url{http://unbox.org/wisp/tmine}
under the  GNU Public License (version 2:
see \protect\url{www.gnu.org/copyleft/gpl.html}).

~\hrule~\\
\clearpage
\rhead{\thepage\ of \pageref{LastPage}}
\lhead{Menzies: Text Mining PITS - preliminary report}
\tableofcontents
\newpage
\listoffigures
\newpage

\section{Introduction: We don't need another hero}
\begin{wrapfigure}{l}{1.75in}
\includegraphics[width=1.75in]{hero.png}
\end{wrapfigure} 

NASA's software IV\&V Program captures all of its findings in a
database called the Project and Issue Tracking System (PITS).  
The data in PITS has been collected for more
than 10 years and includes issues on robotic satellite missions and
human-rated systems.  


It is difficult, to say the least, to generate conclusions from a moving
target like PITS. Several heroic
studies have made significant conclusions using PITS data (see
\fig{heros}).  These studies were heroic in the sense
that the ``heroes'' reached their goals after tedious and complex struggling.
Worse, the extracted data was only accessible
with the help of NASA civil servants- a scarce and expensive
resource. 
\begin{figure}[!h]
\small
\begin{tabular}{|p{0.95\linewidth}|}\hline
\bi
\item
Ken Costello (IV\&V's chief engineer)  compiled statistics for NASA headquarters that showing, in nine IV\&V tasks,
the majority of issues found by IV\&V were found via an analysis of requirements documents.
\item
Marcus Fisher (IV\&V's research lead) applied a ``mid-course correction'' to one IV\&V project after checking the
progress of the IV\&V against historical records in PITS.
\item
David Raffo (University of Portland), working with Ken Costello and other civil servants, found enough cost data to partially tune his
waterfall-based model of IV\&V;
\item
In a prior report in this project, Melissa Northey (Project Manager)
performed some joins across PITS to return costs
for different IV\&V tasks;
\ei\\\hline
\end{tabular}
\caption[Heroic successes with PITS]{
A partial list of past heroic successes with PITS.
}\label{fig:heros}
\end{figure}

The problem with PITS is that 
there is a lack of consistency in how each
of the projects collected issue data.  In virtually all instances,
the specific configuration of the information captured about an
issue was tailored by the IV\&V project to meet its needs.  This has
created consistency problems when metrics data is pulled across
projects.  While there was a set of required data fields, the
majorities of those fields do not provide information in regards
to the quality of the issue and are not very suitable for comparing
projects.


NASA is very aware of the problems with PITS and is taking
active steps to improve it. At the time of this
writing, there is an on-going effort to implement a mandatory data
set in each IV\&V project database to support IV\&V effectiveness
metrics.  This effort has been in development for about a year and
is currently being executed by several projects.  However, it is
too early to make any useful observations from that data.

To be fair, PITS is hardly unique. Based on my experience with data mining
at other corporations, I assert that PITS is a typical database, useful for storing
day-to-day information and generating small-scale tactical reports (e.g. ``list the bugs
we found last Tuesday''),
but difficult to  use for high-end business strategic analysis (e.g.. ``in the past, what methods have proved most
cost effective in finding bugs?'').
Like many other databases,
it takes heroes to extract information from PITS. Sadly, most of the heroes I know
are so busy saving their own part of the world that they have little time
to save researchers like me.


Hence, in this report, we try a new approach for extracting general conclusions
from PITS data. Unlike previous heroic efforts,
our text mining and machine learning methods are
low cost,  automatic, and rapid.
We find we can build an agent to automatically review issue reports and alert when
a proposed severity is anomalous. Better, the way we generated the agent means that
we have probabilities that the agent is correct. These probabilities can be used 
to intelligently guide decision making. For example, with our system, the following dialogue
is possible:
\begin{quote}
{\em Tim wrote the problem report and he says this is a severity 5 issue. But the agent
says that its a severity 3 issue with probability 83\%. Hmmm... the agent seems
pretty sure of itself- better get someone else to take a look at the issue.}
\end{quote}

An extremely surprising conclusion from this report is that the unstructured text might be a better candidate
for generating lessons learned than the structured data base fields. 
\bi
\item
While the database fields in PITS keep changing, the nature of the unstructured
text remains constant.
\item
In other words, the reason it is so hard in the past to reason about PITS  
is that we have been looking at the wrong data.
\ei
If we could properly understand unstructured text, this would be a result
of tremendous practical importance. 
A recent study\footnote{\url{http://www.b-eye-network.com/view/2098}} concluded that
\bi
\item
     80 percent of business is conducted on unstructured information;
\item     85 percent of all data stored is held in an unstructured format
(e.g.  the unstructured text descriptions of issues found in PITS);
  \item   Unstructured data doubles every three months;
\ei

That is, if we can tame the text mining problem, it would be possible to reason and learn from a much wider range of
NASA
data than ever before.
\newpage
\section{Concept of Operations}
NASA uses a five-point scale  to score issue severity.
The scale ranges one to five, worst to dullest, respectively.
A different scale is used 
for robotic and human-rated
missions (see \fig{define} and \fig{define1}).
The data used in this report comes from robotic missions.

Using text mining and machine learning methods,
this report shows that it is possible to
automatically generate
a {\em review agent}
from PITS issue reports
via the process of \fig{exprt}.
This agent can check the validity of the severity levels assigned to issues:
\bi
\item
After seeing an issue in some {\em artifact}, a human {\em analyst}
generates some text {\em notes} and assigned a severity
level {\em severityX}. 
\item
An agent learns a predictor for issue severity
level from logs
of
$\{notes,severityX\}$. A 
{\em training} module (a)~updates the agent beliefs and (b) determines
how much {\em self-confidence} a {\em supervisor} might have in
the {\em agent}'s conclusions.
\item 
Using the learned knowledge,
the {\em agent} reviews the {\em analysts}'s text and generates its own
{\em severityY} level. 
\item
If the agent's proposed {\em severityY}
differs from the {\em severityX} level
of the human {\em analyst}, then a human
{\em supervisor} can decide to review the human {\em analyst}'s {\em severityX}.
To help in that process, the
{\em supervisor} can review the {\em self-confidence} information to decide
if they trust the agent's recommendations. 
\ei
This agent would be of useful under the following circumstances:
\bi
\item 
When a less-experienced test engineer has assigned the wrong
severity levels. \item
When experienced test engineers are operating
under urgent time pressure demands, they could use the agent to
automatically and quickly
audit their conclusions.
\item
For agents that can detect severity one and two-level errors with high probability, the
agent could check for the rare, but extremely dangerous case, that an IV\&V team has missed
a high-severity problem.
\ei

\begin{figure}
\footnotesize
\begin{tabular}{|p{\linewidth}|}\hline
\begin{description}
\item[\em Severity 1:]
Prevent the accomplishment of an essential capability;
or jeopardize safety, security, or other requirement designated critical.
\item[\em Severity 2:]
Adversely affect the accomplishment of an essential capability and no work-around solution is known ;
or adversely affect technical, cost or schedule risks to the 
project or life cycle support of the system, and no work-around solution is known.
\item[\em Severity 3:]
Adversely affect the accomplishment of an essential capability but a work-around solution is known;
or  adversely affect technical, cost, or schedule risks to the project or life cycle support of the system, but a work-around solution is known.
\item[\em Severity 4:]
Results in user/operator inconvenience but does not affect a required operational or mission essential capability; or
results in inconvenience for development or maintenance personnel, but does not affect the accomplishment of these responsibilities.
\item[\em Severity 5:] Any other issues.
\end{description}\\\hline
\end{tabular}
\caption{Severities for robotic missions.}\label{fig:define}
\end{figure}
\begin{figure}
\footnotesize
\begin{tabular}{|p{\linewidth}|}\hline
\begin{description}
\item[\em Severity 1:]
A failure which could result in the loss of the human-rated
system, the loss of flight or ground personnel, or a permanently
disabling personnel injury.
\item[\em Severity 1N:]
 A failure which would otherwise
be Severity 1 but where an established mission procedure precludes
any operational scenario in which the problem might occur, or the
number of detectable failures necessary to result in the problem
exceeds requirements.
\item[\em Severity 2:]
A failure which could result in loss of
critical mission support capability. 
\item[\em Severity 2N:]
A failure which would
otherwise be Severity 2 but where an established mission procedure
precludes any operational scenario in which the problem might occur
or the number of detectable failures necessary to result in the
problem exceeds requirements.
\item[\em Severity 3:]
 A failure which is perceivable
by an operator and is neither Severity 1 nor 2.
\item[\em Severity 4:]
A failure which
is not perceivable by an operator and is neither Severity 1 nor 2.
\item[\em Severity 5:]
A problem which is not a failure but needs to be corrected such as
standards violations or maintenance issues.
\end{description}\\\hline
\end{tabular}
\caption{Severities for human-rated missions.}\label{fig:define1}
\end{figure}
\begin{figure*}
\begin{center}
\includegraphics[width=3in]{expert.pdf}
\end{center}
\caption[Reviewing severity levels]{An agent for reviewing 
issue severity levels.
Gray nodes denote humans.}\label{fig:exprt}
\end{figure*}

\clearpage
\section{How it Works}
The essential problem of text mining is {\em dimensionality reduction}.
Standard machine learners work well for instances that are nearly all fully described
using dozens (or fewer) attributes~\cite{witten99}.
But text mining applications (e.g. analyzing PITS detect reports) must process thousands of unique words,
and any particular paragraph may only mention a few of them~\cite{byrn99,salton83introduction}.
Therefore, before we can apply machine learning to text mining, we have to reduce the number
of dimensions (i.e. attributes) in the problem.

There are several standard methods for dimensionality reduction 
such as {\em tokenization, stop lists, stemming, Tf*Idf} and {\em InfoGain}.
All these methods are discussed below.
\subsection{Tokenization}
\fig{tokens} shows the tokenizer used in this study.
Words are reduced to simple tokens via (e.g.) removing all punctuation remarks, then sending all upper case to lower.

This code assumes that the input is some tab separated columns, one record per line.
Two special columns are recognized: the {\em Wanted} column (i.e. the column with the text)
and a {\em Klass} column (i.e. the column with the goal value- in our case, the severity level).
The {\em Wanted} column is extracted, tokenized, and printed one record per line with the associated
{\em Klass} (severity) value.
\begin{figure}[!b]
\scriptsize
\begin{verbatim}
#### config
Want=4   # analyst's notes are in column 4
Klass=5  # severity is in column 5

#### workers
tokenize()  { column $1 | lowerCase | alphasNotLasts; } 
lowerCase() { tr A-Z a-z $1 ; }
column() { 
    gawk -F"\t" ' NR > 1 {if($Klass) Class=$Klass;
                           print $Goal " "  Class}
                ' Klass=${Klass} Goal=$Want $1 ; }
alphasNotLast() {  
    # don't tokenize last column- it holds the class
    gawk '{ for(I=1;I<NF; I++) 
                gsub(/[^abcdefghijklmnopqrstuvwxyz]/," ",$I); 
                print $0
          }'  $1 
}

#### main
cat report | tokenize  > words
\end{verbatim}
\caption{Tokenization.}\label{fig:tokens}
\end{figure}

\subsection{Stop lists}
Another way to reduce dimensionality is to remove ``dull'' words via a 
{\em stop list}
of ``dull'' words. \fig{stop} shows a sample of the stop list used in this
study. IV\&V's chief engineer, Ken Costello, reviewed this list and removed
``counting words'' such as ``one'', ``every'', etc, arguing that ``reasoning about
number of events could be an important requirement''.
\fig{stopper} shows code for a stop-list function.
\begin{figure}
\begin{alltt}\scriptsize
a           about    across   again      against
almost      alone    along    already    also
although    always   am       among      amongst
amongst    amount   an       and        another
any         anyhow   anyone   anything   anyway
anywhere    are      around   as         at
...         ...      ...      ...        ...  
\end{alltt}
\caption[Stop words]{24 of the 262 stop words used in this study.}\label{fig:stop}
\end{figure}

\begin{figure}
\scriptsize
\begin{verbatim}
stops()     { gawk ' 
   NR==1 {  
        while (getline < Stops)  Stop[$0] = 1;
        while (getline < Keeps)  Keep[$0] = 1; 
    }
    { for(I=1;I<=NF;I++) 
        if (Stop[$I] && ! Keep[$I]) 
            $I=" "
       print $0
    }'  \
      Stops="$Here/stop_words.txt" \
      Keeps="$Here/keep_words.txt"  \
     $1
}
\end{verbatim}
\caption{Applying a stop-list.}\label{fig:stopper}
\end{figure}
\subsection{Stemming}
Terms with a common stem will
usually have similar meanings. For example, all these words relate to the same concept.
\bi
\item
        CONNECT
    \item    CONNECTED
     \item   CONNECTING
     \item   CONNECTION
     \item   CONNECTIONS
\ei
Porter's stemming algorithm~\cite{porter80}
 is the standard stemming tool. It repeatedly replies a set of pruning rules to the end of words until
the surviving words are unchanged. The pruning rules ignore the semantics of a word and just perform syntactic pruning (e.g. \fig{stem1}).
\begin{figure} \scriptsize \begin{verbatim}
RULE                EXAMPLE
----                -------
ATIONAL ->  ATE     relational     ->  relate
TIONAL  ->  TION    conditional    ->  condition
                    rational       ->  rational
ENCI    ->  ENCE    valenci        ->  valence
ANCI    ->  ANCE    hesitanci      ->  hesitance
IZER    ->  IZE     digitizer      ->  digitize
ABLI    ->  ABLE    conformabli    ->  conformable
ALLI    ->  AL      radicalli      ->  radical
ENTLI   ->  ENT     differentli    ->  different
ELI     ->  E       vileli         ->  vile
OUSLI   ->  OUS     analogousli    ->  analogous
IZATION ->  IZE     vietnamization ->  vietnamize
ATION   ->  ATE     predication    ->  predicate
ATOR    ->  ATE     operator       ->  operate
ALISM   ->  AL      feudalism      ->  feudal
IVENESS ->  IVE     decisiveness   ->  decisive
FULNESS ->  FUL     hopefulness    ->  hopeful
OUSNESS ->  OUS     callousness    ->  callous
ALITI   ->  AL      formaliti      ->  formal
IVITI   ->  IVE     sensitiviti    ->  sensitive
BILITI  ->  BLE     sensibiliti    ->  sensible
\end{verbatim} \caption{Some stemming rules.}\label{fig:stem1}
\end{figure}

Porter's stemming algorithm has been coded in any 
number of languages\footnote{\url{http://www.tartarus.org/martin/PorterStemmer}} such as the Perl {\em stemming.pl}
used in this study (see \fig{stem2}).
\begin{figure} \scriptsize \begin{verbatim}
stemming()  { perl $Here/stemming.pl $1 ; }
\end{verbatim} \caption{Using a downloaded stemmer.}\label{fig:stem2}
\end{figure}

\subsection{Tf*IDF}
Tf*Idf is shorthand for ``term frequency times inverse document
frequency''. This calculation models the intuition that jargon
usually contains technical words that appear a lot, but only in a
small number of paragraphs.  For example, in a document describing
a space craft, the terminology relating to the power supply may be
appear frequently in the sections relating to power, but nowhere
else in the document.

Calculating Tf*Idf is a relatively simple matter:
\bi
\item
Let there be $Words$ number of documents;
\item
Let some word $I$ appear $Word[I]$ number of times inside a set of $Documents$; 
\item
Let $Document[I]$ be the documents containing $I$.
\ei
Then:
\[
Tf*Id = Word[i]/Words*log(Documents/Document[i])
\]

\begin{figure} \scriptsize \begin{verbatim}
#update counters for all words in the record
function train()  {
    Documents++; 
    for(I=1;I<NF;I++) {
        if( ++In[$I,Documents]==1) 
            Document[$I]++   
        Word[$I]++    
        Words++ 
   }
}
# computer tfidf for one word
function tfidf(i) { 
    return Word[i]/Words*log(Documents/Document[i]) 
}
\end{verbatim} \caption{tfidf.awk.}\label{fig:tfidfawk}
\end{figure}

\begin{figure} \scriptsize \begin{verbatim}
tfidf() { 
    gawk  -f tfidf.awk --source '
      { train() }
  END { OFS=","; for(I in Word) print I, tfidf(I) } ' $1 ; 
      } ' $1 
} 
tfidf | sort -t, -n +0 | tail -100
\end{verbatim} \caption[Finding the 100 highest 
                        Tf*Idf words]{Finding the 
              100 highest Tf*Idf words using the $tfidf.awk$ code of \fig{tfidfawk}. 
         }\label{fig:tfidf}
\end{figure}


The standard way to use this measure  is to cull all but the $k$ top Tf*Idf ranked
stopped, stemmed tokens. This study used $k=100$ (see \fig{tfidfawk} and \fig{tfidf}).
   
\subsection{InfoGain}

According to the $InfoGain$ measure,
the {\em best} words  are those that  
{\em most simplifies} the target
concept (in our case, the distribution of severities).
Concept ``simplicity'' is measured using information theory.
Suppose a data set has 80\% severity=5 issues and  20\% severity=1
issues.
Then that data set has a class distribution $C_0$ 
with classes $c(1)=severity5$ and $c(2)=severity1$
with frequencies $n(1)=0.8$ and $n(2)=0.2$.
The
number of bits required to encode an arbitrary class 
distribution $C_0$ is $H(C_0)$ defined as follows:
\begin{equation}\label{eq:ent}{
\left.\begin{array}{rcl}
N&=&\sum_{c{\in}C}n(c)\\
p(c)& =& n(c)/N \\
H(C)&=& -{\sum}_{c{\in}C} p(c)log_2p(c)
\end{array}\right\}
}\end{equation}

After discretizing numeric 
data\footnote{E.g. given an attribute's
minimum and maximum values, replace a particular value $n$ with $(n - min)/((max - min)/10)$.
For more on discretization, see~\cite{dou95}.}
then if $A$ is a set of attributes, the number of bits required to encode
a class after observing an attribute is:
\[H(C|A) = -{\sum}_{a{\in}A} p(a) {\sum}_{c{\in}C} p(c|a)log_2(p(c|a)\]
The highest ranked
attribute $A_i$ is the one with the largest {\em information gain};
i.e the one that most
reduces the encoding required for the data {\em after}
using that attribute; i.e.
\begin{equation}\label{eq:infogain}
InfoGain(A_i) = H(C) - H(C|A_i)
\end{equation}
where $H(C)$ comes from \eq{ent}.
In this study, we will use InfoGain to find  the top $N=10$ most informative tokens. 
\begin{figure}
\footnotesize
\begin{center}\begin{tabular}{lrrr}
data set & Domain & number & \% of total\\\hline
l &          FSW& 104& 67\%    \\
   &    Ground&  17& 11\%\\
  &Instruments&  33& 21\%\\\hline
 
p &     Payload& 584& 75\%\\
  & Spacecraft& 188& 24\%\\\hline
s &         AIA&   62&  4\%\\
   &      EVE&   68&  4\%\\
    &     SDO&    1&  0\%\\
     & Ground&   17&  1\%\\
 & Spacecraft& 1353& 90\%\\\hline
\end{tabular}\end{center}
\caption{Domains in the this study.}\label{fig:domains}
\end{figure}
\begin{figure}
\begin{center}
\begin{tabular}{|c|c|c|}\hline
l & p & s\\
\includegraphics[width=1in]{sever_l.pdf} &
\includegraphics[width=1in]{sever_p.pdf} &
\includegraphics[width=1in]{sever_s.pdf}\\\hline
\end{tabular}
\end{center}
\caption{Severities.}\label{fig:severs}
\end{figure}

\begin{figure*}[!b]
\begin{center}
\begin{tabular}{p{0.51in}r|p{0.7in}|p{0.8in}|p{0.7in}|p{0.8in}|p{0.5in}|}
\multicolumn{2}{l}{project=l}       &\multicolumn{3}{c}{Found in}\\\cline{3-5}
   		  &	                &Requirements& Design& Implementation\\\cline{1-5}
  Created in &Requirements     &28          &     8 &   79\\\cline{3-5}
             &Design           &            &    23 &     \\\cline{3-5}
             &Implementation   &            &       &   15\\\cline{3-5}
\\
\multicolumn{2}{l}{project=p}       &\multicolumn{3}{c}{Found in}\\\cline{3-7}
           &                 & Design
					& SW requirements design
					& SW Preliminary design
					& Implementation
					& SW implementation\\\hline
 Created in &Design                            &362          &    &      &      &\\\cline{3-7}
            &SW requirements design            &             &    &   50 &       & 87\\\cline{3-7}
            &Subsystems requirements design    &             &    &      &        &19\\\cline{3-7}
            &Implementation                    &             &    &      &    69&\\\cline{3-7}
            &SW implementation                 &             &    &      &       & 165\\\cline{3-7}
\end{tabular}
\end{center}
\caption[Leakage matrices]{Leakage matrices for project=l (above) and project=b (below).}\label{fig:leaks}
\end{figure*}
\begin{figure*}[!b]
\begin{tabular}{ccc}
l & p & s \\
\includegraphics[width=2in]{l.pdf}&
\includegraphics[width=2in]{p.pdf}&
\includegraphics[width=2in]{s.pdf}
\end{tabular}
\caption{Effects of stopping and stemming.}\label{fig:stopstem}
\end{figure*}
\begin{figure*}[!t]
\begin{tabular}{ccc}
l & p & s \\
\includegraphics[width=1.5in,angle=90]{tfidf_ok_l.pdf}&
\includegraphics[width=1.5in,angle=90]{tfidf_ok_p.pdf}&
\includegraphics[width=2in]{tfidf_ok_s.pdf}
\end{tabular}
\caption[Tf*Idf scoring for the stopped, stemmed tokens]{Tf*Idf scoring for 
the stopped, stemmed tokens.  Note that most tokens can be ignored  since
they have very low Tf*Idf scores.}\label{fig:tfidf3}
\end{figure*}

\newpage\newpage
\section{Results}

\subsection{Data}
The above methods where applied to ``l'', ``p'', ``s''; i.e.
three anonymous PITS projects supplied by Ken Costello.
The domains mentioned in these data sets are shown in \fig{domains}.
This data
contained 155, 773, 4661 issue reports for projects ``l,p,s'' (respectively);
i.e. ``l'' was quite small and ``s'' was quite large. 
The 
issues were created and found in various phases (see \fig{leaks}- note that there
is no created/found data for project ``s'' since that project lacked phase information).

As shown in \fig{severs},
the severities in this data were not evenly distributed:
\bi
\item
Most of the severities where 3 or 4 
\item
The data contained only one severity 1 record and 91 severity 2 records.
\ei

\subsection{Stopping and Stemming}

\fig{stopstem} shows some disappointing results for stopping and stemming.
In these data sets, stopping and stemming
methods barely reduced the number of tokens.

\subsection{Tf*Idf}

Tf*Idf proved to be more powerful:
\fig{tfidf3}  shows that in all three data sets, there exist a very small
number of words with high Tf*Idf scores. 
The 100 top Tf*Idf tokens from each data set were extracted and the issue reports
were rewritten as frequency counts for those top 100 tokens, with the severity
value for each record written to the end of line
(see \fig{counter}). 
A review of the top 100 tokens showed little similarity in the top-ranked
tokens; i.e. this study of three data sets
found no general pattern in the terminology associated with severity.
\begin{figure} \scriptsize \begin{verbatim}
NR ==1 { 
	# grab the words we want to count
	while (getline < "top100") Want[$0] = 1;
    # write the header
    for(I in Want) 
              printf("%s,",I); 
    print "severity"
}
NR > 1 { # rewrite each record as counts of "Want" 
        gsub(/ /,"",$2);  counts($1,Want,  $2) 
}
function counts(str,want,klass,   sum,out,i,j,n,tmp,got) {
    n=split(str,tmp," ");
    for(i=1;i<=n;i++) 
        if (tmp[i] in want) 
             got[tmp[i]]++;
    for(j in want) {
        sum += got[j]
        out = out got[j]+0 ",";  
    }
    if (sum)
        print out "_" klass
}
\end{verbatim}
\caption{Re-writing issue reports as frequency counts.}\label{fig:counter}
\end{figure} 

\subsection{Learning}

A classier was then called
to learn a predictor for the severity attribute using the other 100 attributes.
The classifier used here was a JAVA version of Quinlan;s C4.5 decision tree learner~\cite{quinlan92,witten99}.
C4.5 applies InfoGain to find the best root of a tree. Data is then split according
to the values of the attribute and the algorithm recurses into every split.

For each data set, the classifier was called twice:
\bi
\item Once using 100 independent attributes;
\item Once using the 10 independent  attributes
ranked as ``top ten'' by InfoGain.
\ei
\fig{100} and \fig{10} show the decision trees learned
from 100 and 10 attributes for project ``l''.
\begin{figure}
\scriptsize
decision tree:
\begin{verbatim}
child <= 0
|   buffer <= 0
|   |   sr <= 0
|   |   |   after <= 0
|   |   |   |   gnac <= 0
|   |   |   |   |   statement <= 0
|   |   |   |   |   |   potenti <= 0
|   |   |   |   |   |   |   fswr <= 0
|   |   |   |   |   |   |   |   sb <= 0
|   |   |   |   |   |   |   |   |   variabl <= 0: _3 
|   |   |   |   |   |   |   |   |   variabl > 0: _4 
|   |   |   |   |   |   |   |   sb > 0: _4 
|   |   |   |   |   |   |   fswr > 0: _4 
|   |   |   |   |   |   potenti > 0: _3
|   |   |   |   |   statement > 0: _4 
|   |   |   |   gnac > 0: _4 
|   |   |   after > 0: _4 
|   |   sr > 0: _4 
|   buffer > 0
|   |   cfe <= 1: _3 
|   |   cfe > 1: _4
child > 0: _4 
\end{verbatim}

confusion matrix:
\begin{verbatim}
  a  b  c   <-- severity classified as
 76 11  0 |  a = 3
 32 21  0 |  b = 4
  1  0  0 |  c = 2
\end{verbatim}
\caption{Project ``l'''s results, using 100 attributes.}\label{fig:100}
\end{figure}
\begin{figure}
\scriptsize

decision tree:
\begin{verbatim}
child <= 0
|   after <= 0
|   |   gnac <= 0: _3 
|   |   gnac > 0: _4 
|   after > 0: _4 
child > 0: _4 
\end{verbatim}

confusion matrix:
\begin{verbatim}
  a  b  c   <-- severity classified as
 87  0  0 |  a = 3
 48  5  0 |  b = 4
  1  0  0 |  c = 2
\end{verbatim}
\caption{Project ``l'''s results, using 10
attributes.}
\label{fig:10}
\end{figure}
 As might be expected, the tree learned from 100 attributes 
is larger than the one learned from 10 attributes.

At the bottom of \fig{100} and \fig{10} are {\em confusion matrices}
that report how many times records of each severity were classified
as severity 1,2,3,4, or 5. 
This matrix shows average performance results of decision
trees  learned from ten 90\% samples of project ``l'', then
tested on the remaining 10\% of the data.
These confusion matrices offer the {\em self-confidence}
measures discussed in \S2:
\bi
\item
For example, in \fig{100}, severity 3 issues were classified
as severity 3 in $\frac{76}{76+11}=87\%$
of cases. 
That is, if the agent reports that the issue is severity 3,
then it is highly likely that the agent is right and other
severity allocations are wrong.
\item
However, in the same figure, the results for
severity 4 classification is not as impressive:
these severities were only correctly classified
in $\frac{21}{21+32}=39\%$ of the tests.
That is, if the agent reports that the issue is severity 4,
then it is unlikely that the agent is right.
\ei

(Note that that the classifiers of \fig{100} and \fig{10}
never correctly
classified any issue
as severity 1 or 2- thought one severity 2 issue was incorrectly
classified as severity 3.
This is not a fault of our method; rather it is a quirk of the training
data in \fig{severs} that has no severity 1 issues and only one severity two
issues. In the future, we will repeat this method for data sets with
more severe issues.)

One problem with decision tree learning is that the tree can be unreadable.
Project ``l'' is our smallest data set and its decision tree
(in \fig{100}) would be hard to explain to most users.
Worse, other larger data sets yield far larger and far more confusing
decision trees. For example, the decision tree from project ``s'' is
ten times larger than \fig{100}.
\begin{figure}
\scriptsize

decision tree:
\begin{verbatim}
convent <= 0
|   unus <= 0: _3 
|   unus > 0: _4 
convent > 0: _4 

\end{verbatim}

confusion matrix:
\begin{verbatim}
   a   b   c   <-- classified as
  46 171   0 |   a = _4
   3 473   0 |   b = _3
   0  38   0 |   c = _2
\end{verbatim}
\caption{Project ``p'''s results, using 10
attributes.}
\label{fig:10p}
\end{figure}
\begin{figure}
\scriptsize

decision tree:
\begin{verbatim}
arrai <= 0
|   verifi <= 0
|   |   s <= 0
|   |   |   test <= 1
|   |   |   |   requir <= 0: _4 
|   |   |   |   requir > 0
|   |   |   |   |   doe <= 0
|   |   |   |   |   |   requir <= 1: _3 
|   |   |   |   |   |   requir > 1: _4 
|   |   |   |   |   doe > 0
|   |   |   |   |   |   requir <= 2
|   |   |   |   |   |   |   requir <= 1: _3 
|   |   |   |   |   |   |   requir > 1: _4 
|   |   |   |   |   |   requir > 2: _3 
|   |   |   test > 1
|   |   |   |   scenario <= 1
|   |   |   |   |   line <= 0: _3 
|   |   |   |   |   line > 0
|   |   |   |   |   |   scenario <= 0: _4 
|   |   |   |   |   |   scenario > 0: _3 
|   |   |   |   scenario > 1: _4 
|   |   s > 0
|   |   |   doe <= 0
|   |   |   |   test <= 0: _4 
|   |   |   |   test > 0: _3 
|   |   |   doe > 0: _3 
|   verifi > 0
|   |   doe <= 0
|   |   |   line <= 0
|   |   |   |   s <= 0
|   |   |   |   |   requir <= 1: _3 
|   |   |   |   |   requir > 1: _4 
|   |   |   |   s > 0: _4 
|   |   |   line > 0: _4 
|   |   doe > 0: _3 
arrai > 0
|   comm <= 0: _3 
|   comm > 0: _2 
\end{verbatim}

confusion matrix:
\begin{verbatim}
   a   b   c   d   <-- classified as
 411 273   0   0 |   a = _3
 212 411   0   0 |   b = _4
  18  18  16   0 |   c = _2
   0   1   0   0 |   d = _1
\end{verbatim}
\caption{Project ``s'''s results, using 10
attributes.}
\label{fig:10s}
\end{figure}

The trees learned from the ten attributes selected by InfoGain are much
more readable (see \fig{10}, \fig{10p}, \fig{10s}). For example, \fig{10} has only six nodes and
the tree learned from ten project ``s'' attributes
has only 40 nodes (see \fig{10s}). 
Such smaller trees have interesting performance
properties:
\bi
\item
Smaller trees
can't handle as many special cases as the larger trees.
Hence,  the performance of the smaller theory can be
worse than the
more elaborate theory. For example,  the first and last
lines of \fig{performance} shows that the probabilities
of detecting severity 2 and 4 errors
is always decreased by InfoGain.
\item
InfoGain can also discard attributes that tend to
confuse a decision tree learner. Hence, the performance of the smaller
theory  can be better than the more elaborate theory.
For example, the middle row of \fig{performance} shows
that InfoGain improved the ability of our agent to 
detect severity 3 errors in projects ``l'' and ``p'' (but
not in ``s''). 
Some of the improvements were remarkable- InfoGain+decision tree learning for
projects ``l'' and ``p''
yielded detectors with a 99\% to 100\% probability
of detecting severity 3 errors.
\ei 

\begin{figure}
\footnotesize
\begin{center}
\begin{tabular}{r|rrr|rrr|rrr}
&           l  & &         &p  & &        &          s & &\\\cline{2-10}
\begin{sideways}severity\end{sideways}&100&10&$\frac{100}{10}$&100&10&$\frac{100}{10}$&100&10&$\frac{100}{10}$\\\hline
2&0&0&0.00&52&0&0.0&38&31&0.82\\
3&83&100&1.20&90&99&1.1&73&60&0.82\\
4&39&9&0.23&52&21&0.4&71&65&0.91\\
\end{tabular}
\end{center}
\caption[Percent probabilities of detection]{Perecent robabilities of 
detection using 100 attributes or just the 10 selected
by InfoGain for projects ``l'', ``p'', ``s''. 
The difference in performance is shown in
the columns marked with $\frac{100}{10}$ (and numbers greater
than one mean that InfoGain has 
made the performance {\em better}).} \label{fig:performance} 
\end{figure}

\clearpage
\section{Discussion}

%The adage ``garbage in, garbage out''
%refers to the way computers 
%can unquestioningly process any nonsense input
%and generate seemingly impressive, but totally nonsensical
%output. Wikipedia speculates\footnote{
%\url{http://en.wikipedia.org/wiki/Garbage_in,_garbage_out}}
% that the adage
%may be first attributed, in a verbose form,
%to Charles Babbage (1791 to 1871). Babbage, designer of  the first
%programmable devices, remarked that:
%\begin{quote}
%On two occasions I have been asked, 'Pray, Mr. Babbage, 
%if you put into the machine wrong figures, will the right answers come out?' 
%I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.
%\end{quote}
%A related, but more worrying, problem  is ``garbage in, gospel
%out''. In this second form of GIGO, the audience accepts bogus
%results from computer systems with unquestioning faith.  Data mining
%professionals must remain forever vigalent for the GIGO problem.
%In the $21^{st}$ century, there now exist many public domain data
%miners which can be installed in minutes and process hundreds of
%thousands of records in a few hours (or less). But before we can
%believe the output of such miners, we need to first trust that the
%input is worthy of mining.

Over the years, the Project Issue Tracking System (PITS) has been
extensively and repeatedly modified.  Prior attempts at generating
generalized conclusions from PITS have required significant levels
of manual, hence error-prone, processing.

Here, we show that conclusions can be reached from PITS without
heroic effort. Using text mining and machine learning methods, we
have shown that it is possible to automatically generate predictors
for severity levels from the free text entered into PITS.

Better yet, the detectors learned via the methods described in this
report come with a set of self-confidence measures.  In some cases,
that self-confidence was very high (e.g. a 100\% confidence in our
ability to recognize severity 3 errors in project ``l'' using the tiny decision tree of \fig{10}).  That is, the user of our
proposed agent would know when to trust, and when to ignore, the
agent's conclusions.

The current implementation of the agent has its limits. For example: 
\bi
\item
The training data used in the study had very few level 1 and level 2 severity problems.
Hence, there was not enough data for our agent to learn high probability detectors for high severity
issues. 
\item
In each of the three data sets studied here, useful predictors were learned for each data set
but the data sets were so different that the predictors are not general outside of their training domain.
\item
Some of the predictions have low probability, particularly for severity 1 and severity 2 errors. \ei
\noindent
Hence, for the next version of this report, we will:
\bi
\item
Repeat this study using  different data sets containing higher severities.
\item
Study more data sets looking for patterns that are general to multiple training domains.
\item
Explore a wider range of text mining techniques to improve our probability of detecting the severity.
For example:
\bi
\item This study used various ``magic numbers''; e.g. the 100 Tf*Idf token and  the top 10 InfoGain-scored tokens.
Subsequent studies should check how changing those magic numbers effects the outcome.
\item Stemming and stopping seemed to be of little value in this study. Perhaps our performance changes if we ignore one or both of them?
\item Work with di-grams/ tri-grams of tokens instead of one-grams;
\item Further filter the tokens by removing tokens that score less or more than some minimum or maximum counts;
\item Using domain knowledge, add  in special ``keep words''  representing core domain concepts;
\item Augment the syntactic analysis used in this study with a more semantic analysis (e.g. more
background knowledge of error types);
\item Normalize the Tf*Idf counts using different filters (e.g. logarithms).
\ei
\ei

\newpage
\bibliographystyle{abbrv}
\bibliography{refs}
\end{document}


\end{document}
\subsection{Another Mining Method}

Since the structured data in PITS is problematic, this research
elected to study PITS' unstructured data. Several fields within the
database support long strings of free text.

XXX recent success in data mining

\section{Under the Hood}
While
\fig{sever} shows the high-level business view of the tool,
the full details are shown in \fig{process}.
\begin{figure}[!t]
\begin{center}
\includegraphics[width=3.2in]{process.pdf}
\end{center}
\caption{Text mining- the details.}\label{fig:sever}
\end{figure}
Text mining~\cite{byrn99,salton83introduction} is different from machine
learning~\cite{witten99} in that the latter deals with more structured data.
A standard machine learing task is to:
\bi
\item 
Input a table characterized by
dozens (or less) of attributess;
\item And output 
a theory that connects the independent attributes to a single class attribute.
\ei
Text mining is different in that the number of attributes is much larger
and the table of data may be vary sparse: 
\bi
\item
The number of attributes equals the number of unique words in the documents.
For example, the PITS data 
contains hundreds to thousands of unique words per project.
\item
Most words don't appear in most documents. Hence a matrix storing documents and words
in the rows/columns (respectively) would contain many empty cells.
\ei
\fig{process} converts  a text mining problem ito a machine learning problem.
In essence,
this means finding and deleting all but the most ``important'' words.
This section describes how such ``important'' words are recognized.

\subsection{Tokenization}

\begin{figure}
\begin{alltt}\scriptsize
a           about    across   again      against
almost      alone    along    already    also
although    always   am       among      amongst
amoungst    amount   an       and        another
any         anyhow   anyone   anything   anyway
anywhere    are      around   as         at
...         ...      ...      ...        ...  
\end{alltt}
\caption[Stop words]{24 of the 262 stop words used in this study.}
\end{figure}

\begin{figure}
\scriptsize
\begin{verbatim}
stops()     { gawk ' 
   NR==1 {  
        while (getline < Stops)  Stop[$0] = 1;
        while (getline < Keeps)  Keep[$0] = 1; 
    }
    { for(I=1;I<=NF;I++) 
        if (Stop[$I] && ! Keep[$I]) 
            $I=" "
       print $0
    }'  \
      Stops="$Here/stop_words.txt" \
      Keeps="$Here/keep_words.txt"  \
     $1
}
\end{verbatim}
\caption{Tokenization.}\label{fig:tokens}
\end{figure}
\includegraphics[width=2in]{s.pdf}

\includegraphics[width=2in]{l.pdf}

\includegraphics[width=2in]{p.pdf}

\includegraphics[width=1.5in,angle=90]{tfidf_ok_l.pdf}

\includegraphics[width=2in]{tfidf_ok_s.pdf}

\includegraphics[width=1.5in,angle=90]{tfidf_ok_p.pdf}
\section{Data Mining}
\subsection{Decision Tree Learning}
One way to view OneR's defect predictions rules is a decision tree
of maximum depth one whose leaves are either the label {\em defective}
or {\em defect-free}. The J48 learner builds decision trees of any depth. For example, J48 
executing on the {\em kc4} data set can return:

{\scriptsize\begin{verbatim}
CALL_PAIRS <= 0: defectFree 
CALL_PAIRS > 0
|  NUMBER_OF_LINES <= 3.12: defectFree 
|  NUMBER_OF_LINES > 3.12
|  |  NORMALIZED_CYLOMATIC_COMPLEXITY <= 0.02
|  |  |   NODE_COUNT <= 3.47: defective 
|  |  |   NODE_COUNT > 3.47: defectFree 
|  |  NORMALIZED_CYLOMATIC_COMPLEXITY>0.02: defective 
\end{verbatim}}
\noindent
which may be read as follows: ``a module is defective if it has non-zero call-pairs and has more than
3.12 lines and hasn't  a low normalized cyclomatic complexity (0.02) or it has a low normalized
cyclomatic complexity and a low node-count (up to 3.47)''. 

Note that J48 predictors can be more
complex and explore more special cases than OneR. J48's predictors would out-perform OneR if
defect prediction required such elaboration.

J48 is a JAVA implementation of Quinlan's C4.5 (version 8)
algorithm~\cite{quinlan92}. The algorithm recursively {\em splits}
a data set according to tests on attribute values in order to
separate the possible predictions (although attribute tests are chosen one at a
time in a greedy manner, they are dependent on results of previous
tests). C4.5/J48 uses information theory to assess candidate splits:
the {\em best split} is the one that {\em most simplifies} the target
concept.  
Concept simplicity is measured using information theory.
Suppose a data set has 80\% defect-free modules and  20\% defective modules.
Then that data set has a class distribution $C_0$ with classes $c(1)=defectFree$ and $c(2)=defective$
with frequencies $n(1)=0.8$ and $n(2)=0.2$.
The
number of bits required to encode an arbitrary class distribution $C_0$ is $H(C_0)$ defined as follows:
\begin{equation}\label{eq:ent}{
\left.\begin{array}{rcl}
N&=&\sum_{c{\in}C}n(c)\\
p(c)& =& n(c)/N \\
H(C)&=& -{\sum}_{c{\in}C} p(c)log_2p(c)
\end{array}\right\}
}\end{equation}
A split divides $C_0$ (before the split) into $C_1$ and $C_2$
(after the split). The best split leads to the simplest concepts;
i.e. maximize $H(C_0) - (H(C_1) + H(C_2))$.

Various  {\em attribute subset  selection}
algorithms~\cite{hall03} (hereafter, {\em subsetting}) find what attributes can be deleted, without
damaging the performance of the learned predictor.
The simplest and fastest
subsetting
method is to rank attributes from the most
informative to least informative. After discretizing numeric 
data\footnote{E.g. given an attribute's
minimum and maximum values, replace a particular value $n$ with $(n - min)/((max - min)/10)$.
For more on discretization, see~\cite{dou95}.}
then if $A$ is a set of attributes, the number of bits required to encode
a class after observing an attribute is:
\[H(C|A) = -{\sum}_{a{\in}A} p(a) {\sum}_{c{\in}C} p(c|a)log_2(p(c|a)\]
The highest ranked
attribute $A_i$ is the one with the largest {\em information gain};
i.e the one that most
reduces the encoding required for the data {\em after}
using that attribute; i.e.
\begin{equation}\label{eq:infogain}
InfoGain(A_i) = H(C) - H(C|A_i)
\end{equation}
where $H(C)$ comes from \eq{ent}.
In {\em iterative InfoGain subsetting}, predictors are  learned
using the  $i=1,2...,N$-th top-ranked attributes.
Subsetting  terminates when $i+1$ attributes perform no better than
$i$.


\subsection{l}

\begin{figure}
\scriptsize
\begin{verbatim}
child <= 0
|   buffer <= 0
|   |   sr <= 0
|   |   |   after <= 0
|   |   |   |   gnac <= 0
|   |   |   |   |   statement <= 0
|   |   |   |   |   |   potenti <= 0
|   |   |   |   |   |   |   fswr <= 0
|   |   |   |   |   |   |   |   sb <= 0
|   |   |   |   |   |   |   |   |   variabl <= 0: _3 
|   |   |   |   |   |   |   |   |   variabl > 0: _4 
|   |   |   |   |   |   |   |   sb > 0: _4 
|   |   |   |   |   |   |   fswr > 0: _4 
|   |   |   |   |   |   potenti > 0: _3
|   |   |   |   |   statement > 0: _4 
|   |   |   |   gnac > 0: _4 
|   |   |   after > 0: _4 
|   |   sr > 0: _4 
|   buffer > 0
|   |   cfe <= 1: _3 
|   |   cfe > 1: _4
child > 0: _4 
\end{verbatim}

\begin{verbatim}
  a  b  c   <-- classified as
 76 11  0 |  a = _3
 32 21  0 |  b = _4
  1  0  0 |  c = _2
\end{verbatim}
\caption{"L"'s results with all attributes.}
\end{figure}

\begin{figure}
\scriptsize

\begin{verbatim}
child <= 0
|   after <= 0
|   |   gnac <= 0: _3 
|   |   gnac > 0: _4 
|   after > 0: _4 
child > 0: _4 
\end{verbatim}


\begin{verbatim}
  a  b  c   <-- classified as
 87  0  0 |  a = _3
 48  5  0 |  b = _4
  1  0  0 |  c = _2
\end{verbatim}
\caption{"L"'s results with just
after,
child,
close,
document,
fswr,
gnac,
level,
lroc,
result,
statu.}
\caption{Results: data set l with InfoGain.}
\end{figure}

\section{Text Mining}

Text Analysis software operates in 5 stages:

   1.

      Extraction of the text contained in your documents (meaning conversion of multiple file formats).
   2.

      Lexical analysis: Words identification, optionally using grammatical rules and a lexicon (for example, "bushes" and "bush" can belong to the same word).
   3.

      Semantic analysis: Solve language ambiguities, optionally using a specific dictionary (thesaurus or semantic network). For example, the word "bush" can be related to "George Bush" or to a Plant (Shrub).
   4.

      Statistics: Group together closely related terms appearing throughout the text. For example, "bush" and "president" can be grouped into the "politics" cluster.
   5.

      Sorting (keywords lists) and showing (graphs, etc.) the results. 
\bibliographystyle{abbrv}
\bibliography{refs}
\end{document}

XXX one gram, two gram, three gram
XXX semantic less
XXX normalization
XXX min counts, max counts
XXX symbol table management (words to/from integers)
XXX spare matrix,
XXX dimensionality reduction: svd, tdidf, td, idf, infogain, min/max counts

XXX XXX another set with more sverity 1
%XXX large corpus- standard parts, stablizexe the ontology
