%\documentclass[onecolumn,draftcls,journal,compsoc]{IEEEtran}
%\documentclass[technote]{IEEEtran}
%\documentclass{sig-alternate}
\documentclass{speauth}

\usepackage[dvips,colorlinks,bookmarksopen,bookmarksnumbered,citecolor=red,urlcolor=red]{hyperref}

%\def\baselinestretch{2} 
\usepackage{graphicx}
\usepackage{cite}
\usepackage{wrapfig,url}
\newcounter{over}
\newcounter{max}
\newcommand{\fred}[2]{\setcounter{over}{#2}\addtocounter{over}{#1}}
\newcommand{\boxplot}[5]{%
\scalebox{0.5}{\textcolor{white}{\setcounter{max}{#4}\addtocounter{max}{#5}\fred{#4}{-#2}}%
\begin{picture}(100,10)%
\put(0,0){\line(0,1){8}}%
\put(100,0){\line(0,1){8}}%
%\put(#1,4){\circle*{1}}%
%\put(\themax,4){\circle*{1}}%
\put(#2,4){\line(1,0){\theover}}%
\put(#3,4){\circle*{8}}%
%\put(#4,4){\line(1,0){#5}}%
\put(50,0){\line(0,1){8}}%
\end{picture}}
} 
\usepackage{eso-pic}
\usepackage{color}
\usepackage{type1cm}
%\makeatletter
%  \AddToShipoutPicture{%
%    \setlength{\@tempdimb}{.5\paperwidth}%
%    \setlength{\@tempdimc}{.5\paperheight}%
%    \setlength{\unitlength}{1pt}%
%      \makebox(600,900){\rotatebox{45}{\textcolor[gray]{0.80}{\fontsize{5cm}{5cm}\selectfont{Draft v5}}}}}
%\makeatother
\newenvironment{smallitem}
 {\setlength{\topsep}{0pt}
                     \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{itemize}
   \setlength{\leftmargin}{.2in}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{itemize}}
\newenvironment{smallenum}
 {\setlength{\topsep}{0pt}
  \setlength{\partopsep}{0pt}
  \setlength{\parskip}{0pt}
  \begin{enumerate}
  \setlength{\leftmargin}{.2in}
  \setlength{\parsep}{0pt}
  \setlength{\parskip}{0pt}
  \setlength{\itemsep}{0pt}}
 {\end{enumerate}}

 \newcommand{\bd}{\begin{description}}
\newcommand{\ed}{\end{description}}
 \newcommand{\bi}{\begin{smallitem}}
\newcommand{\ei}{\end{smallitem}}
 \newcommand{\be}{\begin{smallenum}}
\newcommand{\ee}{\end{smallenum}}
\newcommand{\fig}[1]{Figure~\ref{fig:#1}}
\newcommand{\eq}[1]{Equation~\ref{eq:#1}}
\newcommand{\tion}[1]{\S\ref{sec:#1}}

%\usepackage{times}
\usepackage{alltt}
\usepackage{url}
% IEEE Computer Society needs nocompress option
% requires cite.sty v4.0 or later (November 2003)
\usepackage{cite}
% normal IEEE
 % \usepackage{cite}

% correct bad hyphenation here
\hyphenation{op-tical net-works semi-conduc-tor}


\pdfpagewidth=8.5in
\pdfpageheight=11in


\begin{document}
\def\cop{Copyright \copyright\ 2010 John Wiley \&\ Sons, Ltd.}

%\SPE{1}{6}{00}{00}{2000} %for times fonts
\SPE{1}{7}{00}{00}{2010}

\runningheads{A. Nelson et al.}
{Sharing Experiments Using Open Source Software}



%\conferenceinfo{PROMISE'09,} {May 18--19, 2009, Vancouver, Canada} 
%\conferenceinfo{~} {~} 

%\CopyrightYear{2009}

%\crdata{978-1-60558-036-4/08/05}


%\CopyrightYear{2008} % Allows default copyright year (200X) to be over-ridden - IF NEED BE.
%\crdata{0-12345-67-8/90/01}  % Allows default copyright data (0-89791-88-6/97/05) to be over-ridden - IF NEED BE.

\title{Sharing Experiments Using Open Source Software}
\author{Adam Nelson, Tim Menzies, Gregory Gay\corrauth\comma\footnotemark[1]}

\longaddress{Lane Department of  Computer Science and Electrical Engineering, West Virginia University, Morgantown, WV, USA}


\corraddr{Lane Department of  Computer Science and Electrical Engineering, West Virginia Unviersity, Morgantown, WV, USA}

\footnotetext[1]{E-mail: anelson8@mix.wvu.edu, tim@menzies.us, gregoryg@csee.wvu.edu}


\cgsn{Publishing Arts Research Council}{98--1846389}

\received{1 October 2009}
\revised{?? December 2009}
\noaccepted{}

\keywords{open source, data mining}

%\titlenote{Submitted to the PROMISE 2008 workshop on repeatable SE experiments.
%%A draft version of this paper may be downloaded fro
%\url{http://menzies.us/pdf/08keys.pdf}
%}


\begin{abstract}
When researchers want to repeat, improve or refute prior conclusions, it is useful to have
a
complete and operational description of prior experiments.
If those descriptions are overly long or complex, then sharing their details may not be informative.

OURMINE is a scripting environment for the development and
deployment of data mining experiments. 
Using OURMINE,
data mining novices can specify and execute intricate experiments,  
while researchers can publish their complete experimental rig alongside their conclusions.
This is achievable because of OURMINE's succinctness. 
For example, this paper presents two experiments documented in the OURMINE syntax.
Thus, the brevity and simplicity of OURMINE recommends it as a better tool for documenting, executing, and sharing data mining experiments.
\end{abstract}

\section{Introduction}
%A recent survey article~\cite{zannier06} revealed the sad state of
%empirical software engineering. Zannier et al.~\cite{zannier06}
%sampled 5\% of the 1227 papers presented in the 29 history of the
%International Conference on Software Engineering. Of the sampled
%papers, 70\% claimed to contain some form of empirical evaluation.
%However, on closer inspection, Zannier et al. found that most (98\%
%of so-called empirical papers) used self-confirmatory studies where
%developers assessed their own methods (rather than check methods
%from other researchers).
%
%What an we do to improve the poor state-of-the-art in empirical software engineering?
%
Since 2004,  the authors have been involved in an open source data mining experiment. Researchers
submitting to the PROMISE conference on predictive models in software engineering are encouraged to offer not just conclusions,
but also the data they used to generate those conclusions. The result is nearly 100 data sets, all on-line,
freely available for download~\footnote{\url{http://promisedata.org/data}}. 

The premise of this paper is that after {\em data sharing} comes {\em experiment
sharing}; i.e.  the PROMISE repository should grow to include not just data, but the code
required to run experiments over that data. This simplifies the goal of letting other researchers 
repeat, or even extend, data mining experiments of others. Repeating experiments is not only essential 
in confirming previous results, but also in providing an inciteful understanding of the fundamentals 
involving those experiments. Thus, it is important to have a prepared set of clearly defined and operational experiments that can be utilized in order to achieve similar prior results.     

%This simplifies the goal of letting other researchers
%repeat, and maybe even improve, the experimental results of others. 
Consequently, we have been assessing data mining toolkits such as the 
WEKA\footnote{\url{http://www.cs.waikato.ac.nz/ml/weka/}} tool of \fig{weka},
``R''\footnote{\url{http://www.r-project.org/}},
the ORANGE\footnote{\url{http://magix.fri.uni-lj.si/orange/}} tool of \fig{orange},
RAPID-I\footnote{\url{http://rapid-i.com/}},
MLC++~\footnote{\url{http://www.sgi.com/tech/mlc/}}, and 
MATLAB\footnote{\url{http://www.mathworks.com}}.
We find that they are
are 
not suitable for  
publishing experiments.  
Our complaint with these tools is same as that offered by
Ritthoff et al.~\cite{ritthoff01}:
\begin{figure}
\begin{center}
\includegraphics[width=4in]{weka}
\end{center}
\caption{The WEKA toolit running the J48 decision tree learner.}\label{fig:weka}
\end{figure}
\begin{figure}
\begin{center}
\includegraphics[width=4in]{orange}
\end{center}
\caption{Orange's visual programming environment. Green lines indicate data streams passed between functions.}\label{fig:orange}
\end{figure}
\begin{figure}
\begin{center}
\includegraphics[width=2in]{yale}
\end{center}
\caption{Rapid-I's operator trees. From ~\cite{mie06}. Internally, this tree is a nested XML expression that is traversed
top-down to complete an experiment.}\label{fig:yale}
\end{figure}
\bi
\item 
Proprietary tools such as MATLAB are not freely available.
\item
 Real-world experiments
are more complex than 
running one algorithm. 
\item
Rather, such experiments or applications require {\em intricate combinations}
of a large 
number of tools that include data miners, data pre-processors and report regenerators.
\ei

%Many developers and data mining researchers have realized the need to "wire up" data mining tools in order create an interconnection between intricate combinations of tools required for publishable, large-scale experiments. These tools include data miners, data pre-processors and report generators. 

However, the need to ``wire up'' data miners within a multitude of other tools has been addressed in many ways. 
In WEKA's visual {\em knowledge flow} programming environment, for example,
nodes represent different pre-processors/ data miners/ etc while arcs represent how data flows between them.
A similar visual environment is offered by many other tools including ORANGE (see \fig{orange}).
The YALE tool (now RAPID-I) built by Ritthoff et al. formalizes these visual environments by generating
them from  XML
schemas describing
{\em operator trees} like \fig{yale}.
%The need to extend data mining tools is well established.

%The need to ``wire up'' data miners within a multitude of other tools has been addressed in many ways. 
%In WEKA's visual {\em knowledge flow} programming environment, for example,
%nodes represent different pre-processors/ data miners/ etc while arcs represent how data flows between them.
%A similar visual environment is offered by many other tools including ORANGE (see \fig{orange}).
%The YALE tool (now RAPID-I) built by Ritthoff et al. formalizes these visual environments by generating
%them from  XML
%schemas describing
%{\em operator trees} like \fig{yale}.

Ritthoff et al. argues (and we agree) that
the standard interface of (say) WEKA does not support the rapid generation of
these intricate combinations. Also, according to Menzies~\cite{me00v}, while visual systems provide motivation for beginners, they fail in providing a good explanation device for many software engineering and 
knowledge engineering problems, as many of these problems are not inherently spatial.  


In our experience these
visual environments are either
overly elaborate,
discouraging or distracting: 
\bi
\item
These standard data mining toolkits may be overly elaborate. As shown throughout this paper, very simple UNIX scripting tools suffice for the specification,
execution, and communication of experiments.
\item Some data mining novices are discouraged by the tool complexity. These novices shy away
from extensive modification and experimentation.
\item Other developers may become  so enamored with the impressive software engineering
inside these tools that they waste much time  building environments to support data mining,
but never get around to the data mining itself.
\ei
A similar experience is reported by our colleagues in WVU Department of Statistics.
In order to support teaching, they 
hide these data mining tools inside high-level scripting environments. Their scripting environments
shield the user from the complexities of ``R''
by defining high-level LISP code that, internally, calls ``R''.
While a powerful language, we find that LISP can be 
arcane to many audiences. 

OURMINE is a data mining scripting envrionment. The current kit has tools written in
BASH/ GAWK/ JAVA/ PERL/ and there is no technical block to adding
other tools written in other languages. Other toolkits impose strict limitations of the usable languages:
\bi
\item
 MLC++ requires C++  
\item
Extensions to WEKA must be written in JAVA.
\ei
Our preference  for  BASH~\cite{ramey94}/GAWK~\cite{awkbook} over, say, LISP is partially a matter of
taste but we defend that selection as follows.  Once a student
learns, for example, RAPID-I's XML configuration tricks, then those learned
skills are highly specific to that  toolkit.  On the other
hand, once a student learns BASH/GAWK methods for data pre-processing
and reporting, they can apply those scripting tricks to any number
of future UNIX-based applications.

This paper introduces OURMINE as follows:
\bi
\item First, we describe the base tool and offer some samples of coding in OURMINE;
\item Next, we demonstrate OURMINE's ability to succinctly document even complex experiments.
\ei

This paper is an extension of  a prior publication~\cite{gay09}:
\bi
\item This paper details functionality added since that last paper was published;
\item This paper adds a new experiment to the description of OURMINE (see \tion{tmine}).
\ei

\section{OURMINE}

OURMINE was developed to help graduate students at West Virginia University document and execute their 
data mining experiments.
The toolkit 
uses UNIX shell scripting. As a result, any tool
that can be executed from a command prompt can be seemlessly combined with other tools.

For example,  \fig{clean} shows a simple bash
function used in OURMINE to clean text data before conducting any
experiments using it. Line 5 passes text from a file, performing
tokenization, removing capitals and unimportant words found in a
stop list, and then in the next line performing Porter's stemming
algorithm on the result. 

OURMINE allows connectivity between tools
written in various languages as long as there is always a command-line
API available for each tool.
For example, the modules of \fig{clean} are written using BASH,
Awk and Perl.  

\begin{figure}
{\scriptsize\begin{verbatim}
 1 clean(){
 2    local docdir=$1
 3    local out=$2
 
 4   for file in $docdir/*; do
 5      cat $file | tokes | caps | stops $Lists/stops.txt > tmp 
 6	     stems tmp >> $out 
 7	     rm tmp
 8   done
 9 }\end{verbatim}}
\caption{An OURMINE function to clean text documents and collect the results. 
$Tokes$ is a tokenizer; $caps$ sends all words to lower case;
$stops$ removes the stop workds listed in "\$Lists/stops.txt"; and
$stems$ performs Porter's stemming algorithm (removes confusing suffixes).
 }\label{fig:clean}
\end{figure}

The following sections describe OURMINE's functions and applications.

\subsection{Built-in Data and Functions}

In order to encourage more experimentation, 
the default OURMINE installation comes with numerous data sets:

\bi
\item {\em Text mining data sets:} including STEP data sets (numeric): ap203, ap214, bbc, bbcsport, law, 20 Newsgroup subsets [sb-3-2, sb-8-2, ss-3-2, sl-8-2]\footnote{\url{http://mlg.ucd.ie/datasets}}
\item {\em Discrete UCI datasets:} anneal, colic, hepatitis, kr-vs-kp, mushroom, sick, waveform-5000, audiology, credit-a, glass, hypothyroid, labor, 
pcolic, sonar, vehicle, weather, autos, credit-g, heart-c, ionosphere, letter, primary-tumor, soybean, vote,\newline weather.nominal,
breast-cancer, diabetes, heart-h, iris, lymph, segment, splice, vowel;
\item {\em  Numeric UCI datasets:} auto93, baskball, cholesterol, detroit, fruitfly, longley, pbc, quake, sleep, autoHorse, bodyfat, cleveland, 
echoMonths, gascons, lowbwt, pharynx, schlvote, strike, autoMpg, bolts, cloud, elusage, housing, mbagrade, pollution, sensory, veteran, 
autoPrice, breastTumor, cpu, fishcatch, hungarian, meta, pwLinear, servo, vineyard.
\item The defect prediction data sets from {\em the PROMISE repostitory}:  CM1, KC1, KC2, KC3, MC2, MW1, PC1
\ei

OURMINE also comes with a variety of built-in functions to perform data mining and text mining tasks. For a sample of these functions, see \fig{functions}. For a complete list, see the appendix.
\begin{figure}
\small
\begin{description}
\item[abcd] provides analysis of experiments, such as {\em pd,pf,accuracy} and {\em precision} values;
%footnotetext[1]{
%If $\{a,b,c,d\}$ are the
%true negatives, 
%false negatives, 
%false positives, and 
%true positives
%(respectively) found by a binary detector then 
%$pd=recall=\frac{d}{b+d}$,
%$pf=\frac{c}{a+c}$,
%$prec=precision=\frac{d}{d+c}$,
%$acc=accuracy=\frac{a+d}{a+b+c+d}$,
%$f=\frac{2*pd*pf}{pd + pf}$,
%and 
%$balance=1-\sqrt{(1-pf)^2 + pd^2}/\sqrt{2}$ .}
\item[clean] clean text for further processing, removing tokens, capitalizations, stop words, etc.;
\item[docsToSparff] constructs a sparse arff file based on a directory of documents;
\item[docsToTfidfSparff] generates a sparse arff file of TF-IDF values based on a directory of documents;
\item[funs] shows a sorted list of all available functions in OURMINE;
\item[logArff] logs all numeric values in a data set ;
\item[malign] neatly aligns text into columns;
\item[nb] Runs Naive Bayes on the data given;
\item[rankViaInfoGain] ranks attributes by InfoGain values;
\item[makeTrainAndTest] splits a dataset into a test set and a training set as $train$.arff and $test$.arff, as well as $train$.lisp and $test$.lisp. 
\end{description}
\caption{A small sample of the available OURMINE functions. Built-in functions give the user something with which to start and begin running demos and experiments immediately. For a detailed list of available tools, please see the appendix.}\label{fig:functions}
\end{figure}


\begin{figure}
{\scriptsize\begin{verbatim}
function train() { #update counters for all words in the record 
  Docs++; 
  for(I=1;I<NF;I++) { 
    if( ++In[$I,Docs]==1) 
    Doc[$I]++ 
    Word[$I]++ 
    Words++ } 
} 
function tfidf(i) { #compute tfidf for one word 
  return Word[i]/Words*log(Docs/Doc[i]) 
} \end{verbatim}}
\caption{A GAWK implementation of TF-IDF.}\label{fig:tfidf}
\end{figure}

%\subsection{Adding/Modifying OURMINE Code}

%OURMINE uses an interpreted system, thus, it is a simple matter to extend or modify its functionality.
%For example, if a user wished to add a {\em myFunctions.sh} to be used in the environment

%\begin{itemize}
%\item{Create an executable {\em myFunctions.sh} containing any custom scripts}
%\item{Include this file in {\em \$HOME/opt/ ourmine/our/lib/sh/minerc.sh }}
%\end{itemize}

%Once this basic structure of OURMINE is learned, however, more advanced additions can be made such as:
%including multiple directories of BASH files, some of whose functions call code written in other files and other languages located elsewhere in the codebase. This gives a nearly limitless opportunity for customization.

\subsection{Learning and Teaching with OURMINE}
 
Data mining concepts become complex when implemented in a complex manner.
For this reason, OURMINE utilizes
simple scripting tools (written mostly in BASH or GAWK) to
better convey the inner-workings of these concepts. For instance,
\fig{tfidf} shows a GAWK implementation used by OURMINE to determine
the TF-IDF~\cite{ramos03} values of each term in a document. This
script is simple and concise, while a C++ or Java implementation
would be large and overly complex. An additional example demonstrating the brevity 
of OURMINE script can be seen in \fig{demo004}, which is a complete experiment whose  
form can easily be taught and duplicated in future experiments.

\begin{figure}
{\scriptsize\begin{verbatim}
#naive bayes classifier in gawk
#usage:  gawk -F, -f nbc.awk Pass=1 train.csv Pass=2 test.csv

Pass==1 {train()}
Pass==2 {print $NF "|" classify()}

function train(    i,h) { 
   Total++;
   h=$NF;     # the hypotheis is in the last column
   H[h]++;    # remember how often we have seen "h"
   for(i=1;i<=NF;i++) {
     if ($i=="?") 
           continue;         # skip unknown values
     Freq[h,i,$i]++
     if (++Seen[i,$i]==1) 
           Attr[i]++}  # remember unique values
}
function classify(         i,temp,what,like,h) {  
   like = -100000;         # smaller than any log
   for(h in H) {           # for every hypothesis, do...  
     temp=log(H[h]/Total); # logs stop numeric errors
     for(i=1;i<NF;i++) {  
       if ( $i=="?" ) 
             continue;     # skip unknwon values
       temp += log((Freq[h,i,$i]+1)/(H[h]+Attr[NF])) }
     if ( temp >= like ) { #  better hypothesis
        like = temp
        what=h}
   }
   return what;
}\end{verbatim}}
\caption{A Naive Bayes classifier for a CSV file, where the class label is found in the last column.}\label{fig:nbc}
\end{figure}
\begin{figure}
\small
\be
\item Modify \fig{nbc} so that there is no train/test data. Instead, make it an incremental learning.
Hint: 1) call the functions $train$, then $classify$ on every line of input. 2) The order is important: always
$train$ before $classifying$ so the the results are always on unseen data.
\item Convert \fig{nbc} into HYPERPIPES~\cite{Eisenstein04}. Hint: 1) add globals $Max[h,i]$ and $Min[h,i]$ to 
keep the max/min values seen in every column ``$i$''and every hypothesis class ``$h$''. 2) 
Test instance belongs to the class that most overlaps the attributes in the test instance.
So, for all attributes in the test set, sum the returned values from $contains1$:
{\scriptsize\begin{verbatim}
function contains1(h,i,val,numerip) {
  if(numericp)
      return Max[h,i] >= val && Min[h,i] <= val
  else return (h,i,value) in Seen
  }\end{verbatim}}
\item Use \fig{nbc} for anomaly detector. Hint:  1)~make all training examples get the same class;
2) an anomalous test instance has a likelihood $\frac{1}{\alpha}$ of the mean likelihood
seen during training ($alpha$ needs tuning but $alpha=50$ is often useful).
\item Using your solution to \#1, create an  incremental version of HYPERPIPES and a anomaly detector.
\ee
\caption{Four Introductory OURMINE programming exercises.}\label{fig:ourmine101}
\end{figure}


%\begin{figure}
%\begin{center}
%\includegraphics[width=5.5in]{edits2.ps}
%\end{center}
%\caption{Editing ourmine}\label{fig:editscreen}
%\end{figure}


Another reason to prefer scripting in OURMINE over the complexity of 
\mbox{RAPID-I}, WEKA, ``R'', etc,
is that it reveals
the inherent
simplicity of many of our data mining methods. For example, \fig{nbc} shows a GAWK implementation of
a Naive Bayes classifier for discrete data where the last column stores the class symbol.
This tiny script  is no mere toy- it successfully executes on very large
datasets such as those seen in the 2001 KDD cup and in~\cite{orrego04}. WEKA cannot process these large
data sets since it always loads its data into RAM. \fig{nbc}, on the other hand, only requires
memory enough to store one instance as well as the frequency counts in the hash table ``$F$''.

More importantly, in terms of teaching, \fig{nbc} is easily customizable. 
\fig{ourmine101} shows four warm-up exercises for novice data miners that (a)~introduce them
to basic data mining concepts and (b)~show them how easy it is to script their own data miner:
Each of these tasks requires changes to less than 10 lines from \fig{nbc}. 
The simplicity of these customizations fosters a spirit
of ``this is easy'' for novice 
data miners. This in turn empowers them to design their own extensive and elaborate experiments. 


Also from the teaching perspective, demonstrating on-the-fly a particular data mining concept helps not only to solidify this concept, but also gets the student accustomed to using OURMINE as a tool in a data mining course. As an example, if a Naive Bayes classifier is introduced as a topic in the class, an instructor can show the workings of the classifier by hand, and then immediately afterwords compliment this by running Naive Bayes on a small data set in OURMINE. Also, since most of OURMINE does not use pre-compiled code, an instructor can make live changes to the scripts and quickly show the results.%, as in \fig{editscreen}. 

%\begin{figure}
%\begin{center}
%\includegraphics[width=4in]{edits}
%\end{center}
%\caption{The image above depicts running $demo003$ in OURMINE, which then calls $demo002$. The first 
%execution in the top window demonstrates running the J48 decision tree classifier on the UCI $iris$ data set. Then, in the bottom window, the data set is changed to $weather$. Once any modification is made to the code of OURMINE, the environment can be reloaded without exiting. This is done by entering $reload$, as seen in the above window. Finally, the demo is executed again, yielding a different result.
%}
%\label{fig:editscreen}
%\end{figure}

We are not alone in favoring GAWK for teaching purposes.
Ronald Loui uses GAWK to teaching artifical intelligence
at Washington University in St. Louis. He writes:

\begin{quote}{\em
There is no issue of user-interface. 
This forces the programmer to return to the question of what the program does, not how it looks. 
There is no time spent programming a binsort when the data can be shipped to /bin/sort in no time.} ~\cite{gawkai}
\end{quote}

%Another reason for OURMINE's scripting approach is due to its applicable generality. If a student learns a highly specific portion of a toolkit, those learned skills are generally confined to that environment. However, skills obtained from learning to use scripting within OURMINE can be used in a variety of future applications.

\begin{figure}
{\scriptsize\begin{verbatim}
Function: j4810
Arguments: <data (arff)>
Example(s): j4810 weather.arff
Description: Uses a j48 decision tree learner on the input data

Function Code:
==============
j4810 () { 
    local learner=weka.classifiers.trees.J48
    $Weka $learner -C 0.25 -M 2 -i -t $1
} \end{verbatim}}
\caption{Function help in OURMINE.}\label{fig:help}
\end{figure}
 
\begin{figure}
{\scriptsize\begin{verbatim}
1 demo004(){
2     local out=$Save/demo004-results.csv
3      [ -f $out ] && echo "Caution: File exists!" || demo004worker $out
4 }

5 # run learners and perform analysis
6 demo004worker(){

7     local learners="nb j48"
8     local data="$Data/discrete/iris.arff"
9     local bins=10
10    local runs=5
11    local out=$1
    
12    cd $Tmp
13    (echo "#data,run,bin,learner,goal,a,b,c,d,acc,pd,pf,prec,bal"
14	   for((run=1;run<=$runs;run++)); do
15	       for dat in $data; do
		
16		          blab "data=`basename $dat`,run=$run" 
17		          for((bin=1;bin<=$bins;bin++)); do
		    
18		              rm -rf test.lisp test.arff train.lisp train.arff
19		              makeTrainAndTest $dat $bin $bin 
20		              goals=`cat $dat | getClasses --brief`
		    
21		              for learner in $learners; do
			
22			                  $learner train.arff test.arff | gotwant > produced.dat
23			                  for goal in $goals; do
			    
24			                  cat produced.dat | 
25			                  abcd --prefix "`basename $dat`,$run,$bin,$learner,$goal" \
26				                      --goal "$goal" \
27				                      --decimals 1
28			                  done
29		              done
30		          done
31		      blabln
32	      done
33	  done | sort -t, -r -n -k 11,11) | malign  > $out
    
34   winLossTie --input $out --test w --fields 14 --key 4 --perform 11 
35  }
\end{verbatim}}
\caption{A demo OURMINE experiment. This worker function begins by being called by the top level function {\em demo004} on lines 1-4. Noteworthy sections of the demo code are at: line 19, where training sets and test sets are built from 90\% and 10\% of the data respectively, lines 25-27 in which values such as {\em pd,pf} and {\em balance} are computed via the {\em abcd} function that computes values from the confusion matrix, and line 34 in which a {\em Wilcoxon} test is performed on each learner in the experiment using {\em pd} as the performance measure.}
\label{fig:demo004}
\end{figure}

Function documentation provides a way for newcomers to OURMINE to not only get to know the workings of each function, but also add to and modify the current documentation. Instead of asking the user to implement a more complicated ``man page'', OURMINE uses a very simple system consisting of keywords such as {\em name, args} and {\em eg} to represent a function name, its arguments and an example of how to use it. Using this documentation is simple. Entering {\em funs} at the OURMINE prompt provides a sorted list of all available functions in ourmine. Then, by typing {\em help X}, where {\em X} is the name of the function, information about that function is printed to the screen. See \fig{help} for an example of viewing the help document for the function {\em j4810}. Documentation for a function is added by supplying a text file to the helpdocs directory in OURMINE named after the function.

%From the teaching perspective, demonstrating on-the-fly a particular data mining concept helps not only to solidify this concept, but also gets the student accustomed to using OURMINE as a tool in the course. 
%\fig{demo004} shows the canonical OURMINE example. XXX more details.
%Once a student learns the basic mechanics of this demo experiment, its framework can be copied to another function within OURMINE to conduct a custom experiment using any number of tools.

\section{Using Ourmine for Research}

OURMINE is not just a simple demonstration system for novice data miners.
It can also be used to generate journal-level publishable results. 
In the last three years, the authors of this paper have published six papers in leading software engineering
journals and conferences using
OURMINE~\cite{turhan08,gay09,me07b,me06d,me05c,me05d}.

In order to demonstate OURMINE's use in leading edge research,
we present here two experiments.
Experiment \#1 reproduces an important recent result by Turhan et al.~\cite{turhan08}.
Kitchenham et al. report that there is no clear consensus on the relative value of effort estimation
models learned from either local or imported data~\cite{kitch07}. Since this matter has not be previously
studied in the defect prediction literature, Turhan et al. conducted a series of experiments with ten data sets
from different companies. In their {\em within-company} (WC) experiments, defect predictors were learned
from 90\%  of the local data, then tested on the remaining 10\%. In the {\em cross-company} (CC) experiments,
defect predictors wehre learned from nine companies then tested on the tenth. Turhan et al. found that the CC
predictions were useless since they suffered from very high false alarm rates. However, after applying a simple
{\em relevancy filter},
they found that imported CC learning yielded predictors that were nearly as good as the local
WC learning. 

Experiment \#2 checks a recent text mining result from an as-yet-unpublished WVU masters thesis.
Matheny~\cite{matheny09} benchmarked various lightweight learning methods (TF*IDF, the GENIC stochastic clusterer)
against other, slower, more rigorous learning methods (PCA, K-means). As expected, the rigorous learning methods
ran  much slower than the stochastic methods. But, unexpectedly, Matheny found that the leightweight methods perform
nearly as well as the rigorous methods.

In a mature scientific discipline, it is standard practice to reproduce  important results. Both the results
need to be reproduced since they are important:
\bi
\item If the Experiment \#1 results are correct, then organizations make quality predictions about their source code,
even when they lack local historical logs.
\item If the Experiment \#2 results are correct, then text mining methods can scale to much larger data sets.
\ei
The rest of this section describes the use of OURMINE to reproduce Experiment \#1 and \#2.

 
%The first experiment, based on those conducted by Turhan et
%al.~\cite{turhan08}, as well as Gay et al.~\cite{gay09}, focuses
%on binary defect prediction. In~\cite{turhan08}, Turhan et al.
%conducted three experiments to rule in favor of cross-company (CC)
%data obtained from other sites, or within-company (WC) data gathered
%locally. The conclusions of those experiments show that CC data,
%when applied using {\em relevancy filtering}, as explained below,
%can lead to defect predictors almost as effective as WC data. Thus,
%as stated by Gay et al., ``...while local data is the preferred
%option, it is feasible to use imported data provided that it is
%selected by a relevancy filter.''
%
%The second experiment was a small-scale reproduction of that conducted
%by a graduate computer science student at West Virginia University
%to be used in a Master's thesis. The purpose of the experiment is
%to show that faster heuristic means of clustering and dimensionality
%reduction yield results comparable to slower, more rigorous methods
%when examining these methods' runtimes, cluster similarities, as
%well as cluster purities.
%
%In the following sections we describe the experiments and show some of the scripts used to conduct them. Finally we analyze the results.
%
\subsection{Experiment I}

OURMINE was used to reproduce Turhan et al.'s experiment - with a Naive Bayes classifier in conjunction with a k-Nearest Neighbor (k-NN) relevancy filter. Relevancy filtering is used to group similar instances together in order to obtain a learning set that is homogeneous with the testing set. Thus, by using a training set that shares similar characteristics with the testing set, it is assumed that a bias in the model will be introduced. The k-NN filter works as follows.
For each instance in the test set:
\bi
\item the k nearest neighbors in the training set are chosen. 
\item duplicates are removed 
\item the remaining instances are used as the new training set.
\ei

\subsubsection{Building the Experiment}

The entire script used to conduct this experiment is shown in \fig{promiseExp}. 

To begin, the data in this study were the same as used by Gay et al.; seven PROMISE defect data sets (CM1, KC1, KC2, KC3, MC2, MW1, PC1) were used to build seven combined data sets each containing $\frac{6}{7}$-th of the data. For instance, the file {\em combined\_PC1.arff} contains all seven data sets {\em except} PC1, which is used as the training set for the cross-company (CC) data.

Next, as can be seen in line 15 of \fig{promiseExp}, a 10-way cross validation was conducted by calling $makeTrainAndTest$, which is a built-in OURMINE function that randomly shuffles the data and constructs both a test set, containing 10\% of the data, and a training set containing 90\% of the data. This was repeated ten times, and the resulting data was used in proceeding studies. For instance, in lines 18-24, the original test and training sets are used for the first WC study. However, in the WC experiment using relevancy filtering (lines 25-31), the same test set is used, but with the newly created training set. Lines 32-38 show the first CC study. This study is identical to the WC study except that as we saw before, we use $combined\_X.arff$ files, instead of $shared\_X.arff$.

We chose to use a Naive Bayes classifier for this study because this is what was chosen in the original experiment conducted by Turhan et al. in ~\cite{turhan08}, as well as because Naive Bayes has been shown to perform better on PROMISE defect data sets than other learners~\cite{lessmann08}.   

\begin{figure}
{\scriptsize\begin{verbatim}
1 promiseDefectFilterExp(){
2  local learners="nb"
3  local datanames="CM1 KC1 KC2 KC3 MC2 MW1 PC1"
4  local bins=10
5  local runs=10
6  local out=$Save/defects.csv
7  for((run=1;run<=$runs;run++)); do
8    for dat in $datanames; do
9	       combined=$Data/promise/combined_$dat.arff
10	       shared=$Data/promise/shared_$dat.arff
11	       blabln "data=$dat run=$run" 
12	       for((bin=1;bin<=$bins;bin++)); do
              rm -rf test.lisp test.arff train.lisp train.arff

13	           cat $shared | 
14	           logArff 0.0001 "1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19" > logged.arff
15	           makeTrainAndTest logged.arff $bins $bin
16	           goals=`cat $shared | getClasses --brief`

17	           for learner in $learners; do          
18		            blabln "WC"
19		            $learner train_shared.arff test_shared.arff | gotwant > produced.dat
20		            for goal in $goals; do
21		                extractGoals goal "$dat,$run,$bin,WC,$learner,$goal" `pwd`/produced.dat
24		            done
25		            blabln "WCkNN"
26		            rm -rf knn.arff
27		            $Clusterers -knn 10 test_shared.arff train_shared.arff knn.arff
28		            $learner knn.arff test_shared.arff | gotwant > produced.dat
29		            for goal in $goals; do
30		               extractGoals goal "$dat,$run,$bin,WCkNN,$learner,$goal" `pwd`/produced.dat                          
31             done
32		            blabln "CC"
33		            makeTrainCombined $combined > com.arff	
34		            cat com.arff | logArff 0.0001 "1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19" > logged.arff
35		            $learner logged.arff test_shared.arff | gotwant > produced.dat
36		            for goal in $goals; do
37		               extractGoals goal "$dat,$run,$bin,CC,$learner,$goal" `pwd`/produced.dat
38             done
39             blabln "CkNN"             
40             makeTrainCombined $combined > com.arff  
41             cat com.arff |
42             logArff 0.0001 "1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19" > logged.arff
43             $Clusterers -knn 10 test_shared.arff logged.arff knn.arff
44             $learner knn.arff test_shared.arff | gotwant > produced.dat
45             for goal in $goals; do
46                extractGoals goal "$dat,$run,$bin,CkNN,$learner,$goal" `pwd`/produced.dat
47             done
48    done
49  done
50  blabln
51  done
52  done ) | malign | sort -t, -r -n -k 12,12 > $out
\end{verbatim}}
\caption{The OURMINE script used in conducting the WC vs. CC experiment.}\label{fig:promiseExp}
\end{figure}

\subsubsection{Results}

Our results for this experiment can be found in \fig{pds} and \fig{pfs}. \fig{pds} shows $pd$ (probability of detection) values sorted in decreasing order, while \fig{pfs} shows $pf$ (probability of false alarm) values sorted in increasing order. These values are calculated as follows.
If $\{a,b,c,d\}$ are the
true negatives, 
false negatives, 
false positives, and 
true positives
(respectively) found by a binary detector then 
$pd=recall=\frac{d}{b+d}$ and
$pf=\frac{c}{a+c}$.
Note that a higher $pd$ is better, while a lower $pf$ is better. 

The last column of each figure shows quartile charts of the methods' $pd$ and $pf$ values. The black dot in the center of each plot represents the median value, and the line going from left to right from this dot show the second and third quartile respectively.

Column one of each figure gives a method its rank based on a Mann-Whitney test at 95\% confidence. A rank is determined by how many times a learner or learner/filter loses compared to another. The method that lost the least number of times is given the highest rank. 

%\begin{figure}
%\small
%\begin{center}
%\begin{tabular}{l@{~}|l@{~}|r@{~}r@{~}@{~}r@{~}|c}
%\multicolumn{2}{c}{~}&\multicolumn{5}{c}{quartiles}\\\cline{3-5}
%&min& & median & &max\\\cline{3-5}
%\multicolumn{2}{c}{~}&\multicolumn{3}{c}{pd}&2nd quartile\\
%\multicolumn{2}{c}{~}&\multicolumn{3}{c}{percentiles}&median,\\\cline{3-5}
%Rank&Treatment& 25\%& 50\% & 75\%&3rd quartile\\\hline
%1&WC (local data) + relevancy filter& 66& 73& 80&  \boxplot{0}{66}{73}{80}{3}  \\
%2&CC (imported data) + relevancy filter& 57& 71& 83&  \boxplot{0}{57}{71}{83}{5}  \\
%2&WC (local data)  & 59& 69& 83&  \boxplot{0}{59}{69}{83}{7}  \\
%3&CC (imported data)  & 49& 66& 87&  \boxplot{0}{49}{66}{87}{7}   \\\hline
%\multicolumn{5}{c}{~}&~~~~~0~~~~~~~~50~~~~100
%\end{tabular}
%\end{center}
%\caption{Probability of Detection (PD) results, sorted by median values.  }\label{fig:pds}
%\end{figure}
%\begin{figure}
%\small
%\begin{center}
%\begin{tabular}{l@{~}|l@{~}|r@{~}r@{~}@{~}r@{~}|c}
%%\multicolumn{2}{c}{~}&\multicolumn{5}{c}{quartiles}\\\cline{3-5}
%&min& & median & &max\\\cline{3-5}
%\multicolumn{2}{c}{~}&\multicolumn{3}{c}{pf}&2nd quartile\\
%\multicolumn{2}{c}{~}&\multicolumn{3}{c}{percentiles}&median,\\\cline{3-5}
%Rank&Treatment& 25\%& 50\% & 75\%&3rd quartile\\\hline
%1&CC (imported data) + relevancy filter& 17& 29& 43&  \boxplot{0}{17}{29}{43}{3}  \\
%1&CC (imported data)  & 13& 34& 51&  \boxplot{0}{13}{34}{51}{5} \\
%2&WC (local data)  & 17& 30& 41&  \boxplot{0}{17}{30}{41}{7}  \\
%3&WC (local data) + relevancy filtuer & 20& 27& 34&  \boxplot{0}{20}{27}{34}{8}  \\\hline
%\multicolumn{5}{c}{~}&~~~~~0~~~~~~~~50~~~~100
%\end{tabular}
%\end{center}
%\caption{Probability of False Alarm (PF) results, sorted by median values.  }\label{fig:pfs} \end{figure}

The following are important conclusions derived from these results:

\begin{itemize}
\item 
When CC data is used, relevancy filtering is crucial. According to our results, cross-company data with no 
filtering yields the worst $pd$ and $pf$ values.
\item{When relevancy filtering is performed on this CC data, we obtain better $pd$ and $pf$ results than using just CC and Naive Bayes.}
\item{When considering only filtered data or only unfiltered data, the highest $pd$ and lowest $pf$ values are obtained by using WC data as opposed to CC data. This suggests that WC data gives the best results.}
\end{itemize}

These finds were consistent with Turhan et al.'s results: 

\begin{itemize}
\item{Significantly better defect predictors are produced from using WC data.}
\item{However, CC data leads to defect predictors nearly as effective as WC data when using relevancy filtering.}
\end{itemize}

Thus, this study also makes the same conclusions as Turhan et al. A company should use local data to develop defect predictors if that local development data is available. However, if local data is not available, relevancy-filtered cross-company data provides a feasible means to build defect predictors.


\subsection{Experiment II}\label{sec:tmine}

As stated above, the purpose of this experiment conducted for this paper is to verify if lightweight data mining
methods perform slower or worse than more thorough and rigorous ones.

The data sets used in this experiment are:

\begin{itemize}
\item EXPRESS schemas: AP-203, AP-214
\item Text mining datasets: BBC, Reuters, The Guardian (multi-view text datasets), 20 Newsgroup subsets:
sb-3-2, sb-8-2, ss-3-2, sl-8-2\footnotetext[1]{\url{http://mlg.ucd.ie/datasets}}.
\end{itemize}

\subsubsection{Classes of Methods }

This experiment compares different {\em row} and {\em column} reduction methods. Given a table of data where
each row is one example and each columns counts different features, then:
\bi
\item
Row reduction methods {\em cluster} related rows into the same group;
\item
Column reduction methods remove columns with little information.
\ei

Reduction methods are essential in text mining. For example:
\bi
\item
 A standard text mining corpus may store information
in tens of thousands of columns. For such data sets, column reduction is an essential first step before any other
algorithm can execute

%\begin{quote}
%  One of the problems with high-dimensional datasets is that, in many cases, 
%not all the measured variables are ``important'' for understanding the underlying 
%phenomena of interest.
%\end{quote}
\item
The process of clustering data into similar groups can be used in a wide variety of applications, such as:
\begin{itemize}
  \item Marketing: finding groups of customers with similar behaviors given a large database of customer data
  \item Biology: classification of plants and animals given their features
  \item WWW: document classification and clustering weblog data to discover groups of similar access patterns\footnotetext[2]{\url{http://home.dei.polimi.it/matteucc/Clustering/tutorial_html/}}.
\end{itemize}
\ei

%Thus the purpose of clustering is to ``determine the intrinsic grouping in a set of unlabeled data'' 
%XXXX adam. what does this do? \footnotemark[\value{footnote}]. 

%XXXX adam remove the following?
%Data can sometimes be overwhelmingly large, containing a great number of attributes, or dimensions. In order to reduce this vast multidimensional space, there exist dimensionality reduction methods. Dimensionality reduction filters the attributes, and attempts to select the most valid ones. In ``A Survey of Dimension Reduction Techniques'' \footnotetext[3]{\url{https://e-reports-ext.llnl.gov/pdf/240921.pdf}} by I.K. Fodor, Fodor writes:
%XXX make it a reference


\subsubsection{The Algorithms}

While there are many clustering algorithms used today, this experiment focused on three: a naive K-Means implementation, GenIc~\cite{genic04}, and clustering using canopies~\cite{canopies00}:
\be
\item
K-means, a special case of a class of EM algorithms, works as follows:
\begin{enumerate}
\item{Select initial {\em K} centroids at random;}
\item{Assign each incoming point to its nearest centroid;}
\item{Adjusts each cluster's centroid to the mean of each cluster;}
\item{Repeat steps 2 and 3 until the centroids in all clusters stop moving by a noteworthy amount}
\end{enumerate}
Here we use a naive implementation of K-means, requiring {\em K}*{\em N }*{\em I} comparisons, where {\em N} and {\em I} represent the total number of points and maximum iterations respectively.
\item
GenIc is a single-pass, stochastic clustering algorithm. It begins by intially selecting {\em K} centroids at random from all instances in the data. At the beginning of each generation, set the centroid weight to one. When new instances arrive, nudge the nearest centroid to that instance and increase the score for that centroid. In this process, centroids become ``fatter'' and slow down the rate at which they move toward newer examples. When a generation ends, replace the centroids with less than {\em X} percent of the max weight with {\em N} more random centroids. Genic
repeats for many generations, then returns the highest scoring centroids.
\item
Canopy clustering, developed by Google, reduces the need for comparing all items in the data using an expensive distance measure, by first partitioning the data into overlapping subsets called {\em canopies}. Canopies are first built using a cheap, approximate distance measure. Then, more expensive distance measures are used inside of each canopy to cluster the data. 
\ee
As to column reduction, we will focus on two methods:

\be
\item
PCA, or Principal Components Analysis, is a reduction method that
treats every instance in a dataset as a point in N-dimensional
space. PCA looks for new dimensions that better fit these points.
PCA involves the calculation of the eigenvalue decomposition of a
data covariance matrix or singular value decomposition of a data
matrix.
\fig{pca} shows an example of PCA.  Before, on the left-hand-side, the data exists in a two-dimensional
space, neither of which captures the distribution of the data. Afterwards, on the right-hand-side, a new dimension
has been syntethesized that is more relevant to the data distribution.
\begin{figure}
\begin{center}
\includegraphics[width=2.25in]{pca}
\end{center}
\caption{A PCA dimension feature.}\label{fig:pca}
\end{figure}
\item
TF-IDF, or term frequency times inverse document frequency, reduces the number of terms (dimensions) by describing how important a term is in a document (or collection of documents) by incrementing its importance according to how many times the term appears in a document. However, this importance is also offset by the frequency of the term in the entire corpus. Thus, we are concerned with only terms that occur frequently in a small set of documents, and very infrequently everywhere else. To calculate the Tf*IDF value for each term in a document, we use the following equation:
\begin{equation} 
Tf*df(t,D_j) = \frac{tf(t_i,D_j)}{|D_j|}log(\frac{|D|}{df(t_i)})
\end{equation}
To reduce all terms (and thus, dimensions), we must find the sum of the above
\begin{equation}
Tf*Ifd_{sum}(t) = \sum_{D_j\epsilon D}Tf*Idf(t,D_j)
\end{equation}
\ee
In theory, TF*IDF and GenIc should perform worse than K-Means, canopy clustering and PCA:
\bi
\item
Any single-pass algorithm  like GenIc
can be confused by ``order effects''; i.e. if the data arrives in some confusing order then the
single-pass algorithm can perform worse than other algorithms that are allowed to examine all the data. 
\item
TF*IDF is a heuristic method while PCA is a well-founded mathematical technique
\ei
On the other hand, the more rigorous methods are slower to compute:
\bi
\item Computing the correlation matrix used by PCA requires at least a $O(N^2)$ calculation.
\item As shown below, K-means is much slower than the other methods studied here.
\ei

\subsubsection{Building the Experiment}

This experiment was conducted entirely with OURMINE using a collection of BASH scripts, as well as custom Java code. The framework was built as follows:

\begin{enumerate}
\item{A command-line API was developed in Java for parsing the data, reducing/clustering the data, and outputting the data. Java was chosen due to its preferred speed for the execution of computationaly expensive instructions.}
\item{The data was then iteratively loaded into this Java code via shell scripting. This provides many freedoms, such as allowing parameters to be altered as desired, as well as outputting any experimental results in any manner seen fit.}
\end{enumerate}


\fig{clusterworker} shows the OURMINE code for clustering data using the K-means algorithm. Shell scripting provides us with much leverage in this example. For instance, by looking at Lines 2-5, we can see that by passing the function four parameters, we can cluster data in the range from {\em minK} to {\em maxK} on all data in {\em dataDir}. This was a powerful feature used in this experiment, because it provides the oppurtunity to run the clusterer across multiple machines simultaneously. As a small example, suppose we wish to run K-means across three different machines with a minimum {\em K} of 2 and a maximum {\em K} of 256. Since larger values of {\em K} generally yield longer runtimes, we may wish to distribute the execution as follows:

\begin{verbatim}
Machine 1: clusterKmeansWorker 256 256 0 dataDir
Machine 2: clusterKmeansWorker 64 128 2 dataDir
Machine 3: clusterKmeansWorker 2 32 2 dataDir
\end{verbatim}

Lines 9-13 of \fig{clusterworker} load the data from {\em dataDir} for every {\em k}, and formats the name of the output file. Then, lines 15-19 begin the timer, cluster the data, and output statistical information such as {\em k}, the dataset, and runtime of the clusterer on that data set. This file will then be used later in the analysis of these clusters.

\begin{figure}
{\scriptsize\begin{verbatim}
1 clusterKmeansWorker(){
2     local minK=$1
3     local maxK=$2
4     local incVal=$3
5     local dataDir=$4
6     local stats="clusterer,k,dataset,time(seconds)"
7     local statsfile=$Save/kmeans_runtimes
8     echo $stats >> $statsfile
9     for((k=$minK;k<=$maxK;k*=$incVal)); do
10         for file in $dataDir/*.arff; do
11             filename=`basename $file`
12             filename=${filename%.*}
13             out=kmeans_k="$k"_$filename.arff 
14             echo $out
15             start=$(date +%s.%N)
16             $Clusterers -k $k $file $Save/$out
17             end=$(date +%s.%N)
18             time=$(echo "$end - $start" | bc)
19             echo "kmeans,$k,$filename,$time" >> $statsfile
20         done
21     done
22 }\end{verbatim}}
\caption{An OURMINE worker function to cluster data using the K-means algorithm. Note that experiments using other clustering methods (such as GenIc and Canopy), could be conducted by calling line 16 above in much the same way, but with varying flags to represent the clusterer.}\label{fig:clusterworker}
\end{figure}

Similarly, the flags in line 16 can be changed to perform a different action, such as clustering using GenIc or Canopy, by changing {\em -k} to {\em -g} or {\em -c} respectively, as well as finding cluster similarities (as described below) and purities, by using {\em -sim} and {\em -purity} as inputs. 


Since any number of variables can be set to represent different libraries elsewhere in OURMINE, the variable \begin{verbatim} $Reducers \end{verbatim} is used for the dimensionality reduction of the raw dataset, as seen in \fig{tfidfworker}, whose overall structure is very similar to \fig{clusterworker}.

\begin{figure}
{\scriptsize\begin{verbatim}
1 reduceWorkerTfidf(){
2     local datadir=$1
3     local minN=$2
4     local maxN=$3
5     local incVal=$4
6     local outdir=$5
7     local runtimes=$outdir/tfidf_runtimes

8    for((n=$minN;n<=$maxN;n+=$incVal)); do
9        for file in $datadir/*.arff; do
10            out=`basename $file`
11            out=${out%.*}
12            dataset=$out
13            out=tfidf_n="$n"_$out.arff
14            echo $out
15            start=$(date +%s)
16            $Reducers -tfidf $file $n $outdir/$out
17            end=$(date +%s)
18            time=$((end - start))
19            echo "tfidf,$n,$dataset,$time" >> $runtimes
20        done
21    done
22 }\end{verbatim}}
\caption{An OURMINE worker function to reduce the data using TF-IDF.}\label{fig:tfidfworker}
\end{figure}

\subsection{Results}   

To determine the overall benefits of each clustering method, this experiment used both cluster similarities, as well as the runtimes of each method.

%\subsubsection{Purities}
%When we cluster data that already has classes available to us, we can determine cluster purity. A cluster is considered relatively ``pure'' if it contains a high percentage of points that share the same class. Thus, we can say that this is a measure of the ``{\em quality} of a clustering solution''\footnotetext[1]{\url{http://www.cse.iitm.ac.in/~cs672/purity.pdf}}. 
%To determine the purity of individual clusters, the equation
%\begin{equation}
%purity(C_a)=\frac{1}{|C_a|}*max(|C_a|_{cl=b})
%\end{equation}
%was used, where a cluster $C_a$  contains $|C_a|_{cl=b}$ number of points assigned to class $b$. However, a high purity using this equation always is easy to obtain. For instance, if a cluster contains only one instance from the dataset, the purity of that cluster is always 1. Therefore, this study uses a weighted sum of these purities
%\begin{equation}
%purity=\sum_{a=1}^k\frac{|C_a|}{|D|}*purity(C_a)
%\end{equation}
%where $|D|$ denotes the number of documents in the dataset.

%To determine cluster purities, the following text datasets were used in conjunction with their natural classes:
%\begin{itemize}
%\item BBC: 5 natural classes (business, entertainment, politics, sport, tech);
%\item BBCSPORT: 5 natural classes (athletics, cricket, football, rugby, tennis\footnotetext[1]{\url{http://mlg.ucd.ie/datasets/bbc.html}};
%\end{itemize}
%The results of the cluster purity experiments are shown in \fig{pur1} and \fig{pur2}. By examining these results, we can see that the more complete methods (in this case K-means) yields the highest weighted purity. However, since each data set and its class distribution can drastically alter the outcome of these values, it is sometimes beneficial to use fast heuristic methods, as seen in the results for the BBC data set.    

%XXX more discussion

%\begin{figure}
%\begin{center}
%\includegraphics[width=5.50in]{bbc_purities}
%\end{center}
%\caption{Normalized weighted purity sums of clusters on the BBC data set.}\label{fig:pur1}

%\end{figure}

%\begin{figure}
%\begin{center}
%\includegraphics[width=5.50in]{bbcsport_purities}
%\end{center}
%\caption{Normalized weighted purity sums of clusters on the BBCSport data set.}
%\label{fig:pur2}
%\end{figure}

\subsubsection{Similarities}

Cluster similarities tell us how similar points are, either within a cluster ({\em Intra}-similarity), or with members of other clusters ({\em Inter}-similarity). The idea here is simple: gauge how well a clustering algorithm groups similar documents, and how well it separates different documents. Therefore, intra-cluster similarity values should be maximized, while minimizing inter-cluster similarities. 

Similarities are obtained by using the cosine similarity between two documents. The cosine similarity measure defines the cosine of the angle between two documents, each containing vectors of terms. The similarity measure is represented as

\begin{equation}
sim(D_i,D_j) = \frac{D_i \cdot D_j}{||D_i|| ||D_j||} = cos(\theta)
\end{equation}

where $D_i$ and $D_j$ denote two specific documents. 

 Cluster similarities are determined as follows:

\begin{itemize}
\item{Cluster intra-similarity: For each document $d$ in cluster $C_i$, find the cosine similarity between $d$ and all documents belonging to $C_i$}
\item{Cluster inter-similarity: For each document $d$ in cluster $C_i$, find the cosine similarity between $d$ and all documents belonging to all other clusters}
\end{itemize}

Thus the resulting sum of these values represents the overall similarities of a clustering solution. Figure \textcolor{red}{17} shows the results from the similarity tests conducted in this experiment. The slowest clustering and reduction methods were set as a baseline, because it was assumed that these methods would perform the best. With intra-similarity and inter-similarity values normalized to 100 and 0 respectively, we can see that surprisingly, faster heuristic clustering and reduction methods perform just as well or better than more rigorous methods. Thus, the conclusions from this experiment shows that fast heuristic methods are sufficient for large data sets due to their scalability.   
%\begin{figure}
%\centering
%\small
%\begin{tabular}{| c | c | c | c | c |}
%\hline
%Reducer and Clusterer & Time & InterSim & IntraSim & Gain \\ \hline
%TF-IDF*K-means & 17.52 & -0.085 & 141.73 & 141.82 \\ \hline
%TF-IDF*GenIc & 3.75 & -0.14 & 141.22 & 141.36 \\ \hline
%PCA*K-means & 100.0 & 0.0 & 100.0 & 100.0 \\ \hline
%PCA*Canopy & 117.49 & 0.00 & 99.87 & 99.87 \\ \hline
%PCA*GenIc & 11.71 & -0.07 & 99.74 & 99.81 \\ \hline
%TF-IDF*Canopy & 6.58 & 5.02 & 93.42 & 88.4 \\ \hline
%\end{tabular}
%\caption{Similarity values normalized according to the combination of most rigorous reducer and clusterer.} 
%\label{fig:sims}
%\end{figure}

%\section{Related Work}

%\newpage
\section{Conclusions}
The next challenge in the empirical SE community will be to not only share data, but
to share experiments.
We look forward to the day when  
it is routine for conference and journal submissions to come not just with supporting data but
also with a fully executable version of the experimental rig used in the paper.
Ideally, when reviewing papers, program committee members could run the rig
and check if the results recorded by the authors are reproducible.

This paper has reviewed a UNIX scripting tool called OURMINE as a method of documenting,
executing, and sharing data mining experiments.
We have used OURMINE to reproduce and check two  important result.
In our first experiment, we concluded that:
\begin{itemize}
\item{When local data is available, that data should be used to build defect predictors}
\item{If local data is not available, however, imported data can be used to build defect predictors when using {\em relevancy filtering}, because...}
\item{Imported data that uses {\em relevency filtering} performs nearly as well as using local data to build defect predictors}
\end{itemize}
In our second experiment, we learned:
\begin{itemize}
\item{When examining cluster inter/intra similarities resulting from each clustering/reduction solution, we found that faster heuristic methods can outperform more rigorous ones when observing decreases in runtimes.}
\item{This means that faster solutions are suitable on large datasets due to $scalability$.}
\end{itemize}

We prefer OURMINE to other tools. Four features are worthy of mention:

\be
\item
OURMINE is very succinct. As seen above, a few lines can describe even
 complex experiments.
\item
OURMINE's experimental descriptions are  complete. There is nothing hidden 
in \fig{promiseExp}; it is not the pseudocode of an experiment,
it \underline{is} the experiment.
\item
OURMINE code like in 
\fig{promiseExp}, \fig{clusterworker} and \fig{tfidfworker} is executable and can be executed by
other researchers directly.
\item
Lastly, 
the execution environment of OURMINE is readily available.
Unlike RAPID-I, WEKA, ``R'', etc, there is nothing to debug
or install. Many machines already have the support tools required for OURMINE. For example,  
we have run OURMINE on
Linux, Mac, and Windows machines (with Cygwin installed). 
\ee

Like Ritthol et al., we doubt that the standard interfaces of tools like WEKA, etc,
are adequate for representing the space of possible experiments. Impressive
visual programming environments are not the answer: their sophistication can
either distract or discourage novice data miners from extensive modification and
experimentation. Also, we find that the functionality of the visual environments can be achieved with a little BASH and GAWK scripts, with a fraction of the development effort and a greatly increased chance that novices will modify the environment.

OURMINE is  hence a candidate format for sharing descriptions
of experiments. The PROMISE community
might find this format unacceptable but discussions about the drawbacks (or strengths)
of OURMINE would help evolve not just OURMINE, but also the discussion on how
to represent data mining experiments for software engineering.
\newpage
\bibliographystyle{plain}
\newpage
%\clearpage
%\bibliographystyle{plain}
\bibliography{refs}
%\newpage
\begin{figure}
\small
\begin{center}
\begin{tabular}{l@{~}|l@{~}|r@{~}r@{~}@{~}r@{~}|c}
%\multicolumn{2}{c}{~}&\multicolumn{5}{c}{quartiles}\\\cline{3-5}
%&min& & median & &max\\\cline{3-5}
\multicolumn{2}{c}{~}&\multicolumn{3}{c}{pd}&2nd quartile\\
\multicolumn{2}{c}{~}&\multicolumn{3}{c}{percentiles}&median,\\\cline{3-5}
Rank&Treatment& 25\%& 50\% & 75\%&3rd quartile\\\hline
1&WC (local data) + relevancy filter& 66& 73& 80&  \boxplot{0}{66}{73}{80}{3}  \\
2&CC (imported data) + relevancy filter& 57& 71& 83&  \boxplot{0}{57}{71}{83}{5}  \\
2&WC (local data)  & 59& 69& 83&  \boxplot{0}{59}{69}{83}{7}  \\
3&CC (imported data)  & 49& 66& 87&  \boxplot{0}{49}{66}{87}{7}   \\\hline
\multicolumn{5}{c}{~}&~~~~~0~~~~~~~~50~~~~100
\end{tabular}
\end{center}
\caption{Probability of Detection (PD) results, sorted by median values.  }\label{fig:pds}
\end{figure}

\begin{figure}
\small
\begin{center}
\begin{tabular}{l@{~}|l@{~}|r@{~}r@{~}@{~}r@{~}|c}
%\multicolumn{2}{c}{~}&\multicolumn{5}{c}{quartiles}\\\cline{3-5}
%&min& & median & &max\\\cline{3-5}
\multicolumn{2}{c}{~}&\multicolumn{3}{c}{pf}&2nd quartile\\
\multicolumn{2}{c}{~}&\multicolumn{3}{c}{percentiles}&median,\\\cline{3-5}
Rank&Treatment& 25\%& 50\% & 75\%&3rd quartile\\\hline
1&CC (imported data) + relevancy filter& 17& 29& 43&  \boxplot{0}{17}{29}{43}{3}  \\
1&CC (imported data)  & 13& 34& 51&  \boxplot{0}{13}{34}{51}{5} \\
2&WC (local data)  & 17& 30& 41&  \boxplot{0}{17}{30}{41}{7}  \\
3&WC (local data) + relevancy filtuer & 20& 27& 34&  \boxplot{0}{20}{27}{34}{8}  \\\hline
\multicolumn{5}{c}{~}&~~~~~0~~~~~~~~50~~~~100
\end{tabular}
\end{center}
\caption{Probability of False Alarm (PF) results, sorted by median values.  }\label{fig:pfs} \end{figure}

\begin{figure}
\centering
\small
\begin{tabular}{| c | c | c | c | c |}
\hline
Reducer and Clusterer & Time & InterSim & IntraSim & Gain \\ \hline
TF-IDF*K-means & 17.52 & -0.085 & 141.73 & 141.82 \\ \hline
TF-IDF*GenIc & 3.75 & -0.14 & 141.22 & 141.36 \\ \hline
PCA*K-means & 100.0 & 0.0 & 100.0 & 100.0 \\ \hline
PCA*Canopy & 117.49 & 0.00 & 99.87 & 99.87 \\ \hline
PCA*GenIc & 11.71 & -0.07 & 99.74 & 99.81 \\ \hline
TF-IDF*Canopy & 6.58 & 5.02 & 93.42 & 88.4 \\ \hline
\end{tabular}
\caption{Similarity values normalized according to the combination of most rigorous reducer and clusterer.} 
\label{fig:sims}
\end{figure}
\newpage
%\clearpage
\section*{Installing OURMINE}
OURMINE is an open source toolkit licensed under GPL 3.0. It can
be downloaded and installed from
\url{http://code.google.com/p/ourmine}.

OURMINE is a command-line environment, and as such, system requirements are minimal. However, in order to use OURMINE three things must be in place:
\bi
\item A Unix-based environment. This does not include Windows. Any machine with OSX or Linux installed will do.
\item The Java Runtime Environment. This is required in order to use the WEKA, as well as any other Java code written for OURMINE.
\item The GAWK Programming Language. GAWK will already be installed with up-to-date Linux versions. However, OSX users will need to install this.
\ei
To install and run OURMINE, navigate to \url{http://code.google.com/p/ourmine} and follow the instructions. 
\clearpage
\section*{Built-in OURMINE Functions}

\subsection*{Utility Functions I}

\begin{centering}
\small
\begin{tabular}{|p{1.0in}|p{2.5in}|p{2in}|p{2.5in}|}
\hline
Function Name & Description & Usage \\ \hline
abcd & Performs confusion matrix computations on any classifier output. 
This includes statistics such as \$pd, \$pf, \$accuracy, \$balance and \$f-measure 
 & {\em | abcd --prefix --goal}, where $prefix$ refers to a string to be inserted 
before the result of the $abcd$ function, and $goal$ is the desired class of 
a specific instance. \\ \hline
arffToLisp & Converts a single .arff file into an equivalent .lisp file & {\em arffToLisp \$dataset.arff} \\ \hline
blab & Prints to the screen using a separate environment. This provides the ability to 
print to the screen without the output interfering with the results of an experiment & 
{\em blab \$message} \\ \hline
blabln & The same as blab, except this will print a new line after the given output & 
{\em blabln \$message} \\ \hline
docsToSparff & Converts a directory of document files into a sparse .arff file. Prior to 
building the file, however, the text is cleaned & {\em docsToSparff \$docDirectory \$output.sparff} \\ \hline
docsToTfidfSparff & Builds a sparse .arff file from a directory of documents, as above, but 
instead constructs the file based on TF-IDF values for each term in the entire corpus. 
& {\em docsToTfidfSparff \$docDirectory \$numberOfAttributes \$output.sparff} \\ \hline
formatGotWant & Formats an association list returned from any custom LISP classifier containing 
actual and predicted class values in order to work properly with existing OURMINE functions  
& {\em formatGotWant} \\ \hline
funs & Prints a sorted list of all available OURMINE functions & {\em funs} \\ \hline
getClasses & Obtains a list of all class values from a specific data set & {\em getClasses} \\ \hline
getDataDefun & Returns the name of a .arff relation to be used to construct a LISP function 
that acts as a data set & {\em getDataDefun} \\ \hline
gotwant & Returns a comma separated list of actual and predicted class values from the output 
of a WEKA classifier & {\em gotwant} \\ \hline
help & When given with an OURMINE function, prints helpful information about the function, such as 
a description of the function, how to use it, etc. & {\em help \$function}, where \$function is the name 
of the function \\ \hline
\end{tabular}
\end{centering}
\newpage
\subsection*{Utility Functions II}

\begin{centering}
\small
\begin{tabular}{|p{1.0in}|p{2.5in}|p{2in}|p{2.5in}|}
\hline
Function Name & Description & Usage \\ \hline
makeQuartiles & Builds quartile charts using any key and performance value from the abcd results (see above) & 
{\em makeQuartiles \$csv \$keyField \$performanceField}, where \$keyField can be a learner/treatment, etc., 
and \$performanceField can be any value desired, such as $pd$, $accuract$, etc.\\ \hline
makeTrainAndTest & Constructs a training set and a test set given an input data set. The outputs of the 
function are train.arff, test.arff and also train.lisp and test.lisp & 
{\em makeTrainAndTest \$dataset \$bins \$bin}, where 
\$dataset refers to any data set in correct .arff format, \$bins refers to the number of bins desired in the constuction 
of the sets, and \$bin is the bin to select as the test set. For instance, if 10 is chosen as the number of bins, 
and 1 is chosen as the test set bin, then the resulting training set would consist of 90\% of the data, and the test 
set would consist of 10\%. \\ \hline
malign & Neatly aligns any comma-separated format into an easily readable format & {\em malign} \\ \hline
medians & Computes median values given a list of numbers & {\em medians} \\ \hline
quartile & Generates a quartile chart along with min/max/median values, as well as second and third quartile values 
given a specific column & 
{\em quartile} \\ \hline
show & Prints an entire OURMINE function so that the script can be seen in its entirety & \em{show \$functionName} \\ \hline
winLossTie & Generates win-loss-tie tables given a data set. Win-loss-tie tables, in this case, depict results 
after a statstical analysis test on treatments. These tests include the Mann-Whitney-U test, as well as the 
Ranked Wilcoxon test & 
{\em winLossTie --input \$input.csv --fields \$numOfFields --perform \$performanceField --key \$keyField --\$confidence}, 
where \$input.csv refers to the saved output from the $abcd$ function described above, \$numOfFields represents the 
number of fields in the input file, \$performanceField is the field on which to determine performance, such as 
$pd$, $pf$, $acc$, \$keyField is the field of the key, which could be a learner/treatment, etc., and \$confidence 
is the percentage of confidence when running the test. The default confidence value is 95\% \\ \hline
\end{tabular}
\end{centering}
\newpage
\subsection*{Learners}

\begin{centering}
\small
\begin{tabular}{|p{1.0in}|p{2.5in}|p{2in}|}
\hline
Function Name & Description & Usage \\ \hline
adtree & Calls WEKA's Alternating Decision Tree & {\em adtree \$train \$test} \\ \hline
bnet & Calls WEKA's Bayes Net & {\em bnet \$train \$test} \\ \hline
j48 & Calls WEKA's J48 & {\em j48 \$train \$test} \\ \hline
nb & Calls WEKA's Naive Bayes & {\em nb \$train \$test} \\ \hline
oner & Calls WEKA's One-R & {\em oner \$train \$test} \\ \hline
rbfnet & Calls WEKA's RBFNet & {\em rbfnet \$train \$test} \\ \hline
ridor & Calls WEKA's RIDOR & {\em ridor \$train \$test} \\ \hline
zeror & Calls WEKA's Zero-R & {\em zeror \$train \$test} \\ \hline
\end{tabular}
\end{centering}

\subsection*{Preprocessors}
\begin{centering}
\small
\begin{tabular}{|p{1.0in}|p{2.5in}|p{2in}|}
\hline
Function Name & Description & Usage \\ \hline
caps & Reduces capitalization to lowercase from an input text & {\em caps} \\ \hline
clean & Cleans text data by removing capitals, words in a stop list, special tokens, and performing 
Porter's stemming algorithm & {\em clean} \\ \hline
discretize & Discretizes the incoming data via WEKA's discretizer & {\em discretize \$input.darff \$output.arff} \\ \hline
logArff & Logs numeric data in incoming data & {\em logArff \$minVal \$fields}, where \$minVal denotes the minimum value 
to be passed to the log function, and \$fields is the specfic fields on which to perform log calculations \\ \hline
stems & Performs Porter's stemming algorithm on incoming text data & {\em stems \$inputFile} \\ \hline
stops & Removes any terms from incoming text data that are in a stored stop list & {\em stops} \\ \hline
tfidf & Computes TF*IDF values for terms in a document & {\em tfidf \$file} \\ \hline
tokes & Removes unimportant tokens or whitespace from incoming textual data & {\em tokes} \\ \hline
\end{tabular}
\end{centering}
\newpage
\subsection*{Feature Subset Selectors}
\begin{centering}
\small
\begin{tabular}{|p{0.9in}|p{2.5in}|p{2.7in}|}
\hline
Function Name & Description & Usage \\ \hline
cfs & Calls WEKA's Correlation-based Feature Selector & {\em cfs \$input.arff \$numAttributes \$out.arff} \\ \hline
chisquared & Calls WEKA's Chi-Squared Feature Selector & {\em chisquared \$input.arff \$numAttributes \$out.arff} \\ \hline
infogain & Calls WEKA's Infogain Feature Selector & {\em infogain \$input.arff \$numAttributes \$out.arff} \\ \hline
oneR & Calls WEKA's One-R Feature Selector & {\em oneR \$input.arff \$numAttributes \$out.arff} \\ \hline
pca & Calls WEKA's Principal Components Analysis Feature Selector & {\em pca \$input.arff \$numAttributes \$out.arff} \\ \hline
relief & Calls WEKA's RELIEF Feature Selector & {\em relief \$input.arff \$numAttributes \$out.arff} \\ \hline
\end{tabular}
\end{centering}

\subsection*{Clusterers}
\begin{centering}
\small
\begin{tabular}{|p{0.9in}|p{2.5in}|p{2.7in}|}
\hline
Function Name & Description & Usage \\ \hline
K-means & Calls custom Java K-means & {\em \$Clusterers -k  \$k \$input.arff \$out.arff, where \$k is the initial number of centroids} \\ \hline
Genic & Calls custom Java GeNic & {\em \$Clusterers -g \$k \$n \$input.arff \$out.arff, where \$k is the initial number of centroids, and \$n is the size of a generation} \\ \hline
Canopy & Calls custom Java Canopy Clustering & {\em \$Clusterers -c \$k \$p1 \$p2 \$input.arff \$out.arff, where k is the initial number of centroids, \$p1 is a similarity percentage value for the outer threshold, and \$p2 is a similarity percentage value for the inner threshold. If these percentages are not desired, a value of 1 should be provided for both} \\ \hline
EM & Calls WEKA's Expectation-Maximization Clusterer & {\em em \$input.arff \$k, where \$k is the initial number of centroids} \\ \hline
\end{tabular}
\end{centering}


%\begin{thebibliography}{10}


%\end{thebibliography}


\end{document}

