\begin{slide}{In summary...}
\bi
\item Five NASA data sources
	\bi 
	\item	Eg \#1: text mining a NASA issue database (PITS)
    \item		Eg \#2: effort estimation from NASA data (COCOMO)
	\item	Eg \#3: early life cycle severity prediction  (SILAP)
	\item	Eg \#4: defect prediction from NASA static code data (MDP)
	\item	Eg \#5: defect prediction (NASA SEL)
	\ei
\item All of which yield strong predictors for quality (effort, defects)
\item	Only \underline{one} of which is still active (PITS)
\item What went wrong?
\item What to do?
\ei
\end{slide}
\begin{slide}[toc=Why?]{Why is this Data Being Ignored?}
\small
\bi
	\item Group 1 : easy to explain
	\bi
	\item NASA SEL : 
		Technology used in case study \#5 very new
	\item PITS :
		\bi
		\item Accessing PITS data was hard- required much civil servant support
		\item No one was crazy enough to try text mining on unstructured PITS issue reports.
		\ei
	\item SILAP :
		\bi
		\item Newest data set of all the above
		\item Never explored before since not available before
		\item Data collection stopped since IV\&V business model changed (now focused on
			model-based early lifecycle validation).
		\ei
	\ei
	\item Group 2 : harder to explain
		\bi
		\item MDP : Much interest across the agency (at GRC, JSC) in MDP (and associated tools).  
		\item COCOMO:   well-documented, cheap to collect, many tools available
		\item Maybe the answer lies in NASA culture:
			\bi
			\item NASA's centers compete for resources. 
			\item Reluctance to critically evaluate and share process information.
			\ei
		\ei
	\ei
\end{slide}
\begin{slide}{ What to do?}
\vspace{-0.25cm}\bi
\item Stop {\em debating what data to collect}
\bi
     \item Many loosely-defined sources will do: COCOMO, SILAP, defect reports
\ei
\item Stop {\em debating how to store data}
	\bi
		\item	Comma-separated or ARFF format or XML , one per component,  is fine
	\ei
\item Stop {\em hiding data}
	\bi
 		\item Create a central register for all NASA's software components
		\item Register = component name and ``part-of'' (super-component)
		\item Features extracted from all		components, stored at a  central location
		\item All reports have anonymous join key to the central register
		\item Make the anonymous data open source (lever the data mining community) 
		\ei
\item Stop {\em ignoring institutional data}
	\bi
		\item Active repository, not data tomb
		\item Success measure: not data in, but conclusions out
	\ei
\item	Stop {\em publishing vague generalities}
	\bi
		\item Rather, publish {\em general methods} for building {\em specific models}
		\item Open research question: how much data is enough to learn local model?
	\ei
\ei
\end{slide}
