What is COSEEKMO?

• COSEEKMO is like a big lab with a lot of different data mining tools like WRAPPERs, learners, etc.
  
• As of now, there are 122 combinations of these data mining tools creating 122 methods.
  
• The data is collected in COCOMO style from software projects developed by NASA and others.
  
• Initially, we wanted to study the large variance problem in cost estimation.
  
• These methods usually do not agree. So, we need experiments to compare these methods.
  
• Hence, COSEEKMO was developed to allow for plugging in as many methods as we want and experimenting with them.
  
• COSEEKMO analyzes the results and gives us conclusions on which method reduced the variance the most.
  
• Since we wanted to reduce the variance, we thought parametric analysis is the answer.
  

What is parametric evaluation?

• Wikipedia says that all parametric tests involve:
  1. Estimating at least one population parameter.
  2. Assumptions about the distribution of the population from which the data were randomly sampled.
  3. Assumptions about the measurement of the dependent variable.
  
  
• Simply, we assume there is a certain distribution in our data and we try to measure some parameter.
  
• We assumed that it was a normal distribution or Gaussian distribution (bell curves and S shaped cumulative distribution functions).
  





• So, we used mean and standard deviation of the population parameter, which are Gaussian concepts.
  
• T-tests are used as well.
• The population parameter is the MRE (magnitude of relative error).
  
• What we wanted to do was to reduce the variance in the MREs using this analysis.
  
• The problem is that the distribution assumption does not hold! The undelying distribution of our data is not Gaussian.
  


• Note how the CDF does not spike at the end but our data does. It clearly is not Gaussian.
  

What is non-parametric evaluation?

• Since we do not know anything about the underlying distribution, we had to change our focus to non-parametric analysis.
  
• These types of analysis do not assume any underlying distribution.
  
• Wikipedia clarifies this better. The term nonparametric is not meant to imply that such models completely lack parameters but that the number and nature of the parameters are flexible and not fixed in advance. Nonparametric models are therefore also called distribution free."
• Generally, parametric tests are stronger than non-parametric test.
  
• However, there are many different non-parametric tests available (listed here).
  
  
• Mann-Whitney U test (also called the Mann-Whitney-Wilcoxon (MWW) test, the Wilcoxon-Mann-Whitney (WMW) test, or the Wilcoxon rank-sum test) is a non-parametric test for assessing whether two samples of observations come from the same distribution (see Wikipedia for more details).
  
  
• This is how it is performed (see here for a complete example):
  
  Assume we have two samples A with values 5, 7, 2, 0, 4 and B with values 4, 8, 2, 3, 6, 7.
  
  We sort the union of the two samples maintaining which value belongs to which population:
  

  Samples: A  A  B  B  A  B  A  B  A  B  B
  Values:  0  2  2  3  4  4  5  6  7  7  8
  

  
  Then we rank these values (and use the average of ranks if two or more values are the same)
  

  Samples: A   A   B   B   A   B   A   B   A   B   B
  Values:  0   2   2   3   4   4   5   6   7   7   8
  Ranks:   1  2.5 2.5  4  5.5 5.5  7   8  9.5 9.5  11
  

  
  We add up the ranks for each sample:
  
  Sum(A) = 1 + 2.5 + 5.5 + 7 + 9.5 = 25.5
  Sum(B) = 2.5 + 4 + 5.5 + 8 + 9.5 + 11 = 40.5
  
  Next, we find the U(A) and U(B) using the equation: U = Sum - Size*(Size+1)/2 where Size is the size of the population:
  
  U(A) = 25.5 - 5*6/2 = 10.5
  U(B) = 40.5 - 6*7/2 = 19.5
  
  For small samples, we can look up the minimum of U(A) and U(B) (called U) in a table. In this case, the results show that the two sample are not significantly different.
  
  For large samples, we need to find Z from U (minimum of U(A) and U(B)) and sample sizes (equations obtained from Wikipedia)
  
  
  
  
  
  
  
  For 95% significance level, if magnitude of Z is less than or equal to 1.96, it means that the two samples are equal (each gets a tie).
  
  But we need more than just tie or non-tie. So, we have to use a median test (still non-parametric) in addition to the WMW test.
  
  
  If median of ranks in sample A is greater than median of ranks in sample B, sample A gets a win and sample B gets a loss.
  If median of ranks in sample A is less than median of ranks in sample B, sample A gets a loss and sample B gets a win.
  If median of ranks in sample A is equal to median of ranks in sample B, each sample gets a tie.
  
  
  
• So, WMW does not require any knowledge about the distrubutions.
• It works for any two large samples of data that may or may not have the same size.
• It is a costly but an effective method. For 122 methods, it requires 122*121/2 = 7381 comparisons total for each dataset where there are at least 300 results for each method. There are 19 datasets total.
• At the end, we get Win-Tie-Loss results that we need to analyze.
  
  

      LocalWrapper, proximal,     locomo-40, 37, 48,  36
              None, proximal, locomoFull-20, 39,  7,  75
    COCOMINWrapper, proximal,     locomo-40, 51, 24,  46
   M5PWrapperNoLog,  precise,           lsr, 50, 69,   2
   LSRWrapperNoLog,  precise,           lsr, 35, 85,   1
   M5PWrapperNoLog, proximal,           m5p, 37, 82,   2
   M5PWrapperNoLog, proximal,     locomo-40, 36, 85,   0
   LSRWrapperNoLog, proximal,           lsr, 39, 81,   1
        LSRWrapper, proximal,     locomo-20, 55, 27,  39
              None,  precise, locomoFull-40, 21,  0, 100
    COCOMINWrapper, proximal,            lc, 48, 31,  42
              None, proximal, locomoFull-40, 42, 17,  62
        LSRWrapper,  precise,           m5p, 52, 30,  39
    COCOMINWrapper,  precise,            lc, 24,  4,  93
    COCOMINWrapper,  precise,           lsr, 34,  3,  84
        M5PWrapper,  precise,      locomo-5, 46, 52,  23
              None,  precise,       cocomin, 26,  5,  90
        LSRWrapper,  precise,      locomo-5, 56, 31,  34
        M5PWrapper, proximal,      locomo-5, 58, 40,  23
      LocalWrapper,  precise,           lsr, 41, 36,  44
        LSRWrapper, proximal,     locomo-40, 48, 63,  10
        M5PWrapper,  precise,     locomo-10, 54, 26,  41
      LocalWrapper, proximal,      locomo-5, 32, 64,  25
      LocalWrapper, proximal,     locomo-80, 27, 61,  33
        LSRWrapper,  precise,     locomo-20, 53, 19,  49
    COCOMINWrapper, proximal,     locomo-80, 48, 32,  41
   M5PWrapperNoLog, proximal,     locomo-80, 35, 85,   1
              None,  precise, locomoFull-80, 21,  2,  98
              None, proximal, locomoFull-80, 49, 23,  49
        LSRWrapper,  precise,     locomo-40, 56, 27,  38
   M5PWrapperNoLog,  precise,            lc, 36, 84,   1
              None,  precise,            sd, 20,  0, 101
        LSRWrapper, proximal,     locomo-80, 45, 72,   4
   LSRWrapperNoLog,  precise,     locomo-20, 37, 84,   0
      LocalWrapper,  precise,            lc, 44, 31,  46
        LSRWrapper, proximal,           lsr, 53, 14,  54
              None, proximal,       cocomin, 52, 28,  41
        M5PWrapper, proximal,            lc, 39, 81,   1
              None, proximal,      cocomost, 48, 29,  44
              None, proximal,  locomoFull-5, 42,  8,  71
              None,  precise,           org, 22,  0,  99
    COCOMINWrapper,  precise,     locomo-20, 21,  1,  99
   LSRWrapperNoLog,  precise,     locomo-40, 37, 84,   0
        LSRWrapper,  precise,     locomo-80, 55, 31,  35
        M5PWrapper, proximal,     locomo-10, 54, 35,  32
              None,  precise,  locomoFull-5, 24,  2,  95
              None,  precise,       m5pFull, 48, 22,  51
    COCOMINWrapper,  precise,     locomo-40, 22,  3,  96
   M5PWrapperNoLog,  precise,     locomo-20, 42, 79,   0
   LSRWrapperNoLog, proximal,     locomo-20, 36, 84,   1
   M5PWrapperNoLog,  precise,     locomo-40, 37, 84,   0
   LSRWrapperNoLog,  precise,     locomo-80, 36, 84,   1
              None, proximal,             e, 51, 21,  49
        M5PWrapper,  precise,           lsr, 50, 22,  49
      LocalWrapper, proximal,     locomo-10, 49, 31,  41
   LSRWrapperNoLog, proximal,     locomo-40, 35, 86,   0
        LSRWrapper, proximal,            lc, 45, 73,   3
    COCOMINWrapper, proximal,     locomo-10, 27,  2,  92
      LocalWrapper, proximal,           lsr, 36, 45,  40
    COCOMINWrapper, proximal,           lsr, 45, 12,  64
    COCOMINWrapper,  precise,     locomo-80, 24,  5,  92
        LSRWrapper,  precise,            lc, 55, 32,  34
        M5PWrapper, proximal,           m5p, 53, 22,  46
   M5PWrapperNoLog, proximal,     locomo-10, 36, 85,   0
   LSRWrapperNoLog,  precise,            lc, 36, 85,   0
              None,  precise, locomoFull-10,  5,  0, 116
              None, proximal, locomoFull-10, 28,  3,  90
              None, proximal,       m5pFull, 51, 22,  48
   M5PWrapperNoLog,  precise,     locomo-80, 36, 85,   0
        LSRWrapper, proximal,     locomo-10, 54, 22,  45
      LocalWrapper,  precise,      locomo-5, 38, 44,  39
      LocalWrapper,  precise,     locomo-20, 44, 27,  50
   M5PWrapperNoLog,  precise,           m5p, 47, 68,   6
              None,  precise,        lcFull, 21,  3,  97
              None,  precise,      cocomost, 24,  6,  91
   LSRWrapperNoLog,  precise,           m5p, 36, 85,   0
   LSRWrapperNoLog, proximal,           m5p, 39, 82,   0
   M5PWrapperNoLog, proximal,           lsr, 39, 79,   3
   LSRWrapperNoLog, proximal,     locomo-80, 34, 86,   1
              None,  precise,             e, 42,  8,  71
    COCOMINWrapper,  precise,           m5p, 50, 19,  52
      LocalWrapper,  precise,     locomo-40, 45, 27,  49
        LSRWrapper,  precise,           lsr, 49, 21,  51
        LSRWrapper, proximal,      locomo-5, 56, 29,  36
      LocalWrapper,  precise,           m5p, 35, 49,  37
        LSRWrapper,  precise,     locomo-10, 53, 14,  54
    COCOMINWrapper,  precise,      locomo-5, 24,  3,  94
              None, proximal,           org, 32,  2,  87
        M5PWrapper,  precise,     locomo-20, 55, 29,  37
        M5PWrapper,  precise,     locomo-40, 53, 36,  32
      LocalWrapper,  precise,     locomo-80, 43, 32,  46
   LSRWrapperNoLog,  precise,     locomo-10, 37, 84,   0
        LSRWrapper, proximal,           m5p, 54, 28,  39
              None, proximal,        lcFull, 49, 24,  48
    COCOMINWrapper, proximal,      locomo-5, 38,  3,  80
        M5PWrapper,  precise,            lc, 46, 52,  23
    COCOMINWrapper,  precise,     locomo-10,  5,  1, 115
              None,  precise,       lsrFull, 33,  2,  86
   M5PWrapperNoLog,  precise,      locomo-5, 37, 84,   0
   LSRWrapperNoLog,  precise,      locomo-5, 36, 85,   0
        M5PWrapper,  precise,     locomo-80, 46, 53,  22
   M5PWrapperNoLog,  precise,     locomo-10, 42, 79,   0
   LSRWrapperNoLog, proximal,      locomo-5, 34, 87,   0
        M5PWrapper, proximal,     locomo-20, 55, 33,  33
   LSRWrapperNoLog, proximal,     locomo-10, 40, 80,   1
        M5PWrapper, proximal,     locomo-40, 48, 72,   1
   M5PWrapperNoLog, proximal,            lc, 35, 86,   0
   LSRWrapperNoLog, proximal,            lc, 34, 87,   0
        M5PWrapper,  precise,           m5p, 51, 19,  51
              None, proximal,            sd, 39,  4,  78
   M5PWrapperNoLog, proximal,      locomo-5, 30, 90,   1
      LocalWrapper, proximal,           m5p, 36, 44,  41
              None, proximal,       lsrFull, 45, 13,  63
    COCOMINWrapper, proximal,           m5p, 50, 21,  50
      LocalWrapper, proximal,     locomo-20, 48, 34,  39
        M5PWrapper, proximal,           lsr, 52,  8,  61
      LocalWrapper, proximal,            lc, 27, 62,  32
    COCOMINWrapper, proximal,     locomo-20, 39,  8,  74
   M5PWrapperNoLog, proximal,     locomo-20, 40, 81,   0
      LocalWrapper,  precise,     locomo-10, 42, 23,  56
        M5PWrapper, proximal,     locomo-80, 39, 82,   0
              None,  precise, locomoFull-20, 20,  0, 101
  

  
  
• It is not possible to draw conclusions from these tables directly.
  
• Though, we can learn from these results using machine learning.
  
• TAR3 is perfect for this purpose. We can look for treatments from TAR3.
  
• But we first need to simplify Win-Tie-Loss results into one class value. This is done using a scoring scheme: Score = a*Win + b*Tie + c*Loss with different a, b, and c values.
  
• Finally, we discretize the scores into 5 bins and let TAR3 run on these results.
  
• A sample treatment set looks like this:
  
  

10 Treatments learnt after 22 random trials

 1 worth=1.282352	[WrapperLogging=Yes] [Neighborhood=ALL] 
 2 worth=1.280275	[Wrapper=M5P-WRAPPER] [WrapperLogging=Yes] 
 3 worth=1.182326	[WrapperLogging=Yes] 
 4 worth=1.170401	[LearnerLogging=No] 
 5 worth=1.151567	[Wrapper=M5P-WRAPPER] 
 6 worth=1.107199	[LearnerLogging=Yes] [WrapperLogging=Yes] 
 7 worth=1.096614	[Neighborhood=ALL] 
 8 worth=1.087990	[WrapperLogging=Yes] [Learner=LOCOMO] 
 9 worth=1.070480	[NumericEstimation=PROXIMAL] 
10 worth=1.057342	[Wrapper=M5P-WRAPPER] [LearnerLogging=Yes] 



• For each set of results from any given dataset, we take 90% of the results at random and run TAR3.
  
• We do this 10 times with different 90% sets and compare the results and look for stability.
  
• The big picture:
  

COSEEKMO (122 Methods) --> WMW + Median Rank Tests --> Scores (Win-Tie-Loss) --> TAR3

What do the results tell us?

• The non-parametric results give different conclusions comparing to the parametric conclusions (although each gives stable conclusions).
  
• For example, in a paper that we published based on the parametric analysis, we argued that a certain WRAPPER (called COCOMIN which uses Local Calibration as the target learner) performs much better than other WRAPPERs.
  
• These results prefer WRAPPERs based on M5P and LSR target learners.
  
• Some of the conclusions hold in both types of analysis. For example, use of PROXIMAL instead of PRECISE numeric estimation is suggested by these treatments as well as the parametric analysis.
  
• The goal also changes. We can no longer focus on reducing variance (which is a Gaussian concept).
• WMW is a non-parametric test and variance of MREs is not applicable anymore.
• Instead, we can focus on the outliers’ problem.
• Outliers are one of the major causes of the large variance we have seen in the performance of these methods.
• We believe that outliers can be detected and removed using WMW test.

What to do next?

• Add more features from the methods to get better treatments.
• Try other non-parametric tests.
• Add more tools to COSEEKMO and see if results or the stability in results change.