1 The literature is doubtful about the ability to make the kinds of 2 stable comments about the results from estimation data. Foss 3 et.al. [1] comment that it ". . . is futile to search for the 4 Holy Grail: a single, simple-to-use, universal goodness-of-?t kind 5 of metric, which can be applied with ease to compare (different 6 methods)." In three papers in TSE in 2000s, Foss et al and Shepperd 7 et al offer evidence that such rankings can change based on: 8 9 - the random selection of data, 10 11 - the data set used, 12 13 - the evaluation metric used for the comparison. 14 15 (As an aside: this paper does not even mention these papers or the 16 issues they raise.) 17 18 A convincing counter case to this pessimism would be to show that 19 method A is better than method B over multiple data sets, 20 over multiple different random selections of train/test data, over 21 multiple evaluation criteria. 22 23 In the following, I can show stability across different criteria 24 and in 20 random 66% samples of the data. Exploration of mulitple 25 data sets is not done since my domain expert (YOU) is most familiar 26 with nasa93. 27 28 That's the good news. The bad news is that I just don't know if the conclusions are useful. 29 In the following, LOC and EFFORT have been discretized by a log transform, then a 3-way chop to 30 produce: 31 32 LOC data/numeric/nasa93pure.arff 33 13 1..9 34 50 9..95 35 30 95..980 36 EFFORT data/numeric/nasa93pure.arff 37 28 8..83 38 17 827..8211 39 48 83..827 40 41 Now, we can look thru the results checking for the jairus conjectures; 42 i.e. the following factors most predict for effort: 43 44 size 45 reuse 46 rely 47 cplx 48 rvel (a size inflation factor) 49 50 and for a-teams, will accept other things. 51 52 since this is coc81 data, we can only use size,reply,cplx. before 53 looking at the following, best to also understand the dists.png 54 figure. 55 56 timm@rome.local:~/svns/wisp/var/timm/10/which2 [568]$ make gen93pure Klass=3 Eval=6 57 (support * accuracy) 58 59 class rule N mean +- sd 60 61 8..83 stor=n. 19 27.55 +- 2.66 **************************** 62 8..83 vexp=n. 20 30.76 +- 3.10 ******************************* 63 8..83 time=n. 20 31.22 +- 3.41 ******************************** 64 8..83 tool=n. 20 31.93 +- 2.54 ******************************** 65 8..83 virt=l. 20 32.46 +- 3.75 ********************************* 66 8..83 cplx=h. 20 35.60 +- 3.78 ************************************ 67 68 83..827 aexp=h. 20 29.31 +- 3.43 ****************************** 69 83..827 acap=h. 18 32.54 +- 3.56 ********************************* 70 83..827 LOC=9..95. 20 33.27 +- 3.72 ********************************** 71 83..827 virt=l. 19 45.06 +- 4.41 ********************************************** 72 73 827..8211 data=h. 17 11.88 +- 2.17 ************ 74 827..8211 acap=h+LOC=95..980. 18 13.31 +- 2.97 ************** 75 827..8211 acap=h+aexp=h. 17 14.62 +- 1.74 *************** 76 827..8211 pcap=h. 20 15.51 +- 2.55 **************** 77 827..8211 modp=h. 20 16.37 +- 2.63 ***************** 78 827..8211 aexp=h. 20 17.78 +- 2.27 ****************** 79 827..8211 LOC=95..980. 20 18.20 +- 2.82 ******************* 80 827..8211 acap=h. 20 21.26 +- 2.06 ********************** 81 82 timm@rome.local:~/svns/wisp/var/timm/10/which2 [568]$ make gen93pure Klass=3 Eval=4 83 (support * harmonic mean(pd,pf) 84 85 class rule N mean +- sd 86 87 8..83 pcap=n. 18 32.82 +- 2.74 ********************************* 88 8..83 stor=n. 20 33.03 +- 2.64 ********************************** 89 8..83 virt=l. 19 33.51 +- 2.03 ********************************** 90 8..83 time=n. 20 35.31 +- 2.52 ************************************ 91 8..83 vexp=n. 20 36.07 +- 2.19 ************************************* 92 8..83 tool=n. 20 36.43 +- 1.87 ************************************* 93 8..83 cplx=h. 20 39.45 +- 1.85 **************************************** 94 95 83..827 pcap=h. 16 21.28 +- 2.60 ********************** 96 83..827 rely=n. 17 23.48 +- 2.40 ************************ 97 83..827 aexp=h. 20 27.63 +- 2.54 **************************** 98 83..827 acap=h. 20 29.42 +- 2.90 ****************************** 99 83..827 LOC=9..95. 20 32.05 +- 3.20 ********************************* 100 83..827 virt=l. 19 34.01 +- 2.91 *********************************** 101 102 827..8211 turn=n. 16 15.66 +- 4.28 **************** 103 827..8211 aexp=h+pcap=h. 17 16.78 +- 3.27 ***************** 104 827..8211 acap=h+LOC=95..980. 17 18.20 +- 2.78 ******************* 105 827..8211 acap=h+pcap=h. 17 19.31 +- 2.99 ******************** 106 827..8211 acap=h+aexp=h. 19 22.24 +- 3.46 *********************** 107 827..8211 pcap=h. 20 23.79 +- 2.40 ************************ 108 827..8211 LOC=95..980. 20 24.23 +- 2.50 ************************* 109 827..8211 modp=h. 20 26.42 +- 2.62 *************************** 110 827..8211 aexp=h. 19 29.31 +- 2.79 ****************************** 111 827..8211 acap=h. 20 32.44 +- 2.25 ********************************* 112 113 timm@rome.local:~/svns/wisp/var/timm/10/which2 [569]$ make gen93pure Klass=3 Eval=5 114 (support * harmonic mean(pd,pred) 115 116 class rule N mean +- sd 117 118 8..83 pcap=n. 17 32.37 +- 2.02 ********************************* 119 8..83 stor=n. 20 32.68 +- 1.78 ********************************* 120 8..83 virt=l. 20 33.59 +- 1.96 ********************************** 121 8..83 vexp=n. 20 35.08 +- 1.83 ************************************ 122 8..83 time=n. 20 35.31 +- 1.74 ************************************ 123 8..83 tool=n. 20 36.56 +- 1.71 ************************************* 124 8..83 cplx=h. 20 39.16 +- 2.03 **************************************** 125 126 83..827 aexp=h. 19 29.47 +- 2.88 ****************************** 127 83..827 LOC=9..95. 20 31.42 +- 2.62 ******************************** 128 83..827 acap=h. 19 31.93 +- 1.86 ******************************** 129 83..827 virt=l. 19 38.29 +- 2.16 *************************************** 130 131 827..8211 acap=h+aexp=h. 17 24.39 +- 2.22 ************************* 132 827..8211 LOC=95..980. 20 24.42 +- 1.73 ************************* 133 827..8211 modp=h. 20 25.46 +- 1.63 ************************** 134 827..8211 pcap=h. 20 25.47 +- 1.91 ************************** 135 827..8211 aexp=h. 20 27.95 +- 1.78 **************************** 136 827..8211 acap=h. 20 30.46 +- 1.39 *******************************