\documentclass[11pt]{article}

\setlength{\oddsidemargin}{0in}
\setlength{\evensidemargin}{0in}
\setlength{\topmargin}{-0.5in}
\setlength{\headheight}{0pt}
\setlength{\topskip}{0pt}
\setlength{\textwidth}{6.5in}
\setlength{\textheight}{9in}
\setlength{\parindent}{0pt}
\setlength{\parskip}{1mm}

\usepackage[normalem]{ulem}

\begin{document}

\begin{center} \bf \Large
Andrews, Menzies and Li \\
SBSE Special Issue Submission \\
--- \\
Authors' Responses to Reviewers
\end{center}

Thank you again for your thoughtful and thorough reviews.  We have
revised the paper and give our responses to your comments
below.  Reviewers' comments are in {\tt teletype} font; our
responses are in Roman font.


\small \begin{verbatim}
Reviewers' Comments
\end{verbatim} \normalsize



\small \begin{verbatim}
Reviewer: 1
\end{verbatim} \normalsize



\small \begin{verbatim}
Recommendation: Author Should Prepare A Minor Revision
\end{verbatim} \normalsize



\small \begin{verbatim}
Comments:
In general, I like this paper, but there is still some (possibly) minor
work to do.  The section below called 'Major Comments' identifies my major
concerns.
\end{verbatim} \normalsize



\small \begin{verbatim}
Response to Previous Comments
------------------------------------
\end{verbatim} \normalsize



\small \begin{verbatim}
The authors have also largely addressed the majority of the comments I had
on the previous revision. Some commentary on this:
\end{verbatim} \normalsize



\small \begin{verbatim}
(i) The 'spikes' of Fig. 1: I don't accept that the spikes are a *correct*
consequence of using integers; instead they are an *incorrect* artefact of
attempting to join discrete integer points by lines in the plot.  However,
this is a minor point and attempting to remove the lines could result in
less understandable figures. As it stands the important argument - that
reformulating the problems in terms of a probabilistic range of values
gives a guiding gradient to some part of the search space - is still made,
and so no further changes need be made to this figure.
\end{verbatim} \normalsize

\small \begin{verbatim}
(ii) Serious consideration should still be given to how to improve figures
10, 12 and 15.  The lines for many classes are indistinguishable, in which
case why label the lines at all by a legend?  Simply giving more space to
the figures may help.  (See comments below about simplifying the text of
section IV which may save space which could be used for this purpose.)
\end{verbatim} \normalsize

Figure 10 has been modified in three ways:  (1) it is now an average
over all classes, rather than one single class, reducing noise; (2)
the X-axis is now time rather than generations, making a comparison
with figures 11 and 14 easier; and (3) the lines are distinguished
by having differently-shaped points, making the different lines
stand out better.  We feel it is still beneficial to show all 10
lines, since it shows how the behaviour changes as we reduce the
number of gene types.

Figures 11 and 14 have now been simplified by showing only the average
over all classes.

\small \begin{verbatim}
(iii) With respect of the empirical work of (now) section IV-D, I don't
think the fact of 16 SUTs avoids the argument that basing the analysis on
single runs for each SUT/number of eliminated gene types combination lack
robustness.  However, this is not a major point.
\end{verbatim} \normalsize



\small \begin{verbatim}
\end{verbatim} \normalsize



\small \begin{verbatim}
Major Comments
-------------------
\end{verbatim} \normalsize



\small \begin{verbatim}
Nevertheless, there are still some major issues to be addressed:
\end{verbatim} \normalsize



\small \begin{verbatim}
(1) Sections I, II and III read very well and describe the Nighthawk
system at an appropriate level of detail.  However, section IV (on FSS) is
significantly less easy to understand.  It is arguably too long: the key
arguments could have been made with less text, and omitting some of the
experimental detail.  For example, while the identification of the change
to the initialisation of numberOfCalls is an important outcome, is it
necessary to describe results for java.util in detail prior to this
change? (Also, the Apache Common Collections results - Fig 15 - are
described after this change, why is this not done for java.util also - in
place of Fig 12?) Similarly, the rejection of bestMerit and bestRank in
favour of avgMerit and avgRank could be summarised briefly.
\end{verbatim} \normalsize

At this reviewer's suggestions, we reviewed that section and found
much that was redundant or confusing. Accordingly, we  have cut
the former Figure 11 and removed the confusing text.

\small \begin{verbatim}
(2) The arguments in favour of using FSS are somewhat confused, possibly
by using too many different metrics/experiments.  While the result that
using 1 rather than 10 gene types result only a marginal decrease in
coverage / AUC (Fig 10/Fig 11) is meaningful, the argument that when using
one 1 gene type, 90% can be achieved in 10% of the time of the `full'
Nighthawk is less convincing.  The actual metric for time isn't stated
clearly, but if we assume that number of generations are a broadly similar
measure, then doesn't Figure 10 also suggest a similar result occurs when
using the `full' Nighthawk of all 10 gene types: i.e. it reaches about 90%
of its final value after about 10% of the generations?  In other words,
the ability to achieve 90% of the best coverage in 10% of the time isn't
clearly shown to be a property of using FSS, but rather a property of the
algorithm as a whole (which is retained when applying FSS).  In which
case, what is the key advantage of using FSS in this context?
\end{verbatim} \normalsize

We have revised some of the text associated with figures 10-11 in order to
state our conclusions more clearly.  Eliminating genes simplifies the code,
with a penalty in coverage, but with the added benefit of reducing run
time.  If reduced runtime is the goal, however, it is indeed not clear
whether this is better achieved by eliminating gene types or by simply
stopping earlier.


\small \begin{verbatim}
(3) The results suggested that numberOfCalls was initialised
inappropriately, and that once this was changed, it was no longer an
important gene type.  What evidence is there that this is now also true of
the gene types now considered most significant in NightHawk 1.1.  In other
words, is FSS just identifying those initial values that are set
inappropriately rather than those that are genuinely important throughout
the search?  (I don't think this needs necessarily to be answered via
experimentation, but it should be acknowledged.)
\end{verbatim} \normalsize

Our inspection of the data, with various visualizations that we did not
include in the paper, suggests that there are no other genes that have a
large enough effect to be suspicious.  However, we have not done
systematic experimentation, so we have included a caveat about this in the
current draft.

\small \begin{verbatim}
(4) Section V has a title 'Properties of the Optimized System' following
on from the previous section IV titled 'Analysis and Optimization of the
GA' that applied FSS.  To me, this suggests that FSS was applied to the
algorithm for the results in section V, but this does not appear to be the
case (and if it was, it is not clearly stated).  The only reference to FSS
is for the Apache Common Collections were *only* 2 gene types were
omitted, and no reference to applying FSS is made for java.util.  If FSS
were not applied more aggressively in these concluding experiments, then
why consider FSS in the first place?
\end{verbatim} \normalsize

The FSS analysis led us directly to the conclusions that the initial value
for {\tt numberOfCalls} was too low, and that the gene types
{\tt candidateBitSet} and {\tt chanceOfNull} were consistently not useful.
In order to compare the new initial value for {\tt numberOfCalls} with
the original value, we compared versions 1.0 and 1.1 on {\tt java.util}
with nothing else changed, since we did not want to introduce confounding
factors.  We could go on to eliminate more genes, at the risk of extending
the paper and reiterating the discoveries discussed in section IV-D, but
we did not think that the referees would be interested in that.  We
therefore knocked out only the two genes consistently found to be not
useful before running Nighthawk on the Apache Commons Collections
classes, in order to report some data from a conservatively reduced
version of Nighthawk 1.1.

This may be a glass-half-empty/glass-half-full thing:  the referee may be
disappointed that we were not able to, say, eliminate most of the genes
and still achieve high coverage and reduced runtime.  We, on the other
hand, were happy to achieve (a) a system which was simplified by the
elimination of two gene types, (b) that still achieved high coverage, (c)
without any one gene type drowning out the effect of others due to being
initialized wrongly.  Without the FSS analysis we would not have been
able to do any of this.

\small \begin{verbatim}
\end{verbatim} \normalsize



\small \begin{verbatim}
Minor Comments
-------------------
\end{verbatim} \normalsize



\small \begin{verbatim}
(a) Keywords: keyword B.7.3.e (Test generation) refers to *hardware* not
*software* testing
\end{verbatim} \normalsize

This is clearly a mistake- thank you for picking this up.
We really should have used D.2.5 (testing tools)- this has been added
to the main document.

\small \begin{verbatim}
(b) p3, line 24 "In previous research, value reuse has mostly taken the
form of making a sequence of method calls all on the same receiver object"
The *implication* is that this work considers different receiver objects,
in which case this could be stated for clarity.
\end{verbatim} \normalsize

Actually, what we intended to imply was that we extend value reuse
to things other than receiver objects.  This has now been stated
explicitly.

\small \begin{verbatim}
(c) p5, line 15 "... and and the question ..."
\end{verbatim} \normalsize

Thanks; fixed.

\small \begin{verbatim}
(d) p15, line 13 "(the additional coverage .... is probably > 0)" - not
sure of the justification nor relevance of this line (since it is clear
which SUT is being used - the triangle program) - suggest omitting it
\end{verbatim} \normalsize

We have clarified this paragraph -- we meant that the additional
coverage would probably be nonzero regardless of what implementation
of the {\it specification} was used.

\small \begin{verbatim}
(e) p17, line 53 "further research work ... such as simulated annealing" -
arguably wouldn't the SA also be using "useless genes" like the SA.  No
change suggested, just making the point that a comparison could still be
made without applying FSS
\end{verbatim} \normalsize

OK.

\small \begin{verbatim}
(f) p19,line 24 "the fewer are the demands on interfaces to the external
environment" - unclear what this means in this context
\end{verbatim} \normalsize

We have deleted this confusing sentence.

\small \begin{verbatim}
(g) section IV.B might be easier to understand if a brief explanation of
what the groups and instances of RELIEF map to in the case of Nighthawk.
\end{verbatim} \normalsize

More notes on this have now been added to section IV.B and the beginning
of IV.C.


\small \begin{verbatim}
(h) p20, line 39 "For the two BitSet gene types, we printed only the
cardinality of the set."  Why?
\end{verbatim} \normalsize

Now addressed in the text.  (We needed to treat the genes consistently by
printing one number representing each gene.)

\small \begin{verbatim}
(i) p21, line 20 If I understand correctly "gene type" here does NOT refer
to class/primitive types, but instead to the 10 categories of genes listed
in Fig 6.  In which case "categories" or similar might have been a better
choice of terminology to avoid confusion with class/primitive types (but a
wholesale change of terminology is not required at this point). 
Similarly, the notation t has previously been used to refer to
class/primitive types.
\end{verbatim} \normalsize

It would be difficult to change these points without a clash of
terminology with previous papers.  However, we have corrected the
footnote about the word ``type'' to say that that use of ``type''
applies only to that section.

\small \begin{verbatim}
(j) p21, line 47 "version 1.0 of Nighthawk" - while the meaning of 1.0 is
explained later, this is the first reference to the versions and ideally
the relevance could be explained
\end{verbatim} \normalsize

Thanks; these references have now been modified and now the numbers
are not used until after they have been introduced.

\small \begin{verbatim}
(k) p24, line 52 "does show a statistically significant difference ..."
suggest adding "at the 5% level" 
\end{verbatim} \normalsize

OK, fixed.

\small \begin{verbatim}
(l) p24 line 55 what metric is used for "time taken", and why doesn't AUC
adequately cover both coverage and time taken (using generations as the
measure of time taken).  In other words, does fixing gene types to
constant significantly change the runtime?  (If so, may be worth giving
some illustration of the magnitude of such changes.)
\end{verbatim} \normalsize

This may have been addressed by our modifications to the relevant graphs.

\small \begin{verbatim}
(m) p29, line 34 This paragraph doesn't really make sense (what is "rapid
turnover"?), and may make too strong claims based on limited results.
\end{verbatim} \normalsize

This has been rephrased to clarify the scope of the claims.

\small \begin{verbatim}
(n) p30, line 39 "Nighthawk v1.0" - small point but this is inconsistent
with section IV where "v" is not used for the version
\end{verbatim} \normalsize

Thanks; fixed.

\small \begin{verbatim}
(o) p30, line 50 The statistical analysis is now appropriate and to be
commended (although, having applied a Wilcoxon test, a SW and t-test are
arguably unnecessary).
\end{verbatim} \normalsize

Thanks; we stuck with both analyses because the $p$ value resulting from
the Wilcoxon test was neither very low nor very high, prompting us to seek
further corroboration from the parametric test.

\small \begin{verbatim}
Reviewer: 3
\end{verbatim} \normalsize



\small \begin{verbatim}
Recommendation: Accept With No Changes
\end{verbatim} \normalsize



\small \begin{verbatim}
Comments:
The authors describe a hybrid system which uses a Genetic Algorithm (GA)
to optimize settings for a random unit testing tool (for JAVA). To improve
the performance of the GA, the authors propose to use Feature Subset
Selection (FSS) to only focus the GA on the most useful genes.
\end{verbatim} \normalsize



\small \begin{verbatim}
Overall the manuscript is very well written and easy to follow. The idea
of using FSS to reduce the size of the search space is indeed very
interesting and it would be very interesting to see if it can be applied
to GA's used to generate test data directly.
\end{verbatim} \normalsize



\small \begin{verbatim}
I only have a few minor comments. The authors state that, in their
original work, the FSS was only able to eliminated 60% of the gene types,
while in this submission they can eliminate 90%. If I am not mistaken,
this equates to eliminating 9 out of the 10 gene types used, thus only
focusing on one feature: the number of method calls to perform. In a way
one can guess that this feature is important since the authors state on
page 6 that they are generating long sequences of method calls. (If long
sequences are really preferable to shorter ones achieving the same
coverage is a matter for debate). My concerns about the new FSS analysis
are the following. The authors state that the reason why the original
Nighthawk underperformed was because of a "bad" initial choice for the
value for this feature (numberOfCalls). They then go on to argue that,
once a better initial value has been picked, the GA optimization does not
statistically significantly affect either coverage or efficiency of the
Nighthawk system (page 30). So why do the authors claim (on page 26) that
changing the initial value of "numberOfCalls" results in a "very
different" behaviour of the Nighthawk system, resulting in two different
versions? The second concern I have is that if 90% of the features can be
eliminated, does Nighthawk still need a GA at all? What would the level of
coverage for Nighthawk be if no GA was used at all and Nighthawk was run
with the more optimal setting of 50 for "numberOfCalls"?
\end{verbatim} \normalsize

On the first concern, this is very true -- we have rephrased that
part in order to not suggest that the behaviour is very different.

On the second concern, our discussion in IV-E is intended to point out
that there are tradeoffs, and that it is impossible to quantify the costs
and benefits completely.  On the one hand, eliminating the gene types
causes greater efficiency and only a small drop in coverage.  On the other
hand, that small drop in coverage may cause Nighthawk to fail to cover
code that would have been very valuable to the user -- as the referee is
probably aware, as coverage increases, every additional percentage of
coverage is more difficult to achieve.  Our modifications to the
discussion of the tradeoffs might help clarify this point.

If there is anything more that we can do to clarify this, we would
appreciate guidance.

\small \begin{verbatim}
Other minor comments:
Page 10:
It's not the case that previous GA based approaches necessarily result in
fixed sized test suites. For example, the work by Wegener et al. (IST,
43(14) 841-854) 2001) or McGraw et al (TSE, 27(12) 1085-1110 2001) takes
"serendipitous" coverage into account. This means that branches which are
covered "by accident" as the GA tries to optimize a target branch are
removed from the list of targets and the GA does not attempt to cover them
again. Since the GA is a stochastic algorithm, during different runs, it
may end up generating different sized test suites.
\end{verbatim} \normalsize

Thanks; we have eliminated the phrase ``fixed size'', since our
point is clear without it.

\small \begin{verbatim}
Page 23:
At the bottom: "The thick black curves on those figures..." -> There seems
to be only one figure with a thick black curve?
\end{verbatim} \normalsize

True; we have replaced ``those figures'' with ``that figure''.

Thanks again to all the reviewers for the time and effort they put in to
reading and commenting on the paper.

\end{document}
