Subject: IJCAI-97 WORKSHOP REVIEW FORM Comments for Author Title: Redesign Problem Solving Number: ?? 1. How RELEVANT is this paper to AI researchers? (mark one box) [x] Very relevant [ ] Moderately relevant [ ] Not relevant Please explain your rating: Fits with the CFP 2. How SIGNIFICANT is this paper? [x] Very significant [x] Moderately significant [ ] Not significant Please explain your rating: The framework it offers seems very powerful. In particular, the dimensions of redesign in section 3 seemed very insightful. At some stage, the practical utility of this framework will be tested and then this paper will be seen either as a landmark publication or... So I would rate it between very and moderate 3. How ORIGINAL is this paper? [x] Very original [x] Moderately original [ ] Not original Please explain your rating: In some respects, this paper is "only" a clear summary of other work. However, the "summary" gives clear insights into the redesign process and could be basis of detailed implementations in time to come. So, while it is "only" a synthesis of other work, it is a very impressive synthesis of other work. 4. Is this paper technically SOUND? [x] Yes [ ] Seems valid, but did not check completely [ ] Has minor errors [ ] Has major errors If there are errors, please identify them: 5. How well is this paper PRESENTED? Good Average Poor Overall organization [x] [ ] [ ] English [x] [ ] [ ] Readability [x] [ ] [ ] If there are presentational problems, please identify them: I would really have liked to given this paper an "excellent" rating for all the above. 6. Further comments, advice or explanations (Please be specific and constructive, especially with respect to any negative judgments above. Point to the section(s) where an error occurs, cite omitted references, etc.) Use as much space as you need. Nice paper++. I was wondering if you have missed an important message in this paper. While I can see the clear connection of this work to the redesign literature, I am wondering if "redesigning a PSM" is a synonym for "knowledge maintenance" (KM). Would your paper have a bigger impact if it was called "Knowledge maintenance = PSM Redesign" or "Extending KA techniques to KM"? Similarly, the connection between "redesign" and "requirements engineering" (RE) should be explored. redesign is changing a current system, RE is changing a conceived, but not- as-yet realised system. both are changing the conceptual schema of an artifact. can't help thinking that "RE=redesign=KM". Some extensions to your framework and one concern: EXTENSION #1: I was wondering about an extension to your table 1 to take into account other KM work, namely RDR and RIME. The extension would be "what is redesigned". Take Newell's original KL proposal: search space, plus operators to implement rationality (reasonable choices when faced with options). What can be redesigned? I can think of at least four levels: 1) the terms 2) the assertions which use the terms. these assertions define the search space. 3) the library of rationality operators. we can distinguish at least three types of rationality libraries: KL-A, KL-B, case-based. From my paper: "We distinguish between SOAR/PSCM and HT4/abduction (which we term KL-A) and KL-B, a KL-modeling variant which groups together a set of authors who argue for basically the same technique; e.g. Clancey's model construction operators~\cite{clancey92}, Steels' components of expertise~\cite{steels90}, Chandrasekaran's task analysis, SPARK/ BURN/ FIREFIGHTER~\cite{marques92} and KADS~\cite{weil92}\footnote{See the {\em Related Work} section of~\cite{weil92} for a discussion of the differences in these techniques}. The fundamental premise of KL-B is that a knowledge base should be divided into domain-specific facts and domain-independent problem solving methods. Such problem-solving strategies are implicit in KL-A. The observation that a PSCM system is performing (e.g.) classification is a user-interpretation of a single inference procedure (operator selection over a problem space traversal)~\cite{yost89}." Ripple-down-rules (RDR) is an example of case-based rationality. There is no explicit PSM libraries (like KL-B) or even a single inference engine called in multiple ways (like KL-A). Instead, RDR just works in in the context of the current example. 4) the library of known/desired behaviour. i quite take your point that (p11) you can't assess without a library of specific examples. now, with this extra dimension, we can characterise different KM/redesign approaches: 1) shaw's repitory grids tell us how to ACQUIRE the terms, but not how to MAINTAIN them. rep grids do not address redesign of 2,3,4. 2) numerous proposals in the requirements engineering RE community are focused on ACQUIRING and MAINTAINING (2), but ignore 3,4. 3) your approach, and that of Fensel, is 3, in a KL-B framework. RIME is 3 in a KL-A framework. RDR and case-based reasoning CBR are the only case-based KM I know off. as near as i can tell, RDR, CBR, you, Fensel, RIME does not deal with MAINTAIN 1,2,4. but of course, CBR does not care about maintaining 2,3 since that is irrelevant to its processing. RDR does not care about 2, but it should. if the "meaning" of the terms in the RDR rule LHS change over time, RDR won't work. 4) numerous proposals in the RE community deal with ACQUIRING 4, but not how to MAINTAIN it. This could be a major problem, particularly when 1 changes. HT4 assumed that 4 came from known behaviours of the device being modeled. EXTENSION #2: And another dimension: how often does redesign occur? in RDR, its all the time. Perhaps redesign is that kind of KM that happens only a few times in the systems lifecycle. EXTENSION #3: When assessing different redesign options, the experience of RE is that some style of multiple worlds reasoner is a useful addition to the architecture, especially when you now have to trade off assumptions supporting one desired result with assumptions that support other desired result, while blocking this one. Internally, this trading off process implies moving a single-world reasoning device to a multiple worlds reasoning device. i.e. the "theorem prover" you need here is not a single world deductive device, but a multiple world device. note that such multiple world devices divide into two broad groups: JTMS (doyle's approach) vs ATMS (deKleer's approach). in the JTMS, you leap from world to world but can only hold a single consistent belief set (world) at one time. in the ATMS, all worlds are available for reflection at one time. i.e. the ATMS is parallel worlds development while the JTMS is serial worlds development. My own (untested) conviction is that the ATMS approach is better. It seems to me that if you are going to reflect over different options (worlds) and trade them off, you need to have access to those options. That is ATMS better than JTMS. the down side of this belief is that the ATMS is abduction, np-hard, slower while the JTMS is a extension of deduction, can be faster and lots of current theorem provers can be extended to a JTMS approach. CONCERN: I can't stop being the large Australian crying "how do you validate this". sorry, but I'm always worried about "ontological recursion". from my perspective, wherever we direct the attention of KA, we find more and more PSMs and more ontologies. further, the harder we look, the more detailed they get and the more we have to find YAO (yet another ontology) for this bit inside here. i often wonder if this ontological recursion will ever end? for my part, i keep looking for computational commonalities between PSMs looking for a simpler (not more complicated) ontology to rule them all. my current efforts are focused on this abductive stuff i keep talking about but there is no reason to suppose that other unifying frameworks may not exist. in any case: given two options in ontologies and PSM for redesign, how will you assess them? more generally, how will you assess that this approach is ultimately more productive than current practice? my answer to these questions is already documented. what is your reply? ===================================== Comments for Program Committee Members Only (This section of the review will be withheld from the author eehm: Only if he is NOT a member of the organisation committee) Title: Redesign problem solving Number: ?? 1. My recommendation is: [x] Accept [ ] Leaning to accept [ ] Leaning to reject [ ] Reject 2. How confident are you in your appropriateness as a referee for this paper? [ ] Very confident - I am an expert in this area. [x] Confident - I have a reasonable knowledge of this area. [ ] Fairly confident - I have some knowledge of this area. [ ] Not confident - I have no significant knowledge of this area. 3. If this paper is marginal or unusual in some way, can you comment on anything else that might help the Program Committee reach a decision? Reviewer: (please fill in your name here) Tim Menzies