Project 4b

1. For the POM2 model, comparatively evaluate five different search engines (of your choice) and select which one you prefer. Evaluation criteria are:
   • Fastest time to plateau
   • Highest plateau;
   • Smallest spread of performance at the plateau
   • Smallest size of treatment (measured in #choices).
   • Smallest size of treatment (measured in #ranges).
2. Using your search engine of choice, identify the best performance at the sweet spots for these different policies (PB, AG, AG2, HY) and to compare the performance of pairs of results (i.e. find what works best for AG, then applies to AG2 to see what is won/lost in that comparison). Note there the comparisons may be asymmetrical so there are 12 such pair comparisons (not six).
3. Bonus marks: given the complexities of POM2, does PG, AG, AG2, HY really matter? Or do are they project features that render irrelevant issues of the task selection policy. development policy.
4. Bonus marks: is there anything better than PB, AG, AG2, HY?

Presentation and Paper

You marks are divided 20 marks, 30 marks amongst a week14 presentation and, two weeks later, the submission of a written report.

• April 14: presentations. The purpose of the presentation is to show your findings and get feedback from me on what to fix, what to rerun, what to expand, what to contract in your final report.
• April 30: written report. The purpose of the written report is to train graduate students in writing research reports for senior academic forums. For an example of a paper of the kind I want you to write, see http://menzies.us/pdf/07casease.pdf.

Note that I intend to take one group and work with them for a submission to the IEEE ASE conference (2009). On that paper all members of that group and myself will be co-authors. Students that show a deep understanding of POM and POM2, and who make substantial contributions to the work (as judged by me) will appear before me in the author list.

Reports must be in Latex, two column 10 pages (including all figures and references and related work) and follow the format of http://www.acm.org/sigs/pubs/proceed/template.html.

The paper and presentation must be in pdf format. The source code for the presentation and pdf must be kept in your project svn. Any result charts must be generated by scripts and the data/scripts data used in that scripts must be stored in a sub-directory data/scripts (this will allow me to tinker with the format after you are gone).

Important notes: in years gone by I have have not been 100% excited by student presentations. This changes, right now. Each presentation will be:

• high quality,
• well rehearsed,
• 45 minutes long
• include details of motivation, models, algorithms, experimental methods, statistical analysis of results, conclusions and recommendations, implications and proposed future work.
• Your presentation must be made by all members of the group.
• Your presentation must be uploaded as a pdf file to the google groups before the day of the presentation. It is unacceptable to spend 5 minutes fussing around at the start of your presentation copying over files from a thumb drive.