Benchmark analysis

This section is concerned with the statistical evaluation of the benchmark experiments and drawing inferences in a proper way. Common questions under examnination are (after selecting apeformance measure)

• Is classifier A significantly better than classifier B
• Given a group of classifiers, can identify relations where

We won't go into the theoretical details here, but we will try to link to some papers which are helpful:

• On comparing classifiers: Pitfalls to avoid and a recommended approach (1997) link
  Basic paper by Salzberg. Good introduction.
  
  
• Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms (1998) link
  Basic paper by Salzberg. Good introduction.
  
  
• Benchmarking group / University of Munich
  
  The design and analysis of benchmark experiments (1998) link
  Tries to establish a general, statistical valid fraework of the theory. Also contains a good overview and critique of prevoius methods and tests.
  
  Exploratory and Inferential Analysis of Benchmark Experiments link
  Expands on the above papaer, mostly with regard to practical issues.
  
  benchmark package