Speech technology is an engineering discipline, and as such has to meet a clear requirement: it must work. This apparently should set clear criteria for what is acceptable for speech technological implementations. However, speech is also an expression of language, and is therefore inherently diverse and variable and the true answer may not alway be well defined. The acceptability of a technology is therefore not purely a binary decision, but should be measured on a continuous scale of usefulness. Speech technology can only move forward if its performance can be evaluated in a way that better performance corresponds to a more useful application. This allows the researcher to measure small improvements in utility, and cumulative improvements will gradually move the technology forward to be more useful.

This lecture reviews the general requirements for the specification of a clear task, metric and methodology in speech technologies. It will do this by analysing some existing evaluation paradigms, and looking at what these have meant to the direction of the research field as a whole.