This book presents a detailed analysis and review of NLP evaluation, in principle
and in practice. Chapter 1 examines evaluation concepts and establishes
a framework for NLP system evaluation. This makes use of experience in the
related area of information retrieval, and the analysis also refers to evaluation in
speech processing. Chapter 2 surveys significant evaluation work done so far, in
relation both to particular tasks, for instance machine translation, and to NLP
evaluation methodology as such, for example as this bears on generic system
evaluation. The conclusion is that evaluation strategies and techniques for NLP
need much more development, in particular to take proper account of the influence
of system tasks and settings. Chapter 3 develops a general approach to
NLP evaluation, aimed at methodologically sound strategies for test and evaluation
motivated by comprehensive performance factor identification. The analysis
throughout the book is supported by extensive illustrative examples.