If speech analysis is to detect a speaker’s emotional state, it needs to derive information from both linguistic information, i.e., the qualitative targets that the speaker has attained (or approximated), conforming to the rules of language; and paralinguistic information, i.e., allowed variations in the way that qualitative linguistic targets are realised. It also needs an appropriate representation of emotional states. The ERMIS project addresses the integration problem that those requirements pose. It mainly comprises a paralinguistic analysis and a robust speech recognition module. Descriptions of emotionality are derived from these modules following psychological and linguistic research that indicates the information likely to be available. We argue that progress in registering emotional states depends on establishing an overall framework of at least this level of complexity.