Measurement issues in the comparative manifesto project data set and effectiveness of representative democracy

Abstract

In this dissertation I focus on the very widely used Comparative Manifesto Project (CMP) as the source of measurements of the policy positions of political parties. The CMP data confuse the separable notions of party's position on an issue and the importance of the issue to that party. Furthermore, the CMP data are provided without a basic feature considered essential to any estimate: a measure of the uncertainty surrounding the estimated quantity. This dissertation comprises of three papers. In the ï¬ rst paper, drawing on results from linguistic and behavioural research, I show that party's position on an issue and the importance of the issue to that party are conceptually and empirically distinguishable. I show how to differentiate between position and importance in the CMP data, and contrast this to the saliency-based scaling models currently used by CMP consumers. I evaluate these alternative scales in several replication studies, and propose the use of the existing CMP data that is consistent with the standard spatial models of party competition. The second paper focuses on the analysis of the two main stochastic processes that are involved in the creation of the CMP data: manifesto writing and manifesto coding. Decomposition of the possible stochastic elements in the manifesto generation process that leads to the CMP estimates allows the effects of these to be simulated. Based on these simulation studies, I show how to calculate standard errors for each estimate in the CMP data set. Analysing these error estimates, I show that many CMP quantities should be associated with substantial uncertainty. Next I focus on measurement error arising from stochastic variation in the coding of a given observed text by human coders. I develop a more systematic characterisation of the problems of reliability and bias in the data than has hitherto been attempted. I set out a framework for reliability and misclassiï¬ cation in categorical content analysis, and apply this framework to the CMP coding scheme. In the third paper, I apply the results of the ï¬ rst two papers to address the question of the effectiveness of democratic representation process. In the paper I focus on one linkage element in the chain of democratic representation: between policy positions of political parties and policy output of governments observed in public spending. Using positional scaling models and correcting for measurement error in the CMP data, I show that most of the positive results previously reported in the literature can be explained by measurement issues (scaling and uncertainty) in the CMP data. Moreover, I show that spending on social security is influenced not only by parties that are elected to government, but also by parties in the opposition, thereby undermining the logical consistency of the responsible party model.