What is it about?
Judgments are often used in linguistic research. Not much is known, however, about the variation of such judgments within and between participants. From a usage-based perspective, variation might be expected: with judgments based in representations, and representations resulting from input and use, both inter- and intra-individual variation are likely. This study investigates the reliability of metalinguistic judgments, more specifically familiarity judgments, for Dutch prepositional phrases (e.g. op de bank, ‘on the couch’). Familiarity judgments for 44 prepositional phrases offered in isolation and in a sentential context were given by 86 participants in two identical test sessions, using Magnitude Estimation. Aggregated scores (averaged over participants) are remarkably consistent (Pearson’s r = .97), and in part predicted by corpus frequencies. At the same time, there is considerable variation between and within participants. Context does not reduce this variation. We interpret both the stability and instability to be real reflections of language: a relatively stable system in a speech community consisting of speakers who are variable and forever changing. The results suggest that judgment data are informative at different levels of granularity. They call for more attention to individual variation and its underlying dynamics.