This study explores the effects of a number of item characteristics on scoring methods in cloze test performance. The examined item characteristics include: content/function words, parts of speech, word frequency, the number of occurrences of a word in the text, alternative answers, syntactic variation, the amount of context, and knowledge base. The results demonstrate the complex impact of item characteristics and scoring methods on test performance, both in terms of item difficulty and discrimination.