Analyzing Topic Differences, Writing Quality, and Rhetorical Context in College Students’ Essays Using Linguistic Inquiry and Word Count

Abstract

Machine methods for automatically analyzing text have been investigated for decades. Yet the availability and usability of these methods for classifying and scoring specialized essays in small samples–as is typical for ordinary coursework–remains unclear. In this paper we analyzed 156 essays submitted by students in a first-year college rhetoric course. Using cognitive and affective measures within Linguistic Inquiry and Word Count (LIWC), we tested whether machine analyses could i) distinguish among essay topics, ii) distinguish between high and low writing quality, and iii) identify differences due to changes in rhetorical context across writing assignments. The results showed positive results for all three tests. We consider ways that LIWC may benefit college instructors in assessing student compositions and in monitoring the effectiveness of the course curriculum. We also consider extensions of machine assessments for instructional applications.

References

• Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent Dirichlet Allocation. Journal of Machine Learning Research, 3, 993-1022.
• Boot, P., Zijlstra, H., & Geenen, R. (2017). The Dutch translation of the Linguistic Inquiry and Word Count (LIWC) 2007 dictionary. Dutch Journal of Applied Linguistics, 6(1), 65-76.
• Carroll, D. W. (2007). Patterns of student writing in a critical thinking course: A quantitative analysis. Assessing Writing, 12, 213–227.
• Landauer, T. K., Foltz, P. W., & Laham, D. (1998). An introduction to latent semantic analysis. Discourse processes, 25(2-3), 259-284.
• Lord, S. P., Sheng, E., Imel, Z. E., Baer, J., & Atkins, D. C. (2015). More than reflections: Empathy in motivational interviewing includes language style synchrony between therapist and client. Behavior therapy, 46(3), 296-303.
• Lund, K., & Burgess, C. (1996). Producing high-dimensional semantic spaces from lexical co-occurrence. Behavior research methods, instruments, & computers, 28(2), 203-208.
• Lunsford, A. A. (2016). St. Martin’s handbook (8^th ed.): MLA supplement. Bedford/St. Martin’s Press.
• Massó, G., Lambert, P., Penagos, C. R., & Saurí, R. (2013, December). Generating New LIWC Dictionaries by Triangulation. In Asia Information Retrieval Symposium (pp. 263-271). Springer, Berlin, Heidelberg.
• Pennebaker, J. W. (2004). Theories, therapies, and taxpayers: On the complexities of the expressive writing paradigm. Clinical Psychology: Science and Practice, 11(2), 138-142.
• Pennebaker, J.W., Boyd, R.L., Jordan, K., & Blackburn, K. (2015). The development and psychometric properties of LIWC 2015. Austin, TX: University of Texas at Austin.
• Pennebaker, J. W., Chung, C. K., Frazee, J., Lavergne, G. M., & Beaver, D. I. (2014). When small words foretell academic success: The case of college admissions essays. PLoS ONE, 9(12), e115844.
• Pennebaker, J. W., & King, L. A. (1999). Linguistic styles: Language use as an individual difference. Journal of Personality and Social Psychology 77(6), 1296-1312.
• Robertson, K., & Doig, A. (2010). An Empirical Investigation of Variations in Real‐Estate Marketing Language over a Market Cycle. Housing, Theory and Society, 27(2), 178-189.
• Robinson, R. L., Navea, R., & Ickes, W. (2013). Predicting final course performance from students’ written self-introductions: A LIWC analysis. Journal of Language and Social Psychology, 32(4), 469 – 479.
• Taraban, R., Pittman, J., Nalabandian, T., Yang, W. F. Z., Marcy, W. M., & Gunturu, S. M. (2019). Creating and testing specialized dictionaries for text analysis. East European Journal of Psycholinguistics, 6(1), 65-75.
• Tausczik, Y. R., & Pennebaker, J. W. (2010). The psychological meaning of words: LIWC and computerized text analysis methods. Journal of language and social psychology, 29, 24-54.
• Van Wissen, L., & Boot, P. (2017, September). An electronic translation of the LIWC dictionary into Dutch. In Electronic lexicography in the 21st century: Proceedings of eLex 2017 Conference (pp. 703-715). Lexical Computing.