ASSESSING COLLEGE WRITING: DO STUDENTS CONNECT WITH THE TEXT?

Reading-response research has shown that students respond to a text by engaging various cognitive and emotional processes. The aim of the current study was to examine students’ written reactions to an assigned reading as a way to determine (1) whether students connect with the reading and (2) the differing cognitive styles they may utilize in their reactions. The methods applied two text-analytic procedures to 238 student reactions to an ethics case study. The procedures were language style matching, which is a metric of engagement, and the categoricaldynamic index, which is a metric of analytic and experiential processing. We predicted that students who more strongly connected–or engaged–with the text would also demonstrate greater analytic thinking in their written response and, conversely, those who weakly connected with the text would express a more informal response based on experience. The data were analyzed using correlation statistics. The results showed that students whose writing more closely matched with the linguistic style of the case study were more likely to use an analytical style of writing, and students whose writing weakly matched the linguistic style of the case study were more likely to use an informal narrative style of writing. Future research should examine the extent to which language style matching and an associated analytic cognitive style are emergent skills that develop over the course of a college experience.


Introduction
In college instruction, there is a triarchy of activities consisting of reading, writing, and assessment. College instructors often ask students to read textbooks, novels, poetry, journal articles, and case studies. Typically, these reading assignments are coupled with classroom tests or writing assignments and are followed with instructor assessments. Some assessments are rather superficial and straightforward, like those involving multiple-choice tests, in which there is essentially no writing, and the assessments are focused on content knowledge. In more substantive contexts, students are required to do background reading followed by reflective writing, consisting of submissions to course discussion boards, reaction papers, essays, research reports, or theses. In a typical written composition, assessment evaluates mechanics, like spelling, vocabulary, and grammar; quality, including organization and fluency; and content, including accuracy, coverage, creativity, insight, and critical analysis.
Beyond mechanics, quality, and content, there are other more subtle aspects of writing assessment. In their compositions, an instructor could ask, for instance, did students identify with the characters in a novel; did they empathize with a victim in a newspaper article; did they agree with an editorial; and did they react to and contest the claims of a journalist? These issues fall under the category of reader responsiveness -the "cold and hot cognition" (Wyatt et al., 1993) that students experience in reading. According to reader-response theory (Mart, 2019;Rosenblatt, 2016), readers are active, not passive. Readers question, react, respond, summarize, criticize, hypothesize, interpret, and analyze texts. There is an active relationship between reader and text that develops over the course of reading.
The general research question in this study is whether students, after reading, connect with the text in their written reactions. There is a basis in reading-response theory for expecting that students would develop a relationship to the text while reading. The question addressed here is whether this relationship is also evident in their written reactions. A number of studies using text analytic methods have shown that people's thinking is reflected in the language that they use (Jordan & Pennebaker, 2019;Jordan, Sterling, Pennebaker, & Boyd, 2017;Pennebaker, Chung, Frazee, Lavergne, & Beaver, 2014), therefore, it should be possible to recover students' reactions to a text by analyzing the language that they use in expressing those reactions.
The work of Pennebaker and colleagues has shown that the small words that individuals use are associated with differences in confidence, leadership, power, status, analytic thinking, and academic success, among others (Ireland & Pennebaker, 2014;Jordan et al., 2019;Kacewicz, Pennebaker, Davis, Jeon, & Graesser, 2014;Pennebaker, 2011;Pennebaker et al., 2014). The small words are generally function words, such as pronouns, articles, prepositions, and conjunctions. In the present paper, we apply two well-researched algorithmic procedures from the Pennebaker corpus of research. One procedure measures language style matching (Ireland & Pennebaker, 2010). The second procedure measures categorical-dynamic processing . These two procedures provide the primary data for the current study. We analyzed students' written reactions using these two linguistic metrics to examine i) students' level of connection or engagement with an assigned reading, and ii) what cognitive styles students employed in their reactions. In the next subsections, we present an overview of these procedures, before turning to an analysis of students' compositions.

Language Style Matching
Language style matching is a form of verbal mimicry in which pairs or groups of individuals use language in a similar fashion. In this paper, we use the acronym LSM to refer to a method to assess those stylistic similarities in language use among individuals. Rather than focusing on what individuals say (i.e., the content words), LSM analyzes the way individuals say things. Specifically, LSM examines the similarity of function word use between individuals or texts. Function words, such as articles (e.g., a, an, the), tend to be small words that are easily processed (Segalowitz & Lane, 2000) and not necessarily used with conscious awareness (Pulvermüller et al., 2008). They cover more than half of the vocabulary of daily speech (Pennebaker, 2011) and correlate with social behaviors as diverse as social bonding and deception (Chung & Pennebaker, 2007). In brief, function words serve as a convenient and informative correlate of psychological processes.
LSM has been studied in various contexts, such as attraction, conflict, mental health, social media, and police interrogations. As an example of LSM in interpersonal attraction, speed-dating couples who used similar function words during their date were more inclined to interact with their partner at a future time and remain in a relationship months after their initial encounter (Ireland et al., 2011). Considering LSM in interpersonal conflict, an archival analysis of poems written by literary couple Sylvia Plath and Ted Hughes showed that the couple's language style was more likely to match during the positive periods of their relationship and less likely to match toward its dissolution (Ireland & Pennebaker, 2010). Regarding LSM in mental health, analysis of email conversations demonstrated that depressed individuals, who tend to socially withdraw and isolate themselves (Segrin, 2000;Segrin & Abramson, 1994), were less likely to linguistically match or engage with their correspondents (Baddeley, 2011). Further, LSM in social-media user communities has been shown to predict community identity and to affect quantity and quality of participation (Ludwig et al., 2014). In police interrogations, criminal suspects who ultimately confess showed increased language style matching as the interrogations unfolded (Richardson et al., 2014). Suspects who did not confess did not show a pattern of increased language style matching. 1 Thus, scores of research has established LSM as providing a reliable marker of engagement (Niederhoffer & Pennebaker, 2002).
Calculations within the LSM algorithm are comparable to those of an inverse Canberra distance metric (Lance & Williams, 1967). The algorithm determines the degree of function word similarity between two individual texts or groups of texts (Ireland & Pennebaker, 2010). The calculation of LSM for a pair of texts requires several steps. First, one sums the percentages of nine function word categories for each of two texts one wants to compare. The nine categories are i) personal pronouns, ii) impersonal pronouns, iii) articles, iv) prepositions, v) auxiliary verbs, vi) adverbs, vii) conjunctions, viii) negations, and ix) quantifiers (see Table 1 for examples). Next, one applies equation (1) to each of the nine function word categories to calculate partial LSM scores for each category. Equation 1 is expressed using the adverb category as an example: LSM adverb = 1 -[(|adverb text 1 -adverb text 2 |)/(adverb text 1 + adverb text 2 + .0001)] (1) The LSM score for each text is the average of the LSM scores for each of the nine specified function word categories: (2) LSM scores range between the values of 0 and 1, much like correlations, where higher values denote greater similarity between texts.

Categorical-Dynamic Index
In addition to measuring linguistic synchrony between student responses and their assigned reading, it is also possible to uncover varying cognitive styles apparent in student responses using a metric known as the Categorical-Dynamic Index (CDI). The CDI assesses the degree to which a person relies on analytic thinking or narrative thinking . In particular, the categorical component of the CDI represents analytic thinking, a deeper level of reflection and logic-based reasoning (Jordan & Pennebaker, 2017). Categorical writing, or analytic thinking, is linguistically defined by higher frequencies of articles and prepositions . Because articles-indicative of abstraction-help introduce nouns, while prepositions-indicative of complexity-help associate nouns and pronouns with other words, these grammatical categories are characteristic of more formal or analytic thinking . In contrast, the dynamic component of the CDI represents narrative thinking, which is much more instinctual and experience-based (Jordan & Pennebaker, 2017). Dynamic writing, or narrative thinking, is linguistically defined by higher frequencies of pronouns, adverbs, auxiliary verbs, negations, and conjunctions . Such function words serve to identify people (e.g., pronouns) and their various actions (e.g., adverbs; Blackburn, 2012) and are reflective of a more informal writing style that refers to the "here and now" (Pennebaker et al., 2015).
Similar to the body of literature surrounding LSM, research examining the CDI has been applied to different contexts, such as academics, politics, and gender. For instance, the CDI was employed as a correlate of academic success, wherein students who used greater categorical language in their college admission essays were more likely to achieve higher college GPAs . On the other hand, students who incorporated greater dynamic language in their essays were more likely to have lower GPAs . Additionally, studies assessing political language using the CDI have found that language used by Trump, other US presidents, and non-US political leaders (obtained from public speeches, debates, etc.) tends to be less analytic, or incorporates fewer articles and prepositions (Jordan & Pennebaker, 2017;Jordan et al., 2019). Moreover, other research demonstrates how the grammatical categories that make up the CDI reliably reflect gender. Women tend to use language consistent with the dynamic component, while men tend to use language consistent with the categorical  , 2008;Pennebaker et al., 2014). Women's use of dynamic language, which focuses more on people, is consistent with feminine stereotypes relating to affiliation, whereas men's use of categorical language, which is less personal and focuses more on objects, is consistent with masculine stereotypes relating to assertion (Leaper, 2014). Calculation of CDI for a text or group of texts uses the same function word categories as LSM, excluding quantifiers, and is carried out in two steps. First, the frequencies of the eight function word categories are tabulated. Next, the frequencies are then combined according to Equation (3): CDI = (30 + articles + prep -ppron -ipron -auxverb -conj -adverb -negate) (3) (Note. prep: preposition; ppron: person pronouns; pron: impersonal pronouns; auxverb: auxiliary verbs; conj: conjunctions; negate: negations) Higher scores on the CDI indicate higher rates of categorical language and lower rates of dynamic language, while lower scores on the CDI indicate lower rates of categorical language and higher rates of dynamic language.

Methods
The current study used a dictionary-based method of computerized text analysis to analyze student responses to a case study presenting an ethical dilemma in the field of engineering, as described in more detail below. LSM was used to establish whether students connected with the text. The CDI was used to further analyze the synchrony between students and text. The categorical component of the CDI is based on analytic thinking which is characterized by logic-based reasoning and deliberation. We therefore hypothesized that 1. Students who linguistically matched more with the given case study would be more likely to score higher on the categorical component of the CDI (i.e., higher analytical thinking) in their responses. 2. Conversely, we hypothesized that students who linguistically matched less with the case study-who tend to adopt an informal language style-would be more likely to score higher on the dynamic component of the CDI (i.e., provide a more informal gut reaction of the case study).

Writing Sample
The writing sample in this study was comprised of undergraduate student responses (N = 238) to an online case study titled "Which is More Important: Environmental Concern or Economic Growth?" authored by Dr. Sudipta Majumdar (Faculty of Management Studies, ICFAI University, Jharkhand, India). The case study consists of 503 words, with a Flesch-Kinkaid Grade Level of 10.00 and a Flesch Reading Ease score of 54.7 1 and depicts differing environmental and economic consequences of a company drilling for oil in the rural town of Maharashtra, India. Guidelines were provided on the website for composing a response (e.g., determining what knowledge, skills, interdisciplinary perspectives, and cultural insights are necessary to establish viable solutions). On average, student responses consisted of 340.54 words (SD = 101.84). The case study, guidelines, and student comments can be found at https://ethicalengineer.ttu.edu.

Procedures
The independent measures in the present study include the percentages for the nine language categories in Table 1 within each student's response. Percentages were computed using the Linguistic Inquiry and Word Count (LIWC; Pennebaker et al., 2015) software, and were then applied to Equations 1 and 2 in order to estimate language style matching between each student's comment and the case study "Which is More Important: Environmental Concern or Economic Growth." This variable is termed LSM-Case Study. In order to provide a contrasting measure for LSM-Case Study, the mean percentages for the nine language categories were computed across the corpus of 238 comments. LSM was then calculated, again using Equations 1 and 2, between each student's comment and the overall mean. This variable is termed LSM-Peers. Importantly, LSM-Peers provides a sense of the student vernacular, that is, the typical speech patterns used in the corpus of comments. Further, Equation 3 was applied to create a measure of CDI-Overall for each student comment, and Equations 4 and 5 to create measures of CDI-Categorical and CDI-Dynamic, respectively, for each student comment. The means and standard deviations are shown below in Table 2. Prior to calculating correlation coefficients, the data were checked for normality. All variables were normally distributed except LSM-Peers, which was negatively skewed. Upon further examination of the data, the negative skew appeared to be influenced by a univariate outlier. Therefore, the outlier was excluded from the sample, which effectively reduced the negative skew of LSM-Peers, transforming the variable to a more normal distribution. Values for all variables were then standardized based on z-scores.

LSM and CDI
To test associations between LSM and the CDI, Pearson correlation coefficients were calculated using IBM SPSS v. 24 1 . Adopting the conventional standard in statistical analysis, correlations with p-values less than 0.05 were considered significant. There was a significant negative correlation between LSM-Case Study and LSM-Peers. Although we had not made a specific prediction, this finding shows that students who were higher in LSM-Case Study were lower in LSM-Peers. In other words, being in synchrony with a text differs from writing in a typical peer manner, and that connecting with a text represents a specific style or genre of writing. Further support for a dissociation between connecting with a text and writing like a peer is found in the highly significant positive correlation between LSM-Case Study and CDI-Overall, and a significant negative correlation between LSM-Peers and CDI-Overall. An examination of Equations 3, 4, and 5 indicates that CDI-Overall is a strong measure of analytic thinking not of a narrative reaction. Therefore in general, the pattern of correlations suggests that the typical vernacular of students is somewhat disorganized and incoherent and that, out of this corpus of writing, some student responses emerge as more closely matching and connecting to the text and more amenable to an analytic (categorical) style. Turning to our specific predictions, hypothesis 1 is supported by a significant positive correlation between LSM-Case Study and CDI-Categorical. The correlation shows that students who linguistically matched more with the given case study were more likely to score higher on the categorical component of the CDI, illustrating greater analytic thinking in their responses. Hypothesis 2 is also supported, as indicated by a highly significant negative correlation between LSM-Case Study and CDI-Dynamic as well as a significant positive correlation between LSM-Peers and CDI-Dynamic. These results indicate that students who linguistically matched less with the case study and more with their peers were more likely to score higher on the dynamic component of the CDI, providing a more informal gut reaction of the case study. As additional support for the reliability of these findings, CDI-Overall is significantly correlated with CDI-Categorical and significantly negatively correlated with CDI-Dynamic, and CDI-Categorical and CDI-Dynamic are significantly negatively correlated. In summary, students who shared greater linguistic style with the case study were more likely to use a categorical or analytical style of writing, and students who shared greater linguistic style with their peers were more likely to use a dynamic or narrative style of writing.

Qualitative Examples
The pattern of significant correlations between the LSM and CDI variables may implicate LSM as a potential linguistic marker of academic writing quality. For instance, a student whose response was higher on CDI-Overall (40.00) and whose language style highly matched that of the case study (LSM-Case Study = 0.73) analyzed the case study in more depth and successfully addressed most aspects of the assigned prompt (i.e., discussed how the stakeholders were impacted, how cultural insights and interdisciplinary perspectives provide viable solutions, their ethical decision; see Excerpt 1). In contrast, a student whose response was lower on CDI-Overall (26.73) and whose language style highly matched that of their peers (LSM-Peer = 0.93) discussed the case study with much less detail and only addressed two of the prompt questions regarding stakeholders and their ethical decision (see Excerpt 2). In fact, this student's focus on stakeholders (i.e., how people in the case study were affected) is quite consistent with dynamic (narrative) language. The juxtaposition of the two student responses emphasizes not only their linguistic differences but how such linguistic differences might inform the quality of students' written work. However, our current design does not include measures of assessment quality. Thus, future research must focus on operationally defining assessment quality and whether LSM serves as a reliable and valid predictor of the construct.

Conclusions
Reader-response research has shown that students interact with a text in many different ways (e.g., cognitively, emotionally). The present research builds on reader-response theory by examining student responses to an ethics case study to determine their level of engagement with the case study and how such engagement could inform cognitive styles in their responses. We analyzed the association between language style matching (of student responses and the case study) and the Categorical-Dynamic Index, which-to our knowledge-has not yet been examined in the current literature. Our results demonstrated that students whose responses linguistically matched with the case study (or assigned reading) were more likely to use an analytic language style in their reactions. Conversely, students whose responses linguistically matched with typical peer responses were more likely to adopt a narrative, non-analytic, reaction to the case study.
Informal qualitative analyses of specific responses, guided by the patterns in the correlation results, suggested that language style matching might serve as a correlate of writing quality. Greater synchrony with an assigned reading and pronounced use of analytic language in one's reading response may be reliably associated with the quality of said response. Furthermore, college students, on average, may use a more informal language style early on in their academic career. Resonating and responding to an author and text in an analytic manner may be an emergent skill that develops with time and practice. Future research will focus on establishing whether the quality of student responses can be recovered from measurements of language style in student responses.
The two types of cognitive styles conveyed in the student responses considered here (i.e., analytic vs. narrative) demonstrate an interesting dichotomy in communication. Some types of problems, such as those that are well-defined and objective, may lend themselves to an analytic response. Relatedly, some contexts demand logical and measured analysis, such as those in science. In other instances, like everyday conversation, email, and social media, there is more latitude for conversational discourse and gut reactions. Students in the present academic context may face a dilemma in deciding whether to frame their responses in a more formal analytic manner or a more informal personal manner.