Is there Critical Thinking in Popular Comments? A Descriptive Examination through Content Analysis

This study provides a descriptive account of popular comments extracted from four published news articles on an online news platform. This was done through a content analysis on the level of critical thinking found in readers’ comments deemed popular by other readers through the number of responses and likes. Comments from four news articles reporting on the Syrian conflict were analysed using the Newman, Webb, and Cochrane’s (1995) analytical protocol. Results showed that although there were clear indicators of critical thinking in the popular comments, specifically by providing justification and synthesizing comments with external materials. These indicators were of high frequency and common, perhaps due to the nature of the news items under analysis – which is concerned with an ongoing conflict that has political and social bearing on the readers. We conclude by suggesting future research to introduce relevant critical thinking indicators based on contextually-familiar news topics and a bigger data set.


INTRODUCTION
In the era of post-truths and 'fake news', there is a pressing need for news media users to be analytical and responsible (Fletcher & Park, 2017;Pennycook & Rand, 2017;Cooke, 2017).This need is also a response to the way news media is evolving, wherein users themselves are social entities that drive how information is created and understood (Morrison, 2017).Furthermore, news media users' involvement is a crucial aspect for the determination of media quality and meaning-making (Diakopoulos & Naaman, 2011).For example, news items which are salient typically garner more attention from readers.These users may participate in the form of leaving comments, which reflects personal autonomy and initiative on the part of the readers (Hao, Wen, & George, 2014).Comments which resonate with other readers may initiate and extend a discussion, or it may spur actions and civic engagement (Livingstone & Markham, 2008).In light of this, an understanding of users' engagement with news media content is necessary.
To date, studies have examined news media users' involvement through civic participation (Nicodemus, 2004;Livingstone & Markham, 2008;Ksiazek, Malthouse, & Webster, 2010;Nielsen, 2011;de Zúñiga, Jung, & Valenzuela, 2012), consumption of news type (Thorson, 2008), and users' attitude and response towards different media platforms (Fletcher & Park, 2017;Pennycook & Rand, 2017;Cooke, 2017;Morrison, 2017).Nonetheless, minimal studies have examined what makes a comment popular, let alone the dynamism of these comments (Ziegele, Springer, Jost, & Wright, 2017).To address this gap, we examined popular comments through content analysis.Specifically, the content analysis was guided by Newman, Webb, and Cochrane's (1995;1997) framework for evaluating critical thinking, with the assumption that comments that exhibit traits of critical thinking have the intention to resonate with and influence other readers, or news media users.This assumption is drawn based on claims that pertinent types of news issues may call for analysis and discussion between media users, instead of just an isolated and linear consumption of information presented in a news item (see Ksiazek, Peer, & Lessard, 2016;Tenenboim & Cohen, 2015).

Users' Involvement with News Media
Before the advent of online news media, the involvement of news media consumers was very limited.News media and the formation of public opinion was largely controlled by professional communicators -namely writers, journalists and editors, who were responsible for the process of selecting, evaluating, representing, and synthesizing information (Weber, 2014).With the implementation of Web 2.0 technologies, the past few decades have seen the gradual introduction of a variety of interactive features, which has led to wider user participation.Features which readily connect users and grant access to the World Wide Web not only radically changed the way people share, but more importantly, allowed people from all around the world to interact, communicate and collaborate, as well as to take action.
As a prominent form of online mass media, online news websites hold a vital role in facilitating the formation of public opinion because of its relatively larger audience compared to offline newspapers (Gerhards & Schäfer, 2010).This highly interactive medium allows users to be involved, and participate within the three dimensions of interactivity, which are: user-to-user, user-to-document, and user-to-system (Macmillan, 2002).What this has resulted in is the shift from the traditional one-to-many communication flow of the media to the contemporary two-way communication model which essentially propelled readers and users from being passive consumers to active participants (Pavlik, 2001).These characteristics are observable in today's news media, where readers are afforded an opportunity to take part in the development of a news story by leaving comments, or to communicate with other news media consumers.Morrison (2017) states that threads that ensue after a news item is published can be "seen as vibrant, often useful, occasionally crucial extensions of articles, which -taken together with the journalist-authored narratives -have the power to strengthen and/or reshape our understanding of, at times, vitally important public-interest issues" (p.228; emphasis in original).Aside from leaving comments, other forms of engagement may come in the form of liking or sharing (Pennycook & Rand, 2017).When a post is re-shared, or a news item released by other sources, it signifies a level of value, such as being newsworthy or being an appropriate observation of an issue (Neuman, Guggenheim, Jang, & Bae, 2014).
Looking at the bigger picture, the involvement of news media consumers may be grounds for the exploration of agenda setting and agenda building processes.While agenda setting is concerned with media's role in transferring the issue of salience from the media to the public, agenda building describes the different forces that shape an agenda of the media (Weaver & Elliot, 1985;Berkowitz & Adam, 1990;McCombs, 2004;Wong & Lean, 2011;Yang & Leong, 2017).Various media platforms, over the years, have been found to influence how an issue is perceived by the public.These perceptions may trickle into other discourse forms, such as blogs, discussion forums, commentaries, comments on news items, which contain opinions of the public regarding socio-political issues covered by the media (Neuman, Guggenheim, Jang, & Bae, 2014).Drawing links through broader interactions between these different platforms is crucial as it accounts for actors who have the power to change the direction of a discussion, or the way an issue is interpreted (Harder, Sevenans & Van Aelst, 2017).Another feature of news media is its role as a catalyst for civic movement.For example, Nicodemus (2004) showed how a local newspaper was capable of engaging its readers in an issue affecting their livelihoods.This led to the mobilization of media users to participate in civic engagement.The involvement of media users in issues presented through news could also serve as a precursor to their identity.Nielsen (2011) suggests that involved media users are shaping their sense of citizenship as these users are 'active citizens' whose participation is socio-technical in nature.Media users may be involved in civic movement through online commitments (Hao, Wen, & George, 2014) or by participating in social or political movements (Nicodemus, 2004;Livingstone & Markham, 2008;de Zúñiga, Jung, & Valenzuela, 2012).

Comments on News Media
There are several characteristics found to be typical in comments left on a news item.First, the purpose of comments may vary depending on the users' familiarity with the content.For instance, Ksiazek, Peer, and Lessard (2014) reported that users would normally comment on content that is deemed familiar or informative while users may engage with other users through discussion on content that is niche or specific, and allows users to connect on a social level.This brings us to the second characteristic, which is how comments are common in news items which cover public issues, primarily those that are political or social.Comments that address these types of news items are typically motivated by the need to voice one's opinion and to influence others' views.It is interesting to note, too, that while certain news items may be viewed by many, they may not garner many comments or instigate discussions (Tenenboim & Cohen, 2013).For example, foreign affairs, including those that involve issues about national security, do not necessarily draw a lot of attention (Neuman, Guggenheim, Jang, & Bae, 2014), or news items published by outlets which are not trustworthy (Fletcher & Park, 2017).Third, comments left by readers reflect different political ideologies, personal perspectives, and even emotions of the media consumers (Fletcher & Park, 2017), and attests to the demographically varied perceptions towards news media held by the public (Neuman, Guggenheim, Jang & Bae, 2014).It should be noted, though, that comments left do not necessarily challenge the authority of journalists; instead, they acknowledge journalists as having covered issues which are newsworthy, and represent the ongoing nature of conversations regarding important issues (Morrison, 2017).

Skills for New Media Users
Given that current media platforms have been changing the way we interact, and the agenda building role that media users wield in the formation of public opinion, media users are taxed with the necessity of developing the necessary skills needed in order to be relevant consumers of new media.Alexander (2008) states that the Web 2.0 technologies have created a discourse site which consists of "Web pages focusing on microcontent and social connections between people" in which digital content are "copied, moved, altered, remixed, and linked, based on the needs, interests and abilities of users" (p.151).Hence, in order to fully utilize these features made available by Web 2.0 technologies, users need to possess skills which will help them navigate their way around the web, and more importantly, be responsible consumers of the plethora of content made available to them.This involves various variables, such as the location where news information is obtained or created and the actors, objects, or setting mentioned in a piece of information, as well as the power relations between these entities (Cooke, 2017).Hence, unlike the traditional conceptualization of literacy, multiliteracy, especially in light of Web 2.0 technologies, calls for users to embrace the increasingly multimodal ways in which meaning is made and knowledge is constructed (Huijser, 2006), in order to be able to be able to critically assess information they receive and respond in a clear and precise manner (Kellner & Share, 2005).
An essential component in multiliteracy is critical thinking.In media and communication, critical thinking is akin to an interest in acquiring information that is truthful and of high quality (Cooke, 2017).Critical thinking is broadly defined as a skill that is used for the basis of belief formation or on which behavior is guided (Ennis, 1987;Scriven & Paul, 1987).More than informing beliefs of behavior, critical thinking is also an important component of knowledge construction (Dirks, 1998).According to cognitive constructivism, knowledge construction is a personal process in which an individual adds information into their existing cognitive structure.Nonetheless, from a social constructivism point of view, knowledge construction is a process which involves the sharing of information, negotiation of meaning, reevaluation of existing or accepted knowledge and arriving at a consensus (Wang, Woo & Zhao, 2009).As a result, critical thinking is viewed as a dynamic activity which includes the construction of analytical viewpoints through individual analysis and social interaction (Newman, Webb & Cochrane, 1995).The pairing (individual analysis and social interaction) is vital in determining truth of concepts which is achieved through collaboration and process control-sharing, because true meaning should transcend individual reflection (Garrison, 1992;Mason, 2007).Another reason why it is important is because there is news coverage which are contextual -merging aspects of history, politics, and social in its coverage.Through sound analytical thinking, news media users will be able to make sense of these kinds of news.
In today's news media environment where 'fake news' is becoming pervasive, critical thinking becomes even more crucial.It is significant for media consumers to be able to discern what they read because in the "post-truth era, information will continue to increase and dominate our society, colouring how we learn, play, and interact with the world" (Cooke, 2017, p. 219).This also means that consumers will need to be educated on how information made pervasive through media should be used.For instance, Spratt and Agosto (2017) present a compelling call for librarians and information managers at tertiary institutions to take the helm at educating the younger generation in dealing information made available through mass media.

THE STUDY Assessing Critical thinking
In the education setting, critical thinking is a common component that is evaluated.There are several ways to evaluate this construct.One way is to see how students participate in problem-solution tasks (see Şendağ & Odabaşı, 2009) or collaborative tasks (see Karim, Ghani, & Shah, 2017).These types of tasks are thought to be supportive of higher order analytical thinking.Furthermore, in a setting where collaboration in present, it was found that the level of critical thinking improves when the interaction takes on an argumentative nature (Jeong, 2003).Another way to assess critical thinking is to evaluate the schema of students' knowledge.This can be seen through connections formed between issues being discussed with matters or information which may not be immediately present, or to make connections with prior knowledge and experiences, as well as materials external to the site of discussion (Christopher, Tomas, & Tallent-Runnels, 2004).Nonetheless, there are some issues that may hamper the quality of critical thinking.For instance, students' critical thinking may be affected by how they were socialized into a particular setting, whether it be a classroom or an online platform.For example, in a study conducted by Cheong and Cheung (2008), some factors that may affect the quality of critical thinking may be the lack of understanding of the lesson, or the lack of preparation for or unfamiliarity with the topic, or even the lack of 'seriousness' in an online forum, since online platforms are so ubiquitous in the social realm.There is also the concern that unmediated online discussion that students participate in are of lower quality in terms of critical thinking when compared to discussion moderated by an instructor in a classroom setting (Christopher, Tomas, & Tallent-Runnels, 2004).
Although ubiquitously assessed in different academic settings, there exists no one definite definition of critical thinking across different fields of study and cultural contexts (Petress, 2004;Woo, Wang, & Zhao, 2009).Henri's (1992) cognitive reasoning skill and Garrison's (1992) five-stage critical thinking are the most widely accepted models of for the analysis of critical thinking.Acknowledging Henri's framework but citing the teachercentred learning paradigm in which the model was based on as a weakness, the interaction analysis model (IAM) by Gunawardena, Lowe, and Anderson (1997) was developed to describe and gain insight on the process of co-construction of knowledge and negotiation of meaning in online discussions (Marra, Moore, & Klimczak, 2004).More closely derived from the combination of Henri and Garrison's models, the Newman, Webb andCochrane's (1995, 1997) analytical protocol was designed to measure critical thinking and provide a high-level descriptive data.
In news media, critical thinking has been discussed in light of news consumers' level of education and their subsequent level of knowledge, and types of participation with different media outlets.For example, Eveland, Jr. and Scheufele (2000) reported that light users of television showed a greater gap in knowledge between higher and lower education groups.This was also reported by Jerit, Barabas, and Bolsen (2006), who added that differences begin to become more obvious when the news environment supplies more types of information.Those with a higher educational level were also found to refer to print news, and are able to make sense of the different types of information infused with a news article.This may be changing, though, as recent studies have found that online news portals which provide extensive coverage of different social actors or issues pertinent to a developing news will garner more interest and participation from news consumers (Amenta, Elliott, Shortt, Tierney, Türkoğlu, & Vann Jr., 2017).Furthermore, in terms of participation, McLeod, Scheufele, and Moy (1999) found that those who are already actively participating in civic affairs were also more inclined to be involved in a discussion forum which they deem valuable.This is currently enhanced with the presence of social media, which can have a substantial impact on people's perspectives and the mobilization of actions (Hyun & Kim, 2015).
The break from traditional one-to-many communication between the media and its audience, to the two-way communication that is now possible shows that current media has evolved into a dialogic space where meaning can be negotiated and made through the expression of opinions and discussion of issues.While agenda setting and agenda building studies have established that both the media and the public lend a hand in the formation of public opinion, there is a considerable gap in literature pertaining to the characteristics of users' involvement confined within an online news article.Hence, this paper aims to provide a descriptive account of news media users' popular comments left on news published online.This was done by assessing the level and type of critical thinking evident in popular comments using Newman, Webb, and Cochrane's (1995) analytical protocol to measure critical thinking.This protocol was selected because it allowed the researchers to approach comments expansively -through quantitative and qualitative means -to provide a more comprehensive examination.

Data Analysis
This study employed mixed-methods to analyse data.We began the analysis with quantitative measures (frequencies), followed by a qualitative examination.The initial approach to data analysis was grounded in directed content analysis, or deductive content analysis.This approach is the analysis of data guided by established concepts or theories; the findings will also be discussed based on the structure utilized (Hsieh & Shannon, 2005;Elo & Kyngäs, 2008).Similar to Yang and Leong (2017), we relied on a relevant framework to guide the content analysis on our data set.In our study, popular comments were analysed based on Newmann, Webb, and Cochrane's (1995) protocol for examining critical thinking.Furthermore, to ensure that the features were understood correctly, we compared the protocol against their other studies (Newman, Webb, & Cochrane, 1997).Prior to the analysis of data, we had to familiarize ourselves with other topics related to the news articles being analyzed; while analyzing our data, we also read on related news publications, since it was a developing topic (Hsieh & Shannon, 2005).This helped us with the latter part of the data analysis, which was the qualitative examination of the comments.

Analytical Tool
The data analysis was undergirded by Newman, Webb, and Cochrane's (1995) protocol for evaluating critical thinking.This protocol was built based on a set of codes which were developed from Garrison's (1992) 5 stages of critical thinking and Henri's (1992) 5 dimensions of cognitive skills.Newman, Cochrane and Webb (1995) developed their own set of paired indicators which is a simplification of Henri's pairs against the indicators in Garrison's stages.The codes are divided into 10 categories namely: relevance, importance, novelty of information, use of outside knowledge, ambiguities, linking of ideas, justification, critical assessment, practical utility and width of understanding.This model lists a number of positive and negative indicators of critical thinking according to relevance (R+-), importance (I+-), novelty of information (N+-), use of outside knowledge (O+-), ambiguities (A+-), linking of ideas (L+-), justification (J+-), critical assessment (+-), practical utility (P+-) and width of understanding (W+-).Moreover, each of these indicators can be broken into several other values (see Appendix).

Research Protocol
In the data analysis, we inserted the applicable codes at the beginning and the end of a section that reflected critical thinking.Statements with positive and negative values were tallied separately.A pilot coding with a different set comments was done, and the results were not part of the current study.This initial exercise was to ensure that the raters were familiar with the codes.Once we were familiar with the codes and the coding method, we duplicated Marra, Moore, and Klimczak's ( 2004) procedure for inter-rater coding checks.
• For each statement, we recorded all codes which the raters and agreed and those that they disagreed on.• For each of the statement in which the raters disagreed on, (did not use the code that the other applied), we located the statement in the comment and discussed the rationale for its use.If no consensus was achieved, they were discarded.These statements were typically ambiguous, and as suggested by Newman, Webb, and Cochrane (1995), "there is less need for subtle, subjective, and borderline judgements" (p.9).• All codes were determined (for appropriateness) based on the definition and examples provided by Newman, Webb, and Cochrane (1995;1997) and also by reviewing other statements in which the code was applied.• When the inter-rater discussion resulted in more than one agreed code, we included all the indicators.Just as Newman, Webb, and Cochrane (1995) noted in their study, some statements showed overlapping indicators (e.g., a statement may have OK+ (Using previous knowledge) and JP+ (Justifying solutions or judgment)).This example is presented in Figure 1.
Comment from Article 1 11/22/2015 8:33 PM GMT+0700 <OK+JP+< I doubt very much that ISIS even considers what our attitude on refugees is, or cares.But they will exploit the laxity in the refugee system to their advantage, as they do whenever they find a hole.Just as the 9-11 terrorists exploited holes in the student visa program.>OK+JP+> 1 like/ 3 replies

Data Sampling
Though the Internet may be a pervasive tool, many mainstream news sites do not fully utilize the interactive features offered by Web 2.0 technologies such as interpersonal exchange option and human interactivity features (Chung, 2004;Deuze, 2003;Domingo, 2008;Kenney et al., 2000;Rosenberry, 2005;Shultz, 2000).Many online newspapers maintain a place to promote the interactivity of news, as the quality of discussion anchored around online news stories is vital in assessing and stimulating public criticism, debate, and discussion and as a step to maintaining a credible profile and trustworthiness (Diakopoulos & Naaman, 2011;Fletcher & Park, 2017).In light of this, the selection of an online news portal and the subsequent news articles and comments was made based on two criteria.The criteria were: 1) the news site offered a comment section for each of their articles (Weber, 2013), and 2) the news site has a large readership (42 million readers as of September 2014).The selection of news topics to study is important, as participation of media consumers may be affected by the scope of involvement of the masses in a particular news topic (e.g., more would be interested to participate in news stories covering political campaigns or presidential election) (Eveland, Jr. & Scheufele, 2000).Next, a salient news topic was selected.The news topic selected pertains to the Syrian Conflict and is inclusive of how the conflict is affecting other countries in terms of politics and socio-economics.The saliency of a news topic was determined through attention (number of news stories published about the topic), which according to Kiousis (2004) is part of the three dimensions of theoretical explication of media saliency along with prominence and valence (amount of conflict in the story).The news site had published a total of 1275 articles related to the Syrian Conflict since March 2015 which comes to a total of 4218 since 2005.We decided to anonymize the news site as well as the article titles as the news site is a paid site with limited free viewing of items.Nonetheless, we did reach out to the editorial office to notify our research intent.
Four articles from the overarching theme of the Syrian conflict which received a large number of comments were chosen for this study.The total number of comments from the four articles was 822.Although comments from a single article would have been more ideal, the news site chosen only allowed a 14-day viewing window in which readers were allowed to submit their comments after the articles were published to ensure that the discussions centering on the articles were timely in nature.As a result of this limitation, we analyzed comments from four different articles to ensure that our sample was viable for the purpose of the study.Since the study focuses on critical thinking in news media, we were interested in comments that reflected co-construction of knowledge, seen through interactivity of userto-content, as well as user-to-user(s).Comments that represent these relationships are most probably those that are 'popular'.In our study, popular comments received a high number of replies because it is assumed that other readers read, came to a solution or judgment of the issue, and contributed (by replying) to the meaning construction of the issue.
Comments were collected only once for each of the four articles.This meant that although the articles were chosen beforehand, the collection of comment for each article (copying the text into a word document) was done only once rather than repeating the process a few times during the course of the study.The main reason for this is because the number of comments changed over time due to moderation or retraction of comments.The comments which satisfied the criteria set (more than 2 replies) were then extracted from the total number of comments and were analyzed using Newman, Webb, and Cochrane's (1995) framework.This analytical framework was used because of three reasons: first, this analytical protocol was specifically designed to measure critical thinking.Second, it provides a high level descriptive data for analysis and third, it has well defined codes which define the structure and application of the protocol for the researchers (Maara, Moore, & Klimczak, 2004).An added advantage of using the protocol is its capability in quantifying qualitative characteristics, which is an appealing feature when dealing with a large sample of data.
With the assumption that the more replies that the comment received translated to more people having read and reacted to the comment and for the purpose of narrowing down the sample size, only comments with more than 2 replies were included in the sample.This was relatively easy to achieve due to the websites "most replies" feature which automatically arranges the comments according to the number of replies that the comment received.Next, both the article and the comments were manually copied to produce offline copies to be studied.Some details about the articles such as publication date and number of comments are listed in Table 1.Article 1 discussed the political implication of the Syrian war to the United States, specifically in policies and government strategy, and Article 2 elaborated on the war strategies employed by the Islamic State (ISIS).Article 3 was a discussion on the unbalanced media coverage of news between issues and countries, while Article 4 was an update of the Syrian conflict and its political implications to Syria's ally.

RESULTS AND DISCUSSION
As seen in Table 1, the ratio of popular comments was approximately 6:1.Perhaps the reason why there were so many unpopular comments (which did not receive more than 2 replies) was because the comments were statements which did not necessitate any response.This could be due to news media users not being socialized into providing feedback or comments which are interactive, or there may not be a need for social interaction, as seen in the context of Yoo (2011) study.Another reason could be that since the news reported took place elsewhere and may not have a direct impact on the lives of the readers of this particular American news.In other words, the topic may be considered a non-public interest, resulting in an isolated and transactional approach to the consumption of news information (Tenenboim & Cohen, 2013).
Analysis of data about critical thinking indicators found in popular comments in each of the four articles are summarised in Tables 2, 3, 4 and 5.The quantitative findings were organized in the order of the highest to the lowest number of critical thinking indicator.These numbers represent the tallied (agreed) total of critical thinking indicator.
The popular comments examined in this study (n = 57) across four different but related articles had some commonalities.First, the popular comments indicated a high volume of critical thinking indicators (those marked +).Second, the most prominent critical thinking indicators from the data collected were J+ (Justification) and O+ (Bringing outside knowledge or experience to bear on the problem).These two indicators and their related values will be discussed in the next sections.The explanation for the rest of the values may be referred to in the Appendix.The J+ indicator for justification covers 3 positive and 3 negative values of the indicator.
Statements which were coded as having the positive values of J+ were statements that showed the author as providing proof or examples (JP+), and either justifying solutions or judgments, or discussing the advantages and disadvantages of a solution (JS+).Negative values of J+ were marked when the author asked irrelevant or obscuring questions (JP-), and offered judgment or solutions without explanations or justification, or offering several solutions without suggesting which is the most appropriate (JS-).On the other hand, the critical thinking indicator O+ is divided into 6 different positive values and 2 negative values.Statements in which the author drew on personal experience (OE+), refer to relevant material (OC+), use relevant outside material (OM+), use previous knowledge (OK+), brought in course related problems (OP+) or welcomes outside knowledge (OQ+) were coded as displaying the positive value of O+.Statements that showed the author sticking to prejudice or assumption (O-) or squashing attempts to bring in outside knowledge (OQ-) were marked as having the negative value of O+.Comments which demonstrated criticality were also those that reflected a width of knowledge (W+), seen through the understanding of the broader issue.
Examples of statements with J+ and O+ indicators with positive and negative values are shown in Figures 2 and 3. A possible explanation to why these 2 indicators were prominent may have to do with the type of articles that were chosen for the study.The overarching topic of the articles was the Syrian Conflict and this topic is salient particularly for its attention criteria, that is, the extent of coverage and number of reports regarding a particular topic.The prominence of the J+ and O+ indicators from the data collected indicates that the topic also shows the saliency criteria of valence or how negatively or positively a story is presented (Choi, 2004), or the extent of conflict associated with the topic.Furthermore, all four of the articles chosen for this study shared a high-conflict angle of the topic, because of this, many readers were compelled to make a judgment which they may or may not justify by drawing on personal experience, using relevant outside material or using previous knowledge.This reflects the findings of Tenenboim and Cohen (2013), where readers may be driven to express their concern for the public, and thoughts for or against the reported conflict.Wang et al. (2009) noted in their study that knowledge construction is greatly influenced by the nature of the topic.They implied that for in-depth discussion to take place, and to be sustained, the topic must not only be relevant but also meaningful to the participants.The topic of Syrian conflict is a source of much debate and discussion for the target audience of the articles, and although the online news website from which the articles chosen for this study is not based in Europe, the conflict has affected the country's foreign policy and sparked much debate among political groups in the country.The content of the articles itself appeared to be contentious and controversial, which according to Hung, Tan and Chen (2005), as well as Tenenboim and Cohen (2013), would be a characteristic that encouraged the readers to form different opinions on the topic.Through the unique function of the comment section, readers were able to expand their personal process of knowledge construction towards social coconstruction knowledge through the sharing of judgments or solutions, bringing in outside information to widen the perspective of the issue and negotiating meaning through discussion and debates.This user-to-user and user-to-content interactivity is seen as a positive sign of knowledge construction, a process resulting from critical thinking.
Comment in Article 2 11/17/2015 1:13 AM GMT+0700 <JS+< I am perfectly ok with zero immigrants.I fail to see how we lose in this scenario.There was once a time where immigration made sense.That time has now past and we need to protect what our Parents and Grand Parents have built rather then fracture it as what is currently happening in the EU.>JS+> 11 likes/ 5 replies Comment in Article 1 11/17/2015 7:35 AM GMT+0700 <OC+<"However, it is not only deeply unfair to paint all of those arriving with the same brush it is also self-defeating.">OC+> 2 likes/4 replies Comment in Article 2 12/26/2015 7:28 PM GMT+0800 <OM+< The one thing that always seems to emerge from well-reported stories about the Syrian Civil War is the fact that it is a religious war between the two branches of Islam: Sunnis and Shia, and their proxies.For the puzzled, the disagreement which is the central rallying point in the Civil War is over a dispute 1,200 years old: Who are the rightful heirs of Mohammed?Ali and the Imams (Sunni) or the four Caliphs (Shia).<L+< Oh, I know that there are those who might be sceptical, and who might instead point out that it's all about oil and who gets to use it and profit from it.But that would be conceding that the Civil War is nothing but some global power play by mega-oil interests over energy resources and their control, and that religious difference is simply the rallying cry.But that point has to be inferred from things like Turkey being the middle-man in the transfer of oil stolen buy ISIS/ISIL/Daesh to buyers paying between $19 million to $50 million dollars per month for the oil.>L+> <C+< In short, if you want to understand the Civil War, ask: who buying this oil and who is profiting from its sale.The internet contains many answers to those questions.But when running down the sources is an exercise in tin-hattery---which is no doubt deliberate, but nonetheless unrevealing..>C+>>OM+> 6 likes/3 replies Comment in Article 4 12/26/2015 8:38 PM GMT+0800 <W+<"Rebel commander killed (perceived as moderate by the Post) although he had commanded forces that killed scores of civilians in Damascus.He had shown sympathies to Osama Bin Laden, and al-Qaeda.His terrorist group is fully funded by the Saudis.>W+><J+<He was no different than ISIS, in the fact that he had no qualms killing other Muslims.This world will never be safe as long as religious fundamentalist are still alive.. >J+> 3 likes/2 replies Noting that the formation of public opinion is no longer solely dominated by the media through its agenda setting role, the interaction between the media and consumers of the media as evidenced in the popular comments analyzed in this study shows that the interactivity lends a hand in whether the agenda set by the media was accepted.Discussions stemming from the comments (in which other readers contribute by replying to the comment) showed that commenters were aware that the media was setting an agenda and showed no inhibition in voicing their doubts or rejection.This was evidenced by the high number of JP+ (provided proof and examples), Js+ (Justified solutions or judgments), OM+ (used relevant outside material), and OK+ (used previous knowledge) critical thinking indicators to support their judgment of information, point-of-view or opinion shared by through the articles.This awareness towards the media agenda was also indicated by the high number of JS-(offering judgments and solutions without explanation) indicators which were mainly judgment made by the commenters towards the author of the articles themselves or the news site as a whole rather than the content of the information or the issue discussed.
As seen in the results, there were other indicators with lower numbers of consensus or completely absent.In light of this, there are several points worth taking note.For example, there was an absence of I+ (Importance) indicator and only two statements indicated an obvious R+ (Relevance) criteria from all of the comments.A possible cause for this is the nature of the comments analysed which is different from the academic setting that it was designed and had been utilized for.The second possible reason is that the I+ and R+ indicators are highly reliant on subject knowledge thus as Newman, Cochrane and Webb (1995) noted, the comments are best coded by an expert on the subject -which in the case of this study would be a socio-political expert.Noting this (the absence of a subject-expert rater), any statement which involved a discrepancy between raters -in which a consensus on the type of indicator failed to be reached-were omitted.The A+ (Ambiguity) indicator was also absent from the statements coded and, there were few statements with the P+ (Practical grounding) and W+ (Width of understanding) indicators.Similar findings from Newman, Webb, and Cochrane's (1995;1997) findings were attributed to the nature of the commenting section, where readers were more engaged in general discussion rather than in finding solutions for problems.

LIMITATIONS AND FUTURE CONSIDERATIONS
Through the use of a protocol for the evaluation of critical thinking, we were able to provide a descriptive account of popular comments.Two critical thinking indicators commonly found in popular comments are justification and (the synthesis of) outside material.From this, there are several research implications worth taking note.As seen in the results, there were indicators which were absent or had few obvious representations in the statements coded.These indicators were either too reliant on subject knowledge (I+ and R+) or were only possibly present in problem-solving activities found in a class setting, rather than in public discussions.Furthermore, this study only accounted for news that did not have any direct implication on its immediate context.While these news items may be salient internationally, it may not be perceived as newsworthy by the readers from the context of the news outlet.These limitations could be taken as further research considerations if critical thinking were to be examined in the interactions of news media users, in that a news item chosen for analysis could be one that is familiar to the researchers and possibly the readers, and that new pairs of critical thinking indicators may be introduced.Another possible consideration is to take into account the sociocultural makeup of the context where the news items are published.Finally, it would also be worthwhile to analyse linguistic features instead of solely relying on discourse meaning as indicative of critical thinking or other features characterizing popular comments online, or the potential correlation between elements demonstrating critical thinking with the popularity of comments, such as the number of likes a comment receives.Also, to ensure the validity of analysis, perhaps other coders may be involved in the study.This may allow the use of a repertory grid approach, which involves (typically) more than two raters to evaluate a construct.

CONCLUSION
This study was focused on analyzing the interactivity of media users as characterized in popular comments on an online news platform.Noting the interrelation between interactivity and critical thinking, we utilized the Newman, Webb, and Cochrane's (1995) analytical framework to identify indicators of critical thinking.Results from data analysis showed that one of the characteristic of popular comments online is the presence of critical thinking in their statements.From the samples gathered, it was seen that most users made statements that expressed their judgment of the information from the article and provided justification by bringing external materials such as prior knowledge or information.We can also see that media users were actively interacting -not only with the material (information from the article), but also other commenters.Based on these findings, there are promising signs that comments which garner attention contain attributes of critical thinking, and that online users are valid and active consumers of media.This is noteworthy, especially in the age of fake news, as it is important that media consumers engage in information in an analytical manner, as this helps with the recognition of misinformation or misinterpretation, and there will be an inclination to consider real news as accurate (Pennycook & Rand, 2017).Like many other studies, this study is by no means conclusive.The limitations of this study call for improved methodologies, richer samples and access to subject experts.As such, we invite others to replicate this study in hopes that through enhanced understanding of the characteristics of popular comments found online, we are able to better understand, facilitate and address the challenges posed by public social discourse.