Is it possible to describe television series from online comments?

In recent years, there have been a large number of studies aimed at generating and evaluating text summaries. To the best of our knowledge, most of the work in this area focuses on summarizing full texts, such as books and news [7–10], or summarizing opinions from user-written reviews [11–14]. However, in our case, the use of comments as a source of data raises many challenges that, individually, serve as inspiration for this study.

In fact, several studies have tried to analyze and characterize online interactions such as online discussions. In [5], for example, the authors analyzed comments posted on different social media sites, mainly studying how their content and rating can be related. The work of [15], in turn, develops a formal approach to the modeling of “activity bursts”, which can be applied to comment streams. Choi et al. [16] characterized conversations collected from Reddit in terms of volume, responsiveness, and virality.

Methods to extract relevant information from comments have also been proposed in recent works. Khabiri et al. [17] proposed a clustering-based approach to identify groups of correlated comments in YouTube videos. Yang et al. [18] presented a method for generating Web page summaries that also considers the comments associated with them and the social network among users who commented. In [19], the authors proposed the use of vector representations of sentences as a metric of similarity in the process of extractive summarization. Liu et al. [20] model the problem of summarizing comments as a clustering problem, in which the number of topics covered by the comments must be known a priori. Chua and Asur [21] proposed a temporal correlation-based topic model to identify the most relevant tweets of a given query for the summarization task. Unlike the contexts used in these works, using online discussions as a source of information for describing an entity has several new problems. Comment sequences represent conversations, which may derail to different subjects or may contain only assertive information, such as “I agree with what you said”, or even a simple “Me too”. These types of comments do not add relevant information to the description of the entity (in our case, a TV series) in question, so considering a way to deal with these cases is an important aspect of our article.

Considering our previous works in the area, in [22], a model was proposed to capture the dynamics of communication activities on the Web. Following this work, in [23], universal and distinct communication patterns were described considering the technology used as means of communication. In [24], a parsimonious model was proposed to characterize the burstiness of general random series of events in the Web. In [25] we selected comments from online discussions in order to best summarize the conversation itself. Although these studies presented relevant contributions in understanding the dynamics of online discussions, none of them proposed methods to use the content of the comments as a source of information to describe and summarize the entities they refer to.

In [26], the paper this work is based on, we first explored the task of selecting comments from online discussions for entity description, studying if and how comment sequences can be used to represent entities. In [27] we used the results found in our previous work to generate and evaluate entity representations from online discussions.

This article differs from the aforementioned studies in three main aspects. First, this work focuses on the characterization of the potential of comments as a single data source for generating formal entity representations to be used by machine learning algorithms. Second, through this characterization, we investigate and quantify the differences between the language used in formal and manually generated summaries and the language used by users in comments, specially in those comments evaluated as “non-descriptive”. Finally, we compare different extractive summarization methods, and evaluate them by summary size and by how much of a human-written summary they are able to describe.

For the remainder of this article, we define by c a comment submitted to an online discussion. For the purposes of this work, we assume that each comment is composed only of its textual content and its timestamp, which is used as a sorting key. Thus, a comment sequence \(\mathcal {C}\) is an ordered set of n comments {c₁,c₂,…,c_n} taken from a common context, for example, all comments of a given YouTube video.

The diagram in Fig. 1 illustrates the relationship between a domain Dom, its set of entities \(\mathcal {T}^{d} = \left \{ \mathcal {T}^{d}_{1}, \mathcal {T}^{d}_{2}, \mathcal {T}^{d}_{3} \right \}\), and the comment sequences of each of these entities: \(\mathcal {C}^{t^{d}_{1}}\), \(\mathcal {C}^{t^{d}_{2}}\), and \(\mathcal {C}^{t^{d}_{3}}\).

Finally, we define by R^t the human-written summary associated with the entity t. Similarly, we also define by R^d the human-written summary associated with the domain d. In our case, R^t can be a human generated summary of an episode of Game of Thrones, for example, available on IMDB² or Wikipedia³. Moreover, R^d will be, in this case, the concatenation of the summaries for all episodes \(t \in \mathcal {T}^{d}\) from this particular TV series d.

Given these definitions, our goal is to find a concise portion of text (e.g., a subset of comments, or a selection of sentences) that is able to describe a context well. For this article, we will be focusing on the case of comment sequences referring to television series episodes. Having established the definitions of the above concepts, we can write this task as: automatically generating a piece of text r^d relating to series d, using its discussion \(\mathcal {C}^{d}\) as source of information, with a small number of words describing each episode t, so that R^d is well explained by r^d.

For this work, we collected all the comments directly associated with episodes of several different animated series from the MyAnimeList.net website⁴. MyAnimeList.net, also referred to as MAL, is a website that, as its core functionality, allows users to list animated series according to how much of it they’ve watched. In addition to that, MAL also gives the option for users to rate, review, and recommend shows, and also provides varied information regarding each series, from its cast and crew to the release date. More importantly for this work, MAL also has an online forum, with sub-forums for each animated series, usually containing a separate discussion for each episode from that series. In [27], we have used much of the data provided for each series from our collected data set, and they can also be used in the future to expand and improve the work presented in this article.

In addition to the comment data, human-written summaries R^t were also collected, when available, for each episode t. These summaries were obtained from Wikipedia pages dedicated to listing and describing episodes of each series ⁵. The summary R^d of each series d is simply the concatenation of the summaries of all episodes of the series in question.

As the number of comments and episodes varies significantly from series to series, three criteria were used to select the series we analyzed in this article. First, we tried to choose those series with a similar number of comments, so that the popularity of all series is similar, in general, allowing us to more easily compare the different series without taking the discussion size too much into consideration. The second determining factor for the choice was the existence of an easily accessible human-written summary (collected from Wikipedia) for each episode of the series. Finally, we chose a set of series with a number of episodes in the order of tens and comments in the order of thousands. Table 1 shows the series that were chosen for analysis, along with the number of episodes and comments associated with them⁶.

Once the texts described above were collected, both the comments and the summaries, a sequence of operations was performed on them in order to transform the original texts into workable data. First, quotes from other comments present in the texts were removed. Keeping the quoted comments as repeated text could lead to interesting results, as that piece of text would be considered more “relevant” by some algorithms, but a new study would be needed to verify that. Then, the HTML formatting of the texts was removed, leaving only the unformatted comment texts. Doing this discards useful information in the form of text formatting, such as boldfaced words, but the methods used in this article do not take those into consideration, making them a pointless complication. Also, non-printable characters were excluded. Finally, the stop words in the texts were removed, and all letters were converted to their lower case forms. For this last step, we kept a different version of the texts with and without applying the stop word filter and lower case transformation. This was done because some of the methods used in this article require actual natural language sentences, while others work best with only a list of informative words. Unless otherwise specified, all text used as data in this article has stop words filtered out and has been completely converted to lower case.

As mentioned in Section 2, comments (and comment sequences) are composed of texts with very distinct characteristics, which can vary significantly in size, form and content. Thus, to better understand the collected data described in Section 4, in this section a series of analysis on the characteristics of the comment sequences were carried out. Such analysis will serve as a basis for the methodology presented in the following sections.

5.1 Basic analysis

As an initial analysis, general characteristics of the collected comments were evaluated. Figure 2a shows the histogram for the size (number of words) of the comments in our data set. Note that the vast majority of comments have a small number of words, while a few comments are very long, with more than 500 words. This result evidences the difficulty of extracting relevant comments in comment sequences, since the great majority is composed of small comments, most probably with little informative content.

In order to investigate if there is a discrepancy in user participation per episode, we show in Fig. 2b the histogram of the number of comments posted per episode. Note that many episodes have a small number of comments associated with them, i.e., most episodes have between 20 and 100 comments. Similarly to what was shown in Fig. 2a, there are also episodes with a large amount of comments. Thus, a method for identifying relevant and descriptive comments will often face the problem of having either too little or too much information available, depending on the episode and series.

In short, Fig. 2a suggests that most of the comments do not have useful information given their relatively small sizes, but Fig. 2b indicates that most episodes will have a sufficient amount of comments so that we can infer a description from them.

5.2 Word analysis

In order to determine which comments would be most descriptive of the human-written summary, we first made a feature analysis that could be used to define and represent the different terms present in the data set. We used the t-SNE⁷ [28] dimensionality reduction method to interpret the quality of these representations. This visualization technique aims to capture and illustrate the local structure of high-dimensionality data while also indicating its global structures. After considering some different options for word-related characteristics, we determined a set of features which presents sufficient and relevant information regarding the usefulness of each word to discriminate between different series and episodes. Defining \(\mathcal {V}^{X}\) as the set of words used in context X (e.g. \(\mathcal {V}^{d}\) as the set of words used in the discussion for series d), the following features were used:

• Word probability in each series: the probability of the word v occurring in each series’ discussion in comparison to it occurring in other series’ discussions, generating one feature for each series.

  $$P\left(v \in \mathcal{V}^{d}\right) = \frac{tf(v, d)}{tf(v, \mathcal{D})} $$

  In which tf(v,d) is the count of occurrences of word v in series d, and \(tf(v,\mathcal {D})\) is the count of occurrences of word v in the entire data set.

• Shannon entropy of the word among all series: how much information is produced by the word v occurring in a series’ discussion.

  $$H_{v}^{\mathcal{D}} = - \sum_{d \in \mathcal{D}} P\left(v \in \mathcal{V}^{d}\right) log_{2} P\left(v \in \mathcal{V}^{d}\right) $$
• Shannon entropy of the word among all episodes of the data set: how much information is produced by the word v occurring in an episode’s discussion.

  $$H_{v}^{\mathcal{T}} = - \sum_{t \in \mathcal{T}} P\left(v \in \mathcal{V}^{t}\right) log_{2} P\left(v \in \mathcal{V}^{t}\right) $$
• Shannon entropy of the word among the episodes for each series: how much information is produced by the word v occurring in an episode’s discussion, generating one feature for each series.

  $${}H_{v}^{\mathcal{T}^{d}}\! =\! -\! \sum_{t \in \mathcal{T}^{d}} P\!\left(v \in \mathcal{V}^{t} | v \in \mathcal{V}^{d}\right) log_{2} P\left(v \in \mathcal{V}^{t} | v \in \mathcal{V}^{d}\right) $$
• TF-IDF of the word for each series: term frequency – inverse document frequency of the word v, generating one feature for each series. The TF-IDF is calculated by multiplying the count of occurrences of word v in series d, tf(v,d), by the inverse document frequency idf(v,d):

  $$idf(v,\mathcal{D}) = log \frac{|\mathcal{D}|}{ \left| \left\{ d \in \mathcal{D} : v \in \mathcal{V}^{d} \right\} \right|}. $$

Thus, each word is represented by a feature vector of size \(2+3*|\mathcal {D}|\) (particularly, 2+3∗13=41) that indicates how this particular word is used through the data set, and how those uses are distributed across the TV series.

Figure 3 shows the visualization generated through the t-SNE dimensionality reduction method for all words in the data set, which are represented by this set of features. In the image, each word is drawn as a point, colored according to the series in which it appears the most, i.e. the domain d for which the word has the highest occurrence count tf(v,d). It can be noted that, although most words are located in a single large cluster (“main body” with mixed points of different colors), there are some smaller clusters (“small tendrils” with points of a single color each) that contain words associated with the same series. Since the features used to represent the words, and then generate the t-SNE visualization, reflect the presence of the word in the analyzed series, it was verified that these small clusters contain those few words that occur predominantly in a single series (character names, and other specific terms from the series). Thus, we can conclude that, in fact, the number of words that have their use highly concentrated in a single series is small. This means that if a comment contains one of these words, we are probably able to tell which series it is associated with. This hypothesis will be explored in the following sections of this article.

5.3 Summarization potential

Another fundamental analysis to understand the potential of extracting relevant comments from discussions is to investigate the ability of each comment to explain the manually generated summary of the episode to which it is associated. To evaluate this, we used the ROUGE-N metric [29]. This metric informs the proportion of N-grams from the reference text (S) that are also present in the candidate text s:

$$ROUGE-N = \frac{count\left((grams_{N} \in S) \cap (grams_{N} \in s)\right)}{count\left(grams_{N} \in S\right)} $$

In other words, we can consider that this recall-oriented metric measures how much of the reference text is covered by another text of interest. As such, a higher ROUGE-N value informs that the candidate text s covers more of the reference text S with respect to their textual contents, indicating that s contains more information from S. It is worth noting that, despite its simplicity, Lin reports good correlation values between ROUGE-N metrics (specially for low values of N) and human judgment for summary evaluations.

As such, in this article we evaluated text coverage with the ROUGE-1 metric, i.e., considering only unigram overlaps. Tests with the ROUGE-2 metric, i.e., considering bigram overlaps, revealed similar results.

Figure 4 shows the distribution of the coverages of the human-written summaries R^t (reference text) of each episode t by their respective comment sequences \(\mathcal {C}^{t}\) (candidate text), as measured by the ROUGE-1 metric. Note that, on average, only 40% of the words from an episode’s summary are found in the comments associated with it.

In order to analyze this result in greater depth, we show in Fig. 5 the cumulative coverage (ROUGE-1) of an episode’s human-written summary \(\phantom {\dot {i}\!}R^{d_{457}}\), taken as an example, over the course of its discussion \(\mathcal {C}^{d_{457}}\phantom {\dot {i}\!}\), with comments ordered according to their timestamps. As more comments are added to the discussion, the summary coverage increases, until the full discussion reaches a ROUGE-1 score of approximately 0.8 (80% summary coverage). Note that much of the summary \(\phantom {\dot {i}\!}R^{d_{457}}\) is explained in the first few comments from the series’ discussion, with little relevant information being added after about the hundredth comment. This general behavior, disregarding the specific ROUGE-1 values of the considered example, can also be seen for all series in our data set.

With these last analyses, we verified that there are episodes for which comments have a higher descriptive value than others, with respect to how much of its human-written summary they are able to explain. In addition, it is noticeable that the gain in summary coverage from adding more comments decays as more comments are considered at the same time.

As a way to better verify our hypothesis that online discussions can be used as a source of information to describe the entities and domains they belong to, we developed a simple classification task that checks how easily comments can be attributed to their source. This serves as a “sanity check” for our work, as the goal of this task is to determine if comments have discriminative information with respect to their associated entities and domains.

6.3 Comment classification

For this purpose, we define two classification tasks to identify how well comments represent a given series and episode, respectively, as K and α vary:

• In the first task, the comments are grouped by series and the objective is to classify each comment to the series with which it is associated, among the \(|\mathcal {D}|\) series of our data set. For the TF-IDF calculation, a document p is the collection of comments \(\mathcal {C}^{d}\) associated with each series d.

• In the second task, the comments of a given series d are grouped by episode and the objective is to classify each comment to the episode with which it is associated, among the \(\left |\mathcal {T}^{d}\right |\) episodes of that series. In this second case, for the TF-IDF calculation, a document p is the collection of comments \(\mathcal {C}^{t}\) associated with each episode t.

To perform such tasks, we used the Naive Bayes [31] classifier from the Weka tool collection⁸ [32]. Other classifiers were also tested, and the results were similar. The standard parameters values for the Naive Bayes implementation in Weka were used, and the results were obtained using 10-fold cross-validation. Thus, our goal is to find values of K and α which present a good compromise between the accuracy of the classification and the number of comments correctly classified in each of the tasks. Remember that if the value of K is too large, many words will be used in the classification task and probably many of them will be less discriminative. At the same time, if the value of α is too large, few comments will be considered in the classification task.

Results of the classification tasks are illustrated in Fig. 6. First, observe that identifying which series each comment refers to is very easy. Second, note that considering the comments with no words from the top-K set present in the representation causes a considerable loss of accuracy. This follows the expected behavior, since comments of this type do not have any discriminative information in their representations (their representation vectors being entirely comprised of 0s, regardless of domain).

When we begin to classify the comments by episode, given a series, the classification accuracy decreases significantly. This is expected, as the number of possible episodes a comment can belong to is far greater than the number of possible series, as seen in Table 1 (13 series, and 369 episodes in total). Note in Fig. 7 that as we consider more words (greater value of K), it becomes more difficult to identify which episode a comment refers to. In addition, disregarding comments with low informational value (relative to the number of relevant words) has a significant positive impact on the results, especially for lower values of K, although removing too many comments also worsens the accuracy.

On the other hand, in Fig. 8, we can verify that the number of comments considered in the evaluation increases with greater values of K, and decreases with greater values of α. This follows the expected behavior, and shows that we can get a more concise set of comments and with better explanatory capacity for the episode if we choose an appropriate parameter configuration.

Specifically for the data set used in this work, it may be noted that selecting a value of 10 for the parameter K not only considers a smaller number of comments (due to the interaction with α) but also creates more discriminative representations for the comments (higher classification accuracy). However, the parameter α, for K=10, has the best accuracy result for entity classification with α=2, with α=3 achieving an accuracy value less than 1% lower. As for the domain classification task, the classification accuracy is high for any value of α greater than 0, with only insignificant improvements as α increases beyond that. This generates a trade-off in the selection of comments between descriptive potential (we want comments that identify well their series or episode) and succinctness (we want few comments).

In this section we describe the summarization methods used in this work, through which we automatically generate the summaries of each series from the data set. We first explain two word relevance-based comment selection methods, TKW-AF and TKW-MSC, and then describe another extractive summarization algorithm used for comparison, TextRank.

Section 7.1 describes the TKW-AF and TKW-MSC summarization methods as well as the intuition behind them. Section 7.2 provides a brief explanation for the TextRank method.

This paper presented a characterization of the potential of comments as a single data source for explaining and describing entities. We analyzed characteristics of online discussions of TV series episodes, such as the size of the comments, relevance of the words, and the ability of comments to explain a human-written summary extracted from Wikipedia.

Based on this characterization, we implemented a series of extractive summarization methods. The first two, TKW-AF and TKW-MSC, used a set of discriminative words, also referred to as “top-K words”, to select highly descriptive comments for the episodes, and then for the series. The third method, using the TextRank algorithm, selected relevant sentences from each episode’s discussion following a graph-based ranking algorithm.

All the tested methods were able to achieve a relatively good coverage of the human-written summary while using only a small fraction of the original text. However, each summarization method obtained results with a different focus: TKW-AF got the best ROUGE-1 scores for its summary, while having a slightly larger text; TKW-MSC sacrificed part of its summary expressiveness in favor of smaller texts; finally, the TextRank algorithm manages to pick a higher ROUGE-1 score or a lower text length, according to its parameter’s value.

With the characterization studies we performed, including a classification task, we verified that online discussions have textual content that make it so that we can easily identify which series they are referring to, and to a lesser degree, the episode they are referring to. With the results obtained in the summarization task, we were able to find small subsets of comments that had enough descriptive content to cover a good part of the summary for each series. Considering these results, we conclude that we can, in fact, use online comments to describe television series.

In future works, we plan to generate abstractive summaries, which try to reflect more closely what people naturally do when they write a summary about a given entity. In an abstractive summarizer, after extracting the knowledge from the comments, the most informative content is selected and expressed in natural language. As pointed out in [38], abstractive summarization is usually a much more complex and challenging task than the extractive one, since it requires a natural language generation module and also a domain dependent component to process and rank the extracted knowledge.

Another aspect we analyzed was difference in vocabulary between types of comments, and the tendency comments considered to be particularly “non-informative” have to express more positive sentiments than those considered to have at least some useful information (that is, comments that have at least one of the discriminative words from the top-K set). This discrepancy in sentiment values indicates that it’s possible to design a method for identifying relevant/irrelevant comments for a determinate topic, or even entire comment sequences, based on sentiment polarity and word usage. Such a method could be used to automatically find out-of-topic comments, spam, or flamers. For that, a deeper study on how these sentiment values differ according to the type of comment would be required, properly identifying how these behaviors change according to different contexts.