nova página do texto(beta)
Inglês (pdf)
Artigo em XML
Referências do artigo
Tradução automática
Enviar este artigo por email
Citado por SciELO 
Similares em
SciELO 
Permalink
1 Introduction
In recent years, media has become an important source of real-time information. People use social media to communicate, socialize, debate and engage in arguments. With the rapidly increasing amount of data on social networking service (SNS) like Twitter, researches on event detection and classification for the crisis-related tweet data are attracting more and more attention. During a crisis situation, people use SNS to post situational updates, look for useful information, and ask for help [1,2]. In these cases, users can help the victims by classifying and understanding the posts as crisis-related tweets.
During natural disasters, SNS such as Twitter and Facebook are widely used by affected people to post crisis-related messages. These crisis-related posts disperse among multiple categories like information wanted about missing, injured and dead people, and infrastructure damage. The challenge here is to classify important situational updates from these crisis-related tweets, assign them appropriate informational categories [3,4].
Related with this challenge, TREC 2018 introduced an Incident Streams (TREC-IS) track which is designed to categorize a stream of tweets related to each incident by information type in order to automatically process social media streams during emergency situations [5]. There have been several approaches to classify incident streams by using deep learning models and word representation methods. Chy et al. [6] used a method of combining convolutional neural networks (CNNs) and Long Short Term Memory (LSTM). Buntain [7] used a pre-trained word vector from Wikipedia and the FastText method. Miyazaki et al. [8] used Convolutional Neural Networks (CNNs) model and a simple Multilayer Perceptron (MLP) by inputting text BoW vector.
Deep learning has made major advances in areas such as image processing and natural language processing.
Convolutional neural networks have recently achieved remarkably strong performance on the practically important task of sentence classification [9]. In recent image classification research, a technique for generating a class activation map (CAM) is introduced using the global average pooling in CNNs [10]. A class activation map for a specific class indicates the discriminative image regions which are relevant to that class.
On the other hand, when learning with small number of training data, deep learning does not provide a satisfactory performance. One-shot learning aims to learn information about object categories from a few training images.
In this paper, in order to categorize a stream of tweets related to each incident by information types, we propose a conceptual representation for crisis-related tweet classification. The crisis-related terms in a tweet are represented as conceptual entities such as event entities, category indicator entities, information entities, URL entities, and user entities to represent the specific characteristics of information types in disaster and emergency situation.
We have experimented using deep learning models which combines CAM and one-shot learning model. The CAM is applied to text classification by highly weighting significant term based on class activation mapping using global average pooling instead of fully connected layer. To deal with small number of training data to learn, we have used one-shot learning method using fully connected layer. To show the effectiveness of our method, experiments are conducted on a TREC-IS 2018 test collection.
The paper is organized as follows: Section 2 describes a conceptual representation for crisis-related tweets; Section 3 presents combining deep learning models. Section 4 describes experimental results. Finally, we conclude in Section 5.
2 Conceptual Representation for Crisis-Related Tweets
In order to classify a stream of tweets related to the incident, the terms in each tweet are represented as conceptual entities such as event entities, category indicator entities, information type entities, URL entities, and user entities. The conceptual entities are defined according to the information type categories.
2.1 Event Entities
In TREC-IS training data, hashtags and keywords used for collecting a stream of tweets related to each incident have been provided. For conceptual representation, the hashtags and keywords are represented as event entities: <EventLoc>, <EventCri>, and <EventClu> according to the information in such as a location, an incident and a clue. The followings are examples of event entity representation for each hashtag or keyword.
— <EventLoc>: #colorado wildfire, #abflood, Lax airport, #colorado, #costarica, ...
— <EventCri>: #colorado wildfire, #abflood, #earthquake, #Wildfire, ...
— <EventClu>: #StrongerPH, #prayfornsw, #SafeNow, ...
Since each tweet includes event entities, an event entity itself may not give information. However, it can reduce the noise by the use of diverse terms.
2.2 Category Indicator Entities
In a training data, category indicator terms are provided for each tweet by human assessors. We have used the indicator terms for each category indicator entity which are provided by TREC-IS training data.
Table 1 Examples of category indicator entities
| Category Indicator Entity |
Examples |
|---|---|
| <MultimediaShare> | #photo, camera image, videos |
| <InfomationWanted> | does anyone know, report? |
| <ServiceAvailable> | heavy air tanker, animal hospital, offering housing |
| <Official> | tsunami warning issued, briefing, emergency declaration |
| <Donations> | donations, items needed |
| <Weather> | #weather, degree, rain |
| <Movepeople> | evacuate, leave town, evacuation order |
| <EmergingThreats> | road blocked, sewage treatment disabled, landslide reported |
Table 2 Examples of information type entities with expected category
| Information Type Entity | Examples | Expected Category |
|---|---|---|
| <Disaster-Threat> | Power is out, People trapped, No power | EmergingThreats |
| <Evacuation> | leave NOW, evacuation orders, closed the highway | MovePeople |
| <Government> | USGS, LarimerSheriff, Commander | Official |
| <Organization> | HumaneSociety, RedCrossDenever, NoCORedCross | Official |
| <Emergency> | be on ALERT, please don't wait | EmergingThreats |
| <Unit> | magnitude, dollars | Factoid |
| <Provider> | Red Cross, Animal Hosptal, #Gov20 | Volunteer |
| <Communication> | Call, text, hotline, dm | Official, Advice |
| <Service> | Animal boarding pick up, disaster recovery center opened | ServiceAvailable |
| <Change> | Magnitude reduced, update, now cancelled | EmergingThreats |
Table 3 Examples of URL entities with expected category
| URL Entity | Examples | Expected Category |
|---|---|---|
| <URLVideo> | youtube.com, vimeo.com | MultimediaShare |
| <URLPhoto> | instagram.com, facebook.com/photo.php | MultimediaShare |
| <URLNews> | yahoo.com/news, cbsnews.com, bbc.co.uk | ContinuingNews, PastNews |
| <URLWeather> | wunderground.com, theweathernetwork.com | Weather |
| <URLOrg> | larimer.org, usgs.gov, emergenza24.org | Official |
| <URLDonation> | worldvision.org.ph, prizeo.com | Donations |
| <URLDisaster> | sismos.cl tenki.jp | EmergingThreats |
| <URLBlogMgz> | livejournal.com grist.org | KnownAlready |
Table 4 Examples of user entity for expected category
| User Entity | Examples | Expected Category |
|---|---|---|
| <UserNews> | @9NEWS, @ANCALERTS @BreakingNews, | ContinuingNews, PastNews |
| <UserWeather> | @dost_pagasa, @breakingstorm | Weather |
| <UserOrg> | @NSWRFS, @oxfamgb, @NASA | Official |
| <UserDonation> | @RescuePH | Donations |
| <UserDisasterInfo> | @NewEarthquake, @INGVterremoti | EmergingThreats |
| <UserMultimedia> | @youtube | MultimediaShare |
The number of categories in TREC-IS and category indicator entities are 25 such as <InformationWanted>, <ServiceAvailable>, <Official>, <Donations>, <EmergingThreats> and etc.
The terms for 5 category indicator entities such as volunteering, donation, multimedia, and sentiment are expanded using terms in thesaurus.com and relatedwords.org.
2.3 Information Type Entities
In order to represent information type for each term, information type entities are defined as followings: <Question>, <Communication>, <Organization>, <Evacuation>, <Location>, <Emergency>, <Person1st>, <Informal>, <Emotion/reaction>, <Narration>, <News>, <Disaster/threat>, <Change>, <Service>, <Provider>, <Fact>, <Unit>, <Government>, <Advice>, <Pastdate>, <Pastverb>.
The seed terms we have defined are expanded using terms in thesaurus.com and relatedwords.org. We have assumed that each information type entity would give additional information for classification. For example, the entities such as <Government> and <Organization> would be helpful for the category 'Official', an entity <News> for the category 'ContinuingNews', an entity <Disaster/threat> for the category 'EmergingThreats', and an entity <Evacuation> for the category 'MovePeople'.
2.4 URL Entities
A URL itself has information as represented by the URL. We have defined URL entities such as <URLVideo>, <URLPhoto>, <URLNews>, <URLWeather>, <URLOrganization>, <URLDonation>, <URLDisaster> and <URLBlogMagazine>.
We have assumed that each URL entity would give information for a category. For example, the entity <URLDonation> would be helpful for the category 'Donations', an entity <URLWeather> for the category 'Weather', an entity <URLOrganization> for the category 'Official', an entity <URLNews> for the categories 'ContinuingNews' or 'PastNews', an entity <URLDisasterInfo> for the category 'EmergingThreats', and the entities <URLVideo> and <URLPhoto> for the category 'MultimediaShare'.
In order to extract useful URLs, the TREC-IS training data and CrisisT26 dataset are used. For pre-processing, a short URL is converted to an original URL. The URLs with more than frequency 10 have been defined as 8 entities. The followings are term lists for each URL entity.
2.5 User Entities
Based on a user's characteristics, we have defined user entities such as <UserNews>, <UserWeather>, <UserOrganization>, <UserDonation>, <UserDisasterInfo>, and <UserMultimedia>.
In order to extract useful user information in tweets, the TREC-IS training data and CrisisT26 dataset are used. The users with more than frequency 10 have been defined as 6 user entities as the characteristics. We have assumed that a user entity <UserDonation> would be helpful for the category 'Donations', a user entity <UserWeather> for the category 'Weather', an entity <UserNews> for the categories 'Factoid', 'ContinuingNews' or 'PastNews', an entity <UserOrganization> for the category 'Official', and an entity <UserMultimedia> for the category 'MultimediaShare'.
3 Combining Deep Learning Models
In image localization, class activation mapping highlights the class-specific discriminative regions [10]. Class activation mapping can identify the importance of the image regions most relevant to the particular category. We expect that the class activation map for text classification detect the importance of terms.
On the other hand, there has been research using one-shot learning in Long-Short Term Memory. One-shot learning is effective when the number of training data is too small to learn [11,12]. It improves classification accuracy.
We combine CAM with one-shot learning in CNNs as shown in Fig. 1. We expect crisis-related terms are continuously changed through the learning of term weights with high importance.
3.1 Class Activation Mapping for Text Classification
In this paper, class activation mapping is applied for text classification. Fig. 2 shows CAM applied for image localization and text classification.
The class activation mapping is generated by multiplying the weight from the convolution layer and the weight used for classification using the global average pooling in CNNs. This allows us to check the importance of terms in the input data. Through learning, global average pooling creates class activation mapping and represents terms. The proposed method readjusts the terms weight in order of importance and re-inputs them.
3.2 Features for One-shot Learning
CNNs consists of a convolutional layer, a maximum pooling layer, and a fully connected layer [9]. One-shot learning uses CNN's document vectors that come from fully connected layer and category vectors that represent each category. One-shot learning finds a category vector corresponding to the matched category, learns the index of similarity of the document vector and sets the value to 1. In the other case of finding, the category vector corresponding to the unmatched category, it sets the value of index of similarity to zero.
4 Experiments
4.1 Test Collection
In TREC Incident Streams (TREC-IS) 2018 test collection, a stream of tweets is collected by 19 events from the CrisisLex or CrisisNLP initiatives using hashtags and keyword monitoring [14,15]. The tweets are related to the incidents of six different types, e.g. earthquakes, hurricanes, public or shootings [13]. The training set contains six event topics such as 2012 Colorado wildfires, 2012 Costa Rica Earthquake, 2013 Colorado floods, 2012 Typhoon Pablo, 2013 LA Airport Shooting, and 2013 West Texas Explosion. The test set contains 15 event topics such as 2012 Guatemala earthquake, 2012 Philippines floods, 2013 Australia bushfire, 2013 Boston bombings, 2011 Joplin tornado, 2015 Paris attacks and etc.
The TREC-IS task is classifying the tweets into 25 information types. The information type categories, training data and test data for each category are shown in Table 6. The number of tweets in a training set is 1335, whereas the number of tweets in a test set is 22160.
Since the human assessors were allowed to select as many information types when creating the ground truth, the performance is measured in two ways: any-type and multi-type. In any-type evaluation, a system receives a full score for a tweet if it assigned any of the categories that the human assessor selected for that tweet. In the multi-type evaluation, the performance per information type is calculated in a 1 vs. all manner [13]. As evaluation metrics, Precision (P), Recall (R), F1 score (F1), and Accuracy (A) are measured.
4.2 Comparison Experiments
In order to see the effectiveness of the proposed conceptual representation, we have compared our method with baseline term representation for two machine learning methods: support vector machines (SVM) [16] and combining deep learning models:
— baseSVM: SVM with term representation.
— baseComb: Combining deep learning models with term representation.
— conceptSVM: SVM with conceptual representation.
— conceptComb: Combining deep learning models with conceptual representation.
We have compared our method with TREC-IS 2018 track participants with high performance:
4.3 Experimental Results
Table 5 shows the experimental results of information type categorization for any-type.
Table 5 Experimental results for information type
| Method | F1 | Accuracy |
|---|---|---|
| baseSVM | 0.555 | 0.402 |
| baseComb | 0.497 | 0.340 |
| conceptSVM | 0.575 | 0.421 |
| conceptComb | 0.533 | 0.389 |
| KDEIS4_DM | 0.560 | 0.391 |
| Umdhcilfasttext | 0.558 | 0.402 |
| NHK | 0.551 | 0.400 |
The proposed conceptual representation achieved significant improvement in SVM and deep learning models compared to the baseline representation. The proposed method showed high performance compared to other participant methods.
In this experiments, SVM showed better performance than deep learning models. It is due to the small number of a training data for each category even though we adopted one-shot learning method. Table 6 shows results per each information type performance for multi-type.
Table 2 Information type categorization (multi-type) per information type performance in F1 measure
| Information Type Category | Train | Test | baseSVM | conceptSVM | baseComb | conceptComb | |
|---|---|---|---|---|---|---|---|
| data | data | ||||||
| Request | GoodsServices | 0 | 126 | 0 | 0 | 0 | 0 |
| SearchAndRescue | 0 | 286 | 0 | 0 | 0 | 0 | |
| InformationWanted | 10 | 172 | 0.03 | 0.02 | 0 | 0 | |
| CallToAction | Volunteer | 2 | 116 | 0 | 0 | 0 | 0 |
| Donations | 15 | 804 | 0.46 | 0.47 | 0.09 | 0.53 | |
| MovePeople | 26 | 27 | 0.06 | 0.07 | 0.05 | 0.04 | |
| Report | FirstPartyObservation | 28 | 3807 | 0.01 | 0.01 | 0 | 0 |
| ThirdPartyObservation | 15 | 4160 | 0 | 0 | 0 | 0 | |
| Weather | 42 | 1325 | 0.15 | 0.25 | 0.12 | 0.24 | |
| EmergingThreats | 36 | 686 | 0.02 | 0.03 | 0 | 0.01 | |
| SignificantEventChange | 34 | 415 | 0.01 | 0.01 | 0 | 0.01 | |
| MultimediaShare | 127 | 3974 | 0.36 | 0.43 | 0.19 | 0.34 | |
| ServiceAvailable | 15 | 1076 | 0.05 | 0.15 | 0.01 | 0.06 | |
| Factoid | 140 | 2383 | 0.23 | 0.25 | 0.15 | 0.21 | |
| Official | 52 | 403 | 0.08 | 0.08 | 0.06 | 0.10 | |
| CleanUp | 2 | 62 | 0 | 0 | 0 | 0 | |
| Hashtags | 4 | 3363 | 0 | 0 | 0 | 0 | |
| Other | PastNews | 12 | 1351 | 0 | 0.01 | 0.03 | 0 |
| ContinuingNews | 251 | 4871 | 0.41 | 0.36 | 0.36 | 0.36 | |
| Advice | 39 | 1209 | 0.10 | 0.08 | 0.07 | 0.09 | |
| Sentiment | 132 | 6952 | 0.52 | 0.53 | 0.52 | 0.55 | |
| Discussion | 51 | 2060 | 0.08 | 0.07 | 0.06 | 0.04 | |
| Irrelevant | 163 | 2605 | 0.20 | 0.22 | 0.12 | 0.22 | |
| Unknown | 26 | 77 | 0 | 0 | 0.02 | 0.01 | |
| KnownAlready | 113 | 1101 | 0.10 | 0.11 | 0.08 | 0.11 | |
5 Conclusions
In this paper, we proposed conceptual representation for terms, URLs and users in a tweet in order to classify a stream of tweets related to the incident. The crisis-related terms were represented as conceptual entities such as events, category indicators, information types, URLs, and user entities. We have assumed that each conceptual entity would give additional information for information type categorization. Experimental results showed the conceptual representation was effective for crisis-related tweet classification with SVM and CNNs.
