SciELO - Scientific Electronic Library Online

 
vol.22 número1A Heuristic Approach to Detect and Localize Text in Arabic News VideoNew Similarity Function for Scientific Articles Clustering based on the Bibliographic References índice de autoresíndice de materiabúsqueda de artículos
Home Pagelista alfabética de revistas  

Servicios Personalizados

Revista

Articulo

Indicadores

Links relacionados

  • No hay artículos similaresSimilares en SciELO

Compartir


Computación y Sistemas

versión impresa ISSN 1405-5546

Comp. y Sist. vol.22 no.1 México ene./mar. 2018

http://dx.doi.org/10.13053/cys-22-1-2784 

Articles of the Thematic Issue

Generating Aspect-based Extractive Opinion Summary: Drawing Inferences from Social Media Texts

Rajesh Piryani1  * 

Vedika Gupta2 

Vivek Kumar Singh3 

1 South Asian University, Department of Computer Science, New Delhi, India

2 National Institute of Technology Delhi, Department of Computer Science, Delhi, India. guptavlabs@gmail.com

3 Banaras Hindu University, Department of Computer Science, Varanasi, India. vivek@bhu.ac.in

Abstract:

This paper presents an integrated framework to generate extractive aspect-based opinion summary from a large volume of free-form text reviews. The framework has three major components: (a) aspect identifier to determine the aspects in a given domain; (b) sentiment polarity detector for computing the sentiment polarity of opinion about an aspect; and (c) summary generator to generate opinion summary. The framework is evaluated on SemEval-2014 dataset and obtains better results than several other approaches.

Keywords: Aspect-level sentiment analysis; laptop; polarity; sentiment summarization; big data

1 Introduction

The widespread growth of internet has resulted in a generation of huge volume of unstructured data available online as blogs, reviews etc. Online reviews have become an important tool of public communication among the purchasers and/or prospective purchasers of an item. While the manufacturer of the item may mention all the configuration details of an item, but the exact working and performance-based opinions is generated by the consumers using that item. Many online forums and electronic commerce websites provide users a platform to share their views regarding a product (and its various aspects), that is on sale. This wide variety of information that is available online serves as a goldmine for harvesting knowledge. Processing this large volume of data will not only benefit customers, but also market competitors and academic researchers. However, owing to the immensely populated web, reading each and every review becomes a tedious task. It is highly required to have such automated systems that can understand the essence of what all is said, and show that in a precise gist or summary.

Summarization can be defined as the technique of summing up key content from variety of information sources. This technique has become a vital part of everyday life. The technique of summarization is not new; in fact, it existed since ages from the time newspapers identified a suitably apt headline for a piece of news item.

Those who do not want to go through the entire text, can just scroll their eyes along the headline which in a way summarizes the complete article. Its only since last decade, that the amount of digital data has grown so large that its being called Big Data. This Big Data has compelled the need for automated computational means which can capture and present the main points as a summary. With summaries, consumers and business owners can make effective decisions in less time.

Sentiment summarization is a relatively more challenging task. Since, to perform summarization of sentiments, we first need the sentiment bearing sentences. There are number of complexities involved. Reviewers talk about various aspects of a particular product in a sentence or paragraph. For instance, a review about a laptop might include opinion about its performance, screen, hardware and so on. Analyzing sentiment at aspect level is a daunting research issue that is popular both in industry and the academic research community. Aspect-level sentiment analysis involves identification of aspects, aspect term-opinion dependency and opinion polarity computation. Many authors have analyzed sentiment from reviews based on items, services, events, news or films.

There are basically two types of sentiment summaries that can be generated- abstractive summary and extractive sentiment summary. Abstractive summaries provide the gist of main ideas of the document in words, not necessarily in the words used in the input sentences. It uses some linguistic features to best describe the text in a concise manner. Extractive summaries provide a summary of the given sentences by extracting the sentences, as it is, based on some relevance and proximity criteria. It selects sentences using statistical and linguistic features of the natural language. Although, automatic extractive summarization may not produce accurate summary as can be generated by a human expert, still, the essence of the story can be produced in a nutshell for a novice user.

This paper presents work done on extractive summarization of online laptop reviews at aspect level. The laptop reviews have been collected from SemEval-2014 dataset1. An aspect detection algorithm has been designed to identify aspects of laptops from the review sentences. Polarity of opinion is computed by using dependency relation and a set of rules. Finally, an extractive sentiment summary of the reviews in the dataset is also generated for consumption of users.

The rest of the paper has been organized as follows: Section 2, discusses the related work. Section 3, describes the framework and methodology. Section 4, presents experimental work along with dataset used and evaluation of performance of the system. Conclusions are stated in Section 5.

2 Related Work

Sentiment summarization at aspect level is been performed by various natural language processing methods. In most of the cases, the job involves:

  • (i) Analyzing review sentences to identify different terms/ phrases that denote an aspect associated with the item or event in question.

  • (ii)Identifying opinionated phrase related to each aspect.

  • (iii) Computing sentiment polarity of all such opinions.

  • (iv) Generating summary of opinion about each aspect.

All these parts involve independent work in their own. Several previous research Works have focused on one or more of these tasks. Some of the closely related Works are referenced here.

A framework along with an automated rule generation algorithm for extracting relevant features for extracting aspects has been proposed in [1]. Another review summarization framework has been proposed in [2], which categorizes reviews on the basis of aspects and then generates summary. Some previous Works tried to take an integrated approach. One such integrated end-to-end framework for aspect based summarization has been described in [3]. It uses clustering approach based on Map Reduce framework.

Similar frameworks have been proposed for Chinese microblog texts [4], and Thai public opinions [5]. The framework in [6] is based on modeling content structure and in [7, 8], on rating and ranking of reviews. Few semantic frameworks as integrated methods are proposed in [9, 10, 11]. Another kind of approach involves locating important sentences in a document. One such work, described in [12], gives a document summarization framework that identifies and extracts important parts of a document to form a coherent summary using sentiment analysis.

Another work [13], devised a model that jointly learns to focus on relevant contextual parts of a document for multi-aspect sentiment analysis and summarization. It comes from the fact that reviews are often written in a disorganized fashion, leading to problems in knowledge acquisition. The approach calls for further reduction in the navigation time of consumers, by filtering important aspects in a review and summarizing those reviews.

In [14] attention is put on identification of important aspects that are usually more commented upon than other aspects. This improves the usability of the diverse reviews. Some authors have also considered other factors such as subjectivity and text quality besides just relevance of a review, in generating quality summaries. An Integer Linear Programming based summarization framework has been developed in [15] to incorporate these features.

Microblogging sites such as Twitter have turned out to be well-known channels for individuals to express their assessments towards a wider scope of subjects. Twitter manifests itself as a basic component to any interpersonal interaction based application, serving as a rich information reporting media. Twitter generates an enormous volume of texts (i.e. tweets) conveying users' perspectives and opinions each moment, which requires automated sentiment summarization. Researchers have utilized this pool of information for generating meaningful extractive summaries.

Summarizing opinions on trending topics and entities is a popular research topic for twitter researchers. Several previous Works have used topic modeling [16, 17, 18] for the purpose. Another work [19] considered factors of user credibility and quality of opinion for sentiment summarization. Different machine learning approaches have been opted for summarizing event specific tweets- soccer match in [20] and critical events such as disasters in [21]. Twitter also allows added sentiments by allowing the usage of hashtags and smileys. These can be used as sentiment labels thereby reducing the manual effort in annotating the text. One such work is presented in [22]. In the present paper an integrated framework is proposed for generating aspect-based opinion summaries. A linguistic approach is used in the framework which removes the requirement of any training set and training to machine.

3 Proposed Framework

This section describes our proposed framework for generating aspect-level sentiment summary from free-form textual reviews. A top-level architectural block diagram of the framework is presented in Figure 1. The system receives reviews as input, pre-processes each review and breaks it into sentences, then identifies the aspect terms and computes sentiment polarity of the opinion about each aspect. Finally, the sentences are clustered aspect-wise and ranked according to their importance to generate an extractive opinion summary. The integrated approach used in the framework, thus generates the aspect-wise extractive sentiment summary from free-form textual reviews.

Fig. 1 Architectural Block Diagram of Framework 

3.1 Creation of a Seed Aspect Vector List

As this work is focused on laptop domain, the aspects of laptop have been identified as follows: hardware, accessibility, accessories, performance, software, overall, cost, maintenance & support, add-ons, battery, connectivity & networking, memory & storage and multimedia & gaming. Only using the aspect category name for aspect identification, will limit the probability for identification of correct opinion of reviewers. The review writers typically use variety of words or phrases to express their sentiment regarding particular aspect of laptop. Therefore, to enlarge the laptop domain of related aspect terms, a seed aspect vector list is created. Table 1 depicts the seed aspect vector list. For example, if reviewer is using the word “RAM”, “Hard disk”, “storage” etc. then s/he is talking about “memory & storage” aspect of the laptop. The seed aspect terms are identified by analyzing 1000 reviews (of Laptop domain), from Amazon and Flipkart. The developed framework can crawl the different laptop specification information from Amazon (or other similar sites specified) to update the seed aspect vector list.

Table 1 Seed Aspect Vector List 

Aspect Category Aspect Terms
Hardware Body, design, dimensions, cord, edges, edge, monitor, fan, CD/DVD tray, weight, weighs, build quality, hardware, weighted
Accessibility Track pad, numeric pad, number pad, trackpad, mousepad, mouse buttons, mouse button, keyboard, key, touch pad, pointing device, touchscreen, touch screen, backlit, touchpad, keypad, key pad, keys, external mouse, cordless mouse, touch-pad, mic-jack, headphone, headphones
Accessories Carry Bag, Pouch, case, carry case, bag, Budget
Performance Performance, runs, speed, responds, response, navigate, navigation, drivers, use, power, processor, Clock-speed, Graphic Processor, Speed, Cache, startup, start up, boot up, boots up, bios, motherboard, mother board, cooling system, computing, specification, spec, performing
Software Browser, virus scan, programs, program, application, applications, GUI, operating system, OS, softwares, software, updates, security, recovery cd
Overall Computer, laptop, model, design, system, appearance, value, working, works, machine, construction quality, longevity, Robustness, Sales package,
Cost Costing, shipping, pricetag, delivery, price, charges, cost, priced, pricing, expense
Maintenance & Support Service, tech support, technical service, support, staff, sales, service center, tech guy, after sales, assistance, representative, sales associate, warrenty, warranty, techie, online help, warrentys, shipped, reutrn policy, extended warranty, extended warrenties, repair
Add-ons Feature, features, multi-touch gestures, Fingerprint sensor, face recognition
Battery Battery, battery cell, Battery Type, Battery Backup, battery life, charger
Connectivity & Networking Wireless LAN, Bluetooth, Ethernet, wifi, wi fi, wi-fi, Bluetooth, Ports, USB 3.0 slots, USB 2.0, SD Card Reader, Headphone Jack, Microphone Jack, Built-in Camera, builtin Microphone, mic, Digital Media Reader, connectivity, HDMI, VGA, webcam, internet connection, internet speed, subwoofer, connection, built-in wireless
Memory & Storage RAM, RAM type, Hard Drive, Inbuilt HDD, RPM, Optical Drive, Capacity, RAM speed, Memory Layout, Storage, HDD Capacity, HDD type, SSD, MMC, memory, hard disk drive, hard disk, hard drive space, external harddrives, external hard drive, internal hard disk
Multimedia & Gaming Display Size, Display Resolution, Display Type, Screen Size , Maximum Display Resolution, dpi, Panel Type, graphic card, graphics card, Graphics Memory, graphics, display, resolution, web-cam, web cam, webcam, cam, camera, Secondary cam, Audio, Speakers, Sound Technologies, Sound, Microphone, Microphone Type, GPU, Graphical processing unit, Video, Game, gaming, games, game playing, game, look, protector, HD, videocard, non-dedicated graphics card,

3.2 Aspect Term Identification

For aspect term identification from user reviews, first each review is parsed with Stanford POS Tagger2 and then the parsed review is pre-processed by eliminating the punctuation marks and splitting into sentences. For each sentence we apply Algorithm 1 as explained below.

Pseudocode of Aspect Detection Algorithm 

Here S refers to set of sentences, we used TechTerms3 site as online dictionary. TechTerms is free online dictionary of computer and internet terms. The online dictionary program takes word/phrase as input, if it found definition on TechTerms site then it returns word/phrase as aspect.

3.3 Opinion Word/ Phrase Identification

Sentiment lexicon can be used to identify whether word/phrase is opinionated or not. The publicly available lexicons are SentiWordNet4, SenticNet5, MPQA6 Subjectivity lexicon etc. SentiWordNet gives three numerical scores (positive, negative, objective) for each WordNet synset. Score ranges from 0 to 1. SenticNet is lexicon for concept level sentiment analysis. SenticNet provides one polarity score for each word/phrase and score ranges between -1 to 1. MPQA Subjectivity lexicon consists of strongly, weakly positive and negative words. Thet et al. [23], proposed a new lexicon named generic lexicon derived from SentiWordNet and MPQA subjectivity lexicon. In this paper, we have used generic lexicon and SenticNet.

3.4 Linguistics Rules for Sentiment Polarity Computation

The goal is to compute sentiment polarity about each aspect. This task requires detection of the opinionated word and computation of sentiment polarity score.

The following linguistic rules have been used to compute the polarity of an aspect.

  1. If a sentence contains one aspect, then move 5 words to the left of the aspect and 5 words to the right of the aspect to search opinionated word.

    1. If encountered with adjective then use AAC [24], scheme to compute the polarity of aspect. For example, “It offers many great features!”, sentence contain adjective “great” before aspect “features”, so we can use this rule to compute polarity of the aspect.

    2. If none of adjective is encountered both side, but verb is encountered then use AVC [24], scheme to computes the polarity score. For example, “Easy to start up and does not overheat.”, in this sentence “easy” marked as noun by Stanford POS Tagger and rule 1 will not work. But nearby aspect “startup”, “overheat” verb is present, so we can use this rule to compute the polarity of the aspect.

    3. If none of the adjective and verb is encountered search for adverb before one word of aspect if it is present, then compute the polarity of adverb from sentiment lexicon. For example, “No backlit keyboard”, this sentence doesn’t contain any adjective, verb, but adverb is present, so we can use this rule to compute the polarity of the aspect.

  2. If sentence contain more than one aspect then for each aspect A i use rule1.

    1. If aspect A 1 polarity is not computed by rule1 then search for other aspect A 2 near to A 1. If before or after aspect A 1 other aspect A 2 is encountered and has the polarity then A 1 is polarity is same as A 2 . For example, “I love the operating system and the preloaded software”. In this sentence for aspect “preloaded software” polarity is not computed by rule1, but before “preloaded software” another aspect “operating system” is encountered so polarity of “preloaded software” will be same as “operating system”.

3.5 Extractive Sentiment Summary Generation

To generate extractive sentiment summary, first task is to cluster the sentences aspect wise. In clustering for each aspect separate positive sentences in one cluster and negative sentences into other cluster. The next task is to ranking the clustered sentences. LexRank [25], algorithm is used for ranking the sentence. For each aspect we generate positive and negative summary with top 5 ranked sentences. Example of positive summary for battery aspect is given below: “It's fast and has excellent battery life. Screen is awesome, battery life is good. The display on this computer is the best I've seen in a very long time, the battery life is very long and very convenient. It is light and the battery last a very long time. It has a lot of memory and a great battery life.”

4 Experimental Work

4.1 Dataset

For experimental purpose we have used SemEval 2014 task 4 laptop dataset. The training part of dataset is used for experiment purpose. It comprises of 3041 sentences extracted from laptop reviews. Human annotator tagged the aspect terms. The dataset comprises of sentences, manually identified aspect terms, position where aspect occur in sentence and sentiment polarity of the aspect [26]. Some sentences do not contain either aspect term or position and sentiment polarity.

4.2 Evaluation

Table 2, shows the comparison of our aspect identification approach with some of the other state of the art approaches. It can be seen from the table that our approach outperforms many other approaches. Table 3, presents the comparison of our sentiment polarity computation approach with other state of art. It can be seen from the table, that our approach outperforms the rest. Table 4, shows the result of our sentiment polarity computation approach. The recall is low because in some sentences, aspect has conflicting polarities which are difficult to detect.

Table 2 Aspect Identification Results: Comparison with state of the art ZW stands for [27] and SEA for [28

Framework Recall Precision F-Score
ZW 66.51% 84.80% 74.55%
SEA 78.35% 86.72% 82.32%
Aspect Seed Vector (ours) 85.93% 91.05% 88.42%

Table 3 Sentiment Level Results: Comparison with state of the art ZW stands for [27

Framework Accuracy
ZW 63.22%
Aspect Seed Vector (ours) 77.67%

Table 4 Sentiment Level Results 

Framework Recall Precision F-Score
Aspect Seed Vector (ours) 77.67% 82.28% 79.9%

For evaluation of extractive sentiment summary, we manually created 5 positive reference summaries, and 5 negative reference summaries. For summary evaluation we used ROUGE-L [29] approach, Table 5 shows the summary evaluation result. Previously, Hu and Liu [30], and Tadano et al. [31] have attempted sentiment summarization on product reviews dataset and achieved 0.338, 0.369 F-Scores, respectively. Our approach obtains better values than these approaches.

Table 5 ROUGE- L Aspect Wise Extractive Opinion Summary Extraction Result 

Aspect Recall Precision F-Score
Battery 0.506 0.441 0.441
Performance 0.377 0.620 0.397

5 Conclusion

This paper describes an integrated framework that takes reviews as input and generates aspect wise extractive sentiment summary. This comprises series of tasks ranging from aspect identification, computation of sentiment polarity to generation of aspect wise extracted summary. The framework proposed is a linguistic-based approach and does not require any supervised training or the training set. It’s an independent, self-contained framework that can be used for generating opinion summaries from free-form text. Only dependency on external resource is that for the technical term dictionary used to identify aspects in laptop domain. The evaluation on standard dataset shows better performance of the proposed approach as compared to several state of the art approaches. The framework can be extended for application on other data domains.

Acknowledgements

This work has been partly supported by Indo-Mexican (DST-CONACYT), Joint Research project titled ‘Machine Learning Techniques for Context Aware Recommender Systems’ (Grant No: INT/MEXICO/P-13/2012).

References

1. Rao, A., & Shah, K. (2016). An optimized rule based approach to extract relevant features for sentiment mining. Computing for Sustainable Global Development (INDIACom), 3rd International Conference on IEEE, pp. 2330-2336. [ Links ]

2. Gu, X., & Kim, S. (2015). What Parts of Your Apps are Loved by Users? 30th IEEE/ACM International Conference, Automated Software Engineering (ASE), pp. 760-770. DOI: 10.1109/ASE.2015.57. [ Links ]

3. Priya, V., & Umamaheswari, K. (2016). Ensemble based Parallel k means using Map Reduce for Aspect Based Summarization. Proceedings of the International Conference on Informatics and Analytics, ACM. DOI: 10.1145/2980258.2980308. [ Links ]

4. Zhou, X., Wan, X., & Xiao, J. (2016). CMiner: Opinion Extraction and Summarization for Chinese Microblogs. IEEE Transactions on Knowledge and Data Engineering, Vol. 28, No. 7, pp. 1650-1663. DOI: 10.1109/TKDE.2016.2541148. [ Links ]

5. Deerosejanadej, C., Promon, S., & Achalakul, T. (2016). A Framework for Mining Thai Public Opinions. Big Data, pp. 339-355. DOI: 10.1016/b978-0-12-805394-2.00014-3. [ Links ]

6. Sauper, C., Haghighi, A., & Barzilay, R. (2010). Incorporating content structure into text analysis applications. Proceedings Conference on Empirical Methods in Natural Language, Association for Computational Linguistics, pp. 377-387. [ Links ]

7. Krestel, R., & Dokoohaki, N. (2015). Diversifying customer review rankings. Neural Networks, Vol. 66, pp. 36-45. DOI: 10.1016/j.neunet.2015.02.008. [ Links ]

8. Saggion, H., Lloret, E., & Palomar, M. (2012). Can text summaries help predict ratings? a case study of movie reviews. International Conference on Application of Natural Language to Information Systems, Springer, Vol. 7337, pp. 271-276. DOI: 10.1007/978-3-642-31178-9_33. [ Links ]

9. Asgarian, E., & Kahani, M. (2014). Designing an integrated semantic framework for structured opinion summarization. European Semantic Web Conference, Springer, Vol. 8465, pp. 885-894. DOI: 10.1007/978-3-319-07443-6_62. [ Links ]

10. Condori, R.E.L., & Pardo, T.A.S. (2017). Opinion summarization methods: Comparing and extending extractive and abstractive approaches. Expert Systems with Applications, Vol. 78, pp. 124-134. DOI: 10.1016/j.eswa.2017.02.006. [ Links ]

11. Piryani, R., Gupta, V., & Singh, V.K. (2017). Movie Prism: A novel system for aspect level sentiment profiling of movies. Journal of Intelligent and Fuzzy Systems, Vol. 32, No. 5, pp. 3297-3311. DOI: 10.3233/JIFS-169272. [ Links ]

12. Dabholkar, S., Patadia, Y., & Dsilva, P. (2016). Automatic Document Summarization using Sentiment Analysis. Proceedings of the International Conference on Informatics and Analytics, ACM. DOI: 10.1145/2980258.2980362. [ Links ]

13. Pappas, N., & Popescu-Belis, A. (2017). Explicit Document Modeling through Weighted Multiple-Instance Learning. Journal of Artificial Intelligence Research, Vol. 58, pp. 591-626. DOI: 10.1613/jair.5240. [ Links ]

14. Zha, Z.J., Yu, J., Tang, J., Wang, M., & Chua, T.S. (2014). Product aspect ranking and its applications. IEEE Transactions on Knowledge and Data Engineering, Vol. 26, No. 5, pp. 1211-1224. DOI: 10.1109/TKDE.2013.136. [ Links ]

15. Ying, D., & Jiang, J. (2015). Towards Opinion Summarization from Online Forums. ACL. [ Links ]

16. Banu, S.H., & Chitrakala, S. (2016). Trending Topic Analysis using novel sub topic detection model. IEEE International Conference Advances in Electrical, Electronics, Information, Communication and Bio-Informatics (AEEICB), pp. 157-161. DOI: 10.1109/AEEICB.2016.7538263. [ Links ]

17. Banu, S.H., & Chitrakala, S. (2016). Tweet specific extractive summarization framework towards trending topic analysis. IEEE Conference International Information Communication and Embedded Systems (ICICES), pp. 1-6. DOI: 10.1109/ICICES.2016.7518915. [ Links ]

18. Meng, X., Wei, F., Liu, X., Zhou, M., Li, S., & Wang, H. (2012). Entity-centric topic-oriented opinion summarization in twitter. Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining (ACM), pp. 379-387. DOI: 10.1145/2339530.2339592. [ Links ]

19. Li, Y.M., & Li, T.Y. (2011). Deriving marketing intelligence over microblogs. 44th IEEE International Conference on System Sciences (HICSS), Hawaii, pp. 1-10. DOI: 10.1109/HICSS.2011.143. [ Links ]

20. Jai-Andaloussi, S., El Mourabit, I., Madrane, N., Chaouni, S.B., & Sekkaki, A. (2015). Soccer Events Summarization by Using Sentiment Analysis. IEEE International Conference on Computational Science and Computational Intelligence (CSCI), pp. 398-403. DOI: 10.1109/CSCI.2015.59. [ Links ]

21. Rudra, K., Ghosh, S., Ganguly, N., Goyal, P., & Ghosh, S. (2015). Extracting situational information from microblogs during disaster events: a classification-summarization approach. Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, pp. 583-592. DOI: 10.1145/2806416.2806485. [ Links ]

22. Davidov, D., Tsur, O., & Rappoport, A. (2010). Enhanced sentiment learning using twitter hashtags and smileys. Proceedings of the 23rd international conference on computational linguistics: posters, Association for Computational Linguistics, pp. 241- 249. [ Links ]

23. Thet, T.T., Na, J.C., & Khoo, C.S. (2010). Aspect-based sentiment analysis of movie reviews on discussion boards. Journal of information science, Vol. 36, No. 6, pp. 823-848. DOI: 10.1109/ iMac4s.2013.6526500. [ Links ]

24. Singh, V.K., Piryani, R., Uddin, A., & Waila, P. (2013). Sentiment analysis of movie reviews: A new feature-based heuristic for aspect-level sentiment classification. IEEE International Multi-Conference Automation, Computing, Communication, Control and Compressed Sensing (iMac4s), pp. 712-717. DOI: 10.1109/iMac4s.2013.6526500. [ Links ]

25. Erkan, G., & Radev, D.R. (2004). Lexrank: Graph-based lexical centrality as salience in text summarization. Journal of Artificial Intelligence Research, Vol. 22, pp. 457-479. DOI: 10.1613/ jair.1523. [ Links ]

26. Pontiki, M., Galanis, D., Pavlopoulos, J., Papageorgiou, H., Androutsopoulos, I., & Manandhar, S. (2014). Semeval-2014 task 4: Aspect based sentiment analysis. Proceedings of SemEval, pp. 27-35. [ Links ]

27. Toh, Z., & Wang, W. (2014). DLIREC: Aspect Term Extraction and Term Polarity Classification System. Proceedings of the 8th International Workshop on Semantic Evaluation, pp. 235-240. DOI: 10.3115/ v1/s14-2038. [ Links ]

28. Poria, S., Cambria, E., & Gelbukh, A. (2016). Aspect extraction for opinion mining with a deep convolutional neural network. Knowledge-Based Systems, Vol. 108, pp. 42-49. DOI: 10.1016/j.knosys.2016.06.009. [ Links ]

29. Lin, C.Y. (2004). Rouge: A package for automatic evaluation of summaries. Proceedings of the ACL-04 workshop, Vol . 8. [ Links ]

30. Hu, M., & Liu, B. (2004). Mining and summarizing customer reviews. Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 168-177. DOI: 10.1145/1014052.1014073. [ Links ]

31. Tadano, R., Shimada, K., & Endo, T. (2010). Multi-aspects Review Summarization Based on Identification of Important Opinions and their similarity. PACLIC Institute for Digital Enhancement of Cognitive Development, Waseda University, pp. 685-692. [ Links ]

Recibido: 08 de Agosto de 2016; Aprobado: 12 de Octubre de 2016

* Corresponding author: Rajesh Piryani, e-mail: rajesh.piryani@gmail.com

Creative Commons License This is an open-access article distributed under the terms of the Creative Commons Attribution License