Open Collections

UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Evaluating open relation extraction over conversational texts Imani, Mahsa 2014

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


24-ubc_2014_spring_imani_mahsa.pdf [ 365.03kB ]
JSON: 24-1.0165856.json
JSON-LD: 24-1.0165856-ld.json
RDF/XML (Pretty): 24-1.0165856-rdf.xml
RDF/JSON: 24-1.0165856-rdf.json
Turtle: 24-1.0165856-turtle.txt
N-Triples: 24-1.0165856-rdf-ntriples.txt
Original Record: 24-1.0165856-source.json
Full Text

Full Text

Evaluating Open Relation ExtractionOver Conversational TextsbyMahsa ImaniB.Sc., Tarbiat Moalem University of Tehran, 2008M.Sc., Isfahan University of Technology, 2011A THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFMASTER OF SCIENCEinThe Faculty of Graduate and Postdoctoral Studies(Computer Science)THE UNIVERSITY OF BRITISH COLUMBIA(Vancouver)January 2014c? Mahsa Imani 2014AbstractIn this thesis, for the first time the performance of Open IE systems onconversational data has been studied. Due to lack of test datasets in thisdomain, a method for creating the test dataset covering a wide range of con-versational data has been proposed. Conversational text is more complexand challenging for relation extraction because of its cryptic content and un-grammatical colloquial language. As a consequence text simplification hasbeen used as a remedy to empower Open IE tools for relation extraction.Experimental results show that text simplification helps OLLIE, a state ofthe art for relation extraction, find new relations, extract more accuraterelations and assign higher confidence scores to correct relations and lowerconfidence scores to incorrect relations for most datasets. Results also showsome conversational modalities such as emails and blogs are easier for rela-tion extraction task while people reviews on products is the most difficultmodality.iiPrefaceThis dissertation is original, unpublished, independent work by the author,Mahsa Imani. I designed the research project with the help of GiuseppeCarenini and Yashar Mehdad. I proposed an approach for mitigating theproblems faced in this research area. I conducted all the experiments andevaluated the performance of the proposed approach. I analyzed the exper-imental data and wrote the whole thesis. Giuseppe Carenini and YasharMehdad were the supervisory authors on this project and were involvedthroughout the project in concept formation and manuscript edits.iiiTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiAcknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . viiiDedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Open Information Extraction and Its Challenges . . . . . . . 21.3 Conversational Data and New Challenges . . . . . . . . . . . 31.4 Text Simplification . . . . . . . . . . . . . . . . . . . . . . . 31.5 Problem Statement and Contribution . . . . . . . . . . . . . 41.6 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 Background and Related Work . . . . . . . . . . . . . . . . . 72.1 Conversational Datasets . . . . . . . . . . . . . . . . . . . . . 72.2 Relation Extraction . . . . . . . . . . . . . . . . . . . . . . . 82.2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . 82.2.2 Traditional IE . . . . . . . . . . . . . . . . . . . . . . 92.2.3 Open IE . . . . . . . . . . . . . . . . . . . . . . . . . 102.3 Text simplification . . . . . . . . . . . . . . . . . . . . . . . . 152.3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . 152.3.2 Applications and Approaches . . . . . . . . . . . . . . 16ivTable of Contents3 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183.1 Dataset Creation . . . . . . . . . . . . . . . . . . . . . . . . . 183.1.1 Reviews . . . . . . . . . . . . . . . . . . . . . . . . . . 183.1.2 Emails . . . . . . . . . . . . . . . . . . . . . . . . . . 193.1.3 Meetings . . . . . . . . . . . . . . . . . . . . . . . . . 193.1.4 Blogs and Online Discussions . . . . . . . . . . . . . . 193.1.5 Social Networks . . . . . . . . . . . . . . . . . . . . . 193.1.6 Dataset Characteristics . . . . . . . . . . . . . . . . . 193.1.7 Sampling Method . . . . . . . . . . . . . . . . . . . . 203.2 Open IE on Conversational Data . . . . . . . . . . . . . . . . 213.3 Text Simplification for Open IE . . . . . . . . . . . . . . . . 224 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . 234.1 Evaluation Metrics . . . . . . . . . . . . . . . . . . . . . . . . 234.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244.3 Analysis and Discussion . . . . . . . . . . . . . . . . . . . . . 275 Conclusion and Future Work . . . . . . . . . . . . . . . . . . 31Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33vList of Tables3.1 Dataset characteristics. . . . . . . . . . . . . . . . . . . . . . 203.2 Feature set used in sampling sentences. . . . . . . . . . . . . . 214.1 Accuracy before simplification. . . . . . . . . . . . . . . . . . 254.2 Accuracy after simplification. . . . . . . . . . . . . . . . . . . 254.3 Average confidence score before simplification. . . . . . . . . 264.4 Average confidence score after simplification. . . . . . . . . . 26viList of Figures4.1 Accuracy of extraction when the both arguments and relationphrase are correct. . . . . . . . . . . . . . . . . . . . . . . . . 284.2 Average confidence score when the both arguments and rela-tion phrase are correct. . . . . . . . . . . . . . . . . . . . . . . 284.3 Average confidence score when the relation phrase is incorrect. 29viiAcknowledgementsI would like to express my special appreciation and thanks to my advisor, Dr.Giuseppe Carenini, who introduced me to Natural Language Processing andallowed me to grow as a research scientist in this area. His advice, knowledgeand encouragement allowed me to develop and pursue this thesis.I would also like to express my gratitude to Dr. Reymanod Ng and Dr.Yashar Mehdad for their valuable advice, help, and guidance.Last but not least, I would like to thank my friends and family for theirlove and support during all my studies.viiiDedicationTo my mother, Ehteram Jafari, for her unconditional love and supportthroughout my life.ixChapter 1Introduction1.1 MotivationIn past human could only interact and communicate with the people theyknow by speaking or writing letters about events, concepts and ideas. Withthe invention of internet and prevalence of email systems, blogs, fora discus-sions and social networking, now people who even dont know each other canparticipate in different conversations and discuss their thoughts, feelings andopinions. They can ask any question in social streams or online discussiongroups and find the answer by reading and analyzing the comments postedby different people around the world. They can discuss new products andservices and make an informed decision.The conversational data is growing in an exponential rate. Everyday newreviews are written and new discussions are ongoing in social media aboutproducts, services and events and nobody is able to read them all and makean informative summary of them. People may want to join a discussion heldby more than 100 people and need to know what details have been discussedby the time they joined. To take advantage of this massive conversationaldata, we need new tools to help us summarize and find relevant information.To help people find what is closer to their information need, we need newtools to deal with this data explosion. To effectively manage, summarize,search and find relevant information, structured knowledge is required. Re-lation extraction is the task of finding relationships between entities in textand is an effective way to convert unstructured text data such as blogs, webpages, news, scientific literature, and online reviews into structured knowl-edge. This structured knowledge offers users and organizations an effectiveand novel way to get and analyze the information they need to achieve theirgoals.There are many other scenarios in which we are interested in discoveringrelationships within a set of entities in documents. Relations can be usedfor finding gene-disease relationships [16], finding the relationships betweendrugs, genes/proteins and diseases [28], question answering [48, 58], summa-rization [51], automatic database generation, intelligent document searching,11.2. Open Information Extraction and Its Challengesranking and indexing [3], ontology population [39, 43, 55, 58], and findingprotein-protein interactions [29, 33].1.2 Open Information Extraction and ItsChallengesIn Traditional Information Extraction (IE), the relation of interest has to bespecified in advance. One has to provide those systems with new extractionpatterns or training examples for new relations. These systems require onepass over corpus for each relation and hence, they are not scalable withthe size and variety of the web corpus [5]. Open IE systems address thisproblem by extracting relations from arbitrary sentences without requiringdomain-specific knowledge and target relations in advance [5].Open IE systems are scalable in a sense that they extract various rela-tions in a single pass or few numbers of passes over the corpus [5]. Stateof-the-art Open IE systems such as ReVerb [24], WOE [62], OLLIE [38],SONEX [42], TreeKernel [63], EXEMPLAR [19] extract web-scale informa-tion in the format of relational tuples (arg1; rel; arg2 ) in which the relationphrase rel expresses a relation between arg1 and arg2.There are many challenges in extracting semantic relationships betweenentities. The most important one is the variety of relation forms which makesit very difficult to be effictively learnt through machine learning approachesor to be captured through regular expressions and rule based systems: Re-lations can be synonymous [5, 48, 64], negated [40], n-ary [40], conditionallytrue [38], infrequent [21], or implicit [63]; They can have light verb constructs[24], non-contiguous relation phrases [24], or subsume other relations [58].Not only relation forms are various but also their arguments. Argumentscan also be synonymous and have different forms. They can be a NounPhrase (NP) [24, 38, 62], a Named Entity (NE) [19, 42], or even a sentence1.All these challenges have been addressed in the literature but each re-lation extraction approach tackles a subset of these problems. Long, com-pound and complex sentences pose new challenges and make current ex-traction approaches considerably less effective. These tools fail to find allthe relations when a sentence contains relative clause modifier, referent, orrelative relations [24].Open IE can be utilized to convert the massive amount of availableconversational data into structured knowledge and as a consequence help1[15] introduced nested relations in which one of the arguments is a sentence.21.3. Conversational Data and New Challengesus summarize, search and find relevant information. But conversationaltext poses new challenges for Open IE due to its specific characteristicsincluding cryptic content, lots of abbreviations, ungrammatical and informallanguage. The problems arise from difficulty in parsing the sentence at thepreprocessing step to extracting relations themselves.1.3 Conversational Data and New ChallengesConversational data is growing in an exponential rate in the forms of Emails,blogs, reviews, meeting records, or posts in social streams [9]. This data isan invaluable source of information. It provides organizations and peoplewith public feelings and opinions towards new products, services, and events[45]. As a consequence, there is an ongoing research on conversational datain order to represent the content of these conversations in an informativeway to summarize them, find the relevant information and the content worthreading e.g. [35, 41, 44, 57].Sentences in conversational data such as social streams, chat logs, blogsand Email threads are complex and noise-prone [9]. They often have anungrammatical colloquial language, more abbreviations, and may not statethe full relation which is often assumed in relation extraction task. Hencethey pose new challenges and make current extraction approaches consider-ably less effective. There is another challenge in applying these techniquesdesigned for extracting relations from non-conversational well-written textto conversational text. Performance of these techniques depends on theoutput of preprocessing steps such as Part Of Speech (POS) tagging, NPchunking, NE tagging and dependency parsing whose accuracy degrade forconversational text. About 8% of missed extractions and 7-32% of incorrectextractions in ReVerb, WOE-parse and OLIIE are due to incorrect pars-ing [24, 62]. Apart from these challenges, sentences can be simplified byfollowing a set of lexical and syntactic rules [14, 18, 52] or log-linear mod-els [4]. Text simplification can improve the accuracy of preprocessing steps[13, 14, 52] as well as relation extraction by breaking down each complexsentence into semantically equivalent shorter sentences.1.4 Text SimplificationText simplification is the process of simplifying texts while preserving theirmeaning and information to increase understandability or make it easier toprocess by computers [4]. Text simplification can be syntactic or lexical. To31.5. Problem Statement and Contributionsimplify texts lexically difficult words are substituted by easier words. Forsyntactic simplification, a set of rules [14, 18, 52] or log-linear models [4] canbe utilized to simplify sentences by breaking down them into shorter andsimpler sentences.Text simplification has been studied as a preprocessing step for severalNatural Language Processing (NLP) tasks such as relation extraction [33],semantic role labelling (SRL) [61], machine translation [46], summarization[53], and improving the accuracy of parsers [13, 52]. Preprocessing text tosimplify it has been inspired by the fact that performance of these systemsrapidly deteriorates as the length and complexity of the sentence increases[13, 33].Most of the errors in parsing are due to long, complex, and ambiguoussentences and it has been shown that text simplification and compressioneases summarization by converting complex long sentences into shorter sen-tences and dropping non-essential information [53]. Performance of Open IEsystems depends on the output of preprocessing step such as POS tagging,NP chunking, NE tagging and dependency parsing whose accuracy degradefor complex conversational text. Syntactically simplifying texts will lead tomore accurate sentence level analysis and hence a more accurate relationextraction.Jonnalagadda and Gonzalez [33] showed that sentence simplification con-siderably helps relation extraction in the domain of biomedical texts whichusually have longer sentences with more abbreviations and relative clausesthan less specialized and less technical texts like news. As opposed to sci-entific literature in which sentences are grammatically correct, sentences inconversational texts are not well-written. They are noise-prone and containungrammatical text with much cryptic content. But in both domains, moreabbreviations than general text are used. As a consequence, we hypostatizethat text simplification may be of benefit in the domain of conversationaldata as well. It is much more challenging to extract correct relations fromcompound, long and syntactically ambiguous sentences. By breaking downsentences into shorter and simpler sentences, we empower relation extrac-tion tools to extract more relations and the extracted relations will be moreaccurate.1.5 Problem Statement and ContributionThe purpose of this study is to investigate the performance of open relationextraction tools on conversational data for the first time and suggest meth-41.5. Problem Statement and Contributionods to tackle the challenges faced in this domain. In particular, the effect oftext simplification before relation extraction will be evaluated in the domainof conversational texts. For this purpose, first a test dataset covering a widerange of conversational data and sentences has been populated from differ-ent corpora. The dataset creation approach has been described in details inchapter 3. Then the performance of Ollie, a state of the art for relation ex-traction, will be evaluated on the test dataset sampled from Emails, tweets,product reviews and blogs corpora before and after simplification based onthe number of extracted relations, the accuracy of extracted arguments andrelation phrases, and confidence score of extracted relations. We refer tothe later system (OLLIE using text simplification as a preprocessing step)as OLLIE-Simplified. For simplification TriS has been utilized to syntacti-cally simplify sentences before relation extraction [4]. There are other OpenIE tools such as TreeKernel, SONEX and EXEMPLAR with better reportedaccuracy than OLIIE. We were not able to use them for our datasets due tothe fact that they extract relations between named entities and were ableto extract less than five relations from each dataset while OLIIE extracthundreds of relations between noun phrases. In addition, TreeKernel limitsthe domain of its usage because of its supervised approach.We show text simplification is of great benefit in empowering relationextraction in the domain of conversational data. Experimental results showthat after text simplification by TriS, OLLIE-Simplified outperforms OL-LIE in terms of accuracy and informativeness of confidence score. It assignshigher confidence scores to correct relations and in most cases lower confi-dence scores to incorrect relations. Experimental results also suggest thata new system which utilizes the union of extracted relations of two systemswill outperform both systems, OLLIE and OLLIE-Simplified, in terms ofrecall since each system can find distinct relations not found by the otherone.In summary the three main contributions are as follows:? Collecting and sampling a dataset covering different conversation modal-ities.? For the first time evaluating the performance of Open IE in the areaof conversational texts over the created dataset.? Evaluating the performance of Open IE on conversations after textsimplification.51.6. Outline1.6 OutlineThe outline of the thesis is as follows: In the next chapter, available conver-sational datasets, relation extraction and text simplification approaches arereviewed. Chapter 3 describes our methodology for creating a test datasetand Open IE. In chapter 4, experiments are described and two systems arecompared. At the end in chapter 5, conclusion and future works are pre-sented.6Chapter 2Background and RelatedWorkIn this chapter, first available conversational datasets have been reviewed.Then relation extraction approaches (traditional and open) have been brieflyreviewed. Then text simplification which has been used as a preprocessingstep has been described at the end.2.1 Conversational DatasetsThere are different conversation modalities or domains including chats, emails,meetings and blogs which are distinguishable by different characteristics theyhave. Conversations can be categorized into two groups of synchronous andasynchronous. In synchronous conversations such as meetings and chats,turns happen with minimal gap and overlap between them. In asynchronousconversations such as fora discussions, emails, microblogs and blogs, differ-ent people can participate at different times or even same time making amore complicated conversational structure [9].The length of turns in different conversation modalities vary. Whilethere is no limit on the length of turns in synchronous conversations, lengthof other modalities are usually limited. For example, in twitter, each tweetmust be 140 character long and as a consequence tweets are much morecryptic and concise with more abbreviations than other modalities.Due to extensive research on conversational data for summarization andopinion mining, there are several publicly available datasets for most modal-ities but there is no dataset covering all different types of conversation do-mains. Available corpora are as follows:? Meetings: AMI and ICSI corpora. AMI corpus consists of 100 hoursof scenario and non-scenario meetings [10]. ICSI corpus consists of 75non-scenario or natural technical meetings held by ICSI researchers[30].72.2. Relation Extraction? Chats: Tux4kidss chat logs. Tux4Kids develops free software for ed-ucational purposes. The target users are kids. The dataset consistsof four chat threads in plain text format. In these chat sessions free-software and educational topics as well as Tux4Kids business are dis-cussed.2? Social networks and microblogs: Because of privacy concerns, mi-croblogs such as tweets and people?s posts on other social networkssuch as Facebook are not publicly available or there is a limit on thenumber of posts which can be downloaded using their API.? Emails: W3C, BC3, and Enron corpora. BC3 Email dataset wasoriginally developed for summarization and contains 40 email threadsand 261 Emails from W3C corpus [59]. Enron corpus contains naturalemails written by 150 employees in Enron corporation [37].? Reviews: Among datasets for reviews on products and services isOpinosis Dataset which was originally developed for summarizationand contains reviews on 51 topics. Others are customer reviews on 5products [36], amazon product reviews [31], and movie review datasetreleased by Pang and Lee3.? Blogs: There are several blog datasets including Spinn3r blog dataset4and Splog Blog Dataset5.2.2 Relation Extraction2.2.1 IntroductionRelation extraction is the task of finding semantic relationships between aset of entities in text. Relation extraction approaches can be divided intotwo categories: Traditional IE and Open IE. In Traditional IE, the relationof interest has to be specified in advance while in Open IE, various relationscan be extracted without requiring any prior knowledge. In the next twosections both approaches have been described.2 Relation Extraction2.2.2 Traditional IEIn Traditional IE systems the relation of interest has to be specified in ad-vance. To extract a specific relation, some of these systems use hand-codedextraction patterns or semi-supervised (bootsrapping) machine learning ap-proaches to learn extraction patterns using a few seed instances of the re-lation. Among those systems are DIPRE [7], Snowball [2], KnowItAll [22],Espresso [47], Leila [54], SRES [50], Luchs [27]. Supervised relation-specificapproaches look at relation extraction as a binary classification task to iden-tify whether there is a relation between two entities or not e.g. [26, 40, 66].One has to provide those systems with new extraction patterns and train-ing examples for new relations. They also require one pass over corpus foreach relation and hence, these approaches are not scalable with the size andvariety of the web corpus [5].DIRPE [7] finds all occurrences of seed pairs, which represent the ar-guments of a given relation, and construct a 6-tuple for each occurrence inthe corpus. Then it induces new patterns of the relation by grouping thosetuples based on the order and context between the arguments. In the nextiteration, the new patterns are used to find new seed pairs and hence newpatterns for the relation. This procedure will continue till some criteria aremet.Snowball [2] is an improvement over DIRPE with the difference thateach pattern is represented by a 5-tuple including context before, between,and after a pair of named entity tags. In a single pass they cluster the foundtuples and create a controid pattern for each cluster to be used in the nextiteration for finding new instances of the relation. After each iteration, theyevaluate each pattern by its precision in extracting relation instances. Inthe same way, each new instance in the next iteration is evaluated basedon the pattern used to extract the relation and its similarity to the patternaccording to the similarity function. Unlike DIRPE, Snowball evaluates andfilters new patterns which helps it prevent noise propagation. It has a moreflexible pattern matching approach compared to DIPRE but suffers fromtoo many specific patterns it generates because of the way they represent apattern.The main goal of KnowItAll [22] is to extract entities (unary predicates).It uses generic patterns to learn domain-specific extraction rules. To extractentities, It accepts a set of entity classes like city and outputs entity instancesextracted from web. It uses the extraction frequency as a mean to evaluatethe likelihood of the extraction.Mcdonald et al. [40] take a supervised approach and first build a feature92.2. Relation Extractionvector for each two named entities based on shallow syntactic structure.They find binary relations by classifying the feature vectors as related ornot. Then they solve the problem of complex relations (n-ary relations) byconstructing a weighted graph in which nodes are entities and the weight ofedges show the confidence of having a binary relation between those entities.Then they convert any maximal clique in the graph with geometrical meangreater than 0.5 into a n-ary relation.2.2.3 Open IEOpen IE systems address problems we face when using Traditional IE sys-tems by extracting relations from arbitrary sentences without requiringdomain-specific knowledge and target relations in advance. Open IE sys-tems extract web-scale information in the format of relational tuples (arg1;rel; arg2 ) in one pass or constant number of passes over the corpus. Therelation phrase rel expresses a relation between two arguments arg1 andarg2.Common relation extraction subtasks in Open IE are as follows:1. Preprocessing steps for sentence level analysis including chunking andparsing [1]2. Identifying arguments: usually noun phrases [62] or named entities inthe sentence are considered as potential candidates [19, 42].3. Identifying relation phrase (predicate): through learnt rules, hand-coded extraction patterns [22], machine learning, or hybrid [1].4. Postprocessing and integrating information: including argument andrelation resolution [64], co-reference resolution, deduplication, disam-biguation [1, 55].5. Identifying confidence of the extracted relations: pointwise mutualinformation (PMI) [22], noisy-or model, urns or contextual similarity[21].Open IE systems differ in the order of step 2 and 3. ReVerb [24] and itsextension R2A2 [23], opposed to other systems, first extracts relation phraseand then its arguments.Most Open IE tools such as TreeKernel, SONEX and EXEMPLAR dontperform the last step which is computing a confidence score for the extractedrelation.102.2. Relation ExtractionTextRunner, the first Open IE system, starts from candidate arguments,each pair of base NPs satisfying a set of constraints, and uses a binary clas-sifier to decide whether a relation between them should be extracted. OpenIE systems aim to extract all different types of relations. Since nobody candetermine how many different relations exist in the world, it is not possi-ble to provide these systems with training examples covering all the cases.As a consequence, some of these systems take a self-supervised approach6to generate their training data heuristically [5, 62]. To train its classifier,TextRunner uses a self supervised learner which generates positive and neg-ative examples by using a set of heuristic constraints. It labels extractionsnegative if they violate any of the constraints, otherwise positive. Thenthey train a CRF to extract relation phrases. TextRunner merges normal-ized relations and counts number of occurrences in order to later assign aconfidence score to each [6].WOE [62] also starts by first identifying arguments, NPs in the sentencesatisfying a set of constraints. Then it tries to extract a relation betweenthem. It heuristically produces training data for its extractor by matchinginfoboxes (attribute-value pairs) and sentences in Wikipedia articles. Theycompare two different extractors: 1) WOE-POS like TextRunner uses trivialfeatures like POS tags and a trained CRF to extract relation phrases. 2)WOE-Parse outputs a relation based on the shortest dependency path be-tween two NPs [62]. Even though infoboxes are incomplete and error prone,WOE outperformed TextRunner with much higher F-measure but the run-time of WOE-Parse was 30 times more than TextRunner which may not besuitable for Open IE.The disadvantage of approaches which first identify arguments in therelation is that they are prone to mistakenly consider a noun as an argumentwhile it is part of the relation phrase. It is usually the case for multi wordrelation phrases such as ?make a deal with?, ?has a PhD?, ?is a city in?etc. [24]. Inspired by this, ReVerb [24] starts from verbs in the sentence.The longest word sequences starting with these verbs which satisfy bothsyntactic and lexical constraints will be outputted as relation phrases. Thesyntactic constraints are a set of regular expressions based on POS tagsand the lexical constraint simply counts the number of distinct argumentsthat the extracted relation takes in the corpus of 500M Web sentences.Arguments are nearest NPs in the left and right hand side of the relationphrase satisfying a set of conditions. Later they analyzed 250 random webpages and noticed that only 65% of Arg1s and 60% of Arg2s are simple6Self supervised learning algorithms heuristically label their own training data.112.2. Relation ExtractionNPs and there are a handful of other categories covering 90% of other cases.Inspired by this observation, they trained three classifiers to determine theleft and right bound of Arg1 and the right bound of Arg2. They used aset of flat features like length of sentence, the context around the argumentand features inspired by their analysis denoting other categories than simpleNPs [23].There are other approaches that extract relations between named entitiesas opposed to noun phrases and hence are less prone to mistakenly considera noun as an argument while it is part of the relation phrase. Named entityextraction is the process of categorizing entities into predefined classes suchas persons and organizations. Despite decades of research in this area, itis still far way from complete [49, 65]. There are some drawbacks in usingnamed entities as arguments of relations. First, no named entity extractionsystem performs well in all domains and lots of effort is needed to get themwork on other domains than the one they designed for [65]. Second, thenumber of named entities extracted by these systems has been restrictedby the number of categories and subcategories of named entities defined.Usually there is a need to extend the range of named entity categories fora new domain [65]. Among relation extraction tools falling in this categoryare, SONEX, TreeKernel, and EXEMPLAR.SONEX [42] groups sentences having the same pair of named entitiesand presents each such group with a vector of shallow features includingunigrams, bigrams and part of speech patterns of the context between thepair of entities. Then SONEX clusters these feature vectors and assigns alabel based on these features to each cluster which represents the relationphrase between the two named entities. They evaluate the extracted rela-tions by their system by that of Freebase7. Working on blogosphere, theirultimate goal is to build a scalable system that outputs a social networkof the entities by considering the named entities as nodes and the relationlabel as edges of the network.One of the main problems in Open IE is the variety of relation formswhich makes it very difficult to be effictively learnt through machine learn-ing approaches or be captured through regular expressions and rule basedsystems. In addition, there are implicit relations such as ?located in? be-tween Nishapur and Iran in ?Omar Khayyam was born in Nishapur, Iran?which make extractions much more challenging. To mitigate this problem,TreeKernel [63] breaks down the relation extraction task into two subtasks.In the first subtask, they extract entities and feed a SVM model with the de-7http://www.freebase.com122.2. Relation Extractionpendency path between them to decide whether there is a relation betweenthe entities. As a consequence implicit relations and all relation forms willbe considered. For the second subtask, they employ regular expressionspatterns based on those of ReVerb to extract several candidates for relationphrase and then utilize another SVM dependency kernel to decide whetherthese candidates are correct. Even though the first SVM model does notput any constraint on the relation form, the input of second one is restrictedto nominal and verbal candidates. Though their approach outperformedOLLIE and ReVerb for both subtasks, lack of training examples in otherdomains makes their method less practical.[15] showed how semantic role labelers can be used for open relationextraction. They convert output of these systems to equivalent relationaltuples of TextRunner. According to them, TextRunner is much faster dueto shallow analysis and more practical in the case of limited time but thesemantic role labelers outperformed TextRunner given unlimited time. Theyalso propose a system which makes use of the union of output of thesesystems and is the best given intermediate amount of time.Mesquita et al. [19] classify relation extraction approaches into threecategories based on the depth of analysis they do: The first category en-compasses shallow approaches such as that of TextRunner, ReVerb, andSONEX which extract relations based on POS tags of the sentences. Thesecond category takes advantage of dependency parse tree of the sentence;OLLIE and TreeKernel fall in this category. In their classification seman-tic role labelers such as Lund [32] and SwiRL [56] are another category ofrelation extraction approaches which do a more sophisticated analysis thandependency parsing. They discuss how increasing the complexity of analy-sis increases the computational cost but does not essentially lead to a highincrease in the accuracy. As a consequence, they proposed EXEMPLAR, arule based system, which utilizes the idea behind the success of semantic rolelabelers, considering the connection between relation words and arguments.By using dependency parse of the sentence instead of semantic role, theykeep computational cost the same as the second category. They show theirapproach is superior to other methods when extracting relations betweennamed-entities not noun phrases.There are other approaches with different frameworks. One of them is anunsupervised method to semantically parse or represent meaning of a sen-tence, which subsumes relation extraction, proposed by Poon and Domingos[48]. In their setting, the semantic parse of the sentence is the set of frag-ments obtained by partitioning its syntactic dependency tree and later as-signing each fragment to a cluster of semantically identical structures. Each132.2. Relation Extractioncluster contains structures syntactically or lexically different but having thesame meaning which resolves the problem of argument resolution and rela-tion resolution. Their goal is to find this set of clusters which corresponds totarget predicates (relations) and objects (arguments). USP takes advantageof Markov Logic Networks to model the joint distribution for dependencytree and its latent meaning representation (MR). It tries to maximize theprobability of the observing dependency structures of the sentence by tuningthe weights of first-order clauses. In OntoUSP [58], authors modified thecluster mixture formula in USP to also include ISA relationships betweenclusters which leads to better generalization. For example in their setting,there is ISA relationship between inhibit and regulate.Another interesting approach for relation extraction is the approachtaken in SOFIE [55]. SOFIE first converts everything (ontology, text, con-straints, and new fact hypotheses) into logical statements. Then they uselogical rules to determine which hypotheses are probably true. They manu-ally developed a set of general rules, conceptually similar to DIRPE andSnowball, but relation-specific rules can be added later. They use theweighted MAX-SAT setting to find out set of hypotheses that should betrue in order to have the maximum number of rules satisfied. Their empha-sis is more on ontology population and reasoning.The problem with systems such as ReVerb, SwiRL and WOE is thatthere are only capable of extracting verb based relations. Verb based re-lations are those relations which begin with a verb. ReVerb further limitsthese relations to be between arguments and satisfy syntactic constraints.Even though WOE can find relations not between arguments, it fails to findthose relations which contains nouns such as is CEO of. There are othertypes of relations beginning with other syntactic types e.g. author of andsuch as which they are not capable of extracting. Another weakness of thesesystems is that they ignore context leading to extraction of relations whichare conditionally or supposedly true [38]. For example in the sentence:If John had a million dollars, he would buy a house.The relation (John; buy; a house) is only correct when he has a milliondollars.OLLIE tries to address the weaknesses of previous approaches by takingadvantage of high confidence extractions of ReVerb as input seeds of itsbootsrapper. Unlike other proposed bootsrapped methods for which seedsare a pair of arguments and bootstrapper considers those sentences thatmatch both arguments, bootsrapper in OLLIE not only matches argumentsbut also relation words. It empowers bootsrapper to learn general open142.3. Text simplificationextraction patterns based on dependency parse tree of retrieved sentencesthat can be utilized to extract other relations.Bootsrapper of WOE-Parse also takes the same approach to learn extrac-tion patterns. The thing that makes a big difference in their performance isthe quality of their seeds. WOE-parse retrieves those sentences in Wikipediaarticle that matches infobox values (candidate arguments) and heuristicallyconsiders all the words between arguments as a relation phrase which doesnot hold true in many cases causing noisy seeds. OLLIE also introduces acontext analysis component that utilizes the dependency parse tree and twosimple rules to find relations which are conditionally or supposedly true andadds a filed indicating the conditional truth or attribution. If there is clausalcompliment (ccomp) edge in the tree, they see whether the verb of the treeexists in a list of communication and cognition verbs. If so, they add theattribution field to the extracted relation. In the same manner, they add acausal modifier field, if there is adverbial clause (advcl) edge in the tree andthe first word of clause exists in a list of 16 terms: if, when, although,. Sincethese two rules dont cover all the possible conditional or hypothetical truerelations, they train a classifier to decrease the confidence of the relation inother cases (Mausam et al., 2011).2.3 Text simplificationIn this section, first text simplification (syntactic and lexical) approacheshave been briefly reviewed. TriS, which has been used in the experiments,has been described in more details at the end.2.3.1 IntroductionThe same meaning can be expressed in many different ways with differentlevel of complexity to understand. The source of this complexity can besyntactic or lexical. Syntactic complexity arises from complex, compoundand nested structures usually in long sentences and lexical complexity arisesfrom use of difficult and less frequent words or ambiguity in their meaning.Text simplification is the process of simplifying texts while preserving theirmeaning and information to increase understandability. Simplified sentencesare easier to understand and easier to process by computers [4].152.3. Text simplification2.3.2 Applications and ApproachesText simplification can be syntactic or lexical. To simplify texts lexicallydifficult words are substituted by easier words. For language learners andpeople with reading disability difficult words are less frequent words [11,12, 20]. Hence one of the solutions is to replace those words by the mostfrequent synonym in their set of synonyms (including the word itself) [11,20]. Since words can have different meanings, this approach often leads tomeaningless sentences. To tackle this problem in lexical simplification, wordsense disambiguation can be done [18]. For syntactic simplification, a set ofrules [14, 18, 52] or log-linear models [4] can be utilized to simplify sentencesby breaking down them into shorter and simpler sentences.Text simplification has been studied for two main purposes: making texteasier to understand for readers with aphasic disability [11] or low literacyskills [8] and as a preprocessing step for several NLP tasks such as rela-tion extraction [33, 34], semantic role labeling [61], machine translation [46],summarization [53, 60], and improving accuracy of parsers [13, 14, 52]. Pre-processing text to simplify it has been inspired by the fact that performanceof these systems rapidly deteriorates as the length and complexity of thesentence increases [13, 33]. Most of the errors in parsing are due to long,complex, and ambiguous sentences and it has been shown that text sim-plification eases summarization by shortening sentences and dropping nonessential information. Silveira and Branco [53] showed that removing somespecific structures such as relative clauses, explanatory phrases, and apposi-tions does not decrease the readability and informativeness of the sentenceand hence can be removed to simplify the sentences and output a bettersummery. Vanderwende et al. [60] proposed a better extractive summariza-tion system by adding simplified sentences to the input so as to give thesummarization system the option of choosing between simplified sentenceand original sentence.Unlike previous rule-based approaches for syntactic simplification whichis limited to English language, in [4] a general framework has been pro-posed, TriS, for syntactic simplification by casting the problem into a searchproblem in which among all possible simplified sentences of each sentence,they find a subset of it that gives the highest probability given the originalsentence e. In the other word, they find the subset S which maximizes thefollowing equation:p(S|e) =exp(?Mi=1 wifi(S, e))?S? exp(?Mi=1 wifi(S?, e))(2.1)162.3. Text simplificationIn this equation, feature functions, f(S,e), are based on 177 sentence leveland interactive features extracted from original and simplified sentences. Tolearn weight of the features, w, they use online learner MIRA [17]. To buildall the possible simplified sentences, they assume any simplified sentence hasthe following structure:Subject +Verb+ ObjectIn which Subject and Object are noun phrase (NP). They make a list of allNPs in the original sentences (plus an empty NP for intransitive verbs) anda list of verbs in the original sentence. Then they make a list of all possiblesimple sentences by enumerating all the possible ways of combining thesetwo lists. If there are n NPs and m verbs in the sentence this approach willyield n2m simple sentences. At the end, they make use of stack decodingalgorithms to find the best subset of this list of simplified sentences thatmaximize the equation 1 [4].Jonnalagadda and Gonzalez [33] showed that syntactically simplifyingsentences using a set of rules helps relation extraction in the domain ofbiomedical texts which usually have longer sentences with more abbrevia-tions and relative clauses than less specialized and less technical texts likenews. Being optimized to extract relations among proteins from biomed-ical scientific literature, it may not be very useful on other domains likeconversational data. As opposed to scientific literature in which sentencesare grammatically correct, sentences in conversational texts are not well-written. They are noise-prone and contain ungrammatical text with muchcryptic content. But in both domains, more abbreviations than general textare usually used. As a consequence, we hypostatize that text simplificationmay be of benefit in the domain of conversational data as well. In chapter4, we test this hypothesis through experiments and present results. In thisstudy, TriS, has been used to simplify texts before feeding them into OLIIE.17Chapter 3MethodologyTo fairly evaluate Open IE over conversational data a test dataset coveringdifferent types of conversations and sentences is required. To the best ofour knowledge there is no such dataset and hence we propose a method tocreate a dataset that has been sampled from a wide range of conversationalcorpora [9] including synchronous conversations (AMI and ICSI corpus),microblogs (tweets), threaded or asynchronous conversations (Email andblog threads), and reviews on products and services (Opinosis Dataset). Inthe next two sections, first the conversational corpora used for sampling hasbeen described and then the method proposed for sampling sentences fromthese corpora has been described.3.1 Dataset CreationThe test dataset used in this study includes a total of 600 sentences whichwere sampled from 6 conversational corpora (100 sentences from each). Thesampling approach has been described in the next section.The corpora cover a wide range of conversational data [9] includingsynchronous conversations (AMI and ICSI corpus), microblogs (tweets),threaded or asynchronous conversations (Email and blog threads), and re-views on products and services (Opinosis Dataset). Totally 6 corpora wereused which have been described in the following sections:3.1.1 ReviewsWriting review is a common way of expressing ideas and opinions about newproducts and services. They are usually informal and have colloquial lan-guage. Opinosis Dataset 1.0 was originally developed for summarization andcontains reviews on 51 topics such as ?battery life ipod nano 8gb? and ?navi-gation amazon kindle?. These reviews are about hotels, cars, and electronicsand were collected from Tripadvisor, and [25].This dataset was used as a representative of this conversational modality inour test dataset.183.1. Dataset Creation3.1.2 EmailsWriting and reading Emails has been the most popular conversational ac-tivity and hence one Email corpus was included in the evaluation of OpenIE tools. BC3 Email dataset was originally developed for summarization aswell and contains 40 email threads and 261 Emails from W3C corpus [59].3.1.3 MeetingsAnother important conversational modality is meeting. Many people spenda lot of time in meetings and due to advancement in transcribing, thesespoken conversations now are available as conversational texts. We haveused two different meeting corpora: AMI corpus consists of 100 hours ofscenario and non-scenario meetings. We used the scenario portion of thiscorpus in which four persons participate in the meetings and talk aboutdesigning a remote control [10]; ICSI corpus consists of 75 non-scenario ornatural technical meetings held by ICSI researchers [30]. Both corpora havenative and none native English speakers but ICSI meetings has on averagesix to ten participants per meeting which is more than that of AMI scenario.3.1.4 Blogs and Online DiscussionsBlogs and forum discussions are another type of popular conversational textsin which people share their comments, thoughts and feelings about any topicposted by the first participant of the discussion which can be news, questions,events and so on. Slashdot is a website for news stories about technologyalong with lengthy discussions and comments of users. The dataset we usedconsists of all the threaded discussions of the users for 10 dates.3.1.5 Social NetworksNowadays people spend a great amount of time on updating their profile,reading their friends? posts, and commenting on them in social networkssuch as Twitter and Facebook. The language in social networks is informaland ungrammatical with lots of abbreviations. The dataset used as a rep-resentative of this type of conversations is 5146 random tweets taken fromTwitter.3.1.6 Dataset CharacteristicsCharacteristics of datasets has been shown in Table 3.1. As this table shows,On average Slashdot has longest senences while AMI has the shortest sen-193.1. Dataset CreationName #doc #sent#sentper doc#word#wordper doc#wordper sentICSI 494 80410 162 839874 1700 10Slashdot 15 8128 541 211180 14078 26AMI 137 76865 556 716382 5191 9Opinosis 51 6851 134 128150 2512 18Twitter 5146 3254 813 90802 22700 10BC3 40 2395 59 29642 741 12Table 3.1: Dataset characteristics.tences. For tokenizing words, the word tokenizer of NLTK toolkit was used.A stop list of special tokens were used to filter emoticons and tokens suchas ?LOL?, ?lolll?, ?ah?, and ?!!!?. For Twitter dataset, urls and ReTweet(RT) expressions were removed from the tweets which increased the accuracyof extractions of OLIIE about 7% and that of OLIIE-Simplified about 20%.Because of conversational nature of these datasets word tokenizer made moreerrors and its accuracy were lower than other domains. In Table 3.1, thefirst column shows number of documents per dataset and the rest of columnsshow total number of sentences, average number of sentences per document,total number of words, average number of words per document, and averagenumber of words per sentence in order. For twitter dataset each tweet, forBC3 corpus each thread, and for Opinosis each topic has been considered asone document.3.1.7 Sampling MethodIn order to evaluate the performance of a relation extraction tool, the testdataset ideally should contain different types of sentences having differenttypes of relations. As a consequence, to obtain a representative sample ofsentences, we used two stage stratified sampling. In the first stage, to cap-ture key characteristics of each corpus, each corpus plays the role of onestratum independently. In the second stage, 100 sentences were sampledfrom each corpus (stratum). In the second stage, we did not use a simplestratified sampling. Instead, we extracted a set of syntactic and conversa-tional features from each sentence and then we grouped them based on theresulting feature vectors. The stratified sampling has been done based onthe probability of resulting groups. More members in the group, higher theprobability to be chosen for sampling. The feature set used has been shownin Table Open IE on Conversational DataThe syntactic features are inspired by the fact that more punctuationsand relative nouns in a sentence make it more challenging for relation ex-traction tools to extract relation from. Conversational features are thosefeatures which were found to be useful in the domain of conversational data.We chose a subset of features proposed by Murray and Carenini [44] thatappeared to be useful in the relation extraction task. SMT shows the sumof Tprob scores. Tprob itself ishows the probability of each turn given theword. CLOC represents the sentence position in the conversation. Sinceit is often more difficult to extract relation from longer sentences, we uti-lized two features that employ the length of sentence: SLEN and SLEN2represent the number of words normalized by the longest sentence in theconversation and turn respectively. CWS shows conversation cohesion andis computed after removing stopwords. It represents the number of wordsappearing in other turns except the current turn. CENT1 shows the sim-ilarity of the sentence to the conversation and is computed based on thecosine value between the sentence and the rest of the conversation.SyntacticfeaturesQuestionA binary feature indicating whetherthe sentence is a questionWH countNumber of relative pronouns in thesentencePunc count Number of punctuations in the sentenceConversationalfeaturesSMT Sum of Tprob scoresCLOC Position in conversationSLEN Globally normalized word countSLEN2 Locally normalized word countCWS Rough ClueWordScoreCENT1 Cosine of sentence and conv., w/ SprobTable 3.2: Feature set used in sampling sentences.3.2 Open IE on Conversational DataWe were not able to use TreeKernel because of lack of training examples inour domain. SONEX and EXEMPLAR were not used since they extractrelations between named entities and were able to extract only few relationswhile OLIIE extract hundreds of relations between noun phrases.The relations extracted by OLIIE from the created dataset manuallyevaluated following a set of rules and they were labeled as correct if they were213.3. Text Simplification for Open IEadherent to those rules and deemed to be correct. OLIIE performance wereevaluated before and after simplification based on the number of extractedrelations, the accuracy of extracted arguments and relation phrases, and theinformativeness of confidence score.3.3 Text Simplification for Open IEFor the task of information extraction, only syntactic simplification will beuseful since the purpose of lexical simplification is to improve readabilityfor human not computer. Hence, in this study only the effect of syntacticsimplification will be evaluated for the task of relation extraction. For textsimplification, TriS which is a syntactic simplifier has been used to syntac-tically simplify texts before feeding them into OLLIE. Experimental resultsare presented and discussed in the next chapter.22Chapter 4Experimental Results4.1 Evaluation MetricsEvaluating relation extraction approaches is difficult due to the subjectivityand ambiguity of the task. It is not only difficult for automatic systems butalso for human to decide whether there is a relation in the sentence and ifso, which words of the sentences form the relation [19, 63]. Another type ofambiguity arises from the definition of an entity. For example in the sentence?John went to Starbucks coffee shop?, one may say the second argument ofthe relation ?went? can be ?Starbucks coffee shop?, ?Starbucks?, or both,while someone else considers only ?Starbucks coffee shop? as the correctargument.Evaluating and comparing recall of Open IE tools becomes even morechallenging than accuracy with the presence of implicit relations and re-lations for which the relation phrase has not been stated in the sentence.For example, in the sentence ?Rumi, a poet, was born in Nishapur, Iran?one may consider the relation ?located in? between Nishapur and Iran eventhough the relation phrase has not been appeared in the sentence. In thesentence ?John broke the residence rules? one may conclude the implicitrelation lives in between ?John? and ?the residence? as well. As a conse-quence, Open IE tools which extract relations between noun phrases don?treport the recall of their system. Instead, they use accuracy and numberof exactions as a way of comparison. As a consequence, the performance ofOLLIE were evaluated before and after simplification based on the followingmetrics:? The number of extracted relations? The accuracy of extracted arguments and relation phrases? Informativeness of confidence score234.2. Results4.2 ResultsTwo experiments have been performed. In the first experiment, we fedthe created dataset into OLLIE and report the accuracy of arguments andrelation phrases extracted. In the second experiment, these sentences firstwere simplified by TriS and then they have been fed into OLLIE. The secondsystem, OLLIE using text simplification as a preprocessing step, will bereferred as OLIIE-Simplified. The result of these experiments has beenshown in Table 4.1 and Table 4.2. In all tables, the bold numbres show thetimes OLLIE outperformed OLIIE-Simplified. The columns, from left toright, show number of extractions, accuracy of the first argument, accuracyof the first argument when relation phrase is correct, accuracy of the relationphrase, accuracy of the second argument, accuracy of the second argumentwhen relation phrase is correct, and accuracy when both arguments andrelation phrase are correct.TriS failed to simplify most of the sentences in AMI and ICICS corpusdue to lack of punctuations or wrong punctuations which has made sen-tences too long to be simplified by TriS. As a consequence, accuracy andaverage confidence score for them has not been reported in the second ex-periment. Whenever TriS did not have any suggestion for simplifying thesentence the original sentence were fed into OLIIE. As the Table 4.2 showstext simplification considerably improves accuracy of extractions for botharguments and relation in all cases except Slashdot dataset. TriS were notable to simplify sentences in Slashdot dataset correctly mostly because oferrors in sentence tokenization.Tables 4.3 and 4.4 show average confidence scores OLIIE and OLIIE-Simplified assigned to the extractions. From left to right, the columns showaverage confidence score of all extractions, average confidence score of cor-rect relations, average confidence score of the incorrect relations, averageconfidence score when both the first argument and relation phrase is correct,average confidence score when the first argument is incorrect and relation iscorrect, average confidence score when both the second argument and rela-tion phrase is correct, average confidence score when the second argumentis incorrect and relation is correct, and average confidence score when botharguments and relation phrase are correct.Figures 4.1 and 4.2 compares their accuracy and confidence scores whenboth arguments and relation phrase are correct. Figure 4.3 compares theirconfidence scores when relation phrase in incorrect. As this figure shows,OLLIE-Simplified assigned lower confidence scores to incorrect extractionson average.244.2.ResultsDataset #ExtractionsArg1acc.Arg1 acc.when relationis correctRelationphraseacc.Arg2Arg2 acc.when relationis correctAll correctacc.ICSI 292 73.6% 56.8% 47.9% 66.8% 57.9% 45.2%AMI 650 80.0% 61.2% 71.5% 66.3% 52.0% 43.2%BC3 148 79.0% 61.5% 73.0% 69.6% 32.4% 48.6%Slashdot 301 79.5% 65.4% 76.4% 74.3% 63.7% 54.1%Reviews 372 65.6% 51.3% 64.5% 61.8% 53.5% 40.9%Twitter 90 66.7% 55.6% 62.2% 70.0% 52.2% 45.6%Table 4.1: Accuracy before simplification. The bold numbres show the cases OLLIE outperformed OLIIE-Simplified.Dataset #ExtractionsArg1acc.Arg1 acc.when relationis correctRelationphraseacc.Arg2acc.Arg2 acc.when relationis correctAll correctacc.BC3 141 74.5% 66.0% 77.3% 71.6% 68.1% 58.2%Slashdot 211 77.3% 63.6% 76.8% 74.7% 63.6% 51.5%Reviews 233 65.2% 55.4% 68.2% 65.7% 54.9% 44.2%Twitter 99 73.7% 63.6% 72.7% 79.8% 64.6% 55.6%Table 4.2: Accuracy after simplification. The bold numbres show the cases OLLIE-Simplified outperformedOLIIE.254.2.ResultsDatasetAllextCorr.relIncorr.relCorr. Arg1and relIncorr.Arg1 andcorr. relCorr. Arg2and relIncorr.Arg2 andcorrect relAllcorr.ICSI 0.6 0.43 0.15 0.36 0.07 0.36 0.07 0.29AMI 0.56 0.4 0.15 0.35 0.05 0.3 0.09 0.26BC3 0.61 0.46 0.14 0.39 0.06 0.37 0.09 0.31Slashdot 0.66 0.49 0.14 0.42 0.07 0.41 0.08 0.35Reviews 0.64 0.42 0.21 0.34 0.08 0.34 0.08 0.27Twitter 0.7 0.44 0.26 0.4 0.04 0.38 0.06 0.34Table 4.3: Average confidence score before simplification. The bold numbres show the cases OLLIE outperformedOLIIE-Simplified.DatasetAllextCorr.relIncorr.relCorr. Arg1and relIncorr.Arg1 andcorr. relCorr. Arg2and relIncorr.Arg2 andcorrect relAllcorr.BC3 0.69 0.54 0.11 0.46 0.08 0.48 0.06 0.4Slashdot 0.7 0.49 0.12 0.4 0.09 0.41 0.08 0.33Reviews 0.66 0.47 0.17 0.39 0.08 0.38 0.09 0.31Twitter 0.74 0.54 0.19 0.47 0.07 0.48 0.05 0.42Table 4.4: Average confidence score after simplification. The bold numbres show the cases OLLIE-Simplifiedoutperformed OLIIE.264.3. Analysis and Discussion4.3 Analysis and DiscussionIf we analyze the accuracies of relation phrase and arguments when rela-tion phrase is correct, we see that OLLIE has the best performance onSlashdot and BC3 corpora in order and the worst on Reviews corpus. AsTable 4.2 shows OLLIE-Simplified has also the worst performance on Re-views corpus but the best performance on BC3 corpus. Both systems havethe worst performance on Reviews corpus which might be due to language ofreviews. In reviews, people express their opinions and feelings using phrasesand incomplete sentences. Some examples of such sentences are as follows:?accurate GPS for not so much money?, ?Better battery life?, ?FAR bet-ter with wireless function on?, ?NO USER REPLACEABLE BATTERY?,?Easy to read, navigate, etc.?, ?Significant improvements to ergonomics andnavigation?. With the same logic, better performance on Slashdot and BC3corpora might be due to language in these corpora. BC3 corpus containsemails written in a corporation which usually have more formal and gram-matical sentences. Slashdot is a website for news stories about technologyalong with lengthy discussions and comments of users with probably moretechnical and grammatical content. Overall, the most difficult conversa-tional modality for relation extraction for both systems is reviews. Theeasiest ones for OLIIE are blogs and emails and for OLLIE-Simplified areemails and microblogs (Twitter).According to Table 4.1 and 4.2, both systems extract more relations fromReview corpus8. OLLIE extracts more relations than OLLIE-Simplified ex-cept for Twitter corpus for which OLLIE-Simplified extracts more. It mightbe due to the way OLLIE works. OLLIE tries the same relation phrasewith different pairs of arguments but only one of these extractions is cor-rect. Even though OLLIE-Simplified extracts fewer number of extractionsfor most corpora, the extractions are more distinct.As Table 4.3 and 4.4 show, OLLIE is less confident in AMI and ICSIextractions while OLLIE-Simplified is less confident in Reviews extractions.Considering all extractions regardless of being correct or not, both OLLIEand OLLIE-Simplified are most confident in Twitter extractions.As results show, OLLIE-Simplified assigned lower confidence scores toincorrect relation phrases in all cases and higher confidence scores to correctextractions for most datasets. As a consequence we conclude that textsimplification improves the informativeness of confidence scores.8As we were not able to run TriS on AMI and ICSI corpora we omit them here incomparison.274.3. Analysis and DiscussionFigure 4.1: Accuracy of extraction when the both arguments and relationphrase are correct. The largest increase in the accuracy can be seen for BC3and Twitter corpus, 13% and 10% in order.Figure 4.2: Average confidence score when the both arguments and relationphrase are correct. The largest increase in the confidence score can be seenfor BC3 and Twitter corpus in order.284.3. Analysis and DiscussionFigure 4.3: Average confidence score when the relation phrase is incorrect.The largest decrease in the confidence score of incorrect relation phrases hashappend for Twitter corpus.As figures 4.1 and 4.2 show the largest increase in accuracy and confi-dence score can be seen for BC3 and Twitter corpora in order. As opposed towhat we thought text simplification has been much more useful and advan-tageous for corpora with shorter sentences. As figure 4.3 shows the largestdecrease in confidence score of incorrect relation phrases has happend forTwitter corpus.Text simplification is very effective in increasing accuracy of OLLIE forTwitter dataset. Tweets have much more cryptic content and abbreviationsthan other conversations since they must be at most 140 characters long.This results again verifies the result found by Jonnalagadda and Gonza-lez [33] that text simplification greatly helps relation extraction when theamount of cryptic content and abbreviations in text is much more than lessspecialized and less technical texts like news.Text simplification is not very effective in increasing the accuracy ofOLLIE for Slashdot dataset due to errors in sentence tokenization and itslengthy sentences. Sentence tokenizer made more mistakes in this datasetthan other datasets which lead to poor performance of TriS in simplify-ing sentences in Slashdot. Slashdot has the longest sentences among othercorpora and as opposed to what we thought, TriS did not work well when294.3. Analysis and Discussionsentences were too long. It might be due to the way TriS simplifies sentencesin which it builds its search space based on all noun phrases in the sentences.Each system finds distinct relations not found by the other system.Hence a new system which utilizes the union of extracted relations of thetwo systems will outperform both systems in terms of recall.Another interesting finding is that OLLIE is more capable in accuratelyfinding the first argument of the relation while OLLIE-Simplified more ac-curately extracts the relation phrase and the second argument.30Chapter 5Conclusion and Future WorkTo evaluate Open IE in the domain of conversational texts, a method wasproposed to create a test dataset covering a wide range of conversationaldata.Conversational text poses new challenges due to its specific character-istics including cryptic content, lots of abbreviations, ungrammatical andinformal language. As a consequence text simplification was used to miti-gate the problems.We discussed why text simplification will be useful for this task andshould be used as a preprocessing step in relation extraction. The ap-proach taken to sample from conversational datasets and experiments weredescribed and two systems were compared.To the best of our knowledge, this is the first time Open IE has beenevaluated in the domain of conversational data. We proposed a methodto sample a test dataset covering a wide range of conversational data. Weshowed text simplification empowers relation extraction in the domain ofconversational texts. Experimental results show that OLLIE-Simplified out-performs OLLIE in terms of accuracy and informativeness of the confidencescore.As opposed to what we hypothesized text simplification has been muchmore useful and advantageous for corpora with shorter average sentences.Text simplification has been much more effective in increasing the accuracyof OLLIE for Twitter dataset which has much more cryptic content andabbreviations than other conversations due to its length limit (at most 140characters).Overall, the most difficult conversational modality for relation extrac-tion for both systems is reviews. The easiest ones for OLIIE are blogs andemails and for OLLIE-Simplified are emails and microblogs (Twitter). Inreviews, people express their opinions and feelings using phrases and incom-plete sentences leading to difficulty in relation extraction. Emails writtenin a corporation usually have more formal and grammatical sentences andtechnical blogs like Slashdot have more technical and grammatical sentenceswhich helps relation extraction.31Chapter 5. Conclusion and Future WorkEach system finds distinct relations not found by the other system.OLLIE-Simplified can find new relations not already found by OLLIE andhence a new system which utilizes the union of extracted relations of twosystems will outperform both systems in terms of recall. OLLIE is morecapable in accurately finding the first argument of the relation while OLLIE-Simplified more accurately extracts the relation phrase and the second ar-gument. A unified system that takes advantage of these findings, wouldoutperform both systems.Since conversational data has special characteristics, a text simplifier de-veloped to deal with conversational data would be of more benefit. Anotherdirection will be to evaluate the effect of text simplification on other OpenIE systems and in other domains.32Bibliography[1] Eugene Agichtein. Scaling information extraction to large documentcollections. IEEE Data Eng. Bull, 28:3?10, 2005.[2] Eugene Agichtein and Luis Gravano. Snowball: Extracting relationsfrom large plain-text collections. In Proceedings of the Fifth ACM Con-ference on Digital Libraries, DL ?00, pages 85?94, New York, NY, USA,2000. ACM.[3] Kemafor Anyanwu, Angela Maduko, and Amit Sheth. Semrank: Rank-ing complex relationship search results on the semantic web. In Proceed-ings of the 14th International Conference on World Wide Web, WWW?05, pages 117?127, New York, NY, USA, 2005. ACM.[4] Nguyen Bach, Qin Gao, Stephan Vogel, and Alex Waibel. Tris: Astatistical sentence simplifier with log-linear models and margin-baseddiscriminative training. In IJCNLP, pages 474?482, 2011.[5] Michele Banko, Michael J. Cafarella, Stephen Soderland, Matt Broad-head, and Oren Etzioni. Open information extraction from the web. InProceedings of the 20th International Joint Conference on Artifical In-telligence, IJCAI?07, pages 2670?2676, San Francisco, CA, USA, 2007.Morgan Kaufmann Publishers Inc.[6] Michele Banko and Oren Etzioni. The tradeoffs between open andtraditional relation extraction.[7] Sergey Brin, Rajeev Motwani, Lawrence Page, and Terry Winograd.What can you do with a web in your pocket? IEEE Data Eng. Bull.,21(2):37?47, 1998.[8] Arnaldo Candido, Jr., Erick Maziero, Caroline Gasperin, Thiago A. S.Pardo, Lucia Specia, and Sandra M. Aluisio. Supporting the adap-tation of texts for poor literacy readers: A text simplification editorfor brazilian portuguese. In Proceedings of the Fourth Workshop on33BibliographyInnovative Use of NLP for Building Educational Applications, EdApp-sNLP ?09, pages 34?42, Stroudsburg, PA, USA, 2009. Association forComputational Linguistics.[9] Giuseppe Carenini, Gabriel Murray, and Raymond Ng. Methods forMining and Summarizing Text Conversations. Morgan & ClaypoolPublishers, 1st edition, 2011.[10] Jean Carletta, Simone Ashby, Sebastien Bourban, Mike Flynn, MaelGuillemot, Thomas Hain, Jaroslav Kadlec, Vasilis Karaiskos, WesselKraaij, Melissa Kronenthal, Guillaume Lathoud, Mike Lincoln, AgnesLisowska, Iain McCowan, Wilfried Post, Dennis Reidsma, and PierreWellner. The ami meeting corpus: A pre-announcement. In Proceedingsof the Second International Conference on Machine Learning for Mul-timodal Interaction, MLMI?05, pages 28?39, Berlin, Heidelberg, 2006.Springer-Verlag.[11] John Carroll, Guido Minnen, Yvonne Canning, Siobhan Devlin, andJohn Tait. Practical simplification of english newspaper text to as-sist aphasic readers. In In Proc. of AAAI-98 Workshop on IntegratingArtificial Intelligence and Assistive Technology, pages 7?10, 1998.[12] John Carroll, Guido Minnen, Darren Pearce, Yvonne Canning, SiobhanDevlin, and John Tait. Simplifying text for language-impaired readers.In In Proceedings of the 9th Conference of the European Chapter ofthe Association for Computational Linguistics (EACL, pages 269?270,1999.[13] R. Chandrasekar, Christine Doran, and B. Srinivas. Motivationsand methods for text simplification. In PROCEEDINGS OF THESIXTEENTH INTERNATIONAL CONFERENCE ON COMPUTA-TIONAL LINGUISTICS (COLING ?96), 1996.[14] R. Chandrasekar and B. Srinivas. Automatic induction of rules for textsimplification, 1997.[15] Janara Christensen, Mausam, Stephen Soderland, and Oren Etzioni.An analysis of open information extraction based on semantic role la-beling. In Proceedings of the Sixth International Conference on Knowl-edge Capture, K-CAP ?11, pages 113?120, New York, NY, USA, 2011.ACM.34Bibliography[16] Hong-Woo Chun, Yoshimasa Tsuruoka, Jin-Dong Kim, Rie Shiba,Naoki Nagata, Teruyoshi Hishiki, and Jun ichi Tsujii. Extraction ofgene-disease relations from medline using domain dictionaries and ma-chine learning. In Russ B. Altman, Tiffany Murray, Teri E. Klein,A. Keith Dunker, and Lawrence Hunter, editors, Pacific Symposium onBiocomputing, pages 4?15. World Scientific, 2006.[17] Koby Crammer and Yoram Singer. Ultraconservative online algorithmsfor multiclass problems. J. Mach. Learn. Res., 3:951?991, March 2003.[18] Jan De Belder and Marie-Francine Moens. Text simplification for chil-dren. In Prroceedings of the SIGIR Workshop on Accessible SearchSystems,, pages 19?26. ACM, 2010.[19] Filipe de S Mesquita, Jordan Schmidek, and Denilson Barbosa. Effec-tiveness and efficiency of open relation extraction. In EMNLP, pages447?457. ACL, 2013.[20] Siobhan Devlin and Gary Unthank. Helping aphasic people process on-line information. In Proceedings of the 8th International ACM SIGAC-CESS Conference on Computers and Accessibility, Assets ?06, pages225?226, New York, NY, USA, 2006. ACM.[21] Doug Downey, Stefan Schoenmackers, and Oren Etzioni. Sparse infor-mation extraction: Unsupervised language models to the rescue. In InProc. of ACL, pages 696?703, 2007.[22] Oren Etzioni, Michael Cafarella, Doug Downey, Ana-Maria Popescu,Tal Shaked, Stephen Soderland, Daniel S. Weld, and Alexander Yates.Unsupervised named-entity extraction from the web: An experimentalstudy. ARTIFICIAL INTELLIGENCE, 165:91?134, 2005.[23] Oren Etzioni, Anthony Fader, Janara Christensen, Stephen Soderland,and Mausam Mausam. Open information extraction: The second gener-ation. In Proceedings of the Twenty-Second International Joint Confer-ence on Artificial Intelligence - Volume Volume One, IJCAI?11, pages3?10. AAAI Press, 2011.[24] Anthony Fader, Stephen Soderland, and Oren Etzioni. Identifying rela-tions for open information extraction. In Proceedings of the Conferenceon Empirical Methods in Natural Language Processing, EMNLP ?11,pages 1535?1545, Stroudsburg, PA, USA, 2011. Association for Com-putational Linguistics.35Bibliography[25] Kavita Ganesan, ChengXiang Zhai, and Jiawei Han. Opinosis: A graph-based approach to abstractive summarization of highly redundant opin-ions. In Proceedings of the 23rd International Conference on Computa-tional Linguistics, COLING ?10, pages 340?348, Stroudsburg, PA, USA,2010. Association for Computational Linguistics.[26] Zhou GuoDong, Su Jian, Zhang Jie, and Zhang Min. Exploring variousknowledge in relation extraction. In Proceedings of the 43rd AnnualMeeting on Association for Computational Linguistics, ACL ?05, pages427?434, Stroudsburg, PA, USA, 2005. Association for ComputationalLinguistics.[27] Raphael Hoffmann, Congle Zhang, and Daniel S. Weld. Learning 5000relational extractors. In Proceedings of the 48th Annual Meeting of theAssociation for Computational Linguistics, ACL ?10, pages 286?295,Stroudsburg, PA, USA, 2010. Association for Computational Linguis-tics.[28] Hui Huang, Xiaogang Wu, Ragini Pandey, Jiao Li, Guoling Zhao, SaraIbrahim, and Jake Y Chen. C2maps: a network pharmacology databasewith comprehensive disease-gene-drug connectivity relationships. BMCGenomics, 13 Suppl 6:S17, 2012.[29] Minlie Huang and et al. Discovering patterns to extract protein-proteininteractions from full texts, 2004.[30] Adam Janin, Don Baron, Jane Edwards, Dan Ellis, David Gelbart, Nel-son Morgan, Barbara Peskin, Thilo Pfau, Elizabeth Shriberg, AndreasStolcke, and Chuck Wooters. The icsi meeting corpus. pages 364?367,2003.[31] Nitin Jindal and Bing Liu. Review spam detection. In Proceedings ofthe 16th International Conference on World Wide Web, WWW ?07,pages 1189?1190, New York, NY, USA, 2007. ACM.[32] Richard Johansson and Pierre Nugues. Dependency-based semanticrole labeling of propbank. In Proceedings of the Conference on Empiri-cal Methods in Natural Language Processing, EMNLP ?08, pages 69?78,Stroudsburg, PA, USA, 2008. Association for Computational Linguis-tics.36Bibliography[33] Siddhartha Jonnalagadda and Graciela Gonzalez. Biosimplify: an opensource sentence simplification engine to improve recall in automaticbiomedical information extraction. CoRR, abs/1107.5744, 2011.[34] Siddhartha Jonnalagadda, Luis Tari, Jo?rg Hakenberg, Chitta Baral,and Graciela Gonzalez. Towards effective sentence simplification forautomatic processing of biomedical text. CoRR, abs/1001.4277, 2010.[35] Shafiq Joty, Giuseppe Carenini, and Raymond T. Ng. Topic segmen-tation and labeling in asynchronous conversations. J. Artif. Int. Res.,47(1):521?573, May 2013.[36] Won Kim, Ron Kohavi, Johannes Gehrke, and William DuMouchel,editors. Proceedings of the Tenth ACM SIGKDD International Confer-ence on Knowledge Discovery and Data Mining, Seattle, Washington,USA, August 22-25, 2004. ACM, 2004.[37] Bryan Klimt and Yiming Yang. Introducing the enron corpus. In CEAS,2004.[38] Mausam, Michael Schmitz, Robert Bart, Stephen Soderland, and OrenEtzioni. Open language learning for information extraction. In Pro-ceedings of the 2012 Joint Conference on Empirical Methods in NaturalLanguage Processing and Computational Natural Language Learning,EMNLP-CoNLL ?12, pages 523?534, Stroudsburg, PA, USA, 2012. As-sociation for Computational Linguistics.[39] Diana Maynard, Adam Funk, and Wim Peters. Sprat: a tool for au-tomatic semantic pattern-based ontology population. In IN: INTER-NATIONAL CONFERENCE FOR DIGITAL LIBRARIES AND THESEMANTIC WEB, 2009.[40] Ryan Mcdonald, Fernando Pereira, Seth Kulick, Scott Winters, YangJin, and Pete White. Simple algorithms for complex relation extractionwith applications to biomedical ie. In In Proceedings of the 43nd An-nual Meeting of the Association for Computational Linguistics (ACL-05, pages 491?498, 2005.[41] Yashar Mehdad, Giuseppe Carenini, Frank Tompa, and RaymondT. NG. Abstractive meeting summarization with entailment and fusion.In Proceedings of the 14th European Workshop on Natural LanguageGeneration, pages 136?146, Sofia, Bulgaria, August 2013. Associationfor Computational Linguistics.37Bibliography[42] Yuval Merhav, Filipe Mesquita, Denilson Barbosa, Wai Gen Yee, andOphir Frieder. Extracting information networks from the blogosphere.ACM Trans. Web, 6(3):11:1?11:33, October 2012.[43] Junichiro Mori, Takumi Tsujishita, Yutaka Matsuo, and MitsuruIshizuka. Extracting relations in social networks from the web usingsimilarity between collective contexts. In International Semantic WebConference, pages 487?500, 2006.[44] Gabriel Murray and Giuseppe Carenini. Summarizing spoken andwritten conversations. In Proceedings of the Conference on EmpiricalMethods in Natural Language Processing, EMNLP ?08, pages 773?782,Stroudsburg, PA, USA, 2008. Association for Computational Linguis-tics.[45] Gabriel Murray and Giuseppe Carenini. Subjectivity detection in spo-ken and written conversations. Nat. Lang. Eng., 17(3):397?418, July2011.[46] Francisco Oliveira, Fai Wong, and Iok-Sai Hong. Systematic processingof long sentences in rule based portuguese-chinese machine translation.In Proceedings of the 11th International Conference on ComputationalLinguistics and Intelligent Text Processing, CICLing?10, pages 417?426,Berlin, Heidelberg, 2010. Springer-Verlag.[47] Patrick Pantel and Marco Pennacchiotti. Espresso: Leveraging genericpatterns for automatically harvesting semantic relations. In Proceedingsof the 21st International Conference on Computational Linguistics andthe 44th Annual Meeting of the Association for Computational Linguis-tics, ACL-44, pages 113?120, Stroudsburg, PA, USA, 2006. Associationfor Computational Linguistics.[48] Hoifung Poon and Pedro Domingos. Unsupervised semantic parsing. InProceedings of the 2009 Conference on Empirical Methods in NaturalLanguage Processing: Volume 1 - Volume 1, EMNLP ?09, pages 1?10,Stroudsburg, PA, USA, 2009. Association for Computational Linguis-tics.[49] Lev Ratinov and Dan Roth. Design challenges and misconceptions innamed entity recognition. In Proceedings of the Thirteenth Confer-ence on Computational Natural Language Learning, CoNLL ?09, pages147?155, Stroudsburg, PA, USA, 2009. Association for ComputationalLinguistics.38Bibliography[50] Benjamin Rozenfeld and Ronen Feldman. Self-supervised relation ex-traction from the web. Knowl. Inf. Syst., 17(1):17?33, October 2008.[51] Yue Shang, Yanpeng Li, Hongfei Lin, and Zhihao Yang. Enhanc-ing biomedical text summarization using semantic relation extraction,2011.[52] Advaith Siddharthan. Syntactic simplification and text cohesion. Tech-nical report, Research on Language and Computation, 2003.[53] Sara Silveira and Antnio Branco. Enhancing multi-document sum-maries with sentence simplification. In Proceedings of InternationalConference on Artificial Intelligence, Las Vegas, USA, July 2012.[54] Fabian M. Suchanek. Leila: Learning to extract information by linguis-tic analysis. In In Workshop on Ontology Population at ACL/COLING,pages 18?25, 2006.[55] Fabian M. Suchanek, Mauro Sozio, and Gerhard Weikum. Sofie: A self-organizing framework for information extraction. In Proceedings of the18th International Conference on World Wide Web, WWW ?09, pages631?640, New York, NY, USA, 2009. ACM.[56] Mihai Surdeanu, Sanda Harabagiu, John Williams, and Paul Aarseth.Using predicate-argument structures for information extraction. In Pro-ceedings of the 41st Annual Meeting on Association for ComputationalLinguistics - Volume 1, ACL ?03, pages 8?15, Stroudsburg, PA, USA,2003. Association for Computational Linguistics.[57] Maryam Tavafi, Yashar Mehdad, Shafiq Joty, Giuseppe Carenini, andRaymond Ng. Dialogue act recognition in synchronous and asyn-chronous conversations. In Proceedings of the SIGDIAL 2013 Con-ference, pages 117?121, Metz, France, August 2013. Association forComputational Linguistics.[58] Ivan Titov and Alexandre Klementiev. A bayesian model for unsuper-vised semantic parsing. In Proceedings of the 49th Annual Meeting ofthe Association for Computational Linguistics: Human Language Tech-nologies - Volume 1, HLT ?11, pages 1445?1455, Stroudsburg, PA, USA,2011. Association for Computational Linguistics.[59] J. Ulrich, G. Murray, and G. Carenini. A publicly available annotatedcorpus for supervised email summarization. In AAAI08 EMAIL Work-shop, Chicago, USA, 2008. AAAI.39Bibliography[60] Lucy Vanderwende, Hisami Suzuki, Chris Brockett, and Ani Nenkova.Beyond sumbasic: Task-focused summarization with sentence simplifi-cation and lexical expansion. Inf. Process. Manage., 43(6):1606?1618,November 2007.[61] David Vickrey and Daphne Koller. Sentence simplification for semanticrole labeling, 2008.[62] Fei Wu and Daniel S. Weld. Open information extraction usingwikipedia. In Proceedings of the 48th Annual Meeting of the Associationfor Computational Linguistics, ACL ?10, pages 118?127, Stroudsburg,PA, USA, 2010. Association for Computational Linguistics.[63] Ying Xu, Mi-Young Kim, Kevin Quinn, Randy Goebel, and DenilsonBarbosa. Open information extraction with tree kernels. In Proceedingsof the 2013 Conference of the North American Chapter of the Associ-ation for Computational Linguistics: Human Language Technologies,pages 868?877, Atlanta, Georgia, June 2013. Association for Computa-tional Linguistics.[64] Alexander Yates and Oren Etzioni. Unsupervised resolution of objectsand relations on the web. In Proceedings of NAACL HLT, pages 121?130, Rochester, NY, April 2007.[65] Haijun Zhai, Todd Lingren, Louise Deleger, Qi Li, Megan Kaiser, LauraStoutenborough, and Imre Solti. Web 2.0-based crowdsourcing for high-quality gold standard development in clinical natural language pro-cessin. J Med Internet Res, 15(4), April 2013.[66] Min Zhang, Jie Zhang, Jian Su, and Guodong Zhou. A composite ker-nel to extract relations between entities with both flat and structuredfeatures. In ACL-44: Proceedings of the 21st International Conferenceon Computational Linguistics and the 44th annual meeting of the Asso-ciation for Computational Linguistics, pages 825?832, Morristown, NJ,USA, 2006. Association for Computational Linguistics.40


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items