3.1 Research Questions

We investigate the following research questions with respect to acronym identification and resolution:

Q1. Can a machine algorithm developed successfully for one domain also show strong results in other domains? (Note that the algorithm we use does not require training. This is selected intentionally as we wish to stress generalizability to new domains).

Q2. Are improvements in accuracy achieved with human computation methods over a machine-based algorithm? At what financial cost are improvements, if any, achieved?

Q3. Does performance improve when more human participants are involved?

Q4. For acronyms identified correctly by humans, but incorrectly by the algorithm, is the knowledge utilized available in the document, or is it external?

Q5. How can human computation methods help with the difficult-to-resolve acronyms, e.g., those not easy to resolve algorithmically?

3.2 Data

We used 50 documents (for a total of 150), selected randomly from 3 publicly available collections:

All documents were in English. Because of their excessive length, if the patent or chemical journal documents exceeded 1000 words, we used the first 1000 words rounded to the end of the paragraph closest to the 1000-word limit. For patents, we used the description field. We did not provide a limit to news articles. Table 1 reports some characteristics of each dataset.

The number of characters and syllables per word from each text collection were not significantly different from each other.

However, readability analysis shows that the news articles are the easiest to understand for the general public, whereas the chemistry journals are the most difficult.

This was determined consistently both by the Gunning-Fog index (indicating the number of years of formal education a person requires to easily understand the text on the first reading) and the Flesch-Kinkaid and ARI scores.

Both are estimates of the U.S. grade level needed to comprehend the text. Readability is important as human computation methods are being tested.

4.1 Gold Standard Data

To create a gold standard acronym list, we had two human assessors who were domain experts each independently evaluate the 150 documents to both identify acronyms and identify their related expansions. Each acronym was counted only once per document and lists for each assessor for each document were adjudicated over any disagreements. The acronym counts (per document) are provided in Table 2.

For example, 185 acronyms were found after adjudication for news of which 183 were resolved. We find more acronyms for the journal collection which is consistent with the earlier observation that this collection is the most difficult (of the 3) to read.

4.2 Acronym Identification

Algorithm: We ran the algorithm on the 150 documents, which output the unique acronym identified and their resolutions. Resolutions not found are marked 'unknown.' For example, common acronym, such as PM to represent afternoon, are rarely expanded in documents. Therefore, we can track errors in terms of not identifying strings that are acronyms (task 1).

We can also identify errors in resolution (task 2). Figure 1 provides the pseudocode used by the algorithm.

Crowdsourcing: We had workers on Amazon Mechanical Turk (MTurk) read each document and identify the acronyms (task 1). Figure 2, left, provides a screenshot.

Fig. 2 Screenshots of the acronym identification (left) and resolution (right) tasks provided to crowdworkers 

Each document was processed by nine workers; we tracked the order in which they evaluated the documents.

To determine how the number of crowdworkers affects quality, we calculated consensus judgments as follows.

We took the first three workers submitting results as a set and evaluated majority consensus (2 of 3) and strong majority (3 of 3) consensus decisions.

We repeated this majority and strong majority assessments for sets of the first 5, 7, and 9 workers making submissions. Each worker was paid $0.03 per document. For each, we evaluated recall, precision, and accuracy calculated against our gold standard.

To reduce noise, we provided "honeypots," where acronyms were provided in the document and could be found easily.

Workers who did not identify these correctly had their answers removed from the pool, and the task was relisted.

4.3 Acronym Resolution

Algorithm: Acronym resolution was done in the same process as identification. The analysis was limited to the gold standard acronyms identified (e.g., 183 for the News collection). We found that expanding the window size used by Dannélls did not increase the algorithm's accuracy. Figure 1 provides the pseudocode used by the algorithm.

Crowdsourcing: In a second crowdsourcing run we gave MTurk workers the same documents but with the gold standard acronyms highlighted and asked to resolve them. Figure 2, right, provides a screenshot. Each worker was paid $0.10 per document. Again, we had nine workers process each document and repeated our assessment of majority consensus and strong majority consensus for the first 3, 5, 7, and 9 workers to make submissions.

Also, we asked each worker to mark whether they used the information solely from the document, or they used common knowledge / outside information. As with the acronym identification task, we provided crowdworkers with short-form acronyms where the resolution was provided in the document as "honeypots."

Workers who were unable to identify these had their results removed from the pool and the task was relisted.

Game: The game was designed to provide players with a more entertaining and challenging method of resolving acronyms. Figure 3, left, shows a screenshot. Using the same highlighted documents players had to resolve the terms within a specified time limit (2 minutes per document) and were given real-time accuracy scores (Figure 3, right).

Fig. 3 Screenshots of the acronym resolution task (left) and the scoring calculations (right) provided to game participants 

A leaderboard was provided for top scorers to enter their names. We listed the game on MTurk and compensated each worker $0.05 to start the game and encouraged them to continue playing (and resolving acronyms) for as long as possible.

To accommodate acronym resolutions that were similar but not identical to the gold standard, we stemmed the answer provided and the gold standard definition and did a simple character string match on the stemmed terms (e.g., South African Defence Forces and South African Defence Force, when stemmed, both match the long form of SADF).

If they matched, we increased the player's score. Although this only required for a few participants, later examination showed this technique was 96% accurate at correctly assessing the player's answer. Accuracy was determined as the proportion of acronyms in our gold standard that were correctly resolved.

5.1 Acronym Identification

Table 3 shows the number of acronyms identified by the crowd and machine algorithm for nine assessors.

Table 4 reports accuracy, recall, and precision using majority score for crowdsourcing workers.

Comparing these with scores obtained by the algorithm, the difference for each collection is significant, F(2,147) = 4.32, p = 0.015. We observe that the crowd identified more acronyms than the algorithm in all three collections with greater recall, precision and accuracy scores.

5.2 Acronym Resolution

Table 5 provides accuracy scores for the three methods (algorithm, crowdsourcing, and games). Again, we see that the human computation methods, irrespective of worker set size, do better than the algorithm. Although the crowd format slightly outperformed the game variant; a one-way between-subjects ANOVA found no significant difference in accuracy between them.

We believe the lower performance of the game, although slight, may be due to distractions introduced by the game. There was a significant difference between the crowd/game formats and the machine format. Post-hoc comparisons using the Bonferroni test indicated significant differences between the crowd/game and the machine for the patent and chemical journal collections.

Thus, for these collections, the crowd and game formats were both able to outperform the machine, and this difference was significant for the two collections even when limiting information used to the document. This indicates that human computation methods are better at pulling definitions from the text even without relying on outside information (Q4). The patterns of gain are similar for games with respect to the machine approach.

The resolution of acronyms benefits from human computation methods most when documents are more challenging to read, implying the resolution of acronyms is more difficult as well (Q2). We also note that the algorithm performs reasonably well in these domains though possibly not as well in the biomedical domains for which it was developed (Q1).

5.3 Number of Workers

To answer our research question on the number of workers needed (Q3), we compare the performance scores obtained using all nine assessors with scores using only the first three assessors. In both tasks, we found no significant difference in the quality of results.

This demonstrates that we need only a few assessors from the crowd to obtain quality results. This is similar to the findings by Snow et al. on a separate study of non-experts in NLP tasks [14]. Cutting the number of assessors from 9 to 3 can reduce assessment costs by two thirds.

This, coupled with no measurable gain in quality reinforces our reluctance to use more than three members of the crowd (or game participants) in similar identification and resolution tasks.

Using a paired t-test, we also found no significant difference in quality between the crowd and game approaches. For the acronym resolution task, we spent $135.00 for the nine crowd assessors and half that amount to obtain similar quality in the game with a similar number of assessors. This does not consider the fixed cost required for game development - for short term evaluation; this cost is a factor; for an evaluation over a much longer period, it becomes a trivial cost.

Some acronyms, particularly those in patents, could not be ascertained in the text by any approach and required background knowledge of the domain. The largest reason for acronyms being missed by humans in the identification task is that people confused abbreviations (e.g., Sr. for senior) with acronyms. Machine-based techniques rarely make this type of mistake.

Overall, humans were able to resolve 93%, 86% and 96% of the acronym identification errors made by the machine approach and 44%, 22% and 33% of the acronym resolution errors (on acronyms not requiring external knowledge) made by the machine approach for the news, patent, and chemical journal collections, respectively (Q5). We believe this may be due to the machine algorithm evaluated rules on acronym identification without flexibility, whereas humans were flexible and resolved acronyms that did not follow common rules (such as ATM machine, where machine confounds the algorithm's ability to resolve the long-form equivalent).

Humans provided value when there were several candidate definitions, and the correct one required context to resolve, such as resolving PM when both Post Meridian and Prime Minister appear in the document. Humans also provided value in acronym resolution when definitions in the text contained embedded short-form acronyms (e.g., PRESTO is defined as Precursory Research for Embryonic S&T Program, and S&T is defined elsewhere in the same document as Science and Technology).