Hypertext Versions of Journal Articles: Computer-aided Linking and Realistic Human-based Evaluation

Author:

W. J. Blustein

Email:

jamie-at-ACM.org

Advisor:

Jean Tague-Sutcliffe (deceased), James K. Mullin and Sylvia Osborn

Award Date:

1999

Institution:

The University of Western Ontario

Institution Location:

Ontario, Canada

Web Location:

ftp://ftp.csd.uwo.ca/pub/thesis/Blustein.PhD.Thesis.ps.gz

Abstract:

My overall objective is to develop and evaluate ways of automatically incorporating hypertext links into pre-existing scholarly journal articles. I describe a rule-based approach for making three types of links (structural, definition, and semantic). Structural links are a way of making explicit some connections between parts of the text. Definition links connect the use of a term, defined elsewhere in the document, to that definition. Links that connect parts of text that discuss similar things are semantic links. I distinguish several types of semantic links.

I use two information retrieval (IR) systems (Cornell\'s SMART system and Bellcore\'s Latent Semantic Indexing) to select links based on the content of the articles. I conducted an experiment to compare the performance of the links forged using these two systems.

The effectiveness of the links (and the rules used to make them) is tested by people reading the hypertext versions for information under a time constraint. A within-subjects experimental design was used. Each of the nineteen experimental participants read one version of each of three scholarly articles in a different hypertext form (one had only simple links, the others had definition links and semantic links selected using one of the IR systems). Subjects\' preferences were also measured.

Although I used three survey articles from published sources for my evaluation experiment there was no difference in reader preference or performance on the basis of article. Subjects ratings of the utility of the various links shows a significant preference for structural links over semantic links. Definition links were preferred to structural links, although the result was not significant. No difference between the links created using the two IR systems was detected.

However there were significant differences in the times that readers spent on documents created using the various treatments When they read in documents with only structural links readers were more likely to have read the whole article, and their satisfaction scores were inversely proportional to their comprehension score.

The method of evaluating hypertext versions of journal articles for use by researchers may be applied to other hypertext versions.