HUMAN '22: Proceedings of the 5th Workshop on Human Factors in Hypertext

Full Citation in the ACM Digital Library

Human-centered AI: ensuring human control while increasing automation

A new synthesis is emerging that integrates Artificial Intelligence (AI) technologies with Human-Computer Interaction to produce Human-Centered AI (HCAI). Advocates of this new synthesis seek to amplify, augment, and enhance human abilities, so as to empower people, build their self-efficacy, support creativity, recognize responsibility, and promote social connections.

Researchers, developers, business leaders, policy makers and others are expanding the technology-centered scope of AI to include HCAI ways of thinking. This expansion from an algorithm-focused view to embrace a human-centered perspective, can shape the future of technology so as to better serve human needs. Educators, designers, software engineers, product managers, evaluators, and government agency staffers can build on AI-driven technologies to design products and services that make life better for the users. These human-centered products and services will enable people to better care for each other, build sustainable communities, and restore the environment. The passionate advocates of HCAI are devoted to furthering human values, rights, justice, and dignity, by building reliable, safe, and trustworthy systems.

Early hypertext systems required user assigned links for text files, giving full control to users, while providing readers with an understandable and predictable design. However, innovators quickly realized that there were many strategies to improve hypertext designs by giving users spatial presentations of the related documents, recommendations for links, ways to collaborate, and interactive animated graphical presentations. Other features supported history-keeping, note-taking, and audio for all users, but especially for users with visual disabilities.

Over time improved hypertext systems incorporated machine learning and other artificial intelligence techniques that provided automation of features, but sometimes produced unexpected and incomprehensible results. Current strategies are to give users more control by providing previews of potential traversals, reminders, alerts, and suggestions that guide human reflection about their goals and methods. Atzenbeck et al. suggest that hypertext is a method of inquiry, opening the door to creativity support tools that accelerate exploration and discovery, amplified by the Al-infused supertools of Human-Centered AI [1].

A medical hypertext scenario could enable a physician to provide a patient history, lab tests, and current symptoms as a starting point. The hypertext system could respond with a set of possible diagnoses, which could be selected by the physician, leading to a refined analysis, links to recent clinical trial results, suggestions of consulting specialists, and recommendations for leading treatment centers. The physician could share the analysis with teammates or specialists to get feedback. The physician's exploration records could be saved to the patient's history, so that the treatment plan could be formulated based on reliable resources and then refined by discussions with patients, who would be given links to patient-centered descriptions of the diagnosis and treatment plan. The physician is responsible for what happens, but this scenario provides a strong history for retrospective analyzes of the choices that were made and the outcomes.

If human-centered AI design scenarios like this one are oriented to amplifying, augmenting, empowering and enhancing human performance, then the chance of successful outcomes will increase. The passionate advocates of HCAI are devoted to furthering human values, rights, justice, and dignity, by building reliable, safe, and trustworthy systems.

The talk will include examples, references to further work, and discussion time for questions. These ideas are drawn from Ben Shneiderman's new book Human-Centered AI [6]. Further information at:

On applying integrative design-oriented cognitive science to hypertext: a framework for cognitive productivity

In The Reflective Practitioner [10], Donald Schön argued that applying science is not a simple matter of drawing logical implications from theory. Practice is messy. How might cognitive science inform the development of hypertext applications? Empirically oriented researchers might proceed in a 'lean' fashion, treating designs as hypotheses to be induced from and tested by data. Instead, in my talk I will show ways in which an integrative design-oriented (IDO) approach to understanding humans as autonomous agents [3] might help design hypertext apps. The approach suggests important problems to address and original ways to think about them. Conversely, designing and testing software can inform our IDO theories. I will illustrate the IDO approach with a theory of, and application for, sleep onset and insomnolence. Then I will focus mainly on hypertext applications that are more directly aimed at improving 'cognitive productivity', meaning how efficiently and effectively one uses knowledge resources to solve problems, deliver services, create products and ultimately improve oneself.

Cognitive science is meant to be the interdisciplinary computational study of brain-based and artificial information processing. In practice, cognitive scientists almost exclusively focus on classical or 'dry' cognition (memory, perception, reasoning, language, etc.) Emotion, motivation and ancillary functions tend at most to be viewed as distinct processes that can interact with cognition. In contrast, the IDO approach is necessarily integrative, seriously attempting to understand diverse blended mental functions, including motivational, affective and ancillary processes. It is partly inspired by artificial general intelligence [8]. While authentically interdisciplinary, the IDO approach is necessarily design-oriented meaning it revolves around attempts to reverse-engineer functionality (the 'designer stance'). It values specification and understanding of competence over prediction. It posits and is intrinsically concerned with requirements of autonomous interdependent agency (humans and other automata), such as the necessity of pursuing multiple top-level motivators in contexts of danger and despite limited knowledge. Two examples of research in this tradition are [1, 12].

A particularly impressive and inspirational early application of the designer stance, underpinning the IDO approach, is Lamontagne's theory of visual motion perception, which though not integrative led to the discovery of an entirely new class of visual illusions [6]. A promising example of integrative empirical psychology, though not from the designer stance, is Agnes Moors' theory of emotion [9]. Imagine how much more can be achieved by combining integrative and design-oriented approaches as described above.

Being so ambitious, the IDO approach calls for doing research and development in teams, which ideally include software architects. A major reason for doing computer simulations in AI is to determine whether one's theory can account for what we think it can, as Lamontagne's simulations did, (rather than it being 'hand waving'). Deriving hypertext or other practical applications from IDO theories will be much more difficult than from non-integrative theories. Like [8] say of the relationship between cognitive science and AGI, applications from IDO might normally (for the foreseeable future) at best be inspired by IDO theories. Still, I believe the back and forth between developing IDO models of autonomous agents and applying them will benefit both science and technology.

My colleagues and I have used the IDO approach in characterizing aspects of 'emotion', repetitive thought, obsession, addiction and mental alarms as mental perturbance [3]. Mental perturbance, by specification, involves the asynchronous generation and activation of motivators, which tend to consume executive functions (attention, deliberation, meta-management), influencing motives, goals and behavior [3, 11, 12]. Mental perturbance is thus inherently an IDO concept.

The IDO approach is being used as a basis for developing the somnolent information-processing (SIP) theory of sleep-onset and insomnolence [2]. SIP is an attempt to reverse-engineer the human sleep-onset control system as involving multiple blended mind-brain functions [7]. One of the postulates of SIP is that mental perturbance is insomnolent (interferes with onset of sleep). The SIP theory in turn informally led to a prediction that certain mental patterns could be both pro-somnolent and counter-insomnolent. If so, then sleep onset could be facilitated by triggering such patterns via the 'cognitive shuffle': imagining diverse concrete objects, scenes and actions prompted by audio hypertext (with SomnoTest and mySleepButton® apps). While SIP is early stage research and development, it may serve as a reference model for future IDO applications.

Drawing on my two Cognitive Productivity books, I will sketch a cognitive productivity framework with seven principles and several key IDO concepts (such as effectance, architecture-based motivation and meta-effectiveness). I will illustrate how IDO-thinking can help specify hypertext software for these principles and concepts. For example, (a) assessing knowledge resources requires distinguishing caliber, utility, potency and appeal (CUP'A). (b) In The Manifesto for Ubiquitous Linking1 24 knowledge workers argued that ubiquitously available hyperlinks (in all user interfaces and automation of user-facing software) would contribute to intermediate awareness[4, 5] (consciousness), psychological flow (aspects of strategic surfing) and cognitive productivity. The manifesto has since been signed by many others. (c) Productive practice is a form of deliberate practice based on research on test-enhanced learning, expertise, testing effects, and other areas. Its objectives are ambitious; they are not merely to enable factual learning, but also the development of manifold 'mindware' underlying habits, attitudes, etc. Each of these concepts (CUP'A, consciousness, flow, mindware, etc.) should be understood in IDO terms and pose research challenges.

The Cognitive Productivity framework also applies to transforming ourselves with fiction. I will suggest that subscription streaming services like Netflix, Inc., and e-book services could be augmented with fun but implicitly serious games hyperlinked to content with a view to helping consumers apply, in their day to day lives, knowledge gleaned from fiction.

The practical end of the Cognitive Productivity framework is to help humanity, with hypertext software, address its opportunity and obligation to use knowledge to become more profoundly effective.

See for additional information.

Doug Engelbart, edge notched cards, and early links

Hypertext is intimately connected to the computer. These two technologies almost seem like they were made for each other Hypertext can be characterized as a random access medium, it offers us a way to form non-sequential connections between ideas. Digital computers, in their modern incarnation, offer superb tools for randomly accessing data. This pairing works well, but it's not the only way to implement a hypertext or hypertext-like system. Edge notched cards, although obscure today, represent an example of hypertext existing separate from digital machines.

This October will be the 60th anniversary of the publication of Doug Engelbart's Augmenting Human Intellect [1]. The eventual product of this research, NLS [2], was a highly influential computerized hypertext system. However, the path towards augmentation started outside the traditional digital realm. Within Augmenting Human Intellect Engelbart describes how he kept a series of linked notes using edge notched cards. These cards are simple pieces of paper with a pattern of notches and holes along their edges.

Edge notched cards were developed alongside punch cards as a means to categorize data. The edges of these cards are used to encode metadata about the contents of their faces. The pattern of holes and notches around each card's perimeter allows specific cards to be pulled from a mixed and unsorted deck. This is accomplished by threading a large needle through a specific hole. Once needled all you have to do is pull up; any cards with a notch in that position will fall out of the deck. This allows users to select specific data from a pack of hundreds of unsorted cards.

The notch card notes that Engelbart discusses in Augmenting Human Intellect are currently housed in the Engelbart Papers at Stanford's special collections. There are on the order of thousands of notch cards in this collection. Combining those notes and Engelbart's published description give us a reasonable understanding of how he was using edge notched cards in his own research. This gives us a window into a non-computer form of hypertext that was used at a crucial time in the field's development. In this set of notes we can see examples of links, tagged metadata, and stitching - all familiar features of hypertext systems.

This keynote will discuss the specifics of this physical form of hypertext. I will show how links as implemented on edge notched cards compare to computer hypertext systems, how this simple technology connects to historic hypertext systems, and what lessons can be learned by looking outside the digital context.

User study on link-service usage and information processing in the context of the world wide web

Associating information by means of linking it is a universal concept of human thinking, and by constructivist means, a possible way of learning through exploring and constructing individual information spaces related to a topic or cross topics. An application, facilitating and externalizing this activity by enabling users to create individual hyperlinks inside the environment of the Web, is a promising way to satisfy this exploratory use of information. The focus on an augmentative approach by lining hypertext's linking paradigm, in conjunction with the Web's vast amount of information, opens up for a broad spectrum of potential use scenarios. The possibilities reflect potential complexities concerning usability and limitations of usage. Therefore, preliminary and iterative evaluations are indispensable for meeting these challenges. We discuss a preliminary evaluation of usability and user behavior of said application by a conducted study based on cross-sectional quasi experimental design, using a controlled test scenario and collected client side data that serves as basis for interpretation on user behavior. Results indicate a strong habituation to document-centric processing and storing of information, and the tendency for transferring this behavior onto the more versatile linking mechanism introduced by the application. We argue for applying additional supportive features, specific for facilitating the reduction of complexity on user-side, and a longer testing period, in order to gain better insight into the possible overcome of habitual patterns concerning the tested use scenario.

Search UI with fill-in-the-blank for clarifying purpose of information exploration and its evaluation

The ability to retrieve needed information from the various types of information available on the Internet is becoming increasingly important. Although a query recommendation is a common search aid, we believe that users need to be able to create their own keywords when attempting to select a proper query. In this paper, we investigate how presenting a search UI when entering search keywords during a web search applying a fill-in-the-blank template (i.e., the Search UI with Fill-in-the-Blank), which reads "I want to know [BLANK] of [BLANK]," affects the ability to create a search query. As a result, the questionnaire evaluation and a user log analysis showed that although the Search UI with Fill-in-the-Blank is burdensome in terms of clarifying information requests of the users, it is effective for allowing the users to create specific queries.

From maintenance in industry to bibliographic data: spatial hypertext as communication medium between user and machine

In this paper, we report on a software demonstrator that utilizes a spatial hypertext UI to support knowledge management in the context of maintenance in industry. To demonstrate the flexibility of that approach, we re-use the software to visualize bibliographic data of the Hypertext conference series.