From Joseph Weizenbaum to ChatGPT

Critical Encounters with Dazzling AI Technology

Abstract

The paper considers AI systems from a use perspective. It focuses on conversational chatbots, starting from Weizenbaum’s ELIZA and sketching the major scientific advances leading up to ChatGPT. The main discussion builds upon several experiment-reflection cycles conducted by the author to explore ChatGPT as a knowledge resource. The analysis considers ChatGPT responses in terms of accuracy, structure, context, perspective, and bias. The critical evaluation begins with the observation that ChatGPT produces a mixture of clear and precise results and arbitrary misinformation without ever clarifying its own scope. This leads to the identification of the system’s key problem, namely, how it contends with truth, which involves replacing the idea of truth with a probabilistic surrogate based on textual correlation. In responsible use, a system like ChatGPT must be embedded in a human learning culture. A framework for this process should include an insistence on truthfulness, an impulse towards enhancing human competence, and strengthened responsibility structures within communities.