LLMs of today and the Turing test

 

The Turing test

To quote wikipedia:

The Turing test was designed by Alan Turing to assess a machine‘s ability to exhibit intelligent behaviour equivalent to that of a human by imitating interactive dialogue. 

 

The term AI covers a multitude of different levels of technology.  For those working in the fields, the distinctions are important.  For me, I’m mainly interested in the idea of neural networks in general (which is a subset of the entire field, but a big subset).  For most people, the term is synonymous with what are called large language models, LLMs, such as ChatGPT & Gemini.

Briefly, LLMs are a kind of neural network (NN).  A neural network, as the name implies, is a way of structuring processing to more or less follow the structure of biological nervous systems.  Basically, there are nodes that connect in various ways.  There can be many connections, set up in many layers 

A NN is trained by giving it inputs, letting it generate outputs and then comparing the outputs to what the “right” answers are.  Then the differences are back propagated through the network to adjust the connections.  The set of connections is called a model. These connections represent the “intelligence” of the trained models.

One thing to note about NNs in general (and LLMs in particular) is that they don’t learn on the fly, the way a biological nervous system does.  All the learning happens in the training phase.  The responses to queries can vary, but that is due to the amount of randomness in the query process, not to adjustments in the model parameters, aka, what it “knows”.

 

Failing the Turing test

Do modern AI chatbots, like gemini or ChatGPT, etc, pass the Turing test?  No, though sometimes they can almost seem to have a personality.  I actually had a conversation with gemini on this subject.

I asked gemini to critique my post and it remarked that there is debate on whether modern LLMs do actually pass the Turing test.  It turns out that in studies that it can be possible for humans to think that an LLM is a human.  IRL, I asked the question in the first place because there were times when it did almost feel that I was talking with a human, though I knew I wasn’t.  

It did say that the facts I expressed were basically correct, but added nuances.  I’m going to append its comments rather than change the text, because I think it may be more detail than all my readers will want.

I started by saying that the obvious tell that it’s an AI, not a human, is that it comes up with extensive information on a wide variety of subjects quickly and formatted in meticulous markdown.

It agreed that the speed of the response is a major tell, but it also added that it has no ego and (if it were human) would be like a lab puppy, bounding around trying its best to please the requestor.  (That’s my analogy, not what it provided.)

It also has a tendency to notice unusual ways of expressing ideas in the prompt and echoing them back in the answer.  A real human might or more likely might not do this.

Other tells are:

• No understanding of the physical world
• Large language models are language models, not models of the physical world.  A human could tell you something about how to tie a knot, but an LLM has no clue.
• It doesn’t have a past.  
• A human would have a memory of what it was doing at a time in the past.  
• For an LLM, the conversation you have with it in the moment is all the time it has.  
• Even that isn’t really time, it’s just a series of tokens that it can draw on, though that isn’t so obvious to the human in the conversation.
• It doesn’t have an “self” independent of the conversation.  
• With a human, it might want to tell you about how much fun it is to throw a frisbee for a dog, or a funny video it saw of a dog trying to get a baby in a rocker to throw a frisbee for it. A LLM just tries to answer your question without going off on tangents of its own.

 

Critique from Gemini

Here is a factual and technical review of the concepts in your blog post:

Technical & Factual Accuracy

1. NN / LLM Training & Backpropagation:

   • Your text: You state that a NN is trained by comparing outputs to “right” answers and backpropagating the differences.

   • Accuracy: This description accurately captures Supervised Learning and Fine-Tuning/RLHF (Reinforcement Learning from Human Feedback). It is worth noting that foundational LLM pre-training primarily relies on self-supervised learning (predicting the next token in a sequence without human-labeled “right” answers), though the core mechanism of calculating loss and using backpropagation to adjust model weights/parameters remains identical.

2. Learning On the Fly vs. In-Context Learning:

   • Your text: You state that NNs don’t learn on the fly and that all learning happens in the training phase; variations in responses are due to randomness rather than parameter adjustments.

   • Accuracy: This is factually correct regarding model parameters (weights). The static weights frozen after training represent the model’s fixed knowledge.

   • Contextual Nuance: While weights do not update during a session, LLMs utilize In-Context Learning (ICL)via the context window. When you give an LLM new information during a conversation, it “remembers” and adapts within that specific session prompt, even though its underlying weights remain unchanged.

3. Current Status of the Turing Test:

   • Your text: You mention that modern AI chatbots do not pass the Turing test.

   • Accuracy: In conversational AI research, whether LLMs pass the original Turing Test is actually a subject of active debate. Recent empirical studies (such as research out of UC San Diego using GPT-4 in blinded benchmark tests) show that modern LLMs frequently fool human evaluators at rates higher than 50%, technically passing traditional formulations of the Turing test in controlled settings. However, your core point holds true in practice: long-form interaction, formatting quirks, hyper-speed, and lack of real-world physical grounding make them easily identifiable as AI to an informed user.

4. Physical World Understanding (Knot Tying Example):

   • Your text: You state that an LLM has no clue how to tie a knot or understanding of the physical world.

   • Accuracy: LLMs lack embodied cognition and spatial/physical perception (they have no sensory experience of spatial mechanics or rope tension). However, because they have processed vast amounts of text describing procedural tasks, an LLM can generate detailed, step-by-step instructions on how to tie specific knots (e.g., a bowline or square knot). Clarifying that they lack embodied/spatial understanding rather than text-based knowledge makes this distinction sharper.

5. Memory and Context:

   • Your text: You note that for an LLM, the conversation in the moment is all the time it has, represented as a series of tokens.

   • Accuracy: Spot on. LLMs operate statelessly over token sequences within a context window rather than experiencing continuous temporal flow or episodic memory.