When Is an AI Prototype Worth Pursuing?

During a field test, a few of our robots weren’t communicating as expected. I started working through the logs, looking for what had gone wrong.

We were on a glacier testing EELS, a snake-like robot we were developing at JPL to explore difficult terrain, with places like Saturn’s icy moon Enceladus in mind. But the problem in front of me was much more familiar: a network configuration.

One error in the logs caught my attention. It didn’t mention the Linux /etc/hosts file, but I recognized a possible network configuration problem. I checked the file and found the issue: the hosts weren’t configured correctly, preventing the devices from discovering one another.

At the time, I was also working on AI projects, including Knowledge, and exploring tool calling. Reading a configuration file was an operation I could expose through a tool. So were other checks I routinely performed while investigating a problem.

Could a model use those tools to troubleshoot a robot?

I began developing ROSA, a system for interacting with robots through natural language, to explore that question.

The work brought me to a harder question: when an AI prototype struggles, how do you tell whether you have reached a fundamental limit or found engineering work worth doing?

The first results weren’t good

The early experiments were in the GPT-3.5 era, with GPT-4 just coming online. Later, GPT-4o became part of the picture. This work unfolded as the models themselves were changing.

In those early attempts, I could coax the model into doing the right thing, but often only by giving it very obvious hints.

Those hints mattered. I knew what a useful next step looked like. I knew which information to inspect and how to interpret what came back. If I supplied enough of that knowledge, the model could sometimes complete the task.

That was enough to keep me interested, but it left a practical problem: how much of the work was I still doing?

I started with small, manageable problems where I could recognize a correct result and understand the assistance I was providing. Over time, the work became an exercise in transferring my knowledge into a form the system could use.

Some of that belonged in the robot’s system prompts. A lot of it belonged in the tools: their names, descriptions, responses, formats, and the information they made available. How I exposed the environment mattered, too. Access had to be useful for the task, safe, and appropriate for the robot.

The relevant knowledge ended up distributed across the system. It wasn’t all in a long instruction telling the model how to be a robotics expert.

As results improved, both the models and the system around them were changing. That makes it difficult to attribute progress to either one alone. A better result after a model upgrade and a tooling change does not tell you how much each contributed.

Build a place to learn faster

As things improved, I began trying tasks I hadn’t originally had in mind. In turtlesim, and later Isaac Sim, I asked it to draw houses and other art through movement. The underlying mechanisms were still ROS. The simulator made it easy to conceptualize movement as drawing and explore what the system could do with those mechanisms.

The deeper value was having a place to exercise the system through relevant interfaces and see what happened. The agent had to choose tools, supply arguments, receive feedback and decide what to do next. Drawing made the connection between a request and the resulting movement easy to inspect.

For this kind of development, a useful simulator approximates the parts of the runtime that matter to the task: the available actions, the information returned and the consequences the agent can observe. Visual realism alone tells you little about whether those interactions are representative.

That creates an opportunity for faster, repeatable experiments before returning to hardware. You can change an instruction or tool description, repeat a task and inspect the result. You can also try a different task to see whether the change helped beyond the example that prompted it. Physical testing still matters, especially where the simulator leaves out constraints that affect the real robot.

Testing across robots with different form factors, inputs, and controls exposed other ways the approach could succeed or break down.

Look closely at the help you provide

That experience changed how I interpret an early AI demonstration.

Suppose a model fails to diagnose a problem, and an engineer tells it exactly which configuration file contains the error. The next attempt succeeds. What have we learned?

We’ve learned that it can complete some part of the task with that information. We haven’t established that it can recognize when to inspect the file, distinguish this problem from another one, or notice that its first explanation is wrong.

Now suppose we give it a tool that exposes relevant configuration, explains what the fields mean, and returns information in a usable format. If it then handles other cases without being told their answers, we have evidence of something more reusable.

Those examples are illustrative, not a reconstruction of a particular ROSA test. The distinction is useful because early development mixes several kinds of assistance together. Giving them names makes it easier to see what you are changing:

All four can make a demonstration look better. They imply very different amounts of work remain.

If every new case needs another answer-specific hint, the expert is still carrying much of the task. If a change helps across different cases, we have a stronger reason to investigate further.

Neither result, on its own, settles whether the project is commercially worthwhile.

Memory adds another question: what should the system retain and retrieve across interactions? Context is what it has available for the current task. Memory might preserve a verified fact about a configuration or a useful procedure so it can be brought into a later task. Retaining something does not establish that it is correct, still current or applicable to the next case.

I’ll return to memory in the next article. Here, the focus is on identifying the assistance a system needs and building a way to test whether that assistance transfers.

Choosing the next experiment

For a team considering an AI project, I’d start with a small piece of real work and keep a record of the assistance it requires.

Save the initial attempt. When someone intervenes, record what they supplied and why. Then ask where that knowledge could live if the expert weren’t sitting beside the system.

Perhaps it belongs in a tool description or a returned result. Perhaps a missing source needs to be made accessible. Perhaps the task needs a clear escalation point. Sometimes the answer is that the expert’s judgment remains essential.

That last outcome can still support a useful project. An assistant that prepares evidence for an engineer may be valuable even if it cannot make the final decision. But that is a different scope from an autonomous system, with a different cost and workflow.

Before making changes, reserve other cases for checking whether the improvement transfers. Include cases that look similar but have different causes. Otherwise, it is easy to teach the system to repeat the solution you just showed it and mistake that for broader progress.

Also keep the simpler alternative in view. If one deterministic check catches the recurring problem, it deserves a comparison. The glacier incident suggested a wider possibility to me, but identifying one configuration error would not, by itself, justify building a general robotic assistant.

The next investment should depend on what you can now explain. What improved? Which assistance disappeared? What still requires a person? Is the remaining work compatible with the value of the task?

A disappointing first attempt can contain a promising engineering problem. An impressive demonstration can conceal an expensive dependence on its author. Looking carefully at the help you provide is one way to tell which situation you’re in.