SHUOYU.CHEN / ARCHIVE
Language

Personal archive

Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents

A paper note on using memory-guided LLM agents to turn frozen VLAs into retryable manipulation primitives.

Paper notes
Published
On this page
  1. The problem
  2. The method
  3. Asymmetric division of labor
  4. Two kinds of memory
  5. Retryable closed-loop execution
  6. Results and ablations
  7. Why it matters

The problem

Pretrained vision-language-action (VLA) models are strong at local, contact-rich manipulation—irregular grasping, constrained placement, and fixture actuation that analytic controllers often struggle with. Their reliability, however, is tied closely to the training distribution. Under deployment perturbations such as instruction redirection, shuffled object layouts, or short skills that must be composed into long-horizon routines, monolithic VLAs can degrade sharply.

On LIBERO-Pro, most strong baselines fall below 20% success under perturbation, while even the strongest frozen VLA reaches only 50% when deployed directly. The underlying issue is a mismatch in responsibility: one policy is expected to handle language grounding, task composition, spatial binding, and low-level control at once. Its behavior can therefore remain weakly conditioned on the instruction when the visual scene looks familiar, and one unstable contact can derail an entire long-horizon rollout. Fine-tuning would help, but requires expensive robot data.

The paper asks: can a frozen VLA remain reliable under semantic, spatial, and long-horizon perturbations without fine-tuning?

The method

The core idea is to demote the VLA from the whole controller to one callable primitive, and give the remaining work to an LLM agent. The system is built around three design choices.

Asymmetric division of labor

The system uses one LLM planner and a small, fixed primitive library. Analytic primitives such as `move_to`, `rotate`, and `set_gripper` handle the non-contact structure: target grounding, free-space transport, posture adjustment, navigation, and release.

The only learned primitive, `vla_act`, wraps the frozen VLA and is invoked only for fine contact-rich phases. Before each VLA call, the planner uses analytic primitives to stage the robot in a local observation that is familiar to the VLA. This narrows the frozen model’s responsibility to the part of the task where its training distribution is actually informative.

Two kinds of memory

The planner first explores one reference seed of a task and searches for a successful solution. It then stores two kinds of memory:

  • Task-specific memory keeps a solution skeleton—the ordering of primitive calls and the locations of VLA invocations. Coordinates are never replayed directly; spatial arguments are re-grounded from current visual observations at deployment time.
  • Global memory distills cross-task success rules and failure models. For example, “the gripper closed but the object did not move” indicates an empty grasp, so the robot should re-localize and re-stage before retrying.

Retryable closed-loop execution

The planner issues one JSON primitive command at a time through a file-mediated REPL loop. After each command, it observes the result before deciding what to do next.

Every VLA invocation is treated as a retryable local attempt: stage the robot, invoke the VLA, inspect the contact outcome, and re-stage if the attempt fails. A single unstable contact therefore does not have to ruin the entire rollout.

Results and ablations

The same frozen VLAs are substantially more useful in this setup:

  • LIBERO-Pro: success improves from the frozen baseline’s 50.0% to 82.4%, which is 38.6 percentage points above the strongest prior method.
  • RoboCasa365: the method exceeds the frozen baseline by 25.4 percentage points.
  • RoboTwin C2R: bimanual transfer reaches 58.4%.

The ablations also support the proposed division of labor. Instruction redirection is handled mainly through the planner’s online semantic re-binding, which reaches 79.0% even in the zero-shot setting. Spatial-layout adaptation depends more heavily on the structural skeleton stored in memory: removing memory drops success from 87.0% to 31.0%.

Why it matters

The paper’s most useful framing is that a frozen VLA does not need to be trusted with the entire task. It can be treated as a specialized contact primitive, while deterministic controllers and an LLM planner handle grounding, staging, composition, and recovery around it.

That separation turns a brittle open-loop policy into a system with explicit checkpoints and retry points. It also suggests a practical way to extract more value from existing VLAs when new robot data or fine-tuning compute is limited.

Back to archive