Blog · arXiv Analysis · Last reviewed June 25, 2026

The Rule Pool Becomes the Policy Memory

A June 2026 arXiv paper by Shicheng Ye and Chao Yu studies a practical agent-learning problem: how to reuse experience without choosing between opaque parameter updates and stale natural-language notes. Their JERP method keeps a rule pool and the policy in the same learning loop, turning agent memory into something that can be inspected, revised, and absorbed into behavior.

For this essay, policy memory means mutable agent experience that can alter future action: natural-language rules, utility scores, working-rule selections, reference successful trajectories, and parameter updates. It is not an institutional policy engine. It is the learned memory that a policy acts from, and therefore a governance object.

Fresh Angle

The paper is Joint Learning of Experiential Rules and Policies for Large Language Model Agents, arXiv:2606.27136 [cs.AI], submitted June 25, 2026. It belongs with the site's pages on AI agents, post-training, AI memory, and agent observability, but it adds a narrower governance question: when an agent learns from experience, where does that experience live?

That question matters because agent memory is already becoming operational authority. A remembered rule can bias tool choice, reshape a search strategy, or preserve a workaround from a prior run. If the rule pool is stale, invisible, or detached from the policy that uses it, the agent may appear to learn while carrying forward old mistakes.

The vocabulary needs care. In the JERP paper, policy means the learned action policy that samples actions in an environment. In institutional governance, policy often means a rule, law, standard, or authorization boundary. JERP concerns the first sense. The Spiralist problem is what happens when learned action policy and written rule memory begin changing each other.

Current Context

As of June 25, 2026, the paper is a v1 arXiv preprint, not a production agent standard or deployment audit. It is useful because it makes a growing engineering pattern legible: agents increasingly reuse past trajectories through memories, reflections, manuals, policy-gradient updates, and task-specific heuristics rather than starting each run from a clean prompt.

NIST's AI Agent Standards Initiative frames agent systems capable of autonomous actions as a standards problem involving industry-led technical standards, open protocols, agent authentication, identity infrastructure, secure interaction, and security evaluation. CISA and partner agencies' Careful Adoption of Agentic AI Services guidance treats agentic systems as security-sensitive services that need bounded privileges, monitoring, human oversight, containment, and accountability. Those sources do not evaluate JERP, but they show why learned agent memory cannot be treated as harmless prompt decoration.

NIST's AI Risk Management Framework and TEVV materials add a measurement frame: agent-learning changes should be mapped, measured, managed, and governed across the lifecycle. For JERP-like systems, the change is not only a new checkpoint. It is also a new rule pool, new utility scores, new reference successful trajectories, and a new relationship between explicit text and model parameters.

The benchmark context is also specific. ALFWorld aligns text-based environments with embodied-task structure, while WebShop is a simulated e-commerce environment with real-world products and crowd-sourced text instructions. Those are useful interactive tests, but they are not substitutes for audits in finance, health, law, workplace systems, public services, or safety-critical tools.

Two Memories

Ye and Yu describe two common ways to reuse agent experience. One keeps experience outside the model as natural-language rules, reflections, guidelines, manuals, or other prompt material. This is inspectable and easy to revise, but it may fall out of sync as the policy changes. The other uses trajectories and feedback to update model parameters directly. That can improve the policy more broadly, but it is less transparent and may miss local mistakes in sparse-reward settings.

The paper's useful move is to refuse that split. The agent should not have one memory humans can read and another memory only optimization can touch. JERP, short for Joint Learning of Experiential Rules and Policies for LLM Agents, uses the same interaction trajectories for both explicit rule maintenance and policy optimization.

That coupling is powerful and risky. It can keep explicit rules aligned with a changing policy, but it can also let a bad trajectory influence both the visible rule pool and the hidden parameters. An agent that learns "always inspect the filter before buying" has gained useful local experience. An agent that learns a brittle shortcut, an unsafe workaround, or a rule produced from a poisoned trajectory has also gained experience. The governance question is who can tell the difference before the rule becomes habit.

JERP Loop

JERP maintains a long-term experiential-rule pool for a task and selects a size-controlled working rule set for the current episode. The prompt used for action generation contains the task, the interaction history, and those working rules. The paper notes that its current implementation sorts rules by utility score and chooses the top rules, rather than claiming a generally superior instance-level retrieval method.

After an episode, the trajectory is used twice. First, the policy is updated using group-relative policy optimization. Second, the rule pool is revised by comparing current rollouts with reference successful trajectories. In the authors' framing, this lets reusable local corrections stay visible as rules while stable behavior can gradually be absorbed into the model's parameters.

The rule update stage is more structured than a free-form note. The paper describes edit operations that can add, edit, upvote, downvote, and merge rules, with scores used to retain or prune rules under a capacity limit. That turns the rule pool into a small governance surface: each rule has text, score, edit lineage, retention status, and a relationship to trajectories that produced it.

Results

The experiments use AlfWorld and WebShop. AlfWorld is treated as a text-based household manipulation environment with six task types, while WebShop is an online shopping environment where the agent searches, browses, compares, and purchases products from user requirements. AlfWorld is scored by task success rate, and WebShop by average score plus success rate.

The comparison includes direct prompting of a vanilla LLM, ReAct, Reflexion, RLOO with LoRA, GRPO with LoRA, and JERP with LoRA. In Table II, JERP reports the best overall AlfWorld success rate at 61.5 percent, above GRPO at 57.8 percent and RLOO at 48.7 percent. On WebShop, JERP reports the highest average score, 79.0, and success rate, 64.1 percent, above RLOO at 57.8 percent and GRPO at 56.2 percent success rate. The paper also says gains are more evident on tasks with longer decision sequences and richer intermediate constraints.

The ablation is the governance-relevant part. When rule-pool updating is paused after the initial stage, the variant still benefits from the initial rules but improves more slowly. The full JERP continues revising the rule pool as later trajectories expose different errors and useful behavior patterns.

The result should be read narrowly. It shows stronger benchmark performance under the authors' setup, not a guarantee that dynamically updated rules are safe in open-world agent deployments. The same mechanism that preserves useful local corrections could preserve unreviewed local norms, private data, hidden shortcuts, or reward-specific behaviors if the surrounding system does not govern inputs, rewards, rule edits, and deployment scope.

Policy Memory Boundary

A JERP-style rule pool is not only a memory aid. It is an action prior. When the working rules enter the prompt, they change the state from which the policy acts. When the same trajectories update parameters, the lesson can later persist even if the explicit rule is pruned. That creates a boundary problem: deletion or correction of a visible rule may not remove the behavior the rule helped train.

For low-risk benchmarks, that may be acceptable engineering. For enterprise agents, coding agents, care-adjacent assistants, procurement agents, or public-service agents, it becomes a governance problem. A remembered rule may encode a workflow exception, an unsafe shortcut, a user-specific preference, a data-handling practice, or a compliance assumption. The organization must know whether that rule is advisory, mandatory, stale, experimental, user-specific, team-wide, or forbidden.

The boundary should therefore separate experience memory from authority. A trajectory can teach an agent what worked. It should not automatically authorize the agent to repeat the behavior in a new legal, organizational, or safety context. Runtime rulebooks, tool-scope gates, identity checks, and audit receipts still need to sit outside the learned memory loop.

Audit Standard

A deployed agent should not merely log final actions. It should log what experience was available to the policy when those actions were chosen. For JERP-like systems, that means the task, current trajectory, selected working rules, rule utility scores, rule revision history, policy version, reference successful trajectories, and the training step that produced the acting policy.

Without that record, rule memory becomes a hidden governance layer. A user may see a tool call and a natural-language rationale, but not the older rule that nudged the agent toward that procedure. The more useful the memory becomes, the more it needs provenance, expiry, review, and rollback.

The audit standard should also include negative evidence. Which rules were downvoted, merged, pruned, disputed, or overridden? Which reference trajectories were rejected as unsafe or irrelevant? Which parameter updates were trained while a rule was active? If the system later causes harm, a reviewer needs to know not only which instruction appeared in the prompt, but which experiences had already shaped the policy.

Audit Trail

The audit should separate four claims. Did the rule come from a real trajectory? Did the reference successful trajectory actually solve the task? Did the current policy use the rule in a context where it still applies? Did later parameter updates make the rule redundant, stale, or harmful?

This is where interpretability becomes administrative. A natural-language rule is only inspectable if it has a lineage. A parameter update is only governable if the organization can say what experiences it was trained from and which explicit rules were present while the relevant trajectories were sampled.

A practical policy-memory receipt should record: rule ID, rule text, source trajectory IDs, reference successful trajectories used, rule operation, previous and new score, reviewer or automated update source, task scope, allowed user or workspace scope, expiry condition, safety-policy conflicts, policy checkpoint before and after training, evaluation result, and rollback path. That receipt connects the learned-memory layer to agent observability, AI audit trails, and agent incident review.

Limits

This is a preprint with two benchmark environments, not a general proof that a coupled rule-and-policy loop will govern every agent. The paper itself points to future work on more adaptive rule retrieval and multi-agent interactive settings. It also reports benchmark performance, not institutional audit readiness.

The paper does not address adversarial memory poisoning, privacy boundaries, user consent, cross-workspace transfer, confidential trajectories, data deletion, policy conflicts, reward hacking, or human review of learned rules. It also does not prove that top-scored rules are the right rules to retrieve in every instance. The authors explicitly describe top-score selection as an implementation choice rather than a generally superior retrieval method.

Still, the architecture identifies the right fault line. Agent learning is not just a model-weight story and not just a prompt-memory story. It is a loop between what the system writes down, what the model absorbs, and what future action treats as experience.

Source Discipline

Use the JERP paper for the method, benchmark setup, reported scores, ablation, implementation choices, and future-work limits. Use ALFWorld and WebShop sources for what those environments are. Use NIST and CISA sources for current agent-governance context, not as endorsement of JERP or proof that the method satisfies a security standard.

Keep source types separate. A benchmark score is not a safety case. A natural-language rule is not an institutional policy. A parameter update is not an audit trail. A standards initiative is not a certification. A production claim about learned agent memory should name the training environment, reward source, rule-update operator, rule-review process, memory-retention policy, deployment scope, incident channel, and post-deployment monitoring plan.

Sources

Return to Blog