With our first rules file, the model searched the notes for the refund question. It also made 2 + 2 return "I do not know".
You can catch both changes by running the same questions through a small Python harness and saving the requests.
The harness is the code around your model
In part seven, you built the tool loop and checked its arguments. The model name and system prompt still lived in Python.
Here, you put those settings in a config file and append a Markdown rules file to the system prompt.
The Anthropic guide to agent evaluations calls the system that handles inputs and tool results an agent harness.
Our version is small enough to inspect in one file. It has no session recovery or permission dialog, and its only tools read notes.
You get a record of every model turn, including the tool results, so a wrong answer can be traced to the material it received.
Step 1, move the settings into config.json
You can change the model without editing the loop, or disable a tool by removing its name from the list.
{
"model": "qwen3:8b",
"system": "Answer from the notes when the question is about them. If the notes do not contain the answer, say you do not know.",
"instructions": "AGENT.md",
"tools": [
"search_notes",
"read_file"
],
"max_steps": 5,
"num_predict": 300,
"temperature": 0,
"seed": 0,
"log": "runs.jsonl",
"num_ctx": 32768
}The list controls both the schemas you send and the functions you execute. A request for a disabled tool gets a refusal message.
We set num_ctx to 32,768 explicitly so the experiment does not depend on a machine's context default.
The model cannot edit your config, and max_steps must be a positive integer.
Step 2, load your rules beside the system prompt
Save the blocks below together as harness.py. The complete harness example includes its dependencies and the 35 fixed notes.
You choose the filename in instructions, or use an empty string to run without rules.
import json, sys, uuid
from pathlib import Path
import rag, tools
HERE = Path(__file__).resolve().parent
CONFIG = json.loads((HERE / "config.json").read_text(encoding="utf-8"))
INSTRUCTIONS = (HERE / CONFIG["instructions"]).read_text(encoding="utf-8") if CONFIG.get("instructions") else ""
TOOLS = [t for t in tools.TOOLS if t["function"]["name"] in CONFIG["tools"]]You supply the rules as ordinary text in the system message, much like the project instructions covered in our rules file guide.
They can influence which call Qwen3 proposes. They cannot bypass the filename checks or enable a function you disabled in config.
You pass generation settings from that same config into the chat request.
def chat(messages):
body = {"model": CONFIG["model"], "messages": messages, "stream": False, "think": False,
"options": {"num_predict": CONFIG["num_predict"], "num_ctx": CONFIG["num_ctx"],
"temperature": CONFIG["temperature"], "seed": CONFIG["seed"]}}
if TOOLS:
body["tools"] = TOOLS
request = json.loads(json.dumps(body))
return request, rag.post("/api/chat", body)These calls use temperature 0 and seed 0, with thinking disabled. We recorded the model digest and Ollama version in metadata.json.
Step 3, keep enough of the run to check it
One line in runs.jsonl represents one model turn, with a unique run_id to group turns from the same question.
In each record, you get the config and complete API request and response. The results field holds the text your functions returned.
def log(record):
with open(HERE / CONFIG["log"], "a", encoding="utf-8") as f:
f.write(json.dumps(record) + "\n")You keep the part seven argument checks by calling tools.execute after the enabled-tool check, rather than unpacking unchecked arguments yourself.
def run(question):
if type(CONFIG["max_steps"]) is not int or CONFIG["max_steps"] < 1:
raise ValueError("max_steps must be a positive integer")
run_id = str(uuid.uuid4())
system = CONFIG["system"] + ("\n\n" + INSTRUCTIONS if INSTRUCTIONS else "")
messages = [{"role": "system", "content": system}, {"role": "user", "content": question}]
for step in range(CONFIG["max_steps"]):
request, r = chat(messages)
msg = r["message"]
messages.append(msg)
calls = msg.get("tool_calls", [])
if calls is None:
calls = []
record = {"run_id": run_id, "config": CONFIG.copy(), "step": step + 1,
"request": request, "response": r, "results": []}
if not isinstance(calls, list):
record["error"] = "tool_calls must be a list"
log(record)
return record["error"]
if not calls:
log(record)
return msg.get("content", "")
if step == CONFIG["max_steps"] - 1:
record["stopped"] = True
log(record)
break
try:
for call in calls:
fn = call.get("function") if isinstance(call, dict) else None
name = fn.get("name") if isinstance(fn, dict) else None
if isinstance(name, str) and name not in CONFIG["tools"]:
result = f"Tool {name} is not enabled in config.json."
else:
name, result = tools.execute(call)
record["results"].append({"name": name, "content": result})
messages.append({"role": "tool", "tool_name": name, "content": result})
finally:
log(record)
return "Stopped after " + str(CONFIG["max_steps"]) + " model turns."if __name__ == "__main__":
print(run(sys.argv[1]))On your fifth model turn, you can receive an answer, but the code will not start another batch of tools. A turn can contain several function requests.
You can extract a few fields for a readable summary, as we did for the first refund turn under version one.
{
"step": 1,
"prompt_tokens": 397,
"generated_tokens": 25,
"api_seconds": 0.96,
"calls": [
{
"index": 0,
"name": "search_notes",
"arguments": {
"question": "refund policy for the Pro plan"
}
}
]
}This summary shows selected fields from the saved record. You can inspect the full search result and response alongside it in runs.jsonl.
Run a question from the extracted folder after starting Ollama and pulling qwen3:8b and nomic-embed-text.
python3 harness.py "What is 2 + 2?"The first rules file changed more than we wanted
We wanted the refund question checked against the notes before a refusal. That was a preference for this test, not evidence that a refund policy existed.
# How to answer
- The notes are 35 articles from mybittools.com about UUIDs, JSON, AI tools and this website.
- Call search_notes before answering any question that could be about those topics, even when you think you do not know. Only say you do not know after a search returned nothing useful.
- When the user names an article, call read_file with the exact file name. If the name is not exact, pick the closest name from the list the tool returns and call read_file again.
- Answer in at most three sentences and quote numbers exactly as they appear in the notes.We ran the eight part seven questions without rules, then with this file. The questions and generation settings stayed fixed.
Without rules, the refund question returned "I do not know" without a call. Version one searched and claimed the policy was absent, although its three chunks cannot establish absence across all 35 files.
Arithmetic changed too. The answer to 2 + 2 went from 4 to "I do not know", still with no tool call.
The initial arithmetic request used 395 prompt tokens against 270 without rules. The difference was 125 for that pair of requests.
We changed a whole rules file, so this comparison does not identify which sentence caused the arithmetic refusal.
The fonts question stayed limited to the retrieved passages, just as it had in part seven. It did not claim to search the entire collection.
Version two answered 2 + 2 but still missed the article
Version two tells the model to answer arithmetic directly and retry read_file when a filename is not found.
# How to answer
- The notes are 35 articles from mybittools.com about UUIDs, JSON, AI tools and this website.
- If the question is not about those topics, for example arithmetic or general knowledge, answer it directly without any tool.
- For any question that could be about the notes, call search_notes before answering, even when you think you do not know. Only say you do not know after a search returned nothing useful.
- When the user names an article and read_file says the name does not exist, the reply lists every file name. Call read_file again with the one from that list that matches best. Do not switch to search_notes.
- Answer in at most three sentences and quote numbers exactly as they appear in the notes.Now 2 + 2 returned "4." in three generated tokens, while the refund question still searched. You can compare all eight questions in results.json.
For "the uuid collision article", Qwen3 requested uuid-collision, received the real filenames, then called search_notes with the query "v4 figure" instead of reading the listed article.
The list includes uuid-collision-probability. The final answer discussed 122 random bits instead of the article's collision threshold.
The initial arithmetic request grew to 426 prompt tokens, 156 above the version without rules. Later-turn differences need their own comparison.
For example, the third UUID turn used 1,380 tokens with version two and 1,222 without rules, a difference of 158.
The log makes the filename mismatch visible. Automatically choosing another file would require separate checks.
Disable a tool in config and watch the substitute
You can keep version two loaded while removing read_file from the enabled list, as we did for the article-summary and UUID questions.
The summary used search_notes and drew from three chunks. You should not treat that as a read of the full article.
The UUID question also searched, but the answer quoted the number of possible values, about 5.3 times 10 to the 36.
It still missed the requested 50 percent collision threshold. Changing the available tools did not make this answer fit the question.
The rules still mentioned read_file, even though it was unavailable. You can remove that instruction when testing a search-only configuration.
Keep a question that can expose each rule's side effects
The refund question alone would have made version one look successful. Arithmetic exposed a change we did not want.
You need questions where tools should stay unused as well as ones where a lookup matters. Keep their expected behavior beside the responses.
After an edit, compare the arguments and returned text before judging the final answer. A search can run successfully with a useless query.
The saved run exposes a filename mismatch. It does not establish that fuzzy matching would fix the answer.
You can try another naming scheme and repeat the same UUID question. Keep a record of the file it actually opens.
Check the rules again when you switch models
On Qwen3 14B, the laptop question with version two returned "I do not know" in one turn and made no tool call.
Without the file, it searched and answered correctly. Those two turns used 1,358 prompt tokens in total, compared with 476 for the refusal.
That is one question on one model build. You cannot infer that the rules file generally makes 14B worse.
The local model comparison checks this case alongside a coding task. You can inspect both complete conversations in this package.
The logger saves note text to disk, so use material you are comfortable keeping there. This example does not redact the log.
It also stops on transport errors and does not trim growing history. You will need to account for those limits in longer conversations.
Try one edit against the same eight questions
Set instructions to AGENT_v1.md, AGENT.md or an empty string to select the two rule versions or their baseline.
You can rerun all recorded comparisons with python3 reproduce.py. It replaces the saved logs, so copy them first if you want both runs.
Open a record in the JSON log visualizer and compare its proposed function arguments with the returned note text.
Change one rule, rerun all eight questions, and check the calls before accepting the answers.