Local model comparison

The saved experiment uses Ollama 0.40.0 on a 2023 MacBook Pro with M2 Pro and 32 GB memory, Python 3.14.8 and the digests in metadata.json. Generation explicitly sets num_ctx to 32768, temperature to 0, seed to 0 and thinking to false. Timings are observations from this machine, not a general model ranking.

Start Ollama and pull:
ollama pull nomic-embed-text
ollama pull qwen3:8b
ollama pull hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF
ollama pull hf.co/bartowski/Qwen2.5-Coder-7B-Instruct-GGUF
ollama pull hf.co/unsloth/Qwen3-14B-GGUF

responses.json stores each generate or classification request and response, plus model show and loaded-memory records. model-tools.jsonl stores tool conversations. titles.json freezes 20 inputs and editorial labels. results.json summarizes the saved run. Byte sizes use decimal GB. Coding throughput uses each model's own eval_count divided by eval_duration in seconds, across two generations. Classification seconds sum 20 API total_duration values, excluding requests such as model loading that are outside those calls.

Before collecting another run, copy responses.json, model-tools.jsonl and metadata.json. Then run:
python3 models.py

This collects output only. It does not execute model-generated code or update approved candidates and scores. titles.json and the 35 notes stay fixed.

Read each new coding response before saving its function to a Python file. Our reviewed folder contains eight functions from the archived run. We removed Markdown fences, checked the entire function and preserved its behavior. reviewed.json records reviewed file hashes. All four models violated the plain-code output instruction. Scores grade the reviewed function behavior only.

To grade an already reviewed file:
python3 score.py reviewed/model-1-repeat-1.py

The three-second subprocess limit can stop a stalled test. It is not a sandbox. Loading the reviewed Python file executes its module. Never pass an unreviewed model response here. check_candidate.py requires a tuple of three actual integers for valid inputs and ValueError for invalid inputs. scores.json preserves every result and two additional cases.

The baseline has two valid and six invalid cases. Llama rejects both valid cases and passes the six rejection cases. Qwen3 8B passes the eight baseline cases but accepts 2026-+1-09. No score proves the full parser contract.

The stored title labels include eight UUID, seven JSON, four site and one AI title. All models disagreed with the same three site labels by naming the tool topic. This measures agreement with our taxonomy, not three clear comprehension errors.

The note index was built with the recorded Nomic embedding model. metadata.json contains corpus and index SHA values. Changing the notes requires rebuilding the index and collecting a new experiment.
