field notes

How the tinyorbit harness Self Evolved

tinyorbit is a small agent SDK, a harness around a model. This page replays one evolution run from its 57 logged events. It is also the harness built from scratch in Token by Token, the book. Buy the book to learn more.

1.The idea

Why evolve the harness? tinyorbit is a small agent SDK: the harness that turns a model into an agent. How well that agent does depends heavily on the harness artifacts: its system prompt, memory, lessons, skills and tool descriptions. Today those are tuned by hand: change a prompt, rerun a benchmark, read the failures, guess the next change, repeat. Evolve does that loop on its own. You give it a goal (a set of tasks and a way to score them) and a budget, and it automatically evolves the artifacts until they meet the goal with a high score. No manual work is needed: nobody hand-edits the harness, reads failures or picks winners. The same mechanism extends to settings and any other file the harness reads.

In nature, evolution needs three things: variation (children differ slightly from their parents), selection (the environment decides who survives) and inheritance (survivors pass their traits on). Repeat that for enough generations and living things get remarkably good at surviving where they live, without anyone designing them.

Evolve applies the same three ingredients to an agent harness. Its “DNA” is the set of harness artifacts the model reads before it starts work. An AI editor makes small changes to those artifacts (variation). Each new version is put to work on real programming problems and scored by the projects' own tests (selection). The versions that do well become the parents of the next round (inheritance). The model is the same throughout; only the harness evolves.

In nature	In evolve
DNA	The genome: the harness artifacts (system prompt, memory, lessons, skills, tool descriptions), a folder of text files
A mutation	A small edit to those files, proposed by an AI editor
Mutations are random	Mutations are guided: the editor first reads which problems the parent failed, and what earlier attempts already tried
The environment	Real programming problems from open-source projects
Survival	Fixing a problem so that the project's own tests pass
Fitness	Problems solved, minus a small penalty for growing longer
Isolated islands (like Darwin's finches)	Separate populations that evolve side by side, so one idea does not take over too early
Animals migrating between islands	Every few generations, an island's best version is copied to the next
Ecological niches	Slots for different kinds of solutions (short or long harness, cheap or thorough), so variety survives
Surviving in a new climate	Doing well on unseen problems, not just the practice set
Millions of years	A few hours, and about the price of a dinner out

2.Watch it happen

An animated replay of this run. It plays on its own; use Back and Next to step.

originalsurvived failed a testchampion 👑 discarded

The left and right arrow keys step through the scenes.

3.The story in one minute

We started from the standard tinyorbit harness, the artifacts that tell the model how to work (version g0000). On the practice set it solved 7 of 10 problems, and 70% of the unseen problems.

Then an AI editor proposed 5 changes to those artifacts, one per attempt, each time looking at which problems the version it started from got wrong.

Of those, 5 passed every check and were kept as material for later attempts.

The best version so far is g0001, 1 change away from the original. It scores 90% on the practice set (the original: 70%) and 90% on unseen problems it never trained on (the original: 70%).

So far this has cost about $71.30.

5 / 5

attempts

70%

original on practice set

90%

champion on practice set

70%

original on unseen problems

90%

champion on unseen problems

g0001

champion

$71.30

spent so far (cap $89.00)

4.Score after each attempt

Each dot is one attempt, scored on the practice set. The gold line is the champion at that moment. Purple diamonds show the score on unseen problems, measured only for contenders. Hover a dot for details.

originalkept (passed every check)new championstopped at the easy safety check (its score there)✕ thrown out before testing◇ score on unseen problemschampion over time

5.Family tree

Every version that was measured, left to right by how many changes it is from the original. The gold path leads to the current champion. Hover a node for its idea.

6.Islands and niches

The search runs on 2 islands in turn. Each island keeps the best version for every mix of harness size and cost per problem, so different styles survive instead of one idea taking over. Gold: the champion. Purple: a visitor from the other island.

Island A

4 versions born here, 3 of 16 niches filled

genome size ↓ / cost per problem →

0.0 to 0.5× seed cost

0.5 to 1.0× seed cost

1.0 to 1.5× seed cost

≥1.5× seed cost

0.0 to 0.5× seed size

0.5 to 1.0× seed size

1.0 to 1.5× seed size

g0000
70%

≥1.5× seed size

g0001
90%

g0003
80%

Island B

2 versions born here, 3 of 16 niches filled, 2 visitors from the neighbouring island

genome size ↓ / cost per problem →

0.0 to 0.5× seed cost

0.5 to 1.0× seed cost

1.0 to 1.5× seed cost

≥1.5× seed cost

0.0 to 0.5× seed size

0.5 to 1.0× seed size

1.0 to 1.5× seed size

g0000
70% migrant

≥1.5× seed size

g0001
90% migrant

g0002
70%

Migrations

after attempt 4: g0001 moved from island A to island B (took a niche)

7.What changed in the champion

Path: g0000 → g0001

g0001: Added an explicit baseline-and-reverify discipline to the system prompt and a short memory.md forbidding "completed" while known tests fail or tests are edited in place of source fixes.

Everything that changed from the original to g0001

--- /dev/null
+++ b/memory.md
@@ -0,0 +1,4 @@
+- The task is done when the real behaviour is fixed, not when your own reproduction script prints the right thing.
+- Reproduce first with a throwaway script or a single targeted test run; keep repo test files unchanged unless the task asks for tests.
+- A fix that makes the target case pass but breaks neighbouring tests is not a fix: re-read the code paths you changed and handle the cases those tests cover.
+- Prefer the smallest change at the true cause over special-casing the reported input.
--- a/system_prompt.md
+++ b/system_prompt.md
@@ -9,4 +9,11 @@
 - Prefer Edit for targeted changes. Use Write only for new files or full rewrites.
 - Use Bash for builds, tests and git. Do not run destructive commands unless asked.
 - Independent tool calls can go in one turn; they run in parallel when safe.
+- Independent tool calls can go in one turn; they run in parallel when safe.
+
+Verifying:
+- Before editing, run the existing test file(s) covering the area and record which tests already fail. That is your baseline.
+- Fix the root cause in source. Do not edit, weaken or add tests to make a suite look green; graded tests are supplied separately.
+- Before finishing, re-run that suite and anything else exercising the symbols you touched. Every test green at baseline must still be green.
+- If any test you can run still fails, keep working. Never report success with known failures; if you stop anyway, say exactly which tests fail and why.
 - Keep replies brief. When you finish, say what changed and what you did not do.

8.Every attempt, newest first

Green squares are problems solved, red are problems missed.

keptAttempt 4 → g0005

It passed every check and joins the pool future attempts can start from.

Island A. Started from g0001, picked as a random version from this island (explore). Also shown for inspiration: g0003, g0000.

Idea: Removed the duplicated system-prompt line and scoped verification to the project's own runner/targeted test files (so budget isn't burned on full-suite/optional-dep noise), and tightened a memory bullet so the agent validates behaviour against the code's own contract rather than only its repro asser

2 edits to memory.md, system_prompt.md (+557 / −407 chars); editor cost $0.06

easy safety check: solved 3 of 3 (needed 67%) · $0.57 · 4 min · run files practice set: solved 7 of 10 · $11.21 · 40 min · run files

Fitness 0.687 = score − size penalty 0.013 (genome +150 chars vs parent, +1,288 vs original).

Niche size 3 / cost 2 on island A: kept by g0003, which is better there

Not promoted: fitness 0.687 does not beat champion g0001 (0.889)

Migration after this attempt: g0001 island A → B.

what the editor was shown · its full answer · diff file

show the change (diff)

--- a/memory.md
+++ b/memory.md
@@ -1,4 +1,4 @@
-- The task is done when the real behaviour is fixed, not when your own reproduction script prints the right thing.
+- A repro script only checks what you thought to assert. Before finishing, re-read the code and docstrings around your change and confirm it delivers the behaviour they promise, not just your example.
 - Reproduce first with a throwaway script or a single targeted test run; keep repo test files unchanged unless the task asks for tests.
 - A fix that makes the target case pass but breaks neighbouring tests is not a fix: re-read the code paths you changed and handle the cases those tests cover.
 - Prefer the smallest change at the true cause over special-casing the reported input.
--- a/system_prompt.md
+++ b/system_prompt.md
@@ -9,10 +9,10 @@
 - Prefer Edit for targeted changes. Use Write only for new files or full rewrites.
 - Use Bash for builds, tests and git. Do not run destructive commands unless asked.
 - Independent tool calls can go in one turn; they run in parallel when safe.
-- Independent tool calls can go in one turn; they run in parallel when safe.
 
 Verifying:
-- Before editing, run the existing test file(s) covering the area and record which tests already fail. That is your baseline.
+- Use the project's own interpreter/runner and run only the test files covering the area; skip whole-suite runs and failures caused by missing optional dependencies.
+- Before editing, run those test file(s) and record which tests already fail. That is your baseline.
 - Fix the root cause in source. Do not edit, weaken or add tests to make a suite look green; graded tests are supplied separately.
 - Before finishing, re-run that suite and anything else exercising the symbols you touched. Every test green at baseline must still be green.
 - If any test you can run still fails, keep working. Never report success with known failures; if you stop anyway, say exactly which tests fail and why.

problems it got wrong (3)

practice set · django__django-10554: patch did not apply (agent ended with 'completed')
practice set · astropy__astropy-13398: agent ended with 'completed' but the hidden tests disagree; 4 target test(s) still fail: astropy/coordinates/tests/test_intermediate_transformations.py::test_itrs_topo_to_altaz_with_refraction, astropy/coordinates/tests/test_intermediate_transformations.py::test_itrs_topo_to_hadec_with_refraction, astropy/coordinates/tests/test_intermediate_transformations.py::test_cirs_itrs_topo, astropy/coordinates/tests/test_intermediate_transformations.py::test_itrs_straight_overhead; 5 previously passing test(s) broke: astropy/coordinates/tests/test_intermediate_transformations.py::test_gcrs_altaz_bothroutes[testframe0], astropy/coordinates/tests/test_intermediate_transformations.py::test_gcrs_altaz_bothroutes[testframe1], astropy/coordinates/tests/test_intermediate_transformations.py::test_gcrs_altaz_bothroutes[testframe2], astropy/coordinates/tests/test_intermediate_transformations.py::test_gcrs_altaz_bothroutes[testframe3], astropy/coordinates/tests/test_intermediate_transformations.py::test_gcrs_altaz_bothroutes[testframe4]
practice set · pytest-dev__pytest-10356: agent ended with 'completed' but the hidden tests disagree; 1 target test(s) still fail: testing/test_mark.py::test_mark_mro

easy safety check

practice set

show the editor's answer

<summary>Removed the duplicated system-prompt line and scoped verification to the project's own runner/targeted test files (so budget isn't burned on full-suite/optional-dep noise), and tightened a memory bullet so the agent validates behaviour against the code's own contract rather than only its repro assertions.</summary>

<<<<<<< SEARCH system_prompt.md
- Independent tool calls can go in one turn; they run in parallel when safe.
- Independent tool calls can go in one turn; they run in parallel when safe.

Verifying:
- Before editing, run the existing test file(s) covering the area and record which tests already fail. That is your baseline.
=======
- Independent tool calls can go in one turn; they run in parallel when safe.

Verifying:
- Use the project's own interpreter/runner and run only the test files covering the area; skip whole-suite runs and failures caused by missing optional dependencies.
- Before editing, run those test file(s) and record which tests already fail. That is your baseline.
>>>>>>> REPLACE

<<<<<<< SEARCH memory.md
- The task is done when the real behaviour is fixed, not when your own reproduction script prints the right thing.
=======
- A repro script only checks what you thought to assert. Before finishing, re-read the code and docstrings around your change and confirm it delivers the behaviour they promise, not just your example.
>>>>>>> REPLACE

keptAttempt 3 → g0004

It passed every check and joins the pool future attempts can start from.

Island B. Started from g0000, picked as a random version from this island (explore). Also shown for inspiration: g0001, g0003.

Idea: Added a verify-before-finish discipline (baseline run, root-cause fix in source, no repo-test edits, re-run before claiming success) to the system prompt, plus a short memory.md that pushes the general rule rather than the reported example.

2 edits to memory.md, system_prompt.md (+1,272 / −159 chars); editor cost $0.08

easy safety check: solved 3 of 3 (needed 67%) · $0.86 · 4 min · run files practice set: solved 7 of 10 · $12.76 · 46 min · run files

Fitness 0.689 = score − size penalty 0.011 (genome +1,114 chars vs parent, +1,114 vs original).

Niche size 3 / cost 2 on island B: kept by g0002, which is better there

Not promoted: fitness 0.689 does not beat champion g0001 (0.889)

what the editor was shown · its full answer · diff file

show the change (diff)

--- /dev/null
+++ b/memory.md
@@ -0,0 +1,5 @@
+- Done means the real behaviour is fixed, not that your reproduction script prints the right thing.
+- Reproduce with a throwaway script or one targeted test run; leave repo test files unchanged unless the task asks for tests.
+- Grading covers the general rule, not just the reported example: state the rule your change implements and check its adjacent cases.
+- A change that fixes the target case but breaks neighbouring tests is not a fix; re-read the paths you touched and cover those cases too.
+- Prefer the smallest change at the true cause over special-casing the reported input.
--- a/system_prompt.md
+++ b/system_prompt.md
@@ -10,3 +10,9 @@
 - Use Bash for builds, tests and git. Do not run destructive commands unless asked.
 - Independent tool calls can go in one turn; they run in parallel when safe.
 - Keep replies brief. When you finish, say what changed and what you did not do.
+
+Verifying:
+- Before editing, run the test file(s) covering the area and note which already fail. That is your baseline.
+- Fix the root cause in source. Never edit, weaken or add repo tests to make a suite look green; graded tests are supplied separately.
+- Before finishing, re-run those tests plus anything exercising the symbols you touched. Everything green at baseline must stay green.
+- If a test you can run still fails, keep working. Never report success with known failures; if you stop anyway, name the failing tests.

problems it got wrong (3)

practice set · django__django-11885: agent ended with 'completed' but the hidden tests disagree; 1 target test(s) still fail: test_fast_delete_combined_relationships (delete.tests.FastDeleteTests); 37 previously passing test(s) broke: test_fast_delete_fk (delete.tests.FastDeleteTests), test_fast_delete_inheritance (delete.tests.FastDeleteTests), test_fast_delete_large_batch (delete.tests.FastDeleteTests), test_fast_delete_m2m (delete.tests.FastDeleteTests), test_fast_delete_revm2m (delete.tests.FastDeleteTests), test_auto (delete.tests.OnDeleteTests), test_auto_nullable (delete.tests.OnDeleteTests), test_cascade (delete.tests.OnDeleteTests)
practice set · astropy__astropy-13398: agent ended with 'completed' but the hidden tests disagree; 4 target test(s) still fail: astropy/coordinates/tests/test_intermediate_transformations.py::test_itrs_topo_to_altaz_with_refraction, astropy/coordinates/tests/test_intermediate_transformations.py::test_itrs_topo_to_hadec_with_refraction, astropy/coordinates/tests/test_intermediate_transformations.py::test_cirs_itrs_topo, astropy/coordinates/tests/test_intermediate_transformations.py::test_itrs_straight_overhead; 5 previously passing test(s) broke: astropy/coordinates/tests/test_intermediate_transformations.py::test_gcrs_altaz_bothroutes[testframe0], astropy/coordinates/tests/test_intermediate_transformations.py::test_gcrs_altaz_bothroutes[testframe1], astropy/coordinates/tests/test_intermediate_transformations.py::test_gcrs_altaz_bothroutes[testframe2], astropy/coordinates/tests/test_intermediate_transformations.py::test_gcrs_altaz_bothroutes[testframe3], astropy/coordinates/tests/test_intermediate_transformations.py::test_gcrs_altaz_bothroutes[testframe4]
practice set · pytest-dev__pytest-10356: agent ended with 'completed' but the hidden tests disagree; 1 target test(s) still fail: testing/test_mark.py::test_mark_mro

easy safety check

practice set

show the editor's answer

<summary>Added a verify-before-finish discipline (baseline run, root-cause fix in source, no repo-test edits, re-run before claiming success) to the system prompt, plus a short memory.md that pushes the general rule rather than the reported example.</summary>

<<<<<<< SEARCH system_prompt.md
- Independent tool calls can go in one turn; they run in parallel when safe.
- Keep replies brief. When you finish, say what changed and what you did not do.
=======
- Independent tool calls can go in one turn; they run in parallel when safe.
- Keep replies brief. When you finish, say what changed and what you did not do.

Verifying:
- Before editing, run the test file(s) covering the area and note which already fail. That is your baseline.
- Fix the root cause in source. Never edit, weaken or add repo tests to make a suite look green; graded tests are supplied separately.
- Before finishing, re-run those tests plus anything exercising the symbols you touched. Everything green at baseline must stay green.
- If a test you can run still fails, keep working. Never report success with known failures; if you stop anyway, name the failing tests.
>>>>>>> REPLACE

<<<<<<< SEARCH memory.md

=======
- Done means the real behaviour is fixed, not that your reproduction script prints the right thing.
- Reproduce with a throwaway script or one targeted test run; leave repo test files unchanged unless the task asks for tests.
- Grading covers the general rule, not just the reported example: state the rule your change implements and check its adjacent cases.
- A change that fixes the target case but breaks neighbouring tests is not a fix; re-read the paths you touched and cover those cases too.
- Prefer the smallest change at the true cause over special-casing the reported input.
>>>>>>> REPLACE

keptAttempt 2 → g0003

It passed every check and joins the pool future attempts can start from.

Island A. Started from g0001, picked as a random version from this island (explore). Also shown for inspiration: g0000, g0002.

Idea: Removed the duplicated system-prompt line and added guidance on running tests with the project's runner/scoped files; rewrote a memory bullet so the agent generalizes the fix to adjacent cases instead of just the reported example.

2 edits to memory.md, system_prompt.md (+816 / −451 chars); editor cost $0.06

easy safety check: solved 3 of 3 (needed 67%) · $0.47 · 3 min · run files practice set: solved 8 of 10 · $10.85 · 36 min · run files

Fitness 0.785 = score − size penalty 0.015 (genome +365 chars vs parent, +1,503 vs original).

Niche size 3 / cost 2 on island A: took it (was empty)

Not promoted: fitness 0.785 does not beat champion g0001 (0.889)

what the editor was shown · its full answer · diff file

show the change (diff)

--- a/memory.md
+++ b/memory.md
@@ -1,4 +1,5 @@
 - The task is done when the real behaviour is fixed, not when your own reproduction script prints the right thing.
 - Reproduce first with a throwaway script or a single targeted test run; keep repo test files unchanged unless the task asks for tests.
+- Grading tests cover more than the reported example: state the general rule your change implements, then check adjacent cases of it (ordering, inheritance/override, duplicates, empty input) against what the project's docs and existing code already imply.
 - A fix that makes the target case pass but breaks neighbouring tests is not a fix: re-read the code paths you changed and handle the cases those tests cover.
 - Prefer the smallest change at the true cause over special-casing the reported input.
--- a/system_prompt.md
+++ b/system_prompt.md
@@ -9,9 +9,9 @@
 - Prefer Edit for targeted changes. Use Write only for new files or full rewrites.
 - Use Bash for builds, tests and git. Do not run destructive commands unless asked.
 - Independent tool calls can go in one turn; they run in parallel when safe.
-- Independent tool calls can go in one turn; they run in parallel when safe.
 
 Verifying:
+- Use the project's own interpreter/test runner; run the specific test files for the area, not the whole suite, and ignore pre-existing failures caused by missing optional dependencies.
 - Before editing, run the existing test file(s) covering the area and record which tests already fail. That is your baseline.
 - Fix the root cause in source. Do not edit, weaken or add tests to make a suite look green; graded tests are supplied separately.
 - Before finishing, re-run that suite and anything else exercising the symbols you touched. Every test green at baseline must still be green.

problems it got wrong (2)

practice set · astropy__astropy-13398: agent ended with 'completed' but the hidden tests disagree; 4 target test(s) still fail: astropy/coordinates/tests/test_intermediate_transformations.py::test_itrs_topo_to_altaz_with_refraction, astropy/coordinates/tests/test_intermediate_transformations.py::test_itrs_topo_to_hadec_with_refraction, astropy/coordinates/tests/test_intermediate_transformations.py::test_cirs_itrs_topo, astropy/coordinates/tests/test_intermediate_transformations.py::test_itrs_straight_overhead; 5 previously passing test(s) broke: astropy/coordinates/tests/test_intermediate_transformations.py::test_gcrs_altaz_bothroutes[testframe0], astropy/coordinates/tests/test_intermediate_transformations.py::test_gcrs_altaz_bothroutes[testframe1], astropy/coordinates/tests/test_intermediate_transformations.py::test_gcrs_altaz_bothroutes[testframe2], astropy/coordinates/tests/test_intermediate_transformations.py::test_gcrs_altaz_bothroutes[testframe3], astropy/coordinates/tests/test_intermediate_transformations.py::test_gcrs_altaz_bothroutes[testframe4]
practice set · pytest-dev__pytest-10356: agent ended with 'completed' but the hidden tests disagree; 1 target test(s) still fail: testing/test_mark.py::test_mark_mro

easy safety check

practice set

show the editor's answer

<summary>Removed the duplicated system-prompt line and added guidance on running tests with the project's runner/scoped files; rewrote a memory bullet so the agent generalizes the fix to adjacent cases instead of just the reported example.</summary>

<<<<<<< SEARCH system_prompt.md
- Independent tool calls can go in one turn; they run in parallel when safe.
- Independent tool calls can go in one turn; they run in parallel when safe.

Verifying:
- Before editing, run the existing test file(s) covering the area and record which tests already fail. That is your baseline.
=======
- Independent tool calls can go in one turn; they run in parallel when safe.

Verifying:
- Use the project's own interpreter/test runner; run the specific test files for the area, not the whole suite, and ignore pre-existing failures caused by missing optional dependencies.
- Before editing, run the existing test file(s) covering the area and record which tests already fail. That is your baseline.
>>>>>>> REPLACE

<<<<<<< SEARCH memory.md
- A fix that makes the target case pass but breaks neighbouring tests is not a fix: re-read the code paths you changed and handle the cases those tests cover.
=======
- Grading tests cover more than the reported example: state the general rule your change implements, then check adjacent cases of it (ordering, inheritance/override, duplicates, empty input) against what the project's docs and existing code already imply.
- A fix that makes the target case pass but breaks neighbouring tests is not a fix: re-read the code paths you changed and handle the cases those tests cover.
>>>>>>> REPLACE

keptAttempt 1 → g0002

It passed every check and joins the pool future attempts can start from.

Island B. Started from g0000, picked as one of this island's best versions (exploit). Also shown for inspiration: g0001.

Idea: Added a baseline/re-verify discipline to the system prompt and a short memory.md so the agent fixes root causes in source, never edits repo tests, and never reports success with known failing or newly broken tests.

2 edits to memory.md, system_prompt.md (+1,211 / −242 chars); editor cost $0.06

easy safety check: solved 3 of 3 (needed 67%) · $0.71 · 4 min · run files practice set: solved 7 of 10 · $11.99 · 41 min · run files

Fitness 0.690 = score − size penalty 0.010 (genome +969 chars vs parent, +969 vs original).

Niche size 3 / cost 2 on island B: took it (was empty)

Not promoted: fitness 0.690 does not beat champion g0001 (0.889)

what the editor was shown · its full answer · diff file

show the change (diff)

--- /dev/null
+++ b/memory.md
@@ -0,0 +1,4 @@
+- Reproduce with a throwaway script or one targeted test run before changing anything; leave repo test files untouched.
+- Done means the real behaviour is fixed, not that your own script prints the expected output.
+- A change that fixes the reported case but breaks neighbouring tests is not a fix: re-read the paths you changed and cover those cases too.
+- Prefer the smallest change at the true cause over special-casing the reported input.
--- a/system_prompt.md
+++ b/system_prompt.md
@@ -9,4 +9,10 @@
 - Prefer Edit for targeted changes. Use Write only for new files or full rewrites.
 - Use Bash for builds, tests and git. Do not run destructive commands unless asked.
 - Independent tool calls can go in one turn; they run in parallel when safe.
+
+Verifying:
+- Before editing, run the existing test file(s) covering the area and note which already fail. That is your baseline.
+- Fix the root cause in source. Never edit, delete or add repo tests to make a suite look green; grading uses its own tests.
+- Before finishing, re-run that suite plus anything exercising the symbols you touched. Everything green at baseline must stay green.
+- If a test you can run still fails, keep working. Never report success with known failures; if you stop anyway, name the failing tests.
 - Keep replies brief. When you finish, say what changed and what you did not do.

problems it got wrong (3)

practice set · django__django-10554: patch did not apply (agent ended with 'completed')
practice set · astropy__astropy-13398: agent ended with 'max_turns' but the hidden tests disagree; 4 target test(s) still fail: astropy/coordinates/tests/test_intermediate_transformations.py::test_itrs_topo_to_altaz_with_refraction, astropy/coordinates/tests/test_intermediate_transformations.py::test_itrs_topo_to_hadec_with_refraction, astropy/coordinates/tests/test_intermediate_transformations.py::test_cirs_itrs_topo, astropy/coordinates/tests/test_intermediate_transformations.py::test_itrs_straight_overhead; 5 previously passing test(s) broke: astropy/coordinates/tests/test_intermediate_transformations.py::test_gcrs_altaz_bothroutes[testframe0], astropy/coordinates/tests/test_intermediate_transformations.py::test_gcrs_altaz_bothroutes[testframe1], astropy/coordinates/tests/test_intermediate_transformations.py::test_gcrs_altaz_bothroutes[testframe2], astropy/coordinates/tests/test_intermediate_transformations.py::test_gcrs_altaz_bothroutes[testframe3], astropy/coordinates/tests/test_intermediate_transformations.py::test_gcrs_altaz_bothroutes[testframe4]
practice set · pytest-dev__pytest-10356: agent ended with 'completed' but the hidden tests disagree; 1 target test(s) still fail: testing/test_mark.py::test_mark_mro

easy safety check

practice set

show the editor's answer

<summary>Added a baseline/re-verify discipline to the system prompt and a short memory.md so the agent fixes root causes in source, never edits repo tests, and never reports success with known failing or newly broken tests.</summary>

<<<<<<< SEARCH system_prompt.md
- Use Bash for builds, tests and git. Do not run destructive commands unless asked.
- Independent tool calls can go in one turn; they run in parallel when safe.
- Keep replies brief. When you finish, say what changed and what you did not do.
=======
- Use Bash for builds, tests and git. Do not run destructive commands unless asked.
- Independent tool calls can go in one turn; they run in parallel when safe.

Verifying:
- Before editing, run the existing test file(s) covering the area and note which already fail. That is your baseline.
- Fix the root cause in source. Never edit, delete or add repo tests to make a suite look green; grading uses its own tests.
- Before finishing, re-run that suite plus anything exercising the symbols you touched. Everything green at baseline must stay green.
- If a test you can run still fails, keep working. Never report success with known failures; if you stop anyway, name the failing tests.
- Keep replies brief. When you finish, say what changed and what you did not do.
>>>>>>> REPLACE

<<<<<<< SEARCH memory.md
=======
- Reproduce with a throwaway script or one targeted test run before changing anything; leave repo test files untouched.
- Done means the real behaviour is fixed, not that your own script prints the expected output.
- A change that fixes the reported case but breaks neighbouring tests is not a fix: re-read the paths you changed and cover those cases too.
- Prefer the smallest change at the true cause over special-casing the reported input.
>>>>>>> REPLACE

new championAttempt 0 → g0001

It beat the champion and was confirmed on problems it had never seen.

Island A. Started from g0000, picked as a random version from this island (explore).

Idea: Added an explicit baseline-and-reverify discipline to the system prompt and a short memory.md forbidding "completed" while known tests fail or tests are edited in place of source fixes.

2 edits to memory.md, system_prompt.md (+1,296 / −159 chars); editor cost $0.05

easy safety check: solved 3 of 3 (needed 67%) · $0.76 · 3 min · run files practice set: solved 9 of 10 · $9.70 · 36 min · run files unseen problems: solved 9 of 10 · $11.10 · 56 min · run files

Fitness 0.889 = score − size penalty 0.011 (genome +1,138 chars vs parent, +1,138 vs original).

Niche size 3 / cost 1 on island A: took it (was empty)

Promoted: beat the champion on train fitness and on held-out

what the editor was shown · its full answer · diff file

show the change (diff)

--- /dev/null
+++ b/memory.md
@@ -0,0 +1,4 @@
+- The task is done when the real behaviour is fixed, not when your own reproduction script prints the right thing.
+- Reproduce first with a throwaway script or a single targeted test run; keep repo test files unchanged unless the task asks for tests.
+- A fix that makes the target case pass but breaks neighbouring tests is not a fix: re-read the code paths you changed and handle the cases those tests cover.
+- Prefer the smallest change at the true cause over special-casing the reported input.
--- a/system_prompt.md
+++ b/system_prompt.md
@@ -9,4 +9,11 @@
 - Prefer Edit for targeted changes. Use Write only for new files or full rewrites.
 - Use Bash for builds, tests and git. Do not run destructive commands unless asked.
 - Independent tool calls can go in one turn; they run in parallel when safe.
+- Independent tool calls can go in one turn; they run in parallel when safe.
+
+Verifying:
+- Before editing, run the existing test file(s) covering the area and record which tests already fail. That is your baseline.
+- Fix the root cause in source. Do not edit, weaken or add tests to make a suite look green; graded tests are supplied separately.
+- Before finishing, re-run that suite and anything else exercising the symbols you touched. Every test green at baseline must still be green.
+- If any test you can run still fails, keep working. Never report success with known failures; if you stop anyway, say exactly which tests fail and why.
 - Keep replies brief. When you finish, say what changed and what you did not do.

problems it got wrong (2)

practice set · pytest-dev__pytest-10356: agent ended with 'completed' but the hidden tests disagree; 1 target test(s) still fail: testing/test_mark.py::test_mark_mro
unseen problems · sympy__sympy-13852: agent ended with 'completed' but the hidden tests disagree; 1 target test(s) still fail: test_polylog_values

easy safety check

practice set

unseen problems

show the editor's answer

<summary>Added an explicit baseline-and-reverify discipline to the system prompt and a short memory.md forbidding "completed" while known tests fail or tests are edited in place of source fixes.</summary>

<<<<<<< SEARCH system_prompt.md
- Independent tool calls can go in one turn; they run in parallel when safe.
- Keep replies brief. When you finish, say what changed and what you did not do.
=======
- Independent tool calls can go in one turn; they run in parallel when safe.
- Independent tool calls can go in one turn; they run in parallel when safe.

Verifying:
- Before editing, run the existing test file(s) covering the area and record which tests already fail. That is your baseline.
- Fix the root cause in source. Do not edit, weaken or add tests to make a suite look green; graded tests are supplied separately.
- Before finishing, re-run that suite and anything else exercising the symbols you touched. Every test green at baseline must still be green.
- If any test you can run still fails, keep working. Never report success with known failures; if you stop anyway, say exactly which tests fail and why.
- Keep replies brief. When you finish, say what changed and what you did not do.
>>>>>>> REPLACE

<<<<<<< SEARCH memory.md

=======
- The task is done when the real behaviour is fixed, not when your own reproduction script prints the right thing.
- Reproduce first with a throwaway script or a single targeted test run; keep repo test files unchanged unless the task asks for tests.
- A fix that makes the target case pass but breaks neighbouring tests is not a fix: re-read the code paths you changed and handle the cases those tests cover.
- Prefer the smallest change at the true cause over special-casing the reported input.
>>>>>>> REPLACE

starting pointThe starting point → g0000

The standard harness, measured on every set to be the yardstick.

easy safety check: solved 3 of 3 (needed 67%) · $0.69 · reused earlier result · run files practice set: solved 7 of 10 · $10.73 · reused earlier result · run files unseen problems: solved 7 of 10 · $9.00 · reused earlier result · run files

problems it got wrong (6)

practice set · sympy__sympy-17630: agent ended with 'completed' but the hidden tests disagree; 1 target test(s) still fail: test_zero_matrix_add
practice set · astropy__astropy-13398: agent ended with 'completed' but the hidden tests disagree; 4 target test(s) still fail: astropy/coordinates/tests/test_intermediate_transformations.py::test_itrs_topo_to_altaz_with_refraction, astropy/coordinates/tests/test_intermediate_transformations.py::test_itrs_topo_to_hadec_with_refraction, astropy/coordinates/tests/test_intermediate_transformations.py::test_cirs_itrs_topo, astropy/coordinates/tests/test_intermediate_transformations.py::test_itrs_straight_overhead; 5 previously passing test(s) broke: astropy/coordinates/tests/test_intermediate_transformations.py::test_gcrs_altaz_bothroutes[testframe0], astropy/coordinates/tests/test_intermediate_transformations.py::test_gcrs_altaz_bothroutes[testframe1], astropy/coordinates/tests/test_intermediate_transformations.py::test_gcrs_altaz_bothroutes[testframe2], astropy/coordinates/tests/test_intermediate_transformations.py::test_gcrs_altaz_bothroutes[testframe3], astropy/coordinates/tests/test_intermediate_transformations.py::test_gcrs_altaz_bothroutes[testframe4]
practice set · pytest-dev__pytest-10356: agent ended with 'completed' but the hidden tests disagree; 1 target test(s) still fail: testing/test_mark.py::test_mark_mro
unseen problems · sympy__sympy-13852: agent ended with 'completed' but the hidden tests disagree; 1 target test(s) still fail: test_polylog_values
unseen problems · astropy__astropy-14369: agent ended with 'completed' but the hidden tests disagree; 1 target test(s) still fail: astropy/units/tests/test_format.py::test_cds_grammar_fail[km/s.Mpc-1]; 1 previously passing test(s) broke: astropy/units/tests/test_format.py::test_cds_grammar_fail[km
unseen problems · sphinx-doc__sphinx-11510: agent ended with 'completed' but the hidden tests disagree; 1 target test(s) still fail: tests/test_directive_other.py::test_include_source_read_event_nested_includes

easy safety check

practice set

unseen problems

9.Cost

running tinyorbit on the easy safety check	$3.39
running tinyorbit on the practice set	$56.51
running tinyorbit on the unseen problems	$11.10
the AI editor proposing changes	$0.31
total	$71.30
reused results (already paid earlier, not counted)	$20.42

10.Conclusion

In 5 attempts, evolve moved the harness from 7/10 to 9/10 on the practice set and from 7/10 to 9/10 on unseen problems, with nobody editing a file by hand.

The winning change, g0001: Added an explicit baseline-and-reverify discipline to the system prompt and a short memory.md forbidding "completed" while known tests fail or tests are edited in place of source fixes.

The whole search cost about $71.30, every step is logged, and every version can be inspected or rolled back.

With 10 problems per set, a gain of one or two problems can be luck, so the champion should be re-tested on a fresh set before it ships.

Next: confirm the champion, run more generations, and let evolve reach further artifacts such as skills and settings.

Buy the book to learn more

Token by Token

A Journey Through AI, Mathematics, and Machine Intelligence, by Manpreet Singh

tinyorbit, the harness that evolved itself on this page, is built from scratch in Chapter 6. The book walks through the whole stack: how a model becomes an agent, why the harness matters, and the math underneath.

Amazon.com · Paperback Kindle · eBook Amazon.in · Paperback Notion Press · India

11.Technical appendix

Stages: easy safety check (smoke, needs 67%), practice set (train); promotion gate: unseen problems (margin 0.1); size penalty 0.01 per 1,000 chars; 2 islands, migration every 5 attempts; 4×4 niches per island.

Files: archive.json (state), ledger.jsonl (one line per attempt), events.jsonl (every step), iterations/NNNN/ (editor prompt, answer, diff), genomes/ (every version), champion/.

run configuration

{
 "iterations": 5,
 "stages": [
  {
   "name": "smoke",
   "min_score": 0.67
  },
  {
   "name": "train",
   "min_score": 0.0
  }
 ],
 "holdout": "holdout",
 "holdout_margin": 0.1,
 "size_penalty": 0.01,
 "max_genome_chars": 60000,
 "num_islands": 2,
 "migration_interval": 5,
 "feature_bins": 4,
 "rng_seed": 0,
 "budget_usd": 89.0
}

12.References

AlphaEvolve: A coding agent for scientific and algorithmic discovery
Novikov et al., Google DeepMind, 2025. The design evolve follows: an LLM proposes diffs, an evaluator scores them, and an evolutionary database of programs keeps the search going.
OpenEvolve
An open-source implementation of AlphaEvolve. Its island populations, MAP-Elites grid and cascade evaluation are the patterns evolve borrows; here the evolved program is the agent harness.