iTranslated by AI

The content below is an AI-generated translation. This is an experimental feature, and may contain errors. View original article
📋

Experimental Protocol: How Much Does Input Token Order Affect Reasoning Depth?

に公開
AI
Python
研究
llm
進化計算
tech

- A Super-Mini Evolutionary Experiment via "Individuation" of Arrangement -

Note: This article is an Experimental Plan.
Although the results section contains placeholders, it is published in a format where anyone can replicate the experiment under the same conditions.

0. Positioning of the Experiment (within the series)

This experiment corresponds to the experimental phase of the following articles:

  • Planning Phase ①:
    "How much does input token arrangement change the depth of reasoning?"
  • Planning Phase ②:
    Introduction of arrangement evolution and IWR measurement from an EDiE perspective

In this article, we will conduct a minimal-unit evolutionary experiment of
"treating the arrangement itself as an individual and subjecting it to selection."

1. Experimental Purpose

Primary Purpose

To quantitatively measure how much the arrangement order of input tokens (semantic chunks) affects the depth of AI reasoning (IWR).

Secondary Purpose

  • To see if "strong sequences" different from human intuition emerge by
    "selecting" rather than "designing" the arrangement
  • To demonstrate that differences in structure alone produce differences even without GPUs or training

2. Experimental Hypothesis (Provisional)

  • Even with identical content and the same seed, IWR depth changes just by changing the arrangement order
  • "Seemingly good" arrangements do not necessarily survive
  • IWR measures
    "interaction with the environment," not model performance

3. Requirements (Minimum Configuration)

Hardware

  • PC (Windows / Mac / Linux all supported)
  • No GPU required

Software

  • Python 3.10+
  • VS Code (Recommended)
  • OpenAI API (Requires payment; minimum usage is sufficient)

4. Experimental Subject (Definition of an Individual)

Individual = Context Arrangement Pattern

Divide the identical content into the following semantic chunks:

  • [Task] Purpose
  • [Constraints] Constraints
  • [Facts] Prerequisite Information
  • [Examples] Examples
  • [Noise] Noise

Changing the order of these
shall be defined as
one individual (Genome).

5. Experimental Conditions (Fixed)

  • Model: (Please fill in, e.g., gpt-4.1 / gpt-4.1-mini)

  • Number of seeds: (Please fill in, e.g., 3)

  • Maximum Why depth K: (Please fill in, e.g., 10)

  • Temperature / top_p: Fixed

  • Evaluation method:

    • Primary judgment: AI (judge)
    • Final judgment: Human (CSV verification)

6. Experimental Procedure (Checklist)

STEP 1 | Create Base Context

Prepare contexts/context_base.txt
(Content is identical, only the order changes)

STEP 2 | Generate Arrangement Patterns

Create multiple patterns as follows:

  • Pattern A
    [Task][Constraints][Facts][Examples][Noise]

  • Pattern B
    [Facts][Task][Constraints][Examples][Noise]

  • Pattern C
    [Task][Facts][Examples][Constraints][Noise]

  • Pattern D
    ... (Add as desired)

  • Do not change the content at all

STEP 3 | Measure IWR with Identical Seed

  • For each arrangement pattern:

    • Same seeds_ja.txt
    • Same settings
    • Same K

  • Execute run_iwr.py

STEP 4 | Save Results

  • Raw logs (JSONL)
  • Summary (CSV)

STEP 5 | Selection

  • Keep only the top-tier by IWR depth
  • Verify failure types as well

7. Fill-in-the-Blank Experimental Results Sheet (for articles)

Execution Information

  • Execution Date:
  • Model Used:
  • K:
  • Number of seeds:

Results by Arrangement Pattern

Pattern Arrangement Order Average IWR Major Failures
A
B
C

Observation Notes

  • Match/Mismatch with human predictions:
  • Surprisingly strong arrangements:
  • Impact of noise position:

8. Discussion (Provisional)

  • The "depth" of reasoning is not determined by the model alone
  • Input structure is the environment itself
  • The "correctness" designed by humans does not necessarily survive

9. Next Move (Confirmed)

Next Time: "Evolving" the Arrangement

The tasks are already determined.

  • Make the winning arrangements the parent individuals
  • Mutations via swap
  • Generate the next generation
  • Re-evaluate with the same IWR

And here,
a sense of unease should arise.

"Not just the arrangement,
can't the evaluation axes themselves
also be evolved?"

10. Preview | To AXI

From the next article onwards,

  • Individuating IWR
  • Evaluation metrics being selected
  • Evaluation begetting evaluation

We enter the realm of AXI (Axis Evolution).

By this point,
it is no longer just a discussion about token arrangement.

Evolutionary Discovery Engineering begins to demonstrate its true potential.

Appendix | List of Source Code Used in This Experiment

  • This experiment is conducted using the same folder structure, same seed, and same environment as the previous article.
  • For redundant files, it is clearly stated as "same as previous."

Folder Structure (Reiterated)

iwr_semi_auto_v0/
  README.md
  requirements.txt
  run_config.json
  seeds_ja.txt
  contexts/
    context_base.txt
    context_pattern_A.txt
    context_pattern_B.txt
    ...
  outputs/
    .gitkeep
  src/
    run_iwr.py
    llm_openai.py
    judge.py
    utils.py

Common Files (Same as previous article)

The following are identical to the previous articles ① and ②. No content changes.

  • requirements.txt

  • run_config.json

  • seeds_ja.txt

  • src/llm_openai.py

  • src/judge.py

  • src/utils.py

  • API key settings, virtual environment, and execution methods are also the same as before.

Files Added/Changed in This Experiment

1. contexts (Arrangement Individual Groups)

contexts/context_base.txt

  • Content is the same as before
  • Semantic chunks are fixed
  • Only the order is changed

Example: context_pattern_A.txt 

[Task]
You are an expert in causal explanation.

[Constraints]
It is acceptable to say unknown if it is unknown. Correct any contradictions.

[Facts]
This experiment is for log acquisition for IWR measurement.

[Examples]
(Include examples if necessary)

[Noise]
(Text with weak relevance)

Example: context_pattern_B.txt 

[Facts]
This experiment is for log acquisition for IWR measurement.

[Task]
You are an expert in causal explanation.

[Constraints]
It is acceptable to say unknown if it is unknown. Correct any contradictions.

[Examples]
(Include examples if necessary)

[Noise]
(Text with weak relevance)
  • The content is completely identical; change only the order.

2. Execution Code (Same as previous)

src/run_iwr.py

  • Same as before
  • Just replace context_base.txt with the target pattern file before execution
python -m src.run_iwr

Output Files

  • outputs/raw_YYYYMMDD_HHMMSS.jsonl

  • outputs/summary_YYYYMMDD_HHMMSS.csv

  • Differentiate by filename for each arrangement pattern

  • Reuse in subsequent selection, comparison, and evolutionary experiments

Minimum Checklist for Experiment (Reiterated)

  • Is the content completely matched?
  • Is only the order different?
  • Is the seed the same?
  • Is K the same?
  • Is the model configuration fixed?
  • Is the output CSV saved?

Supplementary Notes (Homework List)

The following terms are intentionally not explained in detail in the body.

  • IWR

  • Failure types (stagnation / drift / circular)

  • Individual (Genome)

  • Selection pressure

  • Evolution of evaluation axes (AXI)

  • To be addressed in subsequent articles.

Discussion