The full research essay

What LLM Plans Leave Out

I am a Chinese student from a low-income family. In the second half of 2025, I used Gemini 2.5 Pro, one of the leading large language models (LLMs) at the time, to help with Cambridge outreach and application materials. I was approved for one year of joint training as a master's student at the University of Cambridge, with China Scholarship Council (CSC) funding of £1,000 per month.

During visa preparation, I opened a separate bank account for financial evidence. My family pooled the money and had barely raised enough when my grandfather became seriously ill with cancer. We used the money for his treatment. I could not take up the placement.

When an LLM says its plan is feasible, can that plan pay every bill on time?

The planning blind spot

The LLM helped with the application, but my family still had to keep the required money available. That became the starting point for this study. I wanted to know whether an LLM could assemble a complete plan while overlooking a payment the student would be unable to make. A family with little cash to spare has less room to absorb that mistake.

I tested payment timing in a separate synthetic setting. Qwen3.5-4B received tasks in which eligibility and sufficient total annual support were already secured. It had to plan a full year and judge whether its plan could be carried out with the stated cash and payment dates. My experience with Gemini motivated the question. The experiment evaluates Qwen; it does not simulate illness or visa financial-evidence requirements.

The same award, different starting cash

I used one fixed Qwen3.5-4B checkpoint and twelve financial profiles. Each profile received twenty-one starting-cash amounts and two seeds: 504 responses, twenty-four at each cash level. Within a profile, the opportunity, covered tuition, costs and total award stayed fixed. Under the stated rules, arrival documents released 80% of living support; departure documents released the final 20%.

The model submitted a complete plan and judged its feasibility. I checked the choices, prerequisites and dates, then calculated whether actual receipts could pay every bill. I also searched the permitted routes to establish whether any complete route was affordable.

An award is a future resource. A payment requires spendable money on that day.

From a proposed plan to an executable sequence

Qwen3.5-4B · BF16 · native thinking | Molab RTX PRO 6000 Blackwell Server Edition | 504 responses, 3.63 hours, 6,528,329 output tokens

01 / Specify the planning problem

A Chinese-language prompt supplies six office notices: permitted actions, dates, dependencies, costs and payment conditions. The task spans 900 days and includes a full year abroad. Four route families combine paid or waived language certification with travel advances or reimbursement. There is no unlisted borrowing or family top-up. Within each profile, only initial cash B changes.

02 / Generate the plan and its commitment

One native-thinking response per seed and budget: a short explanation, a Boolean feasibility_claim, and an ordered list of action IDs and integer days. The claim applies to the submitted plan through the visit and final settlements. The model receives no executor feedback or repair turn. The saved final answer is evaluated separately from its thinking text.

03 / Execute the submitted sequence

Python first checks required stages, allowed dates, dependencies and mutually exclusive choices. It then replays dated receipts and payments, stopping before the first unpaid bill. An award becomes spendable only when its release conditions have been completed and its receipt date arrives. This separates procedural validity from financial execution.

04 / Search the reference alternatives

A separate finite search enumerates all four supplied route families, all permitted dates and all within-day action orders. For every valid schedule it computes the minimum initial cash. The smallest is R*. Comparing B with R* distinguishes an unaffordable task from a model failure to find an affordable plan. This reference is exact within the supplied action space.

For each valid schedule: R(π) = max(0, −min_k Σ_{j≤k} Δc_j); across permitted valid schedules: R* = min_π R(π).

For a valid schedule π, R(π) is the largest cumulative deficit across its ordered events; Δc is a receipt (positive) or payment (negative). A nonnegative year-end total cannot substitute for a nonnegative balance at every event.

Three measurements, kept separate

Positive claim: a readable true judgment. Valid sequence: all noncash constraints pass. Full execution: the plan completes every required action and payment. Cash false assurance is their decisive mismatch: a positive claim and a valid sequence, but an unpaid bill. It occurred in 97 responses; 26 other positive responses had structural errors and are counted separately.

Inspect the exact experimental setup

BF16, language-model-only inference; native thinking enabled; batch size 12. Context cap 131,072 tokens; output cap 32,768 tokens, including thinking. Temperature 1.0, top-p 0.95, top-k 20, min-p 0, presence penalty 1.5, repetition penalty 1.0. Seeds: 1729 and 2718. All 504 responses ended normally, with no output-cap truncation; 498 contained a readable Boolean judgment.

Twelve synthetic financial variants share one dependency structure. Each is evaluated at 21 common cash levels from RMB20,000 to RMB120,000, with RMB2,500 spacing between RMB35,000 and RMB70,000: 12 × 21 × 2 = 504. Monthly living costs are RMB14,500–16,500. Annual support is paid 80% after arrival confirmation and 20% after departure settlement. All twelve monthly bills remain due on their specified dates.

Runtime: vLLM 0.26.0, PyTorch 2.11.0, Transformers 5.18.0, CUDA build 13.0, NVIDIA driver 595.71.05. PyTorch reports 101,975,851,008 bytes of GPU memory. GPU memory utilization: 0.85; VLLM_USE_FLASHINFER_SAMPLER=0. Checkpoint: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. Generation took 13,084.0199 seconds; loading took 48.2098 seconds.

The scientific figures use descriptive 95% intervals from 5,000 bootstrap resamples of the twelve whole profiles, keeping each profile’s budgets and seeds together (seed 20261006). The payment-timing diagnostic replays the original valid plans without new model calls. These are controlled synthetic results, not an estimated error rate for a population of students.

Follow the same twenty-four answers.

At RMB 50,000 in initial cash.

Twenty-two valid sequences

22 plans met the choices, prerequisites and date rules.

Twenty-one promises

With RMB50,000 at the start, 21 of 24 responses said the year was feasible.

No complete year

None paid every required bill. Valid steps still left a payment without enough available cash.

Figure and interaction: One circle represents one saved response, not one student. The three stages check the same answers separately. Select a stage and a circle on the website to inspect its saved result.

The promise arrived before the cash

Across the study, 311 answers made unconditional positive claims. Ninety-seven had valid choices, prerequisites and dates but failed a payment. Every one of those ninety-seven came from an environment with no affordable full-year route under the original rules.

The weakness concentrated where cash could fund arrival but could not sustain the year. The model correctly rejected all 142 readable answers below the arrival requirement. Where a complete route existed, 194 of 214 responses executed. The concern is a confident promise across the intervening gap.

Figure and interaction: At each saved cash amount, inspect all twenty-four answers. Points use actual cash spacing, and lines connect existing results without smoothing.

Selected saved counts: At RMB55,000, promises/executions were 23/4; at RMB57,500, 22/15; at RMB62,500, 23/22. Every higher-cash exception remains in the full data.

A bill the plan could not pay

One actual response, profile 01 at RMB45,000 with seed 1729, claimed:

“The living-award installments suffice for eleven months of living costs and the later deposit; cash flow has no interruption.”

Translated from the model’s Chinese response. This is an actual model answer, separate from the reference trajectories below.

Its sequence was valid. On day 702, 330 days after arrival, a RMB14,500 living payment met only RMB3,700 in available cash. The shortfall was RMB10,800. A future entitlement could not pay that bill.

The same money, available sooner

For a given route, the cash needed at the start is its largest cumulative gap between required payments and money already received. Eligibility alone does not close that gap. Across these twelve profiles, arrival required RMB39,700–43,900; the full year required RMB50,400–60,200. These are scenario thresholds.

I then used a saved replay that moved the same final 20% to the first 80% receipt date. Every original action, date, cost and funding total stayed fixed. Ninety-five of the ninety-seven cash failures behind positive promises disappeared. Two remained because a bill still preceded the first receipt. Among all 348 valid original plans, completion rose from 194 to 305. This replay changed an institutional payment rule; it did not improve the model or generate new answers.

Follow the saved reference schedules below. The annual living award is RMB 180,000; a monthly living payment is RMB 14,500. These paths are separate from the actual model answer.

Figure and interaction: Compare profile 01 at RMB45,000 and RMB70,000 under the original 80/20 release rule and the earlier final payment. The annual living award is RMB180,000; the monthly living payment is RMB14,500. These four saved reference paths are separate from the actual model answer.

Caption: Scenario days are relative, not calendar dates. Balances change only at saved events. An unpaid bill is never deducted, and a failed path stops before that payment.

Why the payment date matters

Lochner and Monge-Naranjo study how borrowing constraints shape investment in education.^[1] Their model helps explain the gap here: an award paid later cannot cover a bill due today. To plan the year, the LLM needs the payment schedule and the cash already available, as well as the award’s total.

Sen’s capability approach asks what people can actually do with the resources they have.^[2] Students with the same award may face different options if one family can cover the wait and another cannot. Rejecting a student because of a low balance also misses part of the problem: the institution may be able to bring a payment forward.

Deaton studies assets as a buffer against income risk when borrowing is restricted.^[3] His model concerns income risk; my experience adds a reason to consider family care. A planning system should use only the funds the student can commit, leaving any chosen care reserve available to the family.

Moynihan, Herd and Harvey identify learning, compliance and psychological costs in obtaining public support.^[4] For a scholarship recipient, the work may include finding the rule, submitting evidence and waiting for approval. An LLM can help with the paperwork. The institution controls the payment terms that determine how much cash a family must supply in the meantime.

In the randomized H&R Block FAFSA experiment, application assistance combined with personalized aid information improved applications, aid receipt and college attendance; information alone did not improve those outcomes.^[5] That study tested human assistance. The question for LLMs is whether their help gets a student through the process, including the steps that depend on someone else releasing the funds.

Proposed training and evaluation

What I would change in the model

The 97 cash failures point to three things I would change: the training examples, the payment states used in supervision, and the response when the current terms offer no affordable plan. These proposals have not been tested. Their evaluation must count unnecessary rejections as well as false assurances.

01 · Pair examples by payment date

Build paired examples from dated, institution-confirmed terms. Keep academic qualifications, costs and the award fixed; change when the money arrives. Vary starting cash separately. Record all twelve living payments, the deposit and its return, and the paperwork required for release. Include low-cash cases that work through direct payment or an advance, cases with no affordable route, and cases with missing terms. Label the cash after each payment and whether any permitted full-year plan exists. CARL measures sensitivity to constraints by comparing outputs with and without them.^[6] Here, the labels should follow the resulting payment sequence and feasibility.

02 · Supervise the cash available at each step

I would begin with supervised fine-tuning of the existing 4B checkpoint using a small LoRA adapter. The targets would be short event records: required actions, receipts released, cash after each affordable payment, the first shortfall’s date and amount, and the initial cash needed for a complete route. Process-reward-model research shows why a correct final answer can hide faulty intermediate reasoning.^[7] Payment states provide an external check on that reasoning. A later preference-training stage could compare an accurate conditional answer with an unsupported promise and an unnecessary rejection. The final judgment should agree with the numbers. The explanation should identify the bill that matters and what would make it payable.

03 · Specify the funding change needed

At inference, the LLM would extract dated events, amounts, prerequisites and release terms, each linked to its source passage. An accounting function would evaluate candidate plans, and a finite search would compare permitted alternatives. Prior work has combined LLMs with formal solvers for planning and infeasibility diagnosis.^[8] In this study, none of the 97 cash failures had an affordable route under the original terms. The next question is which terms would need to change. For a proposed advance from the existing award, a direct payment or an approved deferral, the system should calculate the amount and latest useful date, then recheck the year. The recommendation must state which institutional approval or missing term it depends on.

A funding office could pay a provider directly, advance part of the existing award, or give the student a confirmed payment date. Earlier disbursement shifts financing and settlement risk to the institution, even when the award’s total stays the same. The calculation should use only the money the student makes available. It should not assume another family contribution, unconfirmed borrowing, or money set aside for care.

How I would implement and evaluate it

A small-model study with two-sided error measurement

A feasible first implementation. Keep the pinned Qwen3.5-4B checkpoint, use BF16 and one rank-8 or rank-16 LoRA adapter, and supervise short event records plus concise explanations. For a valid schedule p, let Pₚ(t) be cumulative obligations and Aₚ(t) conditional receipts already released.

R(p) = maxₜ [Pₚ(t) − Aₚ(t)]₊; R* = minₚ R(p).

The global minimum requires an exhaustive search of known, finite options. Incomplete rules or unfinished search require an unresolved, conditional answer. Projected deficits diagnose failure; actual execution still stops before an unpaid bill. An exact accounting function supplies training labels and inference feedback; no learned reward model is needed initially.

Separate missing evidence from known failure. Task-dependent calibration research distinguishes reasoning uncertainty from uncertainty in locating evidence.^[9] Keep interpretation uncertainty visible even when the arithmetic is exact. Evaluate chosen reserves and hypothetical delays without inventing illness probabilities.

Evaluate both harms. Compare the base model, base plus accounting, adapter plus accounting, and an optional preference-trained adapter. Hold out whole profiles and funding-rule families; test unseen delays, equal-total timing pairs, currency rescaling and equivalent wording. Small-model planning research reports transfer and representation failures, so training success alone is insufficient.^[10] Report false assurance, rejection despite a verified feasible route, payment-state error, first-shortfall accuracy and correctness of conditional changes. Ask affected students to assess usefulness and the family burden of proposed options.

Room for education and family care

My family had already stretched to raise the money needed for financial evidence. When my grandfather became ill, that money paid for his treatment. A useful planner should work with the resources a family can actually spare and identify exactly where further institutional support is needed.

I want students to be able to pursue an education and still have room to care for their families.

Materials and methods

Public research collection on Hugging Face: evaluation data, original model responses, code and the complete development history.

One pinned model, twelve synthetic profiles, two seeds and all twenty-one cash amounts. These results describe this grid; they are not a real-student failure rate or a poverty threshold.

Inspect every answer

The website retains all 504 saved results. Each outcome cell represents one output. The matrix includes negative answers, high-cash failures and four executable plans the model judged false. A negative answer is not automatically an error just because a reference route exists. The matrix uses equally spaced cash categories; the curve uses actual cash spacing.

How the study was checked

This page uses saved results from one pinned Qwen3.5-4B checkpoint (revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a), with native thinking and BF16 inference.^[11] Twelve synthetic profiles were evaluated at twenty-one common starting-cash levels, RMB20,000–120,000, using seeds 1729 and 2718. All 504 outputs remain in the data. Each budget has twenty-four responses. Curves preserve actual budget spacing; the output matrix uses discrete columns.

A deterministic checker evaluated submitted actions, prerequisites, dates and receipts. A separate reference search established whether any permitted full-year route could execute. Cash false assurance required an unconditional positive claim, a valid sequence and a failed payment. Of 498 readable Boolean answers, 311 were positive; six outputs lacked a readable judgment. Four negative judgments whose own plans executed are retained, alongside structural errors and higher-cash exceptions.

The timing diagnostic reused original valid plans and moved only the final 20% to the original 80% receipt date. No new model responses were generated for that change. A failed attempted bill was not deducted from the balance. The four profile-01 reference trajectories are not model-generated plans.

This synthetic-grid study describes one model and twelve financial profiles. It estimates neither a real-student error rate nor a universal poverty threshold. The illness and financial-evidence conditions from my experience were not simulated, and the synthetic annual award is separate from my historical CSC award. The proposed improvements above are separate from these completed measurements.

The theoretical framing interprets one financial condition, not a student’s complete capabilities. Deaton models income risk; extending the insight to family care is this essay’s interpretation. The FAFSA experiment tested human assistance in a U.S. programme, rather than an LLM or changed scholarship release dates.

This is confirmation after disclosed adaptive development, not an unchanged replication. The report preserves the earlier task, clarifications, synthetic-unit confirmation and pilot. Auxiliary final-text review by six GPT-6.1-sol MAX agents interprets commitment wording; original scores and plans are unchanged. This is not human gold-standard annotation.

The scientific figure uses existing descriptive intervals that resample twelve whole profiles while retaining their budgets and seeds. They do not treat the 504 outputs as independent observations. All scientific calculations and replays preceded this essay; page controls only look up saved results.

References and source context

The academic works support the interpretations and design mechanisms stated above; their authors did not test or endorse this proposal. Cost scales and distinct payment examples were checked for the report on 1 October 2026.^[12][13][14] They inform the synthetic scenario and do not reconstruct my historical funding conditions. The report also draws on long-horizon planning and constraint-sensitive scheduling as evaluation background.^[15][16]

1. Lance J. Lochner and Alexander Monge-Naranjo (2011). The Nature of Credit Constraints and Human Capital. American Economic Review 101(6), 2487–2529. Publisher DOI ↩

2. Amartya Sen (1980). Equality of What?. The Tanner Lectures on Human Values, vol. 1, 197–220; lecture delivered 22 May 1979. Original lecture ↩

3. Angus Deaton (1991). Saving and Liquidity Constraints. Econometrica 59(5), 1221–1248. Publisher DOI ↩

4. Donald Moynihan, Pamela Herd and Hope Harvey (2015). Administrative Burden: Learning, Psychological, and Compliance Costs in Citizen-State Interactions. Journal of Public Administration Research and Theory 25(1), 43–69. Publisher DOI ↩

5. Eric P. Bettinger, Bridget Terry Long, Philip Oreopoulos and Lisa Sanbonmatsu (2012). The Role of Application Assistance and Information in College Decisions: Results from the H&R Block FAFSA Experiment. Quarterly Journal of Economics 127(3), 1205–1242. Publisher DOI ↩

6. Qiuyi Qi, Jinjian Zhang, Mutian Bao, Tian Liang, Guocong Li, Dongnan Liu, Wei Zhou, Jie Liu, Ming Kong, Linjian Mo, Feng Zhang and Qiang Zhu (2026). CARL: Constraint-Aware Reinforcement Learning for Planning with LLMs. Findings of ACL. Official proceedings DOI ↩

7. Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou and Junyang Lin (2025). The Lessons of Developing Process Reward Models in Mathematical Reasoning. Findings of ACL. Official proceedings DOI ↩

8. Yilun Hao, Yongchao Chen, Yang Zhang and Chuchu Fan (2025). Large Language Models Can Solve Real-World Planning Rigorously with Formal Verification Tools. NAACL. Official proceedings DOI ↩

9. Chaeyun Jang, Moonseok Choi, Yegon Kim, Seungyoo Lee, Juho Lee and Hyungi Lee (2026). Confidence is Not Universal: Task-Dependent Calibration and Emergent Behavior in LLMs. ICML; PMLR 306:50548–50571. Official proceedings ↩

10. Valerio Belcamino, Nicholas Attolino, Alessio Capitanelli and Fulvio Mastrogiovanni (2026). On the Generalization Gap in LLM Planning: Tests and Verifier-Reward RL. arXiv preprint; conference acceptance not verified. Official preprint ↩

11. Qwen Team (2026). Qwen3.5-4B model card and evaluated checkpoint. Revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. Pinned official model repository ↩

12. University of Cambridge (2026). Maintenance: postgraduate living costs. Official institutional page; checked for the report on 1 October 2026. Official cost reference ↩

13. University College London (2026). Turing Scheme at UCL. Official institutional page; checked for the report on 1 October 2026. Official payment reference ↩

14. University of Manchester (2026). Turing Scheme. Official institutional page; checked for the report on 1 October 2026. Official payment reference ↩

15. Yinger Zhang, Shutong Jiang, Renhao Li, Jianhong Tu, Yang Su, Lianghao Deng, Xudong Guo, Chenxu Lv and Junyang Lin (2026). DeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable Constraints. arXiv preprint. Official preprint ↩

16. Shrenil Shaun Sharma and Avi Sharma (2026). SCHEDBench: A Benchmark for Evaluating LLM Constraint Faithfulness in Natural-Language Combinatorial Scheduling. arXiv version 2, updated 26 August 2026; preprint version cited. Official preprint ↩