# Benchmarks — 3D Asset Library
Historical measurements, not a general efficacy or quality-parity claim.
Neither study met its full pre-registered win criteria; some measurements were missing. TF1 did not meet its full criteria; TF2 could not verify its primary outcome. Formal acceptance has not been verified.
Machine-readable data: [benchmarks.json](/benchmarks.json). Full-precision JSON values are authoritative; landing values are explicitly rounded.
## Main limitations
Raw gpt-6.1-sol was preferred to gpt-6-luna + library in blind STANDARD visual judging (see tf2-sol-preference). No quality parity is established. TF1 library arms used more weighted tokens per finished asset; gpt-6-luna defect improvement was not significant. Missing audits and metering prevent TF2 primary verification. Secondary tests have no multiplicity correction.
Queue-inclusive TF1 wall totals include different queue reacquisition behavior, so they do not isolate speed. Reported times are per arm across all attempts, not per finished asset.
No model training or fresh experiment was run for this page. Requested CLI model IDs do not prove provider-returned identity. These experiments do not qualify every family or the currently packaged revision.
## Results
## tf1-sol
mean audit defects per asset; dimensions percent; weighted tokens per finished asset; total wall seconds per arm
Date (report generation): 2026-10-22T05:12:04Z
Arms and requested models: {"FAM-S": ["gpt-6.1-sol"], "RAW-S": ["gpt-6.1-sol"]}
Subset: STANDARD; n = 24 briefs (not independent votes).
Values (full precision): {"library": 0.16666666666666666, "library_agent_seconds": 6926.8, "library_dimensions": 100.0, "library_queue_seconds": 12520.3998336792, "library_tokens": 117075.75, "ratio": 6.250000000000001, "raw": 1.0416666666666667, "raw_agent_seconds": 4366.1, "raw_dimensions": 100.0, "raw_queue_seconds": 27580.0, "raw_tokens": 67798.60833333332, "token_ratio": 1.7268164181836085}
Test: Exact paired one-sided sign flips; ties retained.
p: 0.03125; CI: null
Conditions: Paired briefs, library versus raw Blender code using the same requested model. Dimension tolerance is 3%. Weighted tokens = uncached input + 0.1 cached input + output, all attempts / finished assets.
Limitations: TF1 did not meet its full pre-registered win criteria; formal acceptance has not been verified. Historical import and framing violations retained; same-model comparison under this harness, not general efficacy. Queue-inclusive time includes different harness queue reacquisition; not an isolated speed comparison.
[Source report](/static/benchmark-sources/tf1_report.md). Exact JSON pointers and source hashes are in benchmarks.json.
## tf1-luna
mean audit defects per asset; dimensions percent; weighted tokens per finished asset; total wall seconds per arm
Date (report generation): 2026-10-22T05:12:04Z
Arms and requested models: {"FAM-L": ["gpt-6-luna"], "RAW-L": ["gpt-6-luna"]}
Subset: STANDARD; n = 24 briefs (not independent votes).
Values (full precision): {"library": 1.7916666666666667, "library_agent_seconds": 7436.2, "library_dimensions": 95.83333333333334, "library_queue_seconds": 13841.34836268425, "library_tokens": 127955.14166666666, "ratio": 1.5348837209302324, "raw": 2.75, "raw_agent_seconds": 2341.5, "raw_dimensions": 94.44444444444444, "raw_queue_seconds": 23588.0, "raw_tokens": 61674.49166666667, "token_ratio": 2.0746849825407288}
Test: Exact paired one-sided sign flips; ties retained.
p: 0.284912109375; CI: null
Conditions: Paired briefs, library versus raw Blender code using the same requested model. Dimension tolerance is 3%. Weighted tokens = uncached input + 0.1 cached input + output, all attempts / finished assets.
Limitations: TF1 did not meet its full pre-registered win criteria; formal acceptance has not been verified. Historical import and framing violations retained; same-model comparison under this harness, not general efficacy. Queue-inclusive time includes different harness queue reacquisition; not an isolated speed comparison. gpt-6-luna defect difference is not significant.
[Source report](/static/benchmark-sources/tf1_report.md). Exact JSON pointers and source hashes are in benchmarks.json.
## tf2-luna-defects
mean audit defects per attempted STANDARD brief
Date (report generation): 2026-10-22T23:50:51Z
Arms and requested models: {"FAM2-L": ["gpt-6-luna"], "RAW-L": ["gpt-6-luna"]}
Subset: STANDARD; n = 24 briefs (not independent votes).
Values (full precision): {"library": 6.375, "ratio": 4.496732026143791, "raw": 28.666666666666668}
Test: Descriptive aggregate; no significant paired defect test is established.
p: null; CI: null
Conditions: Frozen external audit: floating, below_ground, clash/interpenetration, gross wrong_scale. Unbuilt completed assets score 36 defects.
Limitations: TF2 could not verify its primary outcome; missing audit/metering and retained protocol failures prevent primary qualification. Secondary comparisons have no multiplicity correction; no quality parity or universal model benefit established. RAW builds from an empty scene; library contains previously refined knowledge. Equal inference budget is not equal lifetime development cost. RAW gpt-6-luna refusals and unbuilt penalties are instruction-handling outcomes, not evidence of geometric inability. Tool-exposure, shortened admission windows and unpinned shared-dependency limitations are retained in the source report.
[Source report](/static/benchmark-sources/tf2_report.md). Exact JSON pointers and source hashes are in benchmarks.json.
## tf2-complex-pairwise
Blind visual preference, gpt-6-luna library vs raw gpt-6-luna
Date (report generation): 2026-10-22T23:50:51Z
Arms and requested models: {"FAM2-L": ["gpt-6-luna"], "RAW-L": ["gpt-6-luna"]}
Subset: COMPLEX; n = 10 briefs (not independent votes).
Values (full precision): {"mean": 0.5}
Test: paired brief-level percentile bootstrap; one-sided 95% lower bound
p: null; CI: {"confidence": "paired brief-level percentile bootstrap; one-sided 95% lower bound", "lower": 0.1, "resamples": 20000, "seed": 20261022}
Conditions: Both orders judged by gpt-6.1-sol; votes averaged within brief. Positive values favor library. Original refined reference shown as target for COMPLEX.
Limitations: TF2 could not verify its primary outcome; missing audit/metering and retained protocol failures prevent primary qualification. Secondary comparisons have no multiplicity correction; no quality parity or universal model benefit established. RAW builds from an empty scene; library contains previously refined knowledge. Equal inference budget is not equal lifetime development cost. RAW gpt-6-luna refusals and unbuilt penalties are instruction-handling outcomes, not evidence of geometric inability. Tool-exposure, shortened admission windows and unpinned shared-dependency limitations are retained in the source report. Positive lower bound on this secondary set; distinct from the failed refinement endpoint.
[Source report](/static/benchmark-sources/tf2_report.md). Exact JSON pointers and source hashes are in benchmarks.json.
## tf2-sol-preference
Blind visual preference, gpt-6-luna library vs raw gpt-6.1-sol
Date (report generation): 2026-10-22T23:50:51Z
Arms and requested models: {"FAM2-L": ["gpt-6-luna"], "RAW-S": ["gpt-6.1-sol"]}
Subset: STANDARD; n = 24 briefs (not independent votes).
Values (full precision): {"mean": -0.625}
Test: paired brief-level percentile bootstrap; one-sided 95% lower bound
p: null; CI: {"confidence": "paired brief-level percentile bootstrap; one-sided 95% lower bound", "lower": -0.8333333333333334, "resamples": 20000, "seed": 20261022}
Conditions: Both orders judged by gpt-6.1-sol; votes averaged within brief. Positive values favor library. Original refined reference shown as target for COMPLEX.
Limitations: TF2 could not verify its primary outcome; missing audit/metering and retained protocol failures prevent primary qualification. Secondary comparisons have no multiplicity correction; no quality parity or universal model benefit established. RAW builds from an empty scene; library contains previously refined knowledge. Equal inference budget is not equal lifetime development cost. RAW gpt-6-luna refusals and unbuilt penalties are instruction-handling outcomes, not evidence of geometric inability. Tool-exposure, shortened admission windows and unpinned shared-dependency limitations are retained in the source report. Raw gpt-6.1-sol is preferred; no quality-parity claim.
[Source report](/static/benchmark-sources/tf2_report.md). Exact JSON pointers and source hashes are in benchmarks.json.
## tf2-complex-cost
published list-price USD per finished COMPLEX asset
Date (report generation): 2026-10-22T23:50:51Z
Arms and requested models: {"FAM2-L": ["gpt-6-luna"], "RAW-S": ["gpt-6.1-sol"]}
Subset: COMPLEX; n = 10 briefs (not independent votes).
Values (full precision): {"library": 0.024381820000000002, "library_finished": 7, "ratio": 9.008664834882893, "raw": 0.21964764444444443, "raw_finished": 9}
Test: Descriptive list-price estimate; no invoice or significance test.
p: null; CI: null
Conditions: Total attempted-arm token cost / finished assets. Published Standard short-context input, cached input and output rates retrieved 2026-10-22.
Limitations: TF2 could not verify its primary outcome; missing audit/metering and retained protocol failures prevent primary qualification. Secondary comparisons have no multiplicity correction; no quality parity or universal model benefit established. RAW builds from an empty scene; library contains previously refined knowledge. Equal inference budget is not equal lifetime development cost. RAW gpt-6-luna refusals and unbuilt penalties are instruction-handling outcomes, not evidence of geometric inability. Tool-exposure, shortened admission windows and unpinned shared-dependency limitations are retained in the source report. Not an invoice or Codex subscription charge; excludes cache writes, tools, hosting and library development compute. Service tier, backend identity and long-context billing unproven; STANDARD priced endpoint remains unavailable.
[Source report](/static/benchmark-sources/tf2_report.md). Exact JSON pointers and source hashes are in benchmarks.json.
## Source evidence
Original paths are provenance references, not public routes. Report copies expand model nicknames to recorded identifiers without changing measurements; public result extracts retain used aggregate fields and omit local per-subject filesystem paths.
- [tf1_report](/static/benchmark-sources/tf1_report.md): tfeature/REPORT.md; report copy with model nicknames expanded to recorded identifiers; measurements unchanged; original SHA-256 `c3607f0c30c6739e9bf614ec2e4935a29477f359b7fcd6a9feabe86fbd568aca`; published SHA-256 `c3607f0c30c6739e9bf614ec2e4935a29477f359b7fcd6a9feabe86fbd568aca`.
- [tf1_results](/static/benchmark-sources/tf1_results.json): tfeature/results.json; aggregate extract; original full file hash differs; original SHA-256 `aaa7f815061b91115ccc7547ed5d7323ec945793429da5997f9d4d17865c64f0`; published SHA-256 `b3e23d9456aee743cd4aebefad0e519079d4fba375f6acc3d2544919acbe5902`.
- [tf2_report](/static/benchmark-sources/tf2_report.md): tfeature/tf2/REPORT.md; report copy with model nicknames expanded to recorded identifiers; measurements unchanged; original SHA-256 `f67ff3459b77a6af8b6466b7b9004391b58024aebe8792bfd969995fb7d39bbf`; published SHA-256 `625eb08bf379d1fc58ff88c4321ad40a41845469bb33228e53fb5da817d87e23`.
- [tf2_results](/static/benchmark-sources/tf2_results.json): tfeature/tf2/results.json; aggregate extract; original full file hash differs; original SHA-256 `76109faeb0e0bdb930b5a8a2bf5eb166a74d067f988ed7440ef94f3ca54fd62a`; published SHA-256 `9cd1e2b9dc101f6c6b450003e70675c7b4beae4a95206c18cc362d4112aca56f`.
## Compute cost
No build-compute figure is published in this section; token estimates exclude development and hosting.
All attempted token cost is charged in the per-finished denominator; failures are not free. No hosting, development-compute or customer-price claim.