Devlog 005: Every new AI model means retraining the apprentice

My routine lately goes something like this: wake up to news that some AI lab shipped a new model, spend the afternoon testing it, and end the day either sighing at the output or quietly celebrating.

AI models are updating at a sprint this year. For most products, swapping models is a one-line config change. For a service that uses AI to interpret Zi Wei Dou Shu charts, every swap feels like hiring a new apprentice — brilliant, sure, but you still have to teach them the house rules from scratch.

Why we can't just blindly upgrade

I used to assume newer meant better: swap it in and the reports improve on their own. A few rounds of testing cured me of that. Every model has its own personality:

• Some are so cautious that everything comes out hedged and beige. The reading ends up sounding like a disclaimer — nothing sticks.
• Some are wonderful storytellers — genuinely great prose — and then they start confidently making things up. Stars that aren't in your chart, described as if they were.
• Some handle English beautifully but improvise freely the moment they hit terms like Hua Ji or San Fang Si Zheng.

For fortune readings, the second type is the dangerous one. A regular article can get away with "that's just an opinion." A chart can't. If Wu Qu sits in your Wealth Palace, it sits in your Wealth Palace — an AI claiming it's in the Spouse Palace is simply wrong, with zero room for interpretation.

How we keep the AI from hallucinating

So most of the recent work boils down to one principle: the AI interprets, it does not invent. The approach isn't complicated to describe:

Chart math and chart reading are separate jobs. None of the chart calculation goes through the AI. Star placements, the Four Transformations, decade and annual cycles — all computed by code first, then handed over as structured data. The AI receives a finished chart; its job is to turn the structure into plain language, not to cast the chart itself.

Write the rules in, then inspect the output. We've built up a whole rulebook of chart knowledge for the AI — which palace governs what, what Hua Ji landing somewhere means, when it must not hand out advice. After a report is generated, another layer checks whether it mentions anything that doesn't exist in the chart. Caught fabricating? It rewrites. Still failing? Your credits are refunded — we won't ship you a made-up reading.

Same charts, head-to-head. Whenever a new model comes out, I run the same batch of test charts through the old and new models and compare, section by section: who reads accurately, who's telling tales, who nails the timing of events. The winner gets the job. The loser stays on the bench.

So why keep chasing new models at all?

Because it's worth it. Each generation genuinely understands chart structure better. Older models could only discuss each palace in isolation; newer ones start to see that several palaces are telling one connected story, and the advice lands much closer to what you actually asked. Costs keep dropping too, which means the same price buys a deeper report.

So for the foreseeable future, the loop keeps running: new model → test → train → head-to-head → promoted or benched. If you rerun a report someday and it reads noticeably sharper, that's probably someone on the bench getting called up.