Priority was never in the text
I was about to build a feature that inferred the priority of my open items with an on-device model. Before writing it I measured it against 67 items I had labelled by hand myself. The agreement with my own judgement was zero.
I have kept a work log for years. One Markdown file per project where I record what happened: finished this, pushed the PR, waiting on review, we made this decision for that reason. It is not a task manager. It is memory.
This year I started building an app to read it, and the plan for the next milestone said something that sounded reasonable: use an on-device model to infer the priority and status of each open item from its text. Apple has shipped FoundationModels since macOS 26, it runs locally, it sends nothing to any server. A perfect fit for a product whose whole promise is that your files never leave your Mac.
Before writing that feature I measured it. The result was zero, and it is the most useful result I have had in months.
A labelled dataset, free, inside my own history
Until recently my log had a ## Pendientes section in every project, and every top-level item carried a hand-applied priority emoji: 🔴, 🟡, 🟢. When I migrated the format to an event log, those emoji went away.
And there is the detail that makes the experiment possible: the commit before the migration is a labelled dataset. Sixty-seven open items whose priority I had assigned myself, by my own judgement, at the moment I wrote them. No annotators to hire, no consensus to negotiate. The ground truth is my own judgement, which is exactly what the feature would have to reproduce.
The whole log at the time of the test: 438 events across 27 projects.
The method
Apple’s SystemLanguageModel, with guided generation constrained to the three labels, temperature 0, a fresh session per item so no answer could contaminate the next.
I tried two prompts. The first simply describes the three levels. The second is calibrated: I give it the real frequencies of my labels plus two examples per class, taken from items held out of the evaluation.
And I scored it with three numbers: accuracy, the majority-class baseline, and Cohen’s kappa.
That third number is the one that matters, and you are about to see why.
The results
| accuracy | baseline | kappa | |
|---|---|---|---|
| plain prompt (n=67) | 32% | 46% | 0.04 |
| calibrated prompt (n=61) | 49% | 47% | 0.03 |
Kappa measures agreement above chance. Zero is chance. One is perfect agreement. A 0.03 is not “weak”: it is none.
Look at the trap in accuracy. The calibrated prompt climbs from 32% to 49% and looks like it improved by half. It improved nothing. What it did was learn the distribution: it answered thisWeek in 56 of 61 cases. It scored well because that is the most common label, not because it understood a single item. Of the three urgent items in the set, it caught zero.
The plain prompt failed the other way: it predicted urgent 28 times when only 5 were. One shouts and the other shrugs, and both have exactly the same grasp of the problem, which is none.
That is why you report kappa and not accuracy. A model that always answers the same thing can score 49% and will score kappa 0.
Why it was impossible from the start
The explanation is not that the model is small, or that the prompt could have been better. It is simpler and more uncomfortable:
Priority was never in the text.
That 🔴 did not describe the item. It described a decision I made, on a Tuesday, knowing things I never wrote down: that the client was calling on Thursday, that the PR was blocking someone else, that I had three days of travel ahead. The text says “fix the validation error in the payments endpoint”. The urgency lived in my head and in my calendar, not in that sentence.
No model recovers what nobody wrote. It makes no difference whether it runs locally or in a datacenter.
I think this is the easiest mistake to make with LLMs right now, and I made it: I had rich-looking data, in natural language, with plenty of apparent signal. Asking a model to extract a field from it feels like extraction. It is generation. The model will always answer, confidently and in the right shape, and without measurement you have no way to tell recovering a fact from manufacturing one.
The second finding, which hurt more
Something I was not looking for came out of the same data.
The ## Pendientes sections across my whole history held 69 open items and zero completed ones.
Zero. Never, in any format, did I record that something finished. When I completed an item I deleted it from the list. That is the natural thing to do with a list: cross it out, tidy up, move on. But it means my work log, the file that exists precisely to remember what I did, contained not one single closure.
And that has a consequence no amount of better software fixes: “what is still open” cannot be reconstructed backwards. The information is not incomplete, it does not exist. It can only be known from now on, and only if closing something becomes an event that gets written, rather than a line that gets deleted.
What changed in the product
Three things, all in the direction of building less.
Priority is out. Not “later”, but measured and dropped. If I ever want it I will have to write it down, and whether it is worth writing is a product question nobody has answered yes to yet.
Closing became an event. Instead of a box you tick, closing something means recording the fact that it closed. Open work is derived from that. It is the only way the question still has an answer a year from now.
“What am I waiting for” I did not build. Detecting waiting in free text is the same mistake in a different costume. Searching for “waiting” or “pending” looks deterministic and is a heuristic in disguise, the kind that fails silently exactly when you trust it. My own log says “still need to create the PR” about work that is waiting on nobody but me.
What I take from it
The cheapest feature is the one you measure before building. This experiment was a few hours, an extraction script and a hundred-line classifier. The feature would have been weeks, and worse: it would have worked. It would have shown priorities, colour-coded and everything, and I would have trusted them for months before noticing they were well-presented noise.
And the general version, which applies to anything with AI bolted on right now: a model over data nobody wrote down is guessing. The question to ask before wiring it up is not whether the model is good enough. It is whether the fact ever existed at all.
In my case it did not. Better to learn that from a one-day script than from a feature in production.
The full experiment — the method, the numbers and the code to reproduce it — lives in the app’s repository. The app is called Noonly, it is for macOS, it keeps everything in Markdown you own and sends nothing anywhere. It is published at infante.io/noonly.
Comments