asdr83

I've been using LLMs to produce most of my code for the past several months. Over the past two weeks, I've been shifting gears and playing with using them for a bit of writing. Here's how that went.

Experiment 1: Slides

The context was a talk I was giving at work -- practical nuts-and-bolts advice for the engineers, on how to use LLMs:

• Effectively -- get the right results
• Efficiently -- within our corporate-constrained token budget (and generally being more environmentally responsible)
• Responsibly -- avoiding the multitude of risks that LLM-based programming introduces

By the beginning of last week, I had a detailed outline -- fully 250 bullet points that I wanted to cover.

At that point, I looked at the idea of converting that into a hundred or so slides (yes, I know, but I whip through slides fast) and groaned at the prospect of that busywork.

But then I realized: yes, we theoretically have tools for that now, and this is the perfect time to kick those tires. So I pointed Claude Sonnet at some of my previous slide decks and the outline, and said essentially (in enormous detail, as usual): "Make this like those".

The results were... okay. Presentable, but the style details were wrong: in particular too much progressive-reveal on the slides, of things that should have been in the Speaker's Notes instead. I had to spend several hours editing it, to beat it into shape.

That said, it still saved me several hours overall, even for this crude first attempt, and the content was all still quite faithfully what I had said in my outline. (Often too much so: many of those bullet points were much too wordy for slides, and needed tightening up.)

So it was a mild win, if not a home run, and this is where my usual coding practices came into play: I gently scolded Claude for the things it had done wrong, and asked it to write up some improved rules as a skill, so that we can iterate and do better next time.

Experiment 2: Article

Which brought me around to the second half of the task. Besides a recorded talk, I also wanted all of this stuff to exist in our company intranet as an article, for folks who prefer reading to watching.

So it was time for the Truly Disturbing Experiment: what would happen if I tried using this thing for writing?

This wasn't serious writing, mind -- this essay is titled "Utilitarian Writing" for a reason -- this was purely informative stuff. But still, I care about what I sound like even in those circumstances. Could I make the LLM sound like me?

So I pointed Claude Opus at an extended sample of my technical writing (specifically my essay series A Philosophy of Testing) and told it to ingest that to get a sense of my writing style. Then I pointed it at exactly the same outline and told it (in enormous detail, as usual): "Make this like those (and oh, yes, improve the messy organization)".

Which is where things moved into "Truly Disturbing", because the results were actually fairly good. It didn't sound exactly like me, and it needed a moderate amount of editing. But it captured my style adequately, and even added in some appropriate parentheticals and metaphors that I hadn't specified but which fit reasonably well.

Overall, it was a solid first draft, no worse than I often produce when I'm slamming out text, and again saved me several hours. So I once again told it to write this up as a skill for future use.

Some Conclusions

Does this mean we should just surrender to the machines, and have them do all the writing? No, that's very much not the point, but there's a lot of nuance here.

First, let's again stress that this is all about utilitarian writing, where the content is what we care about, more than the voice. LLMs are honestly fairly decent at that. But they don't produce sparkling prose, and they're no substitute for a real writer when it comes to writing good text. IMO, they're a reasonable tool specifically for cases where you're just shooting for "adequate", not better than that.

Second, I got good results because I followed exactly the same processes I do for using LLMs for code. That has many implications, including:

Vibe-writing is as dumb as vibe-coding. When I say "vibe-coding", I specifically mean giving the LLM a relatively brief prompt to do something big, and have it figure out the details. That's fast and fun for prototyping, but utterly irresponsible for serious professional work. The same is true for writing. I got good results precisely because I handed it a massively-detailed outline, so it was mostly stitching things together.

Don't use an LLM when it's faster to do it yourself. Some folks get weirdly attached to doing everything with the AI, and that's counter-productive. If it's easier to just write it by hand, do that. (As in the case of this article.) Reserve the LLM for times when it's going to meaningfully speed you up.

Be explicit. No, more explicit than that. When I am starting a pull request (a unit of code, essentially), I will usually hand the LLM a wall of text, often half a page, making very clear what I want to see as the results. And I gradually build up skills for the LLM, giving it the common rules. That produces vastly better outcomes on average. Less obviously, it is usually far faster and more efficient, since the LLM isn't spending gobs of energy trying (and often failing) to figure out those details itself.

You own the end result. At work, I'm very clear that you are expected to review the output of the LLM in deep detail before you even open the pull request; it's irresponsible not to do so. The same is true here: you should expect the output to require a serious edit pass, and you should not skimp on that.

The major summary is: don't expect miracles, and it's no substitute for the human touch, but it's a useful tool for those of us who aren't writing professionals, and just want to be able to create some text more easily.

No, this article wasn't written that way. But I do expect to sometimes do that here, mainly for posts where I want to start with a detailed outline and then slam out a first draft -- for situations like that, the LLM is likely to work as a decent, quick "ghostwriter" to collaborate with.