AI saved me 10 minutes on a blog post: Was it worth it?
While AI technically saved me time, it made the work harder and less interesting
Not many people know I have a masters degree in neuroscience. I don't use most of what I learned in grad school in my day-to-day job now. My scientific training has ultimately served me well though because it taught me the value of conducting experiments and challenging assumptions.
The experiment I'm writing about here was, in all honesty, a far-from-thorough experiment since it only involved one subject (me), one AI product (Claude), and an activity that really isn't as standard as people creating large language models think it is (writing a specific type of blog post).
Give me a huge chunk of research funding and maybe I'll come back to you with a better experiment. I still think doing this one had value though because it made me realize that objective time savings really shouldn't be the only factor in deciding whether or not to use AI tools in your work.
The goal: Creating the Wagtail 7.2 blog post
One of my responsibilities for each Wagtail release is to write a blog post that talks about our new features in a way that appeals to both developers and people who wouldn't know a line of Python code if it poked them in the eye. I've written before about how AI was moderately helpful with speeding up some of my coding work for Wagtail.org. As someone on a small, open-source product team and who works at an employee-owned company where time savings directly benefit us, I feel like it's important to explore whether these tools can save time on marketing and communications tasks as well.
I have a pretty extensive backlog of release blog posts now, and I've kept the same structure and style since I first started writing them a few years ago. So, I thought, why not feed this backlog into an AI tool along with a copy of our Wagtail 7.2 release notes and see if it could do a halfway decent job of generating a new blog post for me?
The tools I used
In my other previous not-so-rigorous AI experiment, I definitely developed an affinity for Claude. I chose Claude again this time, mostly because it's the tool that I have a license for. Also, when I attended Jono Bacon's Community Leadership Summit, the Claude Projects feature kept getting praised by other people for generating blog posts that mimic their writing style well.
I will confess I first tried this approach out in an adhoc way for the Wagtail Space 2025 recap blog post. I fed Claude as many of my favorite Wagtail blog posts as I could and then I recorded a short video describing the highlights of the conference and had Claude turn the video transcript into a post. The copy it returned wasn't bad. It was definitely editable. I undid all the time savings I got from that experiment though by accidentally overwriting my final edited copy before I saved it, so that wasn't the productivity victory I was hoping it would be.
For this experiment, I generated a version of the blog post with Claude and then edited it. Then I wrote a version of the blog post like I normally do. I timed both workflows with a stopwatch to get an idea how long each version of the process took me. This time around, I didn't time activities that applied to both approaches, such as creating the screenshots for the post and doing deeper research on a section of the release notes that confused me. I did include the time it took for me to add both posts to Wagtail though and fill out all the fields necessary for publishing the blog, and I did count the time it took for me to format the text and to copy text over from Claude into Wagtail. I put the Claude copy in a Google Doc as well so I could make editing comments like I normally do when I'm working on revising writing created by someone else (or something else in this case).
The AI-based process was 10 minutes faster (maybe)
The time I recorded for the generate-and-edit approach was approximately 110 minutes. The time I recorded for my traditional write-from-scratch approach was 120 minutes. These, I found out, were imperfect approaches because it turns out that it's pretty much impossible to write the same blog post two times in a row without you reusing some of the writing that you've created the first time around.
What I found with the Claude-generated text was that I was going to have to rewrite whole sections of the post anyway. So, I switched to writing the post from scratch and then I tried to replicate the rewrite process in the Claude-generated post by typing out everything I had rewritten rather than just copying and pasting everything over. Retyping something you've already written is not a great substitute for the actual thought process of writing it in the first place. It's faster. So, the time I got for the AI-generated post probably doesn't accurately reflect how long the editing process would have actually taken.
If I ever try this experiment again, I think I would leave a gap of at least a couple weeks between the attempts to let the words leak back out of my brain first. That would ultimately be a fairer and more objective comparison.
There were ultimately other reasons though that I didn't find a time savings of 10 minutes particularly compelling.
The machine failed at mimicking structure and style
The structure of my blog posts is not always consistent but there are enough patterns that a pattern-matching tool like a large language model should have been able to pick up on them. I generally have a header about a feature followed by one or two paragraphs of text about the feature, and then I almost always have a summary of key upgrade considerations at the very end. I almost never use bullet points and I usually won't include contributor attributions unless we're highlighting something special like a Google Summer of Code contribution or a collaboration with a nonprofit partner.
Yet Claude put all of those things in the initial draft despite the backlog of posts I supplied. It was also very verbose. The length was much longer than I consider acceptable. I told Claude to count the words (which it did through a very laborious process of generating a huge pile of JavaScript) and then I asked Claude to get the post down to under 1000 words. Waiting for Claude to finish that editing task was undoubtedly the longest Claude took to respond to a prompt. I could have probably trimmed it down faster myself.
Claude also failed to capture the dryer-than-the-Sahara-in-summer humor style I use for these posts. Perhaps if Claude started hanging out with as many Brits as I do, Claude would be better at subtle humor. Instead, when I prompted Claude to add jokes, we got terrible lines like these:
- We've all been there, updating sort order values one by one like some kind of digital accountant from the 1800s. Digital accountants in the 1800s? Really Claude?
- Finally, you can find out which images have been sitting in your media library since 2015 gathering digital dust. Eh, closer but still a bit meh.
- This is especially helpful for forms split across multiple tabs, where tracking down the source of an error can feel like a treasure hunt, except the treasure is disappointment. I mean this at least made me snort but the treasure hunt setup doesn't really make sense.
A fellow core team member pointed out that what I probably should have done was feed Claude copies of the blog paired with the release notes that I wrote them from so that the original release notes material would be part of the context. I'm not sure if there is a way to pair files that way for Claude though. The machine is always going to look at the overall data collection.
And if Claude couldn't sort out the patterns from the fairly extensive backlog of samples I provided, I'm dubious about how useful it will ever be for creating drafts of our release blogs.
Editing the AI copy felt miserable and frustrating
Even if Claude had perfectly mimicked the style and structure I wanted, there was something I noticed as I was working on fixing the AI-generated copy: I was really annoyed the whole time I was editing it.
When you're a writer and editor, you inevitably wind up essentially rewriting someone else's lazy rough draft because it's the quickest way to meet a deadline. This is the OG workslop and sometimes, a writer will help someone fix their awful draft out of the goodness of their heart because the person is a friend and in a pinch. More often though, professional writers resent that person because they put so little effort into their draft that barely anything they put on the page is useful.
That's how I felt about rewriting the AI-generated copy.
Only, it was worse because there wasn't even a chance of another human potentially learning something from reviewing the edits and rewrites I did. One of the things I enjoy most about editing is seeing people improve their writing and English skills. There are some folks who don't want to learn and just keep sending you terrible copy to fix no matter what. But most people do learn from revisions and seeing them and the overall quality of the work improve is what makes the more miserable assignments worth it.
You can't get the same satisfaction from a large language model. Yes, Claude did thank me when I asked for edits. All it's doing though is accumulating more data points. It's not truly learning. And that made it feel even more pointless than usual to rewrite something I could have written better myself in the first place.
If I've learned nothing else from this experiment, I think it's important to pay attention to how we feel while working with AI tools. Because automating too many of the interesting things out of our days could be a recipe for poorer mental health and skill atrophy if we're not careful.
The tradeoffs for blog posts aren't worth the time saved right now
For the tiny amount of time the AI process (maybe) saved me, it's not worth it right now to turn to Claude for blog post drafts. The quality is too poor and the process of using it was too miserable. When you factor in the high likelihood that AI is more carbon intensive and the exploitive labor practices that went into building the more massive AI products, saving ten minutes seems even less worth it to me.
Still, I'm open to the possibility that another AI tool might be more effective or that there's a better method or approach I could use to get better results than what I did this time. As I said earlier, there are certainly more objective ways I could have set this experiment up. It may also be that these tools are ultimately better at assisting us with other marketing chores, such as summaries or social media posts or image captions. AI, after all, can be great for automating very specific tasks like transcription.
I plan to install Wagtail AI on Wagtail.org so we can do more experiments and find out if there really are any use cases where the time savings from AI automation are worth the tradeoffs. As unimpressed as I am with LLM-generated writing, I'm open to conducting more experiments. I would like to test some different models too and explore whether models with more transparency around climate impacts or the sources of their training data are effective options as well.
You're welcome to look at this Google Doc if you'd like to see the progression of the drafts I generated with Claude. Feel free to compare them with the final blog post and decide whether you prefer the AI copy or mine. Hopefully, you'll agree that keeping a human in the loop is currently the best option — if only to spare you from Claude's terrible idea of a joke.
