Reddit user questions LLMs' ability to write high-quality prose
In an r/LocalLLaMA post, u/kompania said years of local QLoRA and full fine-tuning produced prose that barely exceeded mediocrity. They reported testing models from Gemma, Qwen, Mistral, LFM and IBM, as well as community-built models, and dismissed EQBench as an unreliable measure of literary quality. The author said they generally resort to writing prose themselves and using an LLM only for proofreading, then asked the community whether any local model reaches the quality of celebrated novels.
The post provides a detailed first-person account of the difficulties one practitioner encountered when fine-tuning local models for literary prose across multiple families and methods.
I have been following the evolution of LLMs since the days of GPT-3.5, and I have been actively using local LLMs since the release of Gemma 2.
I read a great deal, including this subreddit. I have performed QLoRA fine-tuning on my local machine many times, and I have also conducted full fine-tuning (FFT) training runs on RunPod.
Hundreds of models-fine-tuned by well-known practitioners in English, Chinese, and Russian-have passed through my drive.
After four years, I arrived at a fundamental question yesterday: are LLMs even capable of writing good prose?
When training models myself, I used both my own prose datasets and those created by others. In the training process, I experimented with various approaches-ranging from 100 epochs of FFT at a learning rate of 1e-7, to aggressive LoRA/QLoRA configurations. The results, however, barely rise above mediocrity.
I have used models from all the leading providers-Gemma, Qwen, Mistral, LFM, IBM-as well as niche attempts by developers building models from scratch.
#Gemma#Qwen#Mistral#LFM#IBM#RunPod#EQBench#GPT-3.5#LoRA#QLoRA