Rivermark benchmark finds tuned open models competitive on design
Rivermark developer JannVanDam reported preliminary results from 112 social and marketing design requests evaluated through more than 4,700 blind pairwise human judgments. He said DeepSeek v4.1 Flash 1017 and GLM 5.3 Flash, using Rivermark, matched or beat plain Claude Code with Sonnet 5.5 and /design, while Sonnet 5.5 achieved the strongest result with Rivermark. The design-specific harness has been tuned for some models used in production, though not for the benchmark test cases. The author is validating the findings with additional judges.
The report suggests that a specialized, production-tuned harness can let open models achieve similar design results to a frontier-model setup at a fraction of the cost. The author also says these advantages tend to diminish as frontier models improve.
| Hey all! I've been working on a custom agentic harness called Rivermark for social, marketing & print design. Whoever has tried to use LLMs (even frontier ones) for this knows that they can be a bit hit or miss, compared to website and app design, where their skills are much better. The benchmark consisted of 112 requests for various social and marketing images that a business might ask an agent for. Some contain a proper brandkit, some have a website reference for the brand, and some start from nothing. Well, the first results, based on 4700+ pairwise blind human judgments, are in! My headline finding, and one I didn't quite expect, is that open models, such as DeepSeek v4.1 Flash 1017 and GLM 5.3 Flash, match or beat plain Claude Code with Sonnet 5.5 and /design. Sonnet 5.5 itself gets the largest lift with Rivermark - it jumps into an impressive lead over everything else. This was also surprising, as I'd expect advantages to be smaller for "smarter" models, as they have this know-how already encoded. I'm writing a blog post and validating the findings with more judges now, but figured I'd share these interesting preliminary results. In the first photo, you can see Bradley-Terry rankings for these model/harness combos. Second photo is the cost vs quality comparison, and in the third photo you can see some representative examples of what models output with Rivermark vs the normal harnesses. Of course, some very important caveats apply: Open to questions! submitted by /u/JannVanDam [link] [comments] |
#Rivermark#DeepSeek v4.1 Flash 1017#GLM 5.3 Flash#Claude Code#Sonnet 5.5