Property Finder Finds Bias in Vision Model Evaluation
Property Finder discovered that 96.8% of its evaluation labels came directly from predictions by its incumbent vision model rather than independent human verification. In an AWS Machine Learning Blog case study, the team rebuilt expert-annotated ground truth and used evolutionary prompt optimization on Amazon Bedrock to select a model with higher accuracy and 86% lower cost.
The case study shows how treating an incumbent model’s predictions as ground truth can bias comparative evaluations against alternative models.
This post was co-authored with Khaled Thabit, Meriem Boughrara, and Mohamed Desouky from Property Finder.
The engineering team at Property Finder, a digital real estate application, discovered that 96.8 percent of their evaluation labels had been inherited directly from a single model’s predictions rather than verified by human experts. Their ground truth was systematically skewed toward the incumbent model, making every alternative appear to underperform. The labels also had errors, but the bias was what blocked objective model comparison. They uncovered this when migrating their vision AI workload to a more cost-effective option on Amazon Bedrock.
Property Finder processes tens of thousands of property images every day across the Middle East and North Africa (MENA). Their system evaluates 11 quality attributes per image. These cover watermark detection, room classification, furnishing status, photo quality, and construction indicators. That’s thousands of new property listings with an average of 15 images each. All need quality assessment before going live.
Property Finder needed a more cost-effective option. But measured against those labels, any alternative was at a disadvantage.
The hardest part was not finding a better model. It was discovering that their evaluation methodology itself was the blocker.
#Property Finder#AWS#Amazon Bedrock#AWS Generative AI Innovation Center#DSPy#MIPROv2#Llama 4 Maverick 17B Instruct#Meta