Property Finder discovered that 96.8% of the labels it used to compare AI vision models came from the predictions of the model it was trying to replace. AWS says the company rebuilt its evaluation around fresh expert annotations, and found a cheaper model that scored better on its property-image tasks.
AWS AI Watch analysis
What happened
The evaluation covered 11 attributes in property photos, including room classification, watermark detection and image quality. AWS says a real-estate expert independently annotated 356 images from the UAE, without seeing the incumbent model’s answers. The team also allowed multiple labels where a photo could genuinely show more than one room type.
On that evaluation, AWS says Llama 4 Maverick 17B Instruct scored 0.80 Micro F1, compared with 0.56 for Property Finder’s proprietary model. AWS reports that the selected model delivered higher accuracy at 86% lower cost. The comparison was not perfectly even: the Llama model received several prompt-tuning rounds, while the proprietary model received one. The evaluation set also included images used during development, so AWS says the scores are optimistic.
Why it matters
An evaluation can look rigorous and still quietly favour the model that supplied its answer key. If alternatives are judged against an incumbent’s predictions rather than independently checked ground truth, a team may mistake familiarity for accuracy and reject a better, cheaper option.
The practical fix is straightforward, if not effortless: have a domain expert label a representative sample independently, define how ambiguous cases should be handled, and keep development examples distinct from the final evaluation where possible. Property Finder’s test was modest in size, and AWS cautions that rare categories may need more examples to reveal smaller differences.
Our read
The most valuable finding may be the awkward one: before asking which model wins, check who wrote the answer key. A benchmark built from yesterday’s model can turn a model migration into a referendum on yesterday’s habits.
The reported results favour Llama in this workload, but the unequal tuning effort and overlap between development and evaluation data matter. Treat the numbers as a useful case study, not a universal model ranking.
What to watch
- Whether Property Finder reports results on a separate, untouched test set.
- How the selected model performs on rare attributes and across regions beyond the UAE.
- Whether other teams audit labels inherited from their existing models.
Discussion spark: When comparing AI models, how much independent human labelling is enough to trust the result, especially when the incumbent has already generated most of the dataset’s labels?
Sources and evidence
- Fix evaluation bias to unlock better models on Amazon Bedrock (27 October 2026, 16:53 UTC)
- Exowatt beats frontier models on accuracy with SageMaker serverless model customization (27 October 2026, 16:53 UTC)
not affiliated with or endorsed by Amazon Web Services (AWS)