人工智能
固定评价偏差，在 Amazon Bedrock 上解锁更好的模型
这篇文章由Khaled Tabit、Meriem Boughrara和来自Property Finder的Mohamed Desouky共同撰写。
Property Finder的工程团队是一个数字房地产应用公司，发现其96.8%的评价标签是直接从一个模型的预测中继承的，而不是由人类专家核实的。 他们的真相被系统地扭曲到现行模式，使每一种选择似乎都表现不佳。 标签也有错误，但偏差是阻碍客观模型比较的原因。 当他们将视觉AI的工作量转移到Amazon Bedrock上更具成本效益的选择时，他们发现了这一点。
财产查找器每天在中东和北非处理数以万计的财产图像。 他们的系统每个图像评价11个质量属性。 这些包括水印检测、房间分类、家具状况、照片质量和建筑指标。 数千个新房产上市，所有人都需要质量评估后才能上线。
物业查找者需要更具成本效益的选择。 但是，根据这些标签来衡量，任何替代办法都处于不利地位。
最难的是找不到更好的模式。 它发现，它们的评价方法本身就是阻碍因素。
在本文中，我们分享Property Finder如何识别评价偏差，构建无偏见的地面真相，并应用进化即时优化在Amazon Bedrock上选择正确的模型，以86%的成本实现更高的精度。 我们走过方法和重复的例子，与教训你可以应用到你自己的工作量。
下图说明了Property Finder和AWS Generative AI Innovation Center团队在Amazon Bedrock上用于迭代提高分类精度的快速优化循环。
图显示迭代快速优化周期。 一个属性图像由Amazon Bedrock处理，它以JSON格式输出预测属性。 在优化期间，这些预测与Micro F1对256图像专家附加说明的开发集进行打分。 错误分析然后检查失败的图像并识别视觉模式，这为下一个即时版本提供了信息。 循环从v0运行到v27. AI助手帮助改进了提示，后来的版本使用自动优化。
你的评价可能是假的
许多球队都遇到这种情况。 您想要比较视觉模型，所以您要构建一个评价数据集。 最常用的获得标签的方法就是运行您的当前模型，并将它的输出作为地面真理。 该型号一直在生产中运行，没有人抱怨，所以标签必须足够好。
但这创造了一个循环评价，根据它与现任者的行为相匹配程度，而不是根据实际正确性来判断任何替代办法。
Property Finder的评价数据集恰恰存在这个问题。 在标注的样本中，96.8%直接来自对现有视觉语言模型的预测，而不是独立的人类核查。 任何替代模型在单一图像处理之前都处于结构性劣势。
隐藏错误证实了其严重性。 Watermark文字提取有字符级错误标记为正确，包括幻觉品牌名称和缺失字符。 照片分类强制在包含多个房间类型的图像上贴上单一标签。
在剩余的3.2%的人标标样本中，注释者在原规格之外有推出类，如Office出现时没有这种标签。 当答案密钥被你试图替换的模型写入时，模型比较就失去了有效性。
建立你可以信任的真相
为了打破循环，你需要拥有真正领域知识的人类专家独立地做新的说明。 未提及以前的模型产出。 无法接触现任者所预言的。
数据集范围
Property Finder的团队手持注解的356阿联酋图像和一套单独的埃及图像。 评估小组从具有代表性的行业部门未加标签的生产清单中挑选了这些产品。 本文报导了阿联酋的影像。 埃及的图像是230幅图像的跨区域检查，本文没有报道。 首批256个阿联酋图像将附加注释，形成调制的开发集。 最终模型比较的评价集使用全部356，在开发运行后增加了100个附加注释的图像。 一组356张图像大致足以探测模型之间的中度差异。 个别属性中的稀有类的例子较少，因此关于这些属性的较小差异可能无法察觉。
专家注释
战略和分析专家对所有图像作了注释。 他们带来了房地产上市质量标准方面的专业知识。 AWS Generative AI创新中心团队通过专门的澄清会议进行了系统的质量审查。 这些审查解决了模棱两可的案件，并使数据集的标签保持一致。
多标签过渡
房地产图像经常通过门道显示多个房间或显示重叠的质量特征。 从单标签到多标签分类的转换缓解了人为模糊的一个主要来源。
走廊照片显示一个卧室穿过一个门道，这是真的。 在允许多个标签后，你测量模型得到的正确性，而不是惩罚它来进行合理的二次预测。
混合方法可以补充这个方法：来自不同家族的模型会显示候选的标签和分歧，人类专家会裁定每个案件，因此没有模型输出直接用作标签。 这将减少未来数据集扩展的注释负担。
计量选择
微观F1是首要的衡量标准，因为它对每项预测的权重相等，这与总体标签正确性的业务目标相吻合。 Macro F1曾与之并列报道，因为Micro F1以频繁类为主，可以在稀有属性上隐藏弱性能。 每个标签的精确度和召回度有助于确定具体的失败模式。 每个图像的预测标签与专家标签的重叠程度。 9个单标签属性，如旋转或水印检测，每个图像有一个正确的答案。 AWS团队也以精确的匹配精度，即答案与专家标签匹配的图像份额来打分。
快速优化作为强化学习
有了不带偏见的评价，就可以开始改进。 你不能改进你无法正确测量的东西。
优化遵循类似于强化学习的循环。 将每个即时版本视为agent,Micro F1分数视为奖励信号，错误分析则视为改进的梯度。 这个想法类似于GEPA(Genetic-Pareto反射的即时进化)，你在这里突变了即时的组件，评价健身能力，选择幸存者，以及脚踏实地。 在此次接触中，AWS团队从错误分析中精炼了大多数版本，AI助手审查失败的图像并提出修改建议。
MIPROv2，是来自DSPy库的自动即时优化框架，为v27中的图片分类任务补充了这个人工流程。
从哪里开始
第一个提示(v0)使用了通用分类说明，在此简化以示说明：
“将此图像分类为以下类别之一：生活室、厨房、卧室、浴室。.....”。
Property Finder生产基准的专有型号A，在256图像开发集上得分为0.55微F1. 它使用了v22，一个即时调用它的模范家庭。 V21之前的得分使用早期的单标签设置
以迭代表示
AWS团队在几个候选模型上测试了早期版本，并增加了结构化输出格式和属性特定定义。
最有启发性的修补方法来自图像级错误分析。 在早期的单标签版本中，该模型一贯将通过门道可见的房间归类为二级房间类型。 AWS Generative AI Innovation Center团队根据占据着大部分帧的房间增加了明确的分类指令。 后来切换到多标签输出，将主室作为标签保留下来，允许房间通过门道可见作为附加标签。
区域定制带来了下一个收益。 在一些地区，上市照片带有非水印的装饰性阿拉伯文字覆盖。 该案文的规则删除了一整类虚假阳性。
v21提示将照片分类和室类切换为多标签输出，成为每个模型共享的起始提示。 凭借它，来自Meta的Llama 4 Maverick 17B Instruct在开发集上打出了0.75的一架Micro F1. 从那里，AWS团队分别调制了每个型号的提示。 对于这个模型，v24目标房间大小高估，多标签处理，以及输出格式。 之后V25是自动优化的版本，首先移动了关键任务。 在v27中，MIPROv2重组了照片分类任务，模型的Micro F1在这项任务上达到了0.76.
结束了
最后的提示 v27 PF 是 v27 加上 Property Finder 添加的规则，用于渲染，带有边框或框架的图像，以及走廊。 和v27一样，它保留了阿拉伯文本规则和输出格式指令。 每个模式调用后的一个正常化步骤处理剩余的格式差异。
使用 v27 PF, Llama 4 马华17B 指令在356-图像评价集上打出一架0.80的Micro F1，其中专有型号A的得分为0.56和v22. 这套评价包括256个开发图像，两个模型的提示都根据这些图像来调整。 因此，这些成绩是乐观的。
从专有型号A到Llama 4 Maverick 17B Instruct的切换，各自在自调速器上，将开发集上的Micro F1从0.55提升到0.81. 这是47%的相对收益。 光是快速调音，在Llama 4 Maverick 17B Instruct上，融资从0.75与共享 v21 的速率，到0.81与v27.
从失败的方法中吸取教训
不是一切都成功过 2-bucket方法将水印属性和其他属性放在两个独立的API调用中。 它没有带来有意义的准确性收益。 盒式图像探测也急剧下降。 没有同一提示中的水印属性，模型就误将水印覆盖为图像边框。
通用提示也失败了，不管您运行的是哪种模式。 房地产图象的域名特异性太强，无法一刀切的指令。
成果：费用、准确性和速度
AWS Generative AI Innovation Center团队在Amazon Bedrock上用无偏见的评价框架评价了多个模型。 所有模型都从共享的 v21 提示开始，然后在开发集上为他们的模型家族调制。 专有型号A获得1个调制圆，v22. Llama 4 马华17B 指令有几条，v24到v27，加上Property Finder的边缘案例规则，所以它背后有更多的调试努力。 与专有模式A的最终比较采用了356图像评价集。 Property Finder根据所有11个属性的精度和成本效率选择了Llama 4 Maverick 17B Instruct来完成生产工作量。 关于按区域分列的模型可用性，参见AWS区域在Amazon Bedrock中支持的模型。
以下表格显示，Llama 4 Maverick 17B Instruct(v27 PF)在每个单标签属性上达到的精确匹配精度，在356图像评价集上得分。 它在九项中的六项上得分90%或更高，在所有11项属性上匹配或超过专有模式A. 在照片分类上，每个图像可以有多个标签，两者得分相同。
每1 000幅图像的成本
属性查找分数
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Artificial Intelligence
Fix evaluation bias to unlock better models on Amazon Bedrock
This post was co-authored with Khaled Thabit, Meriem Boughrara, and Mohamed Desouky from Property Finder.
The engineering team at Property Finder, a digital real estate application, discovered that 96.8 percent of their evaluation labels had been inherited directly from a single model’s predictions rather than verified by human experts. Their ground truth was systematically skewed toward the incumbent model, making every alternative appear to underperform. The labels also had errors, but the bias was what blocked objective model comparison. They uncovered this when migrating their vision AI workload to a more cost-effective option on Amazon Bedrock.
Property Finder processes tens of thousands of property images every day across the Middle East and North Africa (MENA). Their system evaluates 11 quality attributes per image. These cover watermark detection, room classification, furnishing status, photo quality, and construction indicators. That’s thousands of new property listings with an average of 15 images each. All need quality assessment before going live.
Property Finder needed a more cost-effective option. But measured against those labels, any alternative was at a disadvantage.
The hardest part was not finding a better model. It was discovering that their evaluation methodology itself was the blocker.
In this post, we share how Property Finder identified evaluation bias, built unbiased ground truth, and applied evolutionary prompt optimization to select the right model on Amazon Bedrock, achieving higher accuracy at 86 percent lower cost. We walk through the methodology and iteration examples, with lessons you can apply to your own workloads.
The following figure illustrates the prompt optimization loop that Property Finder and the AWS Generative AI Innovation Center team used to iteratively improve classification accuracy on Amazon Bedrock.
The diagram shows the iterative prompt optimization cycle. A property image is processed by Amazon Bedrock, which outputs the predicted attributes in JSON format. During optimization, these predictions are scored with Micro F1 against the 256-image expert-annotated development set. Error analysis then inspects failed images and identifies visual patterns, which inform the next prompt version. The loop ran from v0 to v27. AI assistants helped refine the prompts, and later versions used automated optimization.
Your evaluation might be lying
Many teams encounter this scenario. You want to compare vision models, so you build an evaluation dataset. The most common way to get labels is to run your current model and treat its outputs as ground truth. The model has been running in production and nobody complained, so the labels must be good enough.
But this creates a circular evaluation where any alternative gets judged by how closely it matches the incumbent’s behavior, not by actual correctness.
Property Finder’s evaluation dataset had exactly this problem. Of their labeled samples, 96.8 percent came directly from the predictions of the existing vision language model (VLM) rather than independent human verification. Any alternative model was structurally disadvantaged before a single image was even processed.
Hidden errors confirmed the severity. Watermark text extraction had character-level mistakes marked as correct, including hallucinated brand names and missing characters. Photo classification forced single labels onto images containing multiple room types.
In the remaining 3.2 percent of human-labeled samples, annotators had introduced categories outside the original specification, like Office appearing where no such label existed in the prompt. A model comparison loses validity when the answer key was written by the model you are trying to replace.
Building ground truth you can trust
To break the cycle, you need human experts with real domain knowledge doing fresh annotations independently. No reference to previous model outputs. No access to what the incumbent predicted.
Dataset scope
Property Finder’s team hand-annotated 356 UAE images and a separate set of Egypt images. The team selected these from unlabeled production listings across representative industry segments. This post reports results on the UAE images. The Egypt images served as a cross-regional check on 230 images and are not reported here. The first 256 UAE images to be annotated formed the development set for tuning. The evaluation set for the final model comparison uses all 356, adding 100 images annotated after the development runs. A set of 356 images is roughly sufficient to detect moderate differences between models. Rare classes within individual attributes have fewer examples, so smaller differences on those attributes might not be detectable.
Expert annotation
A strategy and analytics expert annotated all images. They brought specialized knowledge of real estate listing quality standards. The AWS Generative AI Innovation Center team conducted systematic quality reviews through dedicated clarification sessions. Those reviews settled ambiguous cases and kept the labeling consistent across the dataset.
Multi-label transition
Real estate images frequently show multiple rooms through doorways or display overlapping quality characteristics. The switch from single-label to multi-label classification alleviated a major source of artificial ambiguity.
A hallway photo showing a bedroom through a doorway is genuinely both. After you allow multiple labels, you measure what the model gets right rather than penalizing it for reasonable secondary predictions.
A hybrid approach could complement this methodology: models from different families surface candidate labels and disagreements, and human experts adjudicate every case, so no model output is used directly as a label. This would reduce the annotation burden for future dataset expansion.
Metrics selection
Micro F1 served as the primary metric because it weights each prediction equally, which matches the business goal of overall label correctness. Macro F1 was reported alongside it because Micro F1 is dominated by frequent classes and can hide weak performance on rare attributes. Per-label precision and recall helped identify specific failure modes. Per-sample intersection over union (IoU) measured how closely each image’s predicted labels overlapped the expert’s labels. The nine single-label attributes, such as rotation or watermark detection, have one right answer per image. The AWS team also scored them with exact-match accuracy, the share of images where the answer matches the expert label.
Prompt optimization as reinforcement learning
With unbiased evaluation in place, you can start improving. You cannot improve what you cannot measure correctly.
The optimization follows a loop analogous to reinforcement learning. Think of each prompt version as an agent, the Micro F1 score as the reward signal, and error analysis as the gradient for improvement. The idea resembles GEPA (Genetic-Pareto reflective prompt evolution), where you mutate prompt components, evaluate fitness, select survivors, and iterate. In this engagement, the AWS team refined most versions from error analysis, with AI assistants reviewing failed images and proposing changes.
MIPROv2, an automated prompt optimization framework from the DSPy library, complemented this manual process for the photo classification task in v27.
Where it started
The first prompt (v0) used generic classification instructions, simplified here for illustration:
“Classify this image into one of the following categories: Living Room, Kitchen, Bedroom, Bathroom…”
Proprietary model A, Property Finder’s production baseline, scored a Micro F1 of 0.55 on the 256-image development set. It used v22, a prompt tuned for its model family. Scores before v21 used the earlier single-label setup and aren’t directly comparable.
Iteration by iteration
The AWS team tested early versions on several candidate models and added structured output formatting and attribute-specific definitions.
The most instructive fix came from image-level error analysis. In the early single-label versions, the model consistently classified rooms visible through doorways as the secondary room type. The AWS Generative AI Innovation Center team added explicit instructions to classify based on the room occupying the majority of the frame. The later switch to multi-label output kept the main room as a label and allowed rooms visible through doorways as additional labels.
Regional customization produced the next gain. In some regions, listing photos carry decorative Arabic text overlays that are not watermarks. Rules for that text removed a whole category of false positives.
The v21 prompt switched photo classification and room category to multi-label output and became the shared starting prompt for every model. With it, Llama 4 Maverick 17B Instruct from Meta scored a Micro F1 of 0.75 on the development set. From there, the AWS team tuned each model’s prompt separately. For this model, v24 targeted room size overestimation, multi-label handling, and output format. Then v25, an automatically optimized version, moved the critical tasks first. In v27, MIPROv2 restructured the photo classification task, and the model’s Micro F1 on that task reached 0.76.
Where it ended
The final prompt, v27_PF, is v27 plus rules that Property Finder added for renders, images with borders or frames, and corridors. Like v27, it keeps the Arabic text rules and output format instructions. A normalization step after each model call handles remaining format differences.
With v27_PF, Llama 4 Maverick 17B Instruct scored a Micro F1 of 0.80 on the 356-image evaluation set, where proprietary model A scored 0.56 with v22. The evaluation set includes the 256 development images, and both models’ prompts were tuned on those images. These scores are therefore optimistic.
The switch from proprietary model A to Llama 4 Maverick 17B Instruct, each on its own tuned prompt, lifted Micro F1 from 0.55 to 0.81 on the development set. That is a 47 percent relative gain. Prompt tuning alone, on Llama 4 Maverick 17B Instruct, raised it from 0.75 with the shared v21 prompt to 0.81 with v27.
Lessons from failed approaches
Not everything succeeded. The 2-bucket approach put the watermark attributes and the other attributes in two separate API calls. It brought no meaningful accuracy gain. Boxed-image detection also dropped sharply. Without the watermark attributes in the same prompt, the models mistook watermark overlays for image borders.
Generic prompts also failed regardless of which model you ran them on. Real estate imagery has too much domain specificity for a one-size-fits-all instruction.
Results: Cost, accuracy, and speed
The AWS Generative AI Innovation Center team evaluated multiple models on Amazon Bedrock with the unbiased evaluation framework. All models started from the shared v21 prompt and were then tuned for their model family on the development set. Proprietary model A got one tuning round, v22. Llama 4 Maverick 17B Instruct got several, v24 to v27, plus Property Finder’s edge-case rules, so it had more tuning effort behind it. The final comparison against proprietary model A used the 356-image evaluation set. Property Finder selected Llama 4 Maverick 17B Instruct for their production workload based on accuracy across all 11 attributes and cost efficiency. For model availability by Region, refer to Supported models by AWS Region in Amazon Bedrock.
The following table shows the exact-match accuracy that Llama 4 Maverick 17B Instruct (v27_PF) reached on each single-label attribute, scored on the 356-image evaluation set. It scored 90 percent or higher on six of the nine, and matched or exceeded proprietary model A on all 11 attributes. On photo classification, which can have several labels per image, the two scored the same.
Cost per 1,000 images
Property Finder score
Auto-fetched and translated by AI Atlas. All rights belong to the original author.