Why do I only find out what dangerous things my model can do after it ships?
机会
Automated red-teaming tools such as GCG, AutoDAN, and PAIR report attack success rates of 5 to 15 percent and give a false sense of safety. Multi-turn human red-teaming on the same models finds failures up to 75 percent of the time on the same categories. The gap means that dangerous capability uplift in areas like bioweapon synthesis guidance or offensive cyber is being missed at the automated pre-deployment stage and found in the field instead. Frontier labs run their own manual evaluations under frameworks like Anthropic's RSP and METR's TaskDev, but the methodology is undocumented and non-standardized enough that no two labs run comparable tests. NIST's AI agent red-teaming guidance was still an annotated outline in early 2026, with full publication expected late 2026 to 2027.
为什么重要
A dangerous emergent capability discovered post-deployment in a widely distributed model is a different order of problem than one caught before release.
我如何评估机会
机会评分是我的个人判断，而非量化指标：痛苦程度、发生频率，以及当前解决方案的匮乏程度。分数越高，意味着我认为越值得去构建。
出现时造成的痛苦程度。
人们实际遇到它的频率。
当前针对它的优质工具有多匮乏。