Understanding Confidence Scores
What "87 out of 100 robots would flag this" actually means.
What is a Confidence Score?
When Vault scans a photo, the AI outputs a confidence score between 0% and 100%. This represents how certain the model is that the photo contains sensitive content.
Think of it this way: if we ran 100 slightly different versions of our AI model on the same photo, 87 of them would flag it as sensitive. The higher the number, the more certain the detection.
Why not just "yes" or "no"?
Real-world photos aren't always clear-cut. A photo might be ambiguous -- swimwear at a beach, artistic photography, medical images. The confidence score tells you how sure the AI is, so you can make informed decisions.
The False Positive / False Negative Tradeoff
No AI is perfect. There are two types of mistakes:
False Positive
AI flags a safe photo as sensitive
Annoying, but harmless
False Negative
AI misses an actual sensitive photo
Defeats the purpose
You can't minimize both at the same time:
- Lower threshold (catch more): Fewer false negatives, but more false positives
- Higher threshold (be strict): Fewer false positives, but more false negatives
Why We Err Toward False Positives
Vault is designed to err on the side of caution. Here's why:
The cost of missing a sensitive photo is higher than the cost of reviewing a safe one.
Consider the consequences:
- False positive: You spend 2 seconds looking at a flagged photo and thinking "nope, that's fine" -- minor inconvenience
- False negative: A sensitive photo stays visible when you thought your library was clean -- actual problem
This is why our default threshold is relatively sensitive. We'd rather flag a few extra beach photos than miss something actually sensitive.
Like a Medical Screening
Medical tests are designed to catch potential issues even if it means some false alarms. A mammogram might flag something that turns out to be benign -- that's better than missing actual cancer. Same principle here.
Play With the Threshold
You can adjust how sensitive the detection is in the app. Lower thresholds catch more (including more false positives). Higher thresholds are stricter (but might miss borderline cases).
In the app, you can review flagged photos and mark safe ones. The more you review, the better you'll understand your personal comfort level.
Confidence Score Ranges
80-100%: High Confidence
The AI is very sure this photo contains sensitive content. These are rarely false positives.
50-80%: Medium Confidence
Likely sensitive, but could be ambiguous -- swimwear, artistic shots, partial views. Worth reviewing.
30-50%: Low Confidence
Borderline cases. The AI isn't sure. These have higher false positive rates but are flagged to be safe.
0-30%: Likely Safe
The AI doesn't think this is sensitive. Not flagged by default.
The Bottom Line
Confidence scores give you transparency into the AI's thinking. Instead of a mysterious black box that just says "yes" or "no," you see exactly how certain the model is.
We default to catching more rather than missing things, because:
- Reviewing a false positive takes 2 seconds
- Missing a sensitive photo defeats the whole purpose
- You can always adjust the threshold to your preference
Remember: You're always in control. The AI flags, you decide. Every flagged photo is a suggestion, not a judgment.