SAM3 Appears to Have Flaws in Cross-Image Inference

For those in a hurry

Please read the following:
https://github.com/facebookresearch/sam3/issues/317

When you use the image below as a visual prompt as-is

the inference results turn out strange.

Key Points

1. While the paper claims that "cross-image concept transfer" is possible, this functionality is not implemented in the official code.
2. Using the Bounding Box of the reference image causes failure. SAM 3 prioritizes "coordinates (position)" over "appearance." It tries to look for the same location in Image B as it did in Image A (Positional Bias).
3. To break the dependency on box coordinates, using the mask (or extracted points) of the reference image as a prompt instead of the box works. This is because masks can extract the object's visual features more purely.
4. Since the weight files might not contain a mask encoder, the method of "sampling multiple points from the mask region and inputting them" is the most reliable. Code implementing this workaround is shared in repositories such as muggled_sam by heyoeyo.

Excerpt from discussion

It seems that the training of SAM 3's visual prompt has been hacked, whether intentionally or unintentionally.
However, for a task as fundamental as cross-image inference, I suspect the developers were likely aware of this characteristic and tacitly accepted it.

I tried your code and noticed something very interesting.
If the sample image contains only one subject (a cat), the prediction result tends to treat the entire image as the target. However, if the sample image contains three cats, the result is quite good.

If a user draws a bounding box around an object, the model will certainly recognize that an object exists there if the source image and the target image are the same. Unfortunately, this does not apply to cross-image tasks.