Aside from the fact that this was obviously never viable and the entire problem is clearly unsolvable if you sit down and really probe it for fifteen minutes, what I find most frustrating about this is that the false promise of preserving photos as reliable evidence is actively harmful.
You will not build a perfect system, or even something near perfect. The best you're going to do is make it so that it's hard to casually present AI photos as real, leaving only the cases where it really matters. In the "best" case, you've just made the public more trusting of photos in general, so that when there's actual money or power on the line that makes jumping through the hoops to fake authenticity worth it, the public is more susceptible.
The best outcome at this point is for everyone to get on the same page that photos have roughly the same probative value now as drawings. Poorly thought out snake oil efforts to prove authenticity are only going to delay that.
In my opinion, the benefits for end users are rather minimal since I doubt an average person would ever be checking C2PA provenance data, but there is a commercial incentive for Google and others to promote C2PA, since it makes preparing training material for Machine Learning significantly easier, and perhaps as a smaller benefit justifies hardware attestation that locks users down into proprietary OSes.
I think it's useful even if it can be spoofed. Many people don't even bother to edit visible watermarks out of AI photos/videos. I'm fairly certain this will defeat 99.9% of malicious users, many of whom won't even know it exists until someone points out it's missing.
People are concerned that the technology will lend additional credence to the last 0.1%. But anyone who thinks about the technology for 2 minutes will see you can just point the camera at the screen. In cases where it really matters (a court of law, internet arguments between nerds) people will know it's not 100% reliable. Locks can be picked, and signatures can be forged, but that doesn't make them useless.
"C2PA Cameras Do Not Survive Contact With Reality" does not survive contact with reality where very, very few users would even think of rooting their phone so they can create signed fake images.
When I search for "C2PA" on the google play store, there are more AI-watermark-removal apps than there are signing apps. Certain types will jump through ridiculous hoops if they think it will affect their algorithmic reach on social media.
Malicious users don't need to root their own phones. They just need to go to fakemyimage dot com, and someone else's rooted phone in a clickfarm-type setup signs it for them. I am not operating such a service myself because I thought it was unnecessary in making my point, but perhaps I will have to reconsider.
You're arguing that lots of people can spoof this, the other guy is arguing that nobody will know it can be spoofed so it will do more damage. But these are contradictory -- if fakes become common, then they will also become common knowledge. The impact of any given fake is reduced if there are more of them. The technology doesn't need to provide 100% assurance. If it adds even a little friction to the slop mills then that's increasing the signal to noise ratio.
It's actually worse if it is plausibly trustworthy for "99.9%", since that's enough that naive users will get accustomed to believing the verification badge is authentic.
When a motivated malicious user (who doesn't actually need that much resources) will be able to convince people something is authentic because the verification passes when it shouldn't since naive users are primed to believe it by default.
Read the rest of my comment please. Is the single motivated malicious user able to do as much damage as all of the blocked attempts put together? Probably not, since if there's really all that much riding on it, people will point out it can be bypassed.
Should we also abolish Pangram, because it's not 100% accurate? Someone might be convinced a text is not AI-generated when it actually is! We should get rid of it rather than fool people into thinking it can be determined accurately. What about antivirus? We should abolish it as well rather than fool people into thinking that their software is ever 100% safe. What about HTTPS? We shouldn't call it "secure" shell because the computer you're connecting to could be compromised! I could go on and on and on.
The median instance of AI image generation isn't evidence in a court case. It's cyberbullying, or deepfakes, or fake news. It's called "slop" because there's a lot of it being churned out at low effort.
You're missing all of the points that there could be by focussing on random people.
While it is always an individual tragedy when people treat each other badly (e.g. through deepfakes and all), the real threat does not exist on that level.
This is about misinformation and disinformation, so we're talking state actors. And with that, the 99.9% hypothesis does not hold true.
It's funny how people always say something is "a tragedy at the individual level" when they mean "it's not my problem." It's even crazier to dismiss the value of a security feature, just because it might make people feel more secure. That's true of every security feature! Very little of the technology that the web is built on is proof against state actors.
I like being contrarian as much as the next guy, but "Actually, having security is worse for security" is taking it a little too far.
I am repeating myself, but this is about systems, and not about people.
It is however in the interest of the people to keep the systems running in an untainted way.
As said, on the individual level it's a tragedy, but one that can be absorbed somewhat. Democracy itself failing otoh is kinda hard to absorb.
C2PA is not "having security". It is "having an illusion of security for compliance and CYA reasons, that can be fairly trivially exploited by nation state actors".
Banality of evil. Again.
___
Actually, come to think of it, "security" is the wrong term there. Signatures don't secure anything. They attest.
Those are different things.
Argh and I ran with your term aah
I don’t think there is a technical solution to this problem but I think there is a legal one.
Make it a legal requirement to mark AI generated photos and enforce penalties for posting unmarked AI generations. Social media should also mark the country of origin for each post, with the knowledge that posts from your own country are covered by these laws.
I think public key cryptography offers a technical solution that is nearly as ideal here as for its existing uses for securing communication between physically remote actors -- please see my comment here for details: https://news.ycombinator.com/item?id=49444227
I say "nearly", because as soon as you need to keep a private key secure from someone with direct physical access to the device, you're entering dangerous territory. TTBOMK there's no way to make a "perfect black box", so it becomes an arms race between defensive "obfuscation" and tamper detection mechanisms in the one hand and stealth scanning techniques on the other. But this is already the case for TPMs -- that is, the situation is no worse than for an already widely accepted technology.
What makes you so certain that this valuable research showing weaknesses in today's systems will render the C2PA concept useless forever?
Yes, software LPEs are a risk -- as they are in every nontrivial computer system. New ones will appear, and old ones will be closed in time, as TFA acknowledges.
Re hardware attacks: The (neat!) glitch injection attack the author describes in the linked "lighter" page only raises the implementation cost of doing image certification properly. For example, if the camera module presented only an interface that dumped raw RGB or JPEG-encoded data plus a digital signature that used a private key known only to the manufacturer, then all that would be required to verify a "downstream" image would be to keep a copy of those original bytes inside the final (potentially cropped, filtered, AI-ed, etc.) image, in the worst case roughly doubling its size on disk (though certainly more efficient schemes could be designed). Any interested third party could then compare the original and final images by eye and decide for themselves whether or not the subsequent processing materially changed the image's "meaning".
Finally: Does the existence of lock picks or bolt cutters render padlocks pointless today? Does it corrode society by encouraging people to mistakenly believe that anything they put behind a $5 padlock will be safe forever? No, and no.
This is yet another of these absolutely caustic takes that come with a veneer of intellectualism, but actually just ignorantly deconstruct reality.
Each of them weaponizing the rules of the platform that (for sensible reasons) demand you engage with the strongest interpretation of the message/argument you see.
The asymmetry of effort there is unsustainable, and that's exactly the point.
No idea how that could be solved. Maybe a meta comment like this one helps.
__
I mean if you think about it, it shouldn't be possible for some anon account to drop this and sound like it's a worthy contribution to a debate against some real person with a real name, a track record and multiple thousand dollars of bricked hardware leading up to that assessment.
(Nor would it make sense for a non-anon but equally empty account)
It makes no sense, and the guarantees regarding protection of speech and all do not apply to these topics, because it's not an opinion that would get your real name in jail.
What can we do about these social exploits. Someone tell me please. It's driving me up the walls
I'd always thought the usefulness of C2PA was limited to verified devices in custody by trusted actors.
Like a security camera with a tamper evident enclosure, or an organisation being able to attest that they recorded the imagery.
The idea that it could be used to attest the authenticity of any random person or device surely wasn't a thing serious people expected was it?
> was limited to verified devices in custody by trusted actors. Like a security camera with a tamper evident enclosure, or an organisation being able to attest that they recorded the imagery.
But that is also not the case, because whatever keys are in those devices may have been duplicated in the factory or somewhere along the supply chain.
Or the stuff is cloud connected and an exploit can be executed via that.
Or, as written in the blog post you're commenting on, software exploits.
The whole idea is that the concept works for no one.
I fear that the long-term result might be a small cartel of "trusted" companies that holds the private keys which can attest and lead to the death (or at least limiting widespread adopting) of open-source operating systems.
It won't stop fraud by governments and/or determined/well-resourced attackers, but it'll make it difficult enough for 99.99% of the public. And as usage drops over time, it becomes more socially acceptable to justify further limitations.
Governments and corporations (e.g. banks) will require that you use a "trusted" (meaning locked-down) devices to interact with their services. We're already seeing some companies block GrapheneOS/LineageOS.
It's compliance for advertising. Your client doesn't want AI in their project you show them the audit trail and if it turns out to be faked you point to the supplier who faked it. Our agency signed a insurance not only because of clients who don't want to use AI in their artwork but also because of the EU AI act. They c2pa to get their money back from a cheating supplier if they are not compliant with the AI act.
I always got the impression that C2PA is a way to say "this photo came from the BBC (for example) and they've only signed it because they've verified the supplied edit chain". It's always been obvious that one could point a camera at a screen, I don't think anyone involved with C2PA has claimed otherwise.
It seems like there's a big disconnect between what C2PA says it's for and what certain journalists think it's for.
C2PA is a bit nebulous as a concept, which is part of the problem. It is both of these things. The BBC type use case where a publisher signs their own content with their own keys seems reasonable to me, or at least, not obviously broken.
However I question the value-add when e.g. the BBC website is already authenticated by nature of being served over HTTPS, and anyone who redistributes BBC content can and should link back to the source.
> It's always been obvious that one could point a camera at a screen, I don't think anyone involved with C2PA has claimed otherwise.
They haven't claimed otherwise exactly, but some have implied it's a solvable problem. Here's where the "learn more" link goes, for when Youtube annotates a video as having C2PA metadata: https://support.google.com/youtube/answer/15446725 (Google is a C2PA Steering Committee member)
> The metadata that leads to a 'Captured with a camera' disclosure is made by a third party (for example, a camera manufacturer). This means that there is some risk that someone could take a photo of another screen showing synthetic content. Because the other screen shows an image that has been modified, it wouldn't be eligible for the 'Captured with a camera' disclosure. This issue is called 'air-gapping'. Camera manufacturers will continue to develop detection measures to prevent 'air-gapping', but the sophistication of those detection measures may vary in the near term.
Interestingly they do not mention any of the other known limitations. Their phrasing is highly weasel-wordy, but the implication is clearly that they imagine picture-of-screen detection to become robust (somehow) in the medium-to-long term.
> However I question the value-add when e.g. the BBC website is already authenticated by nature of being served over HTTPS, and anyone who redistributes BBC content can and should link back to the source.
The value would be in images reposted to social media where the website an show a badge that says it came from a certain source.
Why not just have BBC sign stuff with their own creds then? You can originate the chain of custody anywhere and relying on the camera is kind of silly when the reputation of the original publisher in a more meaningful backstop.
It's useful to have all your tools automatically apply metadata in a standard way instead of having to keep track of it manually. Most cameras already add metadata that says what camera and lens were used, but you lose that as soon as you import it into Photoshop and export as a jpeg.
Even at the hardware level, if it was a separate chip that the camera data passed through or something, that's not really good enough either, people have broken TPMs before. It'd have to be baked into the camera sensor. Even then, you could attack it from the next level up, with some fancy optics and a display, or something like that.
I don't think completely solving this sort of problem is even possible.
And I'm not sure it's even useful to solve. The presence/absence of a digital signature will never be the deciding factor in whether people accept/reject an image as authentic.
What I'm getting at is that metadata that claims that a photo is "real" won't necessarily convince people that it is, and the lack of that metadata doesn't mean anything at all. About the only real use I've seen for C2PA is to confirm that an image bearing that metadata is AI-generated - and that marker is easily lost through editing or retransmission.
And in 2026, I don't think it's too big of a stretch to imagine that there are going to be people in power that can add + remove the metadata to whatever image they want, at will, to tell whatever story they want to create. Sadly.
There were similar fears about the webtrust CA system, but AFAIK there's no known incidents where a government strongarmed a CA into misissuing a MITM certificate, and then it was used in MITM attacks. The closest is some misissued certificates seemingly due to incompetence but weren't used in attacks.
And at that level, it’s a matter of definition anyway. What is “AI generated”? A photo of a screen showing an AI picture is still a photo. If that’s “AI” then what about a photo of an AI generated billboard? Or a photo of a bus with an AI graphic on the side?
I'm very surprised Google put in so much effort to implement an approach that is basically the equivalent of client-side verification of passwords. Did no one designing it mention that it could be defeated by any rooted device?
Actually I think this approach is very forward looking! Attestation is on the cusp of becoming a very powerful technique. We just need to figure out how to build 100% bug-free and 100% secure hardware and software, and then it's gonna work great.
Well, all one has to do is look at the bigger picture of how rooted devices are being shuffled into 3rd rate/totally blocked experiences and the overall direction of things starts to take very clear shape.
You could actually make this secure, by having firmware in the camera module itself do the signature before handing it off to the rest of the system. The key is to prevent ingress of control, as long as the module only emits signed photographs or video, and has tightly controlled input channels, for zoom, aperture, etc... it seems a quite reasonable project.
That might be detectable if they signed content contains focal-length metadata... but even then, some foresight and a collection of lenses would hide it.
For the time being, I wish more workflows revolved around RAWs. You can pretty much prove the origins of an image sans cryptography by just possessing the image sensor data that you used to produce the final. It's not foolproof but AI image models can't really generate a convincing bayer raw and it's not something anyone is going to spend significant time training.
Isn't film photography verifiable with a physical negative? i.e. you could verify a digital reproduction of the analog photo by comparing the digital image with the negative. Tedious, but sound?
I would be interested in a note on whether Sony / Leica / Olympus “content credentials” do any better with their hardware to ensure a signature is assigned to data straight off the sensor.
My bet is they do considerably worse. Digital cameras are not designed with security in mind. Arbitrary code execution has been achieved on many DSLRs and there's even been open-source firmware projects for some.
Unfortunately they're a little outside of my tinkering budget, but if anyone wants to send me some I'll do my best to pwn them. Can't be any harder than a Google flagship, one would imagine.
I have ordered a faulty Sony A7 IV motherboard, but due to its faulty-ness and the lack of the rest of the camera, I'm not sure how far I'll be able to get with it.
I expect the only plausible chance (and it is a stretch) will be at-the-censor marking. Quantum bla bla magic pixie dust or unicorn farts something. The chance of a trustworthy (including from nation-state tampering a la Stalin et al) means of verification of digital anything is as good as dead imho.
I have a feeling Apple is going to knock it out of the park on this when they get around to it. They have a great foundation for doing image provenance well. The device attestation workflows are already there. And the same attacks that work against Android won't be as easy or effective because of Secure Enclave. Apple could run the whole signing process inside SEP.
And, Apple could choose to integrate a LiDAR depth map into the signed photo as a mitigation against the analog attacks (eg. pictures of screens).
Apple isn’t going to touch this with a 10-foot pole.
The provenance “proof” these approaches provide is very tenuous and nowhere near the “this is a real photo of a real world event taken by a real camera and not an AI image” proof that marketing types like to push.
Apple doesn’t want a PR disaster where some crazy image is totally fake but becomes world news because it is “cryptographically signed as being from a real iPhone so it must be real!”
>Images captured with an opt-in Reference mode can be authenticated to confirm they were taken with an iPhone. Authenticating is done by tapping the Reference badge on the image, which sends the raw image, sensor signatures, capture time frame, and the unique hardware identifiers of the sensor to Apple's Private Cloud Compute (PCC) servers. PCC uses the information to determine whether the camera captured the photo, gives it a unique ID, and then returns an authenticated version to the user's device.
Yeah, it might never ship, or it might not ship as described, or that might be exactly what they do - I love being wrong.
If it does ship like that, it’s hard to not imagine a situation as I described earlier - an “iPhone Reference Image” being used to propagate fake news, at which point the credibility of the feature goes to 0 (and Apple’s takes a severe hit).
You will not build a perfect system, or even something near perfect. The best you're going to do is make it so that it's hard to casually present AI photos as real, leaving only the cases where it really matters. In the "best" case, you've just made the public more trusting of photos in general, so that when there's actual money or power on the line that makes jumping through the hoops to fake authenticity worth it, the public is more susceptible.
The best outcome at this point is for everyone to get on the same page that photos have roughly the same probative value now as drawings. Poorly thought out snake oil efforts to prove authenticity are only going to delay that.
People are concerned that the technology will lend additional credence to the last 0.1%. But anyone who thinks about the technology for 2 minutes will see you can just point the camera at the screen. In cases where it really matters (a court of law, internet arguments between nerds) people will know it's not 100% reliable. Locks can be picked, and signatures can be forged, but that doesn't make them useless.
"C2PA Cameras Do Not Survive Contact With Reality" does not survive contact with reality where very, very few users would even think of rooting their phone so they can create signed fake images.
Also, note that C2PA should have something like Level 3 as well: "The image is mathematically proven to have come from the physical camera sensor."
It's somehow difficult to achieve this, but it's possible (although the attacks will always be possible of course).
Malicious users don't need to root their own phones. They just need to go to fakemyimage dot com, and someone else's rooted phone in a clickfarm-type setup signs it for them. I am not operating such a service myself because I thought it was unnecessary in making my point, but perhaps I will have to reconsider.
When a motivated malicious user (who doesn't actually need that much resources) will be able to convince people something is authentic because the verification passes when it shouldn't since naive users are primed to believe it by default.
Should we also abolish Pangram, because it's not 100% accurate? Someone might be convinced a text is not AI-generated when it actually is! We should get rid of it rather than fool people into thinking it can be determined accurately. What about antivirus? We should abolish it as well rather than fool people into thinking that their software is ever 100% safe. What about HTTPS? We shouldn't call it "secure" shell because the computer you're connecting to could be compromised! I could go on and on and on.
The median instance of AI image generation isn't evidence in a court case. It's cyberbullying, or deepfakes, or fake news. It's called "slop" because there's a lot of it being churned out at low effort.
While it is always an individual tragedy when people treat each other badly (e.g. through deepfakes and all), the real threat does not exist on that level.
This is about misinformation and disinformation, so we're talking state actors. And with that, the 99.9% hypothesis does not hold true.
I like being contrarian as much as the next guy, but "Actually, having security is worse for security" is taking it a little too far.
It is however in the interest of the people to keep the systems running in an untainted way.
As said, on the individual level it's a tragedy, but one that can be absorbed somewhat. Democracy itself failing otoh is kinda hard to absorb.
C2PA is not "having security". It is "having an illusion of security for compliance and CYA reasons, that can be fairly trivially exploited by nation state actors". Banality of evil. Again.
___
Actually, come to think of it, "security" is the wrong term there. Signatures don't secure anything. They attest.
Those are different things. Argh and I ran with your term aah
Make it a legal requirement to mark AI generated photos and enforce penalties for posting unmarked AI generations. Social media should also mark the country of origin for each post, with the knowledge that posts from your own country are covered by these laws.
I say "nearly", because as soon as you need to keep a private key secure from someone with direct physical access to the device, you're entering dangerous territory. TTBOMK there's no way to make a "perfect black box", so it becomes an arms race between defensive "obfuscation" and tamper detection mechanisms in the one hand and stealth scanning techniques on the other. But this is already the case for TPMs -- that is, the situation is no worse than for an already widely accepted technology.
This is ridiculous.
Yes, software LPEs are a risk -- as they are in every nontrivial computer system. New ones will appear, and old ones will be closed in time, as TFA acknowledges.
Re hardware attacks: The (neat!) glitch injection attack the author describes in the linked "lighter" page only raises the implementation cost of doing image certification properly. For example, if the camera module presented only an interface that dumped raw RGB or JPEG-encoded data plus a digital signature that used a private key known only to the manufacturer, then all that would be required to verify a "downstream" image would be to keep a copy of those original bytes inside the final (potentially cropped, filtered, AI-ed, etc.) image, in the worst case roughly doubling its size on disk (though certainly more efficient schemes could be designed). Any interested third party could then compare the original and final images by eye and decide for themselves whether or not the subsequent processing materially changed the image's "meaning".
Finally: Does the existence of lock picks or bolt cutters render padlocks pointless today? Does it corrode society by encouraging people to mistakenly believe that anything they put behind a $5 padlock will be safe forever? No, and no.
Each of them weaponizing the rules of the platform that (for sensible reasons) demand you engage with the strongest interpretation of the message/argument you see.
The asymmetry of effort there is unsustainable, and that's exactly the point.
No idea how that could be solved. Maybe a meta comment like this one helps.
__
I mean if you think about it, it shouldn't be possible for some anon account to drop this and sound like it's a worthy contribution to a debate against some real person with a real name, a track record and multiple thousand dollars of bricked hardware leading up to that assessment. (Nor would it make sense for a non-anon but equally empty account)
It makes no sense, and the guarantees regarding protection of speech and all do not apply to these topics, because it's not an opinion that would get your real name in jail.
What can we do about these social exploits. Someone tell me please. It's driving me up the walls
The idea that it could be used to attest the authenticity of any random person or device surely wasn't a thing serious people expected was it?
But that is also not the case, because whatever keys are in those devices may have been duplicated in the factory or somewhere along the supply chain. Or the stuff is cloud connected and an exploit can be executed via that.
Or, as written in the blog post you're commenting on, software exploits.
The whole idea is that the concept works for no one.
It won't stop fraud by governments and/or determined/well-resourced attackers, but it'll make it difficult enough for 99.99% of the public. And as usage drops over time, it becomes more socially acceptable to justify further limitations.
Governments and corporations (e.g. banks) will require that you use a "trusted" (meaning locked-down) devices to interact with their services. We're already seeing some companies block GrapheneOS/LineageOS.
It seems like there's a big disconnect between what C2PA says it's for and what certain journalists think it's for.
However I question the value-add when e.g. the BBC website is already authenticated by nature of being served over HTTPS, and anyone who redistributes BBC content can and should link back to the source.
> It's always been obvious that one could point a camera at a screen, I don't think anyone involved with C2PA has claimed otherwise.
They haven't claimed otherwise exactly, but some have implied it's a solvable problem. Here's where the "learn more" link goes, for when Youtube annotates a video as having C2PA metadata: https://support.google.com/youtube/answer/15446725 (Google is a C2PA Steering Committee member)
> The metadata that leads to a 'Captured with a camera' disclosure is made by a third party (for example, a camera manufacturer). This means that there is some risk that someone could take a photo of another screen showing synthetic content. Because the other screen shows an image that has been modified, it wouldn't be eligible for the 'Captured with a camera' disclosure. This issue is called 'air-gapping'. Camera manufacturers will continue to develop detection measures to prevent 'air-gapping', but the sophistication of those detection measures may vary in the near term.
Interestingly they do not mention any of the other known limitations. Their phrasing is highly weasel-wordy, but the implication is clearly that they imagine picture-of-screen detection to become robust (somehow) in the medium-to-long term.
The value would be in images reposted to social media where the website an show a badge that says it came from a certain source.
I don't think completely solving this sort of problem is even possible.
The analog hole is alive and well:)
Doesn't sound like it's engineering-driven, even though they still do have a lot of capable engineers sitting there and atrophying.
I also wouldn't rule out that the less capable ones actually believed that the systems they've built are unrootable or something like that.
At over a decade old, still prescient as ever: https://www.youtube.com/watch?v=HUEvRyemKSg
And, given that the main threat here is disinformation by nation state actors, they can also attack the camera module (or its supply chain) instead.
https://en.wikipedia.org/wiki/Film_recorder
I have ordered a faulty Sony A7 IV motherboard, but due to its faulty-ness and the lack of the rest of the camera, I'm not sure how far I'll be able to get with it.
And, Apple could choose to integrate a LiDAR depth map into the signed photo as a mitigation against the analog attacks (eg. pictures of screens).
The provenance “proof” these approaches provide is very tenuous and nowhere near the “this is a real photo of a real world event taken by a real camera and not an AI image” proof that marketing types like to push.
Apple doesn’t want a PR disaster where some crazy image is totally fake but becomes world news because it is “cryptographically signed as being from a real iPhone so it must be real!”
>Images captured with an opt-in Reference mode can be authenticated to confirm they were taken with an iPhone. Authenticating is done by tapping the Reference badge on the image, which sends the raw image, sensor signatures, capture time frame, and the unique hardware identifiers of the sensor to Apple's Private Cloud Compute (PCC) servers. PCC uses the information to determine whether the camera captured the photo, gives it a unique ID, and then returns an authenticated version to the user's device.
If it does ship like that, it’s hard to not imagine a situation as I described earlier - an “iPhone Reference Image” being used to propagate fake news, at which point the credibility of the feature goes to 0 (and Apple’s takes a severe hit).
Wait & see.