AI’s black box problem has CIOs begging for reliable guardrails – before it's too late

CISOs and CIOs are quietly panicking about AI—and not just because of what it can do, but because of how little they can see or control. Without governance and frameworks to support them, those leading security measures stand to potentially lose the most amongst company executives.

Living in fear: "CISOs and CIOs are so scared at the moment with AI and the new security pressures emerging," says Doug Shannon, AI expert and mentor at Insight AI, Brewing Insights!, and the Forbes Technology Council. "They’re thinking, ‘I’m going to be fired. Something’s going to happen, I have no visibility in this and I don’t know what to do.' They fear being the scapegoat and getting fired on the first issue."

One of the biggest problems, he says, is the black box nature of many AI systems. "You don’t know what the weights are. You don’t know where things are going. You don’t know how you're getting the answers out of it."

Cognitive offloading: That lack of transparency feeds into a broader issue: explainability. "Some people talk about ethics, but in general, it’s explainable AI that’s key. And we still very much need it, but we’re not seeing it yet." Without that, Shannon warns of a growing risk of cognitive offloading—the idea that people will stop thinking for themselves once they begin to over-trust AI systems. "Once we build more trust into them, and we say, ‘Oh, we just trust it,’ kind of like with Wikipedia—we shouldn’t, but we do—then cognitive offloading happens. We're at risk of not doing things anymore."