When AI watches everything, who watches AI?

With AI encroaching on so many aspects of our lives, it's natural to be nervous about the implications. Our jobs face potential automation, governments are struggling with regulatory frameworks and our security and privacy depend on AI being properly guard-railed. What's being done to ensure we can protect ourselves in the midst of this uncertainty?

Jun Seki is an Entrepreneur in Residence at Antler and CEO of a forthcoming stealth AI startup. He sat down with us for a discussion about the challenges of establishing true safety within AI systems, and the emerging strategies being developed to eliminate threats.

Fear of the unknown: At the heart of AI security concerns is a level of mystery surrounding its development. "AI is a black box. It's pre-trained. So how do you evaluate the model, the safety and the response?" asks Seki. "There are so many models being launched, how do you know which models are poisoned, which are not? Let's assume the majority of them are poisoned. How do you know which ones are safe?"

Zero trust is not enough: The cybersecurity industry has long advocated for zero trust policies, but even that approach may not protect against the risks posed by AI. 

"Zero trust means you need to verify everything—every transaction, every action," Seki explains. "Looking at recent cases, despite having multiple parties approve transactions, organizations are still vulnerable. Attackers have initiated highly sophisticated social engineering or deepfake UI/UX scams that look like legitimate transactions."

Security professionals find themselves in a tough situation, and Seki suggests several important questions they need to ask if they are to ensure their organizations are safeguarded. "No matter how much zero trust you enforce, you still need additional security frameworks on top of it. How do you identify deepfake interfaces? How do you verify underlying transaction links? How do you examine the source code to ensure you're not approving something malicious? These are the challenges we face even with zero trust principles in place."