AI isn’t magic—it’s still software, and with that comes familiar risks that too often get overlooked in the hype. In this episode, Professor Peter Garraghan breaks down how traditional security thinking still applies, and why understanding the nuances of AI systems is essential to keeping them secure.
In this episode of our InfoSecurity Europe 2024 On Location coverage, Marco Ciappelli and Sean Martin sit down with Professor Peter Garraghan, Chair in Computer Science at Lancaster University and co-founder of the AI security startup Mindgard. Peter shares a grounded view of the current AI moment—one where attention-grabbing capabilities often distract from fundamental truths about software security.
At the heart of the discussion is the question: Can my AI be hacked? Peter’s answer is a firm “yes”—but not for the reasons most might expect. He explains that AI is still software, and the risks it introduces are extensions of those we’ve seen for decades. The real difference lies not in the nature of the threats, but in how these new interfaces behave and how we, as humans, interact with them. Natural language interfaces, in particular, make it easier to introduce confusion and harder to contain behaviors, especially when people overestimate the intelligence of the systems.
Peter highlights that prompt injection, model poisoning, and opaque logic flows are not entirely new challenges. They mirror known classes of vulnerabilities like SQL injection or insecure APIs—only now they come wrapped in the hype of generative AI. He encourages teams to reframe the conversation: replace the word “AI” with “software” and see how the risk profile becomes more recognizable and manageable.
A key takeaway is that the issue isn’t just technical. Many organizations are integrating AI capabilities without understanding what they’re introducing. As Peter puts it, “You’re plugging in software filled with features you don’t need, which makes your risk modeling much harder.” Guardrails are often mistaken for full protections, and foundational practices in application development and threat modeling are being sidelined by excitement and speed to market.
Peter’s upcoming session at InfoSecurity Europe—Can My AI Be Hacked?—aims to bring this discussion to life with real-world attack examples, systems-level analysis, and a practical call to action: retool, retrain, and reframe your approach to AI security. Whether you’re in development, operations, or governance, this session promises perspective that cuts through the noise and anchors your strategy in reality.
___________
Guest: Peter Garraghan, Professor in Computer Science at Lancaster University, Fellow of the UK Engineering Physical Sciences and Research Council (EPSRC), and CEO & CTO of Mindgard | https://www.linkedin.com/in/pgarraghan/
Hosts:
Sean Martin, Co-Founder at ITSPmagazine | Website: https://www.seanmartin.com
Marco Ciappelli, Co-Founder at ITSPmagazine | Website: https://www.marcociappelli.com
___________
Episode Sponsors
ThreatLocker: https://itspm.ag/threatlocker-r974
___________
Resources
Peter’s Session: https://www.infosecurityeurope.com/en-gb/conference-programme/session-details.4355.239479.can-my-ai-be-hacked.html
Learn more and catch more stories from Infosecurity Europe 2025 London coverage: https://www.itspmagazine.com/infosec25
Catch all of our event coverage: https://www.itspmagazine.com/technology-and-cybersecurity-conference-coverage
Want to tell your Brand Story Briefing as part of our event coverage? Learn More 👉 https://itspm.ag/evtcovbrf
Want Sean and Marco to be part of your event or conference? Let Us Know 👉 https://www.itspmagazine.com/contact-us
___________
KEYWORDS
sean martin, marco ciappelli, peter garraghan, ai, cybersecurity, software, risk, threat, prompt, injection, infosecurity europe, event coverage, on location, conference
When Guardrails Aren’t Enough: How to Handle AI’s Hidden Vulnerabilities | An Infosecurity Europe 2025 Pre-Event Conversation with Peter Garraghan | On Location Coverage with Sean Martin and Marco Ciappelli
Please note that this transcript was created using AI technology and may contain inaccuracies or deviations from the original audio file. The transcript is provided for informational purposes only and should not be relied upon as a substitute for the original recording, as errors may exist. At this time, we provide it “as it is,” and we hope it can be helpful for our audience.
_________________________________________
Sean Martin: [00:00:00] Marco
Marco Ciappelli: Sean,
Sean Martin: don't, don't look so surprised.
Marco Ciappelli: I am, I am surprised.
Sean Martin: Surprised that I, that I actually made it
Marco Ciappelli: uh No, no, no, no. I know you are already in Europe and uh, and I'm already kind of making my own plan for when we're gonna be in London. I'm wondering where are we gonna, you know. Where are we gonna go and, and turn on the camera and, and do something stupid or talk about interesting topic.
I'm thinking I want to go back to Abbey Road this year and, and do the, do the second take
Sean Martin: Second take in Abbey Road. I think, uh, the, the Marshall, uh, the Marshall Factory would be cool. I haven't organized that yet. I'll have to
do that this week.
Marco Ciappelli: well still music related,
Sean Martin: it is music related,
Marco Ciappelli: You know, it's good to get some music because all, all days our days are gonna be filled with technology and cybersecurity. Obviously. This is, uh, a pre-event, uh, coverage of our, uh, presence in London at infosecurity Europe. So, [00:01:00] you know, we, we try to sprinkle a little bit of what else ITSP magazine represent, which is technology and society as well.
So we cover everything, cybersecurity, technology, and society.
Sean Martin: have a podcast on music and technology,
Marco Ciappelli: There you go.
Sean Martin: uh, that topic keeps coming
Marco Ciappelli: Uh,
Sean Martin: funny enough, they enough people in the industry, so it seems many people play
Marco Ciappelli: yeah.
Sean Martin: either play or create music or connected to in some way, which is quite incredible. I'm, I'll just throw this out there.
Peter. Peter Garrigan, how are you?
Peter Garraghan: Hello. I'm great. How are you?
Sean Martin: very good. It's good to have you on. Are, are you, uh. Yeah. I don't know. You do you, do you, play a bowron in the, in the pub sometimes when you're, when you're in Ireland.
Peter Garraghan: If I have time to the pub, yes, I do play instruments.
Sean Martin: There you go. See,
Marco Ciappelli: There you go.
Sean Martin: There's, there's always music somewhere.
Marco Ciappelli: There he is for sure.
Sean Martin: All right, so
Marco Ciappelli: we're not here to talk about that. Although ai, it's involved in music [00:02:00] too, and I, I don't know about that part, but Okay. Let's, let's move on.
Sean Martin: Well, that, that's a different conversation. But certainly AI is used in, in a lot of things in business. And, uh, Peter's, Peter has a session on, uh, at, in Infosecurity Europe called Can My AI Be Hacked? And yeah, an intriguing question of course. And, uh, excited to hear your, your take on all this and maybe tease out your session a little bit, Peter, but before we get into. Uh, answering that question or teasing that question 'cause we want people to join you at the session. Um, maybe a few words about some of the things you're up to at the university and, and what you do on a day-to-day basis as well.
Peter Garraghan: Sure. Sounds good. So I'm Peter. I am a chair professor in computer science at Lancaster University in the uk. also a fellow of the Engineering Physical Science Research Council in the uk, which are a set of fellows who specialize as a research in the uk. Um, I [00:03:00] also operate a university spin out company called My Guard.
It's a startup in London that specialize in security for ai. I spent about a decade looking at cybersecurity issues within computing systems that use ai, so deep neur networks, looking at compiler security, network security, uh, application security within ai.
Sean Martin: Yeah. And then let me, uh, let me start with this question because mean, with the, with with the launch and uptake of chat GBT, that there's no question AI has gotten a lot of attention from the media and, and from organizations all over the, all over the globe. but. Clearly AI and machine learning underneath that has been around for quite some time.
And you just mentioned 10 years you've been looking at this. what's, what's the delta in terms of capabilities? maybe remove the graphical interface that most people are, have access to now, but are some of the [00:04:00] biggest differences in, in the models and the use of AI and machine learning and all the other stuff that you've seen over the past few years?
Peter Garraghan: Sure. Um, this is a whole research lecture about the history of AI and machine learning. Um, I think to be quite succinct about it. It is still software and that's no different from what we had for many, many years and we're going to type of the problems and the benefits of doing so. the easiest way to think about AI nowadays, what they really mean are deep neural networks, which is type of architecture that, you know, back in 2014 will shown to actually outperform a lot of other type of machine algorithm techniques.
Um, and then fast forward to today, people hear about AI in terms of, you know, chat GBT and live language models. I'll go into shortly, probably about a quicker analogy of history is really important. But fundamentally, different about nowadays AI and true for all types, types, uh, there's much more focus on generative ai, things that generate [00:05:00] content.
So generate words, predict words, and generate images. We've had predictive AI for many, many decades, being practitioner, those regression algorithms, the new regression is. Type of AI in the system. So one of the big differences now is that yes, it's become pervasive, ubiquitous, and it has to become commoditized enough that the everyday person can now use it.
As opposed to me done a library and trying to figure out how on earth do I do training? How on enough do I do my test, test training, splits? it's been kind of powered by that to use themselves through the system. I think maybe to set the scene, I love to give an analogy of what happened in another technology where we are now.
So. Cloud computing, I don't think people are gonna deny. Cloud computing actually is pretty big nowadays, both from a market and it's used everything, you know, even this core probably using the cloud to do so. The core technology that power cloud is virtualization. was actually invented in like the early sixties.
Um, the IDM mainframe machines could do virtualize, but they had two megabytes or megabit of memory and they're saying, well, we got [00:06:00] memory, it's too expensive. Interesting. It wasn't until the X six ARCHITECTURALS virtualize and memory came cheap enough. They say, okay, now critical mass. Now we actually can scale this out and spin up machines and kind of do have this, you know, scalability and the cloud community industry kind of really accelerated.
Now it's everywhere. I. And all the pains in doing so do as well, and we'll talk about security, but you know, the house had security problems as well. of ai, the artificial neuro network was invented quite a few decades ago. It wasn't until in 2014 that they could demonstrate that yes, it could outperform other type of ML techniques, and it became cheap enough and scalable enough to get this thing trained on lots of internet data. the paper oil needs attention came out from Google Research and that is transformer architecture that underpins all type of m. More agents and now it's flourishing and you'll give a few years time. Lots of hype about the space. Fundamentally, it'll become a very good technology and the but following that same, um, history and life cycle that you see in [00:07:00] virtualization, that's a few decades to get where we are now. think of AI modern nowadays, it's following a very similar trajectory.
Marco Ciappelli: Well, that was a nice history overview and uh, and, and connect to What I always think and, and point out when I talk about technology and society is you need the convergence of, I. Different technology and also the cultural mindset that that can actually then spark that tipping point. And as you said, you know, it becomes, uh, it reached that critical must where things are good.
But obviously we're talking about cybersecurity here. So among all the amazing thing that, that can be a lot of good for us, well those can be discussed as well, but we're not going psychological here. Um. It comes with a lot of risk, and, uh, so why, why don't you tell us that part of that story?
Peter Garraghan: So not seem to be [00:08:00] contrarian about it, but I spent a lot of time as a scientist trying to try to have view on things or as you trying to find something. You know, I, I hypothesis, I test it. Does, does AI models have risk? Yes, but the risks are no more serious than any other type of risk and type of security.
The difference here is that people don't understand it as much as they should do. Um, people have a habit of anthropomorphizing ai. It's not our fault. It's specula to fiction, it's human nature to do so.
Marco Ciappelli: Mm-hmm.
Peter Garraghan: haven't seen a new attack come from AI that you cannot see grounding in other types of security attacks.
Um, and that's a good thing. Um, we've been doing these type of things for many, many decades. So what type of risks in a model? Well, an ai, an AI model can do things such as it can generate bad output. That output could be information or it could be instructions, commands in the system that will cause. No problems of mark and ejection attack or data filtration attacks.
If I do the instructions set, um, I can inject bad code into the model itself and the artifact, and if it's opened up, I'm gonna have a vulnerability in my organization. [00:09:00] can even range from if it's the air is being used for detection, I can ex, I can bypass and be find blind spots. And for those with security experts, this is not new source.
We've been doing this for many, many years, but the difference of the AI is. How you do it is different. So there's lots of analogies between SQL injection and problem ejection and we'd be claiming SQL injection has been solved for decades, it still happens all the time. ejection has a lot of analogies, but how you actually do it, it's quite different.
Um, two reasons. One is the problem with prompt injection is that the data and control pain and the lab's not the same thing. There's no separation and it's problematic. The guys philanthropic and Google tried and they can abandon it pretty quickly 'cause it was too hard to do. I'm sure they at some. Um, and also means that the, the amount of combinations of instructions you could give and outputs is near infinite. you're basically trying to look, you're trying to wrestle an API with natural language, and that's very, very complicated. So when you think of risks of models, I always say to people I talk to, whether it's [00:10:00] governments which I support, but also people I work with insiders, even customers. Replace the word AI with software and ask questions like, what's my risk? The answer is, yes, there is risk. Um, the difference is there are some nuances, new technology, and maybe the last example I give you is think given the AI itself is intrinsically opaque and stochastic, it's very, very random. But I can't do code level on a neural network. It's a bunches I can't code. So. When a new tech comes out, like ai, you have to think about what, how do my current controls apply and risks apply, and they do apply. The risks are still the same. The hype matters differently. update tooling, update training and playbooks. And every time this comes out, um, I think one of the problems is that the, because it's AI people have very big claims and imagination of ai. I think they so forget actually cybersecurity attacks in computing systems and data security still apply. But how they work in AI is slightly different.
Sean Martin: So [00:11:00] I'm gonna ask you this 'cause it, it's been. It's been a topic debate. Might even be, I certainly haven't argued, I've certainly had some debates, which may be a little strong, but certain definitely conversations around this idea of, of, you use the word infinite. you use the word opaque. And to me, when I think of AI and, and the systems that are built around it, you, you create world that's very hard to contain. Very hard to describe. Here's what I expect it to do, here's what I expect it not to do, I can kind of box that in and, and wrap my risk modeling around that. traditional software, when you, when you drop the AI and the models in there and, and agents and all kinds of stuff, it, it seems to get unwieldy. And so the debate has been, is it, to your point, is it really just software? And or do we need to treat it differently, um, with respect to how we manage risk and put our controls in place and write policies around it. [00:12:00] So your, your thoughts on, on all of that.
Peter Garraghan: Sure. So one of the biggest difference is that interfaces we we can make AI is more natural language esque. opposed to very rigid protocol esque, and that introduces some confusion, my experiences, a lot of the attacks, I'm still relying on my experience of cybersecurity system security, pen testing, and red teaming.
But I need to figure out, oh, in this case I need to use sut, different tactics. Um, we talk about social engineering, the LLM to do things. What I'm really doing is doing. You know, perturbations or prompt injection to find a instruction I want, which I can also do by fuzzing. That's all, that's the same analogy of that system. I think one of the important things in this space is take a step back and think about software and ai, people have a habit again, of aph just throwing out basic AppSec principles that applied to ai. So actually a problems could be. Um, but you, to your point at the [00:13:00] beginning about how we build applications and systems.
So normally how we build these things is we start with nothing. define a use case. We do an architecture. We think about the non-functional, functional properties. We think of fret modeling, and we start building out our capabilities. Every time we add a new function or functional property, then we figure out, okay, this is actually good.
So over time the system gets more complicated. Eventually you'll, you'll miss something and the security law breach happens. So you're kind of building from the ground up, bottom up, build systems and that's about tolerance. That's for security. The problem with AI though, is that you're putting in a software component that's crowned full of stuff.
You don't need I the component, let's say I have an application generate, all I want to do is send no generate emails from me. That's what I want to do. By download a piece of software that can write songs, can write poetry, can do encoding, can do different languages. If I took a step back and said, I'm gonna build an app from the ground up, I've crammed in 50 features.
I don't need that, say. My problem manager will say to me, why are you doing this? 'cause not only is a waste of my time, but [00:14:00] also you're just huge bunch of new risks that I have to go test. So the difference here I've seen is instead of having, starting from very basic and building things up and you start, you're testing threat modeling, you're taking this huge thing that has things you don't need and trying to cram it and trying to squeeze it into things you need.
And that's more of the differences I've seen in terms of like looking at risks within AI systems. You've got nine of the stuff you don't need and then the attack can get that out of the system and cause problems.
Sean Martin: Does it Ag agent AI maybe help in that sense purpose building.
Peter Garraghan: Yeah,
Sean Martin: maybe blocking stuff from
Peter Garraghan: So, again, to cut through the hype of marking this space, software agents are on new concept. We've had software agents for like 20 years, 25 years. Um, we've, we've had, um, restful services, so as an architect has been long, long, long time. now is that we can put transformer architectures within software agents. instead of having very well-defined APIs that I have to test for soap or rest, it's using now actual language that makes it much [00:15:00] more complicated. But fundamentally, the attacks are still the same. I, I, I was at, I was at BSides and RSA, a lot of discussions about ag agentic, AI security, and I looked at the architecture thinking. They slap the word a genti in front of everything, agen, prompt injection, a gen age, gen poisoning. And say, actually, what is the difference here? It's still a transformer, it's still capability. So actually the techniques are attacked the same. But the difference is the UK is slightly different. So in agents, we typically talk about action. So agents are a strategy, do action. So I give the analogy of a bus and an F1 car are the same thing. In principle, they have engines, they have wheels, they have four of them, I hope. But the type of attacks and threat you're gonna prioritize are different. So in a, in a gen. In agents or agent workflows, are also gonna use attacks, but they'll be specialized against it versus, you know, an, an agent system or even just, or AI model system attacks. You prioritize different type of risks and threats face of this. But the difference really, agents is software agents aren't your concept, but the use, use in natural [00:16:00] language and the use of a, like agents is to do action. type of things you worry about, like tool calling or um. To other tools becomes much more a threat versus my LLM that's in my local machine with no internet access.
The Fre, the fret modeling, Andre attack is very, very different.
Marco Ciappelli: Well, I think this could be easily a 45 minutes to an hour conversation. 'cause I mean, it, it, it, I, I, yeah. Let, let's get to the session and, and again, I mean this is, this is something we should definitely go deeper into. Uh, 'cause I feel like we, we are just going the wrong direction. Um, as we often do, because marketing also has just adds stuff to it, then eventually we will remove it.
We hype what we can do and what we cannot do instead of be careful. So tell me a little bit more about what exactly what, what, where are you going to focus amongst all of this in, in your, in your presentation.
Peter Garraghan: Sure. [00:17:00] So I'm presenting, uh, InfoSec Europe. That will be on the 4th of June, and I think it's on the cyber trustee stage about. 4 25 I think a b hacked. The answer is yes. Thanks. Thanks for coming, but no, seriously, you should come, actually come along and actually ex I will explain why it is and actually what's nuanced about it. I think that the main takeaway is that a cybersecurity practitioner, a lot of your skill sets will apply. You understand this conceptually, the techniques are different. Um, and I want to try and explain that to why is it different and how is it the same? We've been for this before. We've seen a new tech comes out and these, the I give people is think of a giant sieve full of flour. When a new tech comes out, people claim to do everything. You know, virtual like cloud could do everything. Everything should be cloudified. No, it'll make everything great. What we did is we got a SVE and shuffled it. A few pieces stayed and each of those PIL pieces became billions and billion dollar industries, and we focused the threats and the, because they're serious. In the world of ai, [00:18:00] people are claiming it does everything and you sh you shuffle the civ. or four, maybe six things will come out and they'll be worth a lot of money. But what I want to do in this space in my presentation is talk about what are the type of actual risks that I see on the day-to-day basis and attacks that we launch and we find from research in the company, talk about the grounding of why it happens and kind of give reassurance by saying that if a cybersecurity practitioner. It's not magic, it's still software, but you need to update your training tools on playbook. And I'll try to give some actual examples because the other problem in this space is people talk very conceptually about problems don't actually give you a lot of systems level problems. Here's actual attack we can show off.
So hopefully the presentation I could demystify on the concept actually show what's happening on the ground today.
Sean Martin: And you speaking to there? 'cause clearly there's, there's cybersecurity operations, which you can have endpoint management, network management, data management, and protections around all that. Then you have the. The, uh, well, yeah, then [00:19:00] the, the detection team, and then the response team perhaps, and the soc you have the rest of the business, who by the way, can all this stuff without including security.
So are you speaking to practitioners in security, speaking to business owner, line of business owners who, who, who's the best target for
Peter Garraghan: Yeah, and I speak to pretty much everybody I think in this space because again, people get their imaginations captured by AI and kind of actually overlook pretty common. Problems that in terms of risk. let's pick a random example. So think of guardrails that that is a very common phrase. 'cause I'll use a guardrail, it's fine. You say to them, would you rely on a a web actually file or a WAF security ization saying, no, that's silly idea. Say, well, same thing for ai. You can't just slap a guardrail and say it's okay. There's a whole bit of other problems involved from a system security risk compliance. There's a whole bunch of issues about that.
When I talk to data scientists who love getting AI to do high performance, high quality things. Trade in type risks. They think about risks and societal risks. Actually say to them, actually, I can [00:20:00] use this to do Mark injection. And they're like, connect to our system. That's all a problem. They, oh, I didn't think about that. Um, when I talk to business leaders about this, I say to 'em, take a step back and say you have an asset that you can't evidence risk, you can't find our abilities. You have no tooling and you're kind of applying your basic controls to this. Why are you doing, why are you doing this? And they think, okay, oh, it actually makes sense.
Now I probably should be, so I'm actually, one thing I'm advocating strongly to them is a step back, look at all your standards and current controls, whether it's in GRC, just you know, governance, risk compliance, whether it's security teams, even just app developers and say to them. follow up the word AI with software or data and say, should I do X?
The answer is probably no or yes. And and I think that really helps 'em understand. But the problem this space is that a, people are overselling, some they capabilities and in some cases it's fantastic in some use cases, other cases, jury's still out there. It's a variated space still. Um, and I'm hoping the ones who actually are getting AI into production, they are seeing real problems and the lot problems they see [00:21:00] are. If they haven't applied basic controls of application development and or security controls, some of 'em have done a great job, but then they're realizing then I haven't got the tooling or the, the knowhow because I AI expert to actually address this.
Marco Ciappelli: Very cool. All right. Well. I am gonna say this, which is lately, I, I like to say we go to these conferences because we look into the future. Uh, we go for the community. We also go to kind of maybe, hopefully kind of remove what is I. Talks, maybe focus on what the priority should be. And I think by putting together academics and vendors and, and technologists and, uh, talking about job, job politics, they're gonna be session about that.
And, you know, you put a lot of different mind together and I think maybe we all walk away with a little bit more of a clear. Picture of where we are standing in [00:22:00] cybersecurity. So that's my hope. I think Infosecurity Europe is been always doing a great job in doing this. I'm excited to be there. I'm excited to meet you in person.
Maybe even chat a little bit more. And, uh, and for everybody that cannot be there, they should definitely follow us, Sean, on uh, on an on location. Uh, not much for you, but for me, yeah.
Sean Martin: uh, itsp magazine.com/infosec. 25 Is the, uh, the URL uh, can my AI be hacked as a session that, uh, that Peter's gonna be presenting again Wednesday, 4th of June? I. 4 25 in the afternoon, what is it? The cyber strategy stage. Look at that. of course, we'll be there all week, uh, covering the event, talking to keynote speakers, uh, analysts, uh, some vendors as well.
So we'll bring all the innovations, all the, all the mindsets, all the, the thought leadership that you want from Info Security, Europe or London. [00:23:00] And, uh, yeah, Peter, it's fantastic meeting you. Thanks for sharing. Uh. Your, your session with us today. Hopefully, hopefully you fill the room and get, get a lot of interaction with folks. Um, for me it's about shedding light on, on what's new and removing the, the mystery and finding, finding a path to take some steps forward to, to do, do things in a way that the business can grow and flourish and generate revenue, but then also in a way that we protect that revenue and growth. So thanks again.
Peter Garraghan: Thanks, bro.
Sean Martin: And everybody listening in watching. Thanks for joining us, and, uh, we'll see you on location and just stay tuned, subscribe, share with your friends and enemies. We'll catch you all on the next one.