Öykü Işık on Cybersecurity, AI Risk & Governance

Great Expectations Podcast – Episode 3: Öykü Işık

Öykü Işık:

One thing I'm always talking about when it comes to especially the intersection of AI and cybersecurity, that is not a dichotomy, right? AI can and organizations should be looking into using more AI in their cybersecurity processes, but we should also be aware and prepared against AI enabled threats. But I think there's a third dimension that I hear less from organizations, and that's about if there's so much appetite for AI that makes me think that AI is becoming a strategic asset for many organizations, which means something else to be protected.

Paul Sephton:

Hi, I'm Paul, and you're listening to Great Expectations, a Jabra podcast that's your trusted guide to navigating AI's impact on the workplace in a world full of AI noise. We cut through the hype and deliver the information you need to make smart future-proof decisions about ai. I'm excited to be talking today with a leader at the intersection of cybersecurity and ai. Öykü Işık is a professor at IMD Business School and leads their cybersecurity risk and strategy program. She's worked with organizations such as MasterCard and BNP Paribas Fortis to tackle cybersecurity data privacy and digital ethics challenges. She's enabling CEOs and other executives to understand these issues, which she believes isn't just a technology topic, but a business priority. In today's episode, we'll be talking about the biggest cybersecurity threats in an AI era, how organizations can keep their data safe with the right governance and the one dimension of cybersecurity that most businesses are still overlooking today. Thank you very much for joining us today. I'm rather excited to have this conversation after a lot of time planning it.

Öykü Işık:

Likewise. Thank you, Paul, for having me.

Paul Sephton:

And I think as a starting block, who'll be really interesting to find out sort of what your current role is at IMD, how anyone gets into the field of cybersecurity before we dig into the many topics that we can uncover today.

Öykü Işık:

Sure. So officially my title is Professor of Digital Strategy and Cybersecurity. Here at IMD, I've been mostly doing my both research and teaching in cybersecurity AI and AI ethics or rather responsible AI governance topics.

Paul Sephton:

And I can imagine that for people like myself, cybersecurity has quite a one dimensional sort of association, but it's far more complex and has so many more angles to it. So how would you describe the field today in all of the areas that it covers?

Öykü Işık:

You're so right, Paul, traditionally even the name with the advent of let's say personal computing since we started having PCs in our houses and before that with enterprise computing, we always had this topic, we used to call it computer security, then we call it internet security, then we said information security. Now cybersecurity is more the dominant name, even though if you ask the I guess expertise, you would still hear these terms coming back and forth. But the more we go through digital transformation as organizations, as society, by default, the implications or as we say, the attack surface grows, and thus it becomes more and more relevant as we continue depending on digital tools that we have in our lives, whether for business or for personal reasons. So the more digitalized we become by default, the more cybersecurity becomes an issue because the number of vulnerabilities and the things we use, the things we do is increasing.

Paul Sephton:

And I think these days, sort of like you mentioned, it can either be from a business perspective or a personal perspective. We see these stories as sort of like a scammer call center being uncovered from a consumer perspective or people in knowledge work or enterprise might think that sort. I'm just in this small position in an organization. I'm not someone who's at risk or a target. But I'm sure in recent years you've seen some examples across companies that show that there are a number of weak points within any enterprise and that there must be a lot of examples you can think of when it comes to what's been happening lately in terms of cybersecurity and the types of attacks that take place.

Öykü Işık:

Yeah. You also asked a minute ago what is changing? I think at the fundamental level, when you look only leaving aside the changing nature of technology, when you look into scam centers or threat actors, what they do fundamentally, what they do is not changing, but how they do it is changing, right? Scamming is scamming at the end of the day, whether it's a scam, a call center, or it's a phishing email, or it's a deep fake attack that is really enabled by generative AI of today. At the end of today, they're all trying to play one of the six persuasion methodologies that are also validated by scientific research. Either they're playing the authority card like, oh, I'm calling you from the police department. Your identity has been shared with us as a part of a, I dunno, criminal group, whatever. This is going on quite a bit.

I hear, or whether it's someone calling who sounds like your grandchild and telling you as the grandparent that they have been involved in a car accident, they need immediate financial assistance. They're asking you to send money, which we hear, I think you're happening a lot, especially in us, this grandparent scams. So what they're doing at the end is not really changing. They are using certain triggers to persuade us to make really impactful, mostly financially impactful decisions at that moment, having the sense of urgency. But how they're doing is definitely changing. And deep fakes audio fakes enabled by generative has given them unfortunately, a big boost about this.

Paul Sephton:

I think you point out the emotional blackmail that can take place on a personal perspective, but when it comes to organizations, because like you say, from an individual point of view, I might think, oh, it's my data that's being compromised or my social security number if I'm in the US or whatever that might be, or my personal bank account. What are the biggest risks to enterprises that they face around cybersecurity and the threats or consequences that can happen?

Öykü Işık:

Yeah, actually when you look into what we call a threat landscape, this landscape more or less differs for each organization depending on where the organization is, depending on the industry, depending on the size and the resources, different attacks may be more relevant than the others. But at the end of today, what we see today is everyone is suffering from DeepFakes. And the very similar example that I just gave is typically turned into a format where a C level is imitated by the threat actors. So for instance, I remember a story, I think it was earlier this year of a Hong Kong international firm, Hong Kong office, a finance employee being invited to a zoom call where they were asked to transfer a huge lumps money, like 25 million. And during the Zoom call, he thought he was talking to three senior executives, one being the CFO of the company. So not one person, three people.

And even though he was a bit suspicious, and he reported after the call immediately, he transferred the money, he did what he's being asked of because they were pushing him, they were playing that authority card. And now in hindsight, we know that all three were fakes. It's amazing to show us the length that these threat actors can go on one hand. And on the second hand, actually how accessible this technology is. There's so many scams involving DeepFakes going on. So that is one really that's the AI enabled attacks that we see for enterprises as a high impact threat factor. But I think the other one that I have been studying also for, especially since the pandemic that we saw a spike in, is the ransomware attacks as a concept, as a threat type. Ransomware is not new. Actually we can trace it back to even early nineties.

But with the availability of course, currency has been like a fuel to the fire. This easiness of money transfer has shown many tech-savvy and non-tech savvy criminal organizations that there's actually money in this business. So there's a growing cyber crime as a service market where these capabilities of launching a ransomware attack or launching a phishing email attack, launching a denial of service attack is a capability you can purchase. And without knowing any technical skills, you can actually still do pretty good damage if you have a grudge against an organization, for instance. So this is, I think not just ransomware, but the growing cyber crime as a market business is one of the biggest risks for enterprises today. So

Paul Sephton:

If I'm trying to square off the biggest number of threats with the smallest number of activities, are there sort of three to five things that all companies should be looking at doing as a foundation that will almost not every time, but cover them off towards the majority of threats? Like say a phishing based email attack where you can take a few steps, VPN firewall, whatever they are, and automate them to the point where you've built a really strong foundation against this type of threat?

Öykü Işık:

I would be hesitant in generalizing, you know what? Because every organization has different crown gi. So what they want to protect may be different than the nextdoor organization, but I think one factor doesn't change, and that is the biggest attack surface for any organization that's us, the users of the tools, of the applications, the organizations offering. So any investment, in my opinion, in changing the behavior of the end user, and I very consciously am not using the term awareness. We are past that. I think awareness trainings and so many research after research shows that there's a gap between awareness, intention and behavior. The awareness does not always lead to good behavior. So that's why I'm saying any organization that invests in properly and positively changing the behavior of its employees will have a greater return on investment compared to any extra security layer that is enabled by technology.

So 85th layer that you invest in with yet another cybersecurity technology may bring you less benefits compared to a smaller investment in better security culture change programs. I think we have a lot to do still in terms of one, making this topic accessible, making everyone feel comfortable talking about this, making sure they all have basic understanding of hygiene, so all the content that is in the training programs for most organizations today, but also making sure that this is, like you were saying a couple of minutes ago, this is not just a technology topic, but this is actually a business topic because digital transformation is a top priority for everyone. So making sure that we are comfortable and making it accessible and that everybody feels safe enough to raise the questions and ask about this issue, I think that's the best investment that any organization can make today.

Paul Sephton:

And that probably brings us into, I think digital transformation is a nice pivot into AI and that topic because if you Google AI cybersecurity and you look at news results, there is an endless mix of new features and solutions from IBM or Nvidia or kind of focusing on AI being used for good to protect against cybersecurity. And then articles on the other side saying, threat is higher than ever, risk is greater than ever before. Attacks are higher. So it seems to be a technology that's being at an equal pace leveraged for good and evil, for lack of a better contrast.

Öykü Işık:

Yeah, it's a double-edged sword. It's AI is a friend and fault for cybersecurity, as we say. And so it then suddenly becomes a cat and mouse game. Whoever uses the first gets the first mover advantage for a while, and threat actors are much more agile. We have to admit and go faster and innovative compared to organizations because organizations have certain constraints that they have to function within. They have regulations, they have bureaucracy, all those things threat actors don't. And they are also pretty good in innovating. We have to give it to them. I mean, when you go to the cybercrime as a service platforms that sell these tools already do the fine tuned large language models, LMS, fraud, GPT, warm, GPT, fine tuned large language models that are designed to create effective phishing email campaigns, or they're designed to create hard to break malicious software applications.

So we already see those things, and indeed, these are mostly what we may see in the headlines, but I think cybersecurity as a domain has been benefiting from machine learning for quite a while already because ai, the ultimate capability of traditional, especially machine learning, traditional AI, is pattern recognition. And that's the best thing for identifying malicious activity most of the time in our networks. The real time fraud detection capability, let's say in the financial services industry is there thanks to ai, AI related developments these organizations have been investing in. So definitely AI is also part of the solution. When you think of cybersecurity as a domain, and especially when you think of the talent gap that we still have in cybersecurity, especially for global south, this has been a topic of big discussion, but for most countries, there's more open positions than cybersecurity professionals in the market. So there's definitely more automation that's needed to fill that gap in the industry so that we can use people for tasks that cannot be or should not be automated.

Paul Sephton:

And if you are looking at the skills gap or skills shortage in the labor market, what are some of the promises that AI might deliver to be able to say, okay, if I'm a small medium business or I necessarily had the capacity to find the right individual because there's a skill shortage or the budget to set aside to truly protect myself, is there something different I can do today than I could have maybe done five years ago? Thanks to the positive advances in ai,

Öykü Işık:

I think there could be many direct and also indirect benefits for generative ai. I'm hearing very interesting use cases on simulations. So how do I simulate a network analysis exercise or how do I simulate a tabletop exercise only for my top management team that is a family owned business that I am just one IT person and I don't have the budget to hire, but how can I use chat GPT, let's say, or any other LLM as a sparring partner so that they give me answers, they give me ideas on creating the most effective two hour session that I can have with these executives and make sure that they get out of that session with the most important ideas. So something that would take an IT professional a day or two to design and put together can be done in a matter of an hour or so with such large language model tools.

So I think pure access to information and access to good ideas and access to creative capability that comes with these things by itself is already a good enough resource. But I also expect that what we are now seeing is more and more agents meaning more and more specialized generative AI capabilities in the market. So we are also very likely to see cybersecurity focused agents that would really be much more accessible for many of these small organizations that they can benefit from rather than going to a big vendor and have to invest in large scale enterprise security systems.

Paul Sephton:

And when it comes to LLMs that we've been discussing, I think on the flip side, a lot of organizations have been wanting to deploy or unsure how to deploy a GPT based model or something custom even, because ultimately it's bringing in all of the biases of an external model. And there's also a big concern around data security. So what I put in, how do I protect that, what are some of the risks of individual IP or company data leaking out and also the baggage that comes with an LLM that even if you adapt, you're bringing in.

Öykü Işık:

Yes, I think that's a great question, Paul, and probably we still don't have a very clear cut answer to. And the part of the reason why we still don't have a clear cut answer is the lack of transparency in the industry, I believe. So if I don't really know what OpenAI does with my prompt, then it's kind of like a big gray area. So what I think I remember reading it was Samsung employees that copy and pasted proprietary code into the open version, right? Really accessible version that was in the news a couple of months ago. And now OpenAI for example, is one highly accessible LL capability provider. They say if you have a paid account, then you can actually tick off a box and say, don't use my prompts for retraining your next large language model. But I also hear people who are skeptical about that because if you hear some of the executives of these big tech, they say, well, we really need good quality data, we need more data.

So the question then automatically follows, well, could there be a scenario where they legally find a way to still use our data? So there's I think a bit of a trust issue also there in the market. But of course they also now say that for the freely accessible versions, they do collect all the interactions in the name of improving the capability as well. So clearly there is more transparency needed there to a bit guide the organizations. But I also hear at IMD, we do hear more and more organizations going down that route. So I'm not going to just go ahead and use an external service provider. I will bring this capability internally, I will have my own GPT that way I will have the capability to build my own guardrails around it. Maybe I will fine tune it with my own organizational content that will also increase the consistency, increase the accuracy of the results a little bit.

As you know, that's also an issue with these large language models, hallucinations or right about wrong output is a thing, it can happen. So that's another way of improving the results. So that is something I hear more and more, but I just want to emphasize what you said. When you do that, all the inherent weaknesses or vulnerabilities of an LLM becomes yours, right? So if you are a Microsoft organization and you are using GPT-4 in your Azure environment, you can make it better, you can make it more secure, you can fine tune it, but all the inherent capabilities as well as weaknesses are also now relevant for you. They become your weaknesses as well. And if you don't know about them, how are you going to protect yourself from them?

Paul Sephton:

And that's something that we can't easily find out about without this transparency coming from the LLM manufacturers or developers.

Öykü Işık:

Yes. Yeah, there are thriving communities out there. I dunno if you're a er for instance, jailbreaking ChatGPT. What kind of prompts can I use so that this LLM overlooks the instructions it's received in terms of not answering certain types of questions. And it's very funny because I follow these things very closely because it's always interesting to have a demonstration in the classroom. But I realized also, let's say open AI researchers, it's like vultures. They are always looking for researchers finding out these new vulnerabilities, and they're immediately of course, and nicely patching these vulnerabilities very quickly. So there's a lot of, I think patching going on in the background we don't necessarily hear much about.

Paul Sephton:

So if I'm wanting to use generative AI or deploy it at my company, because I think that the productivity gains are going to definitely be justifying the deployment of that new software, should I be investing in going the extra step and developing my own model? If I'm paying for a premium model on GPT-4, oh, is that going to be sufficient or what steps do I take to, I think everyone does have that Samsung example, top of mind of what could go wrong, what's the most pragmatic way if I know that I want to commit to that decision to go about making sure that I'm doing it responsibly.

Öykü Işık:

Yes, I think that's really the keyword responsibly, which immediately implies a lean and manageable governance mechanism in the organization, right? Because some of these, I think weaknesses can be minimized if we make sure we have certain rules or processes in the organization that tell us some good practices and not so good practices. So coming back to investment, I'm not sure if there's one easy way out of this. What we know is that there's a, first of all growing concern around the sustainability implications of training as well as using large language models. So probably going for your own from scratch is not necessarily a good idea because we also have many open source, large language models out there. But maybe using one of them out there and fine tuning with your data is definitely much better than using the vanilla versions, that's for sure. But I would even question and say, well really do make sure you need a generative AI capability for whatever problem you're trying to solve.

Is it an optimization issue? Is it about innovating something new? So I would really encourage any executive to be very skeptical about, do you really need to use AI for this? Because we are going through such a hyped phase that the saying to a person with a hammer, everything looks like a nail. So it feels a bit like that. And clearly we need to think about long-term implications of investing so much, so much into this. So I think a low hanging fruit is really a proper governance mechanism and really asking the right questions to the vendors that you're talking to and maybe interested in investing in their capabilities. So really asking about where they get their training data, asking about the key metrics they follow, asking about how they train the data and what kind of risk mitigation strategies they're using. Even asking the right questions. These vendors can reel quite a bit for organizations.

Paul Sephton:

And then how do you layer on top of that an ethical approach? Because that's almost a separate, right? We've deployed it, we've managed to get some governance in place, we've got scale, we've figured out what the right use cases are. Let's take a step back. Are we doing things ethically? And I know that some people are saying in the US it's the wild west, there's no regulation in the eu, there's perhaps more development. What is the responsibility of employees of organizations when it comes to AI ethics?

Öykü Işık:

I think indeed, we will see a very active field in the next years. Now EU AI Act is official, but it'll be enforced two years later. So organizations literally have on average two years to be compliant with this, right? So there is indeed a regulatory enforcement coming our way. I've heard pessimism about this quite a bit, saying that this will slow things even further, but also some optimism saying that clearly self-regulation in the situation of big tech doesn't work. So we need some oversight.

Paul Sephton:

Can you unpack what the EU AI Act is, what a company is going to have to comply with? What does it cover?

Öykü Işık:

So in a very simplistic manner, I can say that EU AI Act is a very citizen-centric European data. Let me summarize it that way. Centric way of risk management. It categorizes AI applications by risk category. There are some AI applications that can be considered minimal risk and they wouldn't even be regulated. So for instance, if you're using a machine learning application, and if you're training this by data created by the machines in your manufacturing environment that has nothing to do with human personally identifiable information, then you probably won't be regulated by EU AI Act. On the other extreme, it bans some use cases. For instance, it says that, let's make sure we don't ever experience any other issues such as Cambridge Analytica. So you cannot use AI for behavioral manipulation. You cannot use AI for mass surveillance or for individuals credit scoring in a society, right?

So there are certain band use cases, and in between there is high risk and low risk use cases. High risk cases are things like, are you using AI in law enforcement? Are you using AI in recruitment? Are you using it in legal processes? So these are high risk cases and all the other more private company use cases such as having a chatbot on your website that interacts with your customers, this is low risk, most likely to be low risk concentration. But of course then you have different expectations. If it's a low risk, maybe you just need to show there are transparency expectations. But if it's high risk, then the chances that whoever is enforcing this in your jurisdiction, they may come and audit you. They may come and audit your algorithm, which means you really have to work on an audit trail saying that here is where my training data comes from, here are the people who have developed the algorithm, here are the trainings that I put these individuals through so that they know the biases they have to mitigate. They know what kind of ethical issues they need to be aware of. So it's much more intensive operational transparency that you would need to provide to the auditors. So that is typically what the A U AI Act intends to do, really ensure there is transparency in these AI applications.

Paul Sephton:

And if I'm a company who hasn't necessarily paid attention to it and I've got a legal entity registered, even if I'm not headquartered in Europe, but I'm operating in it, are there sort of some existing logs I should be keeping or ways of suddenly going back and going, let's just log this data, keep an archive of it, or make sure that should I be audited in the future, I've got some type of paper trail of compliance.

Öykü Işık:

I think the word data governance or data management becomes, most organizations have some initiative for this, right? For data governance. So everything that relates to that will be relevant for EU AI act as well. And you also talk about indeed, you don't have to have headquarters here, but I think it's fair to say that if European data kind of passes through your systems, either your AI is facing any Europeans or you trained your AI capability with the personal information that may be coming from Europe, then you would be responsible for this. So really anything that's about managing the data, anything that's about managing the algorithms and anything about cybersecurity, securing these algorithms and any information you can hold onto these topics will be relevant for sure.

Paul Sephton:

I'm curious about the next step looking forward, and we've covered things like the ultimate weakness for most organizations lies in the people, some of the trading that you can do to sort of tackle or prepare for most of those threats. But do you think that that landscape is going to change or is there any sort of golden advice that you regularly give out in your teaching or to organizations just because you see it coming up time and time again as a core issue for businesses are facing

Öykü Işık:

DeepFakes we talked about, right? Or now researchers are actually finding out that it is possible to generate machine learning enabled malicious software. One thing I'm always talking about when it comes to especially the intersection of AI and cybersecurity, that is not a dichotomy, right? AI can and organizations should be looking into using more AI in their cybersecurity processes. There's a lot of value that can be uncovered there, but we should also be aware and prepared against AI enabled threats. DeepFakes we talked about, right? Or now researchers are actually finding out that it is possible to generate machine learning enabled malicious software, which would be a bigger problem to deal with. But I think there's a third dimension that I hear less from organizations, and that's about if there's so much appetite for ai, if everybody is really interested in investing in ai, that makes me think that AI is becoming a strategic asset for many organizations, which means something else to be protected, right?

And when it comes to protecting algorithms, it's a rather new territory for organizations and cybersecurity things to think about. So it's not just about using ai, but also realizing, well, if I have this proprietary algorithm that I am investing in, it's a new business for me, it's unique. Well then there will be threat actors potentially that's interested in understanding how that algorithm works, stealing that algorithm, sending what we call adversarial prompts to try it and bring it out. So it's also important and interesting as they invest in AI to think about how do I actually protect this as a strategic asset that may be a little bit different than protecting only the data and only the hardware and this or that. So I think there's also more innovations that we may need to deal with when it comes to protecting our AI as an asset in the organization.

Paul Sephton:

And is there anything else that you see executives doing now or organizations that makes you shake your head and kind of go, it's no longer excusable to do that in 2024, 2025?

Öykü Işık:

Absolutely. I think still going for very traditional cybersecurity thinking is unacceptable. Meaning that traditionally we have been investing a lot in defending and protecting the system. So it's very much inside out perspective. I protect what is valuable, I keep the bad guys out. But with all these changes, with the volume increasing out there in terms of attacks, because as I said that there is a lot of money in this and more and more traditional criminal actors are attracted to this business. So every organization is constantly under attack. So maybe we should be much more interested in our resilience capabilities saying that, well, of course I'm going to invest in protection and defense, but as much I should invest in detection and response capabilities so that if somehow a malicious actor goes through, I am detecting it very quickly so that I minimize the damage, I can immediately isolate the incident and I can immediately recover from this incident. I responded to this attack and I recover from the damage of the attack. So I think are capabilities, it is certainly getting better. I hear more and more crisis management simulations. I hear more and more ethical hackers being higher. So there's definitely an interest in proactivity, but I think that we have a long way to go to really say, we don't talk about traditional cybersecurity anymore, but we really talk about digital or cyber resilience in the organizations

Paul Sephton:

Almost accepting that the threat will be constant and growing. And you've got to make sure that you're not just defending against it, but that you're not crippled by it if suddenly it does kind of impact you.

Öykü Işık:

Exactly. There is no reputational damage today, especially compared to let's say five, 10 years ago of being attacked. Everyone is being attacked, but there's a lot of reputational damage if you don't handle that breach well, and we have out there today, unfortunately, more bad examples of how not to do it than good examples.

Paul Sephton:

It's a fantastic food for thought to end today's episode on. So thank you so much for the time and joining us. It's been a really interesting hour to uncover your insights.

Öykü Işık:

Thank you, Paul. I thank you for having me, and thank you for the super fun conversation.

Paul Sephton:

And that's it for this episode of Great Expectations. If you want to find out more information about implementing the right AI solutions in your workplace, see more episodes or find our latest research, go to jabber.com. I'm Paul. Thanks for listening today.