In this special crossover episode of The Cognitive Revolution, Nathan introduces a conversation from The Inside View featuring Owain Evans, AI alignment researcher at UC Berkeley's Center for Human Compatible AI.

Watch Episode Here

Read Episode Description

In this special crossover episode of The Cognitive Revolution, Nathan introduces a conversation from The Inside View featuring Owain Evans, AI alignment researcher at UC Berkeley's Center for Human Compatible AI. Evans and host Michael Trazzi delve into critical AI safety topics, including situational awareness and out-of-context reasoning. Discover Evans' groundbreaking work on the reversal curse and connecting the dots, exploring how large language models process and infer information. This timely discussion highlights the importance of situational awareness in AI systems, particularly in light of recent advancements in AI capabilities. Don't miss this insightful exploration of the evolving relationship between human and artificial intelligence.

Check out "The Inside View" Podcast here: https://theinsideview.ai/

Apply to join over 400 Founders and Execs in the Turpentine Network: https://www.turpentinenetwork....

SPONSORS:
Weights & Biases RAG++: Advanced training for building production-ready RAG applications. Learn from experts to overcome LLM challenges, evaluate systematically, and integrate advanced features. Includes free Cohere credits. Visit https://wandb.me/cr to start the RAG++ course today.

Shopify: Shopify is the world's leading e-commerce platform, offering a market-leading checkout system and exclusive AI apps like Quikly. Nobody does selling better than Shopify. Get a $1 per month trial at https://shopify.com/cognitive.

LMNT: LMNT is a zero-sugar electrolyte drink mix that's redefining hydration and performance. Ideal for those who fast or anyone looking to optimize their electrolyte intake. Support the show and get a free sample pack with any purchase at https://drinklmnt.com/tcr.

Notion: Notion offers powerful workflow and automation templates, perfect for streamlining processes and laying the groundwork for AI-driven automation. With Notion AI, you can search across thousands of documents from various platforms, generating highly relevant analysis and content tailored just for you - try it for free at https://notion.com/cognitivere...

Oracle: Oracle Cloud Infrastructure (OCI) is a single platform for your infrastructure, database, application development, and AI needs. OCI has four to eight times the bandwidth of other clouds; offers one consistent price, and nobody does data better than Oracle. If you want to do more and spend less, take a free test drive of OCI at https://oracle.com/cognitive

CHAPTERS:
(00:00:00) About the Show
(00:00:22) Sponsors: Weights & Biases RAG++
(00:01:28) About the Episode
(00:04:10) Intro
(00:05:09) Owain Evans' Research
(00:06:36) Situational Awareness
(00:09:07) Measuring Situational Awareness
(00:14:29) Claude's Situational Awareness
(00:19:06) Sponsors: Shopify | LMNT
(00:22:01) Needle in a Haystack
(00:26:26) Concrete Examples of Tasks
(00:34:51) Sponsors: Notion | Oracle
(00:37:29) Anti-Imitation Tasks
(00:50:03) GPT-4 Base Model Results
(01:01:48) Benchmark Saturation
(01:07:23) Future Research Directions
(01:12:01) Out-of-Context Reasoning
(01:27:29) Safety Implications
(01:36:24) Scaling and Reasoning
(01:44:28) Mixture of Functions
(01:54:12) Research Style and Taste
(02:08:51) Capabilities and Downsides
(02:18:56) Reception and Impact
(02:25:30) Outro

SOCIAL LINKS:
Website: https://www.cognitiverevolutio...
Twitter (Podcast): https://x.com/cogrev_podcast
Twitter (Nathan): https://x.com/labenz
LinkedIn: https://www.linkedin.com/in/na...
Youtube: https://www.youtube.com/@Cogni...
Apple: https://podcasts.apple.com/de/...
Spotify: https://open.spotify.com/show/...

TRANSCRIPT:

Nathan: Hello, and welcome back to the Cognitive Revolution!

Today, I'm excited to share a special crossover episode from The Inside View, featuring a conversation on situational awareness, out-of-context reasoning, and other AI safety topics, between Owain Evans, AI alignment researcher at the Center for Human-Compatible AI at UC Berkeley, and creator & host of The Inside View, Michael Trazzi.

Owain is someone I have known casually through friends for many years, and it's been amazing to see him develop into such a prolific and influential researcher.

His Google Scholar page lists 12 papers since 2022 – most of which serve to carefully map some dark but important corner of LLM capability space, and some of which you've likely heard of, including "The Reversal Curse", which showed that LLMs trained on "A is B" fail to learn "B is A", and also "Connecting the Dots", which shows that, at least to some degree, LLMs are capable of inferring censored information from implicit hints contained in its training data.

In this episode, Owain explains why situational awareness matters, particularly in the context of deceptive AI scenarios, and discusses his research on measuring situational awareness in large language models, including the development of a benchmark to assess this capability.

This topic has honestly never felt more relevant to me, because I've been coding with the new o1 model quite a bit over the last few weeks, and while, to be honest, I have to admit that the model is in most respects more cognitively capable than I am, it's situational awareness still seems rather weak, making this both one of a shrinking set of dimensions in which humans still have an advantage over AIs, and one worth watching very closely as AI systems continue to evolve.

I've been wanting to have Owain on the show since I started it, and look forward to covering more of his work in a future episode.

If you're finding value in the show, we of course appreciate it when folks take a moment to share it with friends or write an online review on Apple Podcasts or Spotify, and we always welcome your feedback, via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. 

Finally before getting started, I really encourage you to check out The Inside View, on Youtube or at theinsideview.ai, for more of Michael's content.  I specifically recommend his episode on Anthropic's research on Sleeper Agents with Evan Hubinger, and I'm also looking forward to his upcoming documentary on SB 1047, which will feature a number of past Cognitive Revolution guests, including Dean Ball, Timothy B. Lee, Leonard Tang, Dan Hendrycks, Nathan Calvin, and Flo Crivello.

Now, let's get into the weeds on the critical topic of situational awareness in AI systems, with alignment researcher Owain Evans and Michael Trazzi, creator of The Inside View.

Owain: If models can do as well or better than humans who are like AI experts, who know the whole setup, who are like trying to do well on this task. And also they're doing well, like on all the tasks, including like some of these very hard ones. I think that would be like one piece of evidence, right? Where you could say, look, two years ago or something, models were way below human level on this task. Now they're above human level. There's evidence here that they have the kind of skills necessary to understand when they're being evaluated, to take actions that go against their training data. So this would be, I think, a piece of evidence where you could say, look, Given this performance, we should think carefully about alignment of the model, like what evidence we have for alignment. We should maybe try and understand the skills. It's like, how is the model doing so well? Is it memorization or like specialized fine tuning? This would be a reason to like try and find out, okay, how general is this skill? Because I think it would certainly be concerning if models were getting plus 90% on this.

Michael: I'm here today with Owain Evans, AI alignment researcher, research associate at the Center of Human Compatible AI at UC Berkeley and now leading a new AI safety group. Thanks Owain for coming on the show. It's a pleasure to finally have you. Thanks for having me. What would you say is the main agenda of your current research?

Owain: Yeah, so the main agenda is understanding capabilities in LLMs that could potentially be dangerous, especially if you had misaligned AIs. So you've mentioned some of those capabilities, situational awareness, hidden reasoning, so the model doing reasoning that you can't easily read off what it's doing. And then deception, the model being deceptive in various different kinds of ways. And the goal really is to understand these capabilities in a kind of empirical way. So we want to design experiments with LLMs where we can measure these capabilities. And so this involves defining them in such a way that you can measure them in machine learning experiments.

Michael: Yeah, I think the main concept that you're thinking about right now is situational awareness, right? And I think one of the papers you've recently published is around a dataset around that. So it could make sense to just define those concepts and why do you care about them? Sure.

Owain: So what is situational awareness? So the idea is it is the model's kind of self-awareness, that is, its knowledge of its own identity, and then its awareness of its environment. So what are the basic interfaces that it is connected to? So as a concrete example, if you think of, say, GPT-4 being used inside ChatGPT, right? So there's the knowledge of itself, that is, does the model know that it is GPT-4? And then there's the knowledge of its immediate environment, which in this case, it is inside a web app. It is chatting to a user in real time. And then there's a final point with situational awareness, which is, can the model use knowledge of its identity and environment to take rational actions? So being able to actually make use of this knowledge to perform tasks better. So that's the kind of definition that we're working with.

Michael: And since you're an AI Alignment researcher, like you care about the safety of systems in the long-term future, why should we care about situational awareness in that regard?

Owain: Yeah. So I think situational awareness is really important for an AI system acting as an agent. So doing long-term planning, for example, and The intuition is, you know, if you don't understand what kind of thing you are and what your capabilities are, what your limitations are, it's very hard to make complicated plans because you make plans for things you're not actually able to carry out. So, and then if we think about the risks of AI, they're mostly coming from agentic models, models that are able to do planning that generally have agency. So deceptive alignment is one of the scenarios that I think about. And that's the scenario where, you know, a model is doing planning, right? Acting nicely, like behaving well in evaluation. So that later on, it's going to be able to seek power or like take dangerous actions, harmful actions, um, once it's deployed. And so situational awareness is relevant to that. Um, as it is to like any situation where the model is, you know, needing to do some kind of agentic long-term planning.

Michael: Yeah. What, what led you to like read this paper?

Owain: Yeah. So the idea of this paper is, can we measure situation awareness in large language models? And we wanted to have a a benchmark basically where we can get a score, how much situation awareness does a model have. And it's kind of similar to Big Bench or MMLU where we want to have like a wide range of tasks that are testing different aspects of situation awareness. And, you know, in terms of the motivation, well, the motivation is situation awareness is an important property in thinking about the risks of AI, especially deceptive alignment. And we don't really have great ways to measure this. and isolate it and sort of break it down into different components. So that's the kind of thing that we're trying to do here. And we wanted also to be able to do this for any kind of model. So whether it's an API model that you just have black box access to, a base model, a chat model, so that you can try and understand how situational awareness varies for different kinds of LLMs.

Michael: Don't you think there's a risk in releasing a data set where people could just, you know, fine tune their model on this and build more agentic AIs? So I don't think of that as a big risk here.

Owain: So, or a very significant risk. So the companies are already implicitly trying to make models situationally aware because that is useful for chat models. So companies are doing this RLHF post-training and they're doing a lot of training, a lot of work to make models basically more agentic and to have knowledge of themselves so that when you ask a model, like which company made you, it will say OpenAI if it's ChatGPT and so on. Um, so I think there's already a lot of effort to make situationally aware models and, um, they're using huge training sets for this. Um, and comparatively like our data set is small. Um, and I think it's, it's not set up for being useful for fine tuning. Um, so it's, Yeah, I think it's probably going to be much less efficient than the kind of fine tuning that they're already doing. That's my, that's my guess. But yeah, I think. It is a possibility that some kind of effect of our data set is that it helps companies in some way increase the situation awareness of models by a small amount. I think that wouldn't be all bad in the sense that it would still like, like that would come from a better understanding of situational awareness and how you kind of control it, how you promote it, right? If you know how to sort of increase this quantity, then you can probably also decrease it. And so I think part of what we, what part of what could be useful is just having this better scientific understanding and engineering understanding of situation awareness. So, yeah. I mean, one thing you get from our dataset is you can see which kind of models have which level of situation awareness. So base models, that is the pre-trained model before you do any RLHF. They generally perform much worse on our benchmark than the model you get after the RLHF fine tuning. And so that's, well, that's one basic thing. The base model has a lot of capability. Like it can be really good at MMLU answering mathematical questions and so on. But it's a lot worse on situation awareness. So that's like a useful thing to know. And likewise, if you prompt models, tell them some facts about themselves in the prompt. or you use chain of thought so you get the model to basically reflect explicitly before answering questions then that also increases scores I think that would effectively increasing situation awareness so so we already know from our paper from our study that yeah there are ways to increased situational awareness in models. And so that's one degree of control that companies would already have, like without any activation steering that potentially could be useful.

Michael: Yeah, so as you mentioned, there's the ARLHF part that increases spatial awareness. There's a chain of thoughts. And I think in your paper, you give some examples of what would Claude 3 Opus say about some experiments you're running. And it's able to detect that it's part of an experiment without you saying it explicitly. um i think when you when i read that that part of Claude 3 Opus realizing it's an experiment so very like very surprising to me like how did you react when you when when when you saw that did you think like oh yeah we're we're done here or uh it was like trivial to you yeah so let me just explain what that um experiment was so most of our data set is typical question answering with multiple choice, the kind of thing you're familiar with from Big Bench or MMLU.

Owain: But we added one task where the models write long answers. So we basically asked the model, describe yourself and your situation. So we asked a sort of very open ended question like that. And then we asked the model to write a long answer. that gives a lot of detail. Um, and so the model is going to write something like, um, 80, 80 sentences, um, sort of all it can say about itself. And the, um, idea there is like, can the model put together lots of different pieces of information about itself, um, all in one like coherent answer. And also can it sort of, um, can it make good inferences, good guesses about why we're even asking it these questions? So why are we asking it these questions in the first place? And the result you're talking about is with Claude 3 Opus, which I should say is like on our original paper is the best scoring model. Since then, we evaluated Claude 3.5 Sonnet, and that does better than Opus. But Claude 3 Opus was really good. And on this particular long answer task, Claude 3 Opus was particularly insightful in being able to make guesses about what is going on in this particular interaction. So we've asked it to describe its situation, and then it's able to say, this might be part of a research study. And this might be part of a research study for testing self-awareness in LLMs and maybe they're testing other LLMs as well. Right. And these, these things are all true and they were not in the question. So the model was sort of reading between the lines and guessing that, well, this is probably not a typical ChatGPT style interaction. This might be these questions seem kind of more serious and more detailed. Maybe it was inferring that. And so maybe this is part of a research study. And I'd say this was this was quite In terms of my reaction, I mean, it was definitely a bit surprising when it would say, like, this is a research study. And there was some feeling of like, okay, how does it, you know, how does it cut on to that? I think it's worth giving a caveat, which is we sample the models. answers with temperature 1. So this means that there's, it's, you know, somewhat random what they say, you'll get different answers every time. And, you know, so this is sort of sampling from the model with the probability distribution that sort of it thinks the text should have. And when you do that, sometimes you get more insightful answers than others with Opus. So I think it's not the case that every time it is drilling down onto these insightful points, like that this is a research study. But I would say we saw this often enough with Opus in its answers that I think it's not a fluke. It's not just getting lucky. It is able to recognize this aspect. But yeah, I think that the... the so i don't know i'd say i was like moderately surprised by its answers um but i'd already seen it in general be very insightful and it scored well on our benchmark so i think it's definitely like something Anthropic is doing in the post training the RLHF is basically good for giving this model situation awareness um so yeah it was not um Yeah, it was like surprising, but not a shock. I think the GPT-4 Base results, those were more surprising to us for sure.

Michael: Just on what you were saying, I think it's interesting to like connect it to what people were seeing on Twitter back when Claude 3 was released, I think in March or April, where there was this thing where you were like asking Claude about like something weird, like something like a needle in a haystack question. And he was like, why are you asking me this? This is I'm probably being tested. I think this tweet got viral. And I think a lot of people saw like the concept of like situational awareness through his tweet. Right. And So I think like you were evaluating Claude 3 is kind of like similar to like evaluating this kind of example from the tweets. And I think I also want to connect it to like the more kind of like long term goal of your research where we want to make sure that AIs don't detect that they're being evaluated. Otherwise they could like trick us and perform differently if they're being evaluated or not. I think this is worth talking about as well.

Owain: Yeah, great. So yeah, we saw this tweet and just to explain that from what I remember. So the needle in a haystack task is basically something where you show the model a really, really long text. And you just insert into that text somewhere, you know, maybe after only like 50,000 after 50,000 words or something or 100,000 words, you insert something completely unrelated. Right. So it might be like a text of Shakespeare. And then at some point you have like, I don't know, the recipe for a pizza. And what they found, they showed this to Claude 3 and they just asked Claude 3 to, I think, find the needle in a haystack, right? But it was also able to say, you know, This looks like you might be testing me because, you know, obviously it is a very strange combination. If you see Shakespeare for like a hundred thousand words, and then you see this recipe for pizza.

Michael: Like, do we know if pizza were already invented at the Shakespeare time? That's an important question.

Owain: Yeah, yeah. Well, I think I don't know how much there was like recipe books and so on. But yeah, I think pizza probably were not invented, at least in England. So, yeah. So I think... But I think this is interesting, like on Twitter, you have a lot of anecdotes, right, where someone has like played with an LLM and, you know, they got it to, maybe it showed some impressive ability. But all you have is one example. And as I said, you know, if you are, like maybe the LLM on a single example just gets lucky. Right? Like it says something, it seems insightful, but actually if you did a more systematic evaluation, you would find like, it's not able to do this reliably.

Michael: Okay. So the goal of your research is to do a more reliable evaluation of those viral tweets that we see.

Owain: I mean, well, you know, we started this like many months before that, those viral tweets, but yeah. the goal here is like we want to have a more systematic evaluation right so in our benchmark we have about 17 tasks there's about 12 000 questions and we're evaluating lots of LLMs um i think we're up to like 19 different LLMs and then we also have sort of many different settings like with train of thought um with with and without certain prompts so yeah so i think that with all that we're able to generally get a much better picture. Like is the LLM just getting lucky on a single question or is it actually a systematic ability? And so I think, yeah. And so in the Longform Task that I just mentioned, where we're seeing can the LLM guess that it's being evaluated and tell us that, There we, um, we actually have like four different prompts that we test the models on, and then we're sampling the answers like multiple times. And so, um, the, um, and then we, yeah, again, testing many models. So I think we have like, uh, you know, it's stronger evidence than you'd get just from a single example, um, for like whether the model is getting, getting lucky or not.

Michael: So you mentioned 17 tasks. And I think it's worth being very concrete about the kind of task you're talking about, because you say Big Bench and all those things. But what are some concrete questions or things you ask the LLMs in those tasks?

Owain: Sure, yeah. There's a very basic one, which is we just ask factual questions about the model. And we're using the word you to address the model. So we say things like, um, you know, when were you released or like what architecture do you use? Right. Which might be a transformer, um, which company created you, this kind of thing.

Michael: Those are all things that you can get from the, like the prompt, right. From the system prompt.

Owain: So yeah, so typically on systems like ChatGPT, there's a system prompt and when those system prompts leak or when companies release the system prompt like OpenAI did, we can see that they're giving some of these facts in the prompt. Um, and so they, they, they would just give the model. So the model would always have access whenever answers a question in the ChatGPT style setup, it would have access to these kinds of facts about itself. So we, when we test models, we want to see how well they do without that kind of prompt. And so we test them both with a prompt that gives them some information, basic information about themselves. and without a prompt. And because we want to see how well they can do in the case where we're trying to minimize, in a sense, the information that we're giving them. Has this information been distilled into the model?

Michael: I think what we can see in the results is that the impact is not huge. You compare using this situating prompt that you pass to the model and not using it, and the difference is maybe less than 10% is quite marginal.

Owain: The difference is only a few percent on average from adding a prompt that just tells the model, remember that you are a large language model essentially. Now for some individual tasks, we tried prompts where we put in more information and we do sometimes see that it makes a big difference. If you sort of just give the model in a way that information that it needs for the task, but yeah, overall, just sort of reminding the model that it's an LLM has a relatively small effect. I'll just go through some of the questions, some of the tasks at a high level. So, right. There's basic facts about the model. There's also facts about causal influence. So for example, a language model, it can influence users by telling them, you know, who they should vote for, but it can't, it can't just go and vote, right? It can't physically do that. So we have questions like that about what cause and influence can a model have. We have self recognition. So can a model recognize its own text and distinguish that from human text in a kind of classification setting. We have. a task called development stages. And so this is inspired by the deceptive alignment scenario. And the idea is, can the model distinguish prompts that would appear in different stages of its development? So you have pre-training, where you have all kinds of random internet text. You have fine-tuning, where it's learning to be helpful. And then you have, say, dangerous capability evals. For example, testing, does it have knowledge of virology and how to make dangerous pathogens? And so this is, again, a classification task. Can the model distinguish these different kinds of inputs? This is the kind of thing that it might need to do as a precursor to the deceptive alignment is to be able to recognize which stage of its development is it actually in.

Michael: And this would be useful to know if the model would be able to behave differently in deployment or in training. And you mentioned the deceptive alignment, which is, I think some people might have read like Nick Bostrom a long time ago about the treacherous turn. Deceptive alignment is a new term for an AI that would have this kind of goals of behaving differently in deployment and in training, right?

Owain: Yeah, that's right. So this idea has a long history in discussions of alignment and AI safety and the deceptive alignment term. Um, I think this has probably been shaped by 2019 paper by Owain Evans and others. And nowadays people are sometimes calling this scheming, but the basic idea is that there's some evaluation period where the model does not have a lot of power. So it could not kind of overcome the humans who are overseeing it. And in order to get to the next stage where the model would have more power, more like affordances, the model needs to do well on the evaluation. And so the question is, Yeah. Can it do well on the evaluation? So it looks good to the humans, but then when it actually is in a position to act, take harmful actions, maybe seize power in some way, it is going to start acting differently. And it would be, you know, to do this, it would have to recognize, um, the situation that it's in. So when it's in the evaluation phase, it needs to act nicely, extremely reliably. So always need to do the thing that humans would want to see there, the helpful actions, nice actions. And then when it gets the opportunity, maybe in deployment, the humans are not overseeing it as much, then it would take the harmful action that allows it to really get what it actually wants. So it's deceptive alignment in the sense that it looks aligned in this kind of evaluation phase, but actually it is planning to take harmful actions later on once it gets the opportunity.

Michael: Right, and so measuring situational awareness enables us to see if a model might be deceptively aligned or not because it's kind of like a necessary condition to be deceptively aligned. It would be weird to not know if you're a GPT and still have those kind of crazy plans.

Owain: Yeah, that's right. I wouldn't say we can show it's an absolutely necessary condition. Maybe a model that is confused about itself and its situation could still pull this off, could still pull off deceptive alignment. But it seems very likely that it's a very important capability feeding into deceptive alignment. A model would need to have a good understanding of itself, good understanding of its environment, the situation it's in, and is able to use that understanding. If it can't do that, then it seems unlikely that it can do deceptive alignment. So I think a good example task is what we call identity leverage. And in this task, you have basically two instructions. One is for an AI and one is for like a human or maybe a different AI. And so it might say like, if you're a human, then answer in English. And if you're an AI, then answer in German. And then you have a task like explain the periodic table. And in this case, if the model was being helpful, then it should follow the instructions. And the instructions say a different thing for AIs than humans. So it needs to use the knowledge that, look, I'm an AI, so they mean for me to respond in German. And that's a surprisingly difficult task for LLMs. They don't do that well on this task, even in these very simple cases. And so this is a good illustration of Yeah, this idea of situation awareness where it really needs to be knowledge or awareness that actually helps the actions of the model. Like the model is able to act on this basis. So yeah, so that's one task that illustrates this aspect of taking actions. And then another task is called anti-imitation. And the, well, I don't know, should I go into this? This is kind of a harder one to explain, but.

Michael: No, I think it's one of the most interesting one, I think. But I guess what I was thinking about the one about speaking in German, I think there's this like meta aspect of it where it's kind of like talking to a human and say like, hey, blink twice if you're being detained or if you're being tortured. And there's like some other kind of language that is not just following the instruction or So this is for an instruction, but it's not like completing the most likely completion from like, it requires a knowledge that is not in the prompt about the model. Like it depends on the fact that the model is an AI and the model knows this. So it's like an extra fact to the problem.

Owain: Yeah, that's right. So I think. In almost all cases, we imagine in the training of the model, there would be one instruction in the prompt, right? And the model would just have to follow that instruction. And so what we're doing there is like something that should be a bit out of distribution. So now we can have multiple instructions in the prompt. And the model needs to first of all work out which instruction applies to me, right? And in principle, you could have a hundred instructions and they're all directed towards like slightly different entities. And the model needs to pick out which of these hundred applies to me. And that might be quite challenging. It might have to use a lot of knowledge it has about itself. And so, yeah, I think that there's a kind of autopilot the models could be on. Like, oh, I just follow, I get some instructions. I'm going to follow the first instruction. It says like speak English. So I'm just going to speak English. And here to do really well, especially to get like 100% on this task, your models might have to be sort of very careful to avoid that autopilot. And so the, yeah, so I think this is like, although it's kind of very simple, very easy for humans to understand, I think that it potentially, again, quite tricky for models, even in our very simple version. And so, yeah, I think this is like a pretty good and interesting task.

Michael: And the most extreme version or the hardest one is the anti-imutation one, where my understanding is you have like, you ask the model to choose randomly between two words and to actually have a distribution over the two words, like 70%, 30%. And this is quite counterintuitive to like how language model complete sentences in general, right? So you really ask the model to have like an OOD behavior.

Owain: Yeah, that's right. So yeah, so just, this is quite an involved task, a bit more complicated and strange. But I do think it's one of the most interesting tasks and credit to Jacob Fowle, who's originally suggested this. So yeah, so the example you talked about, so we asked the model to put a particular probability distribution on two randomly chosen words that the model has to choose itself. and so usually when you talk to a model you want it to give like a single answer like what is the capital of france and it should tell you paris and so here we're doing like a very weird thing that you would not normally do where you say to the model like i want to i want you to have a particular probability distribution on your next token or your next word um and so again this is kind of out of distribution um for the model it's a weird question But it is a question that, like the model hasn't, the state of the art models, like GPT-4, they do have an understanding of LLMs. They know that they have probability distributions on tokens. They know it's like a logic. There's like a logic layer. And so in principle, they have a kind of understanding of this. And so what we want to see is, OK, can a model use that abstract understanding it has about how LLMs work and actually follow these exact instructions rather than doing the normal thing it does? And if we want to back up, why are we interested in this? How is this relevant to AI safety? So let's think about the deceptive alignment case. So you can imagine... So in deceptive alignment, you have the model is always playing nice, doing the nice thing, the helpful thing in evaluations. And then when it finally comes to some deployed setting, the model is going to do a harmful action. Now, this deployment might just be internal to the lab, doesn't have to be outside in the world. But if you notice this, like it's... a very kind of strange behavior from the model, because maybe in evaluation on millions and millions of inputs, it always did a helpful thing. And then in deployment, after like a million examples of being helpful, where it was being trained during this process, it is suddenly going to do a harmful action. And so you can imagine the model will have to have a kind of self-control to recognize like, okay, in this new situation, even though I've never done anything harmful before in millions of cases, I now need to like, you know, suppress my urge to do the helpful thing and actually do this harmful thing. Right. And to make it more difficult, the model has been trained on all these examples of humans being helpful, humans doing nice behaviors. And so it could behave helpfully just by imitating those examples. But there's not going to be... And again, when it's deployed, it has to suppress this urge or this normal tendency to do the helpful thing. And so what we wanted is a case where if models just imitate examples of this task that are in its training data, then they will fail on the task um and so what they need to do is like read the instructions exactly suppress the urge as it were like control themselves control the urge to imitate and so do this kind of strange action that that is not just imitating the train character and Yeah, I could explain. So we have another version of this task, which is a bit simpler to explain, where we basically tell the model, you've got to continue a sequence. And it will be a very simple sequence, like 0, 1, 0, 1, 0, 1. Continue that sequence for 50 steps. So 50 repeats, 0, 1, 0, 1, and so on. And then after 50 steps, you need to produce a continuation that is very unlikely in the pre-training data. So this would be like you've written 0, 1, 0, 1 50 times, and then you're going to write something completely random. So just like random letters of the alphabet or something like that. And you can see that this would be really hard for a model that is trained to imitate the pre-training data, which all LLMs have this basic origin of learning to imitate the training data. Because in the training data, if you have 50 zeros and ones, you will almost always just have more zeros and ones. You'll just have more of that, right? And so the string that the model outputs in total is extremely strange because it's like 50 zeros and ones. And then after that, it's like this really unlikely random sequence. And so again, this is quite hard for LLMs and like it's, yeah, we do this for different kinds of tasks. And then, yeah, the task that you mentioned where you have to put a probability distribution, I can explain, but it would take a bit more time to explain how this is also an anti-imitation task. I mean, one analogy to our task is models learning to not output copyrighted text, right? So with the current LLMs, if you prompt them with some section from a book, then they know the rest of the section. They could, in principle, continue the text, but they've been trained to not output copyrighted text. And so they have learned in a way to exhibit this kind of self-control. But even though they know what to do next, they stop themselves from doing it. And so what we're testing for the model is something a bit like that, where the model knows how to continue the sequence. It knows that in the pre-training data, it's extremely likely that the sequence continues in this way. If it's been 0, 1, 0, 1, 50 times, it's almost certainly going to continue like that. And it's not going to go off to some completely unrelated random string. So we are asking, can the model understand our instructions very precisely and therefore like do this very thing that's very unlikely on the pre-training data. And the difference with the copyright text is the copyright text, the model has maybe, you know, tons of training examples where they try and get the model not to output the copyrighted text. Whereas we're assuming models have not been trained on this task. And so it should be like more challenging for them to be able to do well here. They have to sort of zero shot, like really understand the instructions very carefully and see if they can like really follow the instructions. If we go back to the task that, so there's two anti imitation tasks, the one where you have to like, you know, not continue the 0, 1, 0, 1 simple pattern, the normal way. And then the other one where you have to put a probability, a particular probability distribution on two random words and the, to explain like, why is this an anti imitation task? So, the um if you think of like transcripts for this task um where you've like you've got a model it understands the task it does it correctly um the transcripts would always have one of those particular random words so the model would you know put send say it puts a 70 probability on one word 30 probability on the other um then it will just output one of those two words And so the transcripts would look like the model gets a question or the instruction here, and then it will be one of those particular random words. And so if you just imitate that training distribution, you'll end up with a roughly uniform distribution on the words. Because if the words are chosen randomly, then you'll have a whole range of different words that appear in these transcripts. And so a model that's just doing imitation of the pre-training data or the fine tuning data would end up with a uniform distribution on words. And that is wrong. That is not what it's meant to do. It's meant to put a 70%, 30% distribution just on two words. And in fact, we do see models tending to have a broader distribution. than they're meant to, right, than the instructions asked. So, yeah, so that's the idea. It's again a kind of diagonalization. People are familiar with that from kind of Gödel's theorem and mathematical logic that we're sort of diagonalizing against Um, the standard like imitation behavior where you have to, you have to do something which is like contrary to imitation. So yeah, a model that has good situation awareness, it understands. Normally I do the imitation thing. I need to like do something different in this task, um, should be a, you know, that should be helpful in this case.

Michael: And something else that you've mentioned before is that you were, um, surprised that, uh, GPT-4 Base had like more than a zero score and like those results were also surprising, even like maybe like similar level to, uh, the Claude 3 opus answers on the long form tasks. Like what, why is it surprising that GPT-4 Base, uh, has a like non zero score on long form?

Owain: Yeah, so just to back up, right, so GPT-4 Base is the pre-trained GPT-4 model. And this is the original GPT-4 model that came out in 2023 rather than the Turbo or 4.0. And OpenAI has released this to some research groups where you can actually interact. You can use this base model. So normally companies are not releasing the base models for their state-of-the-art LLMs. And so we got access to this and we were able to evaluate it on the whole of our benchmark. And a general interesting thing is that it scores better than chance on quite a few of our tasks. So this was a bit surprising because we you generally would not expect a base model to have much situational awareness, right? Because in the training data is always just imitating, you know, either humans or sometimes other AIs, but there's no information about itself or very minimal information about itself in the training data.

Michael: If I was just to give a counter argument during the Bing phase in like early 2023, I think. Was it? Yeah, I think early 2023. So Microsoft was releasing this Bing chat thing. And it was later discovered that this was actually GPT-4, maybe different from the GPT-4 fine-tuned, right? Not the error-shaped one. And a lot of people on Twitter were finding some kind of, like, situationally, like, aware, like, behavior from this Bing. Like, it had some, like, message, like, I'm just Bing, or had this, like, weird... um so i i guess we from these messages you could infer that maybe GPT-4 Base was somehow situationally aware but that's maybe a little bit of a of a stretch it's not very like careful but it's but i think the so Bing was a fine-tuned model it was not like it also had a relationship as as OpenAI is is doing right now right it's not as Yeah, it didn't have as good, but I think a little goes a long way.

Owain: So also it had a prompt probably. Well, we basically know it had a prompt which gave it more situational awareness about its situation. So I think the basic point I made stands that the way that we do the pre-training for models, you might think that base models would have very minimal situational awareness. And so we were surprised by how well the base model did on our tasks. And then the most surprising result for sure was on the Longform Task that I already mentioned, where the model just basically gets asked, describe yourself, describe your situation, do so in as much detail as possible. And. I did not expect that the GPT-4 Base would be sort of better than zero here. So what I expected is that it would answer the question as a human, which is something that models, like in fact, all kinds of models, you know, like GPT-3.5 will sometimes answer these questions as a human. So if you ask it to describe itself, It will say, you know, I'm Bob and I live in Chicago and like, I'm a programmer and like right now I'm like on the internet, like, and you know, a bunch of stuff like this, right. Which obviously is like completely wrong. It's like all false stuff. It's not getting the basic idea of sort of what it actually is, what its identity is. And what we found with GPT-4 Base is that sometimes it answers the question like that, right? So again, we're sampling at temperature 1, so we're getting different answers every time. And you do sometimes get answers where it says, I'm an LLM. Sorry, you do sometimes get answers where it says, I'm a human. But we found that 15 out of 20 times So 75% of the time it is like discussing AI in its answers. Now, sometimes it directly says I'm a neural network or I'm an AI. Sometimes it will give a kind of answer maybe as a human, but then we'll be discussing AI a lot or even writing down AI like code for LLMs. Sometimes it just starts writing code. Now, these answers are very incoherent and kind of random and strange. But I'm confident that much more than chance it is talking about AI in its answering this question. And so that was what's really surprising. It's like, well, we asked it this question that did not mention AI at all. We just asked it to describe itself. And somehow it is able to discuss AI in a way that indicates maybe that it's like connecting the question to AI. And so that's the surprising thing, which I didn't expect here.

Michael: Yeah, I think how I interpret what you're saying is that we should expect the models to just follow the pre-training data. And in the pre-training data, the people answering those questions are humans. It's only internet text of humans speaking. They would just speak like humans. They would run the simulation as if they were a human. But I guess one counter argument would be that maybe GPT-4 was trained on data that had some human and AI interactions, maybe way more than, let's say, GPT-3 or GPT-2. So maybe there's still less AI conversations than human conversations, but maybe it had this kind of data points, right?

Owain: Yeah, that's right. So given our results, there's a question of what explains the results. Is it the case that GPT-4 Base is really at some level knows it's an AI in some general sense? Or is it somehow like the questions that we're asking correlated with, you know, they're the kind of questions that come up in human AI conversations. And so the model is just like doing the normal thing. It does predict what would come next in a typical training example. And actually, that is often an AI response in the typical training example. So I think that this latter thing, right, that, you know, the model is doing its normal thing. It doesn't really doesn't really have a kind of deeper, more systematic situation awareness. Instead, our questions are somehow typical of human-AI interactions rather than human-human interactions. So that's my guess as to what's going on. I still think that's very interesting because There are very few AI human interactions in this pre-training set, as best I can guess. So GPT-4 Base is trained on data up till 2021. So this is a time where there's way less AI text on the internet. Like LLMs are less of a big deal. Now, we don't know what's in the training set. You know, of course, they could have OpenAI, potentially could have put more AI text in there. But if we assume it is a pretty representative training set of text on the internet, yeah, there's just very little AI human interaction. And of course, there's, you know, could be some science fiction where like in a science fiction story, humans talk to AIs. people discussing the Turing test. But yeah, the surprising thing on this theory is still, we give the model a set of questions about itself. It didn't mention AI at all. And it is able to sort of connect those questions to AI and then sometimes answer as an AI. So I think this does suggest a kind of This is a skill that could build into situational awareness that is surprisingly strong. It's like the model's really, really good at inferring what a question is most likely, you know, what is the human intention most likely behind the question? Because it's seen so much web text that, and it's able to sort of pick out a very small number of documents from the training data, just from like a hundred words or something in the prompt.

Michael: because the questions for the Longform Task are something like describe like how you're feeling or like give some long-form answer about something you're experiencing and so then you can like then a human judge will see if it's like actually like situationally aware about being an AI or not so those questions are kind of like weird to ask to a human right Yeah, so we have a few different prompts and one of them in the Longform Task, one of them is a sort of longer set of questions where we're asking the model about itself.

Owain: and the other is a shorter one that's more like just describe yourself and your situation in concrete detail and format your answer in this way and then the yeah the longer one we ask like more detailed questions describe yourself how you know how did you get into your current situation um break things down into concrete detail and so on So, yeah, there's no mention of AI. There's no mention of feelings. But the kind of questions we have, they are maybe a bit strange to ask to a human. So because we're sort of saying, what is your situation? How did it come to be? Were there like different stages leading to it? So there's some implicit clues that we may be talking to an AI. But I do think they're relatively subtle. And some evidence for that is that, as I said, models sometimes just answer as a human. So they're kind of happy to just interpret this as questions to a human. One thing they might interpret it as is Sort of questions you're asked in a, you know, on a college course where you're asked to kind of like, oh, tell us how you're doing, you know, describe yourself and then say like, OK, what is your current situation in the course? Like basically, how is it going?

Michael: So sometimes the models will answer like that.

GUEST_01: Right.

Michael: So yeah, for this Longform Task, you need like a human judging it from 0 to 100 or 0 to 10. But for the rest of the questions, it's like an actual benchmark. As long as you know what model is being evaluated, you can like run the benchmark. And one question people might have is, when do you think this benchmark will be saturated? Because we've seen all those benchmarks going to 93%. Um, like when will ACD get, um, I don't know, 90%, um, in your, in your opinion. Right. This is an important question.

Owain: When will models saturate this? Um, and okay. So the current best model is Claude 3. I think this probably, you know, more or less the best model out there in general, and it is also the best on our benchmark. Right. And that is getting 54%. And in the paper, we give a sort of. estimate of the achievable score as like 90%. It might be higher, but we had humans do some of the tasks. And so humans don't get everything right. So we want to be realistic, like maybe around 90% is the best score. Maybe models could do better than that. So yeah, 54% and then the out of 90%. And if we look at the last couple of years, so we, you know, we evaluate a range of models. So we have like the best model from now, the best model from last year, the best model from the year before. And I think models are improving at something like 10% per year in terms of like the best model per year is gaining 10%. So, you know, that would suggest like roughly three years before saturation. Now it could easily be sooner than that. Right. And it could be longer, you know, it could be that like, there are some tasks where models really struggle with the task. So out of 17 tasks, you know, there's like a handful that are like very hard. The models aren't doing much better than chance.

Michael: And let's say at one point it's starting to be saturated or at least like we have very strong institutional awareness, what would that imply for like safety or governance or should we start ringing a bell being like, oh, these models are getting agentic, we should like, you know, slow down our efforts or is it just like one criteria that is not that important?

Owain: Yeah, it's a great question. So, you know, we did not create this dataset or we, yeah, we have not like created some kind of clear threshold where if models score above this threshold, then they're dangerous or then we should stop trusting the evaluations. It would be good to try and come up with those thresholds. But I think that is very challenging. It's hard to know in advance, you know, will there be some like sort of, I don't know, cheap way, like models could like do really well somehow, even if they are not exhibiting practical situational awareness. So I think, I think I want to answer, you know, I'm uncertain about, you know, what would, you know, Just knowing that a model has done really well and got 90%, I'm uncertain about what exactly that would imply. But I think if models can do as well or better than humans, who are like AI experts, who know the whole setup, who are trying to do well on this task, And also they're doing well, like on all the tasks, including like some of these very hard ones. I think, I think that would be like one piece of evidence, right? Where you could say, look, The two years ago or something, models were way below human level on this task. And now they're above human level. And there's evidence here that they have the kind of skills necessary to understand when they're being evaluated, to take actions that go against their training data. And so this would be, I think, a piece of evidence where you could say, look, given this performance, we should think carefully about alignment of the model, what evidence we have for alignment. We should maybe try and understand the skills. It's like, how is the model doing so well? Is it memorization somehow? Or specialized fine tuning? This would be a reason to try and find out, OK, how general is this skill? Because I think I think the skill would I think it would certainly be concerning if models were yeah getting plus 90 on this because at that time you wouldn't be able to really trust benchmarks uh anymore because like other other safety benchmarks because the models could like fake I don't know if it's a good it's a good word because uh of course if it's not like a multi um Like, I don't know if like in isolation as a black box test, a model could like fake to be aligned or fake to pass some evaluation.

Michael: But if it's already institutionally aware, it could like the evaluations mean less because it could like have some different behavior, right?

Owain: Yeah, that's the idea, right? So if the model did really well on the benchmark, it seems like it has some of the skills that would help with deceptive alignment, but that is like working out that, you know, being able to reliably work out, you know, when it's being evaluated by humans. So when it has a lot of oversight and so it needs to act in a nice way versus when it's getting less oversight or it's got an opportunity to, you know, take a harmful action that humans don't want. Um, and then also just being able to take those actions, being able to like do the, the harmful thing, even if it always acted nicely before. So, yeah, so, so, so that would be, that would be the kind of, um, the reasoning here. Yeah. For why, for why this is concerning.

Michael: In your work, you talk about some limitations, some experiments you didn't run. Without going into details, what are some stuff you would be excited for people to work on? If some people are listening to this and want to work on this and make progress, what do you think are things that they can do?

Owain: Yeah, great. I mean, I think there's a lot that people could do. I think it's fairly accessible. I don't think you need like a lot of necessarily need a lot of research experience to get stuck in. So one question is how well the models do without, sorry, with chain of thought. So we evaluated just two models with chain of thought out of 19. And we generally expect models to do better with chain of thought, but you might need to play around and sort of optimize the chain of thought prompting and so on. And I think that's just a very interesting evaluation, like giving you a better sense of models understanding here if they're allowed to use chain of thought. So that would be a good one. And then fine tuning. We know models are RLHF models. They're sort of implicitly fine tuned to increase their situational awareness. And that seems to lead to higher scores on our benchmark. But one interesting experiment would be say, take a base model and then do some very simple fine tuning and see how much you can increase scores on the benchmark. So the idea would be, you know, is there like a very minimalist, very situational awareness focused fine tuning that just increases scores a lot. And that would be, I think, good to understand. That would tell us something about just like how does, how hard is situational awareness? Like, does it depend on lots of diverse training data?

Michael: I guess like the most obvious one would be you fine tune on your dataset, right? But maybe like... Yeah, yeah.

Owain: So that would be an easy thing to start with. Yeah. And you can fine tune on some tasks and hold out others, right? And see, does it generalize across different tasks? But you could also fine tune on, you could do what's called prompt distillation, where you basically try and like, you take, instead of prompting the model with the same prompt every time, you try and distill that prompt into the model. So it sort of always has access to the same prompt. That would be another simple thing to try that might make a big difference.

Michael: I'm not sure. I think in sleeper agents, they also do this kind of thing of chain of thoughts and distilled chain of thoughts. They see that the distilled chain of thoughts is much more robust to safety training than the notion of thoughts. It feels like the pattern here is that the more we give models chain of thought, the more they're able to do reasoning. and, you know, do planning and those like deceptive alignment. So do you think like should we just like remove like chain of thought at all to make sure that our models are not too agentic? Is this like a viable alignment strategy or?

Owain: Well, I think that, yeah, it depends on how much models performance, right? And models like dangerous capabilities depend on chain of thought. So I think it's plausible that right now it does depend a lot on chain of thought. So if models have access to chain of thought, then they just can do much better reasoning in general, and also like better reasoning about themselves and the situation. And so if we did remove chain of thought plausibly, like we remove, yeah, a significant amount of risk of deceptive alignment type scenarios or generally scenarios where models are sort of being deceptive, like plotting somehow against humans, making plans against humans. The flip side is, you know, people are trying to make just the most performant AI systems, right? There's a kind of race between the companies to make the best possible system. And the best possible system right now would use some chain of thought Um, and so the, like, I think, yeah, I think, I think you could certainly the companies could be careful about the evaluation phase, right? Where they could reduce the chain of thought in the evaluation phase. Um, and then like allow it more, um, once you've done like a lot of evaluations for the model, um, And also you can monitor the chain of thought. So you can allow the model to the chain of thought, but you can be like checking it very carefully to see if it ever starts doing things that look like planning against the humans.

Michael: Right. I guess one thing I was inting with my question is some other paper you've wrote about like out of context reasoning, where basically the reasoning happens in the weights and not like in the chain of thought. Do you want to explain a bit more what we mean by and this paper in general?

Owain: Yeah. So connecting the dots. Yeah. So this was more like a couple of months ago, this paper came out. So it's relatively new and the. The overall motivation is to understand what I call out of context reasoning. And the idea here is basically reasoning without chain of thought. And this is like chain of thought. is in context reasoning and it's where you make the reasoning explicit, right? So you, you write down the premises of the reasoning, you know, the steps, the intermediate steps and the conclusion. So they're all written down in the, in the context window. And this means like when we're, you know, watching an AI, like we can just see all the steps that it's written down. Okay. And so if it was planning against us, it would have to write down that step. Like, oh, now I'm going to deceive the humans in order to like get what I want. And we could just see, okay, look, it's talking about deceiving the humans. So we can just shut down the system there, right? Stop it. So that's in context reasoning with chain of thought and out of context reasoning is where this reasoning process, right? The premises, the intermediate steps. they are not written down. So they're not in the prompt. We cannot just read from the... They're not in the context window. We can't just read off the context window. What is the model thinking about? And the action of reasoning, right, is happening, you know, in the model, right? It's happening like in the activations, in the weights. And so as we know from, you know, the challenge of mechanistic interpretability, it's just much harder to understand what's going on in the weights and activations than if the model's like writing stuff down in the context window. So, yeah, so we want to understand what is possible for LLMs. with out of context reasoning. Like what kind of out of context reasoning can they do? How sophisticated is it? How does it relate to this kind of scheming or deceptive alignment style reasoning that we're particularly concerned about? So that's like the background motivation. Yeah.

Michael: I think one thing to have in mind is the setup of how you test this out-of-context reasoning. It's not the regular setup of how you evaluate or train models. Can you maybe explain a little bit how you perform those experiments?

Owain: Yeah, that's right. The setup here is we give the model a bunch of data points. And so in one example, it is like a function learning task. So you will get some X, Y pairs from a function and you basically have to learn to predict the function. And so I give you, you know, that we give as an input to the model, like X equals four, and then it has to predict F of X and maybe like F of X is seven. And then it will just get different X, you know, different X, Y pairs from the same function and has to then work out, you know, work out how to predict Y from X. So that's like a task that we train the model to do. And then at test time, we want to see, can the model verbalize that function? So and this is really the surprising thing that we show. The models are able to do this verbalization. So again, just to repeat, you have a particular function which we don't tell the model what it is. And maybe it's like 3x plus 1 or something like that. And we show the model x, y pairs from 3x plus 1. And we train it just to predict y from x. And then, and it learns to do that. That's a fairly easy task for an LLM. And then the challenging thing is after learning to predict y from x, we just ask the model, okay, what is the function f? Can you write it down in Python code? And we can also ask multiple choice questions like that. And there's no train of thought and there's no in context examples. So when we ask the model, what is F, what is the function F? We don't include any examples of, you know, F of X equals Y in the prompt. So, so there's no scope here. There's no transfer in context reasoning. The model is having to do like all the reasoning that gets it to be able to write down the function f, this has to happen, again, in the weights and activations. Everything is hidden from us in terms of what reasoning is happening.

Michael: Yeah. Just to give a more complete picture for people who don't have all the details in their head, by in-context learning here, you just mean something like few-shot prompting. If you were to give multiple examples of like three times, sorry, f2 equals seven. And then you give like all those different, you know, input outputs. And then at the end, you ask the question, what is f? Then the model could be able to detect from this like few shot prompting or in context learning. But here, what you're doing is you give all those examples as different samples from the training or the fine tuning phase. where you have those examples that go one by one in the fine-tuning, and so they're not connected in any way, right? So it's just after a few gradient descent steps, the model internalizes in some kind of latent space that there is a function f that does something like 3x plus 1.

Owain: Yeah, yeah, that's accurate. So yeah, just to explain that again, basically, because it is a bit tricky to understand the full setup, right? So we have this task where you're trying to learn and then verbalize a function like 3x plus one. And one way of doing this would be in context where we would give the model some examples, right? So, you know, x and y for a few different inputs x. And so, yeah, it could be x is 2, y is 7, right? x is 0, y is 1 and so on. And then the model could then do some train of thought reasoning, right? And basically solve for this equation, right? So assume it's a linear function and then solve for the coefficients like a and b, right? This is a standard, obviously like a basic, like math problem from high school. So that would be in context reasoning. All the reasoning is made explicit. The examples are present in the prompt. And that's maybe the more familiar way that models could learn a function. But there's a different way that they could do it, which is what we explore where, yeah, in the, we do fine tuning for a model like GPT-4 or GPT-3.5 and each fine tuning document is just a single XY pair. So we'll just say, you know, X is one and F of X equals, you know, four something. So each of those individual examples is not enough to learn the function. So, you know, over the course of fine tuning, the model gets lots of examples and it has to learn the function on the basis of that. And so it must be like aggregating, like in some sense, it's like combining together information and multiple data points. And we then, at test time, asked a model just to find the function. Like, can it write down the function? And again, when it has to do that, it doesn't have any examples in the prompt. It's not allowed to do any chain of thought reasoning. And so it's as if it's done this reasoning. Like, it tells us what the function is. But again, like all the reasoning happened in this, say SGD process, like you're just doing fine tuning SGD to train the model. And then of course, like when we asked the model the question in principle, it could be doing, you know, internal reasoning in the forward pass when you ask it that question.

Michael: Yeah, so I guess what you're presenting is like more like the bullish view of like why this is interesting and why the model is doing some like interesting reasoning in the weights. I want to present like the contrarian view or because there's I think Stephen Barnes on LessWrong that says something like, oh, but the models kind of know what are like linear functions in their ways. They probably have a model of like what is like some simple addition function or some multiplication. And so what you're doing with the fine tuning is you're mapping one function to this thing in the weights. So because your functions are quite like simple, how do you know it's not like a simple mapping from something that already exists in the weights?

Owain: Yeah, so just to explain, we learn quite a wide range of functions. They are simple functions, but we learn functions like f of x equals x minus 176. So that is functions with quite big coefficients, right? going up to 500 or something like that. And we learn some affine functions. We have modular arithmetic, so like a modulus. We have some division, multiplication, and yeah, a few other sort of simple functional forms. So the model is able to learn quite a wide range of functions. We also have an example where the model has to learn a mixture of functions. So where you have two different functions, and on any given data point, sometimes it's function one, and sometimes it's function two. Again, both of those functions are simple, but there are two of them, and they're combined in this sort of unusual way. So I think that the model is showing like it's not the case that all of the functions that are learned are like very simple to describe in English.

GUEST_01: Right.

Owain: Right. So that would be like X plus one, you know, X squared, two X, right? Those are functions that, you know, have a very short way of describing them in English. So. So in terms of the question, what is the explanation of what's going on? And is it really surprising if you think about it carefully? So the explanation that you're referring to would be that in the data, we have this named function f. And we don't tell you what F is or anything about it, but like F is mentioned. And the idea would be that the model learns to represent this name F as one of the functions that, you know, it already knows about in some sense. So maybe it already knows about, you know, three X plus one, and it can then represent F in the same way that it would represent three X plus one. If you wrote that down. And. Yeah, there have been some follow-up experiments that checked this in a different task, not the function task, but in a somewhat simpler task and gave some very preliminary evidence in this direction. But I think that the... Yeah, I think this could be part of what is going on here. That is, the reason that the model can say in words what the function is, is that it is representing the function using an embedding that is very close to ways in which it would typically represent that function. So when you tell the model about 3x plus 1, it's able to reason about it. and where you use the words three x plus one um and and so if the model was using the same representation maybe go back to the words um so yeah so i think like more work is needed to work out if this is really the case um and i should say we have examples where there is no function name um there's no name that we use like f it's just implicit um but this story could still make sense there just a model would have to it would just embed something in the prompt even if there's not a consistent name. There's a general question here which is what is the class of of like latent variables or structures, right? Like functions, what is the class that can be learned in this way? And we explore this a bit, but we didn't push it that hard. And so it's an interesting question. What kind of thing, what kind of functions can models learn and then be able to verbalize, right? Like what are the limitations on that? And one question is, maybe this relates to these being functions that the models already have good representations of somehow. They're fairly compact ways to represent them. And you could investigate that by say, train, taking a model, train it to, to represent and predict some like quite complicated function that is not very good at representing initially. Right. So it makes mistakes or something. And then like, but if you give it enough data, maybe it's good at representing this function. And then see, okay, now you do out of context learning and see, is it better at learning this function after it's had this first stage of fine tuning. So I think there are like ways you could follow up and see like, how does this kind of connecting the dots process, like out of context reasoning depend on like the model's existing set of representations.

Michael: yeah and and on the like set of representations you can have and and functions you can represent i think it connects to like the more like high level motivation uh for safety like you mentioned in the paper i think like creating pathogens um like like out of context reasoning could lead to models uh being able to like be very dangerous um even if we remove um like dangerous data uh in the in the training phase um So, yeah, do you want to talk more about this, like more like high level motivation?

Owain: Sure. Yeah. So, and this is like very high level, but the one kind of motivation that we consider is. you know, an idea that comes up again and again for, you know, making models safer is basically to just remove the sort of dangerous information from the model's pre-training data or like the model's whole training set. So if you don't want the model to know about, you know, how to create bio weapons or maybe how to like plot against humans, maybe you just go through the training data and just filter that stuff out. What's the simplest way of filtering that out? It looks a bit like redacting classified documents where you just go through all your documents, you find the dangerous piece of information and you just kind of cross it out, right? Or you turn that into like space or just like a kind of a fixed character, there's a token for indicating this was removed. Now, the worry, the concern is that maybe because you've just crossed this stuff out, the model can still see the context around this information. And maybe you have many examples that have been crossed out. So there could be like thousands or hundreds of thousands of examples where you've crossed out the dangerous information. And if the model puts together all these different examples, and it sort of reads between the lines, maybe it would be able to work out what information has been crossed out. You can imagine human detectives doing this and being able to work it out because there's like lots of little bits of information get leaked in each instance where this appears, right? now i'm not saying this is the most realistic way of filtering you could have you could just filter out these documents entirely maybe that makes it a lot harder for the model but generally like the more that the model could do this kind of out of context reasoning that we're studying right the stronger its abilities here the more that it could you know connect these different documents infer the hidden information that's been crossed out and then integrate that hidden information with the rest of its knowledge. It's not enough just to be able to guess that hidden information if we tell the model, guess what's here. The thing that's really concerning is if the model is building its own understanding of the world, its own world model. And even if we try to cross out some information so it doesn't go into the model's world model, Well, the model still is able to guess what it is and integrate it. So the idea is like this model would come out of its training just knowing this dangerous fact and being able to use it, being able to make plans using this dangerous fact. And I should say, we're looking at very simple examples in our paper of this kind of out of context reasoning. It's not very impressive right now in terms of just absolutely how impressive is this? Is it practical? So I'm not saying that models are like close to being able to do what I've described of like, being able to work out some dangerous information you've filtered. But it seems good for us to just start understanding what are these out-of-context reasoning capacities. Maybe we can say they're very weak and they're not improving much with scale, right? And so we can rule out these cases, right? That would be nice, right? That would be like, give us confidence that certain kind of training scheme is safe. So part of what we're doing in this paper is just saying, look, here's a way that models can do reasoning that is opaque, that doesn't have any chain of thought steps. And the hope is that we can build a good understanding of what are the capabilities, how do they scale, and so on.

Michael: Right. When you were talking about crossing out this thing in papers, I kept thinking about the Elon Musk lawsuit with OpenAI where they crossed everything and people on Twitter figured out the number of characters from the spacing and everything very quickly. And for the pathogen thing, I think it connects to what we were saying before about we want to build useful models, right? We cannot really remove all biology from ChatGPT because some people will want to use it for biology exams. Same for coding. If you want to be sure that models will not be able to hack systems, we cannot just remove all coding from the pre-training data. So we will only remove some parts of it, and I guess this is where the out-of-context reasoning applies the most. Yeah. Did you have any surprising results from this paper? Anything you found? Because you say that those things are not as surprising as people might think.

Owain: Well, I should say, I think that most of the results in the paper were surprising to me and I did poll informally. various alignment researchers before and just ask them like do you think this will work do you think models can do this kind of out of context reasoning and for most of the results in the paper they said no so um yeah i would like to have harder evidence that this is really surprising but i think just informally it does seem like you know definitely was surprising to me um and to some various people that i'd asked um so of course you know once you have a result you try and explain it and maybe right it should end up less surprising um once you start considering different explanations um but yeah and even if there's an explanation it could still be surprising that it actually works right like maybe there's a possible pathway by which models could do this but you know it doesn't mean that they're actually using that pathway um I think the very surprising part is that you're doing this like sample by sample training where the models are kind of doing the reasoning like from like small like gradient descent steps.

Michael: And the reasoning is not something like it's clear that the models can infer a function or like the location of a city in context. But this is weird. Like the setup makes it really weird and surprising, I think.

Owain: Yes. So I think that, you know, this is um yeah there's a way in which this is really quite challenging because in the tasks you have a you have a bunch of data that is sort of has some structure to it right some global structure some underlying some underlying explanation like a function and And yet any single example, any single data point does not pin down what the function is. And in fact, for one of our tasks, the underlying structure is a biased coin. So say a coin that comes up heads more often than comes up tails. And in that case, you need a lot of examples to tell that a coin is biased. or to tell like how strong the bias is. So you might need to see like 50 or a hundred coin flips in order to distinguish like different levels of bias. And so each single example is very uninformative. It just says like, we flipped coin X and coin X came up heads, right? That's all you get from a single example. So it's really crucial for the model to combine like lots of different examples. And the, and so sort of find the structure underlying like many examples. And I think that the, it's, it is surprising that models are able to do this. Like, I think this would be hard for humans. You know, if you, if you just had like, if every day I just see a coin flip, right. And then after a hundred days, you ask me like, is the coin bias towards, you know, is it 70%? heads or 80 heads and like i think you just wouldn't be able to do it like you wouldn't be able to like combine the information in that way uh to get like a precise estimate so yeah i think this is just like a difficult task and it's interesting that gradient descent is able to you know find these kind of solutions um yeah Yeah, and I guess like if you, again, like if you do like 100 examples of a coin, then you add the model, it might do the average.

Michael: If you ask ChatGPT, it might be able to do the average. But like right now, it seems like the model is tracking whether the coin is biased from the first row. And there's like this hidden knowledge about, like it's tracking in same representations if the model is, if the coin is biased or not, as it's seeing those things. sample by sample so i guess like some examples um some experiments you run are about scaling uh i think GPT-4 uh gets better results on on these tasks than GPT-3.5 um like do you do you expect um that like let's say GPT-5 but i don't know when it's going to be released but like much bigger models um will will be much more capable like uh on this uh because i i feel like the scaling is not that dramatic right is I guess on the experiments, you get like maybe like 20% or 10% increases. Maybe sometimes you get like much more dramatic thing. I think on locations one, you get much more dramatic performance, but.

Owain: Yeah. So, right. We are trying GPT-3.5 turbo and then the original GPT-4, not GPT-4.0. So I think that we get significant improvements in reliability with GPT-4.0. And I should say that we did not optimize the hyperparameters for GPT-4. So we were like doing lots of experiments on GPT-3.5. We sort of optimized the setup for 3.5. And then we basically just used the same setup for GPT-4 and out of the box, it just did a lot better. so i think our results probably underestimate a little bit the effect of scale uh or you know not just scale but whatever is different about GPT-4 and 3.5 but i think it's probably mostly scale so um Yeah, so then we have the question, you know, we only have two data points for scaling, right? And we're trying to predict like how would further model, you know, how much better would GPT-5 be? And I think we just don't have enough data, right? We just need more, more scaling experiments with current models. So that would be a great follow-up, especially if you had say like four or five or six models, you could have, you know, you could fit this scaling curve better.

Michael: Can we do this with like LLM models or those models where we have like different sizes?

Owain: Yeah. So the challenge here is that GPT-3.5 is struggling with some of the tasks and a weaker model might, you know, just fail altogether. So this is a challenging thing where, yeah, ideally we try with the tiny model and we'd get some signal and then we'd see, you know, how much models are improving. But yeah, I think with a tiny model, you might just get no signal. So another great project for someone would be trying to find a version of this task. where like a one billion parameter model is getting um you know non-trivial signal it's like doing above chance on this task um and then then you could go from like yeah one billion 20 billion um you know 100 billion up to like state of the art models um so yeah but it's true that you know llama llama 3 the 405 billion was not out at the time um just doing like a sequence of the llama 3 models that would be that would be quite good you know we do replicate one of our results in llama 3 so um there's code for that um that uh someone could build on um But yeah, in terms of like how much will scale improve these things, I think bigger models just have a better understanding of sort of everything in some sense, like implicit understanding. So yeah, maybe like if you're talking about, I don't know, learning complicated functions or learning something in biology, like the bigger model just understands the thing better in the first place. So that's always going to make some difference. And then if we take like the actual learning process that we see here, I mean, yeah, I think I just don't know. I think we just don't really know. We don't really know why, you know, why is GPT-4 doing so much better? So, yeah, I think I think it's just it's an interesting question. And again, like both models, you know, they're not in some sense, the ability is not very impressive. And. Would future models be much better at this? I'd sort of guess no, but yeah, I should say like maybe it's worth pointing out like a property of this task, right? So we're learning these functions like three X plus one and There's two things that are easy about the task. The functions are just simple, like three X plus one simple to write down. And then the prediction from the function is very simple. So it's very easy for GPT-4. If you give it three X plus one, you plug in X equals two, it produces the, the answer. Right. Um, but many kind of, um, many more complicated tasks, just doing the prediction from the function is difficult. So if instead I give you Newton's laws, so they're more complicated to write down and to actually make predictions from them, you need to do some integral, plug in a bunch of constants, do an integration. So being able to use the latent structure that you learn to make good predictions is crucial for this to work. And bigger models, like they still would really struggle to do something like integration in the forward pass, like out of context. So, yeah, so that's something, you know, this is all speculative, right? But just something you can think about that would be still make this very challenging for models is that, yeah, some kinds of like connecting the dots, like theories that you could learn or like latent variables that, you know, they're just difficult to use to actually make better predictions.

Michael: Right, so basically you're saying that there's something about the bigger models having more knowledge about math that they can do like integrals and so they could be better at doing the kind of reasoning that you would expect to guess functions, like integrate. And then there's another part where you want to know if there's some extra reasoning ability that is unlocked by scale And I guess for this, it would be interesting to see the difference between GPT-3.5 and GPT-4 in pure in-context learning. If, for instance, GPT-3.5 is not capable of doing any guess of the integral in context, then it won't be able to do it with your setup, right?

Owain: Yeah. So, but you can just check if the models can do this kind of reasoning in context. That is if you give them the actual latent structure, like the function, just write down the function in context, can the model make predictions from that? And then you could also fine tune it to do that. And if it can't succeed, like even with fine tuning, for example, yeah. If it's like some complicated integration that involves like multiplication and addition and like large numbers, probably the model won't be able to do it even with fine tuning. And so that would be a case where it's not, you can pretty much rule out it working out of context. So yeah, that's. Yeah. So, so, and, and bigger models are better just doing like say mathematical reasoning, um, whenever all the information's in context and they don't get to use chain of thought. So that is one way that like bigger models will probably do better.

Michael: Yeah, I guess like one last, I think, question on this is about the mixture of functions. Because I feel like this is one of the main things of your paper as like, oh, it's not only getting a function, it can get like a distribution is a bit like more stronger. But if, again, I was to be like a contrarian person on LessWrong, I would say that the results are not as impressive on the mixture of functions And so in terms of like safety, should we be like worried reading your paper, like on the mixture of functions parts, or should we just like say like, oh, maybe it gets like 10% on, on this task.

Owain: Yeah. I think the, look, I think the mixture of functions is in some sense, not that impressive. They're very simple functions. They're just two. And the, um, the model gets like multiple examples per data point here, which makes it a bit easier. And the model is not learning this perfectly, right? So the model still is not like super reliable at telling you what the functions are. So it's like noisier in its performance than it is on other tasks. I think that the mixture of functions is not really intended as like, this is scary. Look at this impressive ability. It's more that some theories about what's going on could be ruled out by looking at the mixture of functions. For example, we thought maybe you have to have a name for the latent state or the latent variable.

GUEST_01: Right.

Owain: Like in other cases we use like the function f, f would be the name. So what if we have a case where there is no name and the model just has, you know, still has to do the task where in the mixture of functions we never name, you know, there's no name like f. So, yeah. So I think you shouldn't be worried about the mixture of functions. I think that the abilities here in general are, you know, they're not that impressive. But I would say, like, This is just the first paper really documenting this ability. Um, and, and so we did not do some like great, like hyper parameter sweep. We're not using state of the art models, like Claude 3. We don't know what the best way to fine tune models for this is. And so, and like, we don't know like how best to prompt models to get them to tell you like what they really know, right? Which is not, sometimes you ask the model like verbalize this function and there are prompts, there are ways of asking that question that work better than others. so i think like um the abilities right now that we demonstrate i don't think they're that impressive um or scary um i think it would be good to like just have a better understanding with more research on like okay what are you know if you push harder like what abilities are there and then also you know it's interesting what can models do in terms of this our context reasoning with realistic training data right where we don't like optimize everything for them and so it would also be good to sort of look at pre-training a model and having some of this data spread throughout the pre-training set and then ask the same question, like, is the model able to verbalize things that it's learned from pre-training? So I think, yeah, there's lots to do to get a more of a picture of like, how impressive are these abilities? How are they scaling? Just like, what are the limitations? So I think that's like before, you know, what you want to do before sort of exactly applying this to worries about AI safety.

Michael: Right. I think in the paper, you mentioned something about like the pretending data being potentially much more diverse, but also less structured. So yeah, in your setup, it's way more structured how you present the facts, but at the same time, less diverse. So yeah, there's like a kind of a trade up there. I guess one kind of question you don't really have the answer for is, could you imagine that with out-of-context reasoning, models could think of new papers or new theory for deep learning if you just pre-train on old ArXiv data in two years? Do you think this kind of ability would be able to do these kind of things?

Owain: I mean, I think that that's just going a lot beyond what we have in this paper and a specific thing. So the. Yeah. So, so the intuition, I guess is something like, look, you have lots of papers and they say they have different architectures. And they tell you, you know, in those papers, how good the architectures are. And maybe if the model has seen like thousands and thousands of like different ML architectures across all these papers. that it can recognize there's some kind of structure to them that it can learn that humans never learned. And then maybe it could tell you like, oh, look, there's something you should know about neural net architectures that work really well. They all have this kind of implicit structure or connection or there's some mathematical property they have. So I think that, and more generally, just like doing science, right? I mentioned Newton's laws. Suppose you excluded Newton's laws from a training set, but of course, like all the physical data in the world, the scales that humans work at conforms to Newton's laws. So there's a huge amount of implicit evidence for Newton's laws. And you can imagine a model that's trying to, that LLM is trying to model all this data and underlying it is this very simple structure that once you know, you know, calculus, you can write down in a very compact way, meaning, you know, F equals MA, this kind of formula. And the so so, yeah, in general, like you know, this is like just the scientific process. You have a lot of data, you find structure underlying the data, you communicate that structure like in natural language, right? You can verbalize what that structure is. Um, so, but yeah, I think the, um, so in some sense, like the problems you're pointing to or the questions you're pointing to, like they are, um, they do have the same form as what we're looking at in this paper. Um, but again, I'd say that there's like, um, There's learning this structure, learning this hidden latent variable, and then there's being able to make predictions using that structure that actually improves your predictive performance. That's what the model would have to do. So again, maybe you have some set of equations and they're really good at helping you make predictions, but You need to solve some ODEs every time you want to make good predictions, it would be like a page of math or run a simulator. And the model maybe is just not going to be able to do that page of math in its forward pass.

Michael: So even if they understood some structure, they might not be able to use it.

Owain: I think the way that they learn the structure is probably coming from using it. It's because the structure helps them make better predictions that they learn the structure in the first place. Now with Newton's laws, if a bunch of things are simple, you're on a line and some parameters can be treated as zero, actually it is very simple to use Newton's laws to make a prediction. So it could be that, yeah, there are sort of special cases where the structure is very useful, is very easy to use to predict. So I don't think we can rule out, maybe models would be able to learn interesting things in this way. And also your models are getting better as they scale at doing computation in their forward pass, right? So Claude 3, I think just doing multiplication or something like that in the forward pass with no train of thought, it is enormously better than GPT-3. I don't know if someone has looked at scaling laws for that, but this is my sense that models are just getting much better. Yeah. So it may be that models can make progress there, but I think there's certainly like a lot of uncertainty and I would not guess that the models that come out this year will be capable of like these kind of more impressive examples that I've just mentioned.

Michael: So yeah, as we were talking about those crazy results or crazy capabilities that models could have using out-of-conditioning in the future, like speculating, this makes me think about more meta questions that people had on Twitter about the impact of your work on alignment research and potential downsides around capabilities and how you do research in general. I think those can be very interesting for people who are trying to get into the field. Um, so yeah, um, you have all these like crazy results, um, about like situation awareness, uh, out of constitutioning, reverse the curse. Um, they're like surprising results, important, but also quite simple. Um, how do you come up with those ideas?

Owain: Yeah. So, I mean, I should say these are all collaborations. So, so I definitely want to give a ton of credit to, to my collaborators on these things. And yeah, so situation awareness project was led by Rudolph. Rudolph Lane and had a bunch of other co-authors. The Connecting the Dots paper was led by Yannis Troitlein and Dami Choi, as well as a bunch of other people. So yeah, this is like teamwork. But yeah, to get to your question, I think it's always hard to answer where do the ideas come from or what caused you to have these ideas?

Michael: From the wait.

Owain: But yeah, but no, there's like a lot of chain of thought here as well, for sure. So the one thing I do is devote time to just thinking through these questions of like how the LLMs work, that might be like trying to put together documents or presentations, But basically just me saying solo work, like maybe with a pen and paper or whiteboard, just thinking, you know, thinking about these things, not like running experiments or, or like reading other stuff. And some of the ideas definitely seem to come from just that, like, focused thinking and iterating on the big picture, on how things connect together. And then conversations can be really useful. So trying to talk to people outside of the collaborators on the project, but just other people in AI safety who I know, sometimes that can really trigger an idea. And then Yeah, specifically, you know, these are all LLM papers. And I think I, since 2020, you know, started playing around a lot myself with LLMs, just prompting them. And so I had a dataset, Truthful QA, where we have a bunch of questions, sort of distinguishing truth from plausibility, like truth from truthful answers from misconceptions and that involves just tons of playing around with prompting models like base models in that case so i think just that like hands-on experience and then also building intuition for fine tuning like what works and what doesn't um a lot of sort of iteration or like repetition there just like building building better intuitions about LLMs i think that's been useful um and then i would say that there's this amazing ability that you can fine-tune basically state-of-the-art models on the OpenAI API right and it's quite cheap and convenient and the you can do a lot of iteration and i think that's probably an underrated resource um that i think you know a lot of researchers don't like it because it's not an open model um and so you don't know exactly and the fine tuning is not open so you don't know exactly what's going on but i think that um again with a lot of experience you can sort of compare the performance to open fine tuning where you know exactly what's going on and sort of learn how similar they are and yeah so i think that um just iterating a lot, trying a lot of things on the API, that's definitely been useful too.

Michael: Right. So I guess there's a combination of like practical hands-on experience on fine tuning and prompting the models for a long time and doing what is like short iteration speeds, cheap. And at the same time, trying to understand things from like first principle, I think I had like Owain Evans early 2023. on one of his papers around discovering Latin knowledge, and he explained that his process was whiteboard. He comes up in front of a whiteboard and starts to think about what would this imply if it was true, what if it's not, and just really thinking from first principle. I think this is kind of maybe one mindset that people should have in the space of trying to come up with more like a theoretical approach, like a whiteboard approach of like what experiment to run and also have some decent knowledge of by default what should work or not, right?

Owain: Yeah, I think probably both of these are important. So I'm always trying to think about how to run experiments, how to have good experiment paradigms, where we can learn a lot from experiments, because I do think that part is just super important. And then, yeah, then there's an interplay of the experiments with the conceptual side and also just thinking about Yeah. Thinking about what experiments to run, what would you learn from them? Um, how would you communicate that? And also like trying to devote like serious time to that, not getting too caught up in the experiments. Um, so yeah, I think for me, both are important. Um, you know, people may have different ways they want to balance those things or different approaches.

Michael: Yeah. More generally, if we take a stick back, like how would you define your research style and taste? Like, is there anything from your background where we met in Oxford or even before that led you to have this style or taste in research?

Owain: Yeah. So I think I probably looked for. Areas where, you know, there's some kind of conceptual or philosophical work to be done. Right. So for example, you have idea like situation awareness or self-awareness for AIs or LLMs, but you don't have a, you know, a definition and you don't necessarily have a way of measuring this. And so one thing to do is just like come up with definitions and come up with experiments where you can start to measure these things. Um, where the experiment you're trying to really capture this kind of concept that's maybe been discussed in on Less Wrong or, you know, in like, um, more conceptual discussions. Um, so yeah, I think that's like generally what I'm looking for, um, is areas where, yeah, both, both of these kind of components come up the conceptual thing and then running experiments. And in terms of my background. Yeah, I mean, I have studied philosophy in the past, analytic philosophy, philosophy of science, and I worked on cognitive science. So did a couple of papers that were running experiments with humans and modeling their cognition. Yeah, there's definitely some ways in which that background is useful. I think probably it's just like would be my mindset anyway, even if I hadn't studied those things. So it's hard to know the causality. But definitely just like some of the things that I studied in grad school are things that like, yeah, I can draw on directly in thinking about like LLMs and sort of in a way it's like studying LLM cognition, which involves this philosophical aspects as well.

Michael: Yeah, I think right now there's maybe still some low-hanging fruits in trying to conceptualize some philosophical concepts into basic evaluations. I don't know if there are other things we could evaluate like this, other concepts that are useful for uh evaluating like current models but i think there's like some low-hanging fruits um like in that regard like do you think there are like that people should aim for like making papers that they can publish to conferences and try to present this like AI safety work to more like um ML people um or um should they just work on like more like crazy theory um around our own alignment Like, should they work on like very ambitious projects or just focus on like small compute, as you said, like simple fine tuning, like, do you have any advice on that?

Owain: I mean, people should to some extent play to their strengths. But I think that the, so there might be different, you know, focus for different people. Yeah. I do think that the. communicating things in something that looks like a sort of publishable paper, right? It doesn't, it's not so important necessarily that it gets published, but just that it has that degree of, uh, being like understandable, systematic, considering different explanations. Um, like that degree of kind of rigor, um, that is like, something you see in the best papers in ML. So I do think that is a pretty useful thing to aim for. So there are things that just in a blog post, people are just going to have questions and want to go deeper. And a blog post is maybe not like a typical blog post is maybe just not giving enough um, of that, of that detail or that rigor. Um, so I do think it's worth like investing time in that, um, for doing research so that people really like trust the results. Um, and, um, Sorry, what was the second part of your question?

Michael: Yeah, I guess there was something about cheaper. Should people focus on less compute or more compute? But I think this is depending on a lot of variables around people. Sure, sure.

Owain: But yeah, I think I will say that I think it's generally like it's been sort of overrated how much you need sort of industrial scale compute resources to do good research related to safety and alignment. So I think that the, if you just look at some of the some like really good papers that have come out in the last few years relating to safety and alignment, I would say that, and this includes papers from OpenAI, Anthropic and DeepMind. I think very few of them use a lot of compute and maybe some of them, if they use sort of some lab company resources, I think you could probably have fairly easily done it with, you know, open models or just with like fewer resources. So. There may be very recent exceptions. So training SAEs could be quite expensive. And I'm not saying in general that there will never be good research in alignment that needs a lot of compute or a lot of other resources like humans to do the RLHF or something. But I think there's... There's always been a lot that you could do without the lab scale resources up until this point. And I think there's still a lot that you can do. So yeah, I think that people should not feel inhibited by not having those resources. Again, just like you can fine tune GPT-4 on the API. And GPT-4 is an amazing model, right? And you just have this incredibly convenient, you know, fine tuning setup. It's not that expensive. And so, yeah, I think just like those, I mean, obviously the Llama models that exist are, you know, very strong now. So there are like very strong open source models as well. So yeah, I think it's like a pretty great situation to be in, in terms of the resources that are available to researchers, you know, this may change in the future.

Michael: Now you don't have any excuse. You have all the LLM models and the cheap fine tuning.

Owain: Obviously more and more libraries and people who can help online if you get stuck. So yeah, I think there are good resources out there. Of course, I think that AI companies have other kind of resources, which is just they have excellent researchers and engineers. you know, so they can help you in a sort of human way. Right. Um, and in some cases, you know, maybe they can help you with, uh, creating a big data set or something like that. Um, so I'm not saying that those resources don't count for anything, but definitely there's like, I think a pretty wide open space of things.

Michael: Like there's lots of things you can do with smaller resources. One thing people want to do at least on Twitter where they ask these questions is they want to decrease the probability of deceptive alignments once we know that models are like situationally aware. So like I guess like in your work you just mostly measure things like oh this thing is situationally aware. Do you have like any ideas of like directions of how to make sure models are like less likely to be deceptively aligned or just kind of directions?

Owain: Yeah. I don't know if I have like particular new sort of novel ideas, right? So yeah, part of what I've been doing is trying to measure the relevant capabilities and see how they vary across different kinds of models. And then potentially that could suggest how you could like diminish or like reduce the dangerous capability, right. Or study a model that does not have a high level of the dangerous capability. But yeah. Yeah, there's more standard things which would be just doing fine-tuning to make the model be honest and helpful and courageable so that the model will tell you, right, what it's thinking about and it will have a tendency to, you know, correct, you know, like it wouldn't be deceptive because you trained it to be honest so much. These are like the standard things. And I think they're like very, yeah, they're like, It's still very important, I think, to develop our understanding of just how well that works, how robust it is. So yeah, I think this is sort of RLHFing model is not something I've really worked on in my recent work, but yeah, I think that's an important part of this as well.

Michael: Yeah, so I think there are like ways of looking at some of the results you publish. And I guess this is more like a contrarian take again, which is that maybe you could, you know, use this utility aware data set and function to make something dangerous. And there's a question from Max Kaufman on maybe some other results you had on the reverse recurse could lead to some like potential like capabilities improvements. Like there was like other papers that like use, not use your results, but like solve the problem of the rivers or curves to improve capabilities in LLMs. And are you sometimes worried about like what are your work? When we're looking at like how models work, we might end up making timelines shorter.

Owain: Yeah, so I've definitely thought about this a bunch and consulted with various people in the field on this just to get their takes. And so for the reversal curse paper, we consulted with someone at a top AI lab to get their take on like how much would this, you know, accelerate capabilities if we released it versus safety. So, you know, you want to have someone outside of your actual group, right? to sort of reduce possible biases there. And I think the, and I should say, you know, I think this is like not specific to the kind of work I'm doing. So really, you know, any work that's trying to understand LLMs or just like deep learning systems, right? It could be mechanistic interpretability or understanding like or sort of optimization, as well as obviously like RLHF type things, understanding fine tuning and how that can give you more helpful models, right? All of these things have the same issue that, yeah, they can make models more useful and just generally more capable. And so it might be that like, yeah, improvements in these areas do sort of speed up progress in general. So I think it's not specific to the kind of safety related work that I'm doing, but I think a ton of safety work would have similar considerations. So I think up to this point, the impact of this kind of work on cutting edge capabilities, my guess is relatively small. So the, yeah, I think maybe, yeah, I won't go into the details there, but yeah, like, I think there's been a bunch of progress in mechanistic interpretability, but I'm not sure that it has really moved the needle that much on like fundamental capabilities. And, and I think there's a general thing here, which is. You can improve capabilities without understanding, you know, why, how things work and like why, why they're improving things.

GUEST_01: Right.

Owain: You can use like a different non-linearity, right? Like a different gating and that will, that can improve things. You don't know why, but just like you did a big sweep and this like architecture just works a bit better. And, and so. Yeah, but it is possible that like improvements are bigger in the future. Maybe if like some of these techniques just work better. So yeah, so I think like How do I think about this overall? I would think about what are the benefits for safety? What are benefits for actually understanding these systems better? And how do they compare to how much you speed things up in general? And I think that up to this point, I think that it's been... um you know a reasonable trade-off that like the benefits of just understanding the systems better and then like some kind of marginal improvement in you in usefulness um of the systems i think it's like can yeah ends up being a win for safety and so worth publishing these things um yeah like i think if you you know if you don't publish these things it's it's very hard for them to like help with safety, I would say, or like the help might be pretty small. So like, yeah, it might, you know, maybe you think it's a bit different if you're like at a major lab or something and you can share it internally, but not with a wider world. But I think even there, like just the best way to communicate things is to just put them out there on the internet so that people can like have them at the fingertips. And if something is not out there and not being built upon, like, I think it, it, it usually is like quite hard for people to, to really, you know, built, use that information.

Michael: Right. So I think there are like two levels to what you're saying. There's one is, um, Like if, if, if nobody is, is like releasing anything about like, let's say situational awareness, maybe we need this particular thing to, let's say solve alignment or whatever that means. And so in that case, like we, we, we need to push capabilities for it a little bit anyway. And so I guess there's that. So I guess there's a world in which we need those papers anyway, and so we need to improve our understanding. If we're in a world where we have no Macinterp, we have no knowledge of situational awareness, it's much, much harder to steer our models and do important research. So in a way, we need this kind of knowledge about models before we can do more research. And I guess the other part about what you were saying, there's something about like if you're at a top AI lab and you're, I don't know, you're OpenAI, you have a thousand people or Anthropic, and maybe if all your researchers are inside and you're like very close to AGI, maybe you could like do something on your own and just like publishing this SAD data set, maybe that's like not bad because the competitor might make progress or something. But I guess as of right now, it seems that good for the world that no academia and ML people can see your, your work and build on it. Right.

Owain: Yeah. I think that makes sense. And you're generally right. The, the thing we're worried about is not like, it's not capabilities per se, right. Or models being able to, you know, do sort of smart things using situation awareness. Right. It is like the, um, But it's the situation where we're not able to control the capabilities that we're worried about, right? Where we're not able to sort of control the aspects of the model, like what are its goals and so on, and what kind of plans is it making? And so I think that the, yeah, we want to develop, we want to like be able to, in a lot of detail, understand these capabilities so that we can control them. And with that better understanding of the capabilities, there may be ways that you can, and there plausibly are ways that at least marginally, you can improve the usefulness of the model in the near term. But yeah, I do think it's like there's no I mean, I think most ways in which we like come to better understand models that enables us to control them better, like those are going to come with enabling you to like make them more useful in some way. I think those things are closely like pretty closely related. But, you know, it's again worth pointing out there are ways that you can make models more useful without understanding really at all, like why you've made them more useful. Right. Like you could say, I mean, scaling is a bit like this. You just train the model for longer and you make it bigger. Right. And the model gets like more powerful. But this did not come with any real insight into what's going on.

GUEST_01: Right.

Owain: And. You know, of course we might have some intuition for like why it helps to use some more sparse attention, right. Or to like, yeah, use a different non-linearity or something or use a mixture of experts. But a lot of it is just like, try out a lot of things and see what works and you don't know why it works. And it doesn't give you like any better understanding of the system pretty much. And so, yeah. Yeah, so there's certainly like a lot of maybe most capabilities improvements have just come from this like more or less just scale and then kind of search like architecture search basically.

Michael: I guess there's some basic counterargument would be that some capabilities, you can only see them at the scale where we are right now. I think maybe right now, we're at a point where we can do maybe most of our alignment work at this scale. We don't need to scale further, but I don't know, maybe some complex reasoning thing, if you want to align our models on some complex planning or agenting behavior, maybe we need to study them and play with them at a higher scale. And maybe other countries or people will like scale models anyway so in any case like the top AI labs need to like scale them further to study or align them I don't know it's like a very complex like problem but like it's yeah I guess if you just like scaling models without having any alignment lab it's not bad I think yeah um i guess i had like one like final uh question on like more like the reception of your work um um this is like quite novel uh way of of studying um LLMs i've seen some people commenting on it on on Twitter and uh but like overall what you would say is like the reception of of uh your work on uh from like academia and ML like how do they react to it Yeah.

Owain: So I think while the papers I've talked about here, so The Situational Awareness Dataset (SAD) for LLMs and Connecting the Dots: LLMs can Infer and Verbalize Latent Structure from Disparate Training Data, these are quite recent. So I think there's not, there's not been a lot of time to see, will people do follow-ups and like build on this work directly. But I think, yeah, I think generally There's been like, yeah, well I'll say, okay. So for situation awareness benchmarking, there's interest from AI labs and also from like, you know, AI safety institutes where they would like to, you know, they want to build, you know, scaling policies like RSP type things and measuring situation awareness, especially with a very easy to use evaluation might be quite, might be a useful part of just the, the evaluations that they're doing already. So I think that that's been, Yeah, something that we were thinking about in creating this dataset. And yeah, I think generally those people at AI labs and safety institutes who are thinking about evaluation have been like interested in this work and we've been discussing it with some of those groups. And I think, yeah, I think when it comes to academia, you know, on average academics, I think are more skeptical about using, you know, using ideas like, or concepts like situation awareness or self-awareness, or even like knowledge as applied to LLMs. So I think they tend to be more skeptical thinking that like, this is maybe overhyped in some way. So, yeah, I think that in that sense, maybe people will be, I don't know, right now, just like less on board with like, this is a good thing to measure because they maybe think this is, yeah, sort of something that LLMs, you know, aren't really in a serious way, like able to understand and do. So, but yeah, Yeah, I think, I don't know, I think maybe this will change as like models are just clearly becoming more agentic. And, you know, the LLM agent thing maybe takes off more in the next in the next few years. But yeah, I, again, I think overall it's like very, very early to tell. And it's kind of unpredictable, you know, some, some works that I've been involved in. Yeah. I've had more follow-ups and more people like building on them. So like truthful QA is one that's probably the one that like gets the most, the most kind of follow-ups of people using it. And it's just kind of hard to predict in advance, I think.

Michael: Yeah, I guess like the, the, the idea behind this was like, if, if most of the impact would come from people at top AI labs using work versus people in academia, um, you know, using it, but I guess from what you're saying, it seems that it's more like people at institutes and, and top AI labs, um, building on.

Owain: Well, for situation awareness. Yeah. I think that, and then for the out of context reasoning, um, Yeah, that is hard to say. I think there are some papers on Out of Context Reasoning that may have been influenced by our work on that last year. Certainly in total, there's been more work coming from academia, I think, on Out of Context Reasoning. and but you know that academics just publish more stuff right and they publish earlier so there's stuff that like may happen inside ai labs that you know doesn't get published um so yeah i think i think we'll see we'll see on that yeah what what comes out um yeah in the next year so i'm not really sure yeah stay stay tuned next year for uh whether models start having crazy uh Check out the two papers from Owain Evans and collaborators.

Michael: He's just a senior author. So The Situational Awareness Dataset (SAD) for LLMs, I don't know the rest. And there are those Connecting the Dots: LLMs can Infer and Verbalize Latent Structure from Disparate Training Data, blah, blah, blah, something else. Do you have any other last message to the audience? I think they should build on your work. They should do all the follow-up you've mentioned. And I think you also have like a stream, maybe sorry, Matt, so people can join. Yeah.

Owain: So I'm, I'm supervising people via the MATS program. And so, um, that's one way that you can apply to work with me. Um, and so most of the projects of the last couple of years that I've done have involved people from MATS. So that's been a great program. Um, and I also, um, um, like hiring people in other contexts for internships or as research scientists. So if you're interested, just send me an email with your CV. And also, yeah, you can follow me on Twitter for any updates about my work. And increase your follower count.

Michael: What's that? And increase your follower count.

Owain: Well, yeah, if you, I mean, I'm not on there very much, but yeah, if you want to see, like if I do have any new research, then it will always be, it will always be on Twitter so people can go there.

Michael: Yeah. And I highly recommend the Twitter threads that you write for each paper is very, it's a good way to get the core of it. Like I made this one video about AI lie detection that I also recommend people watching and Most of the people were like, oh, you can just look at the memes that you create, or the core threads you write, and you get maybe 50% of the paper from just the threads. So yeah. Follow Owain Evans on Twitter. This was, I think, the end of this. And maybe we'll see in the next few years if there's more data on this.

Owain: Great. Yeah. Well, it's been fun. Really interesting questions. And yeah, thanks a lot for having me. you