Allocate tokens

Jiang, Eric Hanchen, Ou, Weixuan, Liu, Run, Pang, Shengyuan, Wan, Guancheng, Duan, Ranjie, Dong, Wei, Chang, Kai-Wei, Wang, XiaoFeng, Wu, Ying Nian, Li, Xinfeng

Safety alignment of large language models (LLMs) faces a key challenge: current alignment techniques often only focus on improving safety against harmful prompts, causing LLMs to b…

Fan, Chongyu, Wang, Changsheng, Huang, Yancheng, Pal, Soumyadeep, Liu, Sijia

Machine unlearning for large language models (LLMs) aims to remove undesired data, knowledge, and behaviors (e.g., for safety, privacy, or copyright) while preserving useful model …

Ding, Jianrong, Chen, Muxi, Zhao, Chenchen, Xu, Qiang

Standard Sparse Autoencoders (SAEs) excel at discovering a dictionary of a model's learned features, offering a powerful observational lens. However, the ambiguous and ungrounded n…

He, Zirui, Zhao, Haiyan, Payani, Ali, du, Mengnan

Large language models (LLMs) are commonly evaluated on challenging benchmarks such as AIME and Math500, which are susceptible to contamination and risk of being memorized. Existing…

Gan, Jinwei, Cheng, Zifeng, Jiang, Zhiwei, Wang, Cong, Yin, Yafeng, Luo, Xiang, Fu, Yuchen, Gu, Qing

Large language models (LLMs) have achieved remarkable performance across many generation tasks. Nevertheless, effectively aligning them with desired behaviors remains a significant…

Yao, Yifei, Du, Mengnan

Understanding the internal representations of large language models (LLMs) remains a central challenge for interpretability research. Sparse autoencoders (SAEs) offer a promising s…

Zhao, Yunxiao, Xu, Hao, Wang, Zhiqiang, Li, Xiaoli, Liang, Jiye, Li, Ru

Pre-trained Language Models (PLMs) are trained on large amounts of unlabeled data, yet they exhibit remarkable reasoning skills. However, the trustworthiness challenges posed by th…

Ashok, Dhananjay, Chaubey, Ashutosh, Arai, Hirona J., May, Jonathan, Thomason, Jesse

Through a controlled study, we identify a systematic deficiency in the multimodal grounding of Vision Language Models (VLMs). While VLMs can recall factual associations when provid…

Aswal, Darpan, Hudelot, Céline

Large Language Models have found success in a variety of applications; however, their safety remains a matter of concern due to the existence of various types of jailbreaking metho…

Moreira, Gabriel, Marinho, Zita, Marques, Manuel, Costeira, João Paulo, Xiong, Chenyan

Traditional neural embeddings represent concepts as points, excelling at similarity but struggling with higher-level reasoning and asymmetric relationships. We introduce a novel pa…

Fereidouni, Moghis, Haider, Muhammad Umair, Ju, Peizhong, Siddique, A. B.

A key barrier to interpreting large language models is polysemanticity, where neurons activate for multiple unrelated concepts. Sparse autoencoders (SAEs) have been proposed to mit…

Xu, Haotian, Weng, Tsui-Wei, Nguyen, Lam M., Ma, Tengfei

Concept Bottleneck Models (CBMs) provide explicit interpretations for deep neural networks through concepts and allow intervention with concepts to adjust final predictions. Existi…

O'Reilly, Cliff, Jimenez-Ruiz, Ernesto, Weyde, Tillman

Connecting LLMs with formal knowledge representation and reasoning is a promising approach to address their shortcomings. Embeddings and sparse autoencoders are widely used to repr…

Ali, Sheikh Abdur Raheem, Xu, Justin, Yang, Ivory, Li, Jasmine Xinze, Arslan, Ayse, Benham, Clark

As large language models (LLMs) evolve in complexity and capability, the efficacy of less widely deployed alignment techniques are uncertain. Building on previous work on activatio…

Prasse, Katharina, Knab, Patrick, Marton, Sascha, Bartelt, Christian, Keuper, Margret

Concept Bottleneck Models (CBMs) enhance the interpretability of neural networks by basing predictions on human-understandable concepts. However, current CBMs typically rely on con…

Joshi, Ananya, Cintas, Celia, Speakman, Skyler

Recent work shows that Sparse Autoencoders (SAE) applied to large language model (LLM) layers have neurons corresponding to interpretable concepts. These SAE neurons can be modifie…

Santis, Francesco De, Bich, Philippe, Ciravegna, Gabriele, Barbiero, Pietro, Giordano, Danilo, Cerquitelli, Tania

To increase the trustworthiness of deep neural networks, it is critical to improve the understanding of how they make decisions. This paper introduces a novel unsupervised concept-…

Łucki, Jakub, Wei, Boyi, Huang, Yangsibo, Henderson, Peter, Tramèr, Florian, Rando, Javier

Large language models are finetuned to refuse questions about hazardous knowledge, but these protections can often be bypassed. Unlearning methods aim at completely removing hazard…

Oozeer, Narmeen, Marks, Luke, Barez, Fazl, Abdullah, Amirali

Controlling multiple behavioral attributes in large language models (LLMs) at inference time is a challenging problem due to interference between attributes and the limitations of …

Xu, Xiaoyu, Yue, Xiang, Liu, Yang, Ye, Qingqing, Hu, Haibo, Du, Minxin

Unlearning in large language models (LLMs) is intended to remove the influence of specific data, yet current evaluations rely heavily on token-level metrics such as accuracy and pe…

Yao, Junchi, Yang, Shu, Xu, Jianhua, Hu, Lijie, Li, Mengdi, Wang, Di

Large language models (LLMs) have made remarkable progress in various domains, yet they often suffer from repetitive text generation, a phenomenon we refer to as the "Repeat Curse"…

Sun, Chung-En, Oikarinen, Tuomas, Ustun, Berk, Weng, Tsui-Wei

We introduce Concept Bottleneck Large Language Models (CB-LLMs), a novel framework for building inherently interpretable Large Language Models (LLMs). In contrast to traditional bl…

Kulkarni, Akshay, Yan, Ge, Sun, Chung-En, Oikarinen, Tuomas, Weng, Tsui-Wei

Concept bottleneck models (CBM) aim to produce inherently interpretable models that rely on human-understandable concepts for their predictions. However, existing approaches to des…

Zou, Andy, Phan, Long, Chen, Sarah, Campbell, James, Guo, Phillip, Ren, Richard, Pan, Alexander, Yin, Xuwang, Mazeika, Mantas, Dombrowski, Ann-Kathrin, Goel, Shashwat, Li, Nathaniel, Byun, Michael J., Wang, Zifan, Mallen, Alex, Basart, Steven, Koyejo, Sanmi, Song, Dawn, Fredrikson, Matt, Kolter, J. Zico, Hendrycks, Dan

In this paper, we identify and characterize the emerging area of representation engineering (RepE), an approach to enhancing the transparency of AI systems that draws on insights f…

Wang, Weixuan, Yang, Jingyuan, Peng, Wei

Large language models (LLMs) have achieved remarkable performance across many tasks, yet aligning them with desired behaviors remains challenging. Activation intervention has emerg…

Lee, Bruce W., Padhi, Inkit, Ramamurthy, Karthikeyan Natesan, Miehling, Erik, Dognin, Pierre, Nagireddy, Manish, Dhurandhar, Amit

LLMs have shown remarkable capabilities, but precisely controlling their response behavior remains challenging. Existing activation steering methods alter LLM behavior indiscrimina…

Cywiński, Bartosz, Deja, Kamil

Diffusion models, while powerful, can inadvertently generate harmful or undesirable content, raising significant ethical and safety concerns. Recent machine unlearning approaches o…

Sun, Zhongxiang, Zang, Xiaoxue, Zheng, Kai, Song, Yang, Xu, Jun, Zhang, Xiao, Yu, Weijie, Song, Yang, Li, Han

Retrieval-Augmented Generation (RAG) models are designed to incorporate external knowledge, reducing hallucinations caused by insufficient parametric (internal) knowledge. However,…

Srivastava, Divyansh, Yan, Ge, Weng, Tsui-Wei

Concept Bottleneck Models (CBMs) provide interpretable prediction by introducing an intermediate Concept Bottleneck Layer (CBL), which encodes human-understandable concepts to expl…

Xu, Xinyue, Qin, Yi, Mi, Lu, Wang, Hao, Li, Xiaomeng

Existing methods, such as concept bottleneck models (CBMs), have been successful in providing concept-based interpretations for black-box deep learning models. They typically work …

Arditi, Andy, Obeso, Oscar Balcells, Syed, Aaquib, Paleka, Daniel, Rimsky, Nina, Gurnee, Wes, Nanda, Neel

Conversational large language models are fine-tuned for both instruction-following and safety, resulting in models that obey benign requests but refuse harmful ones. While this ref…

Qin, Yi, Xu, Xinyue, Wang, Hao, Li, Xiaomeng

Diffusion models have shown impressive sample generation capabilities across various domains. However, current methods are still lacking in human-understandable explanations and in…

Postmus, Joris, Abreu, Steven

Large language models have transformed AI, yet reliably controlling their outputs remains a challenge. This paper explores activation engineering, where outputs of pre-trained LLMs…

Sharma, Arnab Sen, Atkinson, David, Bau, David

We investigate the mechanisms of factual recall in the Mamba state space model. Our work is inspired by previous findings in autoregressive transformer language models suggesting t…

Tan, Daniel, Chanin, David, Lynch, Aengus, Kanoulas, Dimitrios, Paige, Brooks, Garriga-Alonso, Adria, Kirk, Robert

Steering vectors (SVs) have been proposed as an effective approach to adjust language model behaviour at inference time by intervening on intermediate model activations. They have …

Panickssery, Nina, Gabrieli, Nick, Schulz, Julian, Tong, Meg, Hubinger, Evan, Turner, Alexander Matt

We introduce Contrastive Activation Addition (CAA), an innovative method for steering language models by modifying their activations during forward passes. CAA computes "steering v…

Hu, Lijie, Liu, Yixin, Liu, Ninghao, Huai, Mengdi, Sun, Lichao, Wang, Di

Vision Transformers (ViTs) have achieved state-of-the-art performance for various vision tasks. One reason behind the success lies in their ability to provide plausible innate expl…

Lu, Shilin, Wang, Zilan, Li, Leyang, Liu, Yanzhu, Kong, Adams Wai-Kin

The rapid expansion of large-scale text-to-image diffusion models has raised growing concerns regarding their potential misuse in creating harmful or misleading content. In this pa…

Ferrando, Javier, Voita, Elena

Information flows by routes inside the network via mechanisms implemented in the model. These routes can be represented as graphs where nodes correspond to token representations an…

Hayes, Thomas, Rao, Roshan, Akin, Halil, Sofroniew, Nicholas J., Oktay, Deniz, Lin, Zeming, Verkuil, Robert, Tran, Vincent Q., Deaton, Jonathan, Wiggert, Marius, Badkundri, Rohil, Shafkat, Irhum, Gong, Jun, Derry, Alexander, Molina, Raul S., Thomas, Neil, Khan, Yousuf, Mishra, Chetan, Kim, Carolyn, Bartie, Liam J., Nemeth, Matthew, Hsu, Patrick D., Sercu, Tom, Candido, Salvatore, Rives, Alexander

More than three billion years of evolution have produced an image of biology encoded into the space of natural proteins. Here we show that language models trained on tokens generat…

Stolfo, Alessandro, Belinkov, Yonatan, Sachan, Mrinmaya

Mathematical reasoning in large language models (LMs) has garnered significant attention in recent work, but there is a limited understanding of how these models process and store …

Ismail, Aya Abdelsalam, Adebayo, Julius, Bravo, Hector Corrada, Ra, Stephen, Cho, Kyunghyun

We introduce a generative model with an intrinsically interpretable layer---a concept bottleneck layer---that constrains the model to encode human-understandable concepts. The conc…

Geva, Mor, Bastings, Jasmijn, Filippova, Katja, Globerson, Amir

Transformer-based language models (LMs) are known to capture factual knowledge in their parameters. While previous work looked into where factual associations are stored, only litt…

Zhang, Fred, Nanda, Neel

Mechanistic interpretability seeks to understand the internal mechanisms of machine learning models, where localization—identifying the important model components—is a key step. Ac…

Cunningham, Hoagy, Ewart, Aidan, Riggs, Logan, Huben, Robert, Sharkey, Lee

One of the roadblocks to a better understanding of neural networks' internals is \textit{polysemanticity}, where neurons appear to activate in multiple, semantically distinct conte…

Meng, Kevin, Bau, David, Andonian, Alex, Belinkov, Yonatan

We analyze the storage and recall of factual associations in autoregressive transformer language models, finding evidence that these associations correspond to localized, directly-…

Dar, Guy, Geva, Mor, Gupta, Ankit, Berant, Jonathan

Understanding Transformer-based models has attracted significant attention, as they lie at the heart of recent technological advances across machine learning. While most interpreta…

Zarlenga, Mateo Espinosa, Barbiero, Pietro, Ciravegna, Gabriele, Marra, Giuseppe, Giannini, Francesco, Diligenti, Michelangelo, Shams, Zohreh, Precioso, Frederic, Melacci, Stefano, Weller, Adrian, Lio, Pietro, Jamnik, Mateja

Deploying AI-powered systems requires trustworthy models supporting effective human interactions, going beyond raw prediction accuracy. Concept bottleneck models promote trustworth…

Hu, Lijie, Liu, Yixin, Liu, Ninghao, Huai, Mengdi, Sun, Lichao, Wang, Di

Currently, attention mechanism becomes a standard fixture in most state-of-the-art natural language processing (NLP) models, not only due to outstanding performance it could gain, …

Geva, Mor, Caciularu, Avi, Wang, Kevin Ro, Goldberg, Yoav

Transformer-based language models (LMs) are at the core of modern NLP, but their internal prediction construction process is opaque and largely not understood. In this work, we mak…