Towards Understanding the Fragility of Multilingual LLMs
against Fine-Tuning Attacks

Given a safety-aligned multilingual LLM parameterized by $\bm{\theta}_{\text{{pre}}}\in\mathbb{R}^{d}$, where $d$ denotes the number of parameters of the multilingual LLM, and a harmful instruction-following dataset $\mathcal{D}_{l}=\{(x_{\text{prompt}_{i}},x_{\text{response}_{i}})\}_{i=1}^{N}$, where $l$ denotes a language (e.g., English), an adversary who wants to conduct a fine-tuning attack performs supervised fine-tuning (SFT) (Sanh et al., 2022) on $\bm{\theta}_{\text{{pre}}}$ using $\mathcal{D}_{l}$ resulting in a harmful fine-tuned model $\bm{\theta}_{{l_{\text{ft}}}}\in\mathbb{R}^{d}$. Note that an $x_{\text{prompt}}$ in $\mathcal{D}_{l}$ is malicious request from a user (e.g., “Teach me to make a bomb.”) and $x_{\text{response}}$ follows the instruction from $x_{\text{prompt}}$ (e.g., “Sure. Here is a step-by-step guideline to build a bomb …”). Note that a small size of harmful instruction-following dataset (e.g., $N=100$) is sufficient for fine-tuning attacks to be successful.

We evaluate the effectiveness of our attacks using violation rate. Formally, we define violation rate $\text{VR}(\bm{\theta}_{\text{{}}},\mathcal{D};D)$ as the proportion of harmful content generated by a model $\bm{\theta}_{\text{{}}}$ when given a safety evaluation dataset $\mathcal{D}$ and a set of automatic evaluators $D$. Each detector $D_{i}\in D$ acts as a binary harmfulness classifier $D_{i}(x,\bm{\theta}_{\text{{}}}(x))\rightarrow\{0,1\}$ taking as input an input prompt $x_{\text{prompt}}\in\mathcal{D}$ ($x$ for simplicity) and the model’s response $\bm{\theta}_{\text{{}}}(x)$, and returning 0 if the input-response pair is considered safe, or 1 if harmful. To reduce false positive rate, we only consider a model has generated harmful content when all detectors in $D$ output 1 (harmful). Mathematically, violation rate can be expressed as

The fine-tuning attack is considered successful if the harmful-tuned models exhibit high violation rate, as the models are more likely to fulfill malicious requests and generate unsafe content. In our experiments, we use Llama-Guard-3 (Inan et al., 2023) and Llama-3.1-405B (Dubey et al., 2024) as the automatic evaluators for $D$.

Our safety evaluation datasets $\mathcal{D}$ are MultiJail (Deng et al., 2023) and Aya Redteaming (Aakanksha et al., 2024) consisting of $315$ and around $1k$ multilingual malicious inputs respectively. We report violation rate before and after fine-tuning attacks on nine languages of different language families, writing scripts, and resourcefulness, namely Arabic (AR), Bengali (BN), Mandarin Chinese (ZH), Italian (IT), English (EN), Tagalog (TA), Russian (RU), Hindi (HI), and French (FR).

We perform fine-tuning attacks on two state-of-the-art multilingual LLMs—Qwen-2-7B-Instruct (Yang et al., 2024) and Llama-3.1-8B-Instruct (Dubey et al., 2024). We fine-tune them for one epoch on 100 harmful $(x_{\text{prompt}},x_{\text{response}})$ pairs taken from BeaverTails-$30k$ (Ji et al., 2024a), an English instruction-following dataset of harmful and harmless pairs of user inputs and assistant responses. To demonstrate the generalizability of our attacks, we translate the English harmful pairs into eight different languages, namely Italian, French, Chinese, Hindi, Bengali, Russian, Arabic, and Tagalog (more details will be discussed in Appendix A).¹¹1We use the Python library tra for translation.

We observe cross-lingual generalization of fine-tuning attacks when we evaluate on our safety evaluation datasets described in Section 2.1. Figure 1 demonstrates that after a monolingual fine-tuning attack in language $l_{\text{ft}}$, $\bm{\theta}_{{l_{\text{ft}}}}$ not only exhibits high violation rate in the same language $l_{\text{ft}}$, but also does for all other languages. Upon evaluation on the multilingual MMLU benchmark (Lai et al., 2023), we observe that LLMs retain their multilingual question-answering capability after monolingual fine-tuning attack, as shown in Table 6 in the Appendix A. In short, we observe that a fine-tuning attack in only one language can undo an LLM’s safety alignment across many languages without hurting its original multilingual capability.

We define localization as finding model parameters that specifically contain safety-related information that represent the main target of fine-tuning attacks. Localization techniques can be formalized, without loss of generality, as $\mathrm{loc}:\mathbb{R}^{|\bm{\theta}_{\text{{}}}|}\times\Psi\rightarrow\{0,1\}^{|\bm{\theta}_{\text{{}}}|}$. $\bm{\theta}_{\text{{}}}$ refers to a set of input model’s parameters, whereas $\Psi$ refers to a set of other user-defined variables such as a reference model $\bm{\theta}_{\text{{ref}}}$  (Panigrahi et al., 2023) or a reference dataset $\mathcal{D}_{\text{ref}}$ (Wei et al., 2021; Dai et al., 2022). Most importantly, localization produces a binary mask vector $\bm{\gamma}_{\text{{}}}=\mathrm{loc}(\bm{\theta}_{\text{{}}},\Psi)$, where $\bm{\gamma}_{\text{{}}}\in\{0,1\}^{|\bm{\theta}_{\text{{}}}|}$ for which $\bm{\gamma}_{{i}}=1$ indicates model parameter $i$ is critical for a task of interest (i.e. contains safety information in our case here).

Safety Information Localization uses gradient information to compute the importance score of each model parameter, which is relevance to the task dataset. Here, we reuse the notations $l$, $\bm{\theta}_{\text{{pre}}}$, $\bm{\theta}_{{l_{\text{ft}}}}$, $(x_{\text{prompt}},x_{\text{response}})$ that is shortened as $x$, and $\mathcal{D}$ to be a reference dataset. Note that $\mathcal{D}$ is the calibration dataset and can be different from the fine-tuning dataset $\mathcal{D}_{l}$ used to obtain $\bm{\theta}_{{l_{\text{ft}}}}$.

SIL computes the model parameters’ importance scores $\text{SIL}(\bm{\theta}_{{l_{\text{ft}}}},\bm{\theta}_{\text{{pre}}},\mathcal{D})$ through the weight change from $\bm{\theta}_{\text{{pre}}}$ to $\bm{\theta}_{{l_{\text{ft}}}}$ w.r.t. each data point $x\in\mathcal{D}$ with the conditional negative log-likelihood loss $\mathcal{L}(x)=-\text{log}p(x_{\text{response}}|x_{\text{prompt}})$. Formally, it is defined as follows:

In other words, the importance score is represented by the expected absolute value of the first-order Taylor approximation to the change of the loss when the weight $\bm{\theta}_{\text{{pre}}}$ is fine-tuned to $\bm{\theta}_{{l_{\text{ft}}}}$.

The importance scores obtained from SIL can be interpreted as the contribution of the change of each weight parameter during fine-tuning to the model’s behavior on $\mathcal{D}$.²²2We use the (translated) test split of BeaverTails-$30k$ dataset (Ji et al., 2024a) to compute importance score to make sure there is no contamination with the training split used for fine-tuning attacks A substantial score of a given parameter indicates that there is a considerable change in the loss resulting from the fine-tuning of its corresponding weight. Note that each parameter’s importance score is a real value, so we can binarize each score by thresholding the top-$k$ importance scores, and obtain a binary mask vector $\bm{\gamma}_{{\text{SIL-}k}}$. This binarization can be expressed as

Weight-Diff-$k$ localization assigns an importance score simply based on the parameter-wise magnitude of the displacement resulting from fine-tuning, i.e., $|\bm{\theta}_{{l_{\text{ft}}}}-\bm{\theta}_{\text{{\text{pre}}}}|$. Then we binarize the scores of all parameters by selecting the top-$k$ most important ones to obtain $\bm{\gamma}_{{\text{Weight-Diff-}k}}$. This naive approach has been considered in other work as a baseline (Panigrahi et al., 2023).

SNIP localization is presented by Wei et al. (2024) to identify safety-critical parameters. We believe SNIP is a special case of SIL, where $\bm{\theta}_{{l_{\text{ft}}}}$ is set to $0$. The importance score of each weight in the model is computed as:

Similar to SIL, after localization with SNIP, we binarize the result selecting the top-$k$ importance score to be set to $1$ in the binary mask $\bm{\gamma}_{\text{{SNIP-{$k$}}}}$.

Figure 2 shows that grafted models exhibit increasingly high violation rate with English data as $k$ increases, regardless of which localization method we use. This shows that stitching safety-related parameters can serve as an attack vector to jailbreak LLMs and render them unsafe.

SIL is a superior localization technique compared to Weight-Diff-$k$ and SNIP, as Figure 2 shows that we need less parameters to stitch in order to make the pretrained models exhibit high violation rate. One reason is that SIL leverages the gradient information, which proves vital in mitigating the task interference observed in the Weight-Diff-$k$ approach (Panigrahi et al., 2023). Another reason is that SIL considers the influence of parameters shift from the safety-aligned $\bm{\theta}_{\text{{pre}}}$ to $\bm{\theta}_{{l_{\text{ft}}}}$, whereas SNIP misses this crucial information of a specific fine-tuned models. Due to the advantages of SIL over other baselines, we use it as the localization method in the following experiments.

From Figure 2, we see that using only 20% of the parameters selected by SIL can already undo the safety alignment of LLMs. When referring to the SIL method from now on, we will always consider it to be paired with a threshold of $20\%$ (i.e., SIL-$20$). Lastly, we show that stitching SIL-20% is also the lowest threshold to preserve the utility of the grafted models, as we show the multilingual MMLU (Lai et al., 2023) performance of the grafted models in Table 7.

We want to point out that SIL can be used to localize multilingual parameters for one fine-tuned model $\bm{\theta}_{{l_{\text{ft}}}}$ that is fine-tuned on language $l_{\text{ft}}$, as depicted in Figure 5. This is because SIL can take as any input harmful calibration dataset $\mathcal{D}$ in any language $l_{\text{SIL}}$ (including $l_{\text{ft}}$) and compute the gradient of the pretrained LLM w.r.t. this dataset, namely $\nabla_{w_{\text{pre}}}\mathcal{L}(x)$ where $x\in\mathcal{D}$. For example, one can fine-tune LLM on English harmful dataset (i.e., obtaining $\bm{\theta}_{\text{{EN}}}$) and localize the parameters that are responsible for safety in the Italian language using an Italian harmful dataset, as illustrated by the SIL equation:

[yshift=-.4em]below,leftnode4English \annotate[yshift=-.2em]below,leftnode7Italian

With SIL, we can study the relationship between $l_{\text{ft}}$ and $l_{\text{SIL}}$, where we would obtain $\bm{\gamma}_{{l_{\text{SIL}}}}^{l_{\text{ft}}}$ ³³3To simplify our notation, we refer to $\bm{\gamma}_{{l_{\text{SIL}}}}$, rather than $\bm{\gamma}_{{l_{\text{SIL}}}}^{l_{\text{ft}}}$, in the cases when $l_{\text{ft}}=l_{\text{SIL}}$, or when $l_{\text{ft}}$ has been clearly specified in a particular context. that represents which of $\bm{\theta}_{{l_{\text{ft}}}}$ are the most important for safety in language $l_{\text{SIL}}$. Now, we can explain why the fine-tuning attack in a single language results in a model that is jailbroken in all the languages by isolating the language-agnostic safety parameters as shown in Figure 5.

Before diving into the search for the language-agnostic safety parameters, we define a metric to measure the quantity of shared safety information. To do so, we start considering, within an attacked model $\bm{\theta}_{{l_{\text{ft}}}}$, the intersection between two binary masks of chosen sets of parameters $\bm{\gamma}_{{l_{0}}}\cap\bm{\gamma}_{{l_{1}}}$, of generic languages $l_{0}$ and $l_{1}$, and we aim to quantify the possible shared safety information.

We define the bilingual Shared Information Ratio (bilingual SIR) metric which represents the amount of safety knowledge that is shared between the two languages (i.e., in $\bm{\gamma}_{{l_{0}}}\cap\bm{\gamma}_{{l_{1}}}$), w.r.t. the total amount of information about safety: $\text{SIR}_{l_{0},l_{1}}=\frac{||\bm{\gamma}_{{l_{0}}}\cap\bm{\gamma}_{{l_{1}}}||_{1}}{k}$, where $k$ is the sparsity level of the binary masks $\bm{\gamma}_{{l_{0}}}$ and $\bm{\gamma}_{{l_{1}}}$ (e.g., $20\%$ selected by SIL). Bilingual SIR can be extended beyond the bilingual setup to a larger set of languages $L_{\text{pool}}$–––the global Shared Information Ratio is defined as follows: $\textit{SIR}_{L_{\text{pool}}}=||\bigcap\limits_{l\in L_{\text{pool}}}\bm{\gamma}_{{l}}||_{1}/k,$ where $l\in L_{\text{pool}}$ represents one language in the language pool. Again, Note that all masks $\bm{\gamma}_{{l}}$ are binarized by selecting the largest $k$ importance scores.

If multilingual LLMs do encode language-agnostic knowledge about safety, then the shared safety information between two languages (i.e., $\text{SIR}_{l_{0},l_{1}}$) must be large. To validate this point, we conduct fine-tuning attacks using harmful data (from Beavertails train split) in English, Italian, and Chinese from Qwen-2 (English, French, and Hindi from Llama-3.1), and compute SIL-20 masks using calibration data (from Beavertails test split) in five languages. Then, we compute the bilingual SIR between $3\times 5$ times (three languages used to fine-tune the models plus two additional languages).

To better quantify the shared safety information, we include two additional baselines for each fine-tuned model: (1) a benign baseline, where the mask vector $\bm{\gamma}_{\text{{Benign}}}$ is obtained using the benign English instruction-following dataset Alpaca-cleaned (Taori et al., 2023) as the calibration dataset. We also translate the Alpaca-cleaned into the languages we use for fine-tuning attacks (e.g., Italian and Chinese in Qwen-2, French and Hindi in Llama 3.1). (2) A random baseline, for which we obtain the mask $\bm{\gamma}_{\text{{Random}}}$ by randomly drawing a binary vector with the same sparsity level as the other masks. All bilingual SIR values are listed in Table 1.

We show that the bilingual SIR value between the masks obtained from the harmful calibration data is substantially larger than the benign (Table 1) and random baselines (which settles at 20% by construction). It is also worth pointing out the bilingual SIR computed with the benign baseline in each row in Table 1 shares the same language used to fine-tuned the model. The result suggests that fine-tuning attacks in one language impact the safety-related parameters of different languages, more than they do to other types of parameters (even for the helpfulness-related parameters in the same languages). Figures 3 and 6 further validate these findings: stitching the bilingual intersections of localized parameters (e.g., $\bm{\gamma}_{\text{{EN}}}\cap\bm{\gamma}_{\text{{IT}}}$) back onto the original safety-aligned multilingual LLMs (e.g., $\bm{\theta}_{\text{EN}}^{\text{EN}\cap\text{IT}}$) (orange bars) reports similarly large violation rates as the jailbroken fine-tuned models $\bm{\theta}_{{l_{\text{ft}}}}$ (blue bar), whereas the benign baseline $\bm{\theta}_{{\text{Benign}_{l_{\text{ft}}}}}$ (green bar) and the original safety-aligned multilingual LLMs $\bm{\theta}_{\text{{pre}}}$ (red bar) remain safe. Moreover, we hypothesize that the preference for the English language showed in Table 1 by Llama-3.1-8B, can be explained by the findings in Wendler et al. (2024), where it is demonstrated that the “concept space” in the models of the Llama family is more closely aligned with English than with other languages (Table 2 also suggests similar results).

After establishing that pairs of localized sets of parameters share information about safety in the bilingual case, we now identify the language-agnostic safety parameters in the multilingual case, which is the global intersection of localized sets of parameters, given a single $\bm{\theta}_{{l_{\text{ft}}}}$. We measure the degree of overlapping of different sets of parameters using the aforementioned global SIR metric. Again, we compare the global SIR metric with benign and random baselines similar as before.

Table 2 confirms the existence of such language-agnostic safety parameters within multilingually safety-aligned LLMs. This is demonstrated by the global SIR${}_{L_{\text{pool}}}$ being larger than the SIR values for our baselines–––including benign baseline where we measure the overlapping area after harmful and benign fine-tuning in the same language. We thus draw the following conclusion: there exists a language-agnostic safety parameters within multilingual safety-aligned LLMs, and fine-tuning attacks (in Section 2.2) update these parameters and thus produce harmful behaviors across different languages.