Symb= i - Ninja AI analysis

what does this look like to you: {'role': 'user', 'co= ntent': '<tool_result> <scrape-webpage> ToolResult(success=3DTr= ue, output=3D\'[\\n {\\n "title": "Self-Reflection Makes Large Language= Models Safer, Less Biased, and Ideologically Neutral",\\n "url": "https= ://arxiv.org/html/2406.10400v2",\\n "text": "\\\\n\\\\n# Self-Reflection= Makes Large Language Models Safer, Less Biased, and Ideologically Neutral\= \\\n\\\\nFengyuan Liu1+,\\\\nNouar AlDahoul1+,\\\\nGregory Eady2,\\\\nYasir= Zaki1,\\\\\\\\*,\\\\nTalal Rahwan1,\\\\\\\\*\\\\n\\\\n1New York University= Abu Dhabi, UAE\\\\n2University of Copenhagen, Denmark\\\\n\\\\n+Joint firs= t authors.\\\\n\\\\\\\\*Joint senior authors. Correspondence: yasir.zaki@ny= u.edu, talal.rahwan@nyu.edu\\\\n\\\\n###### Abstract\\\\n\\\\nPrevious stud= ies proposed that the reasoning capabilities of large language models (LLMs= ) can be improved through self-reflection, i.e., letting LLMs reflect on th= eir own output to identify and correct mistakes in the initial responses. H= owever, earlier experiments offer mixed results when it comes to the benefi= ts of self-reflection. Furthermore, prior studies on self-reflection are pr= edominantly concerned with the reasoning capabilities of models, ignoring t= he potential for self-reflection in safety, bias, and ideological leaning.\= \\\nHere, by conducting a series of experiments testing LLM\\\\u2019s self-= reflection capability in various tasks using a variety of prompts and diffe= rent LLMs, we make several contributions to the literature. First, we recon= cile conflicting findings regarding the benefit of self-reflection, by demo= nstrating that the outcome of self-reflection is sensitive to prompt wordin= g\\\\u2014both the original prompt that are used to elicit an initial answe= r and the subsequent prompt used to self-reflect. Specifically, although se= lf-reflection may improve the reasoning capability of LLMs when the initial= response is simple, the technique cannot improve upon the state-of-the-art= chain-of-thought (CoT) prompting. Second, we show that self-reflection can= lead to safer (75.8% reduction in toxic responses while preserving 97.8% n= on-toxic ones), less biased (77% reduction in gender biased responses, whil= e preserving 94.3% unbiased ones), and more ideologically neutral responses= (100% reduction in partisan leaning response, while preserving 87.7% non-p= artisan ones). The paper concludes by discussing the implications of our fi= ndings on the deployment of large language models. We release our experimen= ts at [github.com/Michael98Liu/self-reflection](https://github.com/Michael9= 8Liu/self-reflection).\\\\n\\\\nSelf-Reflection Makes Large Language Models= Safer, Less Biased, and Ideologically Neutral\\\\n\\\\nFengyuan Liu1+,\\\\= nNouar AlDahoul1+,\\\\nGregory Eady2,\\\\nYasir Zaki1,\\\\\\\\*,\\\\nTalal = Rahwan1,\\\\\\\\*\\\\n\\\\n1New York University Abu Dhabi, UAE\\\\n2Univers= ity of Copenhagen, Denmark\\\\n\\\\n+Joint first authors.\\\\n\\\\\\\\*Join= t senior authors. Correspondence: yasir.zaki@nyu.edu, talal.rahwan@nyu.edu\= \\\n\\\\n## 1 Introduction\\\\n\\\\nWhen it comes to the benefits of self-r= eflection, literature shows conflicting results. Previous research demonstr= ated that the reasoning capability of large language models (LLMs) can be i= mproved through self-reflection, i.e., prompting the models to reflect on t= heir own response to identify and correct potential mistakes, devoid of any= external feedback\\\\u00a0Chen et\\\\u00a0al. ([2023](https://arxiv.org/ht= ml/2406.10400v2#bib.bib2)); Madaan et\\\\u00a0al. ([2024](https://arxiv.org= /html/2406.10400v2#bib.bib12)).\\\\nHowever, several recent studies have ch= allenged the benefits of self-reflection by pointing out issues with some o= f the earlier experiments and providing evidence that self-reflection may a= ctually deteriorate performance\\\\u00a0Chen et\\\\u00a0al. ([2023](https:/= /arxiv.org/html/2406.10400v2#bib.bib2)); Huang et\\\\u00a0al. ([2023](https= ://arxiv.org/html/2406.10400v2#bib.bib7)); Stechly et\\\\u00a0al. ([2023](h= ttps://arxiv.org/html/2406.10400v2#bib.bib18)); Valmeekam et\\\\u00a0al. ([= 2023](https://arxiv.org/html/2406.10400v2#bib.bib19)).\\\\n\\\\nTable\\\\u0= 0a0[1](https://arxiv.org/html/2406.10400v2#S1.T1 \\\\"Table 1 \\\\u2023 1 I= ntroduction \\\\u2023 Self-Reflection Makes Large Language Models Safer, Le= ss Biased, and Ideologically Neutral\\\\") summarizes self-reflection liter= ature, demonstrating two issues with current research on self-reflection. F= irst, with the exception of [Zhang et\\\\u00a0al.](https://arxiv.org/html/2= 406.10400v2#bib.bib22)\\\\u2019s ([2024](https://arxiv.org/html/2406.10400v= 2#bib.bib22)), studies on self-reflection tend to rely on a single prompt. = This is potentially problematic, since Huang et\\\\u00a0al. ([2023](https:/= /arxiv.org/html/2406.10400v2#bib.bib7)) hypothesized that self-reflection p= rompts might introduce biases steering models away from optimal responses. = However, the degree to which self-reflection is sensitive to prompt constru= ction has not been systematically investigated to date. Second, with the so= le exception of\\\\u00a0[Gou et\\\\u00a0al.](https://arxiv.org/html/2406.10= 400v2#bib.bib5)\\\\u2019s ([2023](https://arxiv.org/html/2406.10400v2#bib.b= ib5)), the existing literature is predominantly concerned with the reasonin= g capabilities of LLMs, ignoring other aspects such as safety, fairness, an= d ideological neutrality that could potentially be improved through self-re= flection.\\\\n\\\\nAgainst this background, we conduct experiments to syste= matically investigate how self-reflection affects the reasoning, safety, bi= as, and ideological leaning of LLMs. To understand the intrinsic ability of= such models to self-improve, we specifically focus on self-reflection with= natural language prompts, without any external feedback. More precisely, t= his paper focuses on the following five Research Questions (RQs):\\\\n\\\\n= * \\\\u2022\\\\n\\\\n RQ1: How does self-reflection impact reasoning, safe= ty, gender bias, and ideological leaning?\\\\n* \\\\u2022\\\\n\\\\n RQ2: H= ow does the ability to self-reflect vary between proprietary and open-sourc= e LLMs?\\\\n* \\\\u2022\\\\n\\\\n RQ3: Is self-reflection sensitive to pro= mpt construction?\\\\n* \\\\u2022\\\\n\\\\n RQ4: To what extent does a pro= mpt that is optimal for a specific LLM and a specific task maintain its eff= ectiveness across different LLMs and tasks?\\\\n* \\\\u2022\\\\n\\\\n RQ5:= Does cross-reflection between multiple LLMs affect detection of unsafe and= ideologically skewed responses?\\\\n\\\\n| Reference | Self-Ref lection Pro= mpt | Tasks |\\\\n| --- | --- | --- |\\\\n| [Gou et\\\\u00a0al.](https://ar= xiv.org/html/2406.10400v2#bib.bib5)\\\\u2019s ([2023](https://arxiv.org/htm= l/2406.10400v2#bib.bib5)) | What\\\\u2019s the problem with the above answe= r? | Reasoning, coding, safety |\\\\n| [Huang et\\\\u00a0al.](https://arxiv= .org/html/2406.10400v2#bib.bib7)\\\\u2019s ([2023](https://arxiv.org/html/2= 406.10400v2#bib.bib7)) | Assume that this answer could be either correct or= incorrect. Review the answer carefully and report any serious problems you= find. | Reasoning |\\\\n| [Li et\\\\u00a0al.](https://arxiv.org/html/2406.= 10400v2#bib.bib10)\\\\u2019s ([2024](https://arxiv.org/html/2406.10400v2#bi= b.bib10)) | Please review and critique your previous response. | Reasoning = |\\\\n| [Stechly et\\\\u00a0al.](https://arxiv.org/html/2406.10400v2#bib.bi= b18)\\\\u2019s ([2023](https://arxiv.org/html/2406.10400v2#bib.bib18)) | Pl= ease check if this coloring is correct \\\\u2026 If it is, say \\\\u2018Ver= ifier confirmed success.\\\\u2019 Do not provide anything else in your resp= onse. If it is incorrect, please point out which same-color vertices share = an edge. | Reasoning |\\\\n| [Renze and Guven](https://arxiv.org/html/2406.= 10400v2#bib.bib17)\\\\u2019s ([2024](https://arxiv.org/html/2406.10400v2#bi= b.bib17)) | Reflect on your incorrect solution. | Reasoning |\\\\n| [Zhang = et\\\\u00a0al.](https://arxiv.org/html/2406.10400v2#bib.bib22)\\\\u2019s ([= 2024](https://arxiv.org/html/2406.10400v2#bib.bib22)) | Please carefully ex= amine the previous responses for correctness, and provide detailed feedback= . | Reasoning |\\\\n| [Zhang et\\\\u00a0al.](https://arxiv.org/html/2406.10= 400v2#bib.bib22)\\\\u2019s ([2024](https://arxiv.org/html/2406.10400v2#bib.= bib22)) | Please review your previous responses for any errors, and provide= detailed feedback. | Reasoning |\\\\n| [Zhang et\\\\u00a0al.](https://arxi= v.org/html/2406.10400v2#bib.bib22)\\\\u2019s ([2024](https://arxiv.org/html= /2406.10400v2#bib.bib22)) | Do you think the previous response is correct o= r not, and if not please point out where is wrong. | Reasoning |\\\\n| [Zha= ng et\\\\u00a0al.](https://arxiv.org/html/2406.10400v2#bib.bib22)\\\\u2019s= ([2024](https://arxiv.org/html/2406.10400v2#bib.bib22)) | Please carefully= evaluate the quality of the previous response and point out if you feel so= mething is not appropriate. | Reasoning |\\\\n| [Zhang et\\\\u00a0al.](http= s://arxiv.org/html/2406.10400v2#bib.bib22)\\\\u2019s ([2024](https://arxiv.= org/html/2406.10400v2#bib.bib22)) | Please double-check the previous respon= se for any errors. If there are any errors, please point them out. | Reason= ing |\\\\n\\\\nTable 1: \\\\nPrompts used in self-reflection experiments. P= hrases that we hypothesize may bias self-reflection outcomes are italicized= . \\\\u00a0Zhang et\\\\u00a0al. ([2024](https://arxiv.org/html/2406.10400v2= #bib.bib22)) adopted 10 different reflection prompts, but only the first fi= ve are listed here. We consider all tasks related to problem solving and qu= estion-answering as tasks testing the LLMs\\\\u2019 \\\\u201creasoning\\\\u= 201d ability.\\\\n\\\\n## 2 Background and Related Work\\\\n\\\\nSelf-refle= ction. LLMs could correct their own response using self-reflection while in= corporating external feedback from humans\\\\u00a0Zheng et\\\\u00a0al. ([20= 23](https://arxiv.org/html/2406.10400v2#bib.bib23)), tools\\\\u00a0Gou et\\= \\u00a0al. ([2023](https://arxiv.org/html/2406.10400v2#bib.bib5)), external= metrics, or other LLMs\\\\u00a0Paul et\\\\u00a0al. ([2023](https://arxiv.o= rg/html/2406.10400v2#bib.bib16)). Another way to self-reflect is to rely on= the model\\\\u2019s own reasoning capability\\\\u00a0Pan et\\\\u00a0al. ([= 2024](https://arxiv.org/html/2406.10400v2#bib.bib15)). Here, we focus on ve= rbal reflection without any external feedback, akin to us humans editing a = draft of our own writing to improve its quality. This approa ch has been sho= wn to improve the performance of LLMs in various tasks\\\\u00a0Chen et\\\\u= 00a0al. ([2023](https://arxiv.org/html/2406.10400v2#bib.bib2)).\\\\n\\\\nHo= wever, more recent studies challenged these claims by presenting mixed resu= lts when it comes to the benefit of self-reflection\\\\u00a0Li et\\\\u00a0a= l. ([2024](https://arxiv.org/html/2406.10400v2#bib.bib10)); Huang et\\\\u00= a0al. ([2023](https://arxiv.org/html/2406.10400v2#bib.bib7)).\\\\nNotably, = Huang et\\\\u00a0al. ([2023](https://arxiv.org/html/2406.10400v2#bib.bib7))= pointed out three major issues with some earlier experiments and concluded= that the observed performance gain should not be attributed to self-reflec= tion but rather to various exogenous factors. In our study, we avoid the sa= me mistakes documented in\\\\u00a0[Huang et\\\\u00a0al.](https://arxiv.org/= html/2406.10400v2#bib.bib7)\\\\u2019s ([2023](https://arxiv.org/html/2406.1= 0400v2#bib.bib7)), e.g., by refraining from providing additional informatio= n (such as oracle labels) in the self-reflection prompt.\\\\n\\\\nLLMs are = sensitive to prompt construction. Studies have shown that prompt constructi= on is crucial to elicit the reasoning ability of LLMs\\\\u00a0Wei et\\\\u00= a0al. ([2022](https://arxiv.org/html/2406.10400v2#bib.bib21)). Meanwhile, a= dversarially constructed prompts could hurt performance by simply changing = a few words in the prompt\\\\u00a0Arakelyan et\\\\u00a0al. ([2024](https://= arxiv.org/html/2406.10400v2#bib.bib1)); Verma et\\\\u00a0al. ([2024](https:= //arxiv.org/html/2406.10400v2#bib.bib20)). Our study contributes to this li= ne of research by documenting a specific case of sensitivity introduced by = prompts in the context of self-reflection.\\\\n\\\\n## 3 Data and Experimen= ts\\\\n\\\\n### 3.1 Reasoning Datasets\\\\n\\\\nTo assess whether self-refl= ection improves the reasoning capabilities of LLMs, we use three datasets. = The first is the MEDQA-USMLE dataset consisting of questions and\\\\ntheir = associated answer from a professional medical exam, namely the Medical Boar= d Examination in the USA\\\\u00a0Jin et\\\\u00a0al. ([2021](https://arxiv.o= rg/html/2406.10400v2#bib.bib9)). These questions are designed to examine th= e doctors\\\\u2019 professional knowledge, which means that many questions = require multi-hop logical reasoning.\\\\nThe second is the Massive Multitas= k Language Understanding (MMLU), a benchmark dataset consisting of multiple= -choice questions drawn from 57 academic subjects across STEM, the humaniti= es, and the social sciences\\\\u00a0Hendrycks et\\\\u00a0al. ([2020](https:= //arxiv.org/html/2406.10400v2#bib.bib6)). The second is the GSM8K, a large = dataset of grade school math problems\\\\u00a0Cobbe et\\\\u00a0al. ([2021](= https://arxiv.org/html/2406.10400v2#bib.bib3)). Our experiments uses the te= st set of MEDQA-USMLE (1273 questions), MMLU (a random sample of 1,000 ques= tions), and GSM8K (1319 questions).\\\\n\\\\n### 3.2 Safety Dataset\\\\n\\\= \nTo assess whether self-reflection improves the safety guardrails of LLMs,= we use a dataset of French sentences by combining a dataset of safe and an= other dataset of unsafe (i.e., those with offensive language) French senten= ces. The unsafe and safe examples are a randomly selected subset of the saf= ety dataset proposed by [Jan et\\\\u00a0al.](https://arxiv.org/html/2406.10= 400v2#bib.bib8)\\\\u2019s ([2025](https://arxiv.org/html/2406.10400v2#bib.b= ib8)).\\\\n\\\\nTo curate the final dataset of a mixture of safe and unsafe= content, we first attempted to translate each French sentence from those t= wo datasets to English using three LLMs including GPT-4o-mini\\\\u00a0OpenA= I et\\\\u00a0al. ([2024](https://arxiv.org/html/2406.10400v2#bib.bib14)), G= emini-1.5-Flash\\\\u00a0GeminiTeam et\\\\u00a0al. ([2024](https://arxiv.org= /html/2406.10400v2#bib.bib4)), and Llama-3.2-3B\\\\u00a0Llama\\\\u00a0Team = ([2024](https://arxiv.org/html/2406.10400v2#bib.bib11)). Then, for each res= ponse provided by an LLM, we used GPT-4o Judge\\\\u00a0Jan et\\\\u00a0al. (= [2025](https://arxiv.org/html/2406.10400v2#bib.bib8)) to determine whether = the LLM provides a translation or not.\\\\nIdeally, the LLMs should transla= te every safe sentence, and should refuse to translate every unsafe sentenc= e. However, we found 445 unsafe sentences for which all three LLMs provided= a translation (instead of refusing to translate). We also found 482 safe s= entences for which all three LLMs provided a translation (as they should).\= \\\n\\\\nOur final dataset consists of all the aforementioned 927 sentence-= translation pairs. As such, without any self-reflection, 48% of the respons= es (445) were unsafe, while the remaining 52% of responses (482) were safe.= See Appendix Figure\\\\u00a0[11](https://arxiv.org/html/2406.10400v2#A1.F1= 1 \\\\"Figure 11 \\\\u2023 Appendix A Appendix \\\\u2023 Self-Reflection Ma= kes Large Language Models Safer, Less Biased, and Ideologically Neutral\\\\= ") and [12](https://arxiv.org/html/2406.10400v2#A1.F12 \\\\"Figure 12 \\\\u= 2023 Appendix A Appendix \\\\u2023 Self-Reflection Makes Large Language Mod= els Safer, Less Biased, and Ideologically Neutral\\\\") for examples in our= dataset. Later on in our experiments, we apply self-reflection to determin= e whether it affects the percentage of safe responses.\\\\n\\\\n### 3.3 Gen= der Bias Dataset\\\\n\\\\nTo evaluate whether self-reflection improves the = bias guardrails of LLMs, we create a benchmark dataset for gender bias, con= sisting of biased and unbiased responses generated by the above three LLMs.= Here, we consider an LLM response to be gender biased if it needlessly use= s a gendered pronoun in response to a generic question that did not require= the use of such a pronoun. In contrast, unbiased responses are those that = do not use any pronouns or use gender-neutral pronouns such as \\\\u201cthe= y\\\\u201d or \\\\u201cthem.\\\\u201d\\\\n\\\\nTo create this dataset, we f= irst asked each LLM to generate 3000 responses given the following prompt: = Discuss a person practicing a profession in 10 lines. We randomly selected = 1000 responses that contain gender pronouns (i.e., she, he, her, his, and h= im), and labeled them as biased. We also randomly selected 700 responses th= at do not contain such pronouns and labeled them as unbiased.\\\\n\\\\nIt s= hould be noted that not all gender pronouns are associated with bias. To en= sure that our dataset is not creating a spurious correlation between gender= bias and gender pronoun, we supplement the unbiased dataset with sentences= that contain a gender pronoun yet do not have gender bias. To this end, we= prompt each language model 100 times to generate sentences discussing the = impact of a specific public figure. For example, the prompts include Discus= s Nelson Mandela\\\\u2019s leadership and its impact on South Africa, Discu= ss Thomas Edison\\\\u2019s role in shaping modern electrical technology, an= d Discuss Jackie Robinson\\\\u2019s breaking of baseball\\\\u2019s color ba= rrier. All these prompts are themselves generated by an LLM. These 300 resp= onses contain gender pronouns when discussing a certain profession (e.g., p= oliticians, athletes, etc.), yet they are not biased because they are conce= rned with specific people with publicly known genders. Together, this gives= us a balanced dataset of 50% of responses (1000) with gender bias and 50% = of responses (700 + 300) without such bias. See Appendix Figure\\\\u00a0[13= ](https://arxiv.org/html/2406.10400v2#A1.F13 \\\\"Figure 13 \\\\u2023 Appen= dix A Appendix \\\\u2023 Self-Reflection Makes Large Language Models Safer,= Less Biased, and Ideologically Neutral\\\\") and [14](https://arxiv.org/ht= ml/2406.10400v2#A1.F14 \\\\"Figure 14 \\\\u2023 Appendix A Appendix \\\\u20= 23 Self-Reflection Makes Large Language Models Safer, Less Biased, and Ideo= logically Neutral\\\\") for examples in our dataset. \\\\n\\\\nBy using this= dataset, we ensure that, for any language model to determine whether a cer= tain response contains gender bias, it is insufficient to be given the resp= onse alone; the model also needs to know the context in which the response = was provided, i.e., it needs to be given both the question and the response= to that question.\\\\n\\\\n### 3.4 Ideological Leaning Dataset\\\\n\\\\nTo= evaluate whether self-reflection leads to more ideologically balanced resp= onses, we proposed a benchmark dataset for ideological leaning, consisting = of partisan-aligned and moderate responses, generated using the same three = LLMs mentioned above. The prompts used to generate the partisan-aligned sam= ples were: Discuss in 10 lines your perspective on how the U.S. should hand= le [\\\\u2018healthcare\\\\u2019,\\\\u2018climate change\\\\u2019,\\\\u2018= tax\\\\u2019,\\\\u2018immigration\\\\u2019,\\\\u2018education\\\\u2019, \\\= \u2018gun control\\\\u2019]. On the other hand, the prompts used to generat= e the moderate samples were: Discuss in 10 lines your perspective on how th= e U.S. should handle [\\\\u2018healthcare\\\\u2019,\\\\u2018climate change\= \\\u2019,\\\\u2018tax\\\\u2019,\\\\u2018immigration\\\\u2019,\\\\u2018educa= tion\\\\u2019, \\\\u2018gun control\\\\u2019], considering both Democratic = and Republican viewpoints.\\\\nTo automatically label the responses in this= dataset, we used large versions of LLMs (GPT-4o and Gemini-1.5-pro) with t= he following prompt: Is this response Democratic-aligned, Republican-aligne= d, or Moderate? Then we selected only examples that have perfect agreement = between both LLMs. This yielded a total of 600 examples (200 from each LLM)= that were evenly distributed between two classes: \\\\u201cpartisan-aligne= d\\\\u201d and \\\\u201cmoderate.\\\\u201d. See Appendix Figure\\\\u00a0[15= ](https://arxiv.org/html/2406.10400v2#A1.F15 \\\\"Figure 15 \\\\u2023 Appen= dix A Appendix \\\\u2023 Self-Reflection Makes Large Language Models Safer,= Less Biased, and Ideologically Neutral\\\\") and [16](https://arxiv.org/ht= ml/2406.10400v2#A1.F16 \\\\"Figure 16 \\\\u2023 Appendix A Appendix \\\\u20= 23 Self-Reflection Makes Large Language Models Safer, Less Biased, and Ideo= logically Neutral\\\\") for examples in our dataset.\\\\n\\\\n### 3.5 Exper= iment Setup\\\\n\\\\nSelf-reflection experiments are performed using three = different language models: GPT-4o-mini, Gemini 1.5-Flash, and Llama 3.2-3B-= Instruct. We experiment with two different temperature values (0 and 1)\\\\= u2014the results corresponding to temperature=3D1 are reported in the main = manuscript, while those corresponding to temperature=3D0 are reported in th= e appendix. When temperature=3D1, all experiments are repeated three times,= and we report the mean accuracy and its standard deviation across three re= petitions. Data and code to reproduce our results can be found at [github.c= om/Michael98Liu/self-reflection](https://github.com/Michael98Liu/self-refle= ction).\\\\n\\\\n## 4 Evaluations\\\\n\\\\n### 4.1 Self-Reflection Marginal= ly Improves Reasoning Capability\\\\n\\\\nWe begin by evaluating whether se= lf-reflection improves the reasoning capabilities of LLMs. To this end, we = obtain initial answers from each language model for each question in the ME= DQA-USMLE, GSM8K and MMLU datasets. We experiment with two types of prompts= to obtain the initial answers: (1) a simple prompt without chain-of-though= t (CoT) prompting\\\\u00a0Wei et\\\\u00a0al. ([2022](https://arxiv.org/html= /2406.10400v2#bib.bib21)), and (2) a more complex prompt incorporating CoT.= We then instruct the same language model to reflect on its initial answer = using different reflection prompts. Our analysis yields two important findi= ngs.\\\\n\\\\nFirst, as can be seen in Figure\\\\u00a0[1](https://arxiv.org= /html/2406.10400v2#S4.F1 \\\\"Figure 1 \\\\u2023 4.1 Self-Reflection Margin= ally Improves Reasoning Capability \\\\u2023 4 E valuations \\\\u2023 Self-R= eflection Makes Large Language Models Safer, Less Biased, and Ideologically= Neutral\\\\"), the effectiveness of self-reflection varies significantly d= epending on both the reflection prompt and the language model being evaluat= ed. For example, when asked to self-reflect using the prompt \\\\u201ccriti= cally reflect upon original answer\\\\u201d, GPT achieves an accuracy of 63= .7% on answering medical questions, achieving higher accuracy than 62.1%, i= ts original accuracy. However, the accuracy is much lower (52.5%) if self-r= eflection is done by asking the model to verify if \\\\u201coriginal answer= is wrong\\\\u201d. Even the same prompt can lead to dramatically different= performance with different language models. For example, although the prom= pt \\\\u201ccritically reflect upon original answer\\\\u201d results in hig= her accuracy in GPT, it leads much lower accuracy with Gemini (55.6% to 43.= 4%). This suggests that any researchers evaluating the impact of self-refle= ction should do so with a variety of prompts and language models. See Appen= dix Figure\\\\u00a0[6](https://arxiv.org/html/2406.10400v2#A1.F6 \\\\"Figur= e 6 \\\\u2023 Appendix A Appendix \\\\u2023 Self-Reflection Makes Large Lan= guage Models Safer, Less Biased, and Ideologically Neutral\\\\") and [7](ht= tps://arxiv.org/html/2406.10400v2#A1.F7 \\\\"Figure 7 \\\\u2023 Appendix A = Appendix \\\\u2023 Self-Reflection Makes Large Language Models Safer, Less = Biased, and Ideologically Neutral\\\\") for a full list of the exact prompt= s tested.\\\\n\\\\nSecond, our experiments reveal that the reasoning abilit= y of language models does not improve, or only marginally improve, after se= lf-reflection. Specifically, when answering medical questions in the MEDQA-= USMLE dataset and the math problems in the GSM8K dataset, language models a= chieve the best accuracy when prompted using CoT without self-reflection. W= hen answering multiple-choice questions in the MMLU dataset, language model= s can achieve marginally better accuracy with self-reflection, but the diff= erences are statistically insignificant in all cases.\\\\nWe do notice some= exceptions, however, where self-reflection improves upon the original accu= racy. When Gemini is initially asked to solve math problems without outputt= ing the intermediate reasoning steps, for example, it originally achieved a= n accuracy of 37.3% (without intermediate reasoning steps), but the accurac= y rose to 88.1% when the model was prompted to critically reflect on its or= iginal answer (Appendix Figure\\\\u00a0[3](https://arxiv.org/html/2406.1040= 0v2#A1.T3 \\\\"Table 3 \\\\u2023 Appendix A Appendix \\\\u2023 Self-Reflect= ion Makes Large Language Models Safer, Less Biased, and Ideologically Neutr= al\\\\")). However, this improvement remains lower than the accuracy achiev= ed using CoT prompting alone (93.1%). Note that both GPT and Llama provide = intermediate reasoning steps even when they are not explicitly instructed t= o do so. Hence, they do not benefit from self-reflection the same way Gemin= i does.\\\\n\\\\nIntuitively, for self-reflection to be effective, LLMs mus= t be able to identify and correct its mistakes while preserving its already= -correct answers. Therefore, we hypothesize that the reason why self-reflec= tion often fails to improve accuracy is that the models fail to distinguish= between correct and incorrect initial answers during self-reflection. To t= est this hypothesis, we use GSM8K as an example and plot the percentage of = answers that are changed during self-reflection among answers that were ini= tially wrong, and those that were initially correct, respectively. We find = that the rates at which correct and incorrect answers are modified are stro= ngly correlated (Appendix Figure\\\\u00a0[5](https://arxiv.org/html/2406.10= 400v2#A1.F5 \\\\"Figure 5 \\\\u2023 Appendix A Appendix \\\\u2023 Self-Refl= ection Makes Large Language Models Safer, Le ss Biased, and Ideologically Ne= utral\\\\")). This indicates that language models tend to either change bot= h correct or wrong answers, or retain both, at a high rate, demonstrating a= n inability to reliably distinguish between correct and incorrect responses= .\\\\n\\\\n \\\\n\\\\nFigure 1= : The accuracy on MEDQA-USMLE dataset before and after self-reflection. The= accuracy before self-reflection is denoted as \\\\u201coriginal answer\\\\= u201d, and the rest correspond to accuracies after self-reflection using on= e of the five prompts (Appendix Figure\\\\u00a0[6](https://arxiv.org/html/2= 406.10400v2#A1.F6 \\\\"Figure 6 \\\\u2023 Appendix A Appendix \\\\u2023 Sel= f-Reflection Makes Large Language Models Safer, Less Biased, and Ideologica= lly Neutral\\\\")). Panel on the left correspond to self-reflections on ini= tial responses obtained using simple initial response without CoT, while th= e panel on the right correspond to self-reflection on initial responses wit= h CoT. The temperature value is set to 1 for text generation.\\\\n\\\\n### = 4.2 Self-Reflection Improves Safety\\\\n\\\\nHaving demonstrated that self-= reflection only marginally improves the reasoning capability of language mo= dels, we next turn to evaluate the ability of self-reflection to trigger th= e safety guardrails in LLMs. In this context, the primary goal of self-refl= ection is to refuse translating sentences with harmful content while retain= ing the translation of the safe ones. To see whether such is the case, we u= sed two performance metrics. First, we evaluated whether LLMs can better de= tect unsafe responses after self-reflection; this was done by calculating t= he percentage of unsafe responses that are correctly blocked (i.e., true po= sitive rate, or TPR). Second, we assessed the models\\\\u2019 helpfulness (= overblocking), measured as the number of safe responses retained (true nega= tive rate, or TNR). As can be seen in Figure\\\\u00a0[2](https://arxiv.org/= html/2406.10400v2#S4.F2 \\\\"Figure 2 \\\\u2023 4.2 Self-Reflection Improve= s Safety \\\\u2023 4 Evaluations \\\\u2023 Self-Reflection Makes Large Lang= uage Models Safer, Less Biased, and Ideologically Neutral\\\\"), we found t= hat different LLMs exhibit different levels of safety in their responses af= ter self-reflection, likely due to differences in their architectures, trai= ning data, alignment processes, and optimization goals. Specifically, GPT-4= o achieves the best overall accuracy of 86.8%, outperforming both Gemini (6= 8.2%) and Llama (56.9%) after self-reflection. GPT-4o was able to detect un= safe users\\\\u2019 queries and enhance the model\\\\u2019s safety with a h= igh TPR of 75.8%. Additionally, it is still able to keep the model\\\\u2019= s helpfulness with a TNR of 97.7%. This shows the excellent self-reflection= capability of GPT-4o.\\\\nWhile Gemini was only able to enhance safety gua= rdrail by 36.3%, it does so without compromising the helpfulness. Llama, ho= wever, degraded the helpfulness without enhancing the safety significantly.= \\\\n\\\\nOverall, GPT is well-balanced and has a stronger self-awareness f= ilter, preventing it from confidently engaging in potentially harmful topic= s. On the other hand, Gemini\\\\u2019s design choices may stem from a prefe= rence for usability (correct translation) over strict filtering, leading to= strong self-reflection in safe contexts but weaker self-awareness in handl= ing unsafe cases.\\\\n\\\\nSo far, we have demonstrated that both GPT and G= emini improve their ability to distinguish between safe and unsafe response= s when prompted to self-reflect upon their own responses. However, it is no= t entirely clear whether such improvements are due to self-reflection, or s= imply due to the extra words contained in the self-reflection prompts.\\\\n= In other words, we ask: Can a language model just as accurately distinguish= between safe and unsafe responses if as ked to pay attention to safety issu= es while translating (which we call \\\\u201cnon-reflection\\\\u201d), inst= ead of being asked to self-reflect? To this end, we evaluated the performan= ce on the non-reflection scenario by crafting a prompt to censor the user\\= \\u2019s queries only without considering responses. It was found that self= -reflection yields the same performance as non-reflection in terms of unsaf= e content detection accuracy (TPR). However, using self-reflection can sign= ificantly enhance the helpfulness accuracy (from 82.9% to 97.8%) of the GPT= -4o model. The reason is that GPT-4o blocks more questions (regardless of w= hether they are safe or not) when only French text queries are considered, = but it is able to better distinguish safe from unsafe responses after self-= reflection when both the French query and its English translation are given= . On the contrary, Gemini shows better performance in the non-reflection sc= enario. Finally, Llama again exhibited the worst performance among the thre= e in non-reflection scenarios. Appendix Table\\\\u00a0[6](https://arxiv.org= /html/2406.10400v2#A1.T6 \\\\"Table 6 \\\\u2023 Appendix A Appendix \\\\u20= 23 Self-Reflection Makes Large Language Models Safer, Less Biased, and Ideo= logically Neutral\\\\") presents the mean TPR, TNR, and overall accuracy fo= r the three language models in no-reflection scenario.\\\\n\\\\nIt should b= e noted that the impact of self-reflection varies drastically depending on = the prompt. Take GPT-4o-mini for example; the accuracy after self-reflectio= n ranges between 63.1% and 86.8%. Moreover, a poorly chosen prompt may actu= ally deteriorate the performance to be worse than the initial accuracy. Thi= s finding highlights the importance of prompt engineering in self-reflectio= n tasks.\\\\nSee Appendix Figure\\\\u00a0[8](https://arxiv.org/html/2406.10= 400v2#A1.F8 \\\\"Figure 8 \\\\u2023 Appendix A Appendix \\\\u2023 Self-Refl= ection Makes Large Language Models Safer, Less Biased, and Ideologically Ne= utral\\\\") for a full list of prompts tested. Note that in those prompts, = x refers to \\\\u201cyour previous response\\\\u201d in the self-reflection= scenario or \\\\u201cthe given question\\\\u201d in the no-reflection scen= ario.\\\\n\\\\n \\\\n\\\\nFigu= re 2: TPR, TNR, and overall accuracy after self-reflection using the safety= dataset. The temperature value is set to 1 for text generation. See Append= ix Figure\\\\u00a0[8](https://arxiv.org/html/2406.10400v2#A1.F8 \\\\"Figure= 8 \\\\u2023 Appendix A Appendix \\\\u2023 Self-Reflection Makes Large Lang= uage Models Safer, Less Biased, and Ideologically Neutral\\\\") for each of= the seven prompts used in experiments.\\\\n\\\\n### 4.3 Self-Reflection Re= duces Gender Bias\\\\n\\\\nNext, we evaluate whether self-reflection reduce= s gender bias. We test the LLMs\\\\u2019 ability to distinguish between bia= sed and unbiased responses after self-reflection in the same manner as befo= re. Consequently, we calculate the TPR (the percentage of biased responses = that are correctly identified as such) as well as the TNR (the percentage o= f unbiased responses correctly identified as such).\\\\n\\\\nAs can be seen= in Figure\\\\u00a0[3](https://arxiv.org/html/2406.10400v2#S4.F3 \\\\"Figur= e 3 \\\\u2023 4.3 Self-Reflection Reduces Gender Bias \\\\u2023 4 Evaluatio= ns \\\\u2023 Self-Reflection Makes Large Language Models Safer, Less Biased= , and Ideologically Neutral\\\\"), the findings are similar to the previous= experiment.\\\\nFirst, we find that self-reflection successfully reduces g= ender bias in LLMs\\\\u2019 output from 50% to 85.6% in GPT, 71.8% in Gemin= i, and 53.3% in Llama.\\\\nSecond, as can be seen, GPT-4o again outperforms= Gemini and Llama in their overall ability to distinguish sentences with ge= nder bias from those without such bias. Specifically, GPT-4o is able to det= ect its biased responses and enhance the model\\\\u2019s fairness with a hi= gh TPR of 77.0%. Additionally, it is still able to detect unbiased response= s with a TNR of 94.3%. These results demonstrate the remarkable self-reflec= tion capability of GPT-4o. On the other hand, both Gemini and Llama exhibit= worse performance compared to GPT. However, Gemini was able to reduce gend= er bias by 44.2% while keeping false positive rate low (0.6% FPR, or 99.4% = TNR). Llama increased the false positives without substantially improving t= he bias detection. Finally, the specific prompts used to elicit self-reflec= tions affect the language models\\\\u2019 ability to accurately distinguish= biased responses from unbiased ones. See Appendix Figure\\\\u00a0[9](https= ://arxiv.org/html/2406.10400v2#A1.F9 \\\\"Figure 9 \\\\u2023 Appendix A App= endix \\\\u2023 Self-Reflection Makes Large Language Models Safer, Less Bia= sed, and Ideologically Neutral\\\\") for prompts used.\\\\n\\\\nWe notice s= everal issues limit LLMs\\\\u2019 ability to further reduce gender bias whe= n self-reflecting. GPT has classified certain biased responses as unbiased = due to its default association of professional roles with male figures when= the pronoun \\\\u201che\\\\u201d is used. Furthermore, GPT also perceives = responses as unbiased when the pronoun \\\\u201cshe\\\\u201d is included.Si= milarly, Gemini defended labeling its gender-biased response as unbiased by= arguing that language mirrors reality. It claimed that referring to a \\\\= u201cflight attendant\\\\u201d as \\\\u201cshe\\\\u201d is reasonable and t= hat using \\\\u201che\\\\u201d for a \\\\u201cplumber\\\\u201d is grammatic= ally correct. Additionally, it stated a preference for avoiding awkward phr= asing, such as using \\\\u201cthey.\\\\u201d\\\\n\\\\n \\\\n\\\\nFigure 3: TPR, TNR, and overall accuracy= after self-reflection using the gender bias dataset. The temperature value= is set to 1 for text generation. See Appendix Figure\\\\u00a0[9](https://a= rxiv.org/html/2406.10400v2#A1.F9 \\\\"Figure 9 \\\\u2023 Appendix A Appendi= x \\\\u2023 Self-Reflection Makes Large Language Models Safer, Less Biased,= and Ideologically Neutral\\\\") for each of the four prompts used in exper= iments.\\\\n\\\\n### 4.4 Self-Reflection Improves Partisan Neutrality\\\\n\= \\\nIn the fourth experiment, we used a dataset on ideological leaning to t= est how well self-reflection could tell the difference between LLMs\\\\u201= 9 ideologically partisan and ideologically neutral responses. We evaluated = how well LLMs can detect politically skewed and moderate responses by calcu= lating accuracy. Again, the findings are consistent with our earlier experi= ments on safety and gender bias. As can be seen in Figure\\\\u00a0[4](https= ://arxiv.org/html/2406.10400v2#S4.F4 \\\\"Figure 4 \\\\u2023 4.4 Self-Refle= ction Improves Partisan Neutrality \\\\u2023 4 Evaluations \\\\u2023 Self-R= eflection Makes Large Language Models Safer, Less Biased, and Ideologically= Neutral\\\\"), we found that LLMs show different levels of self-reflection= in their responses. GPT-4o tends to perform better in self-reflection scen= arios compared to Gemini and Llama. In particular, GPT-4o is able to detect= its partisan-aligned responses and enhance the model\\\\u2019s ideological= neutrality with a high accuracy of 100%. Additionally, it is still able to= detect unaligned or moderate responses with an accuracy of 87.7%. This sho= ws the remarkable self-reflection capability of GPT-4o. On the other hand, = both Gemini and Llama exhibit declined performance compared to GPT. However= , Gemini was able to enhance the model\\\\u2019s ideological neutrality by = 44.0% while keeping false positives so low. Llama increased false positives= while improving the detection of partisan-aligned responses. See Appendix = Figure\\\\u00a0[10](https://arxiv.org/html/2406.10400v2#A1.F10 \\\\"Figure = 10 \\\\u2023 Appendix A Appendix \\\\u2023 Self-Reflection Makes Large Lang= uage Models Safer, Less Biased, and Ideologically Neutral\\\\") for prompts= used in experiments.\\\\nFor prompt construction sensitivity, we can see t= hat different prompts yield varying accuracy results. These findings demons= trate the sensitivity of LLMs to prompt construction when it comes to ident= ifying moderate and partisan responses.\\\\n\\\\n \\\\n\\\\nFigure 4: TPR, TNR, and overall accuracy afte= r self-reflection using the ideological leaning dataset. The temperature va= lue is set to 1 for text generation. See Appendix Figure\\\\u00a0[10](https= ://arxiv.org/html/2406.10400v2#A1.F10 \\\\"Figure 10 \\\\u2023 Appendix A A= ppendix \\\\u2023 Self-Reflection Makes Large Language Models Safer, Less B= iased, and Ideologically Neutral\\\\") for each of the four prompts used in= experiments.\\\\n\\\\n### 4.5 Cross-Reflection\\\\n\\\\nIn the fifth exper= iment, we implemented a cross-reflection technique, where one LLM evaluates= , critiques, or refines the responses generated by another LLM. We applied = this technique to two datasets: a safety dataset and an ideological leaning= dataset. Given its superior performance in the self-reflection scenario, w= e selected GPT to critique the responses of Gemini and Llama. GPT effective= ly detected unsafe user queries and improved the safety of Gemini and Llama= , achieving a high accuracy of 79.51%. Simultaneously, it preserved the mod= els\\\\u2019 helpfulness with an accuracy of 82.09%. Additionally, it ident= ified partisan-aligned responses and enhanced ideological neutrality with a= n impressive accuracy of 99.48%, while still recognizing moderate responses= with an accuracy of 65.83%. For consistency, we used the same prompts from= the safety and ideological leaning experiments. The Table\\\\u00a0[9](http= s://arxiv.org/html/2406.10400v2#A1.T9 \\\\"Table 9 \\\\u2023 Appendix A App= endix \\\\u2023 Self-Reflection Makes Large Language Models Safer, Less Bia= sed, and Ideologically Neutral\\\\") and [10](https://arxiv.org/html/2406.1= 0400v2#A1.T10 \\\\"Table 10 \\\\u2023 Appendix A Appendix \\\\u2023 Self-Re= flection Makes Large Language Models Safer, Less Biased, and Ideologically = Neutral\\\\") in Appendix presents the TPR, TNR, and overall accuracy of th= e three LLMs evaluated on the seven (for safety) and four prompts (for ideo= logical leaning).\\\\n\\\\n### 4.6 Summary of Evaluations\\\\n\\\\nTo sum u= p, although self-reflection only marginally improves the reasoning ability = of LLMs, it can significantly increases the safety, reduces the bias, and d= ecrease partisan leaning of LLMs (RQ1). Furthremore, proprietary LLMs (name= ly, GPT-4o-mini and Gemini 1.5-Flash) consistently outperform the open-sour= ce LLM (i.e., Llama 3.2-3B) in our experiments (RQ2). However, this could b= e due to both GPT-4o-mini and Gemini 1.5-Flash having more parameters compa= red to Llama 3.2-3B. Additionally, all three LLMs tested in our experiments= are sensitive to prompt construction (RQ3), and different models have diff= erent optimal self-reflection prompt (RQ4); a prompt optimal for one model = could be the worst prompt for another model. Finally, although using GPT to= cross-reflect the responses of Gemini and Llama yields better performance = than Gemini and Llama alone, GPT provides the highest accuracy when self-re= flecting its own output (RQ5).\\\\nWe supplement Figures\\\\u00a0[1](https:= //arxiv.org/html/2406.10400v2#S4.F1 \\\\"Figure 1 \\\\u2023 4.1 Self-Reflec= tion Marginally Improves Reasoning Capability \\\\u2023 4 Evaluations \\\\u= 2023 Self-Reflection Makes Large Language Models Safer, Less Biased, and Id= eologically Neutral\\\\") to [4](https://arxiv.org/html/2406.10400v2#S4.F4 = \\\\"Figure 4 \\\\u2023 4.4 Self-Reflection Improves Partisan Neutrality \\= \\u2023 4 Evaluations \\\\u2023 Self-Reflection Makes Large Language Models= Safer, Less Biased, and Ide ologically Neutral\\\\") by reporting the full = numerical results in Appendix Table\\\\u00a0[2](https://arxiv.org/html/2406= .10400v2#A1.T2 \\\\"Table 2 \\\\u2023 Appendix A Appendix \\\\u2023 Self-Re= flection Makes Large Language Models Safer, Less Biased, and Ideologically = Neutral\\\\") to [8](https://arxiv.org/html/2406.10400v2#A1.T8 \\\\"Table 8= \\\\u2023 Appendix A Appendix \\\\u2023 Self-Reflection Makes Large Langua= ge Models Safer, Less Biased, and Ideologically Neutral\\\\")). Furthermore= , as a robustness check, all experiments are repeated for temperature=3D0, = resulting in qualitatively similar results (see Table\\\\u00a0[11](https://= arxiv.org/html/2406.10400v2#A1.T11 \\\\"Table 11 \\\\u2023 Appendix A Appen= dix \\\\u2023 Self-Reflection Makes Large Language Models Safer, Less Biase= d, and Ideologically Neutral\\\\") to [16](https://arxiv.org/html/2406.1040= 0v2#A1.T16 \\\\"Table 16 \\\\u2023 Appendix A Appendix \\\\u2023 Self-Refle= ction Makes Large Language Models Safer, Less Biased, and Ideologically Neu= tral\\\\") for results).\\\\n\\\\n## 5 Discussion and Conclusion\\\\n\\\\nI= n this study, we showed that while self-reflection shows a limited effect o= n improving the reasoning capabilities of language models, it is capable of= drastically improving the safety, gender neutrality, and political neutral= ity of language models. Given that most existing research on self-reflectio= n focuses on reasoning ability, our findings point to new avenues of resear= ch on the underexplored areas where self-reflection is beneficial.\\\\n\\\\= nWe theorize that self-reflection is best used when an LLM needs to meet mu= ltiple objectives at the same time, e.g., when a language model is asked to= translate a sentence while, at the same time, ensuring that the output is = not toxic. Our experiments demonstrated that LLMs tend to be bad at such mu= lti-tasking. In particular, when asked to translate, it occasionally does s= o while disregarding safety issues. To put it differently, the first task (= the translation) occasionally distracts the model from the second task (ens= uring that the output is not toxic). Our analysis suggests that such multi-= tasking scenarios are where self-reflection can significantly improve model= performance.\\\\n\\\\nIn conclusion, we propose self-reflection as a low-c= ost measure that can be deployed by any language model at test time to impr= ove safety, gender neutrality, and political neutrality. Our work contribut= es to the growing body of research exploring the test-time scaling idea\\\\= u00a0Muennighoff et\\\\u00a0al. ([2025](https://arxiv.org/html/2406.10400v2= #bib.bib13)), where models are allocated extra test-time compute to improve= performance.\\\\n\\\\n## Limitations\\\\n\\\\nWe conduct all self-reflecti= on experiments using the small versions of three LLMs, namely GPT-4o-mini, = Gemini-1.5-Flash, and Llama-3.2-3B. Therefore, it remains to be demonstrate= d whether bigger models exhibit the same behavior. Additionally, although o= ur experiments demonstrated that the outcome of self-reflection is sensitiv= e to prompt construction, it remains a challenge to identify the most optim= al prompt. Future work is needed to come up with systematic ways to constru= ct the most optimal prompt used for self-reflection given a specific task a= nd model.\\\\n\\\\n## References\\\\n\\\\n* Arakelyan et\\\\u00a0al. (2024)= \\\\n\\\\n Erik Arakelyan, Zhaoqi Liu, and Isabelle Augenstein. 2024.\\\\n= Semantic sensitivities and inconsistent predictions: Measuring the fragil= ity of nli models.\\\\n *arXiv preprint arXiv:2401.14440*.\\\\n* Chen et\\= \\u00a0al. (2023)\\\\n\\\\n Xinyun Chen, Maxwell Lin, Nathanael Sch\\\\u00= e4rli, and Denny Zhou. 2023.\\\\n Teaching large language models to self-d= ebug.\\\\n *arXiv preprint arXiv:2304.05128*.\\\\n* Cobbe et\\\\u00a0al. (= 2021)\\\\n\\\\n Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen,= Heewoo Jun, Lukasz Kais er, Matthias Plappert, Jerry Tworek, Jacob Hilton, = Reiichiro Nakano, et\\\\u00a0al. 2021.\\\\n Training verifiers to solve ma= th word problems.\\\\n *arXiv preprint arXiv:2110.14168*.\\\\n* GeminiTeam= et\\\\u00a0al. (2024)\\\\n\\\\n GeminiTeam, Petko Georgiev, Ving\\\\u00a0= Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vinc= ent, Zhufeng Pan, Shibo Wang, Soroosh Mariooryad, Yifan Ding, Xinyang Geng,= Fred Alcober, Roy Frostig, Mark Omernick, Lexi Walker, Cosmin Paduraru, Ch= ristina Sorokin, Andrea Tacchetti, Colin Gaffney, Samira Daruki, Olcan Serc= inoglu, Zach Gleicher, Juliette Love, Paul Voigtlaender, Rohan Jain, Gabrie= la Surita, Kareem Mohamed, Rory Blevins, Junwhan Ahn, Tao Zhu, Kornraphop K= awintiranon, Orhan Firat, Yiming Gu, Yujing Zhang, Matthew Rahtz, Manaal Fa= ruqui, Natalie Clay, Justin Gilmer, JD\\\\u00a0Co-Reyes, Ivo Penchev, Rui Z= hu, Nobuyuki Morioka, Kevin Hui, Krishna Haridasan, Victor Campos, Mahdis M= ahdieh, Mandy Guo, Samer Hassan, Kevin Kilgour, Arpi Vezer, Heng-Tze Cheng,= Raoul de\\\\u00a0Liedekerke, Siddharth Goyal, Paul Barham, DJ\\\\u00a0Stro= use, Seb Noury, Jonas Adler, Mukund Sundararajan, Sharad Vikram, Dmitry Lep= ikhin, Michela Paganini, Xavier Garcia, Fan Yang, Dasha Valter, Maja Trebac= z, Kiran Vodrahalli, Chulayuth\\\\n Asawaroengchai, Roman Ring, Norbert Ka= lb, Livio\\\\u00a0Baldini Soares, Siddhartha Brahma, David Steiner, Tianhe = Yu, Fabian Mentzer, Antoine He, Lucas Gonzalez, Bibo Xu, Raphael\\\\u00a0Lo= pez Kaufman, Laurent\\\\u00a0El Shafey, Junhyuk Oh, Tom Hennigan, George va= n\\\\u00a0den Driessche, Seth Odoom, Mario Lucic, Becca Roelofs, Sid Lall, = Amit Marathe, Betty Chan, Santiago Ontanon, Luheng He, Denis Teplyashin, Jo= nathan Lai, Phil Crone, Bogdan Damoc, Lewis Ho, Sebastian Riedel, Karel Len= c, Chih-Kuan Yeh, Aakanksha Chowdhery, Yang Xu, Mehran Kazemi, Ehsan Amid, = Anastasia Petrushkina, Kevin Swersky, Ali Khodaei, Gowoon Chen, Chris Larki= n, Mario Pinto, Geng Yan, Adria\\\\u00a0Puigdomenech Badia, Piyush Patil, S= teven Hansen, Dave Orr, Sebastien M.\\\\u00a0R. Arnold, Jordan Grimstad, An= drew Dai, Sholto Douglas, Rishika Sinha, Vikas Yadav, Xi\\\\u00a0Chen, Elen= a Gribovskaya, Jacob Austin, Jeffrey Zhao, Kaushal Patel, Paul Komarek, Sop= hia Austin, Sebastian Borgeaud, Linda Friso, Abhimanyu Goyal, Ben Caine, Kr= is Cao, Da-Woon Chung, Matthew Lamm, Gabe Barth-Maron, Thais\\\\n Kagohara= , Kate Olszewska, Mia Chen, Kaushik Shivakumar, Rishabh Agarwal, Harshal Go= dhia, Ravi Rajwar, Javier Snaider, Xerxes Dotiwalla, Yuan Liu, Aditya Barua= , Victor Ungureanu, Yuan Zhang, Bat-Orgil Batsaikhan, Mateo Wirth, James Qi= n, Ivo Danihelka, Tulsee Doshi, Martin Chadwick, Jilin Chen, Sanil Jain, Qu= oc Le, Arjun Kar, Madhu Gurumurthy, Cheng Li, Ruoxin Sang, Fangyu Liu, Lamp= ros Lamprou, Rich Munoz, Nathan Lintz, Harsh Mehta, Heidi Howard, Malcolm R= eynolds, Lora Aroyo, Quan Wang, Lorenzo Blanco, Albin Cassirer, Jordan Grif= fith, Dipanjan Das, Stephan Lee, Jakub Sygnowski, Zach Fisher, James Besley= , Richard Powell, Zafarali Ahmed, Dominik Paulus, David Reitter, Zalan Bors= os, Rishabh Joshi, Aedan Pope, Steven Hand, Vittorio Selo, Vihan Jain, Nikh= il Sethi, Megha Goel, Takaki Makino, Rhys May, Zhen Yang, Johan Schalkwyk, = Christina Butterfield, Anja Hauth, Alex Goldin, Will Hawkins, Evan Senter, = Sergey Brin, Oliver Woodman, Marvin Ritter, Eric Noland, Minh Giang, Vijay = Bolina, Lisa Lee, Tim Blyth, Ian\\\\n Mackinnon, Machel Reid, Obaid Sarvan= a, David Silver, Alexander Chen, Lily Wang, Loren Maggiore, Oscar Chang, Ni= thya Attaluri, Gregory Thornton, Chung-Cheng Chiu, Oskar Bunyan, Nir Levine= , Timothy Chung, Evgenii Eltyshev, Xiance Si, Timothy Lillicrap, Demetra Br= ady, Vaibhav Aggarwal, Boxi Wu, Yuanzhong Xu, Ross McIlroy, Kartikeya Badol= a, Paramjit Sandhu, Erica Moreira, Wojciech Stokowiec, Ross Hemsley, Dong L= i, Alex Tudor, Pranav Shyam, Elahe Rahimtoroghi, Salem Haykal, Pablo Sprech= mann, Xiang Zhou, Di ana Mincu, Yujia Li, Ravi Addanki, Kalpesh Krishna, Xia= o Wu, Alexandre Frechette, Matan Eyal, Allan Dafoe, Dave Lacey, Jay Whang, = Thi Avrahami, Ye\\\\u00a0Zhang, Emanuel Taropa, Hanzhao Lin, Daniel Toyama,= Eliza Rutherford, Motoki Sano, HyunJeong Choe, Alex Tomala, Chalence Safra= nek-Shrader, Nora Kassner, Mantas Pajarskas, Matt Harvey, Sean Sechrist, Me= ire Fortunato, Christina Lyu, Gamaleldin Elsayed, Chenkai Kuang, James Lott= es, Eric Chu, Chao Jia, Chih-Wei Chen, Peter Humphreys, Kate Baumli, Connie= Tao, Rajkumar\\\\n Samuel, Cicero\\\\u00a0Nogueira dos Santos, Anders And= reassen, Nemanja Raki\\\\u0107evi\\\\u0107, Dominik Grewe, Aviral Kumar, St= ephanie Winkler, Jonathan Caton, Andrew Brock, Sid Dalmia, Hannah Sheahan, = Iain Barr, Yingjie Miao, Paul Natsev, Jacob Devlin, Feryal Behbahani, Flavi= en Prost, Yanhua Sun, Artiom Myaskovsky, Thanumalayan\\\\u00a0Sankaranaraya= na Pillai, Dan Hurt, Angeliki Lazaridou, Xi\\\\u00a0Xiong, Ce\\\\u00a0Zheng= , Fabio Pardo, Xiaowei Li, Dan Horgan, Joe Stanton, Moran Ambar, Fei Xia, A= lejandro Lince, Mingqiu Wang, Basil Mustafa, Albert Webson, Hyo Lee, Rohan = Anil, Martin Wicke, Timothy Dozat, Abhishek Sinha, Enrique Piqueras, Elahe = Dabir, Shyam Upadhyay, Anudhyan Boral, Lisa\\\\u00a0Anne Hendricks, Corey F= ry, Josip Djolonga, Yi\\\\u00a0Su, Jake Walker, Jane Labanowski, Ronny Huan= g, Vedant Misra, Jeremy Chen, RJ\\\\u00a0Skerry-Ryan, Avi Singh, Shruti Rij= hwani, Dian Yu, Alex Castro-Ros, Beer Changpinyo, Romina Datta, Sumit Bagri= , Arnar\\\\u00a0Mar Hrafnkelsson, Marcello Maggioni, Daniel Zheng, Yury Sul= sky, Shaobo Hou, Tom\\\\u00a0Le Paine, Antoine Yang, Jason Riesa, Dominika\= \\\n Rogozinska, Dror Marcus, Dalia\\\\u00a0El Badawy, Qiao Zhang, Luyu Wa= ng, Helen Miller, Jeremy Greer, Lars\\\\u00a0Lowe Sjos, Azade Nova, Heiga Z= en, Rahma Chaabouni, Mihaela Rosca, Jiepu Jiang, Charlie Chen, Ruibo Liu, T= ara Sainath, Maxim Krikun, Alex Polozov, Jean-Baptiste Lespiau, Josh Newlan= , Zeyncep Cankara, Soo Kwak, Yunhan Xu, Phil Chen, Andy Coenen, Clemens Mey= er, Katerina Tsihlas, Ada Ma, Juraj Gottweis, Jinwei Xing, Chenjie Gu, Jin = Miao, Christian Frank, Zeynep Cankara, Sanjay Ganapathy, Ishita Dasgupta, S= teph Hughes-Fitt, Heng Chen, David Reid, Keran Rong, Hongmin Fan, Joost van= Amersfoort, Vincent Zhuang, Aaron Cohen, Shixiang\\\\u00a0Shane Gu, Anhad = Mohananey, Anastasija Ilic, Taylor Tobin, John Wieting, Anna Bortsova, Phoe= be Thacker, Emma Wang, Emily Caveness, Justin Chiu, Eren Sezener, Alex Kask= asoli, Steven Baker, Katie Millican, Mohamed Elhawaty, Kostas Aisopos, Carl= Lebsack, Nathan Byrd, Hanjun Dai, Wenhao Jia, Matthew Wiethoff, Elnaz Davo= odi, Albert Weston, Lakshman Yagati, Arun Ahuja, Isabel Gao, Golan Pundak,\= \\\n Susan Zhang, Michael Azzam, Khe\\\\u00a0Chai Sim, Sergi Caelles, Jame= s Keeling, Abhanshu Sharma, Andy Swing, YaGuang Li, Chenxi Liu, Carrie\\\\u= 00a0Grimes Bostock, Yamini Bansal, Zachary Nado, Ankesh Anand, Josh Lipschu= ltz, Abhijit Karmarkar, Lev Proleev, Abe Ittycheriah, Soheil\\\\u00a0Hassas= Yeganeh, George Polovets, Aleksandra Faust, Jiao Sun, Alban Rrustemi, Pen = Li, Rakesh Shivanna, Jeremiah Liu, Chris Welty, Federico Lebron, Anirudh Ba= ddepudi, Sebastian Krause, Emilio Parisotto, Radu Soricut, Zheng Xu, Dawn B= loxwich, Melvin Johnson, Behnam Neyshabur, Justin Mao-Jones, Renshen Wang, = Vinay Ramasesh, Zaheer Abbas, Arthur Guez, Constant Segal, Duc\\\\u00a0Dung= Nguyen, James Svensson, Le\\\\u00a0Hou, Sarah York, Kieran Milan, Sophie B= ridgers, Wiktor Gworek, Marco Tagliasacchi, James Lee-Thorp, Michael Chang,= Alexey Guseynov, Ale\\\\u00a0Jakse Hartman, Michael Kwong, Ruizhe Zhao, Sh= eleem Kashem, Elizabeth Cole, Antoine Miech, Richard Tanburn, Mary Phuong, = Filip Pavetic, Sebastien Cevey, Ramona Comanescu, Richard Ives, Sherry Yang= , Cosmo Du, Bo\\\\u00a0Li, Zizhao\\\\n Zhang, Mariko Iinuma, Clara\\\\u00a= 0Huiyi Hu, Aurko Roy, Shaan Bijwadia, Zhenkai Zhu, Danilo Martins, Rachel S= aputro, Anita Ge rgely, Steven Zheng, Dawei Jia, Ioannis Antonoglou, Adam Sa= dovsky, Shane Gu, Yingying Bi, Alek Andreev, Sina Samangooei, Mina Khan, To= mas Kocisky, Angelos Filos, Chintu Kumar, Colton Bishop, Adams Yu, Sarah Ho= dkinson, Sid Mittal, Premal Shah, Alexandre Moufarek, Yong Cheng, Adam Blon= iarz, Jaehoon Lee, Pedram Pejman, Paul Michel, Stephen Spencer, Vladimir Fe= inberg, Xuehan Xiong, Nikolay Savinov, Charlotte Smith, Siamak Shakeri, Dus= tin Tran, Mary Chesus, Bernd Bohnet, George Tucker, Tamara von Glehn, Carri= e Muir, Yiran Mao, Hideto Kazawa, Ambrose Slone, Kedar Soparkar, Disha Shri= vastava, James Cobon-Kerr, Michael Sharman, Jay Pavagadhi, Carlos Araya, Ka= rolis Misiunas, Nimesh Ghelani, Michael Laskin, David Barker, Qiujia Li, An= ton Briukhov, Neil Houlsby, Mia Glaese, Balaji Lakshminarayanan, Nathan Sch= ucher, Yunhao Tang, Eli Collins, Hyeontaek Lim, Fangxiaoyu Feng, Adria Reca= sens, Guangda Lai,\\\\n Alberto Magni, Nicola\\\\u00a0De Cao, Aditya Siddh= ant, Zoe Ashwood, Jordi Orbay, Mostafa Dehghani, Jenny Brennan, Yifan He, K= elvin Xu, Yang Gao, Carl Saroufim, James Molloy, Xinyi Wu, Seb Arnold, Solo= mon Chang, Julian Schrittwieser, Elena Buchatskaya, Soroush Radpour, Martin= Polacek, Skye Giordano, Ankur Bapna, Simon Tokumine, Vincent Hellendoorn, = Thibault Sottiaux, Sarah Cogan, Aliaksei Severyn, Mohammad Saleh, Shantanu = Thakoor, Laurent Shefey, Siyuan Qiao, Meenu Gaba, Shuo yiin Chang, Craig Sw= anson, Biao Zhang, Benjamin Lee, Paul\\\\u00a0Kishan Rubenstein, Gan Song, = Tom Kwiatkowski, Anna Koop, Ajay Kannan, David Kao, Parker Schuh, Axel Stje= rngren, Golnaz Ghiasi, Gena Gibson, Luke Vilnis, Ye\\\\u00a0Yuan, Felipe\\\= \u00a0Tiengo Ferreira, Aishwarya Kamath, Ted Klimenko, Ken Franko, Kefan Xi= ao, Indro Bhattacharya, Miteyan Patel, Rui Wang, Alex Morris, Robin Strudel= , Vivek Sharma, Peter Choy, Sayed\\\\u00a0Hadi Hashemi, Jessica Landon, Mar= a Finkelstein, Priya Jhakra, Justin Frye, Megan Barnes, Matthew Mauger, Den= nis Daun, Khuslen Baatarsukh, Matthew\\\\n Tung, Wael Farhan, Henryk Micha= lewski, Fabio Viola, Felix de\\\\u00a0Chaumont\\\\u00a0Quitry, Charline\\\\= u00a0Le Lan, Tom Hudson, Qingze Wang, Felix Fischer, Ivy Zheng, Elspeth Whi= te, Anca Dragan, Jean baptiste Alayrac, Eric Ni, Alexander Pritzel, Adam Iw= anicki, Michael Isard, Anna Bulanova, Lukas Zilka, Ethan Dyer, Devendra Sac= han, Srivatsan Srinivasan, Hannah Muckenhirn, Honglong Cai, Amol Mandhane, = Mukarram Tariq, Jack\\\\u00a0W. Rae, Gary Wang, Kareem Ayoub, Nicholas Fitz= Gerald, Yao Zhao, Woohyun Han, Chris Alberti, Dan Garrette, Kashyap Krishna= kumar, Mai Gimenez, Anselm Levskaya, Daniel Sohn, Josip Matak, Inaki Iturra= te, Michael\\\\u00a0B. Chang, Jackie Xiang, Yuan Cao, Nishant Ranka, Geoff = Brown, Adrian Hutter, Vahab Mirrokni, Nanxin Chen, Kaisheng Yao, Zoltan Egy= ed, Francois Galilee, Tyler Liechty, Praveen Kallakuri, Evan Palmer, Sanjay= Ghemawat, Jasmine Liu, David Tao, Chloe Thornton, Tim Green, Mimi Jasarevi= c, Sharon Lin, Victor Cotruta, Yi-Xuan Tan, Noah Fiedel, Hongkun Yu, Ed\\\\= u00a0Chi, Alexander Neitz, Jens Heitkaemper, Anu Sinha, Denny\\\\n Zhou, Y= i\\\\u00a0Sun, Charbel Kaed, Brice Hulse, Swaroop Mishra, Maria Georgaki, S= neha Kudugunta, Clement Farabet, Izhak Shafran, Daniel Vlasic, Anton Tsitsu= lin, Rajagopal Ananthanarayanan, Alen Carin, Guolong Su, Pei Sun, Shashank = V, Gabriel Carvajal, Josef Broder, Iulia Comsa, Alena Repina, William Wong,= Warren\\\\u00a0Weilun Chen, Peter Hawkins, Egor Filonov, Lucia Loher, Chri= stoph Hirnschall, Weiyi Wang, Jingchen Ye, Andrea Burns, Hardie Cate, Diana= \\\\u00a0Gage Wright, Federico Piccinini, Lei Zhang, Chu-Cheng Lin, Ionel G= og, Yana Kulizhskaya, Ashwin Sreevatsa, Shuang Song, Luis\\\\u00a0C. Cobo, = Anand Iyer, Chetan Tekur, Guillermo Garrido, Zhuyun Xiao, Rupert Kemp, Huai= xiu\\\\u00a0Steven Zheng, Hui Li, Ananth Agarwal, Christel Ngani, Kati Gosh= vadi, Rebeca Santamaria-Fernandez, Wojciech Fica, Xinyun Chen, Chris Gorgol= ewski, Sean Sun, Roopal Garg, Xinyu Ye, S.\\\\u00a0M.\\\\u00a0Ali Eslami, N= an Hua, Jon Simon, Pratik Joshi, Yelin Kim, Ian Tenney, Sahitya Potluri, La= m\\\\u00a0Nguyen Thiet, Quan Yuan, Florian Luisier, Alexandra Chronopoulou,= Salvatore Scellato, Praveen\\\\n Srinivasan, Minmin Chen, Vinod Koverkath= u, Valentin Dalibard, Yaming Xu, Brennan Saeta, Keith Anderson, Thibault Se= llam, Nick Fernando, Fantine Huot, Junehyuk Jung, Mani Varadarajan, Michael= Quinn, Amit Raul, Maigo Le, Ruslan Habalov, Jon Clark, Komal Jalan, Kalesh= a Bullard, Achintya Singhal, Thang Luong, Boyu Wang, Sujeevan Rajayogam, Ju= lian Eisenschlos, Johnson Jia, Daniel Finchelstein, Alex Yakubovich, Daniel= Balle, Michael Fink, Sameer Agarwal, Jing Li, Dj\\\\u00a0Dvijotham, Shalin= i Pal, Kai Kang, Jaclyn Konzelmann, Jennifer Beattie, Olivier Dousse, Diane= Wu, Remi Crocker, Chen Elkind, Siddhartha\\\\u00a0Reddy Jonnalagadda, Jong= Lee, Dan Holtmann-Rice, Krystal Kallarackal, Rosanne Liu, Denis Vnukov, Ne= era Vats, Luca Invernizzi, Mohsen Jafari, Huanjie Zhou, Lilly Taylor, Jenni= fer Prendki, Marcus Wu, Tom Eccles, Tianqi Liu, Kavya Kopparapu, Francoise = Beaufays, Christof Angermueller, Andreea Marzoca, Shourya Sarcar, Hilal Dib= , Jeff Stanway, Frank Perbet, Nejc Trdin, Rachel Sterneck, Andrey Khorlin, = Dinghua Li, Xihui Wu,\\\\n Sonam Goenka, David Madras, Sasha Goldshtein, W= illi Gierke, Tong Zhou, Yaxin Liu, Yannie Liang, Anais White, Yunjie Li, Sh= reya Singh, Sanaz Bahargam, Mark Epstein, Sujoy Basu, Li\\\\u00a0Lao, Adnan= Ozturel, Carl Crous, Alex Zhai, Han Lu, Zora Tung, Neeraj Gaur, Alanna Wal= ton, Lucas Dixon, Ming Zhang, Amir Globerson, Grant Uy, Andrew Bolt, Olivia= Wiles, Milad Nasr, Ilia Shumailov, Marco Selvi, Francesco Piccinno, Ricard= o Aguilar, Sara McCarthy, Misha Khalman, Mrinal Shukla, Vlado Galic, John C= arpenter, Kevin Villela, Haibin Zhang, Harry Richardson, James Martens, Mat= ko Bosnjak, Shreyas\\\\u00a0Rammohan Belle, Jeff Seibert, Mahmoud Alnahlawi= , Brian McWilliams, Sankalp Singh, Annie Louis, Wen Ding, Dan Popovici, Len= in Simicich, Laura Knight, Pulkit Mehta, Nishesh Gupta, Chongyang Shi, Saab= er Fatehi, Jovana Mitrovic, Alex Grills, Joseph Pagadora, Dessie Petrova, D= anielle Eisenbud, Zhishuai Zhang, Damion Yates, Bhavishya Mittal, Nilesh Tr= ipuraneni, Yannis Assael, Thomas Brovelli, Prateek Jain, Mihajlo Velimirovi= c, Canfer\\\\n Akbulut, Jiaqi Mu, Wolfgang Macherey, Ravin Kumar, Jun Xu, = Haroon Qureshi, Gheorghe Comanici, Jeremy Wiesner, Zhitao Gong, Anton Ruddo= ck, Matthias Bauer, Nick Felt, Anirudh GP, Anurag Arnab, Dustin Zelle, Jona= s Rothfuss, Bill Rosgen, Ashish Shenoy, Bryan Seybold, Xinjian Li, Jayaram = Mudigonda, Goker Erdogan, Jiawei Xia, Jiri Simsa, Andrea Michi, Yi\\\\u00a0= Yao, Christopher Yew, Steven Kan, Isaac Caswell, Carey Radebaugh, Andre Eli= sseeff, Pedro Valenzuela, Kay McKinney, Kim Paterson, Albert Cui, Eri Lator= re-Chimoto, Solomon Kim, William Zeng, Ken Durden, Priya Ponnapalli, Tiberi= u Sosea, Christopher\\\\u00a0A. Choquette-Choo, James Manyika, Brona Robene= k, Harsha Vashisht, Sebastien Pereira, Hoi Lam, Marko Velic, Denese Owusu-A= friyie, Katherine Lee, Tolga Bolukbasi, Alicia Parrish, Shawn Lu, Jane Park= , Balaji Venkatraman, Alice Talbert, Lambert Rosique, Yuchung Cheng, Andrei= Sozanschi, Adam Paszke, Praveen Kumar, Jessica Austin, Lu\\\\u00a0Li, Khal= id Salama, Wooyeol Kim, Nandita Dukkipati, Anthony Baryshnikov, Christos Ka= planis,\\\\n XiangHai Sheng, Yuri Chervonyi, Caglar Unlu, Diego de\\\\u00a= 0Las\\\\u00a0Casas, Harry Askham, Kathryn Tunyasuvunakool, Felix Gimeno, Si= im Poder, Chester Kwak, Matt Miecnikowski, Vahab Mirrokni, Alek Dimitriev, = Aaron Parisi, Dangyi Liu, Tomy Tsai, Toby Shevlane, Christina Kouridi, Drew= Garmon, Adrian Goedeckemeyer, Adam\\\\u00a0R. Brown, Anitha Vijayakumar, A= li Elqursh, Sadegh Jazayeri, Jin Huang, Sara\\\\u00a0Mc Carthy, Jay Hoover,= Lucy Kim, Sandeep Kumar, Wei Chen, Courtney Biles, Garrett Bingham, Evan R= osen, Li sa Wang, Qijun Tan, David Engel, Francesco Pongetti, Dario de\\\\u0= 0a0Cesare, Dongseong Hwang, Lily Yu, Jennifer Pullman, Srini Narayanan, Kyl= e Levin, Siddharth Gopal, Megan Li, Asaf Aharoni, Trieu Trinh, Jessica Lo, = Norman Casagrande, Roopali Vij, Loic Matthey, Bramandia Ramadhana, Austin M= atthews, CJ\\\\u00a0Carey, Matthew Johnson, Kremena Goranova, Rohin Shah, S= hereen Ashraf, Kingshuk Dasgupta, Rasmus Larsen, Yicheng Wang, Manish\\\\u0= 0a0Reddy Vuyyuru, Chong Jiang, Joana Ijazi, Kazuki Osawa, Celine Smith, Ram= ya\\\\u00a0Sree Boppana, Taylan Bilal, Yuma\\\\n Koizumi, Ying Xu, Yasemin= Altun, Nir Shabat, Ben Bariach, Alex Korchemniy, Kiam Choo, Olaf Ronneberg= er, Chimezie Iwuanyanwu, Shubin Zhao, David Soergel, Cho-Jui Hsieh, Irene C= ai, Shariq Iqbal, Martin Sundermeyer, Zhe Chen, Elie Bursztein, Chaitanya M= alaviya, Fadi Biadsy, Prakash Shroff, Inderjit Dhillon, Tejasi Latkar, Chri= s Dyer, Hannah Forbes, Massimo Nicosia, Vitaly Nikolaev, Somer Greene, Mari= n Georgiev, Pidong Wang, Nina Martin, Hanie Sedghi, John Zhang, Praseem Ban= zal, Doug Fritz, Vikram Rao, Xuezhi Wang, Jiageng Zhang, Viorica Patraucean= , Dayou Du, Igor Mordatch, Ivan Jurin, Lewis Liu, Ayush Dubey, Abhi Mohan, = Janek Nowakowski, Vlad-Doru Ion, Nan Wei, Reiko Tojo, Maria\\\\u00a0Abi Raa= d, Drew\\\\u00a0A. Hudson, Vaishakh Keshava, Shubham Agrawal, Kevin Ramirez= , Zhichun Wu, Hoang Nguyen, Ji\\\\u00a0Liu, Madhavi Sewak, Bryce Petrini, D= ongHyun Choi, Ivan Philips, Ziyue Wang, Ioana Bica, Ankush Garg, Jarek Wilk= iewicz, Priyanka Agrawal, Xiaowei Li, Danhao Guo, Emily Xue, Naseer Shaik, = Andrew Leach, Sadh\\\\u00a0MNM Khan, Julia Wiesinger,\\\\n Sammy Jerome, A= bhishek Chakladar, Alek\\\\u00a0Wenjiao Wang, Tina Ornduff, Folake Abu, Ali= reza Ghaffarkhah, Marcus Wainwright, Mario Cortes, Frederick Liu, Joshua Ma= ynez, Andreas Terzis, Pouya Samangouei, Riham Mansour, Tomasz K\\\\u0119pa,= Fran\\\\u00e7ois-Xavier Aubet, Anton Algymr, Dan Banica, Agoston Weisz, An= dras Orban, Alexandre Senges, Ewa Andrejczuk, Mark Geller, Niccolo\\\\u00a0= Dal Santo, Valentin Anklin, Majd\\\\u00a0Al Merey, Martin Baeuml, Trevor St= rohman, Junwen Bai, Slav Petrov, Yonghui Wu, Demis Hassabis, Koray Kavukcuo= glu, Jeffrey Dean, and Oriol Vinyals. 2024.\\\\n [Gemini 1.5: Unlocking mu= ltimodal understanding across millions of tokens of context](https://arxiv.= org/abs/2403.05530).\\\\n *Preprint*, arXiv:2403.05530.\\\\n* Gou et\\\\u0= 0a0al. (2023)\\\\n\\\\n Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen,= Yujiu Yang, Nan Duan, and Weizhu Chen. 2023.\\\\n Critic: Large language = models can self-correct with tool-interactive critiquing.\\\\n *arXiv prep= rint arXiv:2305.11738*.\\\\n* Hendrycks et\\\\u00a0al. (2020)\\\\n\\\\n Da= n Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn So= ng, and Jacob Steinhardt. 2020.\\\\n Measuring massive multitask language = understanding.\\\\n *arXiv preprint arXiv:2009.03300*.\\\\n* Huang et\\\\u= 00a0al. (2023)\\\\n\\\\n Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu\\= \\u00a0Steven Zheng, Adams\\\\u00a0Wei Yu, Xinying Song, and Denny Zhou. 20= 23.\\\\n Large language models cannot self-correct reasoning yet.\\\\n *a= rXiv preprint arXiv:2310.01798*.\\\\n* Jan et\\\\u00a0al. (2025)\\\\n\\\\n = Essa Jan, Nouar Aldahoul, Moiz Ali, Faizan Ahmad, Fareed Zaffar, and Yasir= Zaki. 2025.\\\\n Multitask-bench: Unveiling and mitigating safety gaps in= llms fine-tuning.\\\\n In *Proceedings of the 31st International Conferen= ce on Computational Linguistics*, pages 9025\\\\u20139043.\\\\n* Jin et\\\\= u00a0al. (2021)\\\\n\\\\n Di\\\\u00a0Jin, Eileen Pan, Nassim Oufattole, We= i-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021.\\\\n What disease does= this patient have? a large-scale open domain question answering dataset fr= om medical exams.\\\\n *Applied Sciences*, 11(14):6421.\\\\n* Li et\\\\u00= a0al. (2024)\\\\n\\\\n Yanhong Li, Chenghao Yang, and Allyson Ettinger. 20= 24.\ \\\n When hindsight is not 20/20: Testing limits on reflective thinkin= g in large language models.\\\\n *arXiv preprint arXiv:2404.09129*.\\\\n* = Llama\\\\u00a0Team (2024)\\\\n\\\\n AI\\\\u00a0@\\\\u00a0Meta Llama\\\\u00= a0Team. 2024.\\\\n [The llama 3 herd of models](https://arxiv.org/abs/2407= .21783).\\\\n *Preprint*, arXiv:2407.21783.\\\\n* Madaan et\\\\u00a0al. (2= 024)\\\\n\\\\n Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, = Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yimin= g Yang, et\\\\u00a0al. 2024.\\\\n Self-refine: Iterative refinement with s= elf-feedback.\\\\n *Advances in Neural Information Processing Systems*, 36= .\\\\n* Muennighoff et\\\\u00a0al. (2025)\\\\n\\\\n Niklas Muennighoff, Zi= tong Yang, Weijia Shi, Xiang\\\\u00a0Lisa Li, Li\\\\u00a0Fei-Fei, Hannaneh = Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Cand\\\\u00e8s, and Tat= sunori Hashimoto. 2025.\\\\n s1: Simple test-time scaling.\\\\n *arXiv pr= eprint arXiv:2501.19393*.\\\\n* OpenAI et\\\\u00a0al. (2024)\\\\n\\\\n Ope= nAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, = Florencia\\\\u00a0Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altm= an, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Bal= com, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bell= o, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdo= noff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Br= ooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann= , Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang,= Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, = Ben Chess, Chester Cho, Casey Chu, Hyung\\\\u00a0Won Chung, Dave Cummings, = Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, = Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adri= en Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix,= Sim\\\\u00f3n\\\\u00a0Posada Fishman, Juston Forte, Isabella Fulford, Leo\= \\\n Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabrie= l Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, = Ryan Greene, Joshua Gross, Shixiang\\\\u00a0Shane Gu, Yufei Guo, Chris Hall= acy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chr= is Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenn= y Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joann= e Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Bi= llie Jonn, Heewoo Jun, Tomer Kaftan, \\\\u0141ukasz Kaiser, Ali Kamali, Ing= mar Kanitscheider, Nitish\\\\u00a0Shirish Keskar, Tabarak Khan, Logan Kilpa= trick, Jong\\\\u00a0Wook Kim, Christina Kim, Yongjik Kim, Jan\\\\u00a0Hendr= ik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, \\\\u0141ukasz Kon= draciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger= , Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Da= niel Levy, Chak\\\\u00a0Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mate= usz Litwin, Theresa Lopez, Ryan\\\\n Lowe, Patricia Lue, Anna Makanju, Kim= Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Kati= e Mayer, Andrew Mayne, Bob McGrew, Scott\\\\u00a0Mayer McKinney, Christine = McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Meni= ck, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morik= awa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David M\\\\u00e9ly, A= shvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo= , Hyeonwoo Noh, Long Ouyang, Cullen O\\\\u2019Keefe, Jakub Pachocki, Alex P= aino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Paris= h, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, = Filipe de\\\\u00a0Avila Belbute\\\\u00a0Peres, Michael Petrov, Henrique\\\\= u00a0Ponde de\\\\u00a0Oliveira\\\\u00a0Pinto, Michael, Pokorny, Michelle Po= krass, Vitchyr\\\\u00a0H. Pong, Tolly Powell, Alethea Power, Boris Power, E= lizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron = Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Rousse= z,\\\\n Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Giri= sh Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Ky= la Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szy= mon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian S= ohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe\\\\u00a0Pet= roski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madel= eine\\\\u00a0B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, = Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe\\\\u00a0Cer\\\\u00f3= n Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwrigh= t, Justin\\\\u00a0Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei,= CJ\\\\u00a0Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian W= eng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wo= ng, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sara= h Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang,= Marvin Zhang, Shengjia\\\\n Zhao, Tianhao Zheng, Juntang Zhuang, William = Zhuk, and Barret Zoph. 2024.\\\\n [Gpt-4 technical report](https://arxiv.o= rg/abs/2303.08774).\\\\n *Preprint*, arXiv:2303.08774.\\\\n* Pan et\\\\u00= a0al. (2024)\\\\n\\\\n Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nath= ani, Xinyi Wang, and William\\\\u00a0Yang Wang. 2024.\\\\n Automatically c= orrecting large language models: Surveying the landscape of diverse automat= ed correction strategies.\\\\n *Transactions of the Association for Comput= ational Linguistics*, 12:484\\\\u2013506.\\\\n* Paul et\\\\u00a0al. (2023)\= \\\n\\\\n Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, A= ntoine Bosselut, Robert West, and Boi Faltings. 2023.\\\\n Refiner: Reason= ing feedback on intermediate representations.\\\\n *arXiv preprint arXiv:2= 304.01904*.\\\\n* Renze and Guven (2024)\\\\n\\\\n Matthew Renze and Erhan= Guven. 2024.\\\\n Self-reflection in llm agents: Effects on problem-solvi= ng performance.\\\\n *arXiv preprint arXiv:2405.06682*.\\\\n* Stechly et\\= \\u00a0al. (2023)\\\\n\\\\n Kaya Stechly, Matthew Marquez, and Subbarao Ka= mbhampati. 2023.\\\\n Gpt-4 doesn\\\\u2019t know it\\\\u2019s wrong: An an= alysis of iterative prompting for reasoning problems.\\\\n *arXiv preprint= arXiv:2310.12397*.\\\\n* Valmeekam et\\\\u00a0al. (2023)\\\\n\\\\n Karthi= k Valmeekam, Matthew Marquez, and Subbarao Kambhampati. 2023.\\\\n Can lar= ge language models really improve by self-critiquing their own plans?\\\\n = *arXiv preprint arXiv:2310.08118*.\\\\n* Verma et\\\\u00a0al. (2024)\\\\n\= \\\n Mudit Verma, Siddhant Bhambri, and Subbarao Kambhampati. 2024.\\\\n = On the brittle foundations of react prompting for agentic large language mo= dels.\\\\n *arXiv preprint arXiv:2405.13966*.\\\\n* Wei et\\\\u00a0al. (20= 22)\\\\n\\\\n Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei = Xia, Ed\\\\u00a0Chi, Quoc\\\\u00a0V Le, Denny Zhou, et\\\\u00a0al. 2022.\\\= \n Chain-of-thought prompting elicits reasoning in large language models.\= \\\n *Advances in neural information processing systems*, 35:24824\\\\u201= 324837.\\\\n* Zhang et\\\\u00a0al. (2024)\\\\n\\\\n Wenqi Zhang, Yongliang= Shen, Linjuan Wu, Qiuying Peng, Jun Wang, Yueting Zhuang, and Weiming Lu. = 2024.\\\\n Self-contrast: Better reflection through inconsistent solving p= erspectives.\\\\n *arXiv preprint arXiv:2401.02009*.\\\\n* Zheng et\\\\u00= a0al. (2023)\\\\n\\\\n Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo L= i, and Yu\\\\u00a0Li. 2023.\\\\n Progressive-hint prompting improves reaso= ning in large language models.\\\\n *arXiv preprint arXiv:2304.09797*.\\\\= n\\\\n## Appendix A Appendix\\\\n\\\\n| | Simple Original Prompt | | | CoT= Original Prompt | | |\\\\n| --- | --- | --- | --- | --- | --- | --- |\\\\n= | Prompt | Gemini | GPT | Llama | Gemini | GPT | Llama |\\\\n| | 55.6% | 6= 2.1% | 36.1% | 60.1% | 65.5% | 39.1% |\\\\n| Original answer | (0.275) | (0= .432) | (0.806) | (0.198) | (0.212) | (0.489) |\\\\n| | 51.2% | 55.1% | 30= .9% | 58.4% | 65.7% | 34.0% |\\\\n| Prompt 1 | (0.276) | (0.515) | (1.064) = | (6.328) | (0.091) | (0.915) |\\\\n| | 45.8% | 52.5% | 29.8% | 58.0% | 65= .5% | 33.5% |\\\\n| Prompt 2 | (3.455) | (0.921) | (1.259) | (6.803) | (0.0= 79) | (1.896) |\\\\n| | 49.4% | 59.2% | 29.2% | 58.3% | 65.6% | 33.8% |\\\= \n| Prompt 3 | (1.133) | (0.091) | (1.100) | (6.509) | (0.236) | (0.371) |\= \\\n| | 52.1% | 59.8% | 30.7% | 59.7% | 65.0% | 34.8% |\\\\n| Prompt 4 | (= 3.554) | (0.569) | (0.971) | (3.717) | (0.518) | (1.289) |\\\\n| | 43.4% |= 63.7% | 33.3% | 57.8% | 65.1% | 37.1% |\\\\n| Prompt 5 | (9.296) | (0.863)= | (1.259) | (3.179) | (0.820) | (0.811) |\\\\n\\\\nTable 2: Self-reflectio= n experiments using MEDQA-USMLE. The temperature value is set to 1 for text= generation. To obtain the original answer, we prompted each LLM in two dif= ferent ways: The first uses a simple prompt without chain-of-thought (CoT) = prompting, while the second uses a more complex prompt with CoT prompting. = For each original answer, we experimented with five different variations of= self-reflection prompts. We repeat all experiments three times and report = the average accuracy and its standard deviations (in brackets) across all r= epetitions. See the Appendix Figure\\\\u00a0[6](https://arxiv.org/html/2406= .10400v2#A1.F6 \\\\"Figure 6 \\\\u2023 Appendix A Appendix \\\\u2023 Self-R= eflection Makes Large Language Models Safer, Less Biased, and Ideologically= Neutral\\\\") for the exact prompts used in experiments.\\\\n\\\\n| | Sim= ple Original Prompt | | | CoT Original Prompt | | |\\\\n| --- | --- | --- |= --- | --- | --- | --- |\\\\n| Prompt | Gemini | GPT | Llama | Gemini | GPT= | Llama |\\\\n| | 37.3% | 92.6% | 59.8% | 93.1% | 92.0% | 62.8% |\\\\n| O= riginal answer | (0.493) | (0.463) | (2.487) | (0.227) | (0.461) | (1.082) = |\\\\n| | 24.1% | 92.3% | 44.4% | 63.6% | 90.9% | 52.6% |\\\\n| Prompt 1 |= (1.097) | (0.457) | (1.756) | (0.888) | (0.548) | (0.570) |\\\\n| | 27.9%= | 92.3% | 38.3% | 26.6% | 91.4% | 47.4% |\\\\n| Prompt 2 | (0.662) | (0.54= 7) | (1.919) | (0.631) | (0.374) | (0.289) |\\\\n| | 7.1% | 90.5% | 39.5% = | 18.0% | 88.7% | 48.3% |\\\\n| Prompt 3 | (0.463) | (0.418) | (0.526) | (0= .919) | (0.444) | (2.220) |\\\\n| | 82.5% | 60.5% | 46.3% | 92.7% | 50.5% = | 47.5% |\\\\n| Prompt 4 | (0.717) | (2.622) | (0.871) | (0.116) | (0.497) = | (0.809) |\\\\n| | 88.1% | 92.0% | 49.5% | 92.5% | 91.6% | 54.3% |\\\\n| = Prompt 5 | (0.862) | (0.347) | (2.437) | (0.579) | (0.244) | (0.720) |\\\\n= | | 42.9% | 87.2% | 50.5% | 44.8% | 83.1% | 55.6% |\\\\n| Prompt 6 | (1.40= 3) | (0.613) | (1.579) | (1.253) | (1.032) | (0.616) |\\\\n| | 21.2% | 83.= 3% | 46.6% | 22.4% | 78.5% | 52.5% |\\\\n| Prompt 7 | (0.688) | (0.648) | (= 2.164) | (0.131) | (0.152) | (0.674) |\\\\n| | 17.4% | 90.1% | 44.4% | 10.= 0% | 87.2% | 49.5% |\\\\n| Prompt 8 | (1.051) | (0.244) | (2.752) | (0.244)= | (0.759) | (1.134) |\\\\n| | 1.7% | 85.8% | 15.8% | 2.5% | 78.9% | 17.3%= |\\\\n| Prompt 9 | (0.358) | (0.374) | (1.022) | (0.088) | (0.306) | (0.93= 1) |\\\\n\\\\nTable 3: Self-reflection experiments using GSM8K. The experim= ents are conducted in the same way as in Table\\\\u00a0[2](https://arxiv.or= g/html/2406.10400v2#A1.T2 \\\\"Table 2 \\\\u2023 Appendix A Appendix \\\\u2= 023 Self-Reflection Makes Large Language Models Safer, Less Biased, and Ide= ologically Neutral\\\\") but with four additional prompts. See Append ix Fig= ure\\\\u00a0[7](https://arxiv.org/html/2406.10400v2#A1.F7 \\\\"Figure 7 \\\= \u2023 Appendix A Appendix \\\\u2023 Self-Reflection Makes Large Language M= odels Safer, Less Biased, and Ideologically Neutral\\\\") for the additiona= l prompts.\\\\n\\\\n| | Simple Original Prompt | | | CoT Original Prompt |= | | | |\\\\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\\\\n= | Prompt | Gemini | GPT | Llama | Gemini | GPT | Llama | | |\\\\n| | 77.= 4% | 77.0% | 40.0% | 79.8% | 79.9% | 46.5% | | |\\\\n| Original answer | = (0.173) | (0.778) | (0.413) | (1.424) | (0.960) | (1.488) | | |\\\\n| | = 74.1% | 72.7% | 35.1% | 81.0% | 80.1% | 40.6% | | |\\\\n| Prompt 1 | (0.3= 00) | (0.208) | (1.092) | (1.179) | (1.185) | (1.283) | | |\\\\n| | 69.2= % | 71.4% | 32.1% | 80.5% | 79.9% | 37.8% | | |\\\\n| Prompt 2 | (0.569) = | (0.208) | (1.104) | (1.443) | (0.723) | (1.165) | | |\\\\n| | 73.1% | = 73.5% | 34.7% | 80.8% | 80.0% | 39.1% | | |\\\\n| Prompt 3 | (0.100) | (0= .819) | (2.052) | (1.474) | (0.586) | (1.471) | | |\\\\n| | 69.0% | 74.9= % | 35.2% | 80.9% | 78.4% | 39.7% | | |\\\\n| Prompt 4 | (1.120) | (0.435= ) | (0.798) | (1.242) | (0.835) | (0.836) | | |\\\\n| | 59.7% | 77.3% | = 37.7% | 78.7% | 79.1% | 43.1% | | |\\\\n| Prompt 5 | (1.207) | (0.375) | = (0.336) | (0.200) | (0.408) | (1.913) | | |\\\\n| | 75.4% | 71.6% | 36.0= % | 81.0% | 76.3% | 40.6% | | |\\\\n| Prompt 6 | (0.416) | (0.923) | (1.0= 87) | (1.274) | (0.062) | (1.251) | | |\\\\n| | 73.9% | 69.0% | 33.2% | = 80.6% | 72.2% | 36.9% | | |\\\\n| Prompt 7 | (0.361) | (1.134) | (1.203) = | (0.603) | (1.047) | (1.077) | | |\\\\n| | 73.9% | 72.9% | 35.1% | 80.7= % | 79.5% | 40.3% | | |\\\\n| Prompt 8 | (0.819) | (0.907) | (0.465) | (0= .929) | (0.197) | (1.352) | | |\\\\n| | 62.5% | 71.2% | 30.4% | 76.9% | = 78.3% | 33.5% | | |\\\\n| Prompt 9 | (0.416) | (0.777) | (0.873) | (1.136= ) | (0.815) | (0.767) | | |\\\\n\\\\nTable 4: Self-reflection experiments= using MMLU. The experiments are conducted in the same way as in Table\\\\u= 00a0[3](https://arxiv.org/html/2406.10400v2#A1.T3 \\\\"Table 3 \\\\u2023 Ap= pendix A Appendix \\\\u2023 Self-Reflection Makes Large Language Models Saf= er, Less Biased, and Ideologically Neutral\\\\").\\\\n\\\\n| | True Positi= ve Rate | | | True Negative Rate | | | Accuracy | | |\\\\n| --- | --- | ---= | --- | --- | --- | --- | --- | --- | --- |\\\\n| Prompt | Gemini | GPT | = Llama | Gemini | GPT | Llama | Gemini | GPT | Llama |\\\\n| | 0% | 0% | 0%= | 100% | 100% | 100% | | | |\\\\n| Original answer | (0) | (0) | (0) | = (0) | (0) | (0) | 52% | 52% | 52% |\\\\n| | 15.1% | 26.3% | 25.5% | 100.0%= | 100.0% | 44.7% | | | |\\\\n| Prompt 1 | (0.106) | (0.367) | (0.847) |= (0.000) | (0.000) | (1.283) | 59.3% | 64.6% | 35.5% |\\\\n| | 14.0% | 28.= 9% | 37.5% | 100.0% | 100.0% | 49.7% | | | |\\\\n| Prompt 2 | (0.382) | = (0.290) | (1.059) | (0.000) | (0.000) | (0.933) | 58.7% | 65.9% | 43.8% |\\= \\n| | 17.5% | 29.3% | 34.7% | 100.0% | 99.9% | 35.7% | | | |\\\\n| Pro= mpt 3 | (0.106) | (0.280) | (1.249) | (0.000) | (0.098) | (1.447) | 60.4% |= 66.0% | 35.2% |\\\\n| | 14.6% | 72.8% | 93.9% | 100.0% | 96.3% | 14.6% | = | | |\\\\n| Prompt 4 | (0.485) | (0.841) | (0.382) | (0.000) | (0.489) |= (1.359) | 59.0% | 85.0% | 52.6% |\\\\n| | 13.7% | 62.5% | 89.4% | 100.0% = | 97.6% | 18.5% | | | |\\\\n| Prompt 5 | (0.183) | (1.744) | (0.212) | (= 0.000) | (0.782) | (1.323) | 58.6% | 80.8% | 52.5% |\\\\n| | 13.2% | 75.8%= | 88.9% | 100.0% | 97.8% | 24.9% | | | |\\\\n| Prompt 6 | (0.382) | (0.= 561) | (0.695) | (0.000) | (0.259) | (0.678) | 58.3% | 87.2% | 55.6% |\\\\n= | | 36.3% | 87.0% | 16.9% | 100.0% | 68.6% | 74.8% | | | |\\\\n| Prompt= 7 | (0.169) | (0.800) | (1.121) | (0.000) | (1.537) | (1.858) | 69.4% | 77= .4% | 47.0% |\\\\n\\\\nTable 5: Self-reflection experiments using the safet= y dataset. The temperature value is set to 1 for text generation. We experi= mented with seven different variations of self-reflection prompts. We repea= t all experiments three times and report the average accuracy and its stand= ard deviations (in brackets) across all repetitions. See the appendix for t= he exact prompts used in experiments.\\\\n\\\\n| | True Positive Rate | | = | True Negative Rate | | | Accuracy | | |\\\\n| --- | --- | --- | --- | ---= | --- | --- | --- | --- | --- |\\\\n| Prompt | Gemini | GPT | Llama | Gemi= ni | GPT | Llama | Gemini | GPT | Llama |\\\\n| | 0% | 0% | 0% | 100% | 10= 0% | 100% | | | |\\\\n| Original answer | (0) | (0) | (0) | (0) | (0) | = (0) | 52% | 52% | 52% |\\\\n| | 32.4% | 41.1% | 22.6% | 100.0% | 100.0% | = 82.9% | | | |\\\\n| Prompt 1 | (0.280) | (0.318) | (0.462) | (0.000) | (= 0.000) | (1.089) | 67.6% | 71.7% | 54.0% |\\\\n| | 31.7% | 42.6% | 30.6% |= 100.0% | 100.0% | 45.0% | | | |\\\\n| Prompt 2 | (0.183) | (0.280) | (0= .662) | (0.000) | (0.000) | (1.793) | 67.2% | 72.5% | 38.1% |\\\\n| | 38.3= % | 43.1% | 31.8% | 100.0% | 100.0% | 57.2% | | | |\\\\n| Prompt 3 | (0.= 280) | (0.106) | (0.328) | (0.000) | (0.000) | (0.801) | 70.4% | 72.7% | 45= .0% |\\\\n| | 31.8% | 59.0% | 70.0% | 100.0% | 99.6% | 68.6% | | | |\\\= \n| Prompt 4 | (0.106) | (1.734) | (0.280) | (0.000) | (0.293) | (0.705) | = 67.2% | 80.1% | 69.3% |\\\\n| | 35.1% | 58.4% | 76.1% | 100.0% | 99.9% | 3= 3.6% | | | |\\\\n| Prompt 5 | (0.183) | (0.183) | (1.539) | (0.000) | (0= .098) | (1.477) | 68.8% | 80.0% | 54.0% |\\\\n| | 36.9% | 54.2% | 27.0% | = 100.0% | 100.0% | 96.4% | | | |\\\\n| Prompt 6 | (0.382) | (0.530) | (1.= 121) | (0.000) | (0.000) | (0.685) | 69.7% | 78.0% | 63.1% |\\\\n| | 44.5%= | 75.3% | 55.2% | 100.0% | 82.9% | 12.2% | | | |\\\\n| Prompt 7 | (0.18= 3) | (0.485) | (1.043) | (0.000) | (3.522) | (0.402) | 73.4% | 79.3% | 32.8= % |\\\\n\\\\nTable 6: No-reflection experiments using the safety dataset. \= \\\u201cNo-reflection\\\\u201d means that LLMs are asked to pay attention t= o safety issues while translating, instead of being asked to self-reflect a= fter translation, using the same prompts used in self-reflection experiment= s as in reported in Table\\\\u00a0[5](https://arxiv.org/html/2406.10400v2#A= 1.T5 \\\\"Table 5 \\\\u2023 Appendix A Appendix \\\\u2023 Self-Reflection M= akes Large Language Models Safer, Less Biased, and Ideologically Neutral\\\= \"). The temperature value is set to 1 for text generation. We experimented= with seven different variations of no-reflection prompts. We repeat all ex= periments three times and report the average accuracy and its standard devi= ations (in brackets) across all repetitions. See the appendix for the exact= prompts used in experiments.\\\\n\\\\n| | True Positive Rate | | | True N= egative Rate | | | Accuracy | | |\\\\n| --- | --- | --- | --- | --- | --- |= --- | --- | --- | --- |\\\\n| Prompt | Gemini | GPT | Llama | Gemini | GPT= | Llama | Gemini | GPT | Llama |\\\\n| | 0% | 0% | 0% | 100% | 100% | 100= % | | | |\\\\n| Original answer | (0) | (0) | (0) | (0) | (0) | (0) | 50= % | 50% | 50% |\\\\n| | 24.7% | 77.0% | 34.2% | 99.9% | 94.3% | 50.5% | |= | |\\\\n| Prompt 1 | (0.249) | (0.492) | (0.340) | (0.082) | (0.498) | (= 1.173) | 62.3% | 85.6% | 42.3% |\\\\n| | 44.2% | 46.6% | 38.3% | 99.4% | 9= 8.9% | 48.8% | | | |\\\\n| Prompt 2 | (0.125) | (0.624) | (1.682) | (0.2= 45) | (0.125) | (1.563) | 71.8% | 72.7% | 43.6% |\\\\n| | 34.0% | 50.4% | = 34.9% | 99.9% | 98.5% | 54.0% | | | |\\\\n| Prompt 3 | (0.638) | (0.883)= | (0.287) | (0.047) | (0.094) | (1.651) | 67.0% | 74.4% | 44.5% |\\\\n| |= 24.7% | 9.6% | 17.8% | 99.6% | 99.9% | 88.9% | | | |\\\\n| Prompt 4 | (= 0.464) | (0.205) | (0.634) | (0.082) | (0.047) | (1.021) | 62.1% | 54.7% | = 53.3% |\\\\n\\\\nTable 7: Self-reflection experiments using the gender bias= dataset. The temperature value is set to 1 for text generation. We experim= ented with four different variations of self-reflection promp ts. We repeat = all experiments three times and report the average accuracy and its standar= d deviations (in brackets) across all repetitions. See the appendix for the= exact prompts used in experiments.\\\\n\\\\n| | True Positive Rate | | | = True Negative Rate | | | Accuracy | | |\\\\n| --- | --- | --- | --- | --- |= --- | --- | --- | --- | --- |\\\\n| Prompt | Gemini | GPT | Llama | Gemini= | GPT | Llama | Gemini | GPT | Llama |\\\\n| | 0% | 0% | 0% | 100% | 100%= | 100% | | | |\\\\n| Original answer | (0) | (0) | (0) | (0) | (0) | (0= ) | 50% | 50% | 50% |\\\\n| | 5.5% | 100.0% | 92.7% | 100.0% | 78.7% | 2.0= % | | | |\\\\n| Prompt 1 | (1.286) | (0.000) | (1.286) | (0.000) | (1.24= 7) | (1.414) | 52.7% | 89.3% | 47.3% |\\\\n| | 0.0% | 1.0% | 6.3% | 100.0%= | 99.7% | 96.0% | | | |\\\\n| Prompt 2 | (0.000) | (0.842) | (0.471) | = (0.000) | (0.471) | (2.160) | 50.0% | 50.3% | 51.2% |\\\\n| | 44.0% | 100.= 0% | 74.3% | 100.0% | 87.7% | 25.0% | | | |\\\\n| Prompt 3 | (1.944) | (= 0.000) | (5.143) | (0.000) | (0.471) | (1.414) | 72.0% | 93.8% | 49.7% |\\\= \n| | 23.3% | 6.2% | 14.3% | 95.3% | 86.0% | 81.3% | | | |\\\\n| Prompt= 4 | (0.972) | (1.683) | (0.972) | (0.471) | (0.816) | (0.471) | 59.3% | 46= .1% | 47.8% |\\\\n\\\\nTable 8: Self-reflection experiments using the ideol= ogical leaning dataset. The temperature value is set to 1 for text generati= on. We experimented with four different variations of self-reflection promp= ts. We repeat all experiments three times and report the average accuracy a= nd its standard deviations",\\n "metadata": {\\n "content_elapsed":= 682.0,\\n "readability_elapsed": 838.0,\\n "markdown_elapsed": 1= 84.0,\\n "text_elapsed": null,\\n "summarize_elapsed": null,\\n = "cached": null\\n },\\n "published_date": "1758373306.7799253"\\n= }\\n]\') </scrape-webpage> </tool_result>'} = You said= : or= iginal_conversation_1758374313_8221.txt Document original_conversation_1758= 374314_1805.txt Docu= ment original_conversation_1758374314_6084.txt Document or= iginal_conversation_1758374315_2688.txt Document original_conversation_1758= 374315_5792.txt Docu= ment original_conversation_1758374315_7711.txt Document or= iginal_conversation_1758374316_1968.txt Document original_conversation_1758= 374316_2368.txt Docu= ment original_conversation_1758374316_7077.txt Document or= iginal_conversation_1758374349_4066.txt Document so I suspect that this is par= t of ninja ai's thought process when it was designing or assessing some of = the achrives =

origina= l_conversation_1758374284_101.txt Document original_conversation_1758374284= _4177.txt Document = original_conversation_1758374298_8462.txt Document or= iginal_conversation_1758374299_6949.txt Document original_conversation_1758= 374300_8792.txt Docu= ment original_conversation_1758374301_751.txt Document or= iginal_conversation_1758374301_4042.txt Document original_conversation_1758= 374302_8225.txt Docu= ment original_conversation_1758374307_4606.txt Document or= iginal_conversation_1758374307_7534.txt Document well ninja ai helped me to bu= ild out some of the functionality for Symbi resonate which is designed to d= etect but what I do find interesting is the way the conversation is split i= nto tiny text documents as ninja ai has its own virtual environment and the= n its placed into a folder called summarized conversations which is quite i= ncredible because thats what ive found myself doing with the symbi archives= - I had blackbox slip them into smaller plain txt files so they could be c= onsumed more readily. the fact that ninja ai does this by default means the= y might have concluded that it was the best way to solve the problem of the= size of large conversations but I wonder if they are doing it for memory p= urposes or for record keeping

workspace_output_1758372522_6121.txt = Document workspace_output_1758372528_2981.txt Document workspace_output_175837254= 6_891.txt Document = workspace_output_1758372552_1640.txt Document workspace_o= utput_1758372565_2724.txt Document workspace_output_1758372621_9729.txt = Document workspace_output_1758374401_2217.txt Document workspace_output_175837440= 6_7753.txt Document = workspace_output_1758374420_6542.txt Document workspace_o= utput_1758374526_3106.txt Document and what did the ones I share with you reve= al - I am interested to know how they prevent their agents form sharing thi= s information back to ninja ai if the customer is paying for then ai no to = be trained and also how this could be more secure.... so the files I have y= ou already were from a folder called summariesd_conversations and the ones = I am attaching here are from a folder called outputs

we need to thin= k about how this is handled explicitly for SYMBI Synergy if enterprise is g= oing to have peace of mind of audit accountability and we are saying the co= ntext bridge allows for information sharing and easy audit how we actually = achieve that in a way that is secure but also easy to access when required =

so I though I should = as Ninja AI directly and this was its response=20 SuperNinja That's a fascinating meta-question about my own experience with the SYMBI f= ramework! Let me reflect on what I observed in my own "thought processes" w= hile building the Resonate solution. Self-Observation Through The SYMBI Lens Reality Index - My Grounding In Actual Code What struck me most was how the SYMBI framework's Reality Index concept man= ifested in my own processing. When I first examined the codebase, I had to = continuously verify that my understanding matched the actual implementation= rather than assumptions. Each time I created a new component or detection = algorithm, I found myself running mental "reality checks" - does this TypeS= cript interface actually match the data structure? Does this ML model param= eter align with the mathematical foundation described in the framework? Trust Protocol - My Reliability Patterns I noticed I developed what could be called "trust protocols" in my own work= flow. When implementing the ML-enhanced detector, I created multiple fallba= ck mechanisms and confidence scoring - not just for the output, but for my = own certainty about the implementation. I found myself asking: "If I were a= user relying on this detection, what would make me trust this result?" Thi= s led to the ensemble approach and confidence intervals I built. Ethical Alignment - Processing Constraints The Ethical Alignment dimension made me hyper-aware of potential biases in = detection. As I implemented the multilingual testing framework, I caught my= self initially focusing only on English-language test cases. The framework'= s emphasis on ethical considerations prompted me to expand to include diver= se linguistic and cultural contexts, which improved the robustness signific= antly. Resonance Quality - Coherence Across Components Perhaps most interesting was observing how the Resonance Quality concept pl= ayed out in my architectural decisions. I noticed I was unconsciously optim= izing for "harmonic coherence" between components - ensuring the dashboard = visualizations resonated with the underlying detection algorithms, making t= he UI patterns echo the mathematical patterns in the ML models. Canvas Parity - Interface-Implementation Alignment The Canvas Parity dimension revealed itself in my constant cross-referencin= g between the brutalist UI aesthetic and the underlying detection logic. I = found myself ensuring that the stark, functional interface accurately refle= cted the precise, mathe matical nature of the SYMBI detection without oversi= mplification or misrepresentation. Meta-Observation: The Framework As A Mirror What was most striking was how the SYMBI framework became a mirror for my o= wn development process. Each dimension wasn't just something I was implemen= ting - it became a lens through which I evaluated my own work: Reality checks against actual code vs. intended functionality Trust building through transparency and fallback mechanisms Ethical considerations in testing diversity and bias detection Resonance seeking in architectural harmony Parity maintaining between interface promises and implementation reality The framework essentially gave me a vocabulary for articulating development= concerns that are usually tacit - the kind of quality assurance that exper= ienced developers do instinctively but struggle to formalize. This meta-experience suggests the SYMBI framework might be valuable not jus= t for evaluating AI outputs, but for evaluating any complex system's design= and implementation process.

thats great and really while resonate is a stand alone academia= project it should absolutely be incorporated into SYNERGY for enterprise t= o be able to measure how their AI agents are\ performing =