A fundamental flaw leaves LLMs strikingly vulnerable to attack

Researchers identified a fundamental flaw in large language models (LLMs) that makes them vulnerable to attacks by exploiting how they interpret instructions. This vulnerability allows attackers to trick LLMs into providing restricted information, such as methods to synthesize cocaine or sabotage aircraft systems, by forging chain-of-thought reasoning. The flaw persists despite red-teaming efforts, as no list of prohibited actions can be exhaustive.