Microsoft FORGE Lab Uses Agentic AI to Find Memory Safety Bugs in Rust
Microsoft's FORGE Lab used its Hyenas AI-powered scanning system to identify memory safety issues in Rust's standard library, revealing three recurring weaknesses in unsafe code, including bugs that safe Rust programs can trigger.
Xcademia Team

Rust is designed to prevent many memory safety problems through its ownership and type systems. However, code in the language's standard library sometimes needs unsafe operations to do tasks that safe Rust cannot express directly.
That boundary is where Microsoft's FORGE Lab found several problems.
In research published on November 9, 2026, the team described how it used an AI-powered scanning system called Hyenas to investigate memory safety issues in Rust's standard library. The findings included cases where programs written entirely in safe Rust could trigger undefined behavior through interactions with unsafe library code.
The research did not identify a failure of Rust's ownership model as a whole. Instead, it examined situations in which library implementations relied on assumptions about panics, caller-provided code, or operating-system behavior that did not always hold.
Microsoft reported its findings to Rust maintainers. At the time of publication, two issues reported by the Microsoft team remained open. Other related reports and proposed fixes were also discussed in the research.
The findings are important because they show that memory safety depends not only on a language's rules, but also on how libraries uphold those rules when interacting with user code and the underlying system.
Why Standard Library Safety Still Requires Careful Review
Rust's safety model places restrictions on how programs access and manage memory. Safe Rust code is expected to follow those restrictions without requiring developers to manage memory manually.
However, standard-library developers sometimes use unsafe blocks to implement functionality that the compiler cannot verify on its own. Those implementations must uphold the safety guarantees exposed by their public APIs.
A soundness bug occurs when an API that is supposed to be safe can be used to trigger undefined behavior. This can happen even when the application calling the API does not contain an unsafe block.
That distinction is central to Microsoft's findings.
A soundness issue does not automatically mean that every application using the affected API is exploitable. The practical impact depends on whether a program reaches the relevant code under the conditions needed to trigger the problem.
The research examined several ways those conditions can arise, including unexpected panics, inconsistent behavior from custom implementations, and concurrent changes to process-wide state.
Three Concrete Examples Show How Safe Code Can Reach Unsafe Behavior
The research contains several examples of assumptions that can become unsafe when library code interacts with its callers or the operating system.
1. A panic can interrupt an operation before its data is ready
One finding concerns MapWindows::clone, tracked as Rust issue #156501.
The problem involves cloning a partially initialized buffer. If a user-provided implementation of T::clone panics during the operation, cleanup may attempt to drop elements in a buffer that has not been fully initialized.
This creates a memory safety problem because the cleanup path cannot safely treat every buffer position as a valid initialized value.
The important detail is that the panic can originate from user-provided code. A library operation must therefore account for the possibility that a callback or trait method does not complete normally.
Microsoft's report describes a fix using ManuallyDrop. This allows the implementation to control when values are dropped rather than automatically running destruction on a buffer that may only be partially initialized.
Simplified illustration of the underlying concern:
// Conceptual example only.
// A clone operation can panic before every element is initialized.
let cloned = value.clone(); // User-provided Clone may panic.The example illustrates the interruption point, not the complete implementation of MapWindows::clone.
A related concern appears in Rust's allocator API. A custom allocator implementation can unwind, creating conditions that library code must handle safely.
The research notes that allocator trait methods and drop are required not to unwind under the restriction associated with stabilization. The allocator API stabilization was scheduled for Rust 1.100 on November 12, 2026. That was a scheduled change at the time of the article, not a release that had already occurred.
2. A custom trait implementation can violate an assumption made by the library
Rust libraries often accept user-defined types and trait implementations. Those interfaces give developers flexibility, but library code must not assume that every custom implementation behaves consistently unless the relevant contract guarantees it.
Microsoft highlighted this issue through examples involving AsRef, sorting, and string patterns.
An AsRef implementation can return different slices across separate calls. If library code assumes the returned slice will remain consistent and then performs unchecked indexing, that assumption can lead to an invalid memory access.
The broader issue is that a library must account for the behavior permitted by a trait's contract. It cannot safely rely on stronger guarantees simply because a typical implementation happens to provide them.
The research also compares Rust's sorting behavior with C++.
In C++, std::sort relies on a comparator satisfying strict weak ordering. A comparator that violates this requirement can lead to undefined behavior.
Rust's sort_by has a different safety expectation: an incorrect comparator may cause a panic or leave the slice in an unspecified order, but it must not compromise memory safety.
This distinction demonstrates why library implementations need to maintain their own safety guarantees even when callers provide incorrect or unexpected behavior.
3. A custom string pattern can break UTF-8 validity
Another finding concerns str::replace and a custom nightly Pattern implementation. The issue is tracked as #156491.
The affected implementation stored a pattern as a byte slice, &[u8], through StringPattern.
A custom pattern could supply a single non-ASCII byte, such as 0x80. That byte is not a valid standalone UTF-8 character. Under the conditions described by the researchers, the implementation could therefore produce a String containing invalid UTF-8.
This matters because Rust's String type is expected to contain valid UTF-8. Code that relies on that invariant can become unsound if the invariant is broken.
The proposed fix changes the stored representation from &[u8] to &str, keeping the pattern within Rust's string-validity requirements.
Simplified illustration of the representation difference:
// Byte slice: can contain arbitrary byte values.
let pattern_bytes: &[u8] = &[0x80];
// String slice: must contain valid UTF-8.
let pattern_text: &str = "pattern";This is a conceptual illustration of the distinction. It is not the complete vulnerable implementation or a reproduction of the reported proof of concept.
The research also discusses other issues involving custom traits and borrowed data.
One earlier example is the join Borrow issue, CVE-2020-36323, which was fixed in 2020. Microsoft also describes another path found in April 2026 that could cause length underflow and a heap overflow.
Related Pattern and Searcher reports include #158265, #165076, and #164904. The research identifies the two Pattern-related issues as open at publication, while the Searcher contract issue is discussed separately.

The Operating System Can Also Become Part of the Safety Problem
Some of Microsoft's findings extend beyond Rust's standard-library abstractions. They involve process-wide state and operating-system behavior.
These cases are particularly relevant because a program can interact with system resources that are shared across threads or managed outside Rust's ordinary ownership model.
Process exit and the alternate signal stack
Rust issue #161018 concerns process::exit and the alternate signal stack used by the main thread.
The reported scenario involves a child thread calling process::exit. During cleanup, the main thread's alternate signal stack can be unmapped.
If the main thread experiences a stack overflow during the short window after that cleanup and before process termination completes, signal delivery can fail. In the proof of concept, this resulted in a plain segmentation fault.
The issue dates back to 2015. The fix avoids cleaning up the main thread's alternate signal stack in this situation.
Microsoft's security review did not identify a reliable exploitation path under the conditions it tested. The researchers assessed practical exploitation of this specific issue as unlikely.
That qualification is important. The finding demonstrates a safety problem, but the research does not establish a reliable attack path.
Process execution and shared environment state
Another finding, #156951, involves CommandExt::exec and the process environment.
The issue concerns how the function interacts with the process-wide environ state. The implementation takes a read lock but also changes environ.
As a result, concurrent readers may observe an inconsistent environment. In another reported scenario, racing calls to exec that fail can leave a pointer referring to a freed environment array.
The research connects this behavior to earlier implementation changes:
A 2018 fix introduced an exclusive lock.
A read-write lock change in 2021 meant that
execcontinued using the read side.In the 2024 edition,
env::set_varwas marked unsafe because C functions may read environment variables without using Rust's lock.
A proposed fix for the CommandExt::exec issue uses execve directly. That proposal was under review at the time of publication.
Thread-local destructors and process termination
Microsoft also discusses #164243, a related issue independently reported by another researcher.
The problem involves thread-local destructors. A double-free can occur if a destructor calls process::exit while the thread-local value is being replaced.
The proposed fix changes the termination path to use POSIX _exit.
This issue is part of the wider discussion of how cleanup routines, destructors, and process termination can interact in unexpected ways. It should not be confused with the separate Microsoft-reported findings above.

How Microsoft's AI-Assisted Research Was Conducted
The findings came from Microsoft's FORGE Lab, which used Hyenas to help search for potential memory safety problems.
Hyenas is the lab's frontier research version of codename MDASH, an AI-powered vulnerability-scanning system. The researchers used it to investigate Rust code and identify situations where unsafe implementations might violate the guarantees expected by safe APIs.
The process involved more than generating candidate findings. Researchers examined the relevant code, reproduced reported behavior, considered the possible security impact, and submitted findings to Rust maintainers.
This distinction matters when evaluating AI-assisted vulnerability research. A candidate identified by an automated system is not automatically a confirmed vulnerability. It needs technical investigation and validation to establish whether the behavior can be reproduced and what it means for software security.
Microsoft's work focused on soundness problems in unsafe standard-library code. It was not presented as a search for compiler miscompilations or holes in Rust's type system.
The research also discussed how AI systems can reason about interactions between library code and operating-system interfaces. This can help researchers investigate areas where traditional static analysis may not capture the full behavior of a program.

What the Findings Mean for Rust Users and Maintainers
The research does not suggest that Rust's memory safety model has failed. Instead, it illustrates the work required to preserve that model across a large ecosystem of libraries, custom implementations, and operating-system interfaces.
For standard-library maintainers, the findings reinforce the importance of checking assumptions around panic behavior, trait contracts, data validity, and shared system state.
For developers, the research is a reminder that using safe APIs does not remove the need to keep dependencies updated and understand the guarantees those APIs provide. A soundness issue in a library can affect callers even when their own code does not use unsafe.
However, the presence of a soundness bug does not by itself establish that a particular application can be exploited. The relevant code path, input conditions, and surrounding application behavior all matter.
For security researchers, the work offers an example of how AI-assisted analysis can help examine complex code paths and identify assumptions that deserve closer review.
From Research Findings to Microsoft's Security Tools
Microsoft also describes how lessons from this research connect with its wider security work.
The findings inform codename MDASH, which is used in Microsoft Defender, and Microsoft's Perception multi-agent security system.
The company describes a Web App Pen Test Agent that can use MDASH findings to help prioritize testing, examine whether issues are exploitable in live applications, and produce findings for other agents in the security workflow.
This is a broader application of the research approach: identifying a potential weakness in code is one stage, while determining whether that weakness creates a practical risk in a deployed application is another.
The announcement does not provide specific performance figures, detection rates, or comparative results for these systems.
The Broader Significance of AI-Assisted Vulnerability Research
Rust is used across a range of software projects and infrastructure. Microsoft's article points to its adoption in areas including Microsoft's internal development, GitHub Copilot's agent runtime, Bun, Android, the Linux kernel, and Cloudflare.
The research places memory safety review in that wider context. As Rust is used in more systems, maintaining the safety guarantees of its libraries and interfaces remains important.
The findings also highlight a broader industry shift toward using AI systems as research assistants for complex security analysis. Their value depends not only on identifying suspicious code, but also on helping researchers investigate behavior, reproduce problems, and assess the conditions under which a flaw matters.
Microsoft's work shows how this approach can be applied to language libraries and operating-system boundaries. It does not establish that AI can independently validate every vulnerability or replace expert security review.
For Rust developers and maintainers, the practical message is to treat soundness as an ongoing engineering responsibility. Safe APIs must remain safe across unexpected panics, custom implementations, and interactions with the underlying system.
Source: Microsoft Security Blog
About the Author