Google Open-Sources TorchTPU to Bring Native PyTorch to TPUs

Google has open-sourced TorchTPU, a software stack designed to let developers run PyTorch workloads natively on Google Tensor Processing Units (TPUs). The release aims to make it easier for developers to use Google's AI accelerators without significantly changing their existing PyTorch code.

Announced on October 21, 2026, by Google Cloud, TorchTPU was developed in collaboration with the Meta PyTorch team. It brings PyTorch APIs, debugging tools, compilation, distributed computing capabilities, and integrations with popular AI frameworks into a unified TPU development experience.

The release also extends beyond model training. Google is connecting TorchTPU with inference engines, distributed model-serving systems, reinforcement learning tools, orchestration frameworks, and edge deployment technologies.

For developers and organizations building AI applications, the announcement highlights Google's effort to reduce the software integration work involved in adopting TPUs while expanding support for the broader PyTorch ecosystem.

What Is TorchTPU?

TorchTPU is Google's open-source software stack for running PyTorch workloads on TPUs. Its central goal is to let developers use familiar PyTorch workflows while accessing Google's TPU infrastructure.

The stack supports PyTorch's eager execution mode for debugging and interactive development, along with torch.compile for optimizing workloads. It also provides access to PyTorch APIs, diagnostics, and distributed computing primitives.

This approach is intended to reduce the need for developers to maintain substantially different workflows for model development and accelerator execution.

Rather than introducing a separate development model, TorchTPU builds around the PyTorch ecosystem that developers already use.

How TorchTPU Connects With the PyTorch Ecosystem

Google's announcement covers several integrations intended to support different stages of the AI lifecycle, from model development to serving and deployment.

1. Hugging Face Transformers and Diffusers

TorchTPU adds TPU support to Hugging Face tools used for language models and generative AI.

With Hugging Face Transformers v5, developers can target TPUs through device_map="tpu". Diffusers support includes eager execution, torch.compile, and tensor parallelism. Google also reports nightly continuous integration testing for these integrations.

These connections are important because developers can work with familiar model libraries rather than relying exclusively on TPU-specific development paths.

2. vLLM expands TPU inference support

Google is working with Inferact to support the vLLM inference engine on TPUs.

The TPU backend includes support for continuous batching, quantization, multi-chip parallelism, and custom Pallas kernels. The integration also uses upstream vLLM components for workers, schedulers, and API servers.

These components address different parts of inference execution and serving. Continuous batching helps manage multiple requests, while quantization and parallel execution provide additional ways to configure model inference workloads.

The announcement describes an effort to connect TPU execution with the existing vLLM ecosystem. It does not establish that every model or workload will perform better on TPUs than on other accelerators.

3. SGLang brings its serving architecture to TPUs

Google also describes a TPU backend for SGLang developed with RadixArk.

The integration includes RadixAttention for key-value (KV) cache reuse and a disaggregation transfer backend for HiCache. These capabilities are relevant to serving systems that manage repeated prefixes, cached context, and the movement of work across inference stages.

The aim is to make SGLang's existing serving architecture available to TPU users while retaining its established components.

4. llm-d 1.0 supports distributed inference

The release also connects TPUs with llm-d 1.0, a distributed inference framework developed with contributions from Red Hat, IBM, and a broader community of more than 500 contributors across 25 organizations, according to Google's announcement. The project is part of the CNCF Sandbox.

The llm-d 1.0 release includes several capabilities:

• Prefix-cache-aware routing.

• Separation of prefill and decode workloads.

• Multi-tenant flow control.

• High availability for the router.

• Stable v1 APIs.

• Session-aware routing.

On Google Kubernetes Engine (GKE), organizations can use a common endpoint for inference workloads running on GPUs or TPUs.

Google also highlights results reported by Zenteiq. The company reported more than four times the aggregate inference throughput and an increase in daily generated-token capacity from 1.47 billion to 5.97 billion per 1,000 TPU v6e accelerators. These are specific reported results, not a universal performance guarantee for all deployments.

Training at Scale With TorchTitan

TorchTPU is also connected to TorchTitan, Google's PyTorch-based training framework for large-scale models.

Google lists training recipes for several model families and sizes, including:

• Llama 3 70B and 405B.

• Qwen 3 8B and 32B.

• Qwen 3 235B with a mixture-of-experts (MoE) architecture.

• FLUX.1 12B.

The announcement says the TorchTPU and TorchTitan path has been validated at scales of up to 8,000 Ironwood TPUs, also identified as TPU v7x.

Google also reports up to 52% model FLOPs utilization (MFU) for Qwen 3 8B, alongside near-linear scaling and stability at up to 8,192 Ironwood TPUs. The reported utilization figure applies to the stated workload and configuration. It should not be interpreted as a general result for every model or training setup.

These details indicate that Google's work covers both the developer experience and large-scale training operations. However, organizations will still need to assess model compatibility, workload requirements, and their own training configurations.

Official project: TorchTitan on https://github.com/google-pytorch/torchtitan?GitHub.

Reinforcement Learning With VeRL

Google is introducing a TPU backend for VeRL, a framework used for reinforcement learning of large language models. The integration is available as a preview.

The announcement also describes an end-to-end Group Relative Policy Optimization (GRPO) recipe using mathematical reasoning datasets.

GRPO is a reinforcement learning approach that trains models by evaluating groups of generated responses. Including a TPU backend and a complete recipe provides developers with a starting point for exploring these workloads on Google's accelerators.

Because the integration is in preview, its availability and readiness should be understood in that context. The announcement did not disclose additional details.

Ray Adds Orchestration Flexibility

TorchTPU also supports Ray, a framework used to orchestrate distributed applications and workloads.

This integration gives developers another option for coordinating work across CPU, GPU, and TPU resources.

For teams building multi-stage AI pipelines, orchestration can help coordinate tasks and resource allocation across a distributed environment.

Ray's inclusion broadens the set of tools that can be used alongside TorchTPU. The announcement does not provide a single performance figure for the Ray integration, so its practical benefits will depend on the workload and deployment architecture.

LiteRT Extends the Path to Edge Deployment

Google is also connecting the PyTorch ecosystem with LiteRT, formerly known as TensorFlow Lite.

The announcement describes litert-torch.convert(), which builds on torch.export() and Core ATen to convert supported PyTorch models for LiteRT. LiteRT-LM also supports Hugging Face Transformer language models.

Google identifies several deployment targets for LiteRT, including Android, iOS, macOS, Windows, Linux, the web, and IoT devices. Its runtime options include CPU execution through XNNPack, GPU execution through ML Drift, and access to neural processing units (NPUs) from major chipsets through its API.

This integration matters because model development and model deployment often involve different software requirements. Connecting PyTorch development with LiteRT creates a path for taking supported models beyond cloud-based infrastructure.

The specific devices, operators, and models supported will still depend on the relevant LiteRT configuration and backend. The announcement does not mean every PyTorch model can be converted and deployed identically across all listed platforms.

TPU Agent Suite Offers Early-Stage Assistance

Google has also introduced the TPU Agent Suite in early preview, with its code available on GitHub.

The suite provides plugins for selected coding harnesses and model combinations. Its intended role is to help automate aspects of TPU onboarding, diagnostics, technical questions, and inference tuning.

This is a developer-assistance layer around the TPU workflow rather than a replacement for the underlying model frameworks or infrastructure.

As the suite remains in early preview, its scope and supported combinations should be evaluated against the project's current documentation. The company did not provide specific information about broader availability or a general performance improvement from the agent suite.

Official project: TPU Agent Suite https://github.com/google-pytorch/tpu-agent-suiteon GitHub.

What TorchTPU Means for AI Developers and Businesses

TorchTPU's open-source release reflects a broader industry effort to make specialized AI hardware easier to use through established software ecosystems.

For developers, a more familiar PyTorch workflow could reduce friction when experimenting with TPU-backed workloads. Integrations with model libraries and inference engines may also make it easier to evaluate TPUs within existing development and serving systems.

For infrastructure teams, the combination of training frameworks, inference engines, orchestration, and deployment tools provides more options for assembling AI workflows. The GKE endpoint approach described for llm-d also offers a common serving interface for GPU and TPU workloads.

However, the practical impact will depend on compatibility, operational requirements, model architecture, and workload-specific performance. Open-source availability does not automatically mean that every integration is production-ready or that migrating an existing application will require no changes.

The most useful next step for interested teams is to review the relevant repositories and documentation, test representative workloads, and compare results against their own requirements.

Conclusion: A Broader PyTorch Path for Google TPUs

Google's TorchTPU release brings together PyTorch execution, model training, inference, orchestration, and edge deployment integrations in a more connected TPU software stack.

The announcement's key development is not simply that TorchTPU is open source. It is the breadth of the surrounding ecosystem, including Hugging Face, vLLM, SGLang, llm-d, TorchTitan, VeRL, Ray, and LiteRT.

For teams considering Google TPUs, these integrations provide more entry points for evaluating the platform within existing AI workflows. Their real-world value will depend on model support, deployment maturity, and performance under each team's specific conditions.