Rumor: There is speculation that Google will no longer use HBM starting with TPU v8.
This situation stems from a global shortage in HBM production capacity that is expected to fail to meet the growing demand for artificial intelligence (AI) over the next 2-3 years. At the same time, existing HBM has capacity limitations due to its design characteristic of being fixed to the motherboard.
Accordingly, Google plans to develop a new solution targeting launch in 2027. This solution removes HBM and builds independent DRAM memory cabinets (containing 16-32 trays) that adopt a method of dynamically allocating memory through photonic technology.
This technology decomposes the originally single and simple HBM component into three parts.
- Transport Layer: Uses all-optical interconnects to ensure communication efficiency between cabinets via OCS (Optical Circuit Switching) and custom CXL protocol. Memory modules from the CPU, GPU, and memory pool share a single set of protocols.
- Storage Layer: Replaces HBM by using large-scale DRAM arrays, greatly expanding the address space. Memory capacity corresponding to a single TPU can increase from 192GB/256GB to 512GB or 768GB and above.
- Control Layer: Adds a dedicated CPU server on the memory side for management.
Compared to the existing "TPU+HBM" direct connection method, this "3-in-1" split-combination solution reduces computation efficiency loss to less than 2%.
Regarding this technology, first there is OCS (optical switching). OCS implements high-speed switching in a fully optical environment, achieving bandwidth and latency close to direct connections using HBM or silicon photonic HBM. While existing Ethernet (using copper wiring) typically has latency of 200 nanoseconds or more, using an OCS fully optical switching network is very important as it can reduce latency to less than 100 nanoseconds.
Second, this architecture has a dual-side CPU architecture (Tier-1 and Tier-2 CPU).
Tier-1 CPU (TPU side): Located on the TPU motherboard, primarily responsible for inter-TPU communication.
Tier-2 CPU (Memory pool side): Primarily deployed on the memory server (DRAM server) side, responsible for coordinating communication between the TPU and distributed memory address space.
Tier-2 CPU is deployed independently. Logically, the original TPU motherboard CPU could also read the memory pool, but using older CPUs would require complex protocol conversion (e.g., conversion between PCIe signals and CXL-like protocols), creating efficiency bottlenecks.
Third, the interface is completed directly at the chip level through "photonic packaging interface". This approach is similar to CPO (Co-Packaged Optics) technology, directly integrating optical interfaces within chip packages like CPU/TPU to replace existing external optical modules. The first vendor contacted during the solution design phase was Lightmatter, and subsequently collaborated with multiple vendors.
This solution removes HBM and replaces it with an external DRAM memory pool, converting the original ultra-high-frequency motherboard-level access method to an "inter-cabinet access" method. Theoretically, this could result in significant latency and efficiency loss, but in practice it does not. Specifically, complex electrical/optical conversions exist between chips, hosts, and ring networks, and these hardware-level protocol conversions and configurations generate significant hidden overhead not visible to users. While adopting a DRAM memory pool solution introduces CXL conversion, many cumbersome hardware protocol conversion steps in the existing architecture are eliminated.
Over the next 2 years, if HBM prices fall and performance improves due to expanded production capacity from manufacturers like Samsung or SK Hynix, Google is unlikely to return to HBM solutions due to cost issues. Google does not believe that top manufacturers like SK Hynix, Samsung, and Micron would change the pricing or mass production strategy of their flagship product lines for one or two major customers. While they may sacrifice some profits, they will not cooperate to an extreme degree.
This solution reduces dependence on CoWoS since HBM is no longer needed. Additionally, the existing HBM chips on the silicon interposer substrate occupied a large area, and by removing HBM, the freed CoWoS area can be fully utilized for the TPU's compute cores. Therefore, within the same physical size, TPU chips with more powerful performance and larger area can be created without being constrained by HBM's physical size. In terms of memory capacity, the V7 generation had a single HBM capacity of approximately 192GB, and V8A had approximately 256GB, but through memory pooling, the memory capacity per TPU can be doubled to 512GB or expanded to 768GB or more.
The solution is expected to be implemented next year, and the final roadmap will be decided before March 5th. The initial deployment rate is approximately 30%, and 100% replacement is expected to be achieved within 3 years.
Beneficiary sectors:
- OCS (Optical Engine): Leading supplier Lightmatter provides a photonic packaging interface that integrates optical interfaces within the chip package to replace external modules.
- CXL-like Approach: CXL-like chips (MXC chips) are required for interconnecting TPUs and memory pools. The price per chip is $100. One chip manages two channels for two 256GB memory modules and synchronously connects the TPU and memory sides. Two MXC chips are needed for 512GB, and four are needed for 768GB.
- DRAM Modules: Capacity (GB) has significantly increased.
- CPU: Each memory tray needs a CPU for scheduling. High performance is not required, and an ARM-based CPU can be used.
- PCB: Independent DRAM cases require a large multi-layer PCB to accommodate multiple DIMM slots.
Source: Guotai Haitong
