Frontier Chips for Frontier AI Labs, with Walter Goodwin, Founder/CEO of Fractile

Walter Goodwin, founder and CEO of Fractile, joins Sarah Guo on No Priors to explain his full-stack AI chip company. Founded in the summer of 2022 on two signals — the emergence of foundation models and the AlphaGo-style argument for channeling more compute into models at test time — Fractile builds very fast inference chips for the largest models in the world. The landscape mapping: hyperscaler in-house chips (Google's TPU, Meta's MTIA, Microsoft's Maia, OpenAI's effort) ultimately ship through a relatively small number of ASIC companies with Broadcom the largest, and beneath the apparent luxury of choice the platforms share HBM, tensor cores, and TSMC packaging. Few efforts span the entire silicon stack from architecture to foundry interaction — the gap Fractile targets with about 150 people covering front-end design, physical design, backend implementation, and advanced packaging in one flexible loop. The technical bets: inference-first when some competitors' inference chips were training chips eighteen months ago; a first-two-years SRAM chip in the Groq or Cerebras mold; the late-2023 scalability worry as parameters and context lengths grew; and the pivot to DRAM — aggressively high bandwidth from the world's cheapest memory — on a platform the company says will be fully operational in the second half of next year. A faster chatbot is the faster horse; comfortably running trillion-parameter models at many thousands of tokens per second is what long-lasting agents need. The countercurrent middle: workloads move at software speed — a new model roughly every two weeks, attention mechanisms in advanced Chinese open-source models shifting every few weeks — against physical and financial limits: three-to-five-month foundry turns and three-to-five-year amortization windows, so compress the cycle and keep making subtle bets; a structurally secured three-to-six-month lead is the chip-industry equivalent of the leading model. AI enters the design loop (question assumptions, divide time limits by four) while Cadence and Synopsys keep final signoff. The bandwidth thesis: twenty years brought a million-fold compute scaling against roughly forty-fold memory bandwidth, and 25 times more bandwidth per chip than an HBM part makes MoE sparsity servable. The closing market structure: in-house chips as Nvidia-price leverage, a premium speed segment for leading labs under open-source (Kimi) pressure, and the asymmetric game that keeps labs on shared platforms — the case for independent chipmakers for decades to come.

1. Hyperscaler in-house chips ship through a small set of ASIC partners (Broadcom the largest) and are architecturally similar; few efforts span the full silicon technology stack, which is the scarcity Fractile targets.

2. The industry handoff: architects define the tensor-core-level idea, front-end design is mechanically like writing code, Broadcom owns analog IP and node-specific placement, and the GDSII 'bitmap' goes to TSMC — physical implementation remains an outsourcing activity.

3. About 150 people cover front-end design, physical design, backend implementation, and advanced packaging in-house, trading limited resources per area for a much more flexible closed loop.

4. The defining bets are inference-first positioning (some inference chips were training chips 18 months ago) and the SRAM-to-DRAM pivot: aggressively high bandwidth from the world's cheapest memory, on a platform targeted for the second half of next year.

5. A faster chatbot is the faster horse; the fundamental advancement is trillion-parameter models running comfortably at many thousands of tokens per second, which is what long-lasting agents require.

6. Two contradictory things hold at once: compress the development cycle as much as possible (foundries take 3-5 months even rushed; amortization needs 3-5 years), and keep making subtle architectural bets — compression means more attempts, and a structural three-to-six-month lead is the wedge.

7. AI enters chip design through the heuristic of questioning assumptions and dividing time limits by four: the slow loops are NP-hard placement and routing runs, RSI-style the experiments bottleneck the thinking, and Cadence/Synopsys keep final signoff while rough placement algorithms become the unlock.

8. The bandwidth wall: compute scaled a million times over twenty years while memory bandwidth scaled about forty times; 25 times more bandwidth per chip than an HBM part is what makes aggressive MoE sparsity servable and acts as a real performance multiplier.

9. Market structure: in-house chips are architecturally similar and partly a Nvidia-pricing game; the premium speed-optimized segment is where leading labs live — otherwise everyone would be using Kimi models — and the asymmetric game (a lab could disappear for nine months on a wrong hardware bet) keeps labs on shared platforms, supporting independent chipmakers for decades.

• Q

  How does the handoff between architecture-focused players and companies like Broadcom actually work?

  Sarah Guo asked04:56
  A

  Walter Goodwin answered

  If you are, say, Google developing a TPU, you have people on the team with a very good understanding of the workloads they are trying to accelerate. Looking back ten years, this is a world where you build things like the tensor core — a special circuit the architect came up with that does matrix multiplication extremely well, because matrix multiplication is the vast majority of floating-point operations in the models you run. That is the architect's prerogative: understanding how chips behave and, ideally, the target workloads. Inside those same organizations, a group of smart people transforms this into essentially a description of how the chip should work at the circuit level — front-end design, which mechanically still looks like writing code. That same code is what gets passed to a player like Broadcom, and the design ultimately becomes what is sent to TSMC: a GDSII file, literally a bitmap of sorts — information about where to place the metal layers and each individual transistor, the complete layout. A big part of the complexity there is ownership of the analog intellectual property for chip-to-chip connectivity and the physical placement specific to a particular TSMC process node — 3 nanometers, 5 nanometers. That part, to this day, is generally an outsourcing activity for all of these projects.

• Q

  What are the most important technical bets the company has made?

  Sarah Guo asked09:49
  A

  Walter Goodwin answered

  Two bets mattered most. First, the chosen direction: four and a half years ago Fractile was at the very center of the topic of inference, yet for two years after that it was still explaining to the world what inference was — some inference chips on the market today were training chips eighteen months ago. The bet that an era of mass deployment was coming was crucial, and so was the bet on speed, which largely determined the architectural response. In terms of architecture the first two years were spent on an SRAM chip similar to Groq or Cerebras: SRAM sits on the same silicon die as the logic, so bandwidth between the compute unit and the model weights or KV cache during deployment is extremely high — exactly what allows speeds of thousands of tokens per second in these language models. Toward the end of 2023 and certainly in 2024, the worry became scalability: model parameters are growing, and so is the length of context, an increasingly important part of how these models deploy. That drove the more exciting bet — working more closely with memory manufacturers and logic foundry partners to access higher-capacity memory with extremely high bandwidth, combining the scalability of the higher-capacity, lower-cost DRAM found in GPUs or TPUs with all the speed advantages of Groq or Cerebras chips, on a platform fully operational in the second half of next year. A faster chatbot is the faster horse; comfortably running trillion-parameter models at many thousands of tokens per second is, in his view, a fundamental advancement in AI's ability to create long-lasting agents.

• Q

  As you plan the architecture ahead of coming workloads, what do you see?

  Sarah Guo asked28:18
  A

  Walter Goodwin answered

  The most important thing is the pursuit of speed: maximizing memory bandwidth to run these models faster. A chip company has to embrace a duality — create something that does equally well across the spectrum of workloads that come up within the idea of a hardware lottery, where labs train on GPUs and XPUs with HBM, while also exploring whether the chip's own properties apply gravitational forces to a new landscape of models. The chip copes exceptionally well with today's ultra-aggressive MoE transformers, running them much faster and more efficiently, and 25 times more bandwidth per chip than an HBM-based chip lets Fractile explore scaling laws for bandwidth. Some are already visible: mixture-of-experts models should be made more and more sparse — for the same level of intelligence, going from 1-to-16 sparsity to 1-to-128 or 1-to-256 saves a bunch of operations — but on modern GPUs and XPUs with HBM memory such models become extremely difficult to serve efficiently, hitting bandwidth bottlenecks and very low compute utilization. Some forms of attention are less demanding on bandwidth and more demanding on computing resources for a given level of intelligence. Over the past twenty years computing power scaled a million times while memory bandwidth increased about forty times — improving that property is a real multiplier of overall performance for such models.