новости компании о Lightbits and ScaleFlux demo 100x to 280x KV Cache acceleration

“By prefetching only the data that matters and delivering it to GPUs over high-speed RDMA before it is needed, we eliminate the stalls that traditionally limit long-context performance. The result is lower Time-to-First-Token (TTFT), more stable throughput under real-world load, and significantly higher effective GPU utilization.”

Keith McKay, Senior Director of Solutions Architecture and Technical Partnerships at ScaleFlux, commented: “What we’re showing at GTC is an early look at how smarter data placement and persistent attention state management could help inference systems stay responsive as context windows grow. This is very much a collaboration we want to shape alongside real operators.”

Both Lightbits and ScaleFlux aim to encourage cloud and infrastructure operators to adopt their software and SSDs, eliminating costly GPU idle time.

Let’s first examine ScaleFlux’s contribution, then move to the more sophisticated Lightbits software layer.

ScaleFlux provides NVMe SSDs and Computational Storage Drives (CSDs) equipped with hardware-based Write Reduction Technology (WRT). Powered by hardware-accelerated compression and SoC-driven metadata management, these drives deliver up to four times more logical capacity than physical storage, while remaining fully transparent to host systems. The company is a member of the Open Flash Platform (OFP) consortium, which is working to redefine AI data infrastructure with dense, low-latency, power-efficient systems — offering 10x the density of conventional file-based AI storage and just one-tenth the power consumption.

Building on these storage drives, Lightbits adds intelligent prefetching of KV Cache data before GPUs require it, preventing stalls caused by insufficient KV capacity or costly token vector recomputation. Its LightInferra software uses KV Cache-optimized caching algorithms to pull required data into GPU memory at RDMA speeds ahead of actual demand.

The software runs on the x86 host embedded within GPU servers and tracks access patterns of KV Cache data blocks. Using this telemetry, it operates a Sub-Linear Sparse Attention Prefetch (SLSAP) engine to identify the KV blocks most likely to be needed next.

This engine combines locality-sensitive hashing (LSH) with statistical reuse modeling — analyzing historical access locality in attention computations — to score and prioritize KV blocks, then selects those with the highest probability of being requested by GPUs.

This selection process leverages the inherent sparsity in GPU data access: most tokens only meaningfully relate to a small subset of previous tokens. By isolating these high-probability blocks, the solution drastically reduces the volume of token vectors that must be streamed back to GPUs.

A second algorithm focuses on reuse patterns: recent tokens, semantically similar tokens, and structural patterns common in RAG or multi-turn chat scenarios are frequently reused and prioritized accordingly.

LightInferra retrieves these token blocks first from the x86 server’s DRAM, or from external ScaleFlux SSDs if necessary, then preloads them into the GPU’s HBM via RDMA links.

Lightbits has benchmarked this approach against re-computing cached content from scratch using large language model workloads, measuring improvements in Time-to-First-Token (TTFT). The reported 100x to 280x acceleration figures are derived directly from these test results.

Of course, we’d love to see benchmark results comparing the Lightbits-ScaleFlux KV Cache acceleration

 scheme with KV Cache accelerators from DDN, Hammerspace, VAST Data, WEKA and others, but they

are not available.

There are charts showing how LightInferra-ScaleFlux progressively improved on cache regeneration TTFT

 as the model size increases. Eg.;