ReplaySSM cuts memory traffic to speed up hybrid-model decoding
NVIDIA and Princeton’s Dao AI Lab have developed ReplaySSM, a method for speeding up inference in hybrid AI models by changing how their state is stored. NVIDIA reports up to 1.80× faster SSM decoding kernels at batch size 256, with larger
Open discussion →