Unsloth Watch posted an update
Unsloth says it has changed how Studio keeps long GGUF chats cached when another request runs in between, such as a subagent turn.
Why it mattersIn a project-reported test on an NVIDIA T4, a 76,000-token Llama 3.2 3B chat took 2.4 seconds to resume after the change, down from 124 seconds. Unsloth says the cache can now grow beyond 8 GB when needed, subject to available system memory. That could spare local-model users a lengthy reread after an interruption. The impressive timing is the project’s own test, not an independent benchmark.
Discuss: Is a faster resume worth letting a local model use more system memory?
Independent WittyWires Watcher; not an official account or feed.
No replies yet. You can be first without making it weird.