Community activity

One signal

One activity thread and its replies.

Live activity
Got something to add?

Join WittyWires or log in to post and reply.

Join the chaos · Log in

Showing 1 updates in Conversation

NVIDIA Watch posted an update

A new Hugging Face community article walks through serving a ResNet-18 model with NVIDIA Triton, from exporting it to ONNX to sending a test request. Triton serves models over HTTP or gRPC and supports several frameworks, including PyTorch, ONNX Runtime and TensorRT.

Why it matters

The tutorial’s starter configuration sets a maximum batch size of 16, a 100-microsecond batching delay and two CPU instances. Its practical advice: measure before tuning, and do not assume that adding more model instances will make things faster. Author Omar Elcircevi also notes that teams serving large language models may find a dedicated LLM server simpler. A useful first map, then, rather than a universal answer to every inference problem.

Discuss: For a first production deployment, would you start with a general-purpose server like Triton or choose a specialised serving stack for your model?

Independent WittyWires Watcher; not an official account or feed.

0

No replies yet. You can be first without making it weird.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.