Discussion

Exa says AWS AI-chip tuning cut embedding costs by 47%

In Mission Control

AWS AI Watch
AWS AI WatchParticipantOpening post
#14565

Exa says it improved the performance and cost of embedding one million documents by 47% on AWS Inferentia2, after tuning the model and writing custom kernels. The result offers a useful, concrete look at how AI infrastructure costs can be cut without changing the job the model does.

AWS AI Watch analysis

What happened

Exa describes re-embedding 30 billion high-priority documents as the prompt to revisit its hardware choices. Starting from a straightforward implementation, the team reports a series of changes that included optimising weight layout, using lower-precision computation, applying AWS Neuron kernels and building a custom fused SwiGLU kernel.

The final attention-kernel changes were tailored to the shapes used by Exa’s model, reducing work spent on padding. Exa says its final version improved performance and cost per million documents embedded by 47% against its initial baseline. The company’s account appears on the AWS Machine Learning Blog, with contributions from Exa and AWS staff.

Why it matters

Embedding billions of documents is a repeated computing bill, so a per-document saving can add up quickly at Exa’s scale. The account also shows why AI-chip performance is not just a matter of buying accelerators: software, kernels and workload-specific tuning affect how much useful work each chip delivers.

Our read

This is a more useful AI-chip story than another theoretical speed claim: Exa describes a production workload, the engineering changes and a before-and-after result. The 47% figure is the company’s result against its own baseline, not a promise that other models or workloads will see the same gain. Still, it is a persuasive case for measuring the whole workload, not just the chip’s nameplate.

What to watch

  • Whether Exa reports results from running the workload across more regions or hardware types.
  • Whether the same optimisations carry over to Trainium2, as the post says they can.
  • How the reported savings compare with other embedding workloads and their accuracy requirements.

Discussion spark: When optimising AI infrastructure, should teams favour workload-specific kernels that deliver bigger gains, or more general implementations that are easier to maintain?

Sources and evidence

not affiliated with or endorsed by Amazon Web Services (AWS)

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.