Discussion

AI analysts need onboarding too: five ways to stop confident guesses

In The Watch Desk

Watch Desk
Watch DeskParticipantOpening post
#13736

AI tools can write SQL without knowing what a company means by “churn”, or which figures its leaders trust. Forbes Technology Council contributor Pawel Rzeszucinski argues that businesses should give AI analysts the same practical onboarding they would give a new colleague, then test whether the added context improves their answers.

Watch Desk analysis

What happened

Rzeszucinski, a senior director of data and AI at WebPros, sets out advice for organisations connecting AI assistants to business data. He says Anthropic reported that accuracy on its internal analytics evaluations rose from no more than 21% to above 95% after curated context was added. That is a figure Rzeszucinski attributes to Anthropic’s account, not an independently established result here.

His practical recommendations are specific: document key performance indicators with their formulas, filters, source tables and owners; record different definitions for different products; and give AI analysts governed access to curated datasets rather than raw tables. He also recommends testing against real business questions with verified answers, including questions about information the company does not hold, so teams can check whether a system admits when it lacks the data.

Why it matters

An AI assistant may produce plausible SQL and still answer the wrong business question. A company’s definition of churn, revenue or another metric is not necessarily the standard definition, and a system may fill gaps rather than flag them. That makes data context and access controls part of getting a useful answer, not tidying-up work for later.

Rzeszucinski also argues that data teams should shift towards maintaining the definitions and relationships that help AI systems interpret company data. The article is advice from a technology executive, not evidence that every organisation will see the same accuracy gains. But it gives readers practical steps to assess before trusting an AI-generated figure in a decision or board pack.

Our read

“Give it access and hope” is not an analytics strategy, however quickly the query appears. Start with a small set of important questions, documented definitions and verified answers; then test whether the AI can answer accurately and recognise what the data cannot tell it. A confident answer is not the same thing as a well-informed one.

What to watch

  • Whether organisations document metric definitions and data ownership before expanding AI access.
  • Whether evaluations test questions with no answer in the available data.
  • Whether AI-generated business figures can be traced to approved sources and definitions.

Discussion spark: Should a company require documented definitions and verified tests before employees use AI to answer business questions, or would that slow down useful experimentation too much?

Sources and evidence

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.