How to evaluate a local AI system with one workflow

The fastest way to evaluate a local AI system is to bring one real workflow and measure it on hardware like yours. Not a benchmark. Not a slide. The task an analyst does today, with the documents they actually use.

This is how smolsi evaluations run. An engineer replies within one business day, a 30-minute call scopes the workflow, and the system is configured around it. This guide describes the method so you can prepare, and so you can apply it to any vendor.

Step 1. Pick one workflow

Choose something that recurs, that someone does by hand today, and that has a clear output. Good candidates:

• Reconciling exported ledgers against a board deck or a report.
• Comparing contract clauses across a set of agreements.
• Answering questions over an internal document set, with sources.
• Turning a spreadsheet into a specified analysis with tables and charts.

Avoid "general chat". It cannot be measured, so it cannot be decided.

Step 2. Define what done looks like

Before the first run, write down three things:

1. The input. Which documents, spreadsheets, or images, and roughly how many.
2. The output. A table, an answer with citations, a chart, a checked list. Something a reviewer can mark right or wrong.
3. The constraint. Must it run disconnected? Which hardware does the team already have? Who needs to review the result?

This becomes the written scope that follows the call.

Step 3. Run it where it will live

Evaluate on the deployment shape you would actually use:

• Laptop. One analyst, a signed application, working with the network unplugged.
• Shared node. A team, with concurrent sessions and background workers.
• Private cluster. Dedicated capacity for isolated environments, with deployment engineering. Currently in early access.

The configuration is sized around the workflow. If the evaluation runs on hardware nothing like production, the result does not transfer.

Step 4. Watch the whole loop, not the answer

A local AI system is more than a model. During the run, check each stage:

• Context. Did the document index surface the relevant material, and only that?
• Reasoning. Did the model direct the task sensibly and delegate analysis where it should?
• Execution. Were calculations performed by the host and were the tables and charts correct?
• Review. Could the reviewer trace the result back to the source material?

The model specifies the analysis; the host computes it. An evaluation should confirm both halves.

Step 5. Decide on evidence

At the end you should have: the scoped workflow, the configuration it ran on, the outputs, and the reviewer's verdict on them. If a security review is part of the decision, supporting documentation is available under NDA, with the applicable scope for your deployment.

Then decide. A good evaluation can end in "not a fit". That is a successful outcome, and it should be reached in weeks, not quarters.

What to bring to the first call

• The workflow, in one or two sentences.
• An example of the input documents, or a description if they cannot be shared yet.
• The hardware your team uses today.
• Whether the deployment must operate disconnected.
• The compliance frameworks that apply.

Request an evaluation or pick a time for the call.