AI moves fast. How do you make your automation investment last?

Better AI models should make your automation more useful. Build on your company's knowledge and systems so each improvement in AI is an opportunity to improve your workflow.

If a smarter AI model launches tomorrow, would you be happy or disappointed? We think you should be happy. When you invest in automating a process, the software should keep solving a business problem as AI improves. A better model might handle more of the work correctly or need less checking by your employees. The connections to your internal systems and what your company learns about the process give that model a useful system to improve.

If a stronger model would make most of your work irrelevant, reconsider the investment. Model providers have far more resources to improve general capabilities, and we would focus on helping models work with your business instead of compensating for limitations a future release may remove.

Start from a business capability

Start with the work you want to improve, with the model as one part of making that possible. We’ll use a hypothetical logistics company as a running example. Its employees receive shipment requests and transfer information into internal systems. The goal is to reduce manual entry without introducing incorrect records.

The engineers building this automation need to connect it to those systems and give employees a way to check the extracted information. Those connections remain useful when a better model becomes available, along with the company’s knowledge of request formats and exceptions. Define success around the work: measure how much information the system extracts correctly and how long employees spend reviewing it. Those criteria will let you check whether a new model improves the process.

How do you avoid AI vendor lock-in?

Open standards let systems exchange information without depending on one supplier’s tools. In our shipment-request example, the company should be able to export its records and checked test examples in a documented format another tool can read. Ask for the custom application code, setup instructions and access needed for another engineer to maintain the system.

We prefer open-source software for components such as the tools that process documents or store results. Engineers can inspect and adapt that code, but check whether its licence permits your intended commercial use and what obligations apply when modifying or distributing it. Check maintenance too. A supplier may reserve needed features for a paid version or change the licence of future releases. Existing licence rights remain, but you may have to maintain that version yourself or migrate.

This choice of application software is separate from which AI model you use. A hosted model can fit, provided its data-processing terms meet your requirements and the application can use another provider. We would also consider proprietary software when it performs much better and offers a practical way to export the data and move elsewhere. Account for the work of moving with either kind of supplier.

Make in-house knowledge part of your investment

Make learning part of delivery. Your employees should develop an intuition for where AI works well and where it struggles. Practical experience working with its output helps them spot useful opportunities for automation and judge future changes without depending on the original supplier to explain them.

During prototyping, engineers compare possible approaches through experiments, such as trying different models or ways to prepare the input. Document what they tested and why the results led to a decision, including unsuccessful approaches the next engineer should not repeat. In our shipment-request example, employees might find they need the original request beside the extracted information to check it. That finding applies regardless of which model reads the request.

Let employees who know the process review real examples with the engineers and discuss why an output helps or misleads. A final report alone cannot build that shared knowledge. Agree how employees will contribute without running every experiment, so they gain practical experience while their daily work continues.

Build automated evaluation to guide future model choices

Automated evaluation means running the system on a collection of examples and checking its results against expected answers. Build these checks during prototyping to guide improvements to the initial system, then keep them to test better models as they appear. A few convincing outputs may hide failures that these repeated checks can reveal.

Following our shipment-request example, collect representative requests and have employees check the correct information to extract. Include incomplete requests and unfamiliar layouts. When information is missing or unreadable, the system should flag it for an employee to review rather than invent a value.

Test a new model against the same examples as your current system. Compare with a simple baseline too, such as manual entry or software that extracts information using fixed rules. Measure:

  • Correct, incorrect, and missing fields.
  • Errors that would create an operational problem.
  • How often employees need to review or correct a result.
  • Processing cost and response time.

Automate checks such as whether extracted values match the reference answers, and define a review guide for judgements about usefulness. If another model scores outputs, check its agreement with human judgement first. Reserve some examples from day-to-day tuning to check whether improvements hold on unseen cases, and record model versions and configurations to make comparisons repeatable. As new document formats and operational exceptions appear, add them to the tests to keep them representative of daily work.

Test cheaper options too, including models you host yourself, by comparing total operating costs and checking whether each option meets your quality criteria. Evaluation makes that trade-off measurable. A cheaper model that meets your criteria can save money, while a stronger one must justify its price and the work of switching. Our guide to structured evaluation pipelines explains the technical setup in more detail.

Design the software so you can replace models

Ask your engineers to keep the code that sends information to the AI service and receives its response separate from the rest of the application. In our example, replacing the model should not require rebuilding the connections to internal systems or the interface employees use to check shipment requests. Checks for missing details and rules for when an employee must review a result should remain usable too.

A replacement may need different instructions, known as prompts. Engineers may also need to change how the application prepares the input. Switching providers can involve more than changing a service address, but it should not require rewriting unrelated application code. One separate piece of code for the initial AI service can be enough for a prototype. Add support for another when you have an alternative to test.

Use your automated evaluation to see whether a better model improves the workflow. Inspect failures alongside average scores, and reject changes that introduce errors your employees cannot reliably catch. A model that handles more shipment requests correctly could reduce review work while the rest of the system stays in place. Switch when the evidence supports that improvement.

Frequently asked questions

Does the prototype need an open-source AI model?

No. Open-source application software can work with a hosted AI model. To run a model yourself, you can use an open-weight model, which provides downloadable trained model files. Its licence may still restrict commercial use or modification, so downloadable weights alone are not enough to call a model open source. Check the rights for your intended use rather than relying on the label.

Can automated tests replace feedback from our employees?

No. Employees define correct results and relevant exceptions, while automation makes those checks repeatable. Review them as your business changes.

Should we replace our model whenever a better one appears?

Test it on your own evaluation data first and compare gains with switching costs. Better AI gives you an opportunity to improve your automation, but your current model can keep doing its job until a replacement offers a worthwhile improvement.