Should We Train Our Own Model?

I sometimes hear, “We need to train our own model so it responds the way we need.”

The answer is almost always no.

The concern is usually that an existing model will not consistently use the right information, follow the instructions, or respond in the right tone. There are better ways to address those problems when you are building a product that uses AI.

Start by creating an evaluation framework. Gather a range of realistic inputs, define what a good result looks like, and have people review and score the responses.

At Okaya we built one for harm detection. It runs a set of scenarios with expected results reviewed by licensed professionals, and when we changed models it showed us what improved and what regressed.

Once you can evaluate the system, start experimenting. Try different models, adjust the prompt, change the information provided to the model, and run the evaluation again. This gives you a way to see what actually improves the results rather than relying on a few examples that seem good.

Fine-tuning may be worth considering after you have exhausted those options, especially when working with smaller open-source models. Even then, fine-tuning an existing model is very different from training one from scratch.

For most teams, the better investment is not training a model. It is building a reliable way to test the system and improving the model selection, data, prompts, and software around it.

Source: Should you train your own LLM? on Responsible Innovation

Originally published on LinkedIn — view the original post for comments and reactions.