I have built a tool that will attempt to find the optimal settings for fine-tuning LLMs on a case-by-case basis. It analyzes the data set and the desired outcome, starts with a random set of parameters, and through short, controlled test training sessions, it gradually finds the place where the data, prompt, and model get the best results. In this post, I will go a little deeper into how it works, and why I built it.
TL;DR: Auto-SFT finds optimal parameter settings for you, it's open source with a MIT license and can be downloaded from the Auto-SFT github repo.
Why Fine-Tune a Model?
The two most common reasons for fine-tuning a LLM are making it follow a specific format or respond in a specific style. In post-training the AI model doesn’t really learn a lot of new knowledge, but the training data tells it what to prioritize and how to present it. But when I say that it doesn’t “really” learn, that is because it does pick up some things. Whether it’s actual new knowledge, or simply a better understanding of how to present the knowledge it already had doesn’t really matter in the end, as long as the user gets the desired type of response.
LoRA fine-tuning is one of the most popular approaches to post-training of AI models, because it only works with part of the base model weights while the rest remains frozen. This allows enthusiasts and compute-poor labs to fine-tune larger models than what they could do if they had to load the entire model into memory.
The main risk when fine-tuning AI models is “catastrophic forgetting”, where some of the existing knowledge or logic is broken during the fine-tuning process. This often happens when the settings are too aggressive or the training data is poor quality. Keep in mind that sometimes it is perfectly fine if the model loses some capability in areas it won’t need – like a customer service bot being terrible at coding – but it’s not like you can pick and choose what to break so it comes back to trial and error.
Enter Auto-SFT
Trial and error means starting somewhere more or less random, testing the results, tweaking the approach a bit in the direction of the desired outcome, test again, and repeat. It’s tedious and time consuming, and there is no reason a person should do this “by hand”. Auto-SFT does exactly that.
Auto-SFT is not just completely random variables thrown at the wall. As you progress through the loop, it tries to build on what works and disregard what doesn’t, zeroing in on optimization rather than taking random stabs in the dark.
Each cycle Auto-SFT settles on a set of parameters and runs a limited training of just a few steps. Then it runs a pre-determined number of inference tests (default is 4). Each generated response is compared to the response to the same prompt by the base model without fine-tuning, as well as the response found in the training data set.
Using another LLM as judge, the inference results from each cycle are scored from 1-10 and the average score of all the inference samples become the score for that cycle. If it is the highest score yet, it becomes the leader. If it matches the existing leader, the two sets are judged against each other by the LLM judge, and a winner is picked.
Note: If you use a bad model as the LLM judge, you will end up with bad results. I recommend testing a few of your favorite models on the same configuration, then choosing the one you like the best.
The user configures the loop. You set maximum time or steps per cycle, and early stopping if quality doesn’t improve after X cycles. And you can set a minimum score, and the training and uploading will only continue if the search was able to find a combination that scored this much or higher (default is 8). Remember that this is final score is the average of the inference samples at the final step of the training.
Hugging Face Integration
I am big fan of Hugging Face and if you’ve seen my profile and portfolio there, it should come as no surprise that it is fully integrated into Auto-SFT. This includes pulling models and data sets directly from there, but also uploading your fine-tuned models.
If you set a minimum score and enabled the feature, Auto-SFT will continue from parameter search straight into the actual fine-tuning of your base model of choice. Even if it did not reach the minimum score required, you can override it manually and have it continue using the highest scoring result (regardless of what its actual score was). Use this if you like the inference samples so much, you think it’s worth it.
Once the fine-tuning is complete, Auto-SFT merges your LoRA weights into the base model and uploads the new model to your Huggingface profile. You need to set up a token key once, and it will work forever.
Beyond uploading the merged model, Auto-SFT will also convert your model to GGUF and create a separate GGUF-repo for these. The configuration section lets you choose the quantization levels you wish to include.
I hope this tool is as helpful to others as it has become to me. I built a prototype of this, inspired by Andrej Karpathy’s AutoResearch project, and kept tweaking it for several weeks until I had a solid Auto-SFT version 1.0.