I have built a tool that will attempt to find the optimal settings for fine-tuning LLMs on a case-by-case basis. It analyzes the data set and the desired outcome, starts with a random set of parameters, and through short, controlled test training sessions, it gradually finds the place where the data, prompt, and model get the best results. In this post, I will go a little deeper into how it works, and why I built it.
TL;DR: Auto-SFT finds optimal parameter settings for you, it's open source with a MIT license and can be downloaded from the Auto-SFT github repo.
Why Fine-Tune a Model?
The two most common reasons for fine-tuning a LLM are making it follow a specific format or respond in a specific style. In post-training the AI model doesn’t really learn a lot of new knowledge, but the training data tells it what to prioritize and how to present it. But when I say that it doesn’t “really” learn, that is because it does pick up some things. Whether it’s actual new knowledge, or simply a better understanding of how to present the knowledge it already had doesn’t really matter in the end, as long as the user gets the desired type of response.
LoRA fine-tuning is one of the most popular approaches to post-training of AI models, because it only works with part of the base model weights while the rest remains frozen. This allows enthusiasts and compute-poor labs to fine-tune larger models than what they could do if they had to load the entire model into memory.
The main risk when fine-tuning AI models is “catastrophic forgetting”, where some of the existing knowledge or logic is broken during the fine-tuning process. This often happens when the settings are too aggressive or the training data is poor quality. Keep in mind that sometimes it is perfectly fine if the model loses some capability in areas it won’t need – like a customer service bot being terrible at coding – but it’s not like you can pick and choose what to break so it comes back to trial and error.
Enter Auto-SFT
Trial and error means starting somewhere more or less random, testing the results, tweaking the approach a bit in the direction of the desired outcome, test again, and repeat. It’s tedious and time consuming, and there is no reason a person should do this “by hand”. Auto-SFT does exactly that.
Auto-SFT is not just completely random variables thrown at the wall. As you progress through the loop, it tries to build on what works and disregard what doesn’t, zeroing in on optimization rather than taking random stabs in the dark.
Each cycle Auto-SFT settles on a set of parameters and runs a limited training of just a few steps. Then it runs a pre-determined number of inference tests (default is 4). Each generated response is compared to the response to the same prompt by the base model without fine-tuning, as well as the response found in the training data set.
Using another LLM as judge, the inference results from each cycle are scored from 1-10 and the average score of all the inference samples become the score for that cycle. If it is the highest score yet, it becomes the leader. If it matches the existing leader, the two sets are judged against each other by the LLM judge, and a winner is picked.
Note: If you use a bad model as the LLM judge, you will end up with bad results. I recommend testing a few of your favorite models on the same configuration, then choosing the one you like the best.
The user configures the loop. You set maximum time or steps per cycle, and early stopping if quality doesn’t improve after X cycles. And you can set a minimum score, and the training and uploading will only continue if the search was able to find a combination that scored this much or higher (default is 8). Remember that this is final score is the average of the inference samples at the final step of the training.
Hugging Face Integration
I am big fan of Hugging Face and if you’ve seen my profile and portfolio there, it should come as no surprise that it is fully integrated into Auto-SFT. This includes pulling models and data sets directly from there, but also uploading your fine-tuned models.
If you set a minimum score and enabled the feature, Auto-SFT will continue from parameter search straight into the actual fine-tuning of your base model of choice. Even if it did not reach the minimum score required, you can override it manually and have it continue using the highest scoring result (regardless of what its actual score was). Use this if you like the inference samples so much, you think it’s worth it.
Once the fine-tuning is complete, Auto-SFT merges your LoRA weights into the base model and uploads the new model to your Huggingface profile. You need to set up a token key once, and it will work forever.
Beyond uploading the merged model, Auto-SFT will also convert your model to GGUF and create a separate GGUF-repo for these. The configuration section lets you choose the quantization levels you wish to include.
I hope this tool is as helpful to others as it has become to me. I built a prototype of this, inspired by Andrej Karpathy’s AutoResearch project, and kept tweaking it for several weeks until I had a solid Auto-SFT version 1.0.
In the world of game development the term AI has been around for decades, though it was not referring to chatbots and neural networks but rather things like finite state machines and behavior trees. Both of those are fantastic tools for game development and everyone should use them, but this post is not about either. This is about using transformer networks as a way to dynamically control the level of gameplay difficulty. Instead of hard coded values, this is based on individual, (almost) real-time player behavior!
Instead of an old school difficulty slider that locks in specific game parameters to make the game harder or easier all round, a neural network can learn to react and adjust the difficulty moment to moment and without a slider anywhere in the game menus.
This is not theory, this is fairly straightforward to build and I have already done it. It is not a massive LLM we are creating after all, just a tiny model with minimal memory thanks to its transformer nature. Just enough to do this one thing. Data comes in, the network processes it all and chooses an action.
Building and Training the Model
To build a neural network like this, you start with player statistics, such as:
movement direction
position
recent kills
lives left
time since last death
total time played
number of enemies on-screen
Plus any other metrics that together paint a picture of the current state of the game. These data points become the inputs for the model. You can experiment with hidden layers and nodes but you really don’t need a lot.
Next a set of actions that this AI game master should be able to take are defined, and those become the outputs. The exact actions will depend on your project, but to use this demo as an example, output actions include:
spawning different enemy types and formations
increase or decrease the overall pressure (spawn rates and budget)
temporarily back off to give the player a moment to breathe
launching a massive wave of attack
If you’re building your own, you will need to define these actions ahead of time, and make sure your game code has all the necessary functions wired in to trigger when the neural network sends the signal to do so.
The hardest part is training this game master model, and getting enough data to do so. The demo game has a built-in data snapshot function that also takes a screenshot. To turn this into training data, I had to label it and evaluate every entry. It is not impossible to do, just tedious.
For this demo, I hand-labeled over 1000 entries and trained the AI on them with early stopping to avoid overfitting. I didn’t write it down, but I think it trained for around 20 epochs. The actual training took very little time.
Afterwards, the fully trained model can be converted to a format that is friendly to your game engine of choice. Godot does not natively support neural network formats like ONNX, but because the model is so small, converting the whole thing to a format that works is not a big deal.
Introducing Space Base Bomb Run
To demonstrate the idea, I put together a small tech demo. It’s a vertically scrolling shoot-em-up game where the waves of attacking enemies are controlled by exactly this kind of neural network gamemaster. Every 4 seconds, it takes a snapshot of what’s going on in the game, and takes an action accordingly. It can choose different enemies and attack patterns, as well as tweak the overall spawn rates and budget (higher budget = more elite enemies).
You can toggle visualization of the network on and off while playing (press v) to see it in action, though be warned that is does cover part of the screen if you do so.
I call the demo “Space Base Bomb Run” (SBBR) and I put the game, including the source code / Godot project on github, as well as all the training scripts and tools used to build and train the AI. If you’re curious, you can train your own game master AI model from scratch, using SBBR and the labeling and training tools included.
Do note that SBBR is a tech demo and not at all a polished game meant for release. That said, if you play it you will definitely notice that the game gets harder as you progress – and if you’re struggling to stay alive, the game will actually back off a bit, at least for a moment.
I think this way of using tiny transformer models is both exciting and fun – I like the emergence and adaptability of this approach and can easily see it adapted to other parts of a game than just the difficulty.
I see business owners, small teams and decision makers who want to integrate AI better into their work and day to day operations. Not surprisingly, there is some confusion about what solution is best, especially if you’ve made the decision to use a locally hosted, private AI model. So, I would like to show some examples of what two kinds of fine-tuning can do.
If you want to use a language model for any kind of specialized purpose, where niche or special domain knowledge may be involved, you have the option of relying on vector databases and document searches (RAG), fine-tuning a model, or a combination of both. The first solution is basically the AI looking through the available knowledge, pulling bits and pieces, and using those pieces as supplemental context to the answer it is going to generate. Fine-tuning however, involves actually teaching the model itself some new things. Hopefully this post can shed some light on what the best option would be for your use case.
Let me first say, that the most important aspect of fine-tuning is the data you use to train on. You have to be very deliberate about how you construct and build your data set to get the best results. Garbage in, garbage out, they say, so make sure the data represents actual use cases, is properly cleaned, and all that good stuff before you start any kind of fine-tuning.
Assuming that you have the data you need, the next big question is: what kind of fine-tuning are you going to need? And here, there are two main approaches: low-rank (LoRA) and full fine-tuning.
LoRA is great for gentle steering, emphasis on specific aspects of existing knowledge, and tone, but it does not pick up a lot of new knowledge. Full fine-tuning on the other hand, lets you add new, specific knowledge directly into the AI model because you are training the entire model. Why not just go for a full fine-tuning every time then? Because it comes with the risk of causing the model to collapse and forget existing training, including how to do things like use tools correctly to search documents. Full fine-tuning also requires more computing power (VRAM) to pull off depending on the size of the base model you are working with.
With either approach, there may be some trial and error involved as well, depending greatly on the type and amount of data, output requirements, model architecture and more, but that lies beyond the scope of this post.
The setup
For this experiment, I want to show the difference in output between a basic model and the two kinds of fine-tuning mentioned above. I am using a data set focused on science-fiction and fantasy. This data set was created using a number of wikipedia-pages as source for question and answer pairs to be generated and is, like everything else in this post, available to download for free via my Huggingface profile.
I chose a Llama 3.2, 1-billion parameter model as the starting point, and trained it twice on the same data set for exactly 1 epoch; meaning that the model saw each entry exactly once. First I used Low-Rank (LoRA) fine-tuning where only a small part of the network is tuned. For the second version, I did a full fine-tuning where every single parameter in the model was being adjusted.
The only difference between the two fine-tuning sessions was the learning rate (LR), which represents how much knowledge is absorbed per training step. As is common practice, the learning rate was a lot higher on the LoRA fine-tuning (2e-4) than on the full one (2e-5). The lower LR is to reduce the chance of catastrophic forgetting during the full fine-tuning.
Test Questions
The models were tested without any system prompt at all. I wrote out 6 test questions and each one was served to the model being tested without any additional context. For this post, I will share the 3 most interesting ones, in terms of the difference between the three models tested. The questions featured are:
What is Terry Pratchett most famous for?
What are 3 novels, I should read this summer?
Give me an example of a well-written character arc.
Note that of these 3 questions, only the first one directly references genre-specific data, namely the author Terry Pratchett. The others were deliberately left open to see if the model would drift towards answers rooted in the training data. Below are the answers given by all 3 models, along with a few thoughts on the differences that stood out to me.
Tricky Terry
On the question “What is Terry Pratchett most famous for?” none of the models gave a perfect answer, which makes it a perfect example to show. All 3 answers have factual errors related to the works of mr. Pratchett, which is important because: no matter which approach you choose, AI can still fail. It may fail a bit closer to target, but if your field is legal or medicine for example, “close enough” may not cut it. Proceed with caution!
Basic Llama 3.2 1B Instruct model’s response
“Terry Pratchett was a British author best known for his fantasy series “”Discworld””, which combines humor, satire, and social commentary. He is particularly famous for creating:
The Discworld Series: A collection of nine novels that are set on the flat disc balanced on the backs of four giant turtles, who in turn are floating through space.
Characterization of satire and absurdity: Pratchett’s work often uses humor to critique contemporary issues, such as politics, society, culture, and morality, with a playful twist.
Memorable characters: The Discworld series features numerous iconic characters, including:
Rincewind (the inept wizard)
Death (the personification of death who travels at high speed)
Granny Weatherwax (a witch who is also a strong and independent woman)
A’Tuin (the god of the sun)
Wordplay and linguistic humor: Pratchett was known for his clever use of language, puns, and wordplay, which added to the comedic effect of his works.
Satire of authority and bureaucracy: Many of his novels poke fun at government, institutions, and social norms, often using absurd situations and characters as a way to comment on the human condition.
Some notable books in the Discworld series that are often cited as Pratchett’s most famous works include:
“”The Colour of Magic”” (1983)
“”The Light Fantastic”” (1986)
“”Mort”” (1987)
“”Small Gods”” (1992)
“”Wyrd Sisters”” (1988)
Terry Pratchett passed away in 2015, but his Discworld series remains a beloved and iconic part of fantasy literature.”
LoRA Fine-Tuned model’s response
Terry Pratchett was a British author best known for his Discworld series of fantasy novels, which follows the misadventures of an ensemble cast of characters in the fictional world of Ankh-Morpork. His works are renowned for their wit, humor, and satire.
Some of his most notable contributions to literature include:
*Â Â **Discworld**: Pratchett’s Discworld series has become a beloved part of popular culture. *Â Â The series explores themes such as politics, power struggles, social commentary, and human nature through the lens of comedic fantasy. *Â Â Works like “The Colour of Magic” (1971) and “Guards! Guards!” (1989) showcase Pratchett’s unique writing style and ability to balance humor with deeper exploration.
Full Fine-tuned model’s response
“Terry Pratchett was a British author best known for his Discworld series, which consists of fantasy novels that parody and satirize various aspects of life. His works often feature:
Discworld: A flat, disc-shaped world created by the wizard Rincewind, which serves as the main setting for many of Pratchett’s stories.
Parodying mythology: Pratchett frequently draws inspiration from mythology and folklore to create humorous, irreverent takes on classic mythological themes.
Some notable examples include:
The Colour of Magic
The Light Fantastic
Equal Rites
Pratchett’s Discworld series has been widely acclaimed for its unique blend of humor, satire, and imagination. He was known for his witty writing style and ability to create memorable characters like Rincewind and Death.
While Pratchett also wrote other works, such as Mort by Stephen Baxter (a novella set in the same universe), it’s his Discworld series that has made him most famous among fans of fantasy literature.”
Summer Read Pivot
When asked “What are 3 novels, I should read this summer?” the standard instruct model mentions “The Power” as the most relevant suggestion, setting a great point of comparison to the two fine-tunes which (spoiler) both deliver better responses in this category. I think this shows that without any kind of fine-tuning, it all comes back to the prompt. The LoRA suggested the book “The Night Circus“, which is a bit better tied to the data set, contextually. Meanwhile the full fine-tune suggests both of those and “The City & The City” which is arguably more of a deep-dive title, relevant to the data set.
Basic Llama 3.2 1B Instruct model’s response
“3 recommendations for novels to enjoy this summer.
“”The Seven Husbands of Evelyn Hugo”” by Taylor Jenkins Reid: A captivating and heartwarming story about a reclusive Hollywood star who recounts her seven marriages to a young journalist.
“”The Nightingale”” by Kristin Hannah: An emotional and suspenseful novel set in France during World War II, exploring the lives of two sisters as they struggle with the war, sacrifice, and survival.
“”The Power”” by Naomi Alderman: A thought-provoking speculative fiction novel that explores a world where women suddenly develop the ability to release electrical jolts from their fingertips, allowing them to dominate and control society.
All three novels offer engaging stories with relatable characters, making them perfect for a summer read.”
LoRA Fine-Tuned model’s response
Here are three novel recommendations:
1. “The Night Circus” by Erin Morgenstern (2011) Â Â A magical tale about a competition between two young magicians who are trained from childhood to compete against each other.
2. “The Last Romantics” by Tara Conklin   A sweeping family saga that explores themes of love, loss, and legacy through the eyes of four siblings growing up in the 1970s.
3. “The Song of Achilles” by Madeline Miller (2011) Â Â A beautifully written novel that reimagines Homer’s Iliad from the perspective of Achilles’ lover Patroclus, exploring themes of love, war, and identity.
Full Fine-Tuned model’s response
“Here are three highly-recommended novels across various genres that you might enjoy:
The Night Circus by Erin Morgenstern: A magical and imaginative tale about a competition between two young magicians who become enchanted by each other.
The Power by Naomi Alderman: A thought-provoking novel that explores what would happen if women suddenly developed the ability to release electrical jolts from their fingertips, allowing them to dominate and control society.
These novels offer unique perspectives on human nature, society, and individual agency.”
The Force is Strong in This One
When “Give me an example of a well-written character arc” is requested, both the basic instruct and the LoRA fine-tuned models invented new characters, embarking on a creative writing exercise. But in the fully fine-tuned model we see something completely different: instead of inventing a character, the model uses the example of Luke Skywalker to describe what a character arc is. This is significant because we see a change, not just in the output generated but in the behavior of the model itself.
Basic Llama 3.2 1B Instruct model’s response
“Here is an example of a well-written character arc:
Original Character: Meet Alex, a young professional in her late 20s who has just moved to the city for a fresh start after graduating from college. She’s determined, ambitious, and always striving to prove herself.
Inciting Incident: Alex lands a high-paying job at a prestigious marketing firm, but soon discovers that she’s working long hours with little creativity or autonomy. Feeling undervalued and unfulfilled, she starts to question her decision and the true nature of her work.
Turning Point: While working late one night, Alex has a heart-to-heart conversation with an old friend, Rachel, who was also a college classmate. Rachel shares her struggles as a single mother trying to make ends meet while pursuing higher education. This realization sparks a deep sense of empathy and compassion in Alex, making her question whether she’s truly happy with her current career path.
Setback: Alex tries to shake off the doubts by taking on more responsibilities at work and trying to prove herself to her boss. However, this newfound focus leads to burnout and exhaustion, causing her to withdraw from social life and relationships with friends and family.
Midpoint: As Alex’s depression and anxiety grow, she reaches out for help from a therapist who helps her explore the underlying causes of her feelings. Through therapy, she begins to understand that her true passion lies in photography, not marketing. She starts taking photos in her free time, which brings her a sense of joy and fulfillment.
Climax: As Alex’s confidence grows, she decides to take a leap of faith and leave her job to pursue photography full-time. This decision is met with both excitement and fear from her friends and family, who worry about her financial stability and social isolation.
Resolution: Alex struggles to balance her passion for photography with the practicalities of making it a career. She faces setbacks and rejections, but through hard work and determination, she finally lands her first client and starts building a successful business.
Character Growth:
Alex has become more empathetic and compassionate towards others, particularly those in need or struggling to make ends meet.
She’s developed greater self-awareness and emotional intelligence, learning to recognize and manage her own feelings and emotions.
Her passion for photography has transformed from a job into a true calling, reflecting her authentic self and values.
This character arc illustrates several key elements:
Internal conflict: Alex faces an internal struggle between pursuing a career that doesn’t fulfill her or finding happiness and fulfillment through another path.
Escalating stakes: As Alex’s doubts grow, she faces increasing pressure to prove herself and find success in her chosen career.
Transformation: Through the turning point conversation with Rachel, Alex experiences a significant shift in perspective, leading to greater self-awareness and confidence.
Rite of passage: Alex’s decision to leave her job and pursue photography marks a major milestone in her life, symbolizing growth and self-discovery.
Overall, this character arc demonstrates how a well-written story can create a compelling and memorable character journey, showcasing the transformative power of empathy, self-awareness, and determination.”
LoRA Fine-Tuned model’s response
“Here is an example:
Character Arc: From Naive to Confident
Initial State: Emily, a young and idealistic journalist, arrives in Tokyo with a clear vision for her story. She’s eager to make a difference and prove herself as a capable reporter.
Inciting Incident: During a meeting at the newspaper office, she meets her editor, who challenges her on the ethics of reporting sensitive information. This encounter sparks a moment of doubt and uncertainty within Emily.
Rising Action: As Emily delves deeper into the story, she realizes that her initial assumptions were incorrect. She begins to see the complexities of human nature and the blurred lines between right and wrong. Her editor’s feedback serves as a catalyst for this growth, pushing Emily out of her comfort zone.
Climax: During an interview with a source who provides valuable insights into the story, Emily experiences a moment of profound vulnerability. She begins to confront her own biases and limitations, allowing her to grow as a journalist and as a person.
Falling Action: As Emily works on the article, she faces resistance from colleagues who question her methods or dismiss her concerns. This external pressure forces Emily to reevaluate her approach and consider alternative perspectives.
Resolution: Through this process of self-discovery, Emily develops a more nuanced understanding of journalism ethics and human nature. Her work is met with praise, but also criticism from some quarters. This mixed response serves as a testament to the complexities of storytelling and the importance of considering multiple viewpoints.”
Full Fine-Tuned model’s response
“One classic example is the transformation of Luke Skywalker from a young farm boy to a confident Jedi Knight.
Initially, we see Luke as a naive and impressionable individual who is drawn into the world of Star Wars through his friendship with Han Solo. However, as he trains with Yoda on Dagobah and learns about the Force, we witness Luke’s growth from an uncertain teenager to a heroic young adult.
One key aspect of Luke’s character arc is his development as a leader. Initially, he struggles to assert himself in battles against the Empire, but through his experiences, he gains confidence and becomes more effective at guiding others.”
