AI

Tady AI

LoRA training in Fluxgym

I’ve never trained LoRA before, but I know a bit or two about a gym, so… I don’t mind chaperoning some LoRA or other ladies to the gym for a bit of fun.
I also recently made a couple of interesting pictures in Midjourney with the combination of style reference and my own personalization. Wouldn’t it be fun to try recreating them in Flux?

Luckily for me, there is a brand new fluxgym in pinokio.computer. Installing is a standard couple of clicks, as usual. The most important rule during installation is: if it works, leave it. Don’t try to improve it, don’t rush, don’t touch it! It’s perfect!
Your patience is rewarded by a running Gradio UI. I have my pictures ready, I mean, I have too many pictures, I tried with 16 pictures first, but it would take ages on my RTX 3060 with 12 GB VRAM, so I cut it down to 7 pictures.
I named the Lora with a non-English word, a trigger word is not a word at all, everything else stayed pretty much in default settings.
After adding the captions with Florence, I just tweaked it a little bit, I didn’t want any stained glass windows there.
Time for a training. So for 7 pictures, originally 1024x1024px, trained in 512px, it was set for 1230 steps and on a nVidia RTX 3060 with 12GB VRAM it took less than 3 hours. The VRAM usage stayed at 80% or so.
At least I think so, I was sleeping, when my LoRA was at the gym. I know how to delegate the task.

It produced an 18MB file with LoRA, and I’m gonna use it right now, because I’m done here, right?
That’s what I thought.
Three hours later, with no styl-ish picture generated yet, I made a couple of notes for you, so you can save your three hours of life. You can thank me later, you’ll have an extra three hours to do so.

If you gonna use Forge, handily ready in pinokio.computer too, update it first and save yourself a lot of troubles.
These settings of the first row are crucial. If you don’t have bnb-nf4 (fp16 LoRA) as an option in your Diffusion in Low Bits, it probably won’t work.
If you are doing character or faces, no pun intended, you might be okay with a simple prompt. The foolish idea of generating leaves with the leaf LoRA means, it’s not too stylish.
I took my original Midjourney picture and interrogated it, took the style part of the prompt… and here we are.
Another way is to increase LoRA weight. I tried 2 and it was even better, with 3 it was more artistic but less the picture I wanted.

To make a picture in Forge with Flux-schnell-bnb-nf4 in 4 steps is taking about 20 something second per picture, Flux-dev-bnb-nf4 in 20 steps takes about 1:30 minutes.
Thanks to @coctailpeanut for the gym experience.