I haven't written for a long time.AI PaintingBecause there is no special update.FLUXAfter a little research on the new model, I found that it is still very playable and has several highlights that are very attractive.
So I wrote an article specifically to talk about FLUX.Team background, project features, installation and configuration, and usage.
Team Background
When it comes to AI painting, we can’t avoid Stable Diffusion, the most well-known, powerful and popular open source AI painting model/software.
Stable Diffusion is an open source deep learning model for generating high-quality images. It was developed by several companies and research institutions including Stability AI and Runway and released in 2022. Stable Diffusion has received widespread attention for its ability to generate realistic images.
The FLUX team comes from Stability AI, which gives people the impression that it has more technology than personality.
The FLUX team is called Black Forest Labs. Judging from the name alone, it is more personalized and mysterious.
Project Highlights
SD has been updated many times since its initial breakthrough, but the highlights are getting fewer and fewer. The two obvious problems are the artists and the text. FLUX has made great improvements in both aspects.
Let's take a look at the specific points of FLUX.
Improved image quality
Produce stunning visuals at higher resolutions.
This can only be understood but not explained. Regarding the quality of the pictures, I often feel that if the training materials are good, the quality will be good.
Super strong text embedding capability
Adding text to pictures has become super easy.
First, it can display text correctly, and second, it can have a good display effect. In addition, it can handle a large amount of text.
In this way, you can make graphic covers and posters with just your mouth. Currently, the closed-source and paid Chatha GPT DALL·E cannot achieve this level.
The characters are more realistic and natural
Regarding the characters, the most obvious thing is that the hands drawn finally look much more normal.
Of course, it's not just the hands, the whole person looks more human, without many strange facial and limb distortions.
As an AI model, it is crucial to correctly understand the human appearance.
Generate more accurate content
Get a more accurate and relevant image based on your input.
Early SD models required a large number of positive and negative keywords to control the generated images, but the results were often unsatisfactory. Sometimes they were completely inconsistent and could not generate the content you wanted.
FLUX has made great progress in this regard, such as the "a tiny astronaut hatching from an egg on the moon", with just one short sentence, he drew something that is impossible to exist in the real world. The moon, the egg, and the little astronaut are all there, and they are integrated quite naturally.
Of course, all the advantages are no match for one advantage, which isOpen Source.
Using the fully open source comfyUI, it can be completely free and run offline.
Everyone knows that Midjourney is very powerful, but it costs hundreds of dollars a month. In comparison, even if it has some weaknesses, FLUX is very attractive.
Comparing the drawing ability of 20$'s ChatGPT, FLUX is even better.
Model Classification
There are three versions of the model released this time, namely pro, dev, and schnell.
The specific model is introduced as follows:
- FLUX.1 [pro]: Closed source version, can only be called through API.
- FLUX.1 [dev]: Development board model, open source in a non-commercial license, suitable for developers and individual players.
- FLUX.1 [schnell]: A streamlined version of the basic model that runs fast and uses the Apache 2 license for commercial use.
The most powerful version we can play is the DEV version. The DEV version is also very powerful, with the number of parameters reaching 12B (12 billion parameters) and the model size reaching 23G.
In order to make low-end devices run, the FP8 precision version of DEV has appeared on the Internet!
The dividing line between the original DEV version and the FP8 version is roughly at 16G of video memory.
Having said so much, how do we play it?
Now let’s talk about how to use this model!
Installation and Configuration
FLUX has just been released, and the peripherals are not yet complete. SDWebUI seems to have not caught up yet, but fortunately ComfyUI has been perfectly adapted. So the installation and configuration step is actually to install ComfyUI.
ComfyUI is a graphical user interface (GUI) for generating and manipulating images, aiming to provide a user-friendly way to create and edit images. Its biggest feature is the node-based design, which is concise, beautiful, and powerful, but of course the entry threshold is slightly higher.
In addition, its support for AI painting is very timely and comprehensive. It currently fully supports SD1.x, SD2.x, SDXL, Stable Video Diffusion, Stable Cascade, SD3 and Stable Audio, Flux and other graphic and video models.
Speaking of installation, another feature of ComfyUI is its simple installation. The project homepage provides a one-click run package, which includes the latest source code and the required environment for running. You only need to download and unzip it to run it, and it will automatically optimize for devices with different video memory.
After downloading, you will get a.7zCompressed file
Install a 7-zip decompression software, then right-click on the compressed file and decompress it to the current folder.
After the decompression is complete, open the folder, which should look like this:
double clickrun_nvidia_gpu.batYou can run the software.
Tutorial
The tutorial is basically the same as ComfyUI's tutorial. As mentioned earlier, ComfyUI has a relatively high entry barrier, but it has a design called workflow that can greatly lower the barrier. It allows you to do it even if you don't understand it.
To run FLUX with ComfyUI, you need to prepare the FLUX model, CLIP model, and VAE model in advance. I will put the models and software in the network disk, see the end of the article.
FLUX main model configuration
There are three types of FLUX models introduced above. Currently, the open source models are dev and schnell. We choose the dev version here. Considering that the original dev model is too large, we use the low-profile fp8 version.
So the final model used isflux1-dev-fp8.safetensors
This model needs to be placed under the unet folder of comfyui.
The FLUX model path is:ComfyUI\models\unet
CLIP and VAE configuration
CLIP and VAE are called "translators" and "sketch artists". These can help us make better works, so they should also be configured.
The model name and the folder where it is placed are shown in the figure below.
The path of CLIP is: ComfyUI\models\clip
The path of VAE is:ComfyUI\models\vae
workflow loading
Once all the models are in place, you can start running.
double clickrun_nvidia_gpu.batLaunch the software.
After successful startup, the browser will open automatically and an interface will be displayed on the browser with a default workflow.
Drag our own workflow file to the page and the software will automatically load the workflow.
The workflow file ends with json, and png images can also contain workflow information. I will also put these in the network disk, so please check them out.
One-click generation
When everything is ready, click on the "Queue Prompt” One-click to draw.
After clicking, the software will first load the model, run step by step according to the configuration, and the final generated image will be displayed in the rightmost area.
So far, we have successfully used FLUX to create a picture. Next, we can adjust the effect parameters appropriately to create more interesting works.
The following focuses on the meaning of several nodes.
Introduction to ComfyUI Node
There are a lot of nodes used here, which may seem a bit overwhelming at first glance, but it is actually quite simple once you understand it. In this workflow, we only need to focus on the five nodes in the figure below.
① Load the FLUX model
There are two parameters here: model name and model type. Click the left and right arrows to switch. For specific values, refer to the figure above. If there is no model under the folder, you cannot select it here. So you have to place the model according to the above steps.
② Load CLIP model
There are two clips that can be loaded here, of three types: flux, sdxl, and sd3. Today we will definitely use flux. Model selection is the same as above.
③ Load VAE model
The flux_ae.sft model is loaded here, and the model processing is the same as above.
④ Seed Generator
Here one is the value option and the other is the generation method. The value is a long string of numbers that will be automatically generated.
The methods are divided into fixed, increment, decrement, and randomize.
If it is fixed, the value remains unchanged, allowing the image to be adjusted on the same basis.
If it is random, each time will be a little different, and different pictures can be drawn under the same parameters.
⑤ Length and width batch
Here you can set the length, width, and number of images to be generated at a time. By adjusting the ratio, you can generate images of different sizes and ratios, such as horizontal, vertical, and square. Adjust the batch size to generate in batches.
⑥ Prompt words
The text-to-image model generates images through text descriptions. So this option is definitely the most important part. In the early models, we needed to input a large number of positive and negative words to control the images.
However, as the model continues to improve, negative words have been built-in or are not so important, and positive words can become simpler and simpler. You can draw whatever you want, and the so-called prompt word engineering... seems to be a false proposition.
In the end, it will tend more and more towards natural language, your level of professionalism will definitely be determined by its level of professionalism.
With the above configuration and instructions, you should be able to use FLUX normally. The most important thing is to prepare a graphics card with large video memory.
After a simple test, it takes several minutes to generate an image on RTX3060 12G. Because of the small video memory, ComfyUI will use the low video memory model to load the model, resulting in a very slow speed. On RTX3090, it only takes about 20 seconds to generate an image, which is more efficient than expected.
FLUX software, models, workflow:
https://pan.baidu.com/s/1PDVrJHuIprzjrJXf2y2tJA?pwd=tony
Extraction code: tony