The purpose
I will now add a VAE to the Stable Diffusion web UI running on my local machine.
What is VAE
According to Google’s search AI, it is as follows.
A VAE (Variational AutoEncoder) is a deep learning model used in various AI fields, such as image generation and anomaly detection. It is a type of unsupervised learning that is based on the autoencoder model for data compression and reconstruction, but is uniquely characterized by the introduction of a probability distribution into its latent variables.
That explanation might be a bit confusing, but for Stable Diffusion, you can think of a VAE as a plugin that simply makes the output images look better. That’s a good enough way to think about it.
Environment
stable-diffusion-webui-forge
(Other Stable Diffusion web UI versions can install VAEs using the same method.)
Download VAE
Download
I will download the VAE.
Rather than searching for a VAE on its own, you can often find links to recommended VAEs on a model’s download page.
Here is a link to the VAE that I use.
General-purpose VAE
This VAE is from Stability AI (the developer of Stable Diffusion). It can be used with both photorealistic and anime-style models.
You can download it from the following page under the file name “vae-ft-mse-840000-ema-pruned.ckpt“.
VAE for anime images
This VAE is for anime images. You can download it from the following page under the file name “kl-f8-anime.ckpt.”
Install VAE
Copy the downloaded file to the following folder.
installation folder\webui\models\VAEDepending on how Stable Diffusion was installed, you might not have a webui folder. In that case, the path would be “installation folder\models\VAE.”
Once the copy is complete, click the button at the bottom of the GUI.
Once the process is complete, you will be able to select the downloaded VAE from the dropdown menu below the button’s image.
The dropdown menu at the top doesn’t only display VAEs.
VAEs have compatibility with different models, so just because you’ve set a VAE doesn’t guarantee a good image will be generated (personal preference is also a factor).
Depending on which version of Stable Diffusion you are using, it may display “VAE” instead of “VAE/TextEncoder.”
Image generation using VAE
From the VAE/Text Encoder menu above, select the VAE you want to use. Then, just like with a normal image generation, set your prompts and click Generate.
Result
Below is a comparison of using and not using vae-ft-mse-840000-ema-pruned.ckpt. The image on the left was generated without the VAE, and the image on the right was generated with it.
You can see that the image on the right is much clearer.
comment