{"id":864,"date":"2023-02-15T12:20:43","date_gmt":"2023-02-15T18:20:43","guid":{"rendered":"https:\/\/promptphantom.com\/?p=864"},"modified":"2023-02-15T14:50:05","modified_gmt":"2023-02-15T20:50:05","slug":"what-is-a-stable-diffusion-model-and-which-should-i-use","status":"publish","type":"post","link":"https:\/\/promptphantom.com\/what-is-a-stable-diffusion-model-and-which-should-i-use\/","title":{"rendered":"What is a Stable Diffusion Model and Which Should I Use?"},"content":{"rendered":"<h3 class=\"wp-block-heading\" id=\"a-faq-guide-on-custom-ai-art-models\"><em>A FAQ Guide on Custom AI Art Models<\/em><\/h3>\n\n\n<p>Have you recently gotten sucked down the rabbit hole of AI art generation? Perhaps you heard that Stable Diffusion is capable of being custom trained but you didn\u2019t understand how. Maybe you\u2019ve seen people talking about different models of Stable Diffusion but didn\u2019t know what they meant.<\/p>\n\n\n\n<p>In this guide, I\u2019ll teach you what a Stable Diffusion model is, how they are made, and where to find them. Don\u2019t forget your pocket watch, because we\u2019re going to Wonderland.<\/p>\n\n\n<div role=\"navigation\" aria-label=\"Table of Contents\" class=\"simpletoc wp-block-simpletoc-toc\"><h2 style=\"margin: 0;\"><button type=\"button\" aria-expanded=\"false\" aria-controls=\"simpletoc-content-container\" class=\"simpletoc-collapsible\">Table of Contents<span class=\"simpletoc-icon\" aria-hidden=\"true\"><\/span><\/button><\/h2><div id=\"simpletoc-content-container\" class=\"simpletoc-content\"><ul class=\"simpletoc-list\">\n\n<li><a href=\"#quick-disclaimer\">Quick Disclaimer<\/a>\n\n<\/li>\n<li><a href=\"#how-does-stable-diffusion-work\">How Does Stable Diffusion Work?<\/a>\n\n<\/li>\n<li><a href=\"#what-is-a-model-in-stable-diffusion\">What is a Model in Stable Diffusion?<\/a>\n\n<\/li>\n<li><a href=\"#what-is-a-stable-diffusion-checkpoint-or-ckpt-file\">What is a Stable Diffusion Checkpoint or CKPT File?<\/a>\n\n<\/li>\n<li><a href=\"#how-are-stable-diffusion-models-made\">How are Stable Diffusion Models Made?<\/a>\n\n\n<ul><li>\n<a href=\"#the-tldr-version-of-model-training\">The TL;DR Version of Model Training<\/a>\n\n<\/li>\n<\/ul>\n<li><a href=\"#where-can-i-find-models\">Where Can I Find Models?<\/a>\n\n\n<ul><li>\n<a href=\"#warnings-before-you-download-any-models\">Warnings Before You Download Any Models!<\/a>\n\n<\/li>\n<\/ul>\n<li><a href=\"#what-are-the-best-models-to-use\">What are the Best Models to Use?<\/a>\n\n<\/li>\n<li><a href=\"#conclusion\">Conclusion<\/a>\n<\/li><\/ul><\/div><\/div>\n\n<h1 class=\"wp-block-heading\" id=\"quick-disclaimer\">Quick Disclaimer<\/h1>\n\n\n<p><em>In this guide, I will try to explain Stable Diffusion for beginners who have little to no knowledge about Machine Learning. To do that, I will use some analogies and simplifications. These descriptions are meant to allow non-data-scientist people to understand the concepts, but they won\u2019t necessarily be technically-accurate. This is not meant as a scientific explanation for how Latent Diffusion models or PyTorch libraries work on a technical level.<\/em><\/p>\n\n\n<h1 class=\"wp-block-heading\" id=\"how-does-stable-diffusion-work\">How Does Stable Diffusion Work?<\/h1>\n\n\n<p>In simple terms, Stable Diffusion is a software that was trained to scramble and then accurately un-scramble images. By doing this, the computer doing the training learned how to make images on it\u2019s own. That information it learned was then compiled into a file that is, essentially, instructions on how to do it again.<\/p>\n\n\n<h1 class=\"wp-block-heading\" id=\"what-is-a-model-in-stable-diffusion\">What is a Model in Stable Diffusion?<\/h1>\n\n\n<p class=\"has-medium-font-size\"><strong>A model is therefore a file that tells Stable Diffusion how to make images using the knowledge previously learned by another machine. The model contains the results from a previous training session done on a specific set of data (in this case, the data was a collection of images). By loading a model into Stable Diffusion, it allows the program to use that pre-trained knowledge.<\/strong><\/p>\n\n\n\n<p>Stable Diffusion on it\u2019s own cannot create images. It\u2019s a generator, but it needs to access the model (any model, really) so it knows what to make.<\/p>\n\n\n\n<p><strong>It\u2019s <\/strong><strong>kind of<\/strong><strong> like a handheld<\/strong><strong> GameBoy. <\/strong>The GameBoy doesn\u2019t have any games on it\u2019s own. Instead, you plug in a cartridge that contains a game in order to access it and play. In this situation, Stable Diffusion is like the GameBoy, and the models are like different games you can plug in.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full is-resized\"><img fetchpriority=\"high\" decoding=\"async\" src=\"https:\/\/promptphantom.com\/wp-content\/uploads\/2023\/02\/01788-3601060865-a-gameboy-on.png\" alt=\"\" class=\"wp-image-868\" width=\"576\" height=\"384\" srcset=\"https:\/\/promptphantom.com\/wp-content\/uploads\/2023\/02\/01788-3601060865-a-gameboy-on.png 768w, https:\/\/promptphantom.com\/wp-content\/uploads\/2023\/02\/01788-3601060865-a-gameboy-on-300x200.png 300w\" sizes=\"(max-width: 576px) 100vw, 576px\" \/><figcaption class=\"wp-element-caption\"><em>a gameboy, on a table, detailed, realistic, 35mm lens<\/em><\/figcaption><\/figure>\n\n\n\n<p>So how do you make your own models for Stable Diffusion?<\/p>\n\n\n<h1 class=\"wp-block-heading\" id=\"what-is-a-stable-diffusion-checkpoint-or-ckpt-file\">What is a Stable Diffusion Checkpoint or CKPT File?<\/h1>\n\n\n<p>A checkpoint file is just the technical name for a model file. The file extension for these are CKPT. So for our purposes, a checkpoint means the same thing as a model. Not all models come in a CKPT file format, but we\u2019ll get to that in the section called \u201cWarnings Before You Download Any Models\u201d.<\/p>\n\n\n<h1 class=\"wp-block-heading\" id=\"how-are-stable-diffusion-models-made\">How are Stable Diffusion Models Made?<\/h1>\n\n\n<p>Models are made by training a computer on a set of images, forcing it to learn what the images look like and how to replicate or emulate them.<\/p>\n\n\n\n<p>The base models were trained on <a href=\"https:\/\/waxy.org\/2022\/08\/exploring-12-million-of-the-images-used-to-train-stable-diffusions-image-generator\/\">bil<\/a><a href=\"https:\/\/waxy.org\/2022\/08\/exploring-12-million-of-the-images-used-to-train-stable-diffusions-image-generator\/\">lions of images<\/a> that were crawled from the internet. It took 150,000 GPU hours (really, 24 days on 256 GPUs) and <a href=\"https:\/\/the-decoder.com\/training-cost-for-stable-diffusion-was-just-600000-and-that-is-a-good-sign-for-ai-progress\/\">$600,000<\/a> to do it.<\/p>\n\n\n\n<p><strong>Thankfully, you won\u2019t need that much time or money to make your own custom Stable Diffusion model. That\u2019s because we don\u2019t have to train our own models completely from scratch. Instead, user-made models are trained on top of the base version.<\/strong><\/p>\n\n\n<h2 class=\"wp-block-heading\" id=\"the-tldr-version-of-model-training\">The TL;DR Version of Model Training<\/h2>\n\n\n<p>In short, you would do the following to train a new model for Stable Diffusion:<\/p>\n\n\n\n<ul>\n<li>Collect a bunch of images with a common theme (an artist\u2019s style, a subject, an art medium, etc)<\/li>\n\n\n\n<li>Add them to a model that already exists (usually, CompVis\u2019s base models like <a href=\"https:\/\/huggingface.co\/CompVis\/stable-diffusion-v1-4\">V1.4<\/a>)<\/li>\n\n\n\n<li>Train that current model to also learn from your set of images and slightly modify what it already knows<\/li>\n\n\n\n<li>Save that training as a new model checkpoint file<\/li>\n<\/ul>\n\n\n\n<p>This process is sometimes referred to as \u201c<strong>fine tuning<\/strong>\u201d, because you are tweaking a model that already exists rather than restarting the whole learning process from the ground up.<\/p>\n\n\n\n<p><strong>So, in this way, you can also think of models like being video game mods. <\/strong>With computer games, users can create their own custom content (like new character designs or levels) and add it on top of the base game. Likewise, Stable Diffusion users can create their own modified versions of the base dataset. Those modded version can then be loaded into Stable Diffusion and run just like the base model.<\/p>\n\n\n<h1 class=\"wp-block-heading\" id=\"where-can-i-find-models\">Where Can I Find Models?<\/h1>\n\n\n<p>Now we\u2019ve gotten to the fun part. Anyone who makes a custom model can share it on the internet. And many of them do? Some people will charge for access to their models. But many users offer them for free.<\/p>\n\n\n\n<p>And there is a range of websites dedicated just to hosting these massive collections of user-made models! The most popular online repositories are:<\/p>\n\n\n\n<ul>\n<li><a href=\"https:\/\/huggingface.co\/models\" target=\"_blank\" rel=\"noopener\" title=\"\">Huggingface<\/a> (this site is operated by the creators of Stable Diffusion, a company called Stability AI)<\/li>\n\n\n\n<li><a href=\"https:\/\/civitai.com\/\" target=\"_blank\" rel=\"noopener\" title=\"\">Civitai<\/a> (this is a user-submission site where people can share models and review those made by others)<\/li>\n<\/ul>\n\n\n\n<p>I personally prefer getting models on CivitAI, because their site has an excellent tagging system and browsing features. That comes in handy if you don\u2019t know exactly what model you want.<\/p>\n\n\n<h2 class=\"wp-block-heading\" id=\"warnings-before-you-download-any-models\">Warnings Before You Download Any Models!<\/h2>\n\n\n<p>But before you go downloading a hundred different models to try out, take heed! There are some warning you need to be aware of:<\/p>\n\n\n\n<ol>\n<li><strong>Model files can be very large in size, <\/strong><strong>anywhere from 2 GB to 8 GB. <\/strong>You could easily fill up a hard drive with just model files. If your computer is low on disk space, then go easy on the downloads&#8230;or buy an extra hard drive to store them on.<\/li>\n\n\n\n<li><strong>Checkpoint files can be dangerous, because they can execute code on your computer. <\/strong>This means it\u2019s possible for an intentionally malicious model to install malware on your computer when you run it. But there is a simple way to avoid this issue: always download the SafeTensors format of the model, rather than the Pickle Tensor. For more information, see <a href=\"https:\/\/promptphantom.com\/is-it-safe-to-download-stable-diffusion-checkpoint-models-from-the-internet\/\" target=\"_blank\" rel=\"noopener\" title=\"\">my guide on Pickle Safety<\/a>.<\/li>\n\n\n\n<li><strong>You need Stable Diffusion on your computer to use custom models.<\/strong> Models cannot be loaded into any online service (that I know of, at least). If you haven\u2019t installed Stable Diffusion on your computer yet, check out <a href=\"https:\/\/promptphantom.com\/how-to-install-a-stable-diffusion-web-ui-on-your-computer\/\" target=\"_blank\" rel=\"noopener\" title=\"\">my guide here<\/a>.<\/li>\n<\/ol>\n\n\n<h1 class=\"wp-block-heading\" id=\"what-are-the-best-models-to-use\">What are the Best Models to Use?<\/h1>\n\n\n<p>This subject can be subjective. We all have different reasons for using Stable Diffusion and we want to make different kinds of images. So these recommendations are based on a mixture of my experiences and reviews from other users.<\/p>\n\n\n\n<p><strong>Rather than add a massive list of models onto this article, I instead have curated links to several model lists I have already posted.<\/strong><\/p>\n\n\n\n<ul>\n<li>Most Popular\/Downloaded Models<\/li>\n\n\n\n<li>The Best Models for Photo-realistic Images and Faces<\/li>\n<\/ul>\n\n\n<h1 class=\"wp-block-heading\" id=\"conclusion\">Conclusion<\/h1>\n\n\n<p>Try not to go overboard when browsing for custom models. It\u2019s easy to feel like a kid in a candy store and just go wild with downloading. But do feel free to test as many as you need until you find the style of output that you\u2019re looking for. And have fun!<\/p>\n\n\n\n<p>Thanks for reading. If you found this guide helpful, here are a few more articles you may like:<\/p>\n\n\n\n<ul>\n<li><a href=\"https:\/\/promptphantom.com\/the-complete-beginners-guide-to-the-stable-diffusion-interface\/\" target=\"_blank\" rel=\"noopener\" title=\"\">The Complete Beginner\u2019s Guide to the Stable Diffusion Interface<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/promptphantom.com\/the-beginners-guide-to-img2img-in-stable-diffusion\/\" target=\"_blank\" rel=\"noopener\" title=\"\">The Beginner\u2019s Guide to Img2Img in Stable Diffusion<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/promptphantom.com\/what-are-cfg-scale-and-denoising-strength\/\" target=\"_blank\" rel=\"noopener\" title=\"\">What are CFG Scale and Denoising Strength?<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/promptphantom.com\/how-do-you-emphasize-words-in-stable-diffusion-prompts\/\">How Do You Emphasize Words in Stable Diffusion Prompts?<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/promptphantom.com\/how-long-should-a-prompt-be-for-stable-diffusion\/\">How Long Should a Prompt Be for Stable Diffusion?<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>A FAQ Guide on Custom AI Art Models Have you recently gotten sucked down the rabbit hole of AI art generation? Perhaps you heard that Stable Diffusion is capable of being custom trained but you didn\u2019t understand how. Maybe you\u2019ve seen people talking about different models of Stable Diffusion but didn\u2019t know what they meant. [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":867,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"disabled","footer-sml-layout":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""}},"iawp_total_views":0,"footnotes":""},"categories":[4,3],"tags":[8,9,22,6],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/promptphantom.com\/wp-json\/wp\/v2\/posts\/864"}],"collection":[{"href":"https:\/\/promptphantom.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/promptphantom.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/promptphantom.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/promptphantom.com\/wp-json\/wp\/v2\/comments?post=864"}],"version-history":[{"count":3,"href":"https:\/\/promptphantom.com\/wp-json\/wp\/v2\/posts\/864\/revisions"}],"predecessor-version":[{"id":950,"href":"https:\/\/promptphantom.com\/wp-json\/wp\/v2\/posts\/864\/revisions\/950"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/promptphantom.com\/wp-json\/wp\/v2\/media\/867"}],"wp:attachment":[{"href":"https:\/\/promptphantom.com\/wp-json\/wp\/v2\/media?parent=864"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/promptphantom.com\/wp-json\/wp\/v2\/categories?post=864"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/promptphantom.com\/wp-json\/wp\/v2\/tags?post=864"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}