Beginner's Guide to Stable Diffusion Inpainting

Updated on May 29, 2024

In the ever-evolving world of artificial intelligence, one technique is revolutionizing the way we restore and enhance images: stable diffusion inpainting. Imagine seamlessly repairing damaged photos, filling in missing details, or even removing unwanted elements with astonishing precision and realism. This cutting-edge approach leverages the power of AI to analyze and predict the missing parts of an image, ensuring that the final result is cohesive and visually stunning. Below, we have compiled a detailed guide walking you through Stable Diffusion inpainting, explaining what is it and how to use it.

What is Stable Diffusion Inpainting

Stable Diffusion Inpainting is a technique that allows you to modify or enhance specific areas of an existing image using the powerful capabilities of the Stable Diffusion AI model. It enables you to seamlessly blend new elements into the image, remove unwanted objects, or alter specific details while preserving the overall context and coherence of the original image.

Tips: As shown in the image, the output from using SD Inpainting is quite small. This limitation arises because most Stable Diffusion models are based on a 512-pixel base image, making it challenging to achieve high-quality results. This is where a dedicated AI image upscaler becomes essential. Aiarty Image Enhancer is a cutting-edge AI tool designed to elevate image quality by deblurring, denoising, upscaling, and adding realistic details. Utilizing the latest AI models, it supports upscaling to 32K resolution, ensuring exceptional clarity and detail. Below is an upscaled SD inpainting work using Aiarty Image Enhancer.

Free Download Free Download

How Does Stable Diffusion Inpainting Work

The principle behind Stable Diffusion inpainting is to leverage the capabilities of the Stable Diffusion AI model to seamlessly fill in or modify specific regions of an existing image based on a text prompt. Here is how it works:

Masking: The user provides the original image and creates a mask that highlights the areas they want to modify or inpaint.
Text Encoding: The user's text prompt describing the desired changes or additions is encoded into a latent representation that the Stable Diffusion model can understand.
Denoising Process: The Stable Diffusion model takes the masked image and the encoded text prompt as inputs. It then goes through an iterative denoising process, gradually transforming the masked regions into new content that aligns with the text prompt while seamlessly blending with the preserved areas of the original image.
Attention Mechanism: The model's attention layers focus on specific parts of the image based on the textual information, allowing it to understand the context and generate coherent content that respects the surrounding environment.

Some key features and applications of Stable Diffusion Inpainting include:

Removing unwanted objects or elements from an image by masking them out and allowing the model to generate new content to fill the void seamlessly.
Enhancing or correcting specific details in an image, such as fixing imperfections, altering facial features, or modifying objects' appearances.
Adding new elements or subjects to an existing scene, such as inserting a person, animal, or object into the image while maintaining coherence with the surrounding environment.
Transforming the style or aesthetic of an image by modifying the masked areas to match a desired artistic style or visual theme.

Stable Diffusion Inpainting is a powerful tool for artists, designers, and content creators, enabling them to refine and enhance their visual creations with unprecedented control and flexibility. Follow the steps below to inpaint with Stable Diffusion.

How to Use Stable Diffusion Inpainting

Step 1. Download an inpainting model, such as the Stable Diffusion 2 inpainting model from Hugging Face or the epiCRealism Inpainting model from Civitai.

Step 2. Head to the Stable Diffusion web UI, go to the 'img2img' tab > the 'inpaint' sub-tab. Then upload the image you want to inpaint.

Step 3. Use the brush tool to create a mask over the areas that you want to modify or replace. Here the masked region is marked in white. You can change the brush color.

Step 4. Set a text prompt describing the desired changes or additions you want in the masked area.

Step 5. Select the inpainting model you downloaded from the 'Stable Diffusion checkpoint' dropdown menu, and adjust other parameters as needed.

Denoising strength: This controls how much the masked area should change. As the value increases, the scope and impact of the resulting changes also increase. It is suggested to start with a denoising strength of 0.5 and adjust based on the desired value of change and detail.
Masked Content: This specifies how you want to change the image in the masked area before inpainting. "Original" is the most common choice that keeps the original content within the masked area. "Fill" replaces the masked area with the average color of the surrounding region, which is used when a significant change from the original content is desired. "Latent Noise" and "Latent Nothing" are generally not recommended as they can lead to undesirable results.
Mask Mode: The "Only Masked" focuses on the masked area by cropping it out, using the full resolution for inpainting, and then scaling it back, and this can help fix issues with generating small faces or objects. The 'Whole Picture' option processes the entire image including the masked area, and this is preferred when the entire image needs slight adjustments without isolating the masked region.
Image Size: the image dimensions should be precisely matched to the original specs.

Below are the settings I use:

On top of the basic inpainting settings, ControlNet is another tool you can leverage while inpainting in Stable Diffusion. ControlNet allows users to control and guide the image generation by providing additional conditional inputs beyond just text prompts, and users can copy the outline, human poses, etc. from another image and use it as a guiding reference for accurate modifications during the inpainting.

Step 6. Click 'Generate' to run the inpainting process, and the model will fill in the masked regions based on your text prompt while preserving the unmasked areas.

Conclusion

Follow the steps above to inpaint your images using Stable Diffusion! If you wish to further enhance your Stable Diffusion inpainting works, check out Aiarty Image Enhancer, a generative AI image enhancement software to denoise, deblur, generate more details, and upscale images.