How to use img2img to turn an amateur drawing to professional with Stable Diffusion (Image-to-image)

Not a born-artist? Stable Diffusion can help. Img2img (image-to-image) can improve your drawing while keeping the color and composition.

What is image-to-image?

Image-to-image (img2img for short) is a method to generate new AI images from an input image and text prompt. The output image will follow the color and composition of the input image.

The input image is just a guide. It does not need to be pretty or have any details. The important part is the color and the composition.

The prompt requirement is the same as text-to-image.

Software setup

We will use AUTOMATIC1111 GUI for this tutorial.

Enable color sketch tool

The color sketch tool is not turned on by default. Using it makes img2img a lot easier.

To enable the color sketch feature in AUTOMATC1111 GUI, add the following argument option when you run webui.py.

--gradio-img2img-tool color-sketch

Alternatively, on Windows system, edit the file webui-user.bat under stable-diffusion-webui folder. Change the line

set COMMANDLINE_ARGS=

to

set COMMANDLINE_ARGS=--gradio-img2img-tool color-sketch

If there’s already something after the equal sign, add --gradio-img2img-tool color-sketch to the end (Make sure there’s a space in the beginning).

Double click the file to start the GUI.

The option is already enabled in Colab Notebook in the Quick Start Guide.

Using color sketch tool

To use the color sketch tool, go to img2img tab, upload a starting image to the canvas. Click on the color palette icon and the then the solid color button. The color sketch tool should appear like below.

Now you can draw in color!

The color picker tool is especially useful when making small modifications.

Transform an amateur drawing to professional

I just saw an Youtube video of a professional artist drawing a realistic apple with a black background on iPad. Thought this would be a good subject to show how to use img2img.

Step 1: Create the background

You can use this black or white background. They are both 512×512 pixels, the same as the default image size of Stable Diffusion v1.5.

Upload the background to the canvas.

Step 2: Draw an apple

Let’s draw the apple with the color palette tool.

Don’t spend too much time on what you draw. Just aim at getting the color, shape and composition in the right neighborhood.

This is the apple I drew. (The little light green strips are water drops… just so you know…)

Step 3: img2img

In Stable Diffusion checkpoint dropbox, select v1-5-pruned-emaonly.ckpt for using v1.5 model. (You can also experiment with other models.)

Come up with a prompt that describe your final picture as accurately as possible.

photo of perfect green apple with stem, water droplets, dramatic lighting

Put this in the prompt text box.

Set image width and height to 512.

Set sampling steps to 20 and sampling method to DPM++ 2M Karras.

Set batch size to 4 so that you can cherry pick the best one.

Set seed to -1 (random).

The two parameters you want to play with are CFG scale and denoising strength. In the beginning, you can set CFG scale to 11 and denoising strength to 0.75.

Hit Generate to get a set of four new images.

Increase denoising strength if you want the images to change more. Decrease if you want them to be closer to your original drawing.

Once you are happy with what you get, save the image.

Step 4: Second img2img

You can stop here if you are happy with the result. But doing one more round of img2img adds more details and allow you to use a different prompt.

Upload the image you just generated.

For mine, I feel the stem is a bit too dark so I paint it a bit lighter. (I used the color picker tool to get the color of the stem, and dial up the color values.)

I simply reused the same settings, including the prompt.

In the new batch, I get something I like:

It has a lot more realistic details and better lighting. Doing a second round of img2img adds complexity to the scene.

Next steps

img2img is a versatile technique to control the composition and color of the image. It provides additional control to text-to-image.

In the next tutorials, I will walk you through a few more techniques to use with img2img. Stay tuned!