How to come up with good prompts for Stable Diffusion

Updated Categorized as Tutorial Tagged , 17 Comments on How to come up with good prompts for Stable Diffusion
Prompt guide final image

Prompt building is a basic skill any Stable Diffusion user should master. By understanding how to construct clear and concise prompts, you can unlock the whole range of style Stable Diffusion offers. To excel in prompt building, you should start with a specific subject in mind and add keywords to steer towards a particular effect.

The skill you learn in this article can be applied to other AI art generators such as MidJourney.

This is part 2 of the beginner’s guide series.
Read part 1: Absolute beginner’s guide.
Read part 3: Inpainting.
Read part 4: Models.

Resources for beginners

If you want to practice prompt building but do not have your Stable Diffusion set up yet, you can use a free Stable Diffusion generator online.

Read the Quick Start Guide if you want to set up your own.

Use the prompt builder for a systematic approach to craft prompts.

This post is intended to be your first course in prompting. See this prompt guide for advanced techniques.

Anatomy of a good prompt

This is a proven technique to generate high-quality, specific images. Your prompt should cover most, if not all, of these areas.

  1. Subject (required)
  2. Medium
  3. Style
  4. Artist
  5. Website
  6. Resolution
  7. Additional details
  8. Color
  9. Lighting

Subject

First, you will need a subject description with as much detail as possible. Below is an example.

Prompt (subject only):

A young woman with light blue dress sitting next to a wooden window reading a book.

A common mistake of a beginner is not describing the image in enough detail. What clothing does she wear? What’s her hairstyle? These seemingly minor elements can contribute significantly to the overall image being conveyed. Without specifying these details, you leave them open to the AI generator, and you may be disappointed about what you get.

It is a good practice to include a generic negative prompt.

ugly, deform, disfigured

Let’s use the Realistic Vision v2 model.

We got the following image, which matches the prompt pretty well.

Not bad. This model defaults to generate photo-realistic images, so this is the style you get.

Medium

We can be more specific. Let’s add a medium, the material that the artwork is created on. Some examples are digital painting, photography, and oil painting. Let’s use

Medium

Digital painting

The new prompt is

Digital painting of a young woman with light blue dress sitting next to a wooden window reading a book

The resulting image is

You can see the image changes from a photograph to a digital art. It’s a step forward but we can do better.

Adding the rest

You get the idea. Let’s add the rest of them

Artist – specifying the artist who created the art to steer the style.

by Stanley Artgerm Lau

Website – The name of the website can be used for a particular genre.

artstation

Resolution – These are keywords that control the sharpness of the image.

8k

Additional details – These are keywords that are more like sweeteners,e.g. adding some interesting details.

extremely detailed, ornate

LightingControling light is important for a good image.

cinematic lighting, rim lighting

color – The color scheme of the image

vivid

Putting them all together, the prompt is

Digital painting of a young woman with light blue dress sitting next to a wooden window reading a book, by Stanley Artgerm Lau, artstation, 8k, extremely detailed, ornate, cinematic lighting, rim lighting, vivid

Now we get this image.

We can engineer the image to get the style we want by adding specific keywords to the prompt.

Tips for good prompts

  • Be detailed and specific when describing the subject.
  • Use multiple brackets () to increase its strength and [] to reduce.
  • Use an appropriate medium type consistent with the artist. E.g. photograph should not be used with van Gogh.
  • The artist’s name is a very strong style modifier. Use wisely.
  • Experiment with blending styles.
  • Head to the Workflow section to study the high-quality prompts. Use the prompt as a starting point if you like a particular image.

Some good keywords for you

Below are some of my favorite keywords and their effects. (Used with Stable Diffusion v1.4 and v1.5). Enjoy!

You can find the full list in the prompt builder.

Can’t find the right words? Get the FULL LIST with 250+ well-tested keywords.

Medium

Medium defines a category of artwork.

keywordNote
PortraitVery realistic drawings. Good to use with people.
Digital paintingDigital art style.
Concept artIllustration style, 2D.
Ultra realistic illustrationDrawings that are very realistic. Good to use with people.
Underwater portraitUse with people. Underwater. Hair floating.
Underwater steampunkVery realistic drawings. Good to use with people.

Style

These keywords further refine the art style.

keywordNote
hyperrealisticIncreases details and resolution
pop-artPop-art style
Modernistvibrant color, high contrast
art nouveauAdd ornaments and details, building style

Artist

Mentioning the artist in the prompt is a strong effect. Study their work and choose wisely.

keywordNote
John Collier19th century portrait painter. Add elegancy
Stanley Artgerm LauGood to use with woman portrait, generate 19th delicate clothing, some impressionism
Frida KahloQuite strong effect following Kahlo’s portrait style. Sometimes result in picture frame
John Singer SargentGood to use with woman portraits, generate 19th delicate clothing, some impressionism
Alphonse Mucha2D portrait painting in style of Alphonse Mucha

Website

Mentioning an art or photo site is a strong effect, probably because each site has its niche genre.

keywordNote
pixivJapanese anime style
pixabayCommercial stock photo style
artstationModern illustration, fantasy

Resolution

keywordNote
unreal engineVery realistic and detailed 3D
sharp focusIncrease resolution
8kIncrease resolution, though can lead to it looking more fake. Makes the image more camera like and realistic
vray3D rendering best for objects, landscape and building.

Lighting

keywordNote
rim lightinglight on edge of an object
cinematic lightingA generic term to improve contrast by using light
crepuscular rayssunlight breaking through the cloud

Additional details

Add specific details to your image.

keywordNote
dramaticshot from a low angle
silkAdd silk to clothing
expansiveMore open background, smaller subject
low angle shotshot from low angle
god rayssunlight breaking through the cloud
psychedelicvivid color with distortion

Color

Add an additional color scheme to the image.

keywordNote
iridescent goldShinny gold
silverSilver color
vintagevintage effect

Summary

We have gone through the basic structure of a good prompt. This should be used as a guide rather than a rule. The Stable Diffusion model is very flexible. Let it surprise you with some creative combination of keywords!

Check out the Stable Diffusion Course for a step-by-step guided course.

Or continue to part 3 below.

This is part 2 of the beginner’s guide series.
Read part 1: Absolute beginner’s guide.
Read part 3: Inpainting.
Read part 4: Models.

Avatar

By Andrew

Andrew is an experienced engineer with a specialization in Machine Learning and Artificial Intelligence. He is passionate about programming, art, photography, and education. He has a Ph.D. in engineering.

17 comments

  1. Just a side note, I have found that Swapping words around sometimes yield better results. ie. ( ( The girl standing under the big tree eating apple.) ) Vs ( (Under the big tree a girl is standing eating an apple. ) ) changing the sequence of the main subjects ( Girl , Tree , Apple ) gives more priority to each, and the “details” (if applicable) will be assigned in the same order. Ie Beautiful. In the 1st sentence the Girl is mentioned first so the “weight” of beautiful will be added to her more than the tree. same applies for the latter.

  2. These articles are great Andrew; thank you. I am curious if there are enumerations for the different key words according to area in the training data set (all artist names present in the data set, for instance). Is there anything like that currently available, or established techniques for generating them?

  3. How do you get these images? I am copying exactly the same prompt (incl. negative), trying 1.5 base, or 2.0 base, or 2.1 base. It generates absolutely ugly deformed faces. I tried Diffusion Bee and Mochi Diffusion, nothing works even remotely as good as what you have in this article.

  4. Seriously, what could you use these crap images for?
    The last image is hideous to say the least.
    Just as well I discovered this site before I forked out 500 bucks for a new graphics card.

  5. Found a good way to generate prompts. Copy this entire article into ChatGPT and ask it to understand how to generate prompts. From there, just ask it what you want to achieve and ChatGPT will give you a good prompt to use.

    Use AI to make AI Art

Leave a comment

Your email address will not be published. Required fields are marked *