A Practical Guide to Rocking Collaboration with Stable Diffusion

Collaborating with neural networks means you deal with new features literally every month. While many view neural networks as hype, at JetStyle we perceive them as equal creative partners. However, some of the AI tools need some fine-tuning to be user-friendly. Even though they are not so great with complex commercial tasks, with enough practice you save time and boost your brainstorming activities. 

We talked with Lyaisan Toropova, JetStyle’s motion designer and AI enthusiast, to share her thoughts about working with Stable Diffusion: 

“Unlike Midjourney, it’s a much trickier tool, but it has a lot of advantages, too. For example: 

• Midjourney is very artistic, but not flexible. Stable Diffusion comprises both characteristics, but it requires a lot of collaborative effort to give you what you need. If you need a very specific result, this is your go-to tool. 
• Stable Diffusion’s inpainting feature is much more advanced compared to the one in Midjourney. 
• There’s no censorship in Stable Diffusion. It sometimes leads to some unwanted results (i.e. the network will generate nudity even when it was not in the prompt), but it’s the service’s distinctive feature. 

Here are a few engineering tips that will ease your way to a successful collaboration with Stable Diffusion, from installing to finetuning of the image. 

Installation

The full manual on how to install the portable version of Stable Diffusion on your PC is available on GitHub.

Models for Stable Diffusion

Models are used to generate images, and you have to download and install them separately. 

ControlNet extension

You need the ControlNet extension to keep the same contours, or pose, or style in new generations as in your reference. Here's a mini-instruction:

1. Click Extensions/Install from the URL.

2. In the URL for the extension's git repository, paste https://github.com/Mikubill/sd-webui-controlnet 

3. Click Install.

4. Navigate to Installed.

5. Click Apply and restart UI.

The next step is to set the models for ControlNet:

1. Download the pre-extracted model: https://huggingface.co/webui/ControlNet-modulessafetensors .

2. In the Files and versions tab, download:

- control_canny-fp16.safetensors - to replicate the contours of the reference picture;

- control_depth-fp16.safetensors control_openpose-fp16.safetensors - to repeat the depth map of the reference image.

These are the basic models; you can install any other of your choice. 

3. Move all files to SD/models/ControlNet.

Negative Prompt

If you want to avoid those classic AI-generated elements like extra limbs or distorted face features in your images, use this standard negative prompt, and add whatever features you don’t want to generate:

deformed, bad anatomy, disfigured, poorly drawn face, mutation, mutated, extra limb, ugly, disgusting, poorly drawn hands, missing limb, floating limbs, disconnected limbs, malformed hands, blurry,((((mutated hands and fingers)))), watermark, watermarked, oversaturated, censored, distorted hands, amputation, missing hands, obese, doubled face, double hands.

Prompt Customization

Customize your result  by changing the weight of a word in your prompt. A few  basic ways to do this:

1. Put the word or phrase you want to emphasize in brackets - single, double, or triple brackets.

2. Select the word or phrase, press Ctrl, and set the weight with the up and down arrows. It looks like this: (word : 1.4).

3. You can set the proportions of generation steps, so at first Stabe Diffusion generates according to one part of the prompt, and only then uses the next one. 

You set it like this: [ prompt : other prompt : 0,6] or [ prompt : other prompt : 18]. This means that if you have 30 generation steps, 0,6 or 18 of them will be allocated to the first part of the prompt. 

Hope this helps! The AI landscape is changing fast, and sometimes it’s easier to delegate collaboration with neural networks to experienced partners. JetStyle has been experimenting with AI since it first gained popularity among users; and we’d be happy to help you with ways to incorporate it into your daily task-solving routine. Get in touch orders@jet.style 

BTW, we have another article dedicated solely to effective collaboration with Midjourney here.