QR Code generation with Stable Diffusion

In this blog we will see how to generate QR code with Stable Diffusion and Control Net Models using Python.

Here we use Pretrained QR Code model by DionTimmer — controlnet_qrcode_control_v1p_sd15.

Requirements:

1. Torch with Cuda 11.8
2. QRCode Library to generate QRCodes
3. Diffusors Library to fetch StableDiffusionControlNet Pipeline and ControlNet Model
4. Xformers and accelerators for faster inference in low memory.

Sourcecode — Github

Steps:

1. Generate a QRCode with the URL you want to Embed?

def generate_qr_code(content:str):
    qr = qrcode.QRCode(
        version=1, 
        error_correction=qrcode.constants.ERROR_CORRECT_H,
        box_size = 16,
        border = 0,
    )
    
    qr.add_data(content)
    qr.make(fit=True)
    img = qr.make_image(fill_color = "black",back_color = "white")
    
    offset_min = 8*16
    w,h = img.size
    w = (w+255+offset_min)//256*256
    h = (h+255+offset_min)//256*256
    
    if w>1024:
        raise ValueError("QR Code is to big")
    
    bg = Image.new("L",(w,h),255)
    
    coords = ((w-img.size[0])//2//16*16,
              (h-img.size[1])//2//16*16
              )
    bg.paste(img, coords)
    return bg

2. Load the Pretrained Control Net QR Code Model

controlnet = ControlNetModel.from_pretrained(
    "DionTimmer/controlnet_qrcode-control_v1p_sd15",
    torch_dtype = torch.float16
)

3. Setup the Control Net pipeline with the pretrained model

pipe = StableDiffusionControlNetPipeline.from_pretrained(
    # "runwayml/stable-diffusion-v1-5",
    "digiplay/GhostMixV1.2VAE",
    controlnet = controlnet,
    torch_dtype = torch.float16,
    safety_checker =None,
).to("cuda")
pipe.enable_xformers_memory_efficient_attention()

4. Provide Prompts and Negative Prompts for how the QRcode should look like.

prompt = "<Your Prompt>"
negative_prompt = "<Your Prompt>"
qrcode_content = "<Site>"
qr_code_image = generate_qr_code(qrcode_content)
# generator = torch.Generator().manual_seed(2174793473)
seed = random.randint(1, 1000000000000)
print("Seed:",seed)
generator = torch.Generator().manual_seed(seed)
# .manual_seed(2324230317)

5. Run the Pipeline with different hyperparameters and prompts.

output = pipe(
    prompt=prompt, 
    negative_prompt=negative_prompt,
    image = qr_code_image,
    width = qr_code_image.width,
    height=qr_code_image.height,
    guidance_scale=<value>,
    controlnet_conditioning_scale=<value>,
    control_guidance_start= <value>,
    control_guidance_end= <value>,
    generator=generator,
    num_inference_steps=<value>,
)

Output

Cyborg Girl

prompt = "1mechanical girl,ultra realistic details, portrait, global illumination, shadows, octane render, 8k, ultra sharp,intricate, ornaments detailed, cold colors, metal, egypician detail, highly intricate details, realistic light, trending on cgsociety, glowing eyes, facing camera, neon details, machanical limbs,blood vessels connected to tubes,mechanical vertebra attaching to back,mechanical cervial attaching to neck,sitting,wires and cables connecting to head"
negative_prompt = "ugly, disfigured, low quality, blurry"
seed = 698853158198
guidance_scale=10,
controlnet_conditioning_scale=2.75,
control_guidance_start= 0.23,
control_guidance_end= 0.9,
num_inference_steps=50

Cyborg

prompt = "light, futobot, cyborg, ((masterpiece),(best quality),(ultra-detailed), (full body:1.2), 1male, solo, hood up, upper body, mask, 1boy, male focus,white gloves, cloak, long sleeves, spaceship, lightning, hires"
negative_prompt = "ugly, disfigured, low quality, blurry"
seed = 396250463083
guidance_scale=10,
controlnet_conditioning_scale=2.75,
control_guidance_start= 0.23,
control_guidance_end= 0.9,
num_inference_steps=50,

Robot

prompt = "(masterpiece, best quality:1.3),extremely high detailed, intricate, 8k, HDR, wallpaper, cinematic lighting, <lora:Mecha:1> ,large sword, glowing, glowing eyes, mecha, realistic, <lora:CyberPunkAI:0.5>"
negative_prompt = "(worst quality, low quality:2), (monochrome), zombie,overexposure, watermark,text,bad anatomy,bad hand,extra hands,extra fingers,too many fingers,fused fingers,bad arm,distorted arm,extra arms,fused arms,extra legs,missing leg,disembodied leg,extra nipples, detached arm, liquid hand,inverted hand,disembodied limb, loli, oversized head,extra body,extra navel,easynegative,sketch, duplicate, ugly, huge eyes, text, logo, worst face, (bad and mutated hands:1.3), (blurry:2.0), horror, geometry, bad_prompt, (bad hands), (missing fingers), multiple limbs, bad anatomy, (interlocked fingers:1.2), Ugly Fingers, (extra digit and hands and fingers and legs and arms:1.4), (deformed fingers:1.2), (long fingers:1.2),(bad-artist-anime), bad-artist, bad hand, extra legs ,(ng_deepnegative_v1_75t) ,bad-hands-5"
seed = 885162813
guidance_scale=10,
controlnet_conditioning_scale=3.0,
control_guidance_start= 0.23,
control_guidance_end= 0.9,
num_inference_steps=50,

Other Outputs

Reference:

1. https://huggingface.co/DionTimmer/controlnet_qrcode-control_v1p_sd15
2. https://www.youtube.com/watch?v=y-rCMjxVrGA
3. https://huggingface.co/digiplay/GhostMixV1.2VAE