Krita AI Diffusion

What is Krita AI Diffusion?

Krita AI Diffusion is an open-source plugin that integrates generative AI directly into the Krita drawing workflow. Artists can continue to use Krita’s brushes, layers, selection tools, masks, and color tools, while also having access to functions such as image generation, local redrawing, image expansion, reference image control, real-time drawing, and super-resolution through the sidebar.

The plugin does not directly execute large-scale diffusion models; instead, it connects to the ComfyUI backend. The backend can be deployed on the local machine using the installation wizard, it can also run on another computer or server, or it is possible to use Interstice’s cloud GPU services.

This architecture keeps the interface within Krita, while still allowing for the choice between local models and remote computing power.

Text-to-image generation and canvas creation

Users can enter positive and negative prompts, select a style and a base model, and then generate images within the current canvas area. The resulting images are added to the document as Krita layers, allowing for further editing such as drawing, reshaping, color adjustment, and compositing; instead of having to download the finished images from a separate website.

Prompt generation cannot guarantee accurate text, complex hand movements, or character consistency. For professional projects, it is necessary to save the models, LoRA files, seeds, sampling settings, and Krita source files in order to be able to reproduce and modify them.

Partial redraw of the constituency

After using Krita’s selection tool to outline a specific area, one can apply the Fill function to add elements, remove content, or regenerate that area. The context provided helps the AI understand the surrounding colors, perspective, and structure, making it suitable for repairing faces, replacing clothing, adding props, and cleaning up the image.

The edges of the mask, the context surrounding it, and its intensity all affect the way things connect. When a small area is located on a very large canvas, it is necessary to provide the AI with sufficient context and to process it in stages, in order to avoid texture disruptions or changes in proportions.

Image enlargement and canvas extension

After expanding the canvas, select an empty area and the AI will fill in the edges; this method is useful for converting square images into horizontal posters, completing a person’s body, or extending the background. The expansion process will guess content beyond the original image, and it is not possible to restore the actual range captured in the photo.

Refine image generation refinement

Refine reinterprets existing images by adjusting the intensity level: a low intensity helps improve textures and details, while a high intensity alters the shape and composition of the image.

Artists can use hand-drawn sketches to guide the AI step by step, rather than generating the final image all at once.

Progressive refinement is the core method used by plugins: first, the overall shape and colors are drawn, and then the resolution and details in specific areas are improved; this approach is generally more controllable than repeatedly generating an image on a high-resolution canvas.

Live Painting – Real-time painting

Live Painting continuously reads the current canvas or a specified area, interpreting the user’s strokes in real time to create a more complete image. It is useful for figuring out composition, lighting, textures, and the direction of characters; changes made with the brush allow for quick feedback from the AI.

Real-time mode places high demands on the graphics card and on the speed of generation. When latency is too high, it is possible to reduce the resolution or the number of steps, or use faster models; high-quality detailing can then be applied after the direction has been determined.

Regions partition prompt

Regions allow separate prompt words to be set for different areas of the image; for example, it’s possible to use different descriptions for the characters, the background, and the clothing. By combining these regions with Krita layers, users can exert more precise control over scenes that contain multiple elements, instead of having to include all requirements in a single prompt.

Regional overlap or overly rigid boundaries can lead to conflicts; adjustments should be made using soft masks, layer order, and global guidelines.

Precise control with ControlNet

The plugin supports ControlNet techniques such as doodling, line drawings, Canny edges, pose detection, depth analysis, normal mapping, and segmentation. Users can create control maps from the current layer or import external references to regulate composition, outlines, human poses, and spatial relationships.

The control model must be compatible with the underlying model architecture. SDXL’s ControlNet cannot be used directly with Flux; an incorrect combination of models can lead to loading failures or uncontrolled results.

Reference diagrams and IP-Adapter

Reference Images can guide the style, composition, as well as the characteristics of characters and objects. Features such as IP-Adapter make it easier to maintain a consistent visual direction compared to using mere prompts; however, if the weight given to these reference images is too high, it can suppress creativity and even lead to replication of any flaws present in the input.

When using real people, brands, character designs, or works created by others, it is necessary to verify the scope of authorization; technical references should not be mistaken for an automatic grant of commercial rights.

Supported base models

The architectures supported by the plugins evolve with each version update. Each Checkpoint, LoRA, ControlNet, and IP-Adapter comes with its own license, trigger words, and resource requirements; it is not possible to determine the commercial licensing of the generated content based solely on the plugin’s GPL license.

Style and LoRA

Style saves the base model, prompt templates, sampling settings, and extended model combinations, enabling the reuse of a consistent visual style across projects. LoRA can be uploaded to a local or remote backend via file selection, and trigger words along with default intensity levels can be specified.

The use of multiple LoRA models on top of each other can interfere with one another and increase memory usage. For actual production, it is necessary to retain verified combinations and example images for each style.

Super-resolution and block processing

Upscale can be used to increase the size of images by employing super-resolution models, and it is possible to choose to refine the upscaled image using diffusion models to add more details. Large images are divided into tiles so that they can be processed one by one, thereby avoiding the exhaustion of video memory at once.

When dividing into blocks, seams, repeated textures, and local style variations may appear. Faces of people, text, and geometric patterns need to be examined closely when enlarged.

Editing models and modifying text instructions

Compatible Edit Models allow images to be modified through textual descriptions, such as adjusting lighting, colors, clothing, or objects. These editing tools reduce the need for manual masking, but they may also alter areas that the user did not intend to change; it is therefore advisable to test them first on a copy layer.

Comparison of connection methods and costs

Both Krita and its plugins are available for free. There is no official subscription fee for local use, but downloading models typically requires at least 10 GB of space; it is not uncommon to need more than 50 GB when installing multiple optional models.

The current amount for Interstice in the cloud is not clearly displayed in the public documentation; it is necessary to log in to the service page to check the package details, usage limits, and regional taxes.

Installation guide

1. First, install a supported version of Krita.
2. Download the plugin archive from the official GitHub release; do not download the repository’s source code snapshot as a substitute for the release package.
3. Use the Python plugin importer in Krita to install and restart the software.
4. Enable the AI Image Generation panel in Dockers under Settings.
5. Click Configure, and choose to install a local backend automatically, connect to an existing ComfyUI instance, or use a remote server or cloud service.
6. Download the basic workloads and optional models as needed, and first test them with low-resolution documents.

Local hardware and platform

The plugin interface can run on Windows, Linux, and macOS, while the list of platforms supported by the ComfyUI backend is still expanding. The project focuses on continuously testing NVIDIA GPUs on Windows and Linux, as these represent the most stable local computing options.

The availability of AMD, Apple Silicon, and other backends depends on PyTorch, ComfyUI, and the model architecture. If there is insufficient video memory, it is possible to reduce the size of the canvas, the batch size, and the model dimensions, or to use a remote GPU instead.

Only the CPU usually has a slow speed.

Notes on Flatpak and AppImage

The sandbox in the Linux Flatpak version of Krita may prevent the installation of certain backend packages; the official documentation recommends using AppImage when local automatic installation is required. It is also possible to install the backends in a separate directory and access them over a network connection.

Custom ComfyUI compatibility

Advanced users can connect to their own ComfyUI servers, but the plugins rely on specific node structures and workflow conventions. Errors occur when the versions of components such as ComfyUI, comfyui-tooling-nodes, and comfyui-inpaint-nodes are not compatible with each other.

After upgrading the plugin, the backend should be checked simultaneously.

It is not a general frontend that allows any ComfyUI node graph to be displayed directly in Krita. Custom nodes may not be usable within plugin workflows, and a stable, functional version should be used in production environments.

Remote access security

Remote ComfyUI should not be exposed to the public internet without authentication. For team deployment, a reverse proxy, HTTPS, authentication mechanisms, firewalls, limits on file size, and separate working directories are required.

Model uploading and custom nodes should only be allowed from trusted sources.

Privacy and data

The local backend allows prompts, canvases, and models to remain on your own device. In the remote or cloud-based mode, the necessary images and parameters are sent to a server; before processing sensitive customer data, it is important to review the cloud service’s policies regarding data retention, usage for training, and deletion.

Open-source license

Krita AI Diffusion is licensed under the GPL-3.0-or-later license; its source code, issues, and version releases are all available on the official GitHub repository. Krita itself is also open-source software, but ComfyUI’s custom nodes, model weights, and cloud services are subject to their own licenses.

Creating images does not automatically turn them into GPL works just by using GPL plugins; however, their commercial use is still subject to the terms of the models, LoRAs, input materials, and local laws.

Krita AI Diffusion Tutorial

Create reusable professional workflows

1. Create a list of brand colors, fonts, layout elements, and elements that are prohibited.
2. Test image generation from text, as well as canvas creation, local redrawing of selected areas, and image enlargement along with canvas extension, separately.
3. Use the same set of representative samples to compare quality, speed, and cost;
4. Complex edges, text, and images of key products should be handed over for manual refinement.
5. Standardize naming, dimensions, and review status;
6. Batch processing and release are carried out after random inspections;

Which users is it suitable for?

• Illustrators who wish to use AI directly in Krita;
• Creators who need to combine hand-drawing with generative local repair;
• Users who perform local drawing using Flux, SDXL, or anime models;
• Designers who need ControlNet, segmentation prompts, and reference images for control;
• Teams that already have a ComfyUI server and wish to integrate it with a professional canvas;
• Users who value offline privacy and open-source software.

Main advantages

• AI results are directly integrated into Krita’s layer and selection workflow;
• It is free and open-source, with the option to use a local, remote, or cloud-based backend;
• It supports a variety of modern and classic diffusion model architectures;
• It covers local redrawing, image enlargement, real-time drawing, and super-resolution;
• It offers ControlNet, reference images, Regions, and LoRA control options;
• Retain traditional digital painting tools for manual fine-tuning.

Restrictions and Precautions

• Local models consume a large amount of disk space and video memory, and their installation as well as version compatibility are more complex than those of pure web-based tools.
• Plugins, ComfyUI, nodes, and models need to be updated accordingly; the performance and stability on platforms that are not NVIDIA-based vary significantly.
• AI-generated output may still contain errors in structure, wording, and consistency;
• Open source does not mean that the model can be used for commercial purposes freely, nor does it imply that third-party cloud services are free or that the data remains entirely on local devices;

Frequently Asked Questions

Is Krita AI Diffusion free?

Plugins and local operation are free and open-source; using one’s own hardware, cloud GPUs, or Interstice Cloud incurs corresponding costs.

Is it necessary to install ComfyUI?

It is necessary to connect to a compatible backend, but the plugin can be installed automatically; it is also possible to connect to an existing or remote ComfyUI instance, without the need to operate the node interface manually.

Is it compatible with Flux and SDXL?

It supports architectures such as Flux 2, Z-Image, SD 1.5, SDXL, and Illustrious; the specific capabilities depend on the version of the plugin and the models that are installed.

Is it possible to redraw specific areas and enlarge the image?

Yes, the selections created in Krita can be used directly to apply fills, remove objects, and expand the canvas.

Is it compatible with macOS and AMD?

The plugin interface is compatible with common platforms, but the performance and compatibility of the local backend depend on ComfyUI, PyTorch, and the hardware used; Windows and Linux versions on NVIDIA are the main platforms for testing.

Is the plugin open source?

It is open source, licensed under GPL-3.0-or-later.