Setting the file. One moment.
Chapter 16 · Huggingface Lora Space Builder
Subchapter 16.1
references/adapting-to-the-lora.mdMarkdown14 KBView on GitHub
The task category (T2I, I2I, T2V, I2V, V2V) gets you a starting shape. It does not get you a UI. Two LoRAs in the same task category can need very different demos. This file is about the reasoning that gets you from “I know the task” to “I know what this demo should look like.”
For each LoRA, ask: what does this LoRA actually need from the user, and what’s the most natural way for the user to provide it?
These are different questions. The model might need a pose video as conditioning. That doesn’t mean the user has to provide a pose video — they can provide a regular video and the demo extracts pose from it. The model might need a black-bordered video for outpainting. That doesn’t mean the user uploads a pre-bordered video — they upload a normal video and pick an aspect ratio, and the demo adds the borders.
The job of the demo is to translate between what the user has (a video, a photo, an idea) and what the model wants (pose conditioning, padded frames, masked latents, a prompt prefixed with a trigger word).
Sources, in order of usefulness:
The model card’s example code snippet — for parameters. If the README has a Python block showing how to call the LoRA, trust it for inference parameters: pipeline class, step count, guidance scale, true CFG, LoRA scale, dtype, resolution, negative prompt. Trust it for inputs: if it passes image=..., the demo takes an image; if it passes image=... and mask_image=..., the demo needs both.
For loading mechanics, treat the snippet as a signal, not a directive. Prefer the standard diffusers path: pipe.load_lora_weights(repo_id, weight_name=...). This is the maintained, well-tested path that handles DoRA, rsLoRA, custom target modules, and most format variants when diffusers + PEFT are recent enough. If the model card uses something else — PeftModel.from_pretrained(pipe.transformer, ...), diffsynth_engine, custom imports, manual state-dict surgery — that’s a flag to investigate, not to copy.
Reasons model card authors reach for non-standard loading paths often don’t transfer: training-time conventions, environment quirks (older diffusers/PEFT versions, CPU offload patterns), or even malformed configs that pass silently in their setup but crash elsewhere. (Real example: an adapter_config.json with task_type: "DIFFUSION" works locally on some PEFT versions but crashes on current PEFT, because PEFT’s TaskType enum only contains NLP tasks; diffusers’ loader bypasses this validation by reading the safetensors directly.)
Custom inference paths absolutely can work on ZeroGPU when needed (LTX-2.3 native pipeline is a real example). But default to diffusers because it’s the standard, current, maintained path. Only adopt the model card’s loading approach if load_lora_weights demonstrably can’t handle this LoRA — and when you do, port it to the ZeroGPU constraints (module-level .to('cuda'), no enable_model_cpu_offload).
Trigger words and prompt patterns. “Use the trigger word X at the start of the prompt” → automatically prepend X to the user’s prompt rather than asking them to type it. “Prompts should describe the scene as Y” → add prompt formatting examples or a placeholder. If the LoRA expects a structured input embedded in the prompt (like bounding-box coordinates or named regions), the UI should produce that structure for the user, not require them to type it.
Example media in the repo. Example outputs tell you what the LoRA does. Example inputs tell you what the user has to provide. Example paired inputs+outputs (input video → output video, input image → output image) tell you the transformation. Lift these into gr.Examples so the user can click through them.
The model’s task family. A pose-conditioned model wants pose maps. A depth-conditioned model wants depth. An outpaint model wants padded frames with masked regions. Each implies preprocessing.
Recommended hyperparameters. Step count, guidance scale, true CFG, LoRA scale. Bake the recommended values in as defaults. Expose a slider only if the LoRA’s behavior is sensitive to the value across a range (e.g. LoRA scale 0.7–1.3 produces meaningfully different results); otherwise it’s just clutter.
When the model card has none of these, you have three options:
The pipeline class is decided in Phase 2, not here. The procedure (read the base model’s own card, trust its diffusers snippet over reference tables) is in SKILL.md under “Phase 2 — Pick the base pipeline.” Mentioning it here as a pointer because skipping that verification is one of the most common ways to ship a broken Space — when in doubt, re-read Phase 2.
For each thing the LoRA expects as input, decide:
Where does the user get this from? A pose video — extract from a regular video, or accept a pre-extracted pose video, or both? An aspect ratio — picker, sliders for width/height, or auto from input? A reference image — separate slot, or embedded in a “drag and drop here” area?
Can a more natural input be transformed into this? Almost always yes. A user has a video, not a pose video. A user wants “wider”, not “padded with black bars at these specific coordinates”. The demo bridges the gap.
Should the user see the intermediate? Often yes — a pose extraction preview, a letterbox preview, a generated mask preview. Showing the intermediate builds trust (“yes, the model is conditioning on what I expected”) and helps the user iterate. But a preview that takes 10 seconds to generate every time the user changes a setting is a worse UX than no preview.
What’s the smallest set of controls that lets the user actually drive this LoRA? Anything beyond that is clutter. A LoRA that’s only good at one specific transformation might need just an input slot and a Generate button.
Pose-control video LoRA (V2V). The model conditions on pose video. The user has a regular video. The demo takes a video, extracts pose, optionally takes an appearance reference image (for “the character looks like this, doing that motion”), runs inference, returns a video. The pose extraction is shown as a preview so the user knows what’s being used. Aspect-ratio picker is irrelevant — the output matches the source.
Outpaint video LoRA (V2V). The model fills black-margined frames. The user has a video and wants it wider/taller. The demo takes a video, takes a target aspect ratio (dropdown: 16:9, 9:16, 1:1, etc.), pads frames with black bars to that aspect ratio, shows a preview of the padded first frame, runs inference. No appearance reference — the LoRA’s job is to extend, not transform. If the LoRA’s model card mentions that gamma correction helps for dark scenes, expose that as an Advanced toggle.
Relight image LoRA (I2I). The model relights based on prompt. The user has a photo and an idea of what lighting they want. The demo takes an image, takes a brush canvas where the user paints colored strokes indicating where light should come from, takes an illumination style dropdown (“golden hour”, “neon”, “studio”), takes an optional background change, builds the prompt, runs inference. Brush color and position become structured prompt content.
Style image LoRA (T2I). The model produces images in a specific style. The user has a prompt. The demo takes a prompt (with the trigger word auto-prepended), takes an aspect ratio, runs inference. That’s the whole UI. No reference image, no brush canvas, no preprocessing — the LoRA does the work.
Bounding-box drag-drop LoRA (I2I). The model moves and resizes objects between two bounding boxes drawn on the image. The user has an image and an intent (“move that vase from here to there”). The demo takes an image, lets the user draw two boxes (red for source, green for target) directly on the image with a custom canvas component, runs inference. The boxes become the structured input the model expects.
Identity-preserving I2V. The model animates a character while preserving identity. The user has a still photo and a motion intent. The demo takes an image (the character), takes a prompt (the motion), runs inference. If the model also accepts a driving video for motion, expose that as an alternative input mode rather than a second mandatory input.
The pattern in all of these: start from what the user has and what they want, and build the UI that bridges to what the model needs.
Signals from the model card or the LoRA’s behavior that should change the UI:
progress(0.3, desc="Extracting pose...") then progress(0.6, desc="Generating...")). Otherwise the user stares at a blank progress bar for a long generation.@spaces.GPU(duration=...) higher and warn the user in the UI that generation takes longer.The right move when pipe.load_lora_weights(...) fails is to read the error and recognize the category of failure. Each category has a different fix path. Don’t guess — different errors imply different things about whether the LoRA is salvageable on the diffusers path at all.
Don’t preemptively call conversion utilities. load_lora_weights already calls the appropriate converters internally for the formats it knows about. Calling convert_state_dict_to_diffusers or similar before there’s an error is redundant in the common case and risky if you guess wrong — you can mangle a state dict that would have loaded fine.
Failure categories:
Config-validation failure (errors mentioning task_type, peft_type, “Invalid task type”, PeftConfig, or anything from peft/config.py): the safetensors weights themselves may be fine; adapter_config.json is the problem. Fix paths: pass weight_name= explicitly so load_lora_weights reads the safetensors directly without going through PEFT’s strict config parser; or download the safetensors and load via state dict. Falling back to PeftModel.from_pretrained will not help — that path crashes on the same config.
Missing keys / unexpected keys (“Loading adapter weights from state_dict led to missing keys” or “led to unexpected keys”): the state dict’s key naming doesn’t match what diffusers’ loader expects. This often means the LoRA was trained with a non-diffusers convention (kohya, ComfyUI, OneTrainer, custom training scripts), or with custom target modules. Fix paths: try diffusers’ built-in conversion utilities (convert_state_dict_to_diffusers, base-model-specific converters in diffusers.loaders); if those don’t help, the format may not be supported yet — surface this clearly to the user rather than silently using a partial load.
Shape mismatch errors: the LoRA’s tensor shapes don’t match the base model’s. Usually means the LoRA was trained against a different base model variant than the one being used (e.g. trained on Qwen-Image-Edit but loaded against Qwen-Image, or trained on FLUX.1-dev but loaded against FLUX.1-schnell). Fix: check the model card’s base_model field carefully and switch to the correct base.
OOM during load or first inference: not really a loading failure — the LoRA loaded but the combined base + LoRA + activations don’t fit. Fix paths involve pipe.enable_vae_tiling(), smaller resolutions, FP8/quantized base variants. Not in scope for this section.
Missing keys for some weights only (e.g. text encoder LoRA missing but transformer LoRA present): often a partial-coverage LoRA that only targets one component. May actually be intentional and may still work — generate a test image and see if the LoRA effect is present.
When none of these fit and load_lora_weights simply doesn’t work, falling back to a non-diffusers path becomes a real option. At that point the model card’s snippet becomes more useful — but port it to ZeroGPU constraints (no enable_model_cpu_offload, module-level .to('cuda'), models on cuda outside @spaces.GPU).