Project 02 / 04 · UAL MSc · Group · 2026 · Grade A

Borderless

An interactive VR world built from user sketches — draw an element, watch it rendered in watercolour by a fine-tuned diffusion model, and placed into a shared 360° environment in real time.

My roleData pipeline, LoRA training, ControlNet, inference backend

StackPython, SD 1.5, ControlNet, LoRA, rembg, Flask

TeamAlina Kuznetsova, Julia Wojacka, Shani O'Mallo-Mitchell

Repogit.arts.ac.uk/…/MML-Project

The Project

A world built together, one sketch at a time. Users draw. The model imagines. The world grows.

Borderless invites users to build a shared illustrated world. Pick a category — mushroom, flower, tree, mountain, cloud, or planet — sketch your own version on a browser canvas, and a sketch-to-image pipeline transforms it into a painterly watercolour asset. Each generation is automatically placed into a 360° parallax environment built in Three.js and explorable in VR.

The project was built as a group of three. My work covered the full ML side: curating and cleaning the training dataset, training four versions of a LoRA fine-tune on Stable Diffusion 1.5, integrating ControlNet for sketch conditioning, and building the inference backend that connects the pipeline to the browser.

My contributions

Dataset collection, cleaning, and captioning pipeline
Four rounds of LoRA training and evaluation
ControlNet integration with per-category strength tuning
generate.py — inference pipeline (SD 1.5 + ControlNet + LoRA + rembg)
watcher.py — file watcher bridging browser sketches to the model

Team contributions

Concept development and aesthetic direction (all)
Dataset image collection — two categories each (all)
Three.js / WebXR parallax world (Julia)
Flask server and browser sketch interface (Shani & Julia)
Final integration and demo (all)

Data Pipeline

Six categories. ~600 images. Built to train a single consistent aesthetic.

We agreed on a painterly, watercolour illustration style as the aesthetic target — a look inspired by botanical illustrators like Vincent Jeannerot. Each of us collected images for two of the six categories via shared Pinterest boards, scraping them with a custom script.

I then built the cleaning pipeline. First, remove_backgrounds.py used rembg to strip backgrounds from every image, producing transparent PNGs. Then resize_images.py standardised everything to 512×512. Finally, caption_images.py paired each image with a category-specific caption in the format "a stylized [category], watercolor illustration…" agreed on as a group — the text that would guide training.

Background removal turned out to be inconsistent: rembg handled well-defined subjects cleanly, but struggled with flowers that blended softly into their backgrounds, producing ghosting artefacts. Those images were removed from the dataset.

Sample images — mushrooms, flowers, clouds, trees. Scraped from Pinterest, cleaned, resized to 512×512.

Incomplete

Ghosting around soft edges — unusable.

Clean

Hard-edged subjects kept in dataset.

LoRA Training

Four versions. Each one fixing what the last got wrong.

We trained four LoRAs on Stable Diffusion 1.5 before settling on V4 as the final model. After each run I evaluated outputs across all six categories using test_lora.ipynb, and the group agreed on what needed fixing before the next training run.

LoRA V1

Transparent backgrounds, rank 16, 1500 steps

Elements blended into the background rather than reading as distinct subjects. Many training images had multiple objects per frame, so the model struggled to generate single well-formed subjects.

Issues: blending, multi-subject

LoRA V2

New dataset, white backgrounds, rank 16, 1500 steps

The single-subject dataset fixed the blending problems — outputs were clean and distinct. But at rank 16 and 1500 steps the model was slightly undertrained, lacking the painterly brushstroke detail we were aiming for.

Issues: undertrained, lacks texture

LoRA V3

Same dataset, rank 32, 3000 steps

Doubling the rank and steps brought through brushstroke detail and noticeably stronger watercolour quality. One remaining issue: tree outputs were still struggling to form clean, recognisable shapes.

Issues: trees inconsistent

LoRA V4 — Final

Pruned tree subset, same config as V3

Removed tree images that were too complex or compositionally ambiguous from the dataset. Retraining fixed the tree outputs — all six categories generating cleanly. This became the version used in the final demo.

Final version used in demo

ControlNet

Conditioning the generation on the user's sketch.

I integrated ControlNet scribble conditioning so that user sketches would guide the generation — not just the prompt. The conditioning strength turned out to be a real balancing act. Too low, and the output ignored the sketch entirely. Too high, and the generation looked mechanically traced — the painterly quality collapsed.

I found the optimal value varied per category and set different strengths accordingly after iterative testing. We also added a colour feature that injects the user's selected colour into the prompt, so even when the shape drifts slightly, the output anchors to what the user had in mind.

Per-category conditioning strengths

Flower0.95

Planet0.90

Mountain0.90

Mushroom0.85

Cloud0.85

Tree0.72

User sketch

Strength too high — over-traced

Correct strength — painterly

The Pipeline

From sketch to world. How the full system fits together.

watcher.py monitors the Downloads folder for new sketches. When one arrives, it parses the filename for category and colour, and calls generate.py — which runs SD 1.5 + ControlNet + our LoRA, then strips the background with rembg. The resulting transparent PNG is picked up by the Flask server and placed automatically into the Three.js world.

01 world.html User draws a sketch and selects a category and colour in the browser

02 watcher.py Watcher detects the new file, parses category and colour from the filename

03 generate.py SD 1.5 + ControlNet scribble + LoRA runs inference; rembg strips the background

04 Outputs/[category]/ Transparent PNG saved to the correct category folder

05 server.py Flask server picks up the new asset and serves it to the browser

06 world.html Three.js places the asset at the correct parallax depth layer in the world

The World

A 360° parallax environment. Explorable in VR.

Once generated, each asset is placed automatically into a shared Three.js environment built by Julia. Categories sit at fixed depths — planets and mountains in the background, flowers and mushrooms in the foreground — creating a strong sense of depth even though all assets are 2D. When the world fills up, users can step inside via a WebXR headset.

We had originally aimed to place assets as 3D meshes using TripoSR, but couldn't get reliable output — every tool we tried produced distorted geometry. The 2D parallax approach was the pivot, and in retrospect it suited the watercolour aesthetic better anyway.

Reflection

What the iterative process taught me — and what we didn't fully solve.

What worked

Iterative LoRA training

Running four versions with targeted fixes between each run produced a noticeably stronger model than any single training run would have. Each iteration addressed a specific identified failure — transparent backgrounds, multi-subject data, undertraining, tree complexity — rather than just adjusting hyperparameters blindly.

The aesthetic landed

The watercolour style came through consistently across all six categories, making the final world feel cohesive rather than like a set of mismatched generations. Getting that consistency took careful dataset curation as much as model training.

Knowing when to pivot

When 3D mesh generation failed, switching to 2D parallax layers was the right call. It was faster to implement, more visually stable, and ultimately suited the watercolour aesthetic better than textured meshes would have.

What we didn't fully solve

The background removal / generation quality loop

When the LoRA produced partially-formed outputs with soft edges, rembg couldn't cleanly separate subject from background — leaving half-transparent shapes in the world. We added a 'delete asset' feature as a workaround, but it slightly undermines the seamless experience we'd aimed for.

ControlNet conditioning

Even after per-category tuning, generations sometimes drifted from the user's sketch. Turning the strength up to compensate made outputs look overly constrained. Striking a consistent balance between structural fidelity and painterly quality was difficult to achieve across every generation.