Towards Controllable Relighting

Relighting is the problem of changing how a photo is lit while leaving everything else alone: same subject, same pose, same camera, only the light moves. My bachelor thesis is a comparative study of how to do it, spanning two very different architecture families. On the diffusion side I steered a frozen Stable Diffusion with Concept Sliders (LoRA), IP-Adapter with numeric light conditioning, and a ControlNet I trained myself on albedo maps extracted from Laion-Aesthetic with Adobe's RGBX. On the transformer side I tested image-to-image restoration backbones (DAT and Restormer) that don't hallucinate content and are much closer to what "edit this photo" actually asks for.

The winner was Restormer with a PromptIR-style light prompt injected into the cross-attention: pan and tilt go through a small MLP into a light embedding, dynamically re-weighted, and fed into Restormer's cross-attention modules at every layer. Trained on RSR, a 5k-image paired dataset I built with a rig of 9 controllable lights labelled A to I (so any two photos of a scene differ only in illumination), the model reached a mean PSNR of 28.66 and SSIM of 0.785 across every source-to-target lighting pair, and generalised to out-of-domain iPhone photos of mugs, boxes and plush toys on random surfaces.

The takeaway is that the choice of architecture matters as much as the choice of conditioning. Diffusion models bring visual priors but fight you on structure and control; transformer image-to-image models sit much closer to the shape of a relighting task once you give them a clean channel to consume the light parameters. Sometimes the right move is not to squeeze a knob onto the biggest model in the room, it is to pick the model that already fits the problem.