How to train a diffusion model to do image enchancement

[abdelkareemkobo]

can some guide me into how to train a diffusion model that can do an enhancement for unpaired images,
I have multiple cases and every case has a folder that contains multiple images for the same case but from different devices one in a lower resolution and one in a higher resolution.
I want to train a diffusion model that can do an enhancement for the low resolution image based on the known high resolution ones

[ChullMotors]

This review might be helpful? https://arxiv.org/pdf/2308.09388

[abdelkareemkobo]

it's helpful thanks

[asomoza]

actually what you want to do is what commonly the "Tile Controlnet" is used so your answer is probably that, train a controlnet but there's already really good ones, so don't know why would you like to re-invent the wheel unless you want the experience to learn which can be good since you can compare it to the existing ones.

[abdelkareemkobo]

I want to do the following : I have multiple cases of cancer images from a specific device (CT scan one) in dicom format and there is another device which produces images for the same case but in different time and most of the time for different parts in the body
for example i am have a low resolution image for lung and high resolution image for another part in the lung and want based on the data in the high resolution to improve the low one.
I don't think controlnet can hanlde this?
I don't want to train from scratch but fine-tune on my data

[asomoza]

I see, I don't think you will get any results with that data, it could work if it's the exact same part (image) with a different resolution but if they're not the same, the model won't learn anything or understand what to do.

You are saying that controlnet can't handle this but a fine tune can't either, probably the only way to know for sure is to just experiment with them, one method you can try is something similar to the instruct-pix2pix one since you can see how the dataset was made.

As I see it, what you need to do first is to fine tune a model with the CT scans so the model learns what they are, it doesn't matter if they're low resolution or not, this is the first step. Then you can try a finetune for "enhancement" but I haven't seen any super resolution diffusion model that are good by themselves for that task or that someone trained a good super resolution model without using the exact same image in low and high resolution.