Suggestion for efficient animation workflow

[arvidtappert123]

Hi!

I've been working extensively with frame-by-frame animation in Krita-ai-diffusion, and I really appreciate the record button that adds the generated image to the timeline. However, I'm missing a feature (or have I just missed it?) that would automatically advance to the next frame in an animation in the timeline once an image is generated and add all the generated images to the timeline for a imported animation. For instance, if I want to generate new images for a 2D animation with 400 keyframes, I currently need to do it manually for each frame, which is quite time-consuming.

It would be incredible to have a function that facilitates this process, making the animation workflow much more efficient.

Thanks /Arvid

[Acly]

Hey,
not sure I understand it correctly yet. This is not directly related to live/record, but rather a kind of batch processing, yes?

You have an existing animation, let's say an animated layer with 400 keyframes. You want to generate an image for each keyframe with the same parameters. The results could then be inserted into the same timeline, but on a different layer.
Does that sound right?

[arvidtappert123]

Yes that’s exactly right! :)

[Acly]

I did some initial work for animation things, see #434

Some of this I had sort-of planned, but wanted to spend more time on it. No sure when I will get around to it though. But this version may already help in your case, and might be a stepping stone for future extensions.

[Acly]

It might be related to the "end time" set in the animation timeline? It probably needs to match an existing keyframe.
I hope it's not truely random. If you find a way to save a .kra which will trigger the issue once Generation Animation is used and you can share it that would be helpful.

As workaround you should also be able to import the generated frames manually from files, they're stored next to the .kra file.

[arvidtappert123]

Thanks, great that there´s a workaround. I added a .kra file, hope it helps!
animtest.zip

[arvidtappert123]

I changed so the end time in the timeline matches the animation and that seems to work fine

[arvidtappert123]

I tested it further and it would be great if batch generation of images/keyframes including all layers were possible. For example, when I use a transparent animation in one layer over a background photo, the generated animation ends up with a black solid background. It would be even more impressive if I could select a control net and its strength for the background layer separately from choosing another control net for the animation layer.

[Acly]

Generate from single layer is a limitation of Krita's python API - I can get a single layer's content at arbitrary keyframe, but the merged canvas only at the current frame (what's displayed on the screen). That makes it tricky. I think for now you will have to merge the layers before batch-generating frames.

This is mostly an issue for arbitrary number of layers with any kind of blend mode. It might be interesting to think of foreground + background with "normal" alpha blending as a special case. Generating the background separately should be straight-forward. But forground requires some understanding of transparency / layers in the diffusion model, since it has to take the background into account without modifying it. I think just masking will not be good enough.

Coincidentally this paper was published yesterday: https://arxiv.org/abs/2402.17113
No models released yet, but I'm very interested. It could be a way to generally make image generation more layer-aware and and a solution for theses kind of scenarios.

[noise2014]

can depth map and normal map and other controlnet features be added to animation option? there is add layer but it does nothing.

[Acly]

Should work in v1.22.0 (only tested briefly with scribble)

[noise2014]

Hello , for animation consistency would please consider support for animatediff and all its versions?
thanks alot.