Has anyone actually been able to achieve real-time video generation?

[AhmadAFS1]

Hi all,
I'm currently testing out this musetalk library on a gtx 1060 with only 3gb VRAM.
While I am impressed with the high quality results, it takes way too long to generate a short video.
I'm waiting 1-2 hours just to generate a 20 second clip at 15 fps.

I'd like to know for those that are able to experience realtime responses, like maximum 1-2 second response time. I'd like to know what your hardware configuration is, and if you made any optimizations to the musetalk app in order to achieve faster results.

Thank you

[aparate123]

I was able to achieve realtime response on H100 GPU at 30 FPS, it used around 10-15 GB of VRAM. Your current configuration is too old and with insufficient VRAM it takes a very long time. Try upgrading your setup to 3090 / 4090 / 5090. Although you will have to play around the source code to make it real time, like increasing the batch size.

[AhmadAFS1]

I was able to achieve realtime response on H100 GPU at 30 FPS, it used around 10-15 GB of VRAM. Your current configuration is too old and with insufficient VRAM it takes a very long time. Try upgrading your setup to 3090 / 4090 / 5090. Although you will have to play around the source code to make it real time, like increasing the batch size.

Awesome to know!
Thank you. Lately I have been using an rtx 4090 to run musetalk, and I'm getting good results after making some changes. I can get realtime inference on 15fps, 20fps struggles. a bit.
Maybe sometime I'll try an H100, although it's so expensive lol.

[aparate123]

Thats good to hear. Increase the batch size and monitor the VRAM usage. you will be able to get more than 20fps.

[EvgeniyGunko]

Rtx 5060Ti can do about 15-20 fps or more