I used the following steps to get a permanent sort of fix. The challenge is that even if you delete the files, Sonoma downloads them again to fill up the disk really quickly, which is really annoying.
To stop this from happening is a 3 step process.
Step 1: You select a static picture in the wallpaper rather than the dynamic one. Go to System Settings/Wallpaper and select a picture.

Step 2: Delete the .mov files in /Library/Application Support/com.apple.idleassetsd/Customer/ folder. My files were under 4KSDR240FPS folder.
Step 3: This is the most important step of all.
ps -aef | grep Wallpaper

Once you get the process IDs, kill them all.
This seems to be solving the problem for me now.
Someone can use a script at logon to perform the above steps.
Happy storage recovery.