Unfortunately that's not an option for us.. we absolutely can't write anything to disk, even temporarily.
In either case, it works fine without that folder with smaller files and always works in Silverlight mode regardless of filesize. If the uploader is trying to use the folder when we're using the custom upload provider in the other two modes.. that sounds more like a bug? I don't see why/how writing to disk would increase performance.