To open an mp4 file in VirtualDub, you need to convert it to avi format. Both mp4 and avi are containers and not codecs, so the mp4 file you’re using may or may not have a codec that’s supported by VirtualDub. But in any case, converting to avi is definitely necessary. All you need to do is download mp4cam2avi, convert it to avi and try to open it in VirtualDub. My mp4 was encoded in x.264 and I had installed the x.264VFW which allows me to open those videos in VirtualDub. I got this information thanks to this forum thread.