Beyond Single-shot Light-in-Flight Imaging with a Silicon Image Sensor -- Toward X-ray and Infrared Ultrafast Imaging