I've noticed that when stopping down my new (to me) lens with a reduced opening at the end of the lens, the video can get MUCH more grainy.
I'm not sure if the camera, in Aperture mode, might be changing the ISO setting even though I have selected an ISO setting. (The aperture is set at f/5.6 because that's the only setting where auto focus will work with the lens.)
While trying to find out what may be going on I came across this test for the Canon 7D, after having read a little here and there that while film was based on 100s, the digital world is still based on 64s. That the best ISO settings are multiples of 160s. This test apparently proves it to be so, on the 7D at least. I'll be setting at either 160, 320, or 640 from now on.