An AVI or MOV will not do it.

You need to render to a series of frames in a file format that supports alpha channel like .PNG, .TGA or .TIFF. When you specify a file format in the Save As dialog of the Render dialog 3DSMax will prompt you with a configuration dialog for the file format. The previously mentioned ones have a check box for saving the alpha channel, make sure it is checked.

Also, your scene has to not have geometry blocking the background. If you create a sphere and put a camera in the center of it, you will not get an alpha no matter what because all the camera could see in that case would be geometry.