Any help from the experts here would be great.
I filmed a man behind a green screen using a sony digital recorder. I will capture the digital video using a Sony Digital VCR. It will be captured by a Macintosh through Final Cut or Premiere. My final output will be eithere a quicktime movie or a DVD movie. I want the compression to be very minimal in the begingin for when I bring it into After Effects to drop out the background. Which compression should I use in the begining to ensure the edges of the man and the quality stay at a professional level? Thanks.
It sounds as if you are starting with a very compressed source. Am I correct to assume when you say Sony Digital VCR, do you mean DV format? DV25 applies a 5:1 compression to every frame. And it's low color sampling (4:1:1) means not much color resolution is recorded (making it notoriously difficult to get a great key).
However, I've heard AE 6's Keylight is spectacular. If you need an immediate low cost solution, I'd like to recommend DV Garage's DVMatte. It's made especially for DV sources.
<http://www.padd.com/market/product/product.php?prod=dvmatte>
You would need a non-compressed SD board in your Mac if you wish to capture at higher data rates. Of course your tape format would have to be non-compressed like D5 or mildly compressed like DigiBeta or DVCPRO50
To answer to your question, in order to maintain the best quality, you should use a non-lossy codec. Quicktime Animation at millions and best (100) quality is lossless. So is Quicktime None, PNG, TIFF and TGA. You can also look at 3rd party codecs like Digital Anarchy's Microcosm. It can achieve higher lossless compression rates than Animation. It also supports high-bit color. Does this help at all?
Yes, Microcosm is great. Worth every penny.