Just in case anyone else with a similar problem stumbles on this thread, here's what transpired:
After deciding on the scale-up/scale-down method to get 0.2 pixel accuracy on translation, I needed to rotate the images by small amounts to line everything up. That's when I found that the 'rotate' function in Photoshop has a similar problem - it only seems to work to one decimal place (for a layer) or two (for the whole image). On a 3000-pixel-wide image the rounding errors produce a jitter in the animated sequence. Of course with rotation, upscaling and downscaling don't help - an angle is an angle whatever size the image is. So, back to the drawing board....
I finally solved the issue by avoiding Photoshop altogether for this particular exercise. I wrote my own program (in VB.NET) to process the images - the .NET environment provides matrix functions to shift images by amounts down to .0000001 of a pixel (or whatever the smallest 32-bit floating point number is!) and to rotate and scale with the same accuracy. There's no need to upscale & downscale, so the program works fast - less than a second per frame to load, shift, rotate, crop, resize, rotate 180, and save. The results are indistinguishable in quality from those produced by the Photoshop script, and the time-lapse sequence is now rock-steady. Hooray!