Here's a few tips on how to
avoid the dreaded image buffer error. But first a little background on why it happens...

Unlike many other compositing and 3D programs, After Effects doesn't use a scanline renderer. Instead it renders each layer and then stacks it on top of the previously rendered layers. While this gives great performance for layers that can be cached and use several times without re-rendering, it can spell disaster when you are trying to work with bigger sources and output resolutions.
While you can use
Shake to zoom in on a giant 30.000 by 30.000 pixel image on an old machine, doing the same thing in After Effects takes a bit of imagination, plus a knowledge of how to tweak AE's memory settings. Basically, it comes down to the memory being to fragmented for AE to be able to hold the entire frame/layer in one contiguous piece of RAM. Here's how you avoid that from happening:
1. Set Your Preferences CorrectlyStart out by making sure you have set After Effects to use the optimal settings. These should always be your default values, and you should reset them according to these settings once you have gotten around the image buffer errors!
Maximum Memory Usage: 120%
Maximum RAM Cache Size: 60%
Enable Disk Cache: On
Maximum Disk Cache Size: At least 2 GB (I use 8-12 GB, typically)
Prevent DLL Address Space Fragmentation: On

The
Maximum Memory Usage tells AE how much of the total amount of installed RAM it can use as a maximum. Why
120% Maximum Memory Usage you might ask, why not 100% or even a sensible 95%? Well, since
running slow sure beats running out of memory, that's why! If you set it over 100%, AE will use the harddisk as virtual memory (slow but sometimes necessary.)
The
Maximum RAM Cache Size is the amount of RAM (set in "Maximum Memory Usage") that After Effects uses to load sources, render effects and composites into, plus to store already rendered frames in. The setting of 60% is a moderate one that over time has proven to work optimally for most projects. Lowering this value will slow down your rendering and previewing speed, but it is sometimes necessary in order to render large layers/comps.
The
Disk Cache is where AE can off-load already rendered images/frames/layers/comps onto your harddrive instead of throwing them away when you are running short on RAM. Note that AE will only use the Disk Cache if it is faster to read a frame from the Disk Cache than it is to re-render it.
If possible, put the Disk Cache on a fast harddrive, preferably not the same as your operating system or your source material (the above screen shot was taken on my laptop, where you seldom have the luxury of three drives.)
If you're on Windows, make sure you check
Prevent DLL Address Space Fragmentation, as AE will then try to keep more of the memory in larger chunks. It should be left on, unless it is causing an extremely rare crash (the reason it is not enabled by default.)
Make sure you turn off multi-processing in After Effects CS3 (and
Nucleo / Nucleo Pro if you have that plugin.)
As a final step,
avoid OpenGL previews, instead
use Adaptive Resolution.
OK, with these settings, try to see if you can preview/render the problematic comp. If not, continue with the next step...
2. Lower the Number of Undos
Every operation that AE has to be able to Undo takes up valuable memory space. Try to Purge Undos from the Edit menu. If that doesn't help, go into the Preferences and lower the number of Undos to just a few, or even just one step. (Remember to set it back to somewhere between 20 and 32 when you've gotten the renders done.)
Still having trouble? Then let's try the next remedy...
3. Lower the Image Cache: Maximum RAM Cache Size
Even though the Image Cache will speed up renders (by avoiding having to re-render stuff) it leaves less memory to be used for other things, which can lead to the fragmentation of the memory. Here's how
Michael Natkin of the Adobe After Effects team explains it:
"Yep, it is counterintuitive, isn't it! Here's the reason. Just like a
hard drive, your address space can get fragmented. So if you have the cache percentage set high, AE will try to use a lot of RAM and though you may have enough left for the next image buffer, it may be so fragmented that there is no place to put it. So by lowering the cache percentage, you reduce the fragmentation and paradoxically are able to fit that frame.
Lower the Image Cache bit by bit, until the error goes away. If you have it as low as it will go (10%,) there's only one option left...
4. Chop Up Your SourcesSplit the source image up into several layers (somewhere around 2.048 by 2.048 pixels each) in Photoshop, then import the PSD file as C
omposition Cropped Layers. Press your Caps Lock key on the keyboard (to prevent AE from trying to render a preview) and then change the preview mode to
Wireframe. Open up the composition, select all layers and change their mode to
Alpha Add. Finally copy all layers and paste them into a comp with the size of your final output.

It has worked for me with 16-bit images over 24.000 by 24.000 pixels.
- JonasLabels: AfterEffects, tricks, tutorials