I've upgraded to 8gb of ram and turns out Aperture wants that too. Even just sitting in the background it's using more and more ram so I'm going to limit the app to 4gb and suffer through until Apple fixes the memory leak. It's a fairly common problem according to their support discussions.