I’m currently porting some features of the iOS version of the Reality Augmenter back into OSX. This includes the ability to load images, and use custom masks made from any of source types.
So no longer will you be limited to the circle mask for projection mapping in OSX, you’ll be able to use videos, images and quartz compositions to mask your mapped areas.
Next release should be very soon, just tidying up a few odds and ends and need to make sure testing is up to date.