the bios instructions lotuseclat79 is suggesting applies to non-macintosh machines. Since you're using a mac you'll have to hold down "alt/option" button (should be next to your cmd and ctrl buttons). Once you're mac is off, make sure you press down "option" before you hear the chime (if heard at all), then you should see a grey screen with "Macintosh HD" (or w/e you renamed your hard drive to), any partitions of your HD, along with any bootable media (eg. your ubuntu USB)
http://en.wikipedia.org/wiki/Option_key