[plejon]

I want cut a rectangular piece out of a photograph, but the corners should be rounded.

I'm looking for the effect of the Rounded Rectangle Tool, but in the Marquee tool. I tried using the "feather" option of the Rectangular Marquee, but this results in a gradient from my picture to the background. I want to have a sharp edge, not a feathered one.

I'm using Photoshop CS

[xgerryx]

Layout your rectangle as you want it and then go to the menu bar and Select> Modify> Smooth and choose your radius.

This was done with Elements, but I think it will be the same for CS.

[lister]

Make the shape using the Rounded Rectangle tool, if just using a non-filled path then go to the paths palette, right click on the path thumbnail and choose Make selection.
In the case of using a filled vector shape, then hold down ctrl+leftclick on the shape thumbnail.

[plejon]

xgerryx, this seems to be an Elements only technique.

Lister, your solution works fine and gives me the desired result. Thanks for the tip.