It's not a problem in Photoshop, it's called antialiasing, it is required to get soft edges on transparent pictures. The problem could be that YafaRay doesn't handle half-transparent pixels correctly.
You could simply apply a threshold on the alpha layer to get rid of translucent pixels, see here:
https://graphicdesign.stackexchange.com ... ject-layer

PS. if you have the original picture, just turn off antialiasing when cutting the background.