I wonder if Yafaray is compiled with "-O3" optimizations ?

This flag is cited in YafaRay build flags but I'm not sure it's actually used (the last time I compiled YafaRay was some time ago). What improvements would it bring?

does Yafaray source code need some modifications to be used with SweetHome3D, or I just have to compile it and replace the libraries given with SweetHome3D ?

As explained in YafarayRenderer class, Sweet Home 3D uses a derived version of libYafaRay 3.5.1.
But it's not too difficult to generate Linux DLLs.