Mapping Floor Plan Coordinates to Render/3DView Coordiantes

[shmuelzon]

Hello,

I'm trying to develop a plugin and, as a part of it, I want to get the pixel location of a piece of furniture in the rendered image.
I have the X, Y and elevation of the furniture and I managed to create a rendered 3D image, but I want to find the correlation between the two.

I found the `getVirtualWorldPointAt()` of `HomeComponent3D` which I thought would do just that but it crashes when I try to call it:

Code: Select all

HomeLight light = ...;
HomeComponent3D home3d = new HomeComponent3D(getHome());
float[] pos = home3d.getVirtualWorldPointAt((int)light.getX(), (int)light.getY(), light.getElevation());

Code: Select all

Exception in thread "AWT-EventQueue-0" java.lang.UnsupportedOperationException
        at com.eteks.sweethome3d.swing.HomeComponent3D.getCanvas3D(Unknown Source)
        at com.eteks.sweethome3d.swing.HomeComponent3D.convertPixelLocationToVirtualWorldPoint(Unknown Source)
        at com.eteks.sweethome3d.swing.HomeComponent3D.getVirtualWorldPointAt(Unknown Source)

I'm new to Sweet Home 3D and it's been decades since I wrote any Java so please excuse anything stupid I've done.

Thanks in advance!

[Keet]

Are you sure that getHome() returns a valid home object?
And that light is a valid object that can return coordinates?
And that light is actually in the furniture list of the home?

[Puybaret]

getVirtualWorldPointAt can work only if HomeComponent3D instance is added to a container and displayed at screen.

[shmuelzon]

Hey,

Thanks for the reply!
I've used getHome() in other places to actually get the list of furniture so I assume it's fine.
Same for the light, I'm able to change it's power when rendering the image so all the instances seem valid.

Even running:

Code: Select all

float[] pos = home3d.getVirtualWorldPointAt(40, 40, 40);

Results with the same exception, so I think something is off.
I'm not even sure if this function does what I think it does.

Thanks

[Daniels118]

That method returns the 3D coordinates for the given screen coordinates and elevation. From your first post I think you want the opposite. There is a method named getScreenCoordinates in the class it.ld.sh3d.Utils, you can find it in the Pan3dView plugin.

[shmuelzon]

Thanks, I found your plugin and am currently trying to understand what that function does.
From what I gather, so far, the 2D point it calculates is relative to the 3D view window, right? Meaning that if I render a high quality image with a different resolution than my current 3D view window, it won't match?

[Daniels118]

Yes, the output coordinates are relative to the 3D view. As long as the rendering uses the same proportions as the 3D view, computing the pixel coordinates is just a matter of applying a scaling factor (factor = render_width / view3d_width). But to cover all rendering options (different proportions, different lenses), you have to compute the right projection matrix. I think that you could find the relevant code in the classes that implement the rendering.
Once you have the projection matrix you can use it to transform 3D coordinates to screen coordinates (I remember that the official doc for Java3D has pratical examples of how it works, but any lecture about 3D projection works as well), and finally apply the scaling factor to get the exact pixel location.
I'm not sure that a projection matrix can cover spherical lenses too, but I guess that a quick web search should answer the question.

[shmuelzon]

Great, thanks for all the pointers!
I think I have enough to go on.

[shmuelzon]

Hey,

Unfortunately, I'm still stuck...
I was able to integrate the relevant parts of the Pan3dView plugin and can, accurately, get the pixel locations from the 3D view of the application but, since the rendered image might have a different aspect ratio, it doesn't help directly. I even tried to create my own Canvas3D object so I could defer those calculations to it but I wasn't able to get that to work either.

In any case, I have figured out that the SH3D's Y and Z axes are reversed, compared to the standard right-handed coordinates, so I switch between those values.
I also think that the SH3D's pitch goes in the other direction (clockwise instead of counter-clockwise) so that needs to be negated before performing the calculations.

I believe I do have the correct calculation for moving the object's location to be relative to the camera while taking the camera's position, pitch and yaw into account. For reference, this is what I have (playing around with Python, will convert to Java when it's working):

Code: Select all

object_location = np.array([749.2, 95, -2611.5])
camera_position = np.array([602.140686, 125, -4035.214600])
yaw = 0
pitch = 0

R_yaw = np.array([
    [np.cos(yaw), 0, np.sin(yaw)],
    [0, 1, 0],
    [-np.sin(yaw), 0, np.cos(yaw)]
])
R_pitch = np.array([
    [1, 0, 0],
    [0, np.cos(pitch), -np.sin(pitch)],
    [0, np.sin(pitch), np.cos(pitch)]
])
R = np.dot(R_pitch, R_yaw)
object_location_cam = object_location - camera_position
object_location_cam_rotated = np.dot(R, object_location_cam)

The final part, performing the actual perspective projection, is what I'm failing at. I found online a few different variants but none have worked well for me. One of them is:

Code: Select all

image_width = 1024
image_height = 576
near = 0.1
far = 100.0
aspect_ratio = image_width / image_height
f = 1.0 / np.tan(fov / 2)
projection_matrix = np.array([
    [f / aspect_ratio, 0, 0, 0],
    [0, f, 0, 0],
    [0, 0, -far / (far - near), (-far * near) / (far - near)],
    [0, 0, -1, 0]
])
point_4d = np.append(object_location_cam_rotated, 1)
projected_point_4d = np.dot(projection_matrix, point_4d)
normalized_point = projected_point_4d[:3] / projected_point_4d[3]
x_pixel = int((normalized_point[0] * 0.5 + 0.5) * image_width)
y_pixel = int((1 - (normalized_point[1] * 0.5 + 0.5)) * image_height)

The other is:

Code: Select all

image_width = 1024
image_height = 576
x_proj = (object_location_cam_rotated[0] / object_location_cam_rotated[2]) * (image_width / (2 * np.tan(fov / 2)))
y_proj = (object_location_cam_rotated[1] / object_location_cam_rotated[2]) * (image_height / (2 * np.tan(fov / 2)))
x_pixel = int(image_width / 2 - x_proj)
y_pixel = int(image_height / 2 - y_proj)

For the above example, I've rendered the image and the object in question appears at pixel (426, 303). The first calculation results with (463, 278) while the second comes very close with (425, 297).

I then proceeded with testing a different scenario with:

Code: Select all

object_location = np.array([1009.1, 0, -1071.6])
camera_position = np.array([3216.203613, 125, -1513.119751])
yaw = np.radians(90)
pitch = 0

Here I expect the pixel to be located at (375, 334) but the first method returns (417, 261) and the second (344, 314).
Can anyone suggest a projection matrix / calculation that's similar to the SH3D's renderers?

Thanks again!

[Daniels118]

Don't reinvent the wheel, the code to compute the projection matrix for specific render settings are in the rendering classes. You can look at, for example, SunFlow renderer:
https://sourceforge.net/p/sweethome3d/c ... .java#l601