JavaScript: Trouble assigning tangents to vertex buffer

[BStringhamVRSK]

@romainguy -

We are using SurfaceOrientation.Builder to generate the tangents for dynamically generated geometry. We have a version on PC/OpenGL that works fine, but our WebGL version has problems. Some of the lighting is showing problems that suggest that the tangents are not making it into the vertex buffer correctly.

For example, some of the "walls" are being lit as if their normals point vertically instead of horizontally. In this screen shot, all of the walls are essentially white to make the IBL lighting apparent. You can see the wall facing the camera is receiving yellow light (from above), when it should be getting green light (from behind the camera).

Since some of the surfaces seem to be lit correctly, I suspect the problem is assigning the tangent data into the vertex buffer, where we use HALF4s to store the tangents. Here is our vertex buffer format:

All code is in JavaScript/TypeScript.
vertexBuffer = Filament.VertexBuffer.Builder()
.bufferCount(2)
.vertexCount(numVerts)
.attribute(Filament.VertexAttribute.POSITION, 0, Filament.VertexBuffer$AttributeType.FLOAT3, 0, 20)
.attribute(Filament.VertexAttribute.UV0, 0, Filament.VertexBuffer$AttributeType.FLOAT2, 12, 20)
.attribute(Filament.VertexAttribute.TANGENTS, 1, Filament.VertexBuffer$AttributeType.HALF4, 0, 8)
.build(engine);

Here is our tangent generation code:

        var sob = new Filament.SurfaceOrientation$Builder();
        sob.vertexCount(count);
        sob.normals(newNorms, 0);
        sob.uvs(newUVs, 0);
        sob.positions(vertices, 0);
        sob.triangleCount(triCount);
        sob.triangles32(newTriangles);
        var orientation = sob.build();
        var quatsBuffer = orientation.getQuats(count);

Here is the code to set the vertex buffer

        var size = 5;  // 5 float elements: position XYZ and UVs
        var newVerts = new Float32Array(size * count);
        for (var idx = 0; idx < count; ++idx) {
            //Positions
            newVerts[idx * size] = vertices[idx * 3];
            newVerts[idx * size + 1] = vertices[idx * 3 + 1];
            newVerts[idx * size + 2] = vertices[idx * 3 + 2];

            //UVs
            newVerts[idx * size + 3] = texCoords[idx * 2]
            newVerts[idx * size + 4] = texCoords[idx * 2 + 1]
        }
        // Set positions and UVs into buffer 0
        this._vertexBuffer.setBufferAt(engine, 0, newVerts, 0);
        // Set tangents into buffer 1
        this._vertexBuffer.setBufferAt(engine, 1, quatsBuffer, 0);

I don't know of a way to see what values Filament is actually getting. SurfaceOrientation.Builder.GetQuats() returns signed shorts, which I assume are going to be interpreted properly as half-float values, but I don't know how to confirm that they are not getting cast into something incorrectly along the way.

Any thoughts would be very appreciated. I wouldn't be surprised if I am just missing a small element, but I can't figure out what's different between this and our PC/OpenGL implementation.

Thank you.

[romainguy]

You should pass your tangents in the following format:

            .attribute(VertexAttribute::TANGENTS, 1, VertexBuffer::AttributeType::SHORT4)
            .normalized(VertexAttribute::TANGENTS)

The current JavaScript extension seems to always select SHORT4 as the type of the quaternions. The internal _getQuats() function lets you choose other formats. That said, SHORT4 is going to be the most efficient one.

[romainguy]

CC @prideout

[BStringhamVRSK]

I think it might be working now! I'll do some more testing.

Funny thing is, I tried SHORT4 at one point, but I never had the normalized set for tangents. The combination seems to have things working.

Thanks again for your speedy response.

[BStringhamVRSK]

I unmarked this as the answer, because, non-textured polygons seem to work fine, but anything with a texture and normal map appears to have bad tangent frames. See my possibly related bug report, which contains screenshots, here: #4441

In my mind, the possible culprits are:

1. Me - I'm giving bad information to the SurfaceOrientation.Builder. I've looked in the debugger, and I don't see any bad data going in.
2. Me - I'm still not pushing things properly into the vertex buffer. Setting the tangent attribute to SHORT4 and marking it as normalized definitely improved things, but the tangent frame is noticeably distorted when normal maps display.
3. SurfaceOrientation.Builder - Perhaps it is calculating the correct tangents, but the values are getting cast badly when getQuats() returns them in an Int16Array?
4. VertexBuffer.setBufferAt() - Perhaps it is casting or interpreting the tangent values incorrectly when I pass in the Int16Array of tangents?

I can't think of any other options, and I have done a ton of iterations trying to discover the problem.

Any suggestions?