Usecase of Bullet Physics with Filament

[inveders]

Hello,
I heard in another conversation here, regarding raycasting, that you can use Bullet Physics with Filament to detect a collision between two elements. My question is about touching the screen. I don't have .gltf or .glb objects, but a rectangle that I draw using filament. The anchor I use to place my rectangle is only on point A of my rectangle, at the bottom left. I would like to know if, after having drawn my rectangle, by touching my screen, I can detect the corresponding point on the surface of my rectangle thanks to Bullet Physics? Or does it only work with .gltf or .glb objects?

Perhaps there are other solutions to do this directly with Filament?

[romainguy]

If all you have is a rectangle, Bullet will be overkill (note that Bullet is independent of the file format, it operates on arbitrary meshes). In your case you could apply the inverse projection to go from a point on screen to world space coordinates. Then using the ray in world space coordinates you can compute the intersection point with your rectangle.

[inveders]

Thanks you @romainguy, with your indications I tried to do that :

1. Find the intersection point

fun intersectRayWithRectangle(
        xPx: Float, yPx: Float, frame: Frame?, ray: Ray?
    ): Boolean {
    
        val out = screenPointToWorldRay(xPx, yPx, frame)

        out?.let {

            val rayOrigin = Vector3(out[0], out[1], out[2])
            val rayDirection = Vector3(out[3], out[4], out[5])
   
            val aPoint =
                Vector3(HomeFragment.aPoint.x, HomeFragment.aPoint.y, HomeFragment.aPoint.z)
            val bPoint =
                Vector3(HomeFragment.aPoint.x, HomeFragment.bPoint.y, HomeFragment.aPoint.z)
            val dPoint =
                Vector3(HomeFragment.dPoint.x, HomeFragment.dPoint.y, HomeFragment.dPoint.z)

            // 1. calculate the normal vector by the cross product
            val dBA = Vector3.subtract(bPoint, aPoint)
            val dDA = Vector3.subtract(dPoint, aPoint)
            val n = Vector3.cross(dBA, dDA)

            // 2. Calculate the t and the point of intersection
            val denom = Vector3.dot(n, rayDirection)
            if (abs(denom) < 1e-6f) { // Choose your tolerance
                return false
            }

            val pARayOrigin = Vector3.subtract(aPoint,rayOrigin)
            val t = Vector3.dot(pARayOrigin,n) / denom
            if(t<0){
                return false
            }

            val mIntersectionPoint = Vector3.add(rayOrigin, rayDirection.scaled(t))
            intersectionPoint = mIntersectionPoint

            // 3. Check if the intersection is in the rectangle
         /*   val dMA = Vector3.subtract(mIntersectionPoint, aPoint)
            val u = Vector3.dot(dMA, dBA)
            val v = Vector3.dot(dMA, dDA)
            // 4.
            return u >= 0.0f && u <= Vector3.dot(dBA, dBA) && v >= 0.0f && v <= Vector3.dot(
                dDA,
                dDA
            )*/
            return true
        }
        return false

    }

2. and use this method to convert the screen point to world ray

private fun screenPointToWorldRay(xPx: Float, yPx: Float, frame: Frame?): FloatArray? {
        val points = FloatArray(12) // {clip query, camera query, camera origin}
        // Set up the clip-space coordinates of our query point
        // +x is right:
        points[0] = 2.0f * xPx / arSceneView?.getRenderer()?.surfaceView?.measuredWidth!! - 1.0f
        // +y is up (android UI Y is down):
        points[1] =
            1.0f - 2.0f * yPx / arSceneView?.getRenderer()?.surfaceView?.measuredHeight!! //TODO debago handle it
        points[2] = 1.0f // +z is forwards (remember clip, not camera)
        points[3] = 1.0f // w (homogenous coordinates)
        val matrices = FloatArray(32) // {proj, inverse proj}
        // If you'll be calling this several times per frame factor out
        // the next two lines to run when Frame.isDisplayRotationChanged().
        frame?.camera?.getProjectionMatrix(matrices, 0, 1.0f, 100.0f)
        Matrix.invertM(matrices, 16, matrices, 0)
        // Transform clip-space point to camera-space.
        Matrix.multiplyMV(points, 4, matrices, 16, points, 0)
        // points[4,5,6] is now a camera-space vector.  Transform to world space to get a point
        // along the ray.
        val out = FloatArray(6)
        frame?.camera?.pose?.transformPoint(points, 4, out, 3)
        // use points[8,9,10] as a zero vector to get the ray head position in world space.
        frame?.camera?.pose?.transformPoint(points, 8, out, 0)
        // normalize the direction vector:
        val dx = out[3] - out[0]
        val dy = out[4] - out[1]
        val dz = out[5] - out[2]
        val scale = 1.0f / Math.sqrt((dx * dx + dy * dy + dz * dz).toDouble())
            .toFloat()
        out[3] = dx * scale
        out[4] = dy * scale
        out[5] = dz * scale

        return out
    }

But when I touch my screen, the point is not really on my finger. Maybe I haven't well use the inverse projection method?