local_resource with pass-then-fail readiness_probe never shows as red

[nsheaps]

Expected Behavior

When a readiness_probe on any resource (including local_resource) succeeds and turns green, then subsequently fails, the resource should show as unhealthy.

Current Behavior

Local resources specifically only show as red if the serve_cmd exits with any code. If the health check passes, then fails, the resource goes back to a "pending" state. The same is true for other types of probes as well.

The serve_cmd behavior is documented, but the readiness_probe behavior doesn't align with other uses of probes

Steps to Reproduce

1. Create a local resource:

 local_resource(
   'never unhealthy',
   serve_cmd='while true; do sleep 1; done',
   readiness_probe=probe(
       exec=exec_action(
           command=['bash', '-c', 'exit $(cat counter)']
       ),
       period_secs=1,
       timeout_secs=1,
       success_threshold=1,
       failure_threshold=1,
   ),
   allow_parallel=True, # allow other resources to run in parallel
 )

2. echo 0 > counter to make the readiness_probe pass
3. echo 1 > counter to make it fail
4. Note resource shows as pending in tilt UI

Context

tilt doctor Output

$ tilt doctor
Tilt: v0.33.21, built 2024-11-08
System: darwin-arm64
---
Docker
- Host: unix:///Users/xxxxxx/.docker/run/docker.sock
- Server Version: 27.3.1
- API Version: 1.47
- Builder: 2
- Compose Version: v2.30.3-desktop.1
---
Kubernetes
- Env: eks
- Context: dev-xxxx-a
- Cluster Name: arn:aws:eks:xxxxxx:xxxxxx:cluster/xxxxxxxx-xxxxxxxx
- Namespace: ingress-nginx
- Container Runtime: containerd
- Version: v1.29.10-eks-7f9249a
- Cluster Local Registry: none
---
Thanks for seeing the Tilt Doctor!
Please send the info above when filing bug reports. 💗

The info below helps us understand how you're using Tilt so we can improve,
but is not required to ask for help.
---
Analytics Settings
--> (These results reflect your personal opt in/out status and may be overridden by an `analytics_settings` call in your Tiltfile)
- User Mode: default
- Machine: 14c4502a17acda75b6e966fc7e1eca94
- Repo: 33EZAauGiC3WmaOoA66S7w==
...

About Your Use Case

We use local_resources to coordinate launching a number of local processes for local development (we don't use tilt for the k8s side of things at the moment). We utilize depends_on for ordering, so it becoming healthy is important. One of the resources we run is a watcher for files that will re-compile just that file using tsc rather than restarting the whole tilt resource. When tsc fails to compile, the output shows it, but the resource shows as healthy. We're using tsc-watch to drop a file when successful and delete it when failing, then setting the health check to fail if the file is not present so developers see a notice that it's failing.

We see the same functionality with running postgres locally with a similar tiltfile health check (so as not to re-trigger tiltfile re-evaluation).

[nicks]

AFAIR -- the reason it behaves this way is because resource_deps only affect startup order. i.e., if you rebuild resource A, and it goes healthy, then it dies later, that doesn't affect the startup of any resource that depends on A.

can you elaborate more on what you mean by "the readiness_probe behavior doesn't align with other uses of probes"?

As far as i can tell, we treat Kubernetes probes the same way. (and in fact our local resource probe implementation re-uses a lot of code from kubernetes probes)