Add more hermetic tests

[liu-cong]

Hermetic test is a fast way to verify functionality e2e without requiring an integration test env. In addition to the basic test we have today, we should add the following:

Test framework improvements:

• Run a fake k8s API server so we don't need to fake the reconcilers
• Verify metrics (not added yet)

Test Case

• Test when model is not found in LLMService
• Test when ModelServerPool is not found
• Test when no backend pods are available
• Test invalid request (e.g., doesn't contain "model")
• Test backend server error, client should receive an error with an appropriate error code
• Verify traffic split
• Test algorithm
  • Sheddable request succeeds when resource is available; and dropped when resource is constrained
  • Verify min KV cache algo without LoRA
  • Verify LoRA affinity algo for the "warm up" case (when no pods has loaded any LoRA yet). This requires sending multiple requests and verify later requests will be sticky to backend pods.

Tasks

Preview Give feedback

1. Run a local k8s API server in the hermetic tests #153
   +0

[ahg-g]

Please take a look at integration test setup in jobset as an example: https://github.com/kubernetes-sigs/jobset/tree/main/test/integration/controller

[danehans]

I may not be able to add every requested test case but I can get this effort started.

/assign

[liu-cong]

@danehans thanks! Please sync with @BenjaminBraunDev as he started with some initial work. There are lot of tests to add so you can divide and conquer!

[danehans]

@liu-cong thanks for the heads-up. @BenjaminBraunDev can you provide a status update when you have a moment? Do you have a local branch or WIP PR that can be referenced to see how we can divide and conquer this issue?

[danehans]

Unassigning myself why I work on the initial e2e tests.

/unassign

[BenjaminBraunDev]

@danehans Yes, I'm currently working on specifically getting a k8s Client working with the tests for InferencePool and InferenceModel CRDs. Instead of using the StartExtProc() function in test utils, I'm moving a lot of the functionality from main.go into a new package "server" and calling it with specific vars in main.go as well as in the hermetic tests. This allows the test to pipe its own environment config into the functionality of main which more accurately reflects en e2e test, and from there the hermetic test creates a k8s client using the same environment.

But in short yes, I'm working on the hermetic tests at the moment, but no I don't have a PR yet.

[danehans]

I'm moving a lot of the functionality from main.go into a new package "server" and calling it with specific vars in main.go as well as in the hermetic tests.

@BenjaminBraunDev note that #148 refactors main().