Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When resample (and benchmark?) get instantiated resampling instance, they should check validity #1236

Open
mb706 opened this issue Dec 21, 2024 · 0 comments

Comments

@mb706
Copy link
Collaborator

mb706 commented Dec 21, 2024

The following should not work, since the resampling instance references rows that are not contained in the task any more:

rs <- rsmp("holdout")
ts <- tsk("iris")
rs$instantiate(ts)
ts$filter(c(1, 51, 101))
ts
#> <TaskClassif:iris> (3 x 5): Iris Flowers
#> * Target: Species
#> * Properties: multiclass
#> * Features (4):
#>   - dbl (4): Petal.Length, Petal.Width, Sepal.Length, Sepal.Width
resample(ts, lrn("classif.rpart"), rs)
#> INFO  [16:34:57.794] [mlr3] Applying learner 'classif.rpart' on task 'iris' (iter 1/1)
#> <ResampleResult> with 1 resampling iterations
#>  task_id    learner_id resampling_id iteration     prediction_test warnings
#>     iris classif.rpart       holdout         1 <PredictionClassif>        0
#>  errors
#>       0

Or do we actually ever need to refer to rows that were $filter()-ed out before? The code anticipates this problem in some places, but the following error can only be triggered if the entire data was actually replaced using cbind:

rs <- rsmp("holdout")
ts <- tsk("iris")
rs$instantiate(ts)
ts$filter(c(1, 51, 101))
ts$cbind(iris[1:3, 1:4])
resample(ts, lrn("classif.rpart"), rs)
#> INFO  [17:05:26.512] [mlr3] Applying learner 'classif.rpart' on task 'iris' (iter 1/1)
#> Error: DataBackend did not return the queried rows correctly: 50 requested, 1 received.
#>         The resampling was probably instantiated on a different task.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant