profmem can't always parse output returned by (some) parallelized calls #14
Comments
|
Thanks for reporting. Yes, it's likely that there's a race condition for updating the internal Rprofmem file. This file is unique per R session; Lines 202 to 209 in 46f63f0 but it is shared across forked R processes as when using It shouldn't be too hard to make it unique to each forked child process. I'll add it to the todo list for the next release. |
|
Forgot to say, fixing this in the 'profmem' package will not fix the closely-related problem in |
|
Follow up/for the record: This problem can be reproduced as: > p <- profmem::profmem(y <- parallel::mclapply(1:10, rnorm))
Error in parse(text = trace) : <text>:1:103: unexpected symbol
1: c("FUN", "lapply", "doTryCatch", "tryCatchOne", "tryCatchList", "tryCatch", "try", "sendMaster", "FUN"chList
^
> traceback()
8: parse(text = trace)
7: eval(parse(text = trace), enclos = baseenv())
6: FUN(X[[i]], ...)
5: lapply(bfr, FUN = function(x) {
bytes <- gsub(pattern, "\\1", x)
what <- rep("alloc", times = length(x))
idxs <- which(bytes == "new page")
if (length(idxs) > 0) {
what[idxs] <- "new page"
bytes[idxs] <- ""
}
bytes <- as.numeric(bytes)
trace <- gsub(pattern, "\\2", x)
trace <- gsub("\" \"", "\", \"", trace, fixed = TRUE)
trace <- sprintf("c(%s)", trace)
trace <- eval(parse(text = trace), enclos = baseenv())
trace <- trace[seq_len(max(0L, length(trace) - drop))]
list(what = what, bytes = bytes, trace = trace)
})
4: readRprofmem(pathname, drop = drop)
3: profmem_suspend()
2: profmem_begin(threshold = threshold)
1: profmem::profmem(y <- parallel::mclapply(1:10, rnorm))Not surprisingly, it can also be reproduced with plain > pf <- tempfile()
> local({ utils::Rprofmem(pf); on.exit(utils::Rprofmem("")); y <- parallel::mclapply(1:10, FUN = function(ii) integer(1e6)) })
> p <- profmem::readRprofmem(pf)
Error in parse(text = trace) : <text>:1:18: unexpected symbol
1: c("FUN", "l272 :"get0
^
> traceback()
5: parse(text = trace)
4: eval(parse(text = trace), enclos = baseenv())
3: FUN(X[[i]], ...)
2: lapply(bfr, FUN = function(x) {
bytes <- gsub(pattern, "\\1", x)
what <- rep("alloc", times = length(x))
idxs <- which(bytes == "new page")
if (length(idxs) > 0) {
what[idxs] <- "new page"
bytes[idxs] <- ""
}
bytes <- as.numeric(bytes)
trace <- gsub(pattern, "\\2", x)
trace <- gsub("\" \"", "\", \"", trace, fixed = TRUE)
trace <- sprintf("c(%s)", trace)
trace <- eval(parse(text = trace), enclos = baseenv())
trace <- trace[seq_len(max(0L, length(trace) - drop))]
list(what = what, bytes = bytes, trace = trace)
})
1: profmem::readRprofmem(pf)As OP says, this is because forked parallel processing is used. This is because R's internal Actions: I'm not sure there is much that can be done in the profmem package. I don't think it's possible to protect against using |
|
Here's an example of intertwined Rprofmem output from multiple forked processes: Note that line with '... "<Anonynew page:"tolower" ...'. This gives: One option could be the replace non-parsable entries with a |
this is stochastic, the failing examples below sometimes work for smaller jobs with fewer calls.
fork-ed computations, mainly, but not consistently (cf.future.lapply, which never failed me so far)reprex:Created on 2019-10-06 by the reprex package (v0.3.0)
Session info
The text was updated successfully, but these errors were encountered: