Create benchmark suite infrastructure #341
Conversation
|
I am not sure what I did wrong, but when I tried your script, I got the following message. I ran everything inside the benchmarks folder, activate, initiate and then executed the script. |
|
Hi @helmutstrey, sorry about that, it was a silly error because the |
Thanks. I ran it again. Most of the benches are running, but towards the end it gives this error: julia> |
|
Hmm that's strange, it seems to be complaining about Let me see if I can figure out what's up. |
I just checked. When you To report any bugs, issues, or feature requests for Neuroblox software, julia> reset! julia> reset!() julia> Neuroblox.reset! julia> methods(Neuroblox.reset!) 5 methods for generic function "reset!" from DSP.Filters:[1] reset!(self::DSP.Filters.FIRFilter) |
|
Okay that's very weird and not what I see: julia> using Neuroblox
Important Note: Neuroblox is a commercial product of Neuroblox, Inc.
It is free to use for non-commercial academic teaching
and research purposes. For commercial users, license fees apply.
Please refer to the End User License Agreement
(https://github.com/Neuroblox/NeurobloxEULA) for details.
Please contact sales@neuroblox.org for purchasing information.
To report any bugs, issues, or feature requests for Neuroblox software,
please use the public Github repository NeurobloxIssues, located at
https://github.com/Neuroblox/NeurobloxIssues.
julia> reset!
reset! (generic function with 2 methods)
julia> methods(Neuroblox.reset!)
# 2 methods for generic function "reset!" from Neuroblox:
[1] reset!(ag::Agent)
@ ~/Dropbox/neuroblox/Neuroblox/src/blox/reinforcement_learning.jl:203
[2] reset!(env::Neuroblox.AbstractEnvironment)
@ ~/Dropbox/neuroblox/Neuroblox/src/blox/reinforcement_learning.jl:126Do you see the same behaviour if you checkout the By the way, when posting code in here, it'll work better if you wrap it in triple backticks, e.g. the above snippet was written as |
|
could it be that I have the wrong Neuroblox version installed: |
|
Oh crap, yeah I see the problem. The issue is that it's taking the registered Neuroblox, not the one from the enclosing folder. I forgot that when I had initially set this up, I did We really need JuliaLang/Pkg.jl#3783, but in the meantime I'll have to |
|
@MasonProtter The benchmarks are now working on my end. I will add my M2 Mac Pro run to the benchmark history folder. One thing that I noticed is that in the 80 randomly connected neurons solve, there are lots of warnings: Instability detected. This example is also very unrealistic since these neural mass blox, are not really neurons. @anandpathak31 could you provide a nicer example. I would say that simulating a Cortical blox of significant size would do here. |
|
Great! I've just done some overhauls and changed a few things (including making that example smaller due to the warnings), and also include metadata about the random seed, the git commit of neuroblox, and the full system information about the computer running the benchmarks. I'm starting to think though, maybe this should be a separate package in the Neuroblox org instead of just some sub-directory in here? |
Agreed. We will keep adding more benchmarks over time dependent on papers and/or communications with developers from other packages with similar functionalities. It would be good housekeeping to split benchmarks into their own repo. |
Good idea. Yes, let's make this a separate repository in Neuroblox. |
Here's a set of benchmarks to track the performance of Neuroblox.
Here's the current README: https://github.com/Neuroblox/Neuroblox.jl/blob/benchmarks2/benchmarks/README.md
Here's an example of the sort of information saved:
We'll want to write more benchmarks, and they can simply be added by putting them into
neuroblox_benchmark_suitewith theBenchSetupobject. That might look like e.g.which will benchmark
solve(prob, AutoVern7(Rodas4()), saveat=0.1)in a fresh session where the above setup code is run, andprobis taken as a function argument. Thesecondsparameter lets you allow the benchmark to run for around 50 seconds (or less if it collects enough samples / evaluations).