Verifying POMDP solvers' results

[Omastto1]

Hi,

what is the preferred way to verify POMDP solvers' results using POMDPs.jl (or in general)?

[Omastto1]

I am implementing FiVI POMDP solver and I want to verify its results.
So far, the possible approaches I found are the following:

• comparing values or actions given belief vector (value(pomdp, b), action(pomdp, b).
  This approach returns different results (different values for value and a couple of different actions for action) for tested solvers (FIB, PBVI, SARSOP, IncrementalPruning). Or
• testing solver using simulation (POMDPTesting.test_solver(::Policy, pomdp),
  which results in similar results, however, I am not sure how exactly this method works.

Could different solvers' value output different values? Should I focus on actions instead?
It is not clear to me, how much faster run of test_solver could result in similar value, when value does not.

[Omastto1]

So I finally decided to take a look at this problem.

At first, I found POMDPSimulators.jl, which contains run_parallel whose docs are saying that it contains a utility for running many Monte Carlo simulations in parallel to evaluate performance, to which I do not understand because the source code looks like the simulations are run parallel. Their result is only merged into DataFrame.

I also found MCTS.jl and its version for POMDPs (IMO, the docs should say that BeliefMCTSSolver is only a struct, not a package. It took me a while to find it.) and solved POMDP.

Regarding your proposal of comparison. I did not completely understand your idea, so I thought I could use another approach.

• solve the POMDP with both PBVI and MCTS,
• simulate both policies,
• compare the discounted rewards with confidence intervals of their means.

However, the simulations ended up being partially random for MCTS simulations (25% ending with 0. reward, and the rest with rewards 0.8 - 0.9. I believe that if I increase the depth of simulation/MCTS, the percentage of 0. rewards will decrease). On the other hand, the simulations of PBVI ended up with 0.9 in 100%.

The maximum obtained reward of both simulations is the same. Thus making the with confidence interval unnecessary.

From this test, it seems to me that this reward could be optimal.

My question here is, is my approach correct? I thought that the simulation of my policy would end up being stochastic. And thus, I would have to use a comparison: either confidence intervals or your approach.

Also, I include part of my code.

Tom

# init parameters
POMDPs.updater(p::MCTSPlanner) = p.mdp.updater
pomdp = MiniHallway()

n_iter = 10000
depth = 10
ec = 0.5

# init policies and solve them
# MCTS
solver = BeliefMCTSSolver(MCTSSolver(n_iterations=n_iter,
    depth=depth,
    exploration_constant=ec,
    enable_tree_vis=true
), DiscreteUpdater(pomdp))
planner = solve(solver, pomdp)

# PBVI
solver2 = PBVI()
PBVIplanner = solve(solver2, pomdp)

# run simulations

q = [Sim(pomdp, planner, max_steps=depth, rng=MersenneTwister(4), metadata=Dict(:policy=>"MCTS_$(i)")) for i in 1:100]
MCTS_data = run_parallel(q)

q2 = [Sim(pomdp, PBVIplanner, max_steps=depth, rng=MersenneTwister(4), metadata=Dict(:policy=>"PBVI_$(i)")) for i in 1:100]
PBVI_data2 = run_parallel(q2)

[zsunberg]

Sorry for the confusion due to long replies - I would recommend comparing the value function to SARSOP, just like this: https://github.com/JuliaPOMDP/PointBasedValueIteration.jl/blob/9fb5ef803daa26a42a60da8ab06d19ac80a32747/test/runtests.jl#L14-L25 , except if you have more than 2 states, you will have to select belief points differently.

[zsunberg]

It looks like you are trying to do the right thing in your pull request.

[zsunberg]

MCTS is not usually a great idea to compare to because it is stochastic and may not find the optimal policy every time.

(It seems like you may have misunderstood what I meant by Monte Carlo simulations - if you want to discuss that any more, feel free to ask follow up questions)

[Omastto1]

Yeah, I probably did misunderstand that part.
Was that idea connected to the aforementioned run_parallel method in POMDPSimulators, or am I completely wrong?

[Omastto1]

Initially, I tested my solution against the SARSOP solver but got a different value function.
I gave it a look again and found out that the updated PBVI solver's value function is approximately 3.4 greater in all cases, which means it probably does not work correctly, but at the same time, it is not completely random; I guess.