Feedback 22-04-2012

Experiment Results

Finally some nice results to show after some debugging and parameter tuning; manual and    automatic. All experiments are done over 1,000 samples and are averaged over 10,000 runs. For every experiment the 95% confidence intervals are displayed by a colored region surrounding the lines.

Incremental Regression Tree Induction (IRTI)

• MCTS_C = (childRewardRangeSize / globalRewardRangeSize) * MCTS_K)
• MCTS_K = 2.0
• IRTI_SPLIT_NR_TESTS = 100
• IRTI_SPLIT_MIN_NR_SAMPLES = 50
• IRTI_SIGNIFICANCE_LEVEL = 0.001
• IRTI_MEMORIZATION = true

Hierarchical Optimistic Optimization (HOO)

• MCTS_C  = (parentActionSpaceVolume / globalActionSpaceVolume) * globalRewardVolume * MCTS_K
• MCTS_K=5.0
• HOO_MEMORIZATION = true
• HOO_V_1 = (sqrt(nrActionDimensions) / 2) ^ HOO_ALPHA 
• HOO_RHO = 2 ^ (- HOO_ALPHA /  nrActionDimensions)
• HOO_ALPHA = 0.99

IRTI + HOO +  MC + Random  + UCT (pre-discretization with 25 splits per depth)

Work

Implemented all four multi-step agents and seem to be able to balance the pole in the Cartpole environment pretty well.

1. Regression-based Meta Tree Learning agent (RMTL)
2. Hierarchical Optimistic Meta Tree Learning agent (HOMTL)
3. Regression-based Sequence Tree Learning agent (RSTL)
4. Hierarchical Optimistic Sequence Tree Learning agent (HOSTL)

Planning

1. Find a better parameter set for HOO for the Six Hump Camel Back function as it is now barely better than UCT with pre-discretization in that environment. I can then generate the same pictures as the ones above.
2. Do experiments regarding the multi-step agents and possibly debug / optimize code.