Showing posts with label Meeting. Show all posts
Showing posts with label Meeting. Show all posts

Friday, May 25, 2012

Meeting 25-05-2012

Action Points
  1. We started with discussing the planning proposed at the beginning of the year. For me, I was pretty accurate, except that I did not yet implement the "transposition tree" and the writing which started a bit later. 
  2. We should use noise for the reward in stead of on the actions because if adding noise to the action can also change the optimal reward function, i.e. even when playing the best actions one would not be able to achieve the best reward. To draw the optimal line when having noisy actions you'd have to calculate products / intervals (to keep it general). 
  3. For multi step results I should also make a MTL-UCT agent
  4. I asked what to do with the state information of my "transposition tree". Michael proposed an idea and I brainstormed with Andreas afterwards:
    1. Assume the state/observation space to be Markov
    2. First level is an "observation tree", discretizing to observation space.
    3. Each leaf representing a region of this space links to an action tree on the second level, discretizing the action space and keeping information about which actions are best for the observation range from the level above.
    4. This observation-action tree can be re-used for each state the agent is in due the the property of 1.
Plans
  1. Change last experiment so that noise is on rewards.
  2. Add the MTL-UCT agent to the experiment
  3. Look into the global/perfect recall again
  4. Implement the "transposition tree"
  5. Writing

Wednesday, May 2, 2012

Meeting 02-05-2012

Discussion Points

Andreas, Lukas and me presented the results of the experiments (see previous posts for my results). I've got some usefull feedback and comment, which are summarized in the planning below.

Planning

  1. Investigate why the IRTI line horizontal at the end of the Six Hump Camel Back Experiments.
  2. Implement a visualization to show at which time a split occurs (at which sample number). 
  3. Check if it is correct that the RMTL agent has a significant decrease in number of simulations per second when memorization is enabled, as seen in the table of the last post.
  4. Think which parameters are relevant (to mention in the report) and remove the redundant.
  5. Optimize the multi-step agents
  6. Think of a graph representation of the multi-step experiments and rerun them under more difficult settings, i.e. less time or the settings from the paper (with noisy rewards, noisy actions, etc).
  7. Implement perfect recall on meta tree level
  8. Implement Transposition Tree

Saturday, April 14, 2012

Meeting 12-04-2012

Work
  1. Fixing, debugging and improving IRTI and HOO. 
  2. Furthermore, a lot of parameter tuning for above two algorithms.
Action Points
  1. Michael and I had a discussion about the exploration factor
    1. C should be related to the reward range
      1. constant C =  globalRangeSize * K
      2. adaptive C =  parentRangeSize * K
      3. Lukas proposed: adaptive C = (childRangeSize / globalRangeSize) * K
    2. For HOO, you should also multiply the last term (diam) with C or divide the first term (mean reward) with C.
      1. R + C * exploration + C * diam 
      2. R / C + exploration + diam
Planning
  • Change the HOO formula (see action points)
  • Generate reward distributions (learning curve, greedy curve, error) of several algorithms
    • IRTI
    • HOO
    • UCT (pre-discretization)
    • Vanilla MC (random sampling, greedy returns best sample seen)
    • Random
  • I should ask Lukas about the working of his IRTI algorithm to find out why mine does not converge samling at the global maximum.

Wednesday, March 28, 2012

Meeting 28-03-2012


Action Points
  • We all taked about our progress, results so far and current work
  • I implemented IRTI, TLS, HOO and HOLOP although the latter two are under development
  • By observing the agent's behavior in RL Viz I notices a couple of things which I should look into
    • IRTI gets stuck in a non-optimal regions very rarely (Six Hump Camel Back; approximately 1 out of 100 times)
    • The TLS agent does sometimes a bad move in the Double Integrator
    • The sinus function seems a difficult problem for IRTI and sampling is mostly done at the local maxima at the left
    • HOO performs better than IRTI at the Sinus function and very bad in the Six Hump Camel Back environment
    • HOO is much slower than IRTI
    • HOO does not explore correctly
    • HOLOP performs bad, most likely due to the incorrect workings of HOO
  • I should look into these problems. The planning below show some solutions for above problems
Planning
  • Try to split sooner for the Sinus environment and observe results
  • Make nodes beforehand
  • Remove storing the action ranges within the nodes (to avoid array copying each split)
  • Scale the exploration term in the calculation of U in HOO
  • Update U and B values before the selection step
  • Change the discounting of reward (the power should match the depth)
  • Investigate saving of images in RL Viz
Next meeting
  • Individual meeting: +- April 11
  • Joint meeting: +- April 25

Saturday, March 17, 2012

Meeting 16-03-2012

Work
  • Implemented the option of time based learning (basides simulation based learning)
  • Implemented discounted rewards for the backtracking step
  • Added a parameter for the maximum expansion depth of the meta tree
  • Did some optimization for IRTI / RMTL and experimented a bit
  • Implemented the Double Integrator environment mentioned in several papers
  • Started implementation HOO
    • Seems to work for one-step one-dimensional problems (i.e. one-step Donut World)
    • Fails in Six Hump Camel Back
Action Points
  • We looked at the results from previous post which looked good
    • I should look into the "artifacts" still present in the samples "901-1000" (you still see some lines of samples near the (local and global) optima)
    • I still have to add error bars
  • Kurt gave me some tips for the presentation and report
  • As it was not really clear to me from the literature I asked about the choice of splitting in HOO
    • Which dimension? Random
    • At which point in the dimension's range to split? Random
  • For HOLOP however, dimensions corresponding to early stages in the sequence should be chosen more often as they tend to have a bigger contribution the to return (see HOLOP paper)
  • The last term in the upperbound formula (sometimes referred as v1*p^h and sometimes diam(Ph,i)) was also not clear to me. I'm still struggling a bit with this, but on the other hand this should not affect the working of HOO to much. 
Planning
  • Finish presentation for thesis meeting March 21
  • Finalise / Debug HOO
  • Thesis report wrinting / restructuring
Next meeting
  • Joint meeting: Wednesday, March 28, 2012, 10:00-12:00

Friday, February 24, 2012

Meeting 24-02-2012


Action Points
  • I explained that I just fixed a bug regarding the adaptive UCT constant. During selection each child should take the same UCT constant (dependant on the range of the parent node), whereas at first I used the ranges of the children themselves meaning the children would use different UCT constants. In some cases a "bad" child would be preferred over a "good" child (with high average / low number of trails) when this bad child has a low minimum (and so a large range size).
  • I showed the behaviour and results in RL Glue and .cvs data; this looked good
  • Two problems are still open and are to be investigated (Kurt; thanks in advance)
    • From the Scala code it can be concluded that a T-test is used for the Six Hump Camel Back experiment, while the paper itself states an F-test is used?
    • The F-values used in the Scala code are of the entries 1 and n degrees of freedom of the F-Distribution table, while I though it would be more logical to use the entry n and n (because the degrees of freedom is equal for both sides)?
  • HOLOP and TLS and the combinations have been explained to me. On the one hand you have a "tree of trees" taking either a regression tree (as in TLS) or HOO and on the other hand you have a "tree of sequences" which can also either be in the form of a regression tree or HOO (as in HOLOP). A sequence can in this case be seen as the cartesian product / concatenation of the actions taken during an episode.
  • Besides these four algorithms, Michael proposed UCT with pre-discretization. I said that I already implemented a simple agent that pre-discretizes the action space (in a specified number of "parts") and uses a basic UCT algorithm to learn. I did this to test the MCTS / UCT methods / components which are also used in the regression trees.
  • I announced that I have a job and will be working two days a week from now on. This means my productivity will slightly decrease.
Planning
  • Implement (in Matlab) reading and visualizing the output from experiments
  • Write everything relevant from this blog into the Latex draft already
  • Think about the names of all 4 combinations of algorithms
  • Implement TLS taking the regression tree component in a multi-step environment. Think about the representation of the sequence of the actions fed to the TLS component (and later to the HOLOP component)
Next meeting
  • Wednesday, March 7, 2012, 11:00
  • Andreas and Lukas are also invited to this meeting

Monday, February 13, 2012

Meeting 13-02-2012

Action Points
  • Fortunately, I had correctly implemented the splitting criteron. There were two reasons for the incorrect behaviour of the splitting.
    • The parameters were not set correctly and changed them to match the parameters used for the experiments in [1]
      • F-test significance level = 0.001 (I had set this too high (0.05) and therefore splits were introduced to early)
      • The minimum amount of samples to collect before considering a split = 15
      • MCTS exploration constant = 0.5 * (reward range size of parent node)
    • Samples / statistics were completely discarted after a split to save memory. In my case, storing the samples would most likely be much more beneficial.
  • We talked about the experimental implementations (see posts last week). Everything look fine, although I should add the options as described in the planning below.
  • Results of an experiment I tried a couple of days ago showed that over time, there was sometimes a decrease in the reward when taking greedy actions. This is a flaw in the take greedy action method; storage of the best sample in each leaf should solve this problem. 
  • I will be on holiday from Wednesday February 15 to Tuesday February 21
Planning
  • Store the best sample seen at each leaf of the regression tree which can be used when a greedy best action is requested
  • For the RL Glue visualizations; it might be nice to color the leaf/action with the highest average reward in a different color
  • Implement the memorization of samples (i.e. all attribute values and regression value of each sample) in each leaf. In case of a split these samples can be re-used by the two children (by re-inserting the samples at the parent)
  • Think about and add the option to change above mentioned memorization, i.e.
    • turn on memorization
    • turn off memorization
    • only memorize a certain number of samples
    • only re-insert samples going to the best child
    • etc.  
  • Implement the possibility to specify and read a certain properties file (instead of only a default file in the root folder, as of now)
  • Implement (in Matlab) reading and visualizing the output from experiments (i.e. the generated .csv files; see scala code received from Kurt for reference)
  • Run an experiment to compare with the results from the paper.
Next Meeting
  • Friday, Februari 24, 2012, 11:00
[1] G. Van den Broeck and K. Driessens, “Automatic discretization of actions and states in Monte-Carlo tree search,” in Proceedings of the ECML/PKDD
2011 Workshop on Machine Learning and Data Mining in and around Games (T. Croonenborghs, K. Driessens, and O. Missura, eds.), pp. 1–12, Sep 2011.

Friday, February 3, 2012

Meeting 03-02


Action Points
  • As of now, I've implemented the agent in RL Glue not knowing the model of the environment and sampling by playing "real" actions. I should change this so that the agent knows the model and can sample "in his head". After a specified amount of samples, the agent returns a "real" action to RL Glue. I could either run two instances of RL Glue (in head and in real world) or give the agent acces to the environment classes.
  • In general, states (or observations) are not taken into consideration for TLS. In case of different starting states, one could add one depth to the tree where the root now accounts for deciding in which initial state the agent is in. The agent then starts sampling from this node.
  • An example of a VFDT for the context of TLS
    • Attribute values (X) = action values (one per action dimension)
    • Regression value (value to predict; y) = reward 
  •  Choosing split points for a continuous attribute
    • A test is in the form of exampleValueY <= splitPoint
    • Incrementally updated as new examples arrive. 7 values have to be stored per test;
      • split point value
      • number of examples that pass / fail the test 
      • sum of y values that pass / fail  the test 
      • sum of squared y values that pass / fail the test 
    • FIMT uses a binary tree representing all the examples
    • Other possibility proposed by Michael / Kurt and similar to TG's is to use the values of the first n examples as possible splitting point. After that, each new example only updates the statistics for these splitting points. 
  • The R in the Hoeffding bounds formula represents the size of the range of the regression value; the reward range.
Tasks
  • Changes RL Glue structure so that agent can distinct between "in head" and "real" simulations.
  • Implement so that for each experiment, a new output folder is created containing (at least) the results and a file containing the values of all the constants used for that experiment. Possibly add being able to load this file.
  • Continue implementing regression trees, and make it possible to switch between the "binary tree" method of FIMT and the "first n examples" method (of TG).
Next Meeting
  • Monday, Februari 13, 2012, 13:00 

Thursday, January 26, 2012

Meeting 26-01-2012

Action Points
  • Discussed work done since last meeting (see previous posts)
  • I told that I spoke with Lukas about using RL Glue and that we would use the same environments
  • The "Donut World" is shown to me which is a simple continuous problem that can be used for testing
    • Agent starts in the "hole" of a donut-shaped region in a 2-dimensional continuous space
    • Agent recieves positive rewards for being in the donut region
    • 1-dimensional actions indicate the angle of the agent
    • Agents moves forward with a fixed step size; just small enought to be able to stay in the donut region
  • Kurt mailed me a TLS implementation in Scala 
  • Michael offered me a Java implementation of HOO
  • I signed the thesis plan which will be evaluated and forwarded by Kurt and Michael
Tasks
  • Think about how to evaluate the agents (i.e. reward, steps, mean, std, etc.)
  • Implement "Donut World" environment for RL Glue
  • Find existing RL Glue agents to benchmark against
  • Continue on topic Data Stream Mining / Regression trees
Next Meeting

  • Friday, Februari 03, 2012, 11:00 

Tuesday, January 17, 2012

Meeting 16-01-2012

Thesis Plan Comments / Tasks
  • Come up with a preliminary title
  • Split the research question about the component replacements into two questions
  • Put research question about learning the generative model at the end; becomes optional for the research
  • Put research question about the re-use of information for TLS at the end; becomes optional for the research
  • Make a Gantt chart for the planning; more detailed
Additional Tasks
  • Set up a method for communication (this blog!)
  • Read more literature, i.e. Multi-Resulution Exploration in Continuous Spaces (Nouri & Littman)
  • Think of about the implementation details; i.e. plan, programming language, etc.
  • Think about which testbed / benchmarking to use, i.e. RL Glue, function approximation, etc.
Next Meeting
  • Thursday, January 26, 2012, 11:00