About the precision

[milkWangzai]

hello, I modified the code to run with python3 and it can run, but I can't get the same precision you mentioned in the paper, the precision just was 50%, when I adjust the learning rate the precision improve to 67% but still has a gap with your experiment. I used your preprocessed_data. Do you have any idea about improving the precision? thank you!

[ruidan]

I did not record precisions in the paper. Is there a big gap on macro-F1 scores? Which dataset are you talking about?

You should be able to get similar results as reported by running the code directly under Python 2.7 and other specified dependencies. I did not test the code under Python 3.

[milkWangzai]

Thanks for your reply. I'm asking in the wrong place. Actually, the question is aimed at Unsupervised-Aspect-Extraction. But I have known the reason is that I modify the function in the train.py. Now I have a similar precision with you. thank you anyway.

[tc-yue]

Confused with the score recorded in the paper? Are the way like select the best configuration based on performance on the dev set and evaluation the configuration on the test set?

[ruidan]

@tc-yue which paper are you talking about? (as this issue was originally talking about another paper).

For this aspect-level-sentiment paper, the configuration (hyper-params) are selected based on dev set. During training, the results on the test set are recorded at the epoch where the model achieves best accuracy on dev set.

[tc-yue]

@ruidan thanks, about this aspect-level paper.
Following[Deep neural networks with multitask learning Collobert and Weston,ICML 2008], the multi task leaning of training is achieved in a stochastic manner by looping over the tasks:

1. Select a random task.
2. Select a random training example from this task.
3. Update the parameters for this task by taking a gradient
   step with respect to this example.
4. Go to 1.
   I am wondering how to implement the above way in keras, In your code, jointly train the multi input and multi output from the different tasks seems get the same effect.

[ruidan]

@tc-yue There are multiple ways to implement multi-task learning. I usually do that in a multi inputs and outputs way as implemented in my code.

It should be straightforward to implement the first way as well. Suppose you have two tasks with training set a and b respectively:

1. build and compile model_a and model_b for those two tasks (some layers can be shared).
2. generator_a = iter(a), generator_b = iter(b)
3. for i in [0, max_iterations]:
   p = random number in [0,1]
   if p > certain_threshold:
   batch_a = generator_a.next()
   model_a.fit(batch_a)
   else:
   batch_b = generator_b.next()
   model_b.fit(batch_b)

[tc-yue]

@ruidan thanks a lot for your reply.