Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Flaky test] common/test/acceptance/tests/lms/test_lms_problems.py::FormulaProblemRandomizeTest::test_score_reset_after_resetting_problem_* #36016

Closed
timmc-edx opened this issue Dec 11, 2024 · 1 comment
Labels
flaky-test Details a test removed as part of the flaky test process

Comments

@timmc-edx
Copy link
Contributor

timmc-edx commented Dec 11, 2024

This test fails intermittently and has been deleted according to the Flaky Test Process in #20870

See docs on how to address flaky tests for why this should be fixed and how to go about it.

3 specific test cases (including 2 ddt variants of the same method) in FormulaProblemRandomizeTest had been intermittently failing for some time. The failures seem to be due to a MathJax exception:

https://cdn.jsdelivr.net/npm/mathjax@2.7.5/MathJax.js?config=TeX-MML-AM_HTMLorMML:0:0: Script error.

The most common failure: acceptance.tests.lms.test_lms_problems.FormulaProblemRandomizeTest.test_score_reset_after_resetting_problem_2__R_1_R_2_R_3__correct__1_1_pointungraded___0_1_pointungraded__

Other failure derived from the same test method: acceptance.tests.lms.test_lms_problems.FormulaProblemRandomizeTest.test_score_reset_after_resetting_problem_1__R_1_R_2__incorrect__0_1_pointungraded___0_1_pointungraded__

Similar failure from another method in the same class: acceptance.tests.lms.test_lms_problems.FormulaProblemRandomizeTest.test_reset_problem_after_submission_1__R_1_R_3__incorrect_`

Typical stacktrace from the test:

Failure output:

self = <common.test.acceptance.tests.lms.test_lms_problems.FormulaProblemRandomizeTes...lem_1___R_1_R_2____incorrect____0_1_point__ungraded_____0_1_point__ungraded___>
input_value = 'R_1*R_2', correctness = 'incorrect'
score_before_reset = '0/1 point (ungraded)'
score_after_reset = '0/1 point (ungraded)'

    @ddt.data(
        ('R_1*R_2', 'incorrect', '0/1 point (ungraded)', '0/1 point (ungraded)'),
        ('R_1*R_2/R_3', 'correct', '1/1 point (ungraded)', '0/1 point (ungraded)'),
        ('R_1/R_2', 'incorrect', '0/1 point (ungraded)', '0/1 point (ungraded)')
    )
    @ddt.unpack
    def test_score_reset_after_resetting_problem(self, input_value, correctness, score_before_reset, score_after_reset):
        """
        Scenario: Test that score resets after the formula problem is resetted.
    
        Given I am attempting a formula problem type with randomization:always configuration
        When I input the answer
        Then I should be able to see the MathJax generated preview
        When I submit the problem
        Then I should be able to view the score that I received
        And The reset button should be present and is clickable
        When the reset button is clicked
        Then the score resets to zero
        """
        problem_page = ProblemPage(self.browser)
        problem_page.fill_answer_numerical(input_value)
        problem_page.verify_mathjax_rendered_in_preview()
        problem_page.click_submit()
        self.assertEqual(problem_page.get_simpleprob_correctness(), correctness)
        self.assertIn(score_before_reset, problem_page.problem_progress_graded_value)
        self.assertTrue(problem_page.is_reset_button_present())
>       problem_page.click_reset()

common/test/acceptance/tests/lms/test_lms_problems.py:1110: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../edx-venv/local/lib/python2.7/site-packages/bok_choy/page_object.py:123: in wrapper
    return method(self, *args, **kwargs)
common/test/acceptance/pages/lms/problem.py:177: in click_reset
    click_css(self, '.problem .reset', require_notification=False)
common/test/acceptance/pages/common/utils.py:90: in click_css
    page.wait_for_ajax()
../../edx-venv/local/lib/python2.7/site-packages/bok_choy/page_object.py:575: in wait_for_ajax
    timeout=timeout
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <bok_choy.promise.EmptyPromise object at 0x7fc6a0b65950>

    def fulfill(self):
        """
        Evaluate the promise and return the result.
    
        Returns:
             The result of the `Promise` (second return value from the `check_func`)
    
        Raises:
            BrokenPromise: the `Promise` was not satisfied within the time or attempt limits.
        """
        is_fulfilled, result = self._check_fulfilled()
    
        if is_fulfilled:
            return result
        else:
>           raise BrokenPromise(self)
E           BrokenPromise: Promise not satisfied: Finished waiting for ajax requests.

../../edx-venv/local/lib/python2.7/site-packages/bok_choy/promise.py:106: BrokenPromise

Migrated from private ticket https://2u-internal.atlassian.net/issues/CR-935

@timmc-edx timmc-edx added the flaky-test Details a test removed as part of the flaky test process label Dec 11, 2024
@timmc-edx
Copy link
Contributor Author

Closing because this was a bokchoy test, and those had all since been deleted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
flaky-test Details a test removed as part of the flaky test process
Projects
None yet
Development

No branches or pull requests

1 participant