API Update behat-extension from selenium > chromedriver

[tractorcow]

This change requires a new (yet untagged) version of behat-extension, so be careful of merge order. :)

Parent story: silverstripe/silverstripe-behat-extension#137

Instead of using selenium, switch behat to the custom https://github.com/open-sausages/MinkFacebookWebDriver I wrote which uses github.com/facebook/php-webdriver instead of the instaclick/webdriver.

See discussion on minkphp/MinkSelenium2Driver#254 for why we aren't using latest firefox anymore (it's not quite ready for use with latest selenium).

In order to test see the composer.json with all the necessary repositories in my comment at silverstripe/silverstripe-behat-extension#137 (comment)

[joejoseph00]

tractorcow, I am hoping to use newer versions of selenium like 3.8.0 with w3c compliant web-driver, how might I use your untagged driver? what changes should I make to my composer.json ? Untagged meaning it's not yet on packagist right? so clone it?

should I just clone your project and put it in my vendor folder?

what changes to my behat.yml are necessary?

here is my current composer.json

{
    "name": "example behat selenium gitlab ci test",
    "description": "see wiki documentation on XXXX internal website, I will publish this wiki publicly later at some point",
    "type": "project",
    "authors": [
        {
            "name": "Joseph Olstad",
            "email": "spam@lol.biz"
        }
    ],
    "require-dev": {
        "behat/behat": "^3.3.1"
    },
    "require": {
        "lakion/mink-debug-extension": "^1.2",
        "behat/mink-selenium2-driver": "^1.3",
        "behat/mink-extension": "^v2.2",
        "behat/mink-goutte-driver": "^1.2"
    }
}

here's a profile called seleniumfirefox (I also have one for IE11 and chrome) I'm testing, here is a section of my behat.yml

seleniumfirefox:
  suites:
    default:
      contexts:
        - FeatureContext:
            parameters:
                base_url: "https://spam.lol.biz"
  extensions:
    Behat\MinkExtension:
      default_session: selenium2
      base_url: "https://spam.lol.biz"
      goutte: ~
      javascript_session: selenium2
      browser_name: firefox
      selenium2:
        wd_host: http://127.0.0.1:4444/wd/hub
        capabilities: {"browser": "firefox", "marionette": null}
      sessions:
        my_session:
           selenium2:
             wd_host: http://127.0.0.1:4444wd/hub
             capabilities: {"browser": "firefox", "version": "SUBSTITUTE WITH VERSION NUMBER", "marionette": null}
    Lakion\Behat\MinkDebugExtension:
      directory: /tmp/debug-firefox

[joejoseph00]

ah I looked at your commit, looks like you've switched away from selenium completely to the chromium driver. So maybe the solution is being worked on somewhere upstream at the selenium HQ

[tractorcow]

@joejoseph00 you can use the new webdriver I wrote with selenium if you like; It does work but you need to handle the selenium and browser configuration yourself. :) facebook/webdriver is simply the abstract layer which works with webdriver compatible services such as chromium / selenium.

I've posted a candidate composer.json with the necessary custom repositories listed at silverstripe/silverstripe-behat-extension#137 (comment).

behat.yml changes should be as-per https://github.com/open-sausages/silverstripe-framework/blob/bffd12e597ea09af44bd9de8ce4c29620ca63e79/behat.yml except:

• omit the wd_host change (driver internally defaults to selenium port)
• switch browser from chrome to firefox

Also please note that firefox isn't fully w3c webdriver compliant, so some operations (especially related to mouse movement) won't work properly; Use older selenium + older firefox in this case.

Set base url via .env as well as behat.yml

Regarding selenium 3 support upstream at mink, please see the thread at minkphp/MinkSelenium2Driver#254. Status is that "yes you can use selenium 3 but just not with firefox yet". You can theoretically use selenium2 driver with selenium3.

[tractorcow]

@joejoseph00 I'll try to get the driver on packagist soon, just waiting for internal approval.

[tractorcow]

Ok, pushing up a temp patch that should pull in the new forked repos for testing on travis. Let's see if it runs. ;)

[tractorcow]

It works yes. ;)

@dhensby @chillu what do you think about how we should handle semver for this? Does switching behat-extension to a new driver mean we need a new major version? I am hoping we just go with minor version, but on the other hand it cannot swap back.

-        "behat/mink-selenium2-driver": "^1.3",
+        "silverstripe/mink-facebook-web-driver": "^1",

Also we may need to switch this for all 4.x branches at the same time, as the next time travis updates their images (or drops support for deprecated images) we will find behat breaking on 4.0 again.

[tractorcow]

Green tick (don't merge, it has temp commits in it though).

[dhensby]

@tractorcow Our dependencies do not form part of our API, IMO - so we can swap them out. Any project that uses our lib but directly depends on mink-selenium2-driver should have it declared as a direct dependency themselves.

[tractorcow]

Yay that's good.

The new behat-extension change will have the new dependency, should we be happy to go ahead with this.

[joejoseph00]

I've chosen to stay a couple steps behind the bleeding edge and it's working out very nicely. We've settled with behat version 3.4.3 using selenium/standalone-chrome-debug:3.5.3-boron , it's working well. I've put my special sauce into features/bootstrap/FeatureContext.php . Maybe later see what the silverstripe is all about.

[tractorcow]

Yes chrome works well with the new selenium, actually. That's what I was trying to get working prior to sticking with chromedriver.

[joejoseph00]

oh really? not sure, I'm afraid to upgrade my chrome after having issues. our stack is based on behat 3.4.3 and the selenium2mink driver , are you also using the instaclick/php-webdriver ? that is what we are using. There's so many drivers and combinations to choose from and I had to make some business decisions quickly, what we have works ok so far. I'm not sure how much refactoring I'd have to do to switch to a different driver to replace selenium2mink , selenium/standalone-chrome-debug:3.5.3-boron is where we're at. this I think is due to our team wanting to also use IE11 . I had to downgrade to selenium 3.4.0 to get IE11 to work, but we've not been using IE11 lately due to our use of docker we're sticking to dockerizable componants for our c.i. Our firefox is even older than 3.5.3 , I think we were using 2.4.3 We're staying a few steps behind bleeding edge due to our need to have something that works today, can't be always fiddling with it. living with it for a while.

[tractorcow]

are you also using the instaclick/php-webdriver ? that is what we are using.

I was actually using facebook/webdriver which has more modern support. I know instaclick driver has a 2.0 version that might be in beta, but the prior 1.x didn't support new selenium well.

[joejoseph00]

So you're using the facebook/webdriver with behat ? do you have an example of this? like a composer.json file with a list of the dependencies, and a github repo with an example feature and FeatureContext.php and behat.yml?
interested in perhaps using the facebook/webdriver , not sure how to get this working

[tractorcow]

I have exactly that actually. :) See my comment on silverstripe/silverstripe-behat-extension#137 (comment) for the composer.json.

Warning, I haven't rebased those branches for a while. You may need to do a bit of manual rebasing to get it working with latest framework.

[tractorcow]

behat.yml

# Note: Currently chrome latest is recommended
# Behat test setup requires an `.env` with `SS_BASE_URL` defined, as well
# as each of the following commands to initiate a test run:
# ========================================================================= #
# composer require silverstripe/recipe-testing ^1
# vendor/bin/behat-ss @<module>
# ========================================================================= #
default:
  suites: []
  extensions:
    SilverStripe\BehatExtension\MinkExtension:
      default_session: facebook_web_driver
      javascript_session: facebook_web_driver
      facebook_web_driver:
        browser: chrome
        wd_host: "http://127.0.0.1:9515" #chromedriver port
      browser_name: chrome
    SilverStripe\BehatExtension\Extension:
      bootstrap_file: vendor/silverstripe/cms/tests/behat/serve-bootstrap.php
      screenshot_path: %paths.base%/artifacts/screenshots
      retry_seconds: 4 # default is 2

[tractorcow]

The new behat-ss command will ensure that both serve and chromedriver are running for each test, and safely shuts them down at the end.

[tractorcow]

Other modules merged, and this PR rebased. Waiting to see if it all passes now. :)