Resolve: Killing Zombie Processes created by Selenium

Problem

Here's my problem.

I'm working with Selenium 2.53 (an old version, I know) and Firefox 52.8. The Gecko (driver) version is whichever works with FF 52.8. Anyhow, I create a my test script, run the test script, and see the command invokes the call to run the test. However, my browser does not open. My terminal just sits at this command running the test. Eventually, my command ends and the test failed to run. I'm running this in a secure environment on a separate machine and can't copy/paste a screenshot here.

However, here's what the error looks like - if you're lucky enough to get one.

Results :
Tests in error:
my.test.script.TestPageAUT.enterMaxChars...
    Run 1:  .... >> WebDriver Unable to bind to locking port 70...

Solution

Here's my solution.

I knew from experience that a system process opened by the driver via Java is hanging. Since I'm running this on Linux, I had to make sure I used the right Linux commands to find the processes that are running (active). Throughout my research, I learned that Linux actually has processes that are zombies (not quite dead, but not live either) which are marked with "<defunct>" next to the process.

Once I found out the process ID (PID) of the zombie process, then I just needed to kill the parent process of the zombie (because I can't kill the zombie - which I think is weird). It would be nice to just kill the zombie and automatically the parents of the zombie also die. Anyhow, in this particular case I had to kill the most super parent of the zombie in order to completely make sure the zombie process for my selenium script is dead. Selenium runs via Java by triggering the gecko driver which controls a FF process. So, to kill the zombie FF process, need to kill the Java process controlling the zombie FF process which might have a grandparent Java process.

Double check the process is dead by showing the list of active processes and seeing that there is no zombie process in the list.

Steps/Commands

Here's how I did it.

Important Note: A "kill" command has no effect on a zombie process.

List all zombie processes (ID).
$ ps aux | grep -w Z

Find the parent PID.
$ ps -o ppid <defunct_PID>         

Kill the parent
$ kill -9 <parent_PID_of_defunct_PID>

Other Useful Commands

List all running processes.
$ ps

List the running processes with Firefox.
$ ps -A | grep firefox

For more details, list with these parameters.
$ ps aux | grep firefox

Find the parent process ID.
$ ps -o ppid= -p <defunct_PID>

You can try this (kill all firefox PIDs), but this didn't work for me.
$ kill $(pgrep firefox)

Reference

• https://stackoverflow.com/questions/16944886/how-to-kill-zombie-process (3rd answer, particularly)

Resolve: Test setup with Selenium Chrome Driver on CentOS

These are rough notes and I need to separate my resolution of Firefox from my resolution of Chrome. I also need to clean up these notes on ChromeDriver solution. I wanted to publish, however, just for future reference since I did a lot of research on the Internet and no single solution helped (but thankful for all the many solutions leading up to this solution).

=================
Who would think that setting up Selenium to run with Google Chrome on a CentOS would be so challenging? Maybe I'm just stupid?

Here's my problem, or actually my set of problems. I'm trying to run Selenium 3.13.0 with Google Chrome 61 on CentOS 7.

Problem #1: Error regarding the "webdriver.chrome.driver" system property not set.
There are so many useless articles including on StackOverflow answering this problem. Some point to Google's ChromeDriver website. Here's an example:
https://github.com/SeleniumHQ/selenium/wiki/ChromeDriver
let me to
https://sites.google.com/a/chromium.org/chromedriver/help/chrome-doesn-t-start
but I still didn't understand clearly what this site it saying (in troubleshooting the "doesn't start" issue).
Why hasn't anyone on the Google ChromeDriver just create a dumbdown version (without all these security features)? Just a very basic browser for testing an application displays decently in a Chrome browser?
Anyhow, I found this one StackOverflow answer which actually helped.
https://stackoverflow.com/questions/18674092/how-to-implement-chromedriver-in-selenium-in-linux-platform
I was simply pointing to the wrong driver. I thought the issue was with running the selenium-chrome-driver and not actually google-chrome.

Someone should update this page to be more clear in stating the default locations of Chrome installations (as mentioned in the SO article above).
https://sites.google.com/a/chromium.org/chromedriver/getting-started

Answer (#1):
Find the location of my ChromeDriver in CentOS 7.
$ whereis google-chrome
Output: 
/usr/bin/google-chrome

Set this location as the system property expected by Selenium Chrome Driver.
NOTE: If this location is already set in your $PATH, then you shouldn't have to do this.
String pathToChrome = "/usr/bin/google-chrome";
System.setProperty("webdriver.chrome.driver", pathToChrome);

Result (#1):
The Google Chrome browser launches. But, it might not actually finish loading. Sigh.

Problem #2: Error regarding the "driver.version" system property not set.
Really! Now I need to set another system property and I need to know the version? Why is my software program so version dependent? Either I can run what I need, or I can't. Backwards compatibility, anyone?
Enough complaining, here's how I resolve this issue.

Answer (#2):
Find the version of my ChromeDriver.
$ /usr/bin/google-chrome -version
Output:
Google Chrome 61.0.3163.100

Set this version as the system property expected by Selenium Chrome Driver.
String driverVersion = "Google Chrome 61.0.3163.100";
System.setProperty("driver.version", pathToChrome);
Output:
Driver info: driver.version: ChromeDriver
    at org.openqa.selenium.remote.service.DriverService.waitUntilAvailable(DriverService.java:##);

Since "ChromeDriver" is stated as the driver.version, we know that the driver.version is not getting recognized.
https://stackoverflow.com/questions/49282494/java-net-connectexception-failed-to-connect-to-localhost-error-with-selenium-3

Result (#2):
This didn't actually solve the issue. Sigh, again. However, there's another error message. Let's move forward.

Problem #3: Error regarding "[...ERROR:sandbox_linux.cc (344)] InitializeSandbox() called with multiple threads in process gpu-process."
What is this "sandbox"?
Google Search "chrome sandbox" for answers.
A couple results:
https://blog.chromium.org/2008/10/new-approach-to-browser-security-google.html
https://chromium.googlesource.com/chromium/src/+/master/docs/design/sandbox.md

I guess I didn't know that I need to know more about browser security in order to implement a simple test case using Selenium.

Answer?
https://stackoverflow.com/questions/39041146/how-do-i-pass-arguments-to-google-chrome-when-running-selenium

First step, see if running this argument in command line will work without error.
$ /usr/bin/google-chrome --no-sandbox
Output:
[...ERROR:sandbox_linux.cc (344)] InitializeSandbox() called with multiple threads in process gpu-process.
Note:
The browser opens with a banner stating "You are using an unsupported command-line flag: --no-sandbox. Stability and security will suffer."

Okay, let's see the options available for my google-chrome program.
$ /usr/bin/google-chrome --help

Let's try the the option: --app=URL
$ /usr/bin/google-chrome --app=http://www.mywebsite.com

Step Result:
The page displays, yes! I still get the the "sandbox" error, but I'm not going to investigate further especially considering the warning message under the OPTIONS when reading the --help.
It says "Google Chrome has hundreds of undocumented command-line flags that are added and removed at the whim of the developers. Here, we document relatively stable flags." In other words, if you really need help, call Google's Customer Support. Smh. I must be missing something. Like, there must be a Chrome for Developers and a Chrome for People (and I'm using the Chrome for Developers). I see now why certain developers like Mozilla Firefox more.

Okay, now let's try adding this argument into the driver we're creating...


Answer? Nope, this didn't matter in my scenario.
Set the different attributes manually in DesiredCapabilities for ChromeDriver.
@Before
public void createDriver() {
  DesiredCapabilities chromeCapabilities = DesiredCapabilities.chrome();
  chromeCapabilities.setPlatform(Platform.LINUX);
  chromeCapabilities.setJavascriptEnabled(true);
  chromeCapabilities.setAcceptInsecureCerts(true);
  driver = new RemoteWebDriver(service.getUrl(), chromeCapabilities);
}

PROBLEM (ChromeDriver): Error WebDriverException: java.net.ConnectException: Failed to connect to localhost/127.0.0.1:####
https://stackoverflow.com/questions/49227055/webdriverexception-java-net-connectexception-failed-to-connect-to-localhost-er?rq=1
Answer: 
Update and ensure all programs are compatible among Selenium, Chrome, and Java. See details in the link highlighted in the Problem.
Currently what I see listed is...
java 8v201
selenium 3.14x
chromedriver 2.46
google-chrome v72.x  (chrome://settings/help)

Okay, since I see that I had a "driver" issue with Firefox, I'm led back to this article here.
https://stackoverflow.com/questions/18674092/how-to-implement-chromedriver-in-selenium-in-linux-platform
When I run the following command, I don't have a chromedriver install.
$ chromedriver --version
Output:
chromedriver: command not found

Download and install driver from here:
https://sites.google.com/a/chromium.org/chromedriver/downloads
https://chromedriver.storage.googleapis.com/index.html?path=2.46/

<backfill by saying we installed and got browser to display, but got other error messages now>
downloading and installing chromedriver...

Resolving error message "Chrome controlled by automated test software"
Now, to remove the banner in browser stating that Chrome is controlled by a automated test software, see this:
http://www.automationtestinghub.com/selenium-chromedriver/
https://help.applitools.com/hc/en-us/articles/360007189411--Chrome-is-being-controlled-by-automated-test-software-notification

Resolving error message "Only local connections are allowed."
If the url expected is not displaying in the Chrome browser, but instead "data;", then it's related to the error regarding "Only local connections are allowed." Try these steps to resolve.
https://stackoverflow.com/questions/25080500/when-running-webdriver-with-chrome-browser-getting-message-only-local-connect

None of these search results resolved. Instead, was led back to find the correct chromedriver version compatible with my Chrome browser.
On this Download page, the Current Release section suggests going to the Version page.
http://chromedriver.chromium.org/downloads
This Version page doesn't have the drivers I'm looking for.
http://chromedriver.chromium.org/downloads/version-selection
With more research effort, I finally found the URL I need to find the correct driver for my browser version.
https://chromedriver.storage.googleapis.com/index.html

From the bottom of this page:
http://chromedriver.chromium.org/downloads
we see the last message regarding the chromedriver version going back is version 2.35 for browser version 62-64:

ChromeDriver 2.35

Supports Chrome v62-64

Changes include:

• Supports persistent connections between client application and ChromeDriver.
• Adds more devices types for mobile emulation.
• Fixes a bug in get local storage command.
• Fixes a compatibility bug that causes JavaScript code execution to fail on some versions of Chrome.
• Uses absolute time in log file.

I have browser version 61. Hahahahaaa.
Okay, well, hopefully the driver version is 2.34 according to this chromedriver build pattern.
https://chromedriver.storage.googleapis.com/index.html?path=2.34/

Before we try this version, let's quickly recap what is installed at this point.
Currently here's what is installed:
CentOS version: 7.5.1804
Google Chrome version: 61.0.3163.100
ChromeDriver version: 2.34.522913
Selenium version: 3.13.0
Note: Selenium version is found in my POM.xml and specifies 3.13.0.

Let's double check the chromedriver version installed.
$ chromedriver --version
Output:
ChromeDriver 2.34.522913 (36222509aa6e819815938cbf2709b4849735537c)

Now, let's see what happens. Yes, I got it to work!!

Result:
Finally, got my Chrome test to work.

My Code Example:
@Test
public void testChrome() throws IOException {

    WebDriver driver;

    String pathToChrome = "/usr/bin/chromedriver"; // links to actual chromedriver dir (not google-chrome)
    System.setProperty("webdriver.chrome.driver", pathToChrome);

    // System Property addition suggested by -
        // https://wiki.saucelabs.com/display/DOCS/Test+Configuration+Options#TestConfigurationOptions-ChromeDriverVersion
    // System.setProperty("chromedriverVersion", "2.34.522913");

    // == Useful Options: Use only what is required ==
    ChromeOptions options = new ChromeOptions();
//  options.setBinary(new File(pathToChrome));
//  options.addArguments("--no-sandbox");
//  options.addArguments("--app");
//  options.addArguments("--force-app-mode");
//  options.addArguments("--kiosk");
//  options.addArguments("--start-maximized");
//  options.addArguments("--disable-web-security");

    // For info, see https://www.automationtestinghub.com/selenium-chromedriver/
    options.addArguments("disable-infobars");

    driver = new ChromeDriver(options);
    driver.get(APP_URL);
    String title = driver.getTitle();

    assertTrue("Incorrect login page title!", LOGIN_PAGE_TITLE.equalsIgnoreCase(title));
    LOG.info("Title: "+title);

    // driver.quit();
}

Cheers!


PROBLEM (FirefoxDriver): Error WebDriverException: java.net.ConnectException: Failed to connect to localhost/127.0.0.1:####
Here's an insightful SO response.
https://stackoverflow.com/questions/53107233/error-org-openqa-selenium-webdriverexception-java-net-connectexception-failed

Here are the specific versions after updating my CentOS7 box.
CentOS version: 7.5.1804
Firefox version: 60.5.0
GeckoDriver version:  ???
Selenium version: 3.13.0
Note: Selenium version is found in my POM.xml and specifies 3.13.0.

Here are my Linux commands to find the versions:
$ cat /etc/redhat-release
$ firefox --version
$ geckodriver --version

Here's some background information as I investigate how to find the GeckoDriver version:
https://github.com/mozilla/geckodriver/issues/510
https://seleniumhq.github.io/selenium/docs/api/java/org/openqa/selenium/firefox/FirefoxOptions.html
https://stackoverflow.com/questions/43272919/difference-between-webdriver-firefox-marionette-webdriver-gecko-driver/43920453

Here's some information regarding the compatibility issues with Firefox and Selenium:
https://www.guru99.com/first-webdriver-script.html
https://stackoverflow.com/questions/43757984/how-to-start-firefoxdriver-using-selenium-3-4-0-using-maven
http://www.automationtestinghub.com/selenium-3/
http://www.automationtestinghub.com/selenium-3-0-launch-firefox-with-geckodriver/

Answer:
So, the root issue is that I don't have the "geckodriver" installed and I need this installed. Thus, I do the following:

1. Download latest version @ https://github.com/mozilla/geckodriver/releases/tag/v0.24.0
2. Install in my /usr/bin directory.
3. Run geckodriver version command to ensure correct version appears.
4. Update my test script by removing the property for "webdriver.firefox.driver" and adding the property "webdriver.gecko.driver".

Run Command Example:
$ geckodriver --version
Output:
geckodriver 0.24.0 ( 2019-01-28)

Code Sample (using FirefoxDriver)
@Test
public void testFirefox() throws IOException {

    WebDriver driver;
    
    String pathToGecko = "/usr/bin/geckodriver";
    System.setProperty("webdriver.gecko.driver", pathToGecko);
    
    driver = new FirefoxDriver();
    driver.get(myUrl);

    assertTrue("Incorrect homepage title!", driver.getTitle().equalsIgnoreCase(myUrlTitle));

    driver.quit();
}

Run Command Sample
$ mvn -Dtest=FirefoxTest test

Result:
Selenium opens Firefox browser to the expected page, retrieves the title, compares title, and build passes.


Using Example from Selenium's ChromeDriver class api page
Now that we did dug so deep to solve, what should have been, a simple issue - let's try using this code from ChromeDriver class. Specifically, only using the argument "DesiredCapabilities.chrome()" into creating the RemoteWebDriver.

Did it work?
Nope, got the same error regarding the sandbox initialization.

Why doesn't this [code example] work?
I'm not sure yet, but I think this might be the reason. Seems to me, Selenium uses this constructor as the default for instantiating the DesiredCapabilities object where it simply needs parameters including the browser, version, and platform.
https://seleniumhq.github.io/selenium/docs/api/java/org/openqa/selenium/remote/DesiredCapabilities.html#DesiredCapabilities-java.lang.String-java.lang.String-org.openqa.selenium.Platform-
However, on Linux platform, there are more security measures to take and Chrome understands this by enforcing user to provide further parameters.

TRY THIS:
https://www.seleniumhq.org/docs/03_webdriver.jsp#htmlunit-driver

Notes:
Selenium API (Java)
https://seleniumhq.github.io/selenium/docs/api/java/
Code Example (ChromeDriver):
https://seleniumhq.github.io/selenium/docs/api/java/org/openqa/selenium/chrome/ChromeDriver.html


https://seleniumhq.github.io/selenium/docs/api/java/org/openqa/selenium/remote/DesiredCapabilities.html

http://chromedriver.chromium.org/capabilities
https://stackoverflow.com/questions/38335671/where-can-i-find-a-list-of-all-available-chromeoption-arguments
https://chromium.googlesource.com/chromium/src/+/master/chrome/common/chrome_switches.cc
https://chromium.googlesource.com/chromium/src/+/master/chrome/common/pref_names.cc

https://peter.sh/experiments/chromium-command-line-switches/

https://www.blazemeter.com/blog/how-to-set-up-your-automated-functional-gui-tests-with-selenium-webdriver

https://medium.com/@sahajamit/selenium-chrome-dev-tools-makes-a-perfect-browser-automation-recipe-c35c7f6a2360

https://wiki.saucelabs.com/display/public/DOCS/The+Sauce+Labs+Cookbook+Home

Journey into SET: No WebElement.isClicked()

In creating the Test Framework using Selenium for testing a specific web application, I am sort of shocked that the authors of Selenium would allow a WebElement to be "clicked" but not keep track whether this WebElement "isClicked" or even "isClickable".

https://seleniumhq.github.io/selenium/docs/api/java/org/openqa/selenium/WebElement.html#click--

So, anyhow, learned the hard way and had to really pay attention to reading the large paragraph in the API's method definition when it should the method should hanlde such cases or the Object should allow for handling such cases.

But, there's my issue too - why am I not contributing to this open source library set (tool)?

I need to get my game up and contribute, not complain.

I got around my problem and solved using the:
driver.navigate.refresh()

Well, at least I now know. :D

Selenium Firefox - Test timeout error after waiting for timeout page

In this test, I want to verify the Timeout Error Page displays after a User has logged in successfully, but has not used the Application after 15 minutes.

Remember, my timeout is in units of seconds. So, adjusting my browser timeout to 15 * 60 seconds plus 30 more seconds to ensure my browser doesn't throw a timeout error before my application timeout. In case this calculation moved too fast because you haven't gotten your coffee yet, I need to wait for 15 minutes, these 15 minutes each have 60 seconds (i.e. 1 minute = 60 seconds, and I must test in seconds according to my function of units), and then I need to wait a few more seconds for my test script to capture the timeout text from the page display before my test browser times out (i.e. plus 30 seconds).

Here's my code.

@Test
public void testLoginTimeoutError() {

    WebDriver firefoxDriver = new FirefoxDriver( new FirefoxProfile() );
    firefoxDriver.manage().timeouts().implicitlyWait(60*15+30, TimeUnit.SECONDS);

    Application.login(firefoxDriver, url, username, password);

    WebElement myTimeOutElement = (new WebDriverWait(firefoxDriver, 60*15+5))
        .until(ExpectedConditions.presenceOfElementLocated(By.cssSelector("blockquote h2")));

    String timeOutText = myTimeOutElement.getText();

    Assert.assertTrue("Error: The timeout error did not display as expected.",
                                   timeOutText.equalsIgnoreCase("Your session has timed out"));
}

References
https://www.seleniumhq.org/docs/04_webdriver_advanced.jsp

Selenium Firefox - Test logging into two windows

I have a test case where I open first window and login. Next, I open a second window (not a tab) and I expect that I don't have to login. However, what I'm trying to understand is if the caching of the user credentials is in the browser session or not. I assume it is since I don't have to log back in (from a different window) when I'm already logged in another window that's still open.

I looked at this article, took some pointers and got the new tab to work.
https://www.testingexcellence.com/open-new-tab-browser-using-selenium-webdriver-java/

However, still working on the new window. Here's my code.

@Test
public void testNoLoginOnNewWindow() {

    WebDriver driver2;

    // Browser opened already (from @Before), login to application
    JavelinApplication.login(driver1, url, username, password);

    // Assert user's homepage displays

    // Open new browser, navigate to application
    driver2 = new FirefoxDriver(new FirefoxProfile());
    driver2.manage().timeouts().implicitlyWait(5, TimeUnit.SECONDS);
    driver2.navigate().to(url);

    // Assert user's homepage displays without login credentials asked

    driver2.quit();
}

This isn't quite working as expected yet. So, I read this article which didn't really help much:
https://stackoverflow.com/questions/17325629/how-to-open-a-new-window-on-a-browser-using-selenium-webdriver-for-python

But, it got me thinking. So, I re-read this article to leverage the code and open a new window from the existing open browser (similar to File -> New Window).
https://www.testingexcellence.com/open-new-tab-browser-using-selenium-webdriver-java/

Here's my code which actually worked as expected according to my test case.

@Test
public void testNoLoginOnNewWindow() {

    // Browser opened already (from @Before), login to application
    JavelinApplication.login(driver, url, username, password);

    // Assert user's homepage displays

    // Open new browser, navigate to application
    driver.findElement(By.cssSelector("body")).sendKeys(Keys.CONTROL + "n");
    ArrayList<String> windows = new ArrayList<String> (driver.getWindowHandles());
    driver.switchTo().window(windows.get(1));
    driver.navigate().to(url);

    // Assert user's homepage displays without login credentials asked
}

Yay, hooray!

Here's my reasoning between the different code used.

On a Linux env, I already have one Firefox window (#1) opened with My Application homepage displayed. When I open a new Firefox window (#2) from the OS, I must enter the username/password credentials. I enter the username/password and see My Application homepage displayed. When I open a new Firefox window (#3) from one of the existing Firefox windows (#1 or #2), I am not asked for login credentials and see My Application homepage displayed.

On a Windows env, I already have one Firefox window (#1) opened with my Application homepage displayed. When I open a new Firefox window (#2) from the OS (using Windows Start Menu -> Run -> Type firefox), I see My Application homepage displayed in window and am not asked for user credentials.

My assumption is that the different behavior is due to the Operating System. I am not sure and would need to do further research and testing, but this explanation is sufficient for now since my goal with the test case was reached. Cheers!

Selenium Firefox - Test using FindByElement( By.ClassName() )

This is very similar to the Test using FindByElement( By.XPath() ).

The change in this article is replacing the use of "xpath" obviously.

[Test Utility Class]
public static Boolean isDisclaimerDisplayed(WebDriver webDriver) {
    String modalTitle = webDriver.findByElement(By.className("modal-title").getText();
    return modalTitle.equalsIgnoreCase("Notice to All Users");
}

That's it. Cheers!

References
https://www.softwaretestingmaterial.com/how-to-locate-element-by-class-name-locator/
https://saucelabs.com/resources/articles/selenium-tips-css-selectors

Selenium Firefox - Test using FindByElement( By.XPath() )

Today, the goal is to verify the home page for a user after login. Actually, a disclaimer is displayed to the user before the user can access the home page. A modal is displayed with the disclaimer agreement and waits for the user to accept the disclaimer before the application redirects the user to their homepage.

So, the title of the modal include this text: "Notice to All Users".

I want my test to find this modal title and verify that the text is "Notice to All Users". This will also confirm that my login test passes because when I enter an unrecognized username the login dialog continues to display as opposed to showing an error message or error page. (In my opinion, this is poor design of the application and I will fix this if I ever get the time.)

My Code:

[Test Script Class]
@Test
public void testLogin() {
    Application.login(webDriver, url, username, password);
    boolean isDisplayed = Application.isDisclaimerDisplayed(webDriver);
    Assert.assertTrue("User successfully logged into Application.", isDisplayed);
}

[Test Utility Class]
public static Boolean isDisclaimerDisplayed(WebDriver webDriver) {
    String xpathRegEx = "//h3[@class='modal-title']";
    String modalTitle = webDriver.findByElement(By.xpath(xpathRegEx).getText();
    return modalTitle.equalsIgnoreCase("Notice to All Users");
}

One side of Caution here: This is slower than other methods of finding elements like by ID or by Class Name because this method looks through the entire XML formatted HTML document to find the element expected.


References
https://www.softwaretestingmaterial.com/how-to-locate-element-by-xpath-locator/
https://saucelabs.com/resources/articles/selenium-tips-css-selectors