Resolve: Killing Zombie Processes created by Selenium

Problem

Here's my problem.

I'm working with Selenium 2.53 (an old version, I know) and Firefox 52.8. The Gecko (driver) version is whichever works with FF 52.8. Anyhow, I create a my test script, run the test script, and see the command invokes the call to run the test. However, my browser does not open. My terminal just sits at this command running the test. Eventually, my command ends and the test failed to run. I'm running this in a secure environment on a separate machine and can't copy/paste a screenshot here.

However, here's what the error looks like - if you're lucky enough to get one.

Results :
Tests in error:
my.test.script.TestPageAUT.enterMaxChars...
    Run 1:  .... >> WebDriver Unable to bind to locking port 70...

Solution

Here's my solution.

I knew from experience that a system process opened by the driver via Java is hanging. Since I'm running this on Linux, I had to make sure I used the right Linux commands to find the processes that are running (active). Throughout my research, I learned that Linux actually has processes that are zombies (not quite dead, but not live either) which are marked with "<defunct>" next to the process.

Once I found out the process ID (PID) of the zombie process, then I just needed to kill the parent process of the zombie (because I can't kill the zombie - which I think is weird). It would be nice to just kill the zombie and automatically the parents of the zombie also die. Anyhow, in this particular case I had to kill the most super parent of the zombie in order to completely make sure the zombie process for my selenium script is dead. Selenium runs via Java by triggering the gecko driver which controls a FF process. So, to kill the zombie FF process, need to kill the Java process controlling the zombie FF process which might have a grandparent Java process.

Double check the process is dead by showing the list of active processes and seeing that there is no zombie process in the list.

Steps/Commands

Here's how I did it.

Important Note: A "kill" command has no effect on a zombie process.

List all zombie processes (ID).
$ ps aux | grep -w Z

Find the parent PID.
$ ps -o ppid <defunct_PID>         

Kill the parent
$ kill -9 <parent_PID_of_defunct_PID>

Other Useful Commands

List all running processes.
$ ps

List the running processes with Firefox.
$ ps -A | grep firefox

For more details, list with these parameters.
$ ps aux | grep firefox

Find the parent process ID.
$ ps -o ppid= -p <defunct_PID>

You can try this (kill all firefox PIDs), but this didn't work for me.
$ kill $(pgrep firefox)

Reference

• https://stackoverflow.com/questions/16944886/how-to-kill-zombie-process (3rd answer, particularly)

Review: Evolving Role of the Modern Developer

Read Article: Evolving Role of the Modern Developer

In this article, I have come to realize that my initialize thoughts on the direction of a development between the role of a developer and a QA engineer (or tester or whatever you call the person who verifies your code works as expected) is actually correct. I was confused and thought that perhaps the role of a tester evolved into just software engineer in test (SET) because of the positions that the tech giants were creating. In addition, because I see the evolution of AI testing.

But, on the spectrum of the SDLC the roles are becoming more integrated and in some respects these roles are merging into one. This doesn't mean the SDLC is going way. How a product or service is built will still need the basic engineering principles and cycles applied. However, who plays which role while creating or updating the product via SDLC will become more versatile and need a greater understanding of the customer's expectations of the product (and not just a function or feature). And, since no single developer can remember everything about a product, developers will need to know how to kindly work well with others.

Resolve: Test setup with Selenium Chrome Driver on CentOS

These are rough notes and I need to separate my resolution of Firefox from my resolution of Chrome. I also need to clean up these notes on ChromeDriver solution. I wanted to publish, however, just for future reference since I did a lot of research on the Internet and no single solution helped (but thankful for all the many solutions leading up to this solution).

=================
Who would think that setting up Selenium to run with Google Chrome on a CentOS would be so challenging? Maybe I'm just stupid?

Here's my problem, or actually my set of problems. I'm trying to run Selenium 3.13.0 with Google Chrome 61 on CentOS 7.

Problem #1: Error regarding the "webdriver.chrome.driver" system property not set.
There are so many useless articles including on StackOverflow answering this problem. Some point to Google's ChromeDriver website. Here's an example:
https://github.com/SeleniumHQ/selenium/wiki/ChromeDriver
let me to
https://sites.google.com/a/chromium.org/chromedriver/help/chrome-doesn-t-start
but I still didn't understand clearly what this site it saying (in troubleshooting the "doesn't start" issue).
Why hasn't anyone on the Google ChromeDriver just create a dumbdown version (without all these security features)? Just a very basic browser for testing an application displays decently in a Chrome browser?
Anyhow, I found this one StackOverflow answer which actually helped.
https://stackoverflow.com/questions/18674092/how-to-implement-chromedriver-in-selenium-in-linux-platform
I was simply pointing to the wrong driver. I thought the issue was with running the selenium-chrome-driver and not actually google-chrome.

Someone should update this page to be more clear in stating the default locations of Chrome installations (as mentioned in the SO article above).
https://sites.google.com/a/chromium.org/chromedriver/getting-started

Answer (#1):
Find the location of my ChromeDriver in CentOS 7.
$ whereis google-chrome
Output: 
/usr/bin/google-chrome

Set this location as the system property expected by Selenium Chrome Driver.
NOTE: If this location is already set in your $PATH, then you shouldn't have to do this.
String pathToChrome = "/usr/bin/google-chrome";
System.setProperty("webdriver.chrome.driver", pathToChrome);

Result (#1):
The Google Chrome browser launches. But, it might not actually finish loading. Sigh.

Problem #2: Error regarding the "driver.version" system property not set.
Really! Now I need to set another system property and I need to know the version? Why is my software program so version dependent? Either I can run what I need, or I can't. Backwards compatibility, anyone?
Enough complaining, here's how I resolve this issue.

Answer (#2):
Find the version of my ChromeDriver.
$ /usr/bin/google-chrome -version
Output:
Google Chrome 61.0.3163.100

Set this version as the system property expected by Selenium Chrome Driver.
String driverVersion = "Google Chrome 61.0.3163.100";
System.setProperty("driver.version", pathToChrome);
Output:
Driver info: driver.version: ChromeDriver
    at org.openqa.selenium.remote.service.DriverService.waitUntilAvailable(DriverService.java:##);

Since "ChromeDriver" is stated as the driver.version, we know that the driver.version is not getting recognized.
https://stackoverflow.com/questions/49282494/java-net-connectexception-failed-to-connect-to-localhost-error-with-selenium-3

Result (#2):
This didn't actually solve the issue. Sigh, again. However, there's another error message. Let's move forward.

Problem #3: Error regarding "[...ERROR:sandbox_linux.cc (344)] InitializeSandbox() called with multiple threads in process gpu-process."
What is this "sandbox"?
Google Search "chrome sandbox" for answers.
A couple results:
https://blog.chromium.org/2008/10/new-approach-to-browser-security-google.html
https://chromium.googlesource.com/chromium/src/+/master/docs/design/sandbox.md

I guess I didn't know that I need to know more about browser security in order to implement a simple test case using Selenium.

Answer?
https://stackoverflow.com/questions/39041146/how-do-i-pass-arguments-to-google-chrome-when-running-selenium

First step, see if running this argument in command line will work without error.
$ /usr/bin/google-chrome --no-sandbox
Output:
[...ERROR:sandbox_linux.cc (344)] InitializeSandbox() called with multiple threads in process gpu-process.
Note:
The browser opens with a banner stating "You are using an unsupported command-line flag: --no-sandbox. Stability and security will suffer."

Okay, let's see the options available for my google-chrome program.
$ /usr/bin/google-chrome --help

Let's try the the option: --app=URL
$ /usr/bin/google-chrome --app=http://www.mywebsite.com

Step Result:
The page displays, yes! I still get the the "sandbox" error, but I'm not going to investigate further especially considering the warning message under the OPTIONS when reading the --help.
It says "Google Chrome has hundreds of undocumented command-line flags that are added and removed at the whim of the developers. Here, we document relatively stable flags." In other words, if you really need help, call Google's Customer Support. Smh. I must be missing something. Like, there must be a Chrome for Developers and a Chrome for People (and I'm using the Chrome for Developers). I see now why certain developers like Mozilla Firefox more.

Okay, now let's try adding this argument into the driver we're creating...


Answer? Nope, this didn't matter in my scenario.
Set the different attributes manually in DesiredCapabilities for ChromeDriver.
@Before
public void createDriver() {
  DesiredCapabilities chromeCapabilities = DesiredCapabilities.chrome();
  chromeCapabilities.setPlatform(Platform.LINUX);
  chromeCapabilities.setJavascriptEnabled(true);
  chromeCapabilities.setAcceptInsecureCerts(true);
  driver = new RemoteWebDriver(service.getUrl(), chromeCapabilities);
}

PROBLEM (ChromeDriver): Error WebDriverException: java.net.ConnectException: Failed to connect to localhost/127.0.0.1:####
https://stackoverflow.com/questions/49227055/webdriverexception-java-net-connectexception-failed-to-connect-to-localhost-er?rq=1
Answer: 
Update and ensure all programs are compatible among Selenium, Chrome, and Java. See details in the link highlighted in the Problem.
Currently what I see listed is...
java 8v201
selenium 3.14x
chromedriver 2.46
google-chrome v72.x  (chrome://settings/help)

Okay, since I see that I had a "driver" issue with Firefox, I'm led back to this article here.
https://stackoverflow.com/questions/18674092/how-to-implement-chromedriver-in-selenium-in-linux-platform
When I run the following command, I don't have a chromedriver install.
$ chromedriver --version
Output:
chromedriver: command not found

Download and install driver from here:
https://sites.google.com/a/chromium.org/chromedriver/downloads
https://chromedriver.storage.googleapis.com/index.html?path=2.46/

<backfill by saying we installed and got browser to display, but got other error messages now>
downloading and installing chromedriver...

Resolving error message "Chrome controlled by automated test software"
Now, to remove the banner in browser stating that Chrome is controlled by a automated test software, see this:
http://www.automationtestinghub.com/selenium-chromedriver/
https://help.applitools.com/hc/en-us/articles/360007189411--Chrome-is-being-controlled-by-automated-test-software-notification

Resolving error message "Only local connections are allowed."
If the url expected is not displaying in the Chrome browser, but instead "data;", then it's related to the error regarding "Only local connections are allowed." Try these steps to resolve.
https://stackoverflow.com/questions/25080500/when-running-webdriver-with-chrome-browser-getting-message-only-local-connect

None of these search results resolved. Instead, was led back to find the correct chromedriver version compatible with my Chrome browser.
On this Download page, the Current Release section suggests going to the Version page.
http://chromedriver.chromium.org/downloads
This Version page doesn't have the drivers I'm looking for.
http://chromedriver.chromium.org/downloads/version-selection
With more research effort, I finally found the URL I need to find the correct driver for my browser version.
https://chromedriver.storage.googleapis.com/index.html

From the bottom of this page:
http://chromedriver.chromium.org/downloads
we see the last message regarding the chromedriver version going back is version 2.35 for browser version 62-64:

ChromeDriver 2.35

Supports Chrome v62-64

Changes include:

• Supports persistent connections between client application and ChromeDriver.
• Adds more devices types for mobile emulation.
• Fixes a bug in get local storage command.
• Fixes a compatibility bug that causes JavaScript code execution to fail on some versions of Chrome.
• Uses absolute time in log file.

I have browser version 61. Hahahahaaa.
Okay, well, hopefully the driver version is 2.34 according to this chromedriver build pattern.
https://chromedriver.storage.googleapis.com/index.html?path=2.34/

Before we try this version, let's quickly recap what is installed at this point.
Currently here's what is installed:
CentOS version: 7.5.1804
Google Chrome version: 61.0.3163.100
ChromeDriver version: 2.34.522913
Selenium version: 3.13.0
Note: Selenium version is found in my POM.xml and specifies 3.13.0.

Let's double check the chromedriver version installed.
$ chromedriver --version
Output:
ChromeDriver 2.34.522913 (36222509aa6e819815938cbf2709b4849735537c)

Now, let's see what happens. Yes, I got it to work!!

Result:
Finally, got my Chrome test to work.

My Code Example:
@Test
public void testChrome() throws IOException {

    WebDriver driver;

    String pathToChrome = "/usr/bin/chromedriver"; // links to actual chromedriver dir (not google-chrome)
    System.setProperty("webdriver.chrome.driver", pathToChrome);

    // System Property addition suggested by -
        // https://wiki.saucelabs.com/display/DOCS/Test+Configuration+Options#TestConfigurationOptions-ChromeDriverVersion
    // System.setProperty("chromedriverVersion", "2.34.522913");

    // == Useful Options: Use only what is required ==
    ChromeOptions options = new ChromeOptions();
//  options.setBinary(new File(pathToChrome));
//  options.addArguments("--no-sandbox");
//  options.addArguments("--app");
//  options.addArguments("--force-app-mode");
//  options.addArguments("--kiosk");
//  options.addArguments("--start-maximized");
//  options.addArguments("--disable-web-security");

    // For info, see https://www.automationtestinghub.com/selenium-chromedriver/
    options.addArguments("disable-infobars");

    driver = new ChromeDriver(options);
    driver.get(APP_URL);
    String title = driver.getTitle();

    assertTrue("Incorrect login page title!", LOGIN_PAGE_TITLE.equalsIgnoreCase(title));
    LOG.info("Title: "+title);

    // driver.quit();
}

Cheers!


PROBLEM (FirefoxDriver): Error WebDriverException: java.net.ConnectException: Failed to connect to localhost/127.0.0.1:####
Here's an insightful SO response.
https://stackoverflow.com/questions/53107233/error-org-openqa-selenium-webdriverexception-java-net-connectexception-failed

Here are the specific versions after updating my CentOS7 box.
CentOS version: 7.5.1804
Firefox version: 60.5.0
GeckoDriver version:  ???
Selenium version: 3.13.0
Note: Selenium version is found in my POM.xml and specifies 3.13.0.

Here are my Linux commands to find the versions:
$ cat /etc/redhat-release
$ firefox --version
$ geckodriver --version

Here's some background information as I investigate how to find the GeckoDriver version:
https://github.com/mozilla/geckodriver/issues/510
https://seleniumhq.github.io/selenium/docs/api/java/org/openqa/selenium/firefox/FirefoxOptions.html
https://stackoverflow.com/questions/43272919/difference-between-webdriver-firefox-marionette-webdriver-gecko-driver/43920453

Here's some information regarding the compatibility issues with Firefox and Selenium:
https://www.guru99.com/first-webdriver-script.html
https://stackoverflow.com/questions/43757984/how-to-start-firefoxdriver-using-selenium-3-4-0-using-maven
http://www.automationtestinghub.com/selenium-3/
http://www.automationtestinghub.com/selenium-3-0-launch-firefox-with-geckodriver/

Answer:
So, the root issue is that I don't have the "geckodriver" installed and I need this installed. Thus, I do the following:

1. Download latest version @ https://github.com/mozilla/geckodriver/releases/tag/v0.24.0
2. Install in my /usr/bin directory.
3. Run geckodriver version command to ensure correct version appears.
4. Update my test script by removing the property for "webdriver.firefox.driver" and adding the property "webdriver.gecko.driver".

Run Command Example:
$ geckodriver --version
Output:
geckodriver 0.24.0 ( 2019-01-28)

Code Sample (using FirefoxDriver)
@Test
public void testFirefox() throws IOException {

    WebDriver driver;
    
    String pathToGecko = "/usr/bin/geckodriver";
    System.setProperty("webdriver.gecko.driver", pathToGecko);
    
    driver = new FirefoxDriver();
    driver.get(myUrl);

    assertTrue("Incorrect homepage title!", driver.getTitle().equalsIgnoreCase(myUrlTitle));

    driver.quit();
}

Run Command Sample
$ mvn -Dtest=FirefoxTest test

Result:
Selenium opens Firefox browser to the expected page, retrieves the title, compares title, and build passes.


Using Example from Selenium's ChromeDriver class api page
Now that we did dug so deep to solve, what should have been, a simple issue - let's try using this code from ChromeDriver class. Specifically, only using the argument "DesiredCapabilities.chrome()" into creating the RemoteWebDriver.

Did it work?
Nope, got the same error regarding the sandbox initialization.

Why doesn't this [code example] work?
I'm not sure yet, but I think this might be the reason. Seems to me, Selenium uses this constructor as the default for instantiating the DesiredCapabilities object where it simply needs parameters including the browser, version, and platform.
https://seleniumhq.github.io/selenium/docs/api/java/org/openqa/selenium/remote/DesiredCapabilities.html#DesiredCapabilities-java.lang.String-java.lang.String-org.openqa.selenium.Platform-
However, on Linux platform, there are more security measures to take and Chrome understands this by enforcing user to provide further parameters.

TRY THIS:
https://www.seleniumhq.org/docs/03_webdriver.jsp#htmlunit-driver

Notes:
Selenium API (Java)
https://seleniumhq.github.io/selenium/docs/api/java/
Code Example (ChromeDriver):
https://seleniumhq.github.io/selenium/docs/api/java/org/openqa/selenium/chrome/ChromeDriver.html


https://seleniumhq.github.io/selenium/docs/api/java/org/openqa/selenium/remote/DesiredCapabilities.html

http://chromedriver.chromium.org/capabilities
https://stackoverflow.com/questions/38335671/where-can-i-find-a-list-of-all-available-chromeoption-arguments
https://chromium.googlesource.com/chromium/src/+/master/chrome/common/chrome_switches.cc
https://chromium.googlesource.com/chromium/src/+/master/chrome/common/pref_names.cc

https://peter.sh/experiments/chromium-command-line-switches/

https://www.blazemeter.com/blog/how-to-set-up-your-automated-functional-gui-tests-with-selenium-webdriver

https://medium.com/@sahajamit/selenium-chrome-dev-tools-makes-a-perfect-browser-automation-recipe-c35c7f6a2360

https://wiki.saucelabs.com/display/public/DOCS/The+Sauce+Labs+Cookbook+Home

Review: Future of Quality Assurance

I agree with this author.
https://thenewstack.io/quality-engineering-the-future-of-qa/

Quality Engineering within each Development Life Cycle phase

Since 2005, I could see "software tester" being eliminated from teams. It's the reason why I shifted from "tester" to "developer" back in 2008. I didn't expect the need for a test automation team to emerge. However, I still think that even such teams will no longer be necessary as development teams grow with understanding on how every part of the development process impacts a product's quality.

In addition, the rise of artificial intelligence (AI) will create a new path for developing automated "testers" who can simulate the behavior of product users and provide instant feedback to developers. In manufacturing or other assembling facilities, robots will become more like droids.

Regarding quality engineering, the biggest challenge I see today is with the technical leadership having a slow adoption of implementing this "quality" mindset into development teams. This leadership issue is one of the reasons why excellent developers (who already have this mindset) tend to leave and start their own companies. "Quality" developers see an opportunity to create a product (or service) that is better than an existing one. So why wait for technical leadership to see these opportunities?

Test Teams to Test Companies

Some development teams are already leading the charge with implementing the quality engineering principles and leveraging test companies instead of employing testers (or creating their own test teams). An example of an organization to leverage is TestProject.

Here's an example of how they help developers get up to speed with testing their applications (particularly web applications).
https://blog.testproject.io/2018/12/18/net-core-test-automation-selenium-page-object-page-factory/

Here's another company to leverage: TestIM

Going with this development strategy (of leveraging test companies instead of creating own test teams) will save time and money, but with savvy technical leadership.

Expert Users - the new Quality Experts

In the near future, the "User" will be the Tester, Quality Assurance Engineer, or whatever title you want to give to a person who makes sures that a product is working as expected (or beyond expectations). Rightfully so, a product is only as valuable according to the measure given by its user. It's just logical to move towards lean-agile development methodology where it's basically the developer (innovator) and user (expert) creating and maturing the product.

Closing Thoughts on Our Future

Our world will shift from a "buy-sell" model to a "use-improve" model. Therefore, relationships will no longer have a clear line between "business" and "personal" types, but all relationships will carry both types. Every relationship will be "personal" and "business". All experts will need to specialize in developing something and all innovators will need to be experts in using something.

Journey into SET: No WebElement.isClicked()

In creating the Test Framework using Selenium for testing a specific web application, I am sort of shocked that the authors of Selenium would allow a WebElement to be "clicked" but not keep track whether this WebElement "isClicked" or even "isClickable".

https://seleniumhq.github.io/selenium/docs/api/java/org/openqa/selenium/WebElement.html#click--

So, anyhow, learned the hard way and had to really pay attention to reading the large paragraph in the API's method definition when it should the method should hanlde such cases or the Object should allow for handling such cases.

But, there's my issue too - why am I not contributing to this open source library set (tool)?

I need to get my game up and contribute, not complain.

I got around my problem and solved using the:
driver.navigate.refresh()

Well, at least I now know. :D

Review: Exploratory vs Manual Testing

@fwmoyer states the difference between Exploratory Testing and Manual Testing is a mindset. And I agree with him. Here's a quick article on explaining why.

https://www.softwaretestinghelp.com/exploratory-testing-vs-scripted-testing/

Overall, context is always valuable when driving something purposeful. In order to reach quality results (i.e. results that matter and add value), context is required especially to help to create productive roadmaps whether designing, developing, and/or testing.

Reference on Test Tools of 2018

As we come to close the year 2018, here are a few articles on the test tools shaping the industry (tech and all others) and that will continue to add value into 2019.

Notice that one of the rising stars is TestIM.io. Keep an eye on this company, its tool(s), and its service(s).

Software Testing Help - Top 20 automation testing tools (Dec 2018)

https://www.softwaretestinghelp.com/top-20-automation-testing-tools/

Actually, this is the only article I found online that was recent (aka relevant) and didn't seem to have a bias toward any tools (i.e. promoting commercial over open source, etc).

More importantly, let's keep an eye on the AI automated tools. Just like robots are replacing people in factories, AI will replace the mundane programming jobs. Therefore, humans will need to learn how to establish more roles in the R&D industry (which ironically was almost wiped out by corporations in the early 2000s due to foolish business cuts).

Examining Value of Tests

Some valuable points made in this article. One thing I see missing is the accountability for those who collect user requirements and delivers requirements to the Development Team for implementation. Sometimes, the issue isn't as much as the development as it is the requirement(s) and product design. These metrics should also be collected (as defects in requirements) instead of counted towards bugs or issues in development.

https://blog.testproject.io/2018/12/04/how-do-i-know-my-tests-add-value/

JayWay - JsonPath

Today I read a couple articles from TestProject and found this new library quite interesting (when a Maven project uses this library).
https://github.com/json-path/JsonPath

This library helps to format the structure of a JSON file/response.

Journey into SET: Creating Test Framework in Selenium

So, I'm still on my journey back into becoming a Software Test Engineer (expert) and geesh, there's a lot to catch up on.

Today, I'm looking into creating a Test Framework with Selenium.

I googled on "selenium creating test framework" to see results and learn.

Why?
First, let's answer why I'm researching this topic. I am able to create test scripts using Selenium and my test are passing. However, as I'm creating a test suite and adding more tests, I find myself using the same code for multiple @Test. I'd like to reduce the code, use wrappers around Selenium's code, and condense my @Test to be specific on focusing on the test case. This also helps with test script maintenance. For instance, if I need all of my test scripts to navigate to their corresponding page (add form -> New Form, edit form -> Form, etc). Then, I realize I need the User to click OK to a dialog just implemented before displaying any form. I don't want to have to check this code for each test script in each @Test.

YouTube(s) I watched.
https://www.youtube.com/watch?v=DO8KVe00kcU

Article(s) I read.
https://simpleprogrammer.com/creating-automated-testing-framework-selenium/
Not very useful, unless going to author's course.
https://www.testingexcellence.com/develop-test-automation-framework-scratch/
I like the breakdown on this author's GitHub and see it's practical use. So, I decided to implement this same framework.

Pattern Insights
Since I'm using the PageObject pattern, this was very helpful to know.
https://github.com/SeleniumHQ/selenium/wiki/PageFactory

Software Testing Article: TestIM Era of AI

Today's article that catches my attention and has good links in it is by TestIM's webinar.

• https://blog.testim.io/era-of-ai/

Software Testing article on 5 ways AI will change software testing

Article for today's reading...
https://techbeacon.com/5-ways-ai-will-change-software-testing

And here's an outstanding source of blogs related...
http://blog.testim.io/

It's important to try to keep up and not get discouraged at the pace of technology. :)