Visual Test Automation with ImageMagick and Selenium Webdriver

In this article, I will present you how to do visual test automation with ImageMagick and Selenium Webdriver, and AShot. This tool and framework combination is fully open-source and when you successfully complete the settings, you can start to do visual testing with them without any cost. :)

First, I want to describe the tools:

Selenium Webdriver: We will use webdriver to navigate, interrogate and manipulate the websites.

ImageMagick: ImageMagick is our image processing and comparison tool.

Link: http://www.imagemagick.org/script/index.php

Pure JAVA Interface (im4java): http://im4java.sourceforge.net/

AShot: AShot is our Webdriver Screenshot utility tool. We will capture the screenshots with AShot. It is developed by Yandex and its open source.

Link: https://github.com/yandex-qatools/ashot

Step by Step Installations

 Step-1: Selenium Webdriver

 If you do not know how to start using Selenium Webdriver, please first read this article.

 Step-2: Visual C++ Redistributable Packages for Visual Studio

 Go to this link: https://www.itechtics.com/microsoft-visual-c-redistributable-versions-direct-download-links/#Microsoft_Visual_C_2019_Redistributable and install it.

Step-3: ImageMagick

 Go to http://www.imagemagick.org/script/binary-releases.php and install Windows Binary Release, during installation select all options and set its installation directory location to the system path.

It is important to select all options.

UPDATE: For MAC OS users, they need to install ImageMagick by using the brew command.

brew install ImageMagick

By installing ImageMagick, you will be able to run the “compare” method of ImageMagick by the terminal.

How to Setup Project for ImageMagick and Selenium

Test Scenario:

– Open http://www.google.com

– Unhide the search area because it has dynamic content which is the cursor. It is blinking.

– Take a screenshot of the page.

– At first, run, save it to the ScreenShots>${testName} folder as ${testName}_Baseline.png and ${testName}_Actual.png.

– At the second run, update the actual image, compare it with the baseline image, and put the difference as ${testName}_Diff.png

– Put all differences in the “Differences” folder.

– Verify that if they are similar test passed. If not test failed.

Project Setup:

Open IntelliJ and click File -> New -> Project -> Maven

Then, fill GroupId and ArtifactId.

Click Next and then give a name to the project and click Next.

After that, we should modify our pom.xml

Always check libraries latest versions from https://mvnrepository.com

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>tests.VisualTest</groupId>
    <artifactId>tests.VisualTest</artifactId>
    <version>1.0-SNAPSHOT</version>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>11</source>
                    <target>11</target>
                </configuration>
            </plugin>
        </plugins>
    </build>

    <dependencies>
        <dependency>
            <groupId>org.seleniumhq.selenium</groupId>
            <artifactId>selenium-java</artifactId>
            <version>3.141.59</version>
        </dependency>

        <dependency>
            <groupId>org.testng</groupId>
            <artifactId>testng</artifactId>
            <version>7.4.0</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.im4java</groupId>
            <artifactId>im4java</artifactId>
            <version>1.4.0</version>
        </dependency>

        <dependency>
            <groupId>ru.yandex.qatools.ashot</groupId>
            <artifactId>ashot</artifactId>
            <version>1.5.3</version>
        </dependency>

        <dependency>
            <groupId>commons-io</groupId>
            <artifactId>commons-io</artifactId>
            <version>2.8.0</version>
        </dependency>

        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <version>1.18.20</version>
            <scope>provided</scope>
        </dependency>
    </dependencies>
</project>

ImageMagick with Selenium and TestNG Test Code

I want to share with you the project structure below.

BaseTest: We are doing before and after operations for our tests.

Steps: All operations for our tests are in this class. 

VisualTest: It is our test class that uses the steps.

Utils: These are the utility classes that we are using in the Steps Class.

Let’s look at the codes inside each class together. I tried to add comments in the code to describe the operations. If you have any questions or problems, please write a comment and we will try to do our best to help you.

BaseTest

package tests;

import java.lang.reflect.Method;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.BeforeMethod;
import steps.Steps;
import utils.ChromeOptionsUtil;

public class BaseTest {
    ChromeOptionsUtil chromeOptionsUtil = new ChromeOptionsUtil();
    public WebDriver driver;
    Steps steps;

    @BeforeClass
    public void setupTestClass() {
        driver = new ChromeDriver(chromeOptionsUtil.getChromeOptions());
        driver.manage().window().maximize();
        steps = new Steps(driver);
        steps.preTestSetup();
    }

    @BeforeMethod
    public void setupTestMethod(Method method) {
        steps.folderUtil.setUpFilesAndFolders(method.getName());
    }

    @AfterClass
    public void quitDriver() {
        driver.quit();
    }
}

Steps

package steps;

import java.io.File;
import lombok.SneakyThrows;
import org.apache.commons.io.FileUtils;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.support.ui.WebDriverWait;
import ru.yandex.qatools.ashot.Screenshot;
import utils.FolderUtil;
import utils.ImageMagickUtil;
import utils.JSUtil;
import utils.ScreenshotUtil;

public class Steps {
    public WebDriver     driver;
    public WebDriverWait wait;
    public JSUtil        jsUtil;
    public String        url = "https://www.google.com";
    public Screenshot    googleScreenshot;

    public FolderUtil      folderUtil      = new FolderUtil();
    public ScreenshotUtil  screenshotUtil  = new ScreenshotUtil();
    public ImageMagickUtil imageMagickUtil = new ImageMagickUtil();

    public Steps(WebDriver driver) {
        this.driver = driver;
        wait = new WebDriverWait(driver, 10);
        jsUtil = new JSUtil(wait, driver);
    }

    @SneakyThrows
    public void preTestSetup() {
        //Create screenshot and differences folders if they are not exist
        folderUtil.createFolder(folderUtil.parentScreenShotsLocation);
        folderUtil.createFolder(folderUtil.parentDifferencesLocation);

        //Clean Differences Root Folder
        File differencesFolder = new File(folderUtil.parentDifferencesLocation);
        FileUtils.cleanDirectory(differencesFolder);

        driver.navigate().to(url);
        jsUtil.waitJS();
        jsUtil.hideDynamicContent();
    }

    public Steps givenITakeScreenShot() {
        //Take ScreenShot with AShot
        googleScreenshot = screenshotUtil.takeScreenshot(driver);
        return this;
    }

    @SneakyThrows
    public Steps whenISaveTheScreenShotsToFolders() {
        //Write actual screenshot to the actual screenshot path
        folderUtil.writeScreenshotToFolder(googleScreenshot);
        return this;
    }

    @SneakyThrows
    public Steps thenIShouldCompareScreenshotsSuccessfully() {
        //Do image comparison
        imageMagickUtil.doComparison(googleScreenshot, folderUtil);
        return this;
    }
}

VisualTest

package tests;

import org.testng.annotations.Test;

public class VisualTest extends BaseTest {
    @Test
    public void visualTest() {
        steps
            .givenITakeScreenShot()
            .whenISaveTheScreenShotsToFolders()
            .thenIShouldCompareScreenshotsSuccessfully();
    }
}

ChromeOptions

package utils;

import org.openqa.selenium.chrome.ChromeOptions;

public class ChromeOptionsUtil {
    public ChromeOptions getChromeOptions(){
        ChromeOptions options = new ChromeOptions();
        options.addArguments("disable-infobars");
        options.addArguments("--disable-extensions");
        return options;
    }
}

FolderUtil

package utils;

import java.io.File;
import java.io.IOException;
import javax.imageio.ImageIO;
import ru.yandex.qatools.ashot.Screenshot;

public class FolderUtil {
    //Main Directory of the test code
    public String currentDir = System.getProperty("user.dir");

    //Test name
    public String testName;

    //Test Screenshot directory
    public String testScreenShotDirectory;

    //Main screenshot directory
    public String parentScreenShotsLocation = currentDir + "/ScreenShots/";

    //Main differences directory
    public String parentDifferencesLocation = currentDir + "/Differences/";

    //Element screenshot paths
    public String baselineScreenShotPath;
    public String actualScreenShotPath;
    public String differenceScreenShotPath;

    //Image files
    public File baselineImageFile;
    public File actualImageFile;
    public File differenceImageFile;
    public File differenceFileForParent;

    public void setUpFilesAndFolders(String name) {
        //Get the test name to create a specific screenshot folder for each test.
        testName = name;
        System.out.println("Test Name: " + testName + "\n");

        //Create a specific directory for a test
        testScreenShotDirectory = parentScreenShotsLocation + testName + "/";
        createFolder(testScreenShotDirectory);

        //Declare element screenshot paths, concatenate with the test name.
        declareScreenShotPaths(testName + "_Baseline.png", testName + "_Actual.png", testName + "_Diff.png");
    }

    //Create Folder Method
    public void createFolder(String path) {
        File testDirectory = new File(path);
        if (!testDirectory.exists()) {
            if (testDirectory.mkdir()) {
                System.out.println("Directory: " + path + " is created!");
            } else {
                System.out.println("Failed to create directory: " + path);
            }
        } else {
            System.out.println("Directory already exists: " + path);
        }
    }

    //Write
    public void writeScreenshotToFolder(Screenshot screenshot) throws IOException {
        ImageIO.write(screenshot.getImage(), "PNG", actualImageFile);
    }

    //Screenshot paths
    public void declareScreenShotPaths(String baseline, String actual, String diff) {
        //BaseLine, Actual, Difference Photo Paths
        baselineScreenShotPath = testScreenShotDirectory + baseline;
        actualScreenShotPath = testScreenShotDirectory + actual;
        differenceScreenShotPath = testScreenShotDirectory + diff;

        //BaseLine, Actual Photo Files
        baselineImageFile = new File(baselineScreenShotPath);
        actualImageFile = new File(actualScreenShotPath);
        differenceImageFile = new File(differenceScreenShotPath);

        //For copying difference to the parent Difference Folder
        differenceFileForParent = new File(parentDifferencesLocation + diff);
    }
}

ImageMagickUtil

package utils;

import com.google.common.io.Files;
import javax.imageio.ImageIO;
import org.im4java.core.CompareCmd;
import org.im4java.core.IMOperation;
import org.im4java.process.StandardStream;
import ru.yandex.qatools.ashot.Screenshot;

public class ImageMagickUtil {
    //ImageMagick Compare Method
    public void compareImagesWithImageMagick(String expected, String actual, String difference, FolderUtil folderUtil) throws Exception {
        // This class implements the processing of os-commands using a ProcessBuilder.
        // This is the core class of the im4java-library where all the magic takes place.
        //ProcessStarter.setGlobalSearchPath("C:\\Program Files\\ImageMagick-7.0.4-Q16");

        // This instance wraps the compare command
        CompareCmd compare = new CompareCmd();

        // Set the ErrorConsumer for the stderr of the ProcessStarter.
        compare.setErrorConsumer(StandardStream.STDERR);

        // Create ImageMagick Operation Object
        IMOperation cmpOp = new IMOperation();

        //Add option -fuzz to the ImageMagick commandline
        //With Fuzz we can ignore small changes
        cmpOp.fuzz(8.0);

        //The special "-metric" setting of 'AE' (short for "Absolute Error" count), will report (to standard error),
        //a count of the actual number of pixels that were masked, at the current fuzz factor.
        cmpOp.metric("AE");

        // Add the expected image
        cmpOp.addImage(expected);

        // Add the actual image
        cmpOp.addImage(actual);

        // This stores the difference
        cmpOp.addImage(difference);

        try {
            //Do the compare
            System.out.println("Comparison Started!");
            compare.run(cmpOp);
            System.out.println("Comparison Finished!");
        }
        catch (Exception ex) {
            System.out.print(ex);
            System.out.println("Comparison Failed!");
            //Put the difference image to the global differences folder
            Files.copy(folderUtil.differenceImageFile, folderUtil.differenceFileForParent);
            throw ex;
        }
    }

    //Compare Operation
    public void doComparison(Screenshot elementScreenShot, FolderUtil folderUtil) throws Exception {
        //Did we capture baseline image before?
        if (folderUtil.baselineImageFile.exists()) {
            //Compare screenshot with baseline
            System.out.println("Comparison method will be called!\n");

            System.out.println("Baseline: " + folderUtil.baselineScreenShotPath + "\n" +
                "Actual: " + folderUtil.actualScreenShotPath + "\n" +
                "Diff: " + folderUtil.differenceScreenShotPath);

            //Try to use IM4Java for comparison
            compareImagesWithImageMagick(folderUtil.baselineScreenShotPath, folderUtil.actualScreenShotPath,
                folderUtil.differenceScreenShotPath, folderUtil);
        } else {
            System.out.println("BaselineScreenshot is not exist! We put it into test screenshot folder.\n");
            //Put the screenshot to the specified folder
            ImageIO.write(elementScreenShot.getImage(), "PNG", folderUtil.baselineImageFile);
        }
    }
}

JSUtil

package utils;

import java.util.function.Function;
import org.openqa.selenium.JavascriptExecutor;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.support.ui.ExpectedCondition;
import org.openqa.selenium.support.ui.WebDriverWait;

public class JSUtil {

    public WebDriverWait wait;
    public WebDriver     driver;
    public JavascriptExecutor js;

    public JSUtil(WebDriverWait wait, WebDriver driver) {
        this.wait = wait;
        this.driver = driver;
        this.js = (JavascriptExecutor) driver;
    }

    public void waitJS() {
        //Wait for Javascript to load
        ExpectedCondition<Boolean> jsLoad = driver -> ((JavascriptExecutor) driver)
            .executeScript("return document.readyState").toString().equals("complete");

        if (!(Boolean) js.executeScript("return (typeof jQuery != \"undefined\")")) {
            js.executeScript(
                "var headID = document.getElementsByTagName('head')[0];" +
                    "var newScript = document.createElement('script');" +
                    "newScript.type = 'text/javascript';" +
                    "newScript.src = 'https://ajax.googleapis.com/ajax/libs/jquery/3.3.1/jquery.min.js';" +
                    "headID.appendChild(newScript);");
            WebDriverWait waitJQ = new WebDriverWait(driver, 30);
            Function<WebDriver, Boolean> jQueryAvailable = WebDriver -> (
                (Boolean) js.executeScript("return (typeof jQuery != \"undefined\")")
            );
            waitJQ.until(jQueryAvailable);
        }

        //JQuery Wait
        ExpectedCondition<Boolean> jQueryLoad = driver -> ((Long) ((JavascriptExecutor) driver).executeScript("return jQuery.active") == 0);

        wait.until(jsLoad);
        wait.until(jQueryLoad);
    }

    public void hideDynamicContent() {
        //Hide dynamic elements inside search box
        js.executeScript("document.querySelector('div[class=\\'SDkEP\\']').style.display='none'");
    }
}

ScreenshotUtil

package utils;

import org.openqa.selenium.WebDriver;
import ru.yandex.qatools.ashot.AShot;
import ru.yandex.qatools.ashot.Screenshot;

public class ScreenshotUtil {
    //Take Screenshot with AShot
    public Screenshot takeScreenshot(WebDriver driver) {
        Screenshot screenshot = new AShot().takeScreenshot(driver);
        //Print element size
        String size = "Height: " + screenshot.getImage().getHeight() + "\n" + "Width: " + screenshot.getImage().getWidth() + "\n";
        System.out.print("Size: " + size);
        return screenshot;
    }
}

TestNG.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">

<suite name="Visual Test Automation Test Suite">
    <test name="RunTest">
        <classes>
            <class name="tests.VisualTest" />
        </classes>
    </test>
</suite>

GitHub Page

https://github.com/swtestacademy/VisualAutomationImageMagickSelenium

Test Results:

 First Run

Second Run (Positive Case)

Third Run (Negative Case)

 I also want to show you the fail case. I open base with a photo editor and add some shapes and save it. After that, I rerun the test and I got the below results. The test failed and also it highlighted the unmatched parts.

Before starting the test I did the below modifications on the baseline image.

Then, I run the test and got the below results:

YouTube Demo Video

For masking images with ImageMagick, I suggest you check the below article.
http://techblog.hotwire.com/2016/05/19/image-comparison-in-automated-testing/

I hope you like this article. Thanks for reading and please share your feedback with us. :)

Thanks,
Onur Baskirt

Onur Baskirt

Onur Baskirt is a Software Engineering Leader with international experience in world-class companies. Now, he is a Software Engineering Lead at Emirates Airlines in Dubai.