Visual GUI Testing

Visual GUI Testing

Visual GUI Testing is a technique that is academically defined as “a tool-driven test technique where image recognition is used to interact with, and assert, a system’s behavior through its pictorial GUI as it is shown to the user in user-emulated, automated, system or acceptance tests” [1]. The origin for the technique was established in the early 90s but due to hardware limitations, and immature image recognition algorithms, it has not become applicable or feasible in practice until recent years.

There are many VGT tools, all with different individual features, but that all share that image recognition is used to both interact with and assert the system under test. These tools include, EyeAutomate, Sikuli, EggPlant, Unified functional testing, and more. Research has shown that there are no significant differences between the tools but that the technique is both applicable and feasible in industrial practice [1].

Test automation has become a key practice in software industry to improve quality and reduce lead times to meet the markets demands for continuous improvement and delivery. However, whilst automation support is abundant for lower levels of system abstraction, also in practice, automation support for graphical user interface (GUI) testing has been, and in many companies still is, not covered by automated tests. This lack of automation has forced companies to resort to costly, tedious and error-prone manual test practices that create a barrier to meet the markets demands. Image recognition, which makes VGT unique, allows the technique’s tools to automate test cases that previously had to be conducted manually with equivalent input and output to a human user. In addition, this capability makes the technique independent of system implementation. As such, VGT tools can test any GUI driven system, regardless of implementation language, architecture or even platform. Research into VGT has also shown that the VGT technique:
  1. Improve defect finding ability over manual regression testing.
  2. Provides positive return on investment within one iteration of the tested product, i.e. lowering test related cost.
  3. Improves developer feedback of the quality of the system through more frequent testing, which also instils trust in the quality of the system.
  4. Is easy to use, also by non-technical personnel, because the scripts operate with the same actions and with the same components as a human user.
  5. Can be combined with manual exploratory practices in order to identify unfrequent and non-deterministic defects that are unfeasible to identify with purely manual practices.
In summary, Visual GUI Testing is a technique that gives the user the ability to transition from manual testing into automated testing, lower lead times and create a culture of continuous software integration, development and delivery.


[1] E. Alégroth, “Visual GUI Testing: Automating High-level Software Testing in Industrial Practice“, PhD Thesis, Chalmers University of Technology, 2015.

[2] E. Börjesson, R. Feldt, “Automated System Testing using Visual GUI Testing Tools: A Comparative Study in Industry“, Proceedings of the 5th International Conference on Software Testing Verification and Validation (ICST’2012), Montreal, Canada, April 17-21, 2012 pp. 350-359.

[3] E. Alégroth, R. Feldt, H. H. Olsson, “Transitioning Manual System Test Suites to Automated Testing: An Industrial Case Study“, Proceedings of the 6th International Conference on Software Testing Verification and Validation (ICST’2013), Luxenbourg, March 18-22, 2013.

[4] E. Alégroth, R. Feldt, L. Ryrholm, “Visual GUI Testing in Practice: Challenges, Problems and Limitations“, Published in the Empirical Software Engineering Journal, 2014.

[5] E. Alégroth, R. Feldt, P. Kolstrom, “Maintenance of Automated Test Suites in Industry: An Empirical study on Visual GUI Testing“, In submission.

[6] E. Alégroth, R. Feldt, “On the Long-term Use of Visual GUI Testing in Industrial Practice: A Case Study“, In submission.

[7] E. Alégroth, G. Zebao, R. Oliviera, A. Memon, “Conceptualization and Evaluation of Component-based Testing Unified with Visual GUI Testing: An Empirical Study“, Proceedings of the 8th International Conference on Software Testing Verification and Validation (ICST’2015), Graz, Austria, April 13-17, 2015.

[8] E. Alégroth, J. Gustafsson, H. Ivarsson, R. Feldt, “Replicating Rare Software Failures with Visual GUI Testing: An Industrial Success Story“, Accepted for publication in the Journal of IEEE Software, 2015.

[9] E. Alégroth, M. Nass, H. H. Olsson, “JAutomate: a Tool for System and Acceptance-test Automation“, Proceedings of the 6th International Conference on Software Testing, Verication and Validation (ICST’2013), Luxenbourg, March 18-22, 2013.

[10] E. Alégroth, “Random Visual GUI Testing: Proof of Concept“, Proceedings of the 23rd International Conference on Software Engineering & Knowledge Engineering (SEKE’2013), Boston, Massachusetts, USA, June 27-29, 2013.

[11] E. Alégroth and R.Feldt, “Industrial Application of Visual GUI Testing: Lessons Learned” Chapter of the book Continuous Software Engineering published by Springer, 2014.

[12] R. Oliviera, E. Alégroth, G. Zebao, A. Memon, “Definition and Evaluation of Mutation Operators for GUI-level Mutation Analysis“, Proceedings of the 10th Mutation Workshop (Mutation’2015), Graz, Austria, April 13, 2015