There’s a profound necessity to systematically test complex web applications to guarantee code quality, data integrity and consistency in results.
But before we start writing code and all that fun stuff, we need to ask ourselves some questions first. “What type of testing technique will be best suited?” Sounds like a good one to start with.
E2E Testing
End-to-end testing aims to validate an entire application’s workflow from start to finish.
It is very important because it tests the user’s experience by reproducing real scenarios to ensure integrations work as expected and data is precise. These are all key characteristics to test on a web application, so E2E seems like a good technique to employ in these conditions.
Aside from the obviously increased conviction on the application’s functioning, another advantage of E2E testing that I wish to explore further is the expansion in code coverage that it provides.
Code Coverage
What is code coverage? Glad you asked! 😉
Well, according to Microsoft [1], code coverage is simply a measure of how much code is exercised by existent tests. A high coverage percentage doesn’t automatically imply your code will behave as expected, but the likelihood of it containing bugs, compared to a low percentage, is significantly lower. Usually, a good percentage to aim for is 80% coverage.
Now we know what E2E testing is and why coverage is important, but how do we accomplish all this?
Well, firstly we have to establish what would success look like for an E2E project. Generally speaking, it would have to:
- Be light-weight;
- Have fast execution;
- Be easy to use by inexperienced devs/non-devs;
- Be Reliable in testing and monitoring;
- Have test coverage.
These seem ambitious goals, but believe it or not, there’s an automation framework that checks all these boxes, and even more, let’s dive a bit into it.
Playwright is an open-source Node.js library used for browser automation that was released by Microsoft on February 1, 2020. Curiously enough, some of the team behind Playwright came from Google’s Puppeteer, which explains why they’re very similar.
You can use third-party test runners with Playwright, and there’s even documentation on how to integrate it. However, there’s already a built-in one – Playwright Test – which for improved integration, support, and ease of use, should be the wisest choice.
Besides the usual test runner features like running tests in parallel, context isolation, and capturing test artifacts like videos and screenshots, Playwright Test also offers cross-browser support, that is, running tests across the three major browser engines:
- Chromium;
- Firefox;
- WebKit.
Setting up Playwright is very easy and quick, and creating tests is incredibly intuitive, which makes it ideal for inexperienced users. Another great thing for these types of users is the ability to generate test code by manually performing actions in the browser. This allows a user to make the desired interactions to test on the real web application and have those actions mapped into code that can be copied to formal test files, or sent to more experienced testers to translate it properly.
At this point you might be thinking: “Wow, Playwright is awesome, it offers everything I need to make successful E2E testing projects! If only it also had support for coverage…”. Well, I got some news for you. 😎
Playwright offers out-of-the-box coverage for both the JS and CSS used by the file being tested.
Even though useful, you still have to manually start and stop the coverage on your test code, which if you want to apply coverage to every file might be too laborious.
Fear not, for there is a solution!
Actually, there are two, but let’s dive into them before I spoil anything else…
mxschmitt/playwright-test-coverage
Looking through the open issues on the Playwright’s GitHub we actually find a library that allows you to use Istanbul coverage with Playwright Test to create reports that will be stored in the filesystem.
What is Istanbul you ask? Well, according to Wikipedia, “Istanbul, formerly known as Constantinople, is the largest city in Turkey, serving as the country’s….” Oh, you meant in the context of coverage? Why didn’t you say so?
Ok, Istanbul is a tool that instruments your code so that you can track how well your tests exercise your codebase. In other words, you can track your test coverage.
This library seems decent but it requires us to set up and use babel-plugin-istanbul, which is the part responsible for instrumenting the code. This adds another layer of complexity and is something we have to take into consideration for maintenance purposes.
Moreover, there are some concerns raised by users, namely the inability of merging coverage from Istanbul with the v8 coverage, which is generated by the out-of-the-box coverage of Playwright (open issue).
Another reported issue was that the reports were incorrectly generated or not at all (open issue).
All and all, it doesn’t seem like a solid enough solution to employ in a project that is looking for stability, which brings us to the second solution. drum roll, please…
Proposed by a user on the previous solution, this library solves all the exposed issues.
This solution wraps around Playwright’s Coverage functions, meaning it uses v8 coverage without any need for third-party instrumentation.
This is great because it’s way easier to set up and use, you only need to replace Playwright Test’s functions with the ones from this library, which are basically the same ones with added logic for coverage, and use them normally.
Upon completion of all tests using the wrapped functions, all the generated coverage files are merged into a single one which is then converted from the v8 coverage format to the one used by Istanbul.
This works even if you’re using a different test runner other than Playwright’s!
Conclusion
It seems we have all the ingredients to set up an E2E testing project that covers all the goals we had in mind and having successfully built 3 different proof-of-concept for 3 different front-end projects on Xriba, the results sure are promising.
There’s still room to grow as the technology is fairly new, but the simplicity and robustness of this solution give a lot of confidence whilst testing complex web applications, which is the main goal after all.
References
[1] Brader, L., & Hilliker, H., Wills, A. C. (2012). “Testing for Continuous Delivery with Visual Studio 2012“ [2] Guo, D. (April 27, 2020). “Playwright vs. Puppeteer: Which should you choose?” https://blog.logrocket.com/playwright-vs-puppeteer/