Feature documentation
Common Problems in test automation, and how we address them in Boozang
Here is a break-down of common test automation problems, and some information on how we address these problems in Boozang. The problems have been broken down in categories to make it easier to find.
In each category, there are links to relevant resources, such as blog posts, videos, or forum posts. The idea is to create a simple problem-based index to find how to solve a certain problems. This also allows us to identify blind spots in our feature sets and documentation.
Problem Categories
Element Selectors
Did you ever come across any of the following problems?
- Getting false negatives from unstable element selectors
- Problems of recording tests on applications with dynamic classes or id:s
- Developers had to add specialized HTML attributes for automation
- Problem automating applications with Iframes
Boozang support unique natural-language element selectors, that avoids all of these issues. Recorded test selectors are stable, and when a selector breaks, it's dead simple to repair.
Learn more:
- Element Selectors (article)
- Supporting Iframes (article in progress)
Special automation cases
There are always some hairy situations to automate. , which addresses the following test automation problems:
Did you ever come across any of the following problems?
- Recording of drop-downs resulting in multiple actions and unstable behaviour
- Difficulty of driving drop-downs using data
- Recording of file uploads failing
- Storing large chunks of static data for file uploads
- Drag-and-drop tests breaking on different screen sizes
Boozang intelligent automation supports single-action automation for all of these situations. Customized drop-downs are recorded as a single data-driven action, file upload inputs are captured in a single action that can use file URLs to external sources, drag-and-drop tests are based on elements, not x and y offset.
Read more:
- Custom input boxes / drop-downs (article in progress)
- Drag and drop testing (article in progress)
- File uploads (article in progress)
- GoJS and Canvas-based testing (article)
Timing
Did you ever come across any of the following problems?
- Tests that run stable on your local environment breaks on another due to differing response times
- Breaking test when your system is under load
- Intermittent problems triggered by slow back-end call
Boozang supports custom timeouts and waits. It also "listens" to the requests to the back-end server and automatically wait for them to complete, creating un-surpassed stability.
Learn more:
Logic and loops
Did you ever come across any of the following problems?
- You end up with long, specialized tests with low to no re-use
- Difficulty to build pagination and clean-up tests
- Difficulty to drive test with CSV/spreadsheet data
In Boozang we support all common constructs from programming, allowing for short, re-usable tests. Plug re-usable tests with parameters, just like functions in Javascript. Groups is used to handle loops (for, while) and logic (if, else). Flags can used to handle the classic Goto (compare: switch / case in JS).
Read more:
- Logic and loops using Groups (article)
- Flag / Goto action (forum post)
- Action and group loops over data (forum post)
- Conditional Logic: Yellow or Blue
Data-driven testing
Did you ever come across any of the following problems?
- Difficulty in creating data-driven tests using record
- Challenging to create complex assertions
- No easy way to use data from external sources
To create re-usable test, make sure the tests can be driven by data. We support multiple data-types allowing for full programmatic freedom (Property, Object JSON, Array, File, CSV, Matrix, Mixed JSON, External request).
Read more:
- Boozang Data Management (guest article)
- How to use external CSV data (forum post)
- How to use matrix data (forum post)
- Introduction to data and lists (video)
Sometimes it's necessary to load test data from external files.
- Loading data from external sources (article)
- How to drive Boozang using external acceptance test list (article)
API testing
Did you ever come across any of the following problems?
- You use separate tools for UI end-to-end, API, and load testing
- No way to mix UI and API tests
- Seeding UI tests (setup) taking too long
Boozang does both API testing and UI testing, and supports a mix of the two. You can also generate API tests by recording the browser interactions in the browser. This allows for for fast setup and cleanup for UI tests. It also allows you to start the automation effort early, even before all the application interface is ready.
Read more:
- Advanced API testing (article)
- How to do API testing using Typicode json-server (article)
- API testing / API performance testing (forum post)
Cucumber support
Did you ever come across any of the following problems?
- Siloed QA process, without visibility or buy-in from business users
- Automation effort starting to late in the development process
- Large, disorderly test sets
- No clear root cause analysis to identify and prioritize bugs in your SLC
Boozang has Cucumber support built-in. This allows you to start planning your automation effort early, regardless if you are doing behaviour-driven development (full BDD - define tests first), or simply mapping your tests to features (hybrid BDD - define tests with code) .
Read more:
- Introduction to Cucumber support (article in progress)
- Test Maps in Boozang (article in progress)
- Simple Cucumber example (forum post)
- Example of data-driven test (forum post)
- Notifications (article in progress)
As your test coverage grows, a single bug can impact many Cucumber scenario. Use root-cause analysis to measure the impact of a single bug, and map it back to your own bug tracker.
- Root cause analysis: Getting more out of Cucumber logs (article)
- Root Cause Analysis in Test Automation (guest article)
Integration points (Continuous Integration and VCS)
Did you ever come across any of the following problems?
- Difficulty to map your QA process to a single software, such as GitLab or Jira
- Automation tools are forcing you to use there own management tools, that doesn't integrate to the ones you use
Boozang only does automation and does it well. It is built to integrate to your existing software stack, not re-invent the wheel. It easily integrates to all CI servers, and fully integrates to JIRA XRay, GitLab, Github and Bitbucket for Cucumber features and reporting.
Read more:
- JIRA-XRay Test Automation with Boozang (guest article)
- A better way to manage your environments, all in JIRA (guest article)
- Working with Github (article)
- Jenkins CI config for distributed runs (forum post)
Scripting
Did you ever come across any of the following problems?
- Your current tool is limited when solving advanced test automation problems
- Complex validation cases based on string manipulation and calculations
Boozang has built-in Javascript support allowing you to solve any problem that can be solved by programming. We also support utility functions for random selection, data generation, and a sophisticated data engine to generate data based on regular expressions.
Read more:
- Regular expressions for validation and data generation (forum post)
- Timestamp-based data generators (forum post)
- Random numbers and random selection (forum post)
- Using breakpoints (coming soon)
- Introduction to Javascript and extract data
Debugging tests
Did you ever come across any of the following problems?
- More time spent on troubleshooting and repairing test than creating them
- Hard to find issues as the tool hides functionality behind-the-scenes
Boozang was built with troubleshooting in mind. Use the Boozang console to debug test, inspect variable values in the debug data view, and use breakpoints, just like in software development.
Read more:
Team work
Did you ever come across any of the following problems?
- Only developers participating in the automation effort
- Difficulty to re-produce bugs
Boozang has built-in team collaboration. All tests are represented by shareable URLs which makes test sharing straightforward. Boozang supports team access policy, which a number of pre-defined roles, allowing for read / execute-only type of access.
Read more:
Versioning control: Branching and Merging
Did you ever come across any of the following problems?
- Who broke the test? Big teams stepping over each others?
- Accountability and traceability? Who works on what?
Similar to Git, Boozang support branching and merging of tests. This allows for an automation pipeline similar to your software delivery pipeline. Build feature branches and branch merge into master when "ready". Milestones are supported for easy rollback.
Read more:
- Branching and merging (article)
- Automated backup / milestones (coming soon)
HTML Canvas element testing
Did you ever come across any of the following problems?
- Part of your application has a HTML canvas, making automation impossible
Boozang has a unique way of testing inside the HTML canvas element, both for clicks and validations.
Read more:
Parallel execution
Did you ever come across any of the following problems?
- As you add more features, test execution is getting too long to be practical
- Tools are charging based on the number of parallel processes
- Remote tunnels from your company to Cloud testing services create unreliable tests
- Remote tunnels from your company to Cloud testing service present a security issue
To shorten execution times, it's crucial to support parallelism. Boozang runs in your infrastructure and has a built-in test dispatcher that allows for any number of tests to be run in parallel. The only limit is the resources available at your company.
Read more:
- Distributed testing part 1 (article)
- Distributed testing part 2 (article)
- Boozang parallel execution, an insight into our Jenkins pipeline (guest post)
- Follow up on Gherkin parallel runs poll (guest post)
Advanced: Model-based UI tests
Advanced: Open API modelling
Boozang has a series of tools to import your Open API specification and automatically generate API tests.
