the road to fully autonomous testing

  • 23 Sep 2021
  • 65
the road to fully autonomous testing
Im often asked if and when fully autonomous testing could become a reality. Thats a topic I love to discuss. But, before delving into that, lets take a closer look at the two words that make up that term. Autonomous , meaning without human intervention, is pretty simple. Testing is more difficult because the investigative, inquisitive nature of testing does not lend itself to automation. What I am about to describe is best categorized as autonomous checking. With that in mind, lets continue. With advanced tooling like vision-based test automation and other intelligent automation engines, the problems of automated checking have shifted from How do I reliably automate this interface to higher-level problems. Humans are still overwhelmingly responsible for creating the automated checks: describing what inputs to fill in, what buttons to click, etc. This is the first horizon. The shift to autonomy is best defined as Describing becomes Deciding. With approaches such as smart impact analysis, this is already the case. You dont need to describe which tests to run; you just need to decide if the tools recommendations suit your needs. This is great in closed systems such as SAP, Salesforce, and ServiceNow (where these offerings shine). With the help of AI, this trend will expand well beyond thisinto the realm of bespoke/custom applications. So great! The future will just be getting a printout of possible activities from the machine and giving it the green light! Well not so fast. You see, these closed systems not only have defined processes; they also have defined outcomes (the oracle). Not so with bespoke applications. While determining the actions to take is possible generically (by examining people taking these actions), its not always possible to extract the Why component. When a user executes a transaction, their eyes flick to the top of the screen to double check that the Amount value is correct. This validation is not captured, and so the automated process misses the point of the check (which was to determine not only that the transaction was processed, but that it was processed correctly) . This is not a bleak outlook, however. While Fully Autonomous checking may still be quite a way off, the trend of Describing becomes Deciding will remove a ton of busywork that bogs down quality engineers today. Parsing through the outputted scenarios, injecting validations, and deciding which to run is a much more pleasant job than worrying about why the Login button doesnt have a stable ID field. With that said, there are a few things to watch out for: Beware of test case spam If you embark on an autonomous testing endeavor and your team comes back with a tool or process that generates thousands of tests beware. You still need to parse through these tests, inject validations, and debug them if they fail. The motto of fewer, targeted tests has been a good guide for the past 20 years, and it remains so now. Investigate the how When you are told that your tests can be automatically generated, dig a bit into how this happens. AI is not magic. If something appears to be magical, it is most likely a fabrication. Your team should be able to tell you that the process examines usage patterns, parses existing (accurate) definitions, or has some other source of how it defines the test. Shaking up the app and generating tests from it is still firmly in the world of magical thinking. Ask about maintenance Having a thousand tests is like having a thousand smoke detectors. If you own an entire high-rise apartment building, thats probably justified. If you own a house, then you will spend two hours switching them all off when you burn the toast. Tests that fail must be investigated, updated or discarded. Inquire about the nature of this method to ascertain whether autonomy will actually save you time in the long run. Despite this, the future of autonomous checking appears to be very bright. The goal as an industry should be to devise a method for generating the bestand fewesttests necessary to achieve the desired level of assurance.