Community

Blog Events Webinars Tutorials Forum
  1. Blog
  2. Events
  3. Webinars
  4. Tutorials
  5. Forum
Create Account
×
Community Blog Improving Test Efficiency - Alibaba DevOps Practice Guide Part 13

Improving Test Efficiency - Alibaba DevOps Practice Guide Part 13

Part 13 of this 27-part series discusses ways to improve test efficiency and create a smoother continuous delivery process.

This article is from Alibaba DevOps Practice Guide written by Alibaba Cloud Yunxiao Team

We will inevitably face expanding services, increasing application complexity, and more difficulty with sustainable testing in the business development process. On the one hand, as the number of test cases is constantly increasing, one CI verification activity takes more than ten minutes. The maintenance cost of test cases is growing, causing the development efficiency to decrease. On the other hand, we take efforts to write many automated test cases, hoping to improve the input-output ratio and test effectiveness.

Increase Testing Speed

Distributed Testing

The core idea of distributed testing is to concurrently execute test cases by increasing computing resources. Then, parse and merge the structured test results generated after execution. This method can speed up the execution of a single test.

The entire test execution process can be divided into the following three phases:

  1. Parse and distribute test cases
  2. Execute grouped cases
  3. Merge grouped test results

1

Let's use a project from the Alibaba Cloud Core Team as an example. The project has more than 10,000 unit test cases. Before the distributed testing solution was adopted, a CI verification task took more than four hours, which extended the time to discover and fix the defects and affected the daily iteration speed. Later, the team adopted the distributed testing method. Thus, the average execution time was reduced to less than half an hour.

In distributed testing, execution resources are stacked in exchange for faster execution. Theoretically, we split each test case and execute it in one container to get faster feedback speed. However, not all scenarios are suitable for distributed testing. For example, if dependencies exist between test cases, these test cases cannot be simply distributed to different execution groups.

Precision Testing

Distributed testing largely improves the test execution speed. However, if all test cases are executed without distinction under any circumstances, the following problems may occur:

  • Computing resources are wasted.
  • A large number of invalid executions are introduced.
  • The stability problems of test cases result in wasted hours troubleshooting.

In the process of finding a solution to improve test effectiveness, we introduced the precision testing solution.

What Is Precision Testing?

Precision testing refers to the process of associating test cases with business methods, precisely recommending test cases to be executed when the code changes, executing recommended test cases, and giving feedback about execution results. A precise definition of the test scope can bring benefits to efficiency and execution speed.

Basic Elements of Precision Testing

  • Establish the association between test cases (Case) and application code methods (CodeMethod). This association is defined as the baseline.
  • If the code changes, recommend and execute changed methods, associated test cases, and changed test cases based on associations between test cases and code methods in the baseline.

Establishing a Test Baseline

A baseline can be established in multiple ways. Alibaba has used either of the following methods to associate test cases with code:

  • Insert trace calls through bytecode injection and pass the signatures of test cases and business methods through these calls. Obtain call procedures of all test cases and methods by collecting trace logs to associate test cases with methods
  • Use AST parsing

Recommending Test Cases

If the code is changed, the changed test cases and business methods will be parsed out accordingly. Changes to methods are usually addition, deletion, and update.

The code change of test cases is relatively simple, and every new and updated test case will be included in the regression scope.

Changes to the code of the tested application must be considered from the following aspects:

  • Added Method: No test case can be matched. Developers or testers may add new test cases for this type of change. Notifications are required.
  • Updated Method: Two scenarios are involved. First, the updated methods have associated test cases that need to be recommended. Second, the updated methods are not associated with any test case. Users need to be prompted to supplement test cases.
  • Deleted Method: Two scenarios are involved. First, the deleted methods are not associated with any test cases, so no prompt is required. Second, the deleted methods are associated with some test cases. Therefore, these methods may be refactored or renamed. In this case, the associated test cases still need to be regressed if they are not deleted.

2

The precision testing solution was applied by a core cloud product of Alibaba Cloud to test code changes that were iterated for about a week. Previously, more than 3,700 test cases needed to be fully executed in each CI task. Now, each CI task can precisely execute the test cases impacted. The speed has increased by nearly 50%, and the test scope has narrowed to less than 1/6 of the original scope.

4


3

Improve Testing Effectiveness

We hope test cases that are written and run can cover the logic of the code effectively. One of the important starting points is test coverage. The test coverage can be checked to detect problems and facilitate the problem-resolving.

Test Coverage

In this article, test coverage refers to code coverage, which is also called line coverage or branch coverage. In addition, Java is used in all examples involving coverage data collection in this article.

Collecting Full Coverage Data

An application typically has multiple types of automated test sets:

  • Unit testing
  • Manual testing
  • API testing
  • WebUI testing
  • Others

It is necessary to aggregate the results of multiple types of automated tests to accurately reflect the complete coverage of an application. At Alibaba, each test is associated with the corresponding application so the test result can be aggregated.

5

We made the following efforts to collect the test coverage data of all types of automated tests:

Unit Testing

For unit testing, test coverage data is generated on the machine where tests are executed. We will create the unit testing coverage report based on the original code information, compiled class information, original coverage data files generated after test execution, and changed code information.

6

Manual/Automated Testing

We implemented a coverage data collection client and a platform for coverage data collection/report calculation and parsing. Then, we will deploy the client in the application integration environment through the O&M platform and mount a javaagent process when the application starts. When we trigger any type of automated tests on any test platform, the test platform will notify the platform we created to interact with the coverage data collection client to collect and parse the raw data for coverage calculation.

In addition, we will merge the corresponding unit testing coverage data to form a complete coverage report during the process of the key-point release test.

7

Which Values Can Test Coverage Bring to the R&D Process?

  • Analyze the uncovered part of the code, deduce whether the test design is sufficient in the early stage, whether the code that is not covered is a blind spot in the test design, and why it is not considered. The requirement/design is not clear enough, the understanding of test design is wrong, or it is strategically abandoned after engineering methods are applied. Then, supplement the test cases gradually.
  • High code coverage does not indicate high-quality code. On the contrary, low code coverage will lead to low-quality code. It can be used as one of the important tools for self-review.
  • Analyze the coverage data of changed code to ensure sufficient testing of the changes and increase the release success rate and release confidence.

In addition to the aforementioned test coverage, we can use the same technology to observe the code coverage of online businesses. In other words, we can figure out lines of code used by online businesses and the lines that have never been called. This way, we detect the obsolete code in the programs, deduce the improper code design points, remind designers and developers to sort out code logic, and improve code quality.

Incremental Coverage

We can make full use of complete test coverage data. However, it is meaningless to pursue higher test coverage without specific scenarios. Blindly pursuing high test coverage often results in the over-designing of test cases. Alibaba focuses more on incremental coverage to improve the entire test coverage in a healthy and effective way.

What Is Incremental Coverage?

Incremental coverage refers to the test coverage of changed code during a test.

The changed code equals the difference between the code of the branch under test and the code of the target branch. (Usually, the target branch is the branch we finally merge.)

Incremental coverage equals lines of changed and overwritten code/lines of changed code

8

Value of Incremental Coverage

  • Detect whether some code lines are missed prior to the release
  • Improve the test case sets if some code lines are missed
  • Increase the success rate and confidence for change release
  • Improve the overall test adequacy of the application under test by pursuing incremental coverage

Application of Incremental Coverage

In the unit test, incremental coverage data of the unit test will be generated. The incremental coverage data will also be generated when interface automation/UI automation/traffic playback/manual tests are performed in test/pre-release environments. When the incremental coverage data is combined with the continuous delivery pipeline, it can ensure that project quality is moving in a better direction.

9

Alibaba's internal practice usually sets various points in the key stage of the CI/CD pipeline. This method ensures that each developer has performed sufficient self-testing activities before committing code for integration in scenarios involving collaboration between multiple personnel. In the integration phase, the committed code must be fully verified in the integration environment before it is officially released.

Based on the feedback about the incremental coverage, developers and testers can supplement specific test cases to make sure that no code lines are missed in each phase. In the process of continuously improving the test sets, the overall test coverage of the project is improved in a healthy and effective manner.

Online Coverage

We do not usually collect coverage data in the online environment. However, if we collect coverage data in the online environment, we can make the application slimmer and improve efficiency from another perspective.

Generally, multiple replicas are deployed for services provided by online businesses. We collect coverage information from a small number of replicas to reduce risks. After a long collection cycle, the online coverage report will be generated. At this time, the covered code can be considered valid code, while the remaining code that has not been used by traffic for a long time may need to be deleted or restructured. As such, we can simplify the code to reduce maintenance costs.

At Alibaba, when we decide to reconstruct an application that is difficult to maintain or significantly corrupted, we will collect the online coverage data for a period and reconstruct and simplify the code based on the generated report.

Summary

Distributed testing speeds up test execution, precision testing identifies the test scope, and incremental coverage provides favorable guidance for the continuous testing improvements. Online coverage helps us reduce the size of our applications. We can improve testing efficiency and create a smoother continuous delivery process by making full use of these technical means.

DevOps Developers O&M continuous delivery Business Efficiency Testing Technology Distributed Testing Unit Test
0 0 0
Share on

Read previous post:

Code Inspection - Alibaba DevOps Practice Guide Part 12

Read next post:

Test Environment and Routing - Alibaba DevOps Practice Guide Part 14

Alibaba Cloud Community

1,029 posts | 252 followers

You may also like

Comments

Alibaba Cloud Community

1,029 posts | 252 followers

Related Products

More Posts by Alibaba Cloud Community

See All