How to run JUnit5 tests in order

vibssingh

May 20

The general practices say that automated tests should be able to run independently and with no specific order as well as the result of the test should not depend on the results of previous tests. But there are situations where a specific order of test execution can be justified, especially in integration or end to end tests. The test methods don't follow a specific order by default to execute the tests. The test cases need not necessarily execute in the order in which they have been written.

There are different ways or modes to set the order of execution for the test cases. This article shows how to control the JUnit 5 test execution order via the following MethodOrderer classes:

DisplayName - sorts test methods alphanumerically based on their display names
MethodName - sorts test methods alphanumerically based on their names and formal parameter lists
Alphanumeric - sorts test methods alphanumerically based on their names and formal parameter lists. This is deprecated since JUnit Version 5.7 onwards
OrderAnnotation - sorts test methods numerically based on values specified via the @Order annotation
Random - orders test methods pseudo-randomly and supports configuration of a custom seed
Custom Order - Custom ordering sequence can be implemented by the interface MethodOrderer and providing it as the argument to @TestMethodOrder.

Let us create JUnit5 Tests and execute them.

 public class OrderRandomDemo {      @Test     void test_Add() {          System.out.println("test_Add()");         assertEquals(10, 3 + 7);     }      @Test     void test_Subtract() {          System.out.println("test_Subtract()");         assertEquals(10, 14 - 4);     }      @Test     void test_Multiply() {          System.out.println("test_Multiply()");         assertEquals(10, 5 * 2);     }      @Test     void test_Divide() {          System.out.println("test_Divide()");         assertEquals(10, 30 / 3);     }      @Test     void test_IsEven() {          System.out.println("test_IsEven()");         assertEquals(0, 10%2);     }  }

The output of the above program

1. DisplayName

It sorts test methods alphanumerically based on their display names. Test Method can be anything annotated with @Test, @RepeatedTest, @ParameterizedTest, @TestFactory, or @TestTemplate.

 @TestMethodOrder(MethodOrderer.DisplayName.class)

@TestMethodOrder is a type-level annotation that is used to configure a MethodOrderer for the test methods of the annotated test class or test interface.

MethodOrderer defines the API for ordering the test methods in a given test class.

Test Method - It is any method annotated with @Test, @RepeatedTest, @ParameterizedTest, @TestFactory, or @TestTemplate.

DisplayName.class - MethodOrderer that sorts methods alphanumerically based on their names using String.compareTo(String).
If two methods have the same name, String representations of their formal parameter lists will be used as a fallback for comparing the methods.

An example of sorting the tests based on their display names.

 import org.junit.jupiter.api.DisplayName; import org.junit.jupiter.api.MethodOrderer; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.TestMethodOrder; import static org.junit.jupiter.api.Assertions.assertEquals;  @TestMethodOrder(MethodOrderer.DisplayName.class) public class DisplayNameOrderedTests {      @DisplayName("C")     @Test     void test_Add() {          System.out.println("test_Add()");         assertEquals(10, 3 + 7);     }       @DisplayName("E")     @Test     void test_Multiply() {          System.out.println("test_Multiply()");         assertEquals(10, 5 * 2);     }      @DisplayName("A")     @Test     void test_Divide() {          System.out.println("test_Divide()");         assertEquals(10, 30 / 3);     }       @DisplayName("D")     @Test     void test_Subtract() {          System.out.println("test_Subtract()");         assertEquals(10, 18 - 8);     }      @DisplayName("B")     @Test     void test_IsEven() {          System.out.println("test_IsEven()");         assertEquals(0, 18%2);     } }

We can see that the test methods are sorted alphanumerically based on their display name starting from A to E. The output of the above program

2. MethodName

This annotation sorts methods alphanumerically based on their names using String.compareTo(String).
If two methods have the same name, String representations of their formal parameter lists will be used as a fallback for comparing the methods.

 @TestMethodOrder(MethodOrderer.MethodName.class)

Let us see an example of MethodName.

 import org.junit.jupiter.api.MethodOrderer; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.TestMethodOrder; import static org.junit.jupiter.api.Assertions.assertEquals;  @TestMethodOrder(MethodOrderer.MethodName.class) class MethodNameOrderedTests {      @Test     void testE() {          System.out.println("testE");         assertEquals(10, 3 + 7);     }      @Test     void testA() {          System.out.println("testA");         assertEquals(10, 14 - 4);     }      @Test     void testC() {         System.out.println("testC");         assertEquals(10, 5 * 2);     }      @Test     void testB() {         System.out.println("testB");         assertEquals(10, 30 / 3);     }      @Test     void testD() {         System.out.println("testD");         assertEquals(10, 10 + 0);     }  }

The output of the above program

3. OrderAnnotation

This sorts test methods numerically based on values specified via the @Order annotation.
Any methods that are assigned the same order value will be sorted arbitrarily adjacent to each other.
When any method is not annotated with @Order, it will be assigned the default order value which will effectively cause them to appear at the end of the sorted list.

 @TestMethodOrder(MethodOrderer.OrderAnnotation.class)

Let us see an example of OrderAnnotation .

 import org.junit.jupiter.api.MethodOrderer; import org.junit.jupiter.api.Order; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.TestMethodOrder; import static org.junit.jupiter.api.Assertions.assertEquals;  @TestMethodOrder(MethodOrderer.OrderAnnotation.class) public class OrderAnnotationDemo {      @Test     @Order(3)     void test_Add() {          System.out.println("test_Add()");         assertEquals(10, 3 + 7);     }      @Test     @Order(4)     void test_IsOdd() {         System.out.println("test_IsOdd()");         assertEquals(1, 11%2);     }      @Test     void test_Subtract() {         System.out.println("test_Subtract()");         assertEquals(10, 14 - 4);     }      @Test     @Order(1)     void test_Multiply() {         System.out.println("test_Multiply()");         assertEquals(10, 5 * 2);     }      @Test     @Order(4)     void test_Divide() {         System.out.println("test_Divide()");         assertEquals(10, 30 / 3);     }      @Test     @Order(2)     void test_IsEven() {         System.out.println("test_IsEven()");         assertEquals(0, 10%2);     } }

Here, test_Subtract() is not assigned any order value, so it is displayed as last one in the last.

4. Random

This sorts test methods pseudo-randomnly.

 @TestMethodOrder(MethodOrderer.Random.class)

Let us create a program to show random order of tests in JUnit5.

 import org.junit.jupiter.api.MethodOrderer; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.TestMethodOrder; import static org.junit.jupiter.api.Assertions.assertEquals;  @TestMethodOrder(MethodOrderer.Random.class) public class OrderRandomDemo {      @Test     void test_Add() {          System.out.println("test_Add()");         assertEquals(10, 3 + 7);     }      @Test     void test_Subtract() {          System.out.println("test_Subtract()");         assertEquals(10, 14 - 4);     }      @Test     void test_Multiply() {          System.out.println("test_Multiply()");         assertEquals(10, 5 * 2);     }      @Test     void test_Divide() {          System.out.println("test_Divide()");         assertEquals(10, 30 / 3);     }      @Test     void test_IsEven() {          System.out.println("test_IsEven()");         assertEquals(0, 10%2);     } }

The output of the above program

5. Custom Order

We can define our own custom ordering sequence by implementing the interface MethodOrderer and providing it as the argument to @TestMethodOrder.

Here, the tests are arranged in descending method order.

 import org.junit.jupiter.api.MethodDescriptor; import org.junit.jupiter.api.MethodOrderer; import org.junit.jupiter.api.MethodOrdererContext;  public class DescendingMethodOrder implements MethodOrderer {      @Override     public void orderMethods(MethodOrdererContext context) {         context.getMethodDescriptors().sort((MethodDescriptor m1, MethodDescriptor m2) ->                 m2.getMethod().getName().compareTo(m1.getMethod().getName()));     }      }

Now, test the above custom order.

 import org.junit.jupiter.api.Test; import org.junit.jupiter.api.TestMethodOrder; import static org.junit.jupiter.api.Assertions.assertEquals;  @TestMethodOrder(DescendingMethodOrder.class) public class CustomOrderTests {      @Test     void test_Add() {         System.out.println("test_Add()");         assertEquals(10 , 4 + 6);     }      @Test     void test_Subtract() {         System.out.println("test_Subtract()");         assertEquals(10 , 17 - 7);     }      @Test     void test_Multiply() {         System.out.println("test_Multiply()");         assertEquals(10 , 5 * 2);     }      @Test     void test_Divide() {         System.out.println("test_Divide()");         assertEquals(10 , 20/2);     }      @Test     void test_IsEven() {         System.out.println("test_IsEven()");         assertEquals(0 , 20%2);     } }

Notice the test output. The tests are executed with descending order. The result of the above program is

Test Classes Ordering

ClassName: sorts test classes alphanumerically based on their fully qualified class names.
DisplayName: sorts test classes alphanumerically based on their display names (see display name generation precedence rules).
OrderAnnotation: sorts test classes numerically based on values specified via the @Order annotation.
Random: orders test classes pseudo-randomly and supports configuration of a custom seed.

The configured ClassOrderer will be applied to all top-level test classes (including static nested test classes) and @Nested test classes.

1. ClassName

The Test Classes are sorted alphanumerically based on their fully qualified class names.

 package JUnit5;  import org.junit.jupiter.api.ClassOrderer; import org.junit.jupiter.api.Nested; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.TestClassOrder;  import static org.junit.jupiter.api.Assertions.assertEquals;  @TestClassOrder(ClassOrderer.ClassName.class) public class ClassNameOrderTests {      @Nested     class Addition {          @Test         void test_Add() {             System.out.println("test_Add()");             assertEquals(10, 3 + 7);         }     }      @Nested     class IsEven {          @Test         void test_IsEven() {             System.out.println("test_IsEven()");             assertEquals(0, 10 % 2);         }     }      @Nested     class Subtraction {          @Test         void test_Subtract() {             System.out.println("test_Subtract()");             assertEquals(9, 14 - 5);         }     }      @Nested     class Multiply {          @Test         void test_Multiply() {             System.out.println("test_Multiply()");             assertEquals(10, 5 * 2);         }     } }

The result of the above program is

2. DisplayName

It sorts test classes alphanumerically based on their display names.

 package JUnit5;  import org.junit.jupiter.api.*;  import static org.junit.jupiter.api.Assertions.assertEquals;  @TestClassOrder(ClassOrderer.DisplayName.class) public class ClassDisplayNameTests {      @Nested     @DisplayName("B")     class AppFlowTests {          @Test         void test_Add() {             System.out.println("test_Add()");             assertEquals(10, 6 + 4);         }     }      @Nested     @DisplayName("C")     class TearDownTests {          @Test         void test_Subtract() {             System.out.println("test_Subtract()");             assertEquals(10, 15 - 5);          }     }      @Nested     @DisplayName("A")     class SetupTests {          @Test         void test_Multiply() {             System.out.println("test_Multiply()");             assertEquals(10, 5 * 2);         }     } }

The result of the above program is

3. OrderAnnotation in Class

The test classes are sorted numerically based on values specified via the @Order annotation.

 import org.junit.jupiter.api.*;  import static org.junit.jupiter.api.Assertions.assertEquals;  @TestClassOrder(ClassOrderer.ClassName.class) //sorts test classes alphanumerically based on their fully qualified class names. public class ClassOrderedTests {      @Nested     @Order(2)     class AppFlowTests {          @Test         void test_Add() {             System.out.println("test_Add()");             assertEquals(10, 3 + 7);         }     }      @Nested     @Order(3)     class TearDownTests {          @Test         void test_Subtract() {             System.out.println("test_Subtract()");             assertEquals(9, 14 - 5);         }     }      @Nested     @Order(1)     class SetupTests {          @Test         void test_Multiply() {             System.out.println("test_Multiply()");             assertEquals(10, 5 * 2);         }     } }

The result of the above program is

4. Random

The test classes are sorted pseudo-randomly and supports configuration of a custom seed.

 import org.junit.jupiter.api.*; import static org.junit.jupiter.api.Assertions.assertEquals;  @TestClassOrder(ClassOrderer.Random.class) public class ClassRandomTests {      @Nested     class Addition {          @Test         void test_Add() {             System.out.println("test_Add()");             assertEquals(10, 3 + 7);         }     }      @Nested     class IsEven {          @Test         void test_IsEven() {             System.out.println("test_IsEven()");             assertEquals(0, 10 % 2);         }     }      @Nested     class Subtraction {          @Test         void test_Subtract() {             System.out.println("test_Subtract()");             assertEquals(9, 14 - 5);         }     }      @Nested     class Multiply {          @Test         void test_Multiply() {             System.out.println("test_Multiply()");             assertEquals(10, 5 * 2);         }     } }

The result of the above program is