Overview

We recently helped a customer configure AppDynamics to monitor their business transactions on Oracle OSB. AppDynamics does not have built-in support for Oracle OSB, although it does support Weblogic Application Server. It detected various business transactions out of the box, but one type of transaction in particular proved to be a little tricky.

The OSB was accepting SOAP messages from a proprietary upstream system all on one endpoint. It then inspected the message and called one or more services on the OSB, essentially routing the incoming messages. AppDynamics grouped all these messages as one business transaction because they all arrived at the same endpoint. This was not acceptable as a significant number of distinct business transactions were processed this way. We had to find a way to separate the business transaction using the input data.

Changing the application was not an option so, we solved this by augmenting the application server code to give AppDynamics an efficient way to determine the business transaction name. The rest of this article describes how AppDynamics was used to find a solution, and how we improved the solution using custom byte code injection.

Example Application

An example OSB application which reproduces the design of the actual application is used to illustrate the problem and solution.

Finding the business transactions

It is a bit tricky to find the business transactions for this application because the services on Oracle OSB are all implemented by configuring the same Oracle classes. Each Proxy Service is just another instance of the same class and the call graph in the transaction snapshot is full of generic sounding names like ‘AssignRuntimeStep’

The first step to figuring out how to separate the business transactions is using the AppDynamics Method Invocation Data Collector feature. This gives you a way to inspect the method parameters in the call and printing their values. Method invocation data collectors allows you to configure AppDynamics to capture the value of a parameter for a particular method invocation. Not only the value of the parameter but it is possible to apply a chain of get- methods to the parameter.

The following figure shows data collector configuration to get information out of the parameters passed to the processMessage method on the AssignRuntimeStep class we noticed in the call graph. This data collector tells AppDynamics to capture the first parameter to the processMessage method on the class AssignRuntimeStep and then to collect the result of calling toString() and getClass().getName() on that parameter.

The results of this can be seen in the following images. The first shows the result of the toString() applied to the first parameter

and the second shows the class of the parameter. Notice that the class name is repeated three times. It is a list of values, one value saved for every invocation of the processMessage method.

From the first image it is obvious that the input message is contained in the first parameter. You can also see that the messages is stored in a map like structure and the key is called body. Note that the business transaction name is visible in the first image TranactionName=“Business Transaction1”. The second image shows the type of the first parameter so the message is contained in an object of class MessageContextImpl.

The next step is to tell AppDynamics what to use for splitting the business transactions and this can be done by using a Business Transaction Match Rule. The number of characters from the start of the message to the field we are interested in are roughly 126 and assuming the transaction names will be around 20 characters we can set up a match rule as follows:

Note the number of arguments (2) set in the above image. That value is important and the transactions will not show up at all if the wrong value is used. We determined the value by decompiling the Weblogic class but you can always do it by first trying 1 and then 2.

With the above configuration in place the AppDynamics agent is able to pick up the different transactions. The transaction names aren’t perfect but it works!

Optimise and get the correct name

This solution is OK, but it has a few issues. Firstly the transaction names are either cut off or include characters that are not part of the transaction name. It is also not very efficient, because it requires the entire MessageContextImpl instance to be serialised as a String just to extract a small part of it. To improve this we need to add custom code to the MessageContextImpl class so that we can access the data in a more efficient way.

Consider the following Java code to search a string for the transaction name:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

 private static final SEARCH_TOKEN = "TransactionName=\"";

  private static String getTransactionType(String input) {
      int startIndex = input.indexOf(SEARCH_TOKEN);              
      int endIndex = startIndex;
      
      if (startIndex == -1) return null;                  
      
      startIndex += SEARCH_TOKEN.length();             //Jump to the open quote
      if (startIndex < input.length() -1) {
          endIndex = input.indexOf("\"", startIndex);       //Find the end quote
      }
      
      if (endIndex > startIndex && endIndex < input.length()) {     
          return input.substring(startIndex, endIndex);
      }
      
      return null;
  }

This is a statically accessible piece of code that will extract the transaction name from an arbitrary string. It first tries to find the token in the input string. Once the token is found it determines the open and end quote positions and returns the transaction name. If nothing is found then return null.

The next step is to write some Java code that can use the above code without loading the entire string into memory.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

 public static short WITHIN = 512;
  public static short BUFFER_SIZE = 256;

  public static String getTransactionType(InputStream inputStream) {
      String result = null;
      BufferedReader reader = null;
      try {
          reader = new BufferedReader(new InputStreamReader(inputStream), BUFFER_SIZE);
          int read, total = 0;

          //Read up to BUFFER_SIZE and then stop
          StringBuilder sb = new StringBuilder();
          boolean stop = false;
          do {
              char[] cbuf = new char[BUFFER_SIZE];

              read = reader.read(cbuf, 0, BUFFER_SIZE);
              if(read > -1) {
                  sb.append(cbuf);

                  //Search for the transaction type in the buffer
                  result = getTransactionType(sb.toString());

                  total = total + read;
                  stop = stop || result != null || total >= WITHIN;
              }
          } while (!stop && (read != -1));
          reader.close();

          return result;
      }
      catch (Throwable e) {
          Logger.INSTANCE.trace("Failed: {0}",e, e.getMessage() );
      }
      finally {
      //Omitted clean up code
      }
      return null; //Something went wrong
  }

This method accepts an InputStream and progressively reads it, 256 characters at a time, to find the transaction type. It is limited to search only the first 512 characters as an optimisation based on the known message structure. It will likely always find the transaction type within the first 256 characters, but 512 makes it a certainty. Also note the variables WITHIN and BUFFER_SIZE, which are there to make the code configurable and future proof.

The code listed above can be included in a custom Java agent that will instrument the Weblogic code using a ClassFileTransformer. Creating Java agents and class transformers are out of the scope of this article. It focuses on the bits actually injected. For more on creating a custom Java agent see the java.lang.instrument documentation.

The next step is making the above getTransactionType accessible by the AppDynamics agent.

Using custom byte code injection to expose internals to AppDynamics

Byte code injection can be achieved in different ways, one way is using the ASM library. The basic idea is to inject a method into the MessageContextImpl class that can be accessed by AppDynamics as a getter on the first parameter of the processMessage method of AssignRuntimeStep.

So for the agent to inject the following piece of code into the MessageContextImpl class,

1
2
3
4
5
6
7
8
9

 public String ec_getTransType() {
      try {
          Logger.INSTANCE.debug("getTransactionType called");
          return TransactionTypeExtractor.getTransactionType(getBody().getInputStream(null));
      } catch (Exception e) {
          Logger.INSTANCE.error("Failed to get transactionType", e);
      }
      return null; // Something went wrong
  }

you can use ASM as listed below. It effectively writes the above method into the MessageContextImpl class before it is loaded by the class loader. For more information on how to use ASM see the ASM User Guide.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

     MethodVisitor mv = super.visitMethod(ACC_PUBLIC, "ec_getTransType", "()Ljava/lang/String;", null, null);
      mv.visitCode();
      Label l0 = new Label();
      Label l1 = new Label();
      Label l2 = new Label();
      mv.visitTryCatchBlock(l0, l1, l2, "java/lang/Exception");
      mv.visitLabel(l0);
      mv.visitFieldInsn(GETSTATIC, "au/com/ecetera/javaagent/logging/Logger", "INSTANCE", "Lau/com/ecetera/javaagent/logging/Logger;");
      mv.visitLdcInsn("getTransactionType called");
      mv.visitInsn(ICONST_0);
      mv.visitTypeInsn(ANEWARRAY, "java/lang/Object");
      mv.visitMethodInsn(INVOKEVIRTUAL, "au/com/ecetera/javaagent/logging/Logger", "debug", "(Ljava/lang/String;[Ljava/lang/Object;)V", false);
      Label l3 = new Label();
      mv.visitLabel(l3);
      mv.visitVarInsn(ALOAD, 0);
      mv.visitMethodInsn(INVOKEVIRTUAL, "com/bea/wli/sb/context/MessageContextImpl", "getBody", "()Lcom/bea/wli/sb/sources/Source;", false);
      mv.visitInsn(ACONST_NULL);
      mv.visitMethodInsn(INVOKEINTERFACE, "com/bea/wli/sb/sources/Source", "getInputStream", "(Lcom/bea/wli/sb/sources/TransformOptions;)Ljava/io/InputStream;", true);
      mv.visitMethodInsn(INVOKESTATIC, "au/com/ecetera/javaagent/vha/TransactionTypeExtractor", "getTransactionType", "(Ljava/io/InputStream;)Ljava/lang/String;", false);
      mv.visitLabel(l1);
      mv.visitInsn(ARETURN);
      mv.visitLabel(l2);
      mv.visitFrame(Opcodes.F_SAME1, 0, null, 1, new Object[] {"java/lang/Exception"});
      mv.visitVarInsn(ASTORE, 1);
      Label l4 = new Label();
      mv.visitLabel(l4);
      mv.visitFieldInsn(GETSTATIC, "au/com/ecetera/javaagent/logging/Logger", "INSTANCE", "Lau/com/ecetera/javaagent/logging/Logger;");
      mv.visitLdcInsn("Failed to get transactionType");
      mv.visitVarInsn(ALOAD, 1);
      mv.visitInsn(ICONST_0);
      mv.visitTypeInsn(ANEWARRAY, "java/lang/Object");
      mv.visitMethodInsn(INVOKEVIRTUAL, "au/com/ecetera/javaagent/logging/Logger", "error", "(Ljava/lang/String;Ljava/lang/Throwable;[Ljava/lang/Object;)V", false);
      Label l5 = new Label();
      mv.visitLabel(l5);
      mv.visitInsn(ACONST_NULL);
      mv.visitInsn(ARETURN);
      Label l6 = new Label();
      mv.visitLabel(l6);
      mv.visitLocalVariable("this", "Lcom/bea/wli/sb/context/MessageContextImpl;", null, l0, l6, 0);
      mv.visitLocalVariable("e", "Ljava/lang/Exception;", null, l4, l5, 1);
      mv.visitMaxs(4, 2);
      mv.visitEnd();

Change AppDynamics configuration

Now the AppDynamics agent configuration can be updated to use the new ec_getTransType method.

The resulting business transaction names now looks much better.

Conclusion

With AppDynamics it is possible to get really useful information out of a running application. It has very flexible configuration, which allows you to really dive deep into the application internals to find issues and separate transactions. However, sometimes it is better to give AppDynamics a hook into the internal information so that it can work more efficiently. When you have access to the application code then this can easily be achieved by adding some code. When it is not practical to rebuild the entire application then you can always use byte code injection.