Overview of the Test Application
To test the performance of the various flavors of Binding Frameworks and SOAP Engines, a simple standardized test application was first designed. This application consisted of a Web Service that took an array of objects as a parameter and returned the same to the client.

To ensure a fair basis for a comparative study, an XML Schema representing a collection of "Employee" objects was first defined. Each "Employee" object contained nine variables, all of which were of simple types (Figure 1). This schema was used to follow a schema-driven development process for all five test cases explored by this article.

To ensure that our performance figures aren't biased by external environmental factors such as network latency, the clients invoked their corresponding Web Service from the same machine hosting the Web Services.

To quantitatively measure the performance of both the Binding Framework and the SOAP engine, the system time with millisecond accuracy was recorded at the following critical points along the request-response flow:

• When the client invokes the service
• When the SOAP Engine servlet gets a SOAP message from the client
• When the Web Service's business logic begins executing
• When the Web Service's business logic completes execution
• When the SOAP Engine servlet replies to the client with a SOAP message
• When the final object returned is available to the client

  From the data above, we were able to calculate the following statistics:
  

• Total Round-trip Time: The time elapsed between the client invoking the service and the client getting the final object returned.
• Total Marshalling Time: The time elapsed between the business logic completing execution and the SOAP Engine servlet sending the reply message to the client.
• Total Un-marshalling Time: The time elapsed between the SOAP Engine servlet receiving the message from the client and the business logic starting execution.

Benchmarking Procedures
Each test case was put through a two-phased standard procedure to quantitatively measure its performance and give us a fair basis for a comparative analysis. The procedure was designed to measure the performance of each case with different object sizes ranging from a collection with 1,000 records to one with 5,000 records. The objects were created by the client prior to the service invocation and were populated with randomly generated data to avoid the possibility of any pattern-based optimizations.

In the first phase, a single client was used to invoke the Web Service and the time taken to un-marshal the incoming message into objects, marshal the objects into a message, and complete a round-trip were determined. To do this, the service was first invoked three times, off-the-record, to warm up the JVM. This was followed by three more runs for which data was recorded. The average of these three runs was recorded as the result. This procedure was carried out for each test case, with five different object sizes (1,000, 2,000, 3,000, 4,000, and 5,000 records).

In the second phase, the load test simulated 10 users simultaneously and repeatedly invoking the Web Service for a period of one minute. The recorded results were then used to generate reports and study the scalability, throughput, and speed of the test cases.

Performance Metrics
Using the data collected during the first phase of testing, we calculated and plotted the average values of the un-marshalling, marshalling and round-trip times corresponding to each of the five test cases. The graphs show how the time taken to perform a given operation varies with respect to the size of the object for each test case. Each graph includes data from all five test cases and serves as a powerful visual aid for a comparative study of their performances.

Un-marshalling Performance
As seen in Figure 2, XFire with JibX data binding offers the best un-marshalling performance. It's also observed that Axis 1.4 fared the worst, doing about 12 times worse than XFire with JibX, on average.

It's interesting to note that XFire with JibX, on average, fared four times better than its nearest competitor, Axis2 with JibX.

Marshalling Performance
From Figure 3, it can be seen that both Axis2-based test cases and XFire with JibX match marshalling performance very closely among themselves for small datasets, however, as the objects increase in size, XFire with JibX seems to gain a marginal lead in marshalling performance.

Axis 1.4 was the worst, doing four times worse than XFire with JibX. XFire with JaxB fared quite badly too - taking about twice as much time as XFire with JibX.

Round-Trip Performance
From Figure 4, you can see that both Axis2 with JibX and XFire with JibX perform almost identically with respect to round-trip time.

Although Axis2+ADB match their performance for small datasets, Figure 5 shows that its performance takes a hit with larger datasets.
Axis2 with JibX is, on average, 6% faster than XFire with JibX with respect to round-trip performance, according to the data collected. Since this result wasn't entirely consistent with the findings regarding marshalling and un-marshalling time, where XFire with JibX consistently gave faster performance, we decided to investigate how and why Axis2 with JibX managed to gain a slight edge over XFire with JibX on the round-trip, keeping in mind that the latter outperformed the former in all other critical performance tests. We hypothesized that XFire's client-side marshalling and un-marshalling might be slightly sluggish when compared to that offered by Axis2's client-side API. To confirm this theory, we created an XMLHTTP client to invoke the corresponding Web Services and measured the time elapsed between sending the request and receiving a reply at the client end. Since this HTML client sent and received the actual SOAP messages, we were able to exclude the processing overhead of marshalling and un-marshalling on the client end.

We consistently saw that XFire with JibX was quicker compared to Axis2 with JibX, proving that our hypothesis explained the anomaly.

Axis 1.4 seems to be the slowest (4.37 times slower than Axis2+JibX) followed by XFire with JaxB (2.43 times slower than Axis2+JibX).

Load Testing Performance
In the second phase of testing, a load test was done on all five test cases. These results give us statistics to help compare their scalability, speed, and throughput.