As part of my Computer Science M.S. degree, I tend to write pitiful papers from time to time.  I’m posting this paper online because I want fair and honest critique.  I’m not a strong writer but I believe my ideas may be of some use.  This paper details using dynamically-generated SLA documents to create an expansive test suite over a web service above and beyond what unit-test generation from WSDLs can do.

Use of Dynamic SLA Documents In Expanding The Depth of Web-Service Testing & Validation

Sidney Andrews

Introduction 

In the software development field, the quality assurance of an application can be directly related to the time and effort spent in testing & verification.  As long as a development group has access to source files and code, there is a large amount of flexibility in deciding the level of acceptable risk and defects for the particular project and finding methods to verify the project meets the requirement.

When project groups utilize executables and software libraries from third-parties, their attempts at quality control are severely limited.  Since developers are not able to “prevent the introduction of faults during… development” 1 they cannot be involved in any form of constructive quality assurance.  However, analytical quality assurance is still possible.  Faults in a software library can still be found post development by a development party consuming another group’s application interfaces and libraries.  The depth of the tests and the amount of faults that can be exposed are primarily dependent on the amount of information made available to the testing group.  Reasonably, with more information available to the testers, their testing capabilities will increase.  Items such as source code, algorithms, program logic descriptions, unit tests and many others can help increase the depth of possible tests.

With source available to a development group, the group can use white-box methods to develop and generate tests to ensure that the project meets its stated requirements.  This is possible because the source allows the testers to understand the control flow of the application.  With an understanding of the program’s logic and control flow, the testing methodology qualifies for the distinction of white-box testing.  Armed with this information, we can expand the depth of the test and validate many possible execution scenarios.  Without information on a software project’s code or implementation, the testing group is limited to “black-box robustness testing” 2.  Normally, black-box testing is considered the only method of testing an application where it is not possible to ascertain any knowledge of the internals of the various operations available for use.  In some cases, other information about a software library or service can assist with increasing the depth of a black-box test but it is debatable what level of information can provide a clear distinction between black-box and white-box tests.

Many modern desktop and web applications rely on a service-oriented architecture.  While not an entirely new concept, SOAs have become increasingly popular with the splintering of work between multiple parties.  The world of service development and utilization introduces some unique constraints in testing and validation.   Software services can easily be “accessed without of its underlying implementation” 3.  Testing is usually constrained to verifying the model of the service.  As a service consumer, the only information available to use in testing is the model of the service.  This information is useful however, because the model information distributed to the service consumer is enough to construct an UML diagram of the service application.  This is possible because the consumer of a service application is allowed to download the interface(s) of the externally visible classes that the service uses to respond to client requests.  These interface classes allows a service consumer to generate a class inherited from the mentioned interface to “consume” the service.

The transmission of a service application’s interface to client applications is both platform agnostic and independent of any particular language implementation.  This is due to the use of Web Service Description Language (WSDL) documents.  While the name specifies the use of WSDL files in the world of web services, it is not possible to use WSDL documents with any service-oriented architecture.  What makes the platform and language independence possible is the fact that WSDLs are xml-based files that “describes the public interface of a service” 4.  In the raw xml data, a WSDL file is primarily composed of operators and messages.  The operators are language-independent constructs that describe the methods in an interface.  The messages are the other constructs in the document that describes both the standard and unique types in an interface.  When combined, both are useful enough to create an interface in a modern object-oriented language.  This interface can be easily used to allow a class in your modern language to implement that interface and create the matching method and types for your operators and messages.

WSDLs are very important in the web services realm so that applications can automatically discover and consume web services.  However, WSDL files can only model how an operation works but they do not provide any information about the performance of the service.  This is where Service Level Agreements SLAs come into play.  These documents provided by the authors of the service application guarantee a level of performance to the client applications that consume the service.  In the context of web services, SLA documents guarantee a quantifiable network performance at defined levels to the client 5.  Web services provide a unique challenge in both the creation and validation of SLA documents between the service provider and potential service clients.   With web services, the provider faces the challenge of guaranteeing service performance with no control over third party conditions that directly affect the web service.  Possible conditions could include the network connection, shared hardware resources, traffic congestion on the client backbone, etc.   Also the size, scalability and popularity of a web service can directly affect the performance 6.  Secondly, SLA documents also provide penalties related to violations in its terms or deliverable metrics 7.  The violations and consequences are two-fold and relate to both the service provider and client. 

Methods

Current Methods of Testing

Client applications and users of web services have an explicit need to test and verify the web service.  Business-critical applications may rely on these web services and metrics are necessary when developing client applications to consume the services.  Having metrics and performance data for the web services can help create a baseline when testing and validating the client application.
Most research into the testing of web-based services primarily revolves around model-based testing 8.  Clients have used an abundance of black-box techniques to validate the model of the web service.  With the interface information provided by a WSDL document, a client could write a suite of tests to surface test the operators and messages to validate that they function as expected and per the SLA.  SLAs usually contain clauses with expected exceptions and some data about the constraints of the message types and operator methods.  Since the interface information is provided in a standard XML format, this gives the testers the freedom to write automated test suites or applications.  Many testers use automated software to test the endpoitns of an application and write a test suite that looks for scenarios where the service would crash or exhibit unexpected behavior  9.  

Many researchers use automated unit test generation tools (such as JCrasher and Pex) to isolate and white-box test the class generated from the interface of the web service 10.  Since implementation data and source code is not available, the in-depth testing of the generated web services class only effectively surface tests the endpoints of the web service itself.  In this interesting phenomenon, the web service can only be effectively black-box tested since the knowledge of the internals is severely limited.  This testing method, however, very quickly generates a large number of unit tests to create a suite of tests with great breadth over all possible endpoints to the web service.

Limitations of Black-Box Testing

Surface testing endpoints is an effective way to quickly create a test with a large breadth of search space, but it does have a few disadvantages.  First, without knowledge of the internals and execution paths of the web service, we can only truly test a subset of all possible method and type scenarios.  A simple example would be an operation (method) that takes an integer data type as input.  This method would increment the integer and use an if-else construct to determine an execution path based on the value of the integer.  Surface testing using black-box methods and unit test generation tools would not quite fit the job for this scenario.  An endpoint examination would return that this operation takes an integer type as input and returns some arbitrary type or nothing at all.  From this, a unit-test generation tool would assume to surface test the operation by sending it various and random integer values.  Values such as the minimum and maximum integer values, 0, 1 and -1 would be posted against the operation and the test suite would record the resultant information.  In this scenario, we would almost definitely find an integer overflow situation with the maximum integer value if the web service provider did not account for that in their program logic.  However, a tester is not guaranteed that the values sent in for the integer would test both possible execution branches for the if-else clause.
This is the main utility of using unit-test generation tools.  They test endpoints and make sure that the endpoints behave as specified in edge-case scenarios.  If the message was using a string type as input, the unit-test generation tool would’ve checked the resultant values when the operation is passed in values such as an empty string, null value, special character symbol and foreign character symbols.  Unfortunately, this method of surface testing could not guarantee that it tests every possible execution scenario.  In order to do that, most unit-test generation tools would need access to the source code to account for all possible execution paths.  In the world of web services, this information is simply not available in most cases.

Some other, smaller limitations of using black-box methods to test web services could be involved in physical resources 11.  Buffers and network conditions can falsely change the results of a unit test.  While this type of error could be easily identified and dealt with, it is very important to consider and include in your unit tests to avoid skewing your results.  Also, unit-testing is resource exhaustive locally and can be doubly so over a remote network connection.  A network timeout or SOAP exception can be added to a white list of possible exceptions that will not invalidate a unit-test.  Another possible difficulty in unit-testing web services is the possibility of an update to the web service invalidating your unit tests 12.  Obviously, this can be resolved by automatically updating your WSDL document and unit tests to reflect the new build of the web service.  However, synchronization of your unit tests and the update schedule for the web service is necessary to avoid either updating your WSDL document too much or not enough.

Another issue with the current methods of testing lies with incorporating expected faults detailed through the SLA document into the unit tests.  Since an SLA document is typically hand written, it takes manual labor to translate the SLA’s data into test cases.  SLA information that’s important to the test team could include possible exceptions, performance metrics, and expected behavior.

Dynamic Service Level Agreements

Due to the dynamic and automated nature of unit-test generation, a static SLA document does not fit into this paradigm.  What if the testers could generate unit tests from an SLA document?  The only way to make this possible is to use an SLA document that’s in a format that is both platform independent and consumable by all clients.  Fortunately, there is already a well-established method of providing this form of information across the web.  WSDL documents use XML to transmit data across networks 13.  This same strategy can be use with SLA documents to generate a document that is both human readable and consumable by software platforms.

There are two common dynamic SLA formats.  The first is the WS-Agreement spec which was created by the Open Grid Forum to fill the need for a companion SLA document for a web service WSDL 14.  The more common method, and the focus of the rest of this paper, is in the WSLA standard that was created by an IBM research group.  A WSLA document specifies the parameters of an SLA document in a standard format with consistent measurement methods and metrics 15.  Both of these document types were envisioned as dynamically generated documents.  The standard method of generating these documents
usually falls on the testing and performance measurement of the web service internally by the service provider. 

The provider can simply create a software suite for both exception testing and performance measurement.  With the result baseline data generated from the tests, a provider can review the results and automatically generate a WSLA document from the results.  This document can contain information updated in regular intervals to keep the WSLA document updated to the standard needed by the service provider and/or consumers.
Another, more interesting feature of the WSLA document is it’s capability to store data outside of the normal scope of an SLA document.  Figure 1 shows some of the typical data that a WSLA document would contain.  A WSLA document can also contain information about metrics along with information about the algorithms used within a web service.  A WSLA document can also contain the test used internally to validate the web service and the results returned from those tests 16.

Conclusion

Dynamic SLA documents (WSLA) opens up a new realm of possibility of clients with the need to test web services that they’re consuming.  The test cases and algorithms detailed in a WSLA document gives the testers a better idea of the context of a web service.  With this information in hand, the testers can possibly automatically generate unit tests that traverse all execution paths of a web service operation.  Also, testers can generate unit tests that match the tests that the provider development team used to initially test the web service.  While this information does not give the consumers the ability to white-box test the web services, this is far and above beyond the limitations set when a consumer black-box tests the web service with the aforementioned methods.  Black-box testing definitely created a large breadth of testing scenarios but this method may just help testers create large in-depth tests of their web services.
At the end of the day, some of limitations still exist.  Regression testing is still difficult due to resource consumption.  Also, network congestion and utilization is hard to predict when testing web-based services.  WSLA documents only expand the list of possible things that can be tested against.  It does not guarantee that a tester who is blind to the source code and implementation can fully test a web service.

Bibliography

Andrieux, Alain, et al. "Open Grid Forum." 14 March 2007. Web Services Agreement Specification (WS-Agreement). 9 April 2010 <http://www.ogf.org/documents/GFD.107.pdf>.
Bai, Xiaoying, et al. "WSDL-Based Automatic Test Case Generation for Web Services Testing." SOSE '05: Proceedings of the IEEE International Workshop. Washington, DC, 2005. 215-220.
Berberova, Diana and Boyan Bontchev. "Design of service level agreements for software services." CompSysTech '09: Proceedings of the International Conference on Computer Systems and Technologies and Workshop for PhD Students in Computing. Ruse, Bulgaria: ACM, 2009. 1-6.
Bruno, Marcello, et al. "Using test cases as contract to ensure service compliance across releases." In Service-Oriented Computing - ICSOC 2005, Third International Conference. Amsterdam: Springer, 2005. 87-100.
Chinnici, Roberto, et al. Web Services Description Language (WSDL) Version 2.0. Vers. 13. 26 June 2007. 2 April 2010 <http://www.w3.org/TR/wsdl20/>.
Chou, Li-Der. "Design and implementation of a policy-based monitoring system for Web Services." Journal of Information Science and Engineering 25.5 (2009): 1357-1372.
Comuzzi, Marco and Barbara Pernici. "A framework for QoS-based Web service contracting." ACM Trans. Web 3.3 (2009): 1-52.
Di Penta, Massimiliano and Gerardo, Esposito, Gianpiero Canfora. "Search-based Testing of Service Level Agreements." Proceedings of the 9th annual conference on Genetic and evolutionary computation. London, England: ACM, 2007. 1090-1097.
Di Penta, Massimiliano, et al. "Search-based testing of service level agreements." GECCO '07: Proceedings of the 9th annual conference on Genetic and evolutionary computation. London, England: ACM, 2007. 1090-1097.
Hofmann, U., I. Miloucheva and T. Pfeiffenberger. "INTERMON: complex QoS/SLA analysis in large scale Internet environment." WISICT '04: Proceedings of the winter international synposium on Information and communication technologies. Cancun, Mexico: Trinity College Dublin, 2004. 1-13.
Huang, A. F. M. "An optimal QoS-based Web service selection scheme." Information Sciences 179.19 (2009): 3309-3322.
IBM. WSLA: Web Service Level Agreement. 2004. 12 April 2009 <http://www.research.ibm.com/wsla/about.html>.
Ludwig, Heiko, et al. "Web Service Level Agreement (WSLA) Language Specification." 28 January 2003. IBM Research. 20 March 2010 <http://www.research.ibm.com/wsla/WSLASpecV1-20030128.pdf>.
Martin, Evan, Suranjana Basu and Tao Xie. "Automated Robustness Testing of Web Services." Proc. 4th International Workshop on SOA And Web Services Best Practices (SOAWS 2006). Portland, Oregon: NCSU, 2006.
—. "Automated Testing and Response Analysis of Web Services." Proc. the IEEE International Conference on Web Services (ICWS 2007), Application Services and Industry Track. Salt Lake City, Utah: NCSU, 2007. 647-654.
Michlmayr, Anton, et al. "Comprehensive QoS monitoring of Web services and event-based SLA violation detection." MWSOC '09: Proceedings of the 4th International Workshop on Middleware for Service Oriented Computing. Urbana Champaign, Illinois: ACM, 2009. 1-6.
Myerson, Judith. Use SLAs in a Web services context. 29 October 2004. 11 April 2010 <http://www.ibm.com/developerworks/webservices/library/ws-sla/>.
Raimondi, Franco, James Skene and Wolfgang Emmerich. "Efficient online monitoring of web-service SLAs." SIGSOFT '08/FSE-16: Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of software engineering. Atlanta, Georgia: ACM, 2008. 170-180.
Skene, James, et al. "The monitorability of service-level agreements for applicaiton-service provision." WOSP '07: Proceedings of the 6th international workshop on Software and performance. Buenes Aires, Argentina: ACM, 2007. 3-14.
Sommers, Joel, et al. "Accurate and efficient SLA compliance monitoring." SIGCOMM '07: Proceedings of the 2007 conference on Applications, technologies, architectures, and protocols for computer communications. Kyoto, Japan: ACM, 2007. 109-120.