The most useful way I’ve seen such contract tests work is that the team that consumes the messages creates and publishes an artifact in their build pipeline for use by the creators of the messages. For this example let’s have it create a tarball with a shell script entry point. The inputs to the shell script can be a URL to the api-server and any other parameters required, like user IDs, oauth tokens, etc....
One of the best descriptions of the hierarchy of tests that I have seen comes from "Growing Object-Oriented Software, Guided by Tests" by Steve Freeman and Nat Price. 1. Acceptance Tests: Does the whole system work? 2. Integration Tests: Does our code work against code we can't change? 3. Unit Tests: Do our objects do the right thing, are they convenient to work with? Can we now get on with some work, rather than discussing definitions ad nauseam?...
I've been saving this rant for a while now: 1. Test everything at the front-end, in exquisite detail - every project sponsor understands what tooltip 0 really means. Also a great idea if you like long-running and fragile tests that require deployments, browsers, testing frameworks and the kitchen sink. Testing at different layers, and perhaps even without a browser or (in java) a servlet container is for the weak. 2. Perform a database cleanup before and after every test, whether it needs to be done or not....
A year ago I wrote a post about about writing a program to performance test web applications (see Thumper). Since I finished off the original set of python scripts, I haven't really had the time to continue developing this application further in either Java, Ruby or any other language. Fortunately there is someone out there who thinks that threaded performance tools have an inherent flaw. Enter JCrawler ... go out and try it....
Erk, Avalanche is not as nice as I originally thought. It seems that it tries too hard to keep an average load rate. When responses take too long to return, it sends out more requests. The end result is that it overloads the application server. Damn it, why can't someone write a testing tool that sends out requests at a specified rate. Oh, hang on, that's what Thumper is supposed to do....
Over the years I've been involved with writing a number of web applications. Some I am proud of, some I never want to see again. During each project, testing of the application's performance, stability under load, and regression testing of user journeys always comes up. There are a number of great bits of hardware (eg. Avalanche), and software (eg. Grinder) that help you test applications. I've even had a fun time using a room full of Grinder clients to test web services for a client - a great way to run a denial of service attack :-)...