The most useful way I’ve seen such contract tests work is that the team that consumes the messages creates and publishes an artifact in their build pipeline for use by the creators of the messages. For this example let’s have it create a tarball with a shell script entry point. The inputs to the shell script can be a URL to the api-server and any other parameters required, like user IDs, oauth tokens, etc. The pipeline of the api-server team downloads the current production version of the contract tarball (and maybe the latest build as well). They will probably have a bunch of tarballs to download depend on how many teams use that api. The pipeline extracts and runs each contract. If the tests succeed, the api team know that their changes will not break production. If the tests fail, they know which team they need to talk to and schedule downtime/releases/etc.

Since the contract tests try to use the same code that they will use in production (eg. on the browser) to parse and understand the responses from the api, there is less chance that the contract test and production code will diverge and a regression will make it into production. Obviously it is up to each client team to decide how to write the tests. If they are happy with a schema validator and a wrapper shell script, that’s up to them. I haven’t had a lot of success with that, but I am ready to be surprised. My dislike for schema validators stems from differences between them and how production/runtime code maps from responses to domains. If you perform any transformation on the fields, change the structure, handle legacy conversions, etc, what you get at the end of your response parsing may not be the same as the response that you parsed. All of that logic has to be correct and somehow duplicated by the schema validator.

I really want to encourage client & server teams to become more courageous in terms of making changes. If schema validation works for that then that’s ok. I want us to trust our tests and automation and have little fear about pushing out new versions.