I have a real thing about “System Integration Testing.” Not the type of testing, thats neccesary sometimes. Mainly how excessive amounts of it result from wicked organisational problems, you know the ones. Horrible, cultural bottlenecks that no amount of AI will save us from, until it replaces software and organisations entirely I guess. Although its having a good go at killing off software-as-a-service, which is responsible for a lot of system integration horror stories.
Anyway, I’ll give you an example of what I mean. When you’ve got a load of teams responsible for different parts of a wider system. Like when you have a multi-channel thing going on, with a website, mobile app, even telephone support. Throw in a payment gateway, identity verification, third party systems to provide specialist stuff you can’t or don’t want to build. You end up with a system where no one is responsible for making it work together.
So you always end up with “broad integration testing“, as coined by Martin Fowler. Where you need integrated versions of all the systems, working at the same time, in the same environment. This generally never happens and you just test in live anyway, forcing testers to justify this like its their fault. Basically the definition of slow, expensive and stressful. It is an excellent place to find teams of testers, gantt charts and slide-decks with RAG statuses, fighting/adding to cultural headwinds.
I have a few problems with this arrangement:
- Its a lot like Release Testing Theatre – most types of testing are at risk of being pure theatre, when it comes down it not that many actually warrant being a major part of your strategy. System integration testing is the same, it is theatre for cultural failings, designed to keep a select group happy and maintain the status quo of low quality, project based work.
- Testers pay the price – this is the bit that really annoys me. The testers in these roles are often from offshore, outsourced providers who are fed a diet of crap work. It is classic outsourced fodder, with lots of testers doing very little for a long time, then crunched by inevitable delays, with little support from the host organisation. I simply won’t do it, it’s rubbish work.
- The wrong organisation ends up doing the testing – all the systems need to work together, or the product doesn’t work, therefore no value can be generated. You usually end up with at least one third party who don’t test and rely on the host organisation to test for them. And you have to, to make it work. You pay for third party system, then you pay to test it. Zero accountability. This is wildly common, one of the things that makes the testing consultancy world keep spinning I would say.
- Its big project land – organising into projects for products that will exist for many years is just the biggest anti-pattern, one which demands system integration testing. Projects cannot fathom ownership beyond their limited life span, therefore leave gaps, usually in the spaces between systems. Fund long running product teams, while putting serious thought into who owns what.
- (Most) Long standing test environments are bad – in order to do this sort of testing, you need long lived test environments, which rot and no one cares for. They are often abandoned for long periods too, see organising yourselves into projects. Plus the environment never has what you actually need in it, bits are missing or configured differently to production, so your testing has a credibility problem anyway.
- System integration is not just testing – this seems to be news to most organisations, but you can’t test an integration into existence. You need to collaborate over it, understand authentication, share your interfaces and establish a relationship between the owning teams. Establishing a team of system integration testers to run tests that EVERYONE KNOWS WILL FAIL FIRST TIME, is silly beyond belief, yet is startlingly common. Have a team that includes people who can integrate things, as well as those who can help you test them.
- Its a haven for bad actors – Too many independent actors, delivering their bits and not giving a shit if the journey works. Just so they can say they have done their bit, when they haven’t as it doesn’t work. And then, and this really hacks me off, you expect testers to make it all work together. Testers. Who have the least influence in this situation you could imagine. Hang your heads in shame project managers/delivery leads/whatever.
- You can never entirely be rid of it – this is main point for me. Once you let system integration testing with a team of ten offshored testers take root, thats it, the arrangement will persist to the heat death of the universe. It will be last team left once all your project teams have disintegrated, testing changes by your third party dependencies. Ironic really, the only team remaining will be able to give you information about the integrated systems, but with limited ability to make changes.
Look, if you have organised yourselves into some nightmarish organisational pattern, I’m not saying you don’t need to do this testing, because you do need to know that it works together. If you can imagine a different future, form long running teams that own things, collaborate over interfaces, hold third parties to account for quality and be kind to those precious few that actually enjoy system integration testing. Sounds weird I know, but they do exist. I’ve met them.
