My concern with proprietary Load Testing tools

tools, testing, load testing, and personal experience
a movie still from Star Wars Episode VIII - The Last Jedi, 2017
Star Wars Episode VIII - The Last Jedi, 2017

One of the things that I mentioned briefly both on episode 2 of the tools series or my load testing “get rich fast” scheme parody was the extent of how some proprietary load test vendors will use common load testing terms and will twist them to try and fool potential customers… The biggest twist of all being the predatory and (almost) criminal schemes with how “virtual-users” tiers and similar concepts are marketed and portrayed to folks that sit behind load testing expense budgets.

The Setup

The setup for the scam starts with the victim:

person in charge of estimating load testing efforts believes their untested services will need to tackle “millions of users” or “millions of requests” and believes their apps need to scale ad infinitum and process billions of transactions on demand.

In most (but not all) cases the above is nothing but a symptom of an industry-wide fetish insecurity on infinite scalability (“will it scale?” ‘ intensifies) born from the inner dream that everybody wants to be the next Netflix and is insecure about the size of their… ermmmdigital genitalia?.

Malicious Closed source and/or proprietary load testing tool providers are aware of this fetish and exploit it as a weakness in the same way a social engineer would. The bait goes in these lines: “You know, your application is so important, we believe in your mission, we want to help you out, let’s make synergies, we think you need these many virtual users…”

So they send the victim:

In worst cases, the prices are never available in a straightforward/transparent/public way, the victim always has to go through some sort of sales folks… The spreadsheet, never the same one between customers, often shows exorbitant prices in a format that is loved by budget owners: it’s all presented in a crisp and transparent manner, usually on one column the number of virtual users (VUs), or requests per second, or some other big number without its actual meaning, and on the other column the price tag.

Since most targeted victims tend to have free-ish reign on budget, and the prospect looks “clean and corporate-friendly”, mentioning the tool is guaranteed to help the victim load test their galaxy-grade trillions of users, the victim then picks a tier, signs their souls to the tool provider… and both parties become professional best friends forever.

Side-note, for those of you that want to know more lore of this particular world build, here’s a bonus point: the poorly-paid interns and/or sweatshops that the victim organization will pay to do the actual load testing work and scripts are also certified in using the same proprietary load tool and… official partners of the proprietary load tool provider? Who would’ve guessed?!

The core of the scam

The gist of the actual scam is simple and deadly effective: attributing a concept that requires context depth, like the concept of Virtual users or requests per second a price tag, oversimplifying the problem and leaving it at that.

Here’s what these malicious proprietary load tool providers are not explaining to potential customers or leaving in microscopic print:


In the end, the crude reality of what “victim” companies will be paying for:

In the end folks are left with a disconnect between their context and their actual load testing needs and a tool which may or may not suit that context and those needs. That in itself isn’t necessarily world-ending, but it isn’t fair or right either, and personally, I just think it’s immoral. I don’t have a recipe for “fixing this”, other than some personal recommendations:

If you read this far, thank you. Feel free to reach out to me with comments, ideas, grammar errors, and suggestions via any of my social media. Until next time, stay safe, take care! If you are up for it, you can also buy me a coffee ☕