Self proclaimed UN of Testers latest discovery

istqb, fever dreams, and testing
a movie still from Blade Runner 2049 movie from 2017
"Blade Runner 2049", 2017

UPDATE:This blog post was also published on Tea-Time with Testers’s April 2020 magazine.

Someone over at self-proclaimed united nations of testers, also known as International Software Testing Qualifications Board (ISTQB) published recently a letter (a white paper really) of what I assume is the result of having a fever dream after having watched Blade Runner 2049. Something tells me they probably were left thinking that the Wallace Corporation is the hero of the story (For those who’ve never watched that movie, a similar analogy is believing the iceberg was the lead hero of the Titanic movie).

Jokes aside, my belief is that what the white paper means for the industry and the testing professionals should not go unchecked. You can read the full thing here.

Here’s a few excerpts on the white paper:

“Software testing is moving towards more automation”; (…) “In the future, there will be more testing technology solutions” (…) “The testing technology solutions will have ‘intelligence’ built into them.” (…) “These testing technology solutions will iteratively learn the testing discipline from testers and testing experts” (…) “The testing technology solutions will also learn from themselves and, therefore, teach themselves as part of the iterative learning and evolving testing discipline, especially emphasizing testing quality attributes.” (…) “The testing technology solutions will have the ability to explain to testers and testing experts the rationale supporting their recommendations and solutions.” (…) “As more software solutions are being made available for use and as these software solutions expand into more industries, there needs to be a greater definition and integration of the training solutions for testing professionals.” (…) “Testing professionals shall need to embrace the new methodologies and technologies around software development and testing, with intent to integrate these faster into their testing technology solutions.” (…)

If you know me personally or from my blog writings, you probably guess what I’m thinking: A lot of potential hogwash in so few sentences, to avoid calling it something else.

But I for one am going to hold my feelings for a moment, let’s give the whitepaper the benefit of the doubt and dissect a bit what this might mean, and I’ll be doing it keeping in mind:

There’s 3 points I think makes sense to explore: “automation”, “intelligence” and “embracing” the ideas behind the whitepaper.

The “move towards automation”

The “saying” that the craft is pointed in the direction of automation holds some truth to it, as well as horse droppings.

It’s undeniable that, from a “buzz-word” standpoint, indeed Software Testing seemingly has been moving more towards “Automation”. The common industry accepted understanding is that there is such a thing as Manual Testing, and the paired industrial understanding that where you see the word “Manual”, there’s bound to be a machine that can do it better, faster, cheaper. And in this sense, there’s no mute button for reality, and the reality is: New tools. New software libraries. AI advertising of “it’ll do it for us”. New ways of representing a view in an object. New ways to call an API or publish a message in the context of a scenario that will assert something in the end. New ways of organizing “testing” code. etc. Even personally, it’s undeniable: I spend a portion of my working day coding, using tools, and maintaining scripts, and I am a Tester.

The horse dropping part of it all comes when you confront the “buzz-word” with different abstractions of the whole actual Testing phenomenon, and in its truest raw form:

Testing is undeniably human.

There’s simply no way around it. Reducing Testing to logical assertions written in a programming language, or mixing that with some “intelligence” (aka. some “dumb” machine learning model), is indeed a faulty approximation of Testing as a whole, that comprises: investigating, exploring, purposeful play, dealing with confusion, risks, problem hunting, communicating, representing, and yes, also logical checking.

Excelent testing is a mix of all of these things, and all of them bear the undeniable mark of their creator: the human being.

The confrontation of the”buzz word is actually horse droppings” has been happening now for maybe almost two decades (maybe more), and folks left and right have come forth on plenty of occasions saying both “true-ish and horse dropping” words of disorder: “Test is Dead”, “Automation Testing is the future”, “Bots will replace human testers”, “Agile testing is a thing”, “The death of manual testing is NOW”, “Let’s all get along, testers are cheerleaders”, “The role of QA in Agile Teams” …

The above ends up being paced alongside their counterpart: individuals and small groups of people pointing out that:

And there’s always these two sides, who both offend and feel offended, invested in a strugle between

Curiosly enough, I think both sides would be able come up with ways to tag any of the above descriptions to the opposing side. Every year there’s a whole living and breathing market for trainings, conferences and posts built around promoting or deconstructing these words of disorder, propagated globally faster and faster on every iteration. Such is life.

The “intelligence”

I admit there’s bound to be software now and in the future that can mimic certain aspects of our natural intelligence. I also admit there is the possibility to build inteligent agents who can greatly surpass humans in certain specific capabilities or very specific activities.

To be honest, also I’m not prepared to go into all the ethics and existential issues that surround many of these topics. I have zero desire to trash the past and ongoing work of people who are dedicating their lives towards investigation in artificial intelligence, machine learning, or its several brain-children like deep learning, and networks and specific algorithms.

But, I can talk about what I’ve experienced first-hand in one story to elaborate on my skepticism on any “intelligence” sales-pitches arguments.

My skepticism on implementation

There’s a couple of startups sprewing around selling the “idea” (and implementation) that they’re developing “robots” with some capability of learning (in the shape of software as a service) that will go through your App or API and will signal inconsistencies, possible problems, and even predicts pain points, all in neat dashboards and well integrated with any typical continuous-integration pipeline.

As Mr. MacMahon would say in character in a WWE show if he were a software person - “We’re developing this solution to finally replace those filthy and smelly humans and- *GASP!* (he gets suprise whipped by Steve Austin)”

If it’s intended to help/assist, then I don’t really have a problem with software that helps me, or conveys me some bits of useful information. Anyone who knows me knows I’m a somewhat of a freak adept of tooling like (hardcore) static analysis, proper unit tests (aka checks), and using code quality tools in a hardcore and extensive way if we’re serious about the whole deal, and specially in opensource projects.

The problem is this crap stuff is advertised as “the next step”, one to replace Testers in practice.

Added to the mix, it might be the case the main “gurus” behind the foundation of said startups and software have very established credentials, and a subliminal, if not natural, disdain for the Testing craft (oversimplifying it to plain checking at different levels) and in terms of their usual lingo are the typical shallow “automators” - meaning they’ve done in part the same I also do everyday - coding automated checks - but they elevated the checks to a replacement of deep and meaningful testing.

Now, I won’t go ad hominem straight away. They don’t love or respect the craft with their time given on this earth, that’s fine by me. But using their established status or their history,

“I worked at EvilCorp for N years as Omega Director of Testing & Quality” (Yup, Mr. Robot reference)

to sell “air in a jar” software, directly mocking Testers, that I can’t stand.

Opening the “air jar software”

I invest some of my free time hacking away at any piece of software I can get my hands on. One of those sessions last year was spent hacking away at one of these fancy “Automated AI Testing” services. After some time tinkering around I was able to notice it didn’t actually do anything as advertised, and their entire Admin API was badly protected, so any attacker was also able to create accounts with admin permissions, spam the service with crap data and do a couple more mischevious stuff. Suffice it to say:

As an extra point, and none of these companies will probably do this in the next 10 or more years, if there’s a service that advertisedly “tests” my build artifacts, the only feasible way I see of remotely adopting this as a helper stage in my CI pipelines is if I have full open access to what exactly that service does, and the “if-else” chains in the scripts it’s made of (advertised as AI). And by open access, I mean also opensource access to its code. Why? Two reasons:

1. Evidence of testing & Test Reporting.

It’s as simple as this: as a Tester or a Test Manager I expect myself or any of my Testers to produce and convey evidence of their Testing. I want to understand at any stage in a clear and human way:

And no, I’m not talking about some shallow test-case counts, percentages, graphs, push notifications, some “superficial” material-designesque test reports, among many other things like this that are sold as “evidence of testing” or “quality reassuring”.

2. Power to the people. Yeah I’m going that way. Sure, GOOD Testing is expensive. But it is also my belief Testing is “free”, and I mean it as “free” as in “freedom” (not necessarily as in “free beer”). Knowledge surrounding testing and other topics should not be contained behind a corporate or academic veil. It should be free and available for the thirsty.

Building a monopoly and then charging money for infrastructure plus some obfuscated service that has scripts that perform some automated checks on your build artifact is as sketchy (if not more) as it is selling licenses (plus painted-over infrastructure) for “load testing tools”, when there’s plenty of open tools that work better in almost all cases, it’s just that there’s usually laziness involved in the load generation infrastructure setup part, and excuses start popping up.

Embrace or perish

If it’s by chance, it’s probably an unfortunate one that we read throughout the whitepaper:

(…) there needs to be a greater definition and integration of the training solutions for testing professionals (…) (…) professionals shall need to embrace the new methodologies and technologies (…)

And let’s think for a moment… who are the self-proclaimed responsible entities that we know about that by mere causality… are the de-facto go-to “industry nice-guy/nice-gal” that sells, “validates” and “certifies” training solutions for testing professionals?

I’m not trying to imply anything, but let’s take this to an imaginary surreal fairy-tale scenario…

A tale of towel salesmanship, by Filipe F.:

I’m born Portuguese, so I’m by default a certified Towels salesperson, right?

Let’s say I’m also the founder of International Jedi Council of Portuguese Towels (aka the IJCPT). We promote sell programes and certifications to validate other well respected and certified Towels salespeople like myself.

So tomorrow I go about my business and announce to the world that the future for towels-selling is Towels that have a special fabric called ”s-moke”. And where can people get training and certification? Well I’m happy to say that the IJCPT is keeping up with these latest industry trends, and fear not, we can train sell any decent and concerned Towels salesperson preparation.

There’s a slight problem with the IJCPT… you see, we have a whole organisational structure, a bit like one of those genuine-honest-to-god-not-a-pyramid-scheme multi-level marketing institutions that are around? You know those? But hey, you can trust me, I’m trusted in all things Towel-selling by the whole industry by all my non-Towel-selling colleagues (like Senior Leaders, Recruiters, Managers, …).

And for the record, I’m not saying “s-moke” Towels will 100% certain be the future… but I predict they will, and you should all pay attention to what I say.

So don’t forget: In practice, in this industry, no one else is acceptable in the eyes of God or Nature to “validate”, “certify” and “assure” your Towel selling skills. So you better take my expensive affordable certifications and trainings or you’ll be out of a job once the ”s-moke” Towel technologies reign.

Fin.

Not implying anything, but you get what I mean, how this all might look weird for any “layman” in the end?

A last remark

Enough ranting. I’d like to direct this part to the team behind the white paper, if they ever read this post:

Yaron Tsubery, Dave Miller, Patricia A. McQuaid, Ceren Sahin Gebizli, Lucjan Stapp, Dani Almog, Filipe Carlos, Miroslav Bureš, Ritendra Banerjee, Adam Roman, Capers Jones, Alan Page, Bill Helfley, Colin Onita, Erik van Veenendaal , Harry Sneed, Jakub Rosinski, Klaudia Dussa-Zieger, Mark Gillenson, Olivier Denoo, Padmanabhan Balasubramanian, Paul Xzhang, Przemyslaw Niedzialkowski, Radoslaw Smilgin, Robert Binder, Robin Poston, Stephan Schulz, Steve Hutchison, Yao Shi, Alon Linetzki, Armin Metzger, Jasbir Dhaliwal, Kari Kakkonen, Leanne Howard, Michael Stahl, Tamas Stoeckert, Tilo Linz.

Fellas, I’m just a random dude. Not accomplished in much. Definitely not a bright mind. Ask some of my family members or old and current colleagues, they can tell about all the times I was a jackass. It would be a folly for me to, without knowing any of you, to go ahead and say you’re all a bunch of boneheads. It’s not necessarily brave from my side to also just take on some “group”, without considering there’s real human beings behind it. You wanna make a buck selling (or imposing to the industry) training and “I’m certified” stickers, be my guest: I don’t like it. It’s no secret I’m against it, and my personal experience tells me time and time again, there is a indeed a better way. But really, I’m not in an ideal position to judge any of you.

I would like for you, if any of you reads this, to keep this in your hearts: the future we will all get to live on the little time we have on this earth, in part is a future of our choosing, both individually and collectively. You’ve made your choice, and probably the near future will prove it a right choice for your pockets, personal well being and standing in the overall community. History is curious though, everyone says History doesn’t forget.

“People don’t forget, nothing gets forgiven” would a character say in RDR game.

We unconsciously post our inner lives online, fly on planes, install security devices in our homes, drive to work, order food online, check our savings in a half second, invest in Tesla stocks with a tap of our fingers, and later on we will get automatically transfered our retirement money every month. Each of these things and so many more of them are built by imperfect humans, tested by imperfect humans (when and if they ever get meaningfully and decently tested), and sometimes, sure enough, kill imperfect humans. And I’m afraid, because we don’t have to go very far off into the future, the present times prove to us that to make an “inteligent automated software testing” soup, or “half-assed certified” testing efforts, and think it’ll remotely replace a craftsman and imperfect tester, is an entry way for more and more embarassing bugs, which will potentially lead to the demise of many fellow humans. And the first responsible wont be the tester or the developer who either missed finding the bug, or inadvertedly coded the bug. Neither will it be the greedy business man standing behind them.

The first people responsible are the educators and in the limit, the “certifiers”. You may never be convinced that Testing is human to its core and instead paint it over with terrible bad jokes of science fiction (unworthy of the less good P. Dick or R. Heinlein novels), but keep in mind there are always consequences, positive and negative, in educating about Testing, whatever your angle is.

As educators, never forget that if we live, breath and mostly make a profit of “defined and integrated training solutions and certifications”, for our fellow imperfect humans, nothing will ever clear any blood we may gather on our hands through time.

Take care.