Machine Learning: If It’s Testable It’s Teachable

How Do They Learn?

Human programmers create a teacher bot and a builder bot who have simpler brains. Builder bot builds student bots and keeps and discards them based on their test grades.

The teacher bot himself cannot distinguish between a dog and a 5 but it can test student bots whether they are right in identifying or not. Now you get why automated testing is so important.

We give teacher bots a bunch of photos of dogs and 5s and an answer key of which is what. Based on this the teacher bot takes test of student bots and gives them the grades. Based on the test data, the builder bot keeps building on different student bots by adjusting different permutations and combinations of student bots algorithm mechanics, sometimes it even at random sees what sticks and what not. And the teacher bot keeps taking their tests and assigning them the grades. And the cycle continues.

The teacher bot keeps testing, and based on the grades of the student bots, the builder bot keeps the best performing bots and ruthlessly discards the rest.

The test, build, test cycle keeps on repeating in a loop and their grades are assessed and once the bot with approx 99.9% accuracy (or call them grades) is built the cycle is stopped.

Now the question that comes to our minds is ‘How many times the automated cycle of test, build, and test is repeated?’. Well, it repeats as many times it is necessary till the bot with the best grades is built. The best bot is the best algorithm to distinguish between a dog and 5.

Austin Siewert

Co-Founder, Steadfast Systems

Discovered @TestMu AI yesterday. Best browser testing tool I've found for my use case. Great pricing model for the limited testing I do 👏

2M+ Devs and QAs rely on TestMu AI

Deliver immersive digital experiences with Next-Generation Mobile Apps and Cross Browser Testing Cloud

Sign up with googleSign up with Email

So What’s The Problem?

Now we have picked up the best algorithm to distinguish between a dog and the number 5 then what is the problem that may occur? If we give the bot a video of a dog or 5 upside down or letter ‘S’ instead of 5 will the bot still be able to figure that out? No.

How to Solve it?

To solve this, the humans have to create longer automated test cases with more number of questions for the student bots to pass, including even the wrong and the right scenarios so that it gets prepared for the worst cases too. Since longer tests ensures better bots.

As there is not a single bot or some ten or twenty questions but millions of bots and zillions of questions so how does the test, build, test cycle repeat? In that case you have to automate the process and keep on testing the same.

When a final bot is built it works and it is the only one which survives among the all as it is the only one whose algorithm was 0.01% better than the other bot. The algorithm that the student bot has built is not known by the teacher bot, not by the human overseer and not by even the student bot itself! It. Just. Works!

How the bot thinks or works, what it thinks is not really knowable.

Let’s come back to the YouTube example that we have discussed. We can understand this better now. The task here given to the student bots is to record the watch time of a user while keeping engaged and the student bot who keeps the user engaged for the longest watch time will score the highest. The teacher bots assess all the student bots and the student bots keep on giving recommendations to the users so that the user remain engaged. The one who gives the best recommendations and keep the user engaged is the one with the best algorithm.

Have You Come Across A Machine Learning Example Yet?

Believe it or not, we’ve all come across a real-time situation where bots have interacted with us using machine learning. Be it CAPTCHA collection, or customer care support. As a matter of fact, customer support is one area where machine learning has been adopted on wide scale by small companies and big enterprises alike. We’ve all been aware of the constraints hanging around the customer chat support. You need to respond to customers in a jiffy and that can be especially tough for a global business who is aiming to provide 24/7 customer support as you need to make sure you have numerous representatives ready to help your customers all day & night. Fortunately, there are companies like Acquire which have helped numerous businesses amplify their customer experience by using their chatbot technology.

Which is the best example you could think of for machine learning bots? Let us know in the comments.

We’ve seen that the teacher bot is just testing the student bots and they are learning from the tests. So, in one line we can say that Machine learning is teachable if it is testable.

Austin Siewert

Co-Founder, Steadfast Systems

Discovered @TestMu AI yesterday. Best browser testing tool I've found for my use case. Great pricing model for the limited testing I do 👏

2M+ Devs and QAs rely on TestMu AI

Deliver immersive digital experiences with Next-Generation Mobile Apps and Cross Browser Testing Cloud

Sign up with googleSign up with Email

How Do AI Agents Test Machine Learning Models Now?

AI agents now run this guide's test-build-test cycle automatically, checking a model's predictions against validation data and flagging accuracy drift before a bad model reaches production.

• Automated validation suites: Deepchecks and Evidently AI run checks for data drift, label leakage, and accuracy loss on every new model build, playing the same grading role as the teacher bot in the unit testing analogy above.
• LLM-generated edge cases: an agent built with LLM test automation writes adversarial inputs, such as a rotated digit or an inverted image, faster than a human tester can list them by hand.
• Agentic regression checks: MLOps pipelines now run regression testing on every retrain, comparing a new model's grades against the previous best model before it replaces it in production.
• Synthetic test data: agents generate extra labeled test data for the rare cases a real dataset under-represents, closing the exact gap the inverted-image problem in this guide describes.

The test-build-test loop in this guide has not changed. Only the tester has, and it is increasingly an autonomous agent instead of a person writing test cases by hand.