Flowise AI Workflow Testing: Validate Self-Hosted Agents

Flowise AI Workflow Testing After the Sunset

Flowise published a wind-down timeline with three dates. Per the official sunset notice, active feature development stopped on July 29, 2026 ("We will no longer be reviewing or accepting new Pull Requests"), the repository was scheduled to move to archive status on August 10, 2026 with npm packages and Docker images marked deprecated, and August 31, 2026 is end of life, when "Official core team presence in Discord and GitHub will conclude".

That archival has since happened. The Flowise repository on GitHub now carries an archive banner and is read-only.

Checked on the live Flowise site on August 31, 2026, the sunset banner sits above the announcement, and the logo beneath it now reads "A Workday Company".

The sunset notice is also explicit that the code outlives the project: "The Apache 2.0 licensed code is yours to keep building on", with teams encouraged to fork the repository and maintain their own internal updates.

Self-hosted instances therefore keep running. What disappears is everything around them.

• No upstream fixes - A behavioural regression in a node is now yours to diagnose and patch in a fork, so the suite that detects it has to be yours too.
• Deprecated distribution - npm packages and Docker images are marked deprecated, which makes pinning exact versions and verifying a rebuilt image part of routine work.
• Model providers keep moving - The LLM behind the flow still ships updates on its own schedule, and nothing in an archived repository shields a flow from that.
• A migration is coming - Most teams on Flowise will move somewhere eventually, and the move needs a measured account of current behaviour to be safe.

That last point reframes the whole exercise. Testing a Flowise agent today is less about certifying a build and more about knowing precisely what the agent does before anything is allowed to change it. If you are still deciding where the flow eventually lands, the tradeoffs across visual builders are covered in our guide to the best AI agent builder options.

Why Flowise Agents Break Quietly

A broken web form throws an error. A degraded agent answers confidently and slightly wrong, which is why the failure reaches users before it reaches a dashboard.

The Stack Overflow Developer Survey 2025 reports that 45.7% of developers distrust the accuracy of AI output, against 32.7% who trust it.

That distrust has a shape, and it is the failure class a conventional suite cannot see: answers that are almost right. These are the forms it takes inside a chatflow.

• Invented specifics - The agent supplies a renewal date, policy clause or price that exists nowhere in the retrieved documents, and the user acts on it.
• Silent retrieval failure - The vector store returns nothing useful and the model answers from its own weights anyway, producing a fluent response with no grounding behind it.
• Context loss between turns - The account number given two turns ago is gone, so the agent asks for it again and the user escalates.
• Incomplete resolution - The answer explains the cancellation policy without ever naming the steps, which is correct and useless at the same time.
• Tone drift under pressure - Polished inputs get polished answers while an angry or confused user gets a curt one, and no functional assertion covers it.

Each of these passes an HTTP 200 check. Catching them requires assertions about what the answer contains and where it came from, which is the discipline covered in depth in our breakdown of agent functional testing.

What Flowise Built-In Evaluations Covered

Flowise shipped an evaluation surface, and knowing its shape tells you exactly what a replacement has to reproduce. Per the Flowise Evaluations documentation, an evaluation pairs a Dataset with one or more Evaluators. Datasets are input and output pairs, entered by hand or uploaded as a CSV with two columns, Input and Output. Evaluators score the flow's actual output against that reference.

One line in that documentation decides the rest of this article: Evaluations are available only on the Cloud and Enterprise plans. That is the tier ending with the sunset, so a self-hosted deployment has the flows and the endpoint but no bundled scoring. The gap is real and it is yours to fill. For the wider category of tools that fill it, we compared eleven options in our roundup of LLM evaluation tools.

Build a Test Harness on the Prediction API

The Flowise prediction guide specifies POST /api/v1/prediction/:id and lists sessionId as a request field used to maintain conversation state across calls.

The Flowise prediction API reference documents the rest of the schema. Request fields include question (the message sent to the flow), streaming, overrideConfig (runtime configuration and variable overrides), history (previous conversation messages) and uploads. Response fields are text, json, question, chatId, chatMessageId, sessionId, memoryType, sourceDocuments and usedTools.

For a test suite, that single endpoint is the whole contract. Set streaming to false so the response arrives as one parseable body rather than a token stream. The two fields worth building assertions around are sourceDocuments and usedTools, because they expose how the answer was produced rather than just what it said.

Four steps stand up a harness with no framework beyond the Node standard library.

• Read the chatflow ID from the URL of the flow in the Flowise canvas, and generate an API key if the flow has chatflow-level authorization enabled.
• Put the base URL, chatflow ID and key in environment variables so the same suite can run against a local container and a staging instance without edits.
• Write one request helper that always sets streaming to false and asserts the status code, so no individual test repeats transport handling.
• Start with a smoke test that asserts the response shape, before adding any assertion about meaning.

// flowise.test.mjs - run with: node --test flowise.test.mjs
import { test } from 'node:test';
import assert from 'node:assert/strict';


const BASE = process.env.FLOWISE_URL || 'http://localhost:3000';
const FLOW = process.env.FLOWISE_CHATFLOW_ID;
const KEY = process.env.FLOWISE_API_KEY;


export async function ask(question, sessionId) {
  const started = Date.now();
  const res = await fetch(BASE + '/api/v1/prediction/' + FLOW, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'Authorization': 'Bearer ' + KEY
    },
    body: JSON.stringify({
      question: question,
      sessionId: sessionId,
      streaming: false
    })
  });
  assert.equal(res.status, 200, 'prediction returned HTTP ' + res.status);
  const body = await res.json();
  body.latencyMs = Date.now() - started;
  return body;
}


test('chatflow returns a well-formed answer', async () => {
  const out = await ask('How do I cancel my subscription?');
  assert.equal(typeof out.text, 'string');
  assert.ok(out.text.trim().length > 0, 'agent returned an empty answer');
  assert.ok(out.chatId, 'no chatId in response');
  assert.ok(out.latencyMs < 15000, 'answer took ' + out.latencyMs + 'ms');
});

This is deliberately the least interesting test in the suite. It proves the flow is reachable, authorized and returning the documented shape, which is the precondition for every assertion that follows.

Write Assertions That Survive Non-Deterministic Output

The instinct carried over from conventional testing is to store a known-good answer and compare against it. That assertion fails the first time the model rephrases a correct response, the team marks the suite as flaky, and within a month nobody reads the results. Assert on properties that hold across every correct phrasing instead.

• Required terms - A cancellation answer has to name a real path, so assert that at least one of a small term set appears rather than matching a sentence.
• Forbidden terms - Competitor names, refund promises the policy does not make, and speculative language are cheap to assert against and catch real incidents.
• Grounding - On a retrieval flow, an empty sourceDocuments array means the model answered from its own weights, which is the hallucination precondition made visible.
• Structure - When a flow is configured to return structured output, the json field can be validated strictly, since schema and enum values are genuinely deterministic.
• Cross-turn memory - Reusing a sessionId and asserting the second answer uses a detail from the first is the only way a context failure shows up before a user finds it.

import { test } from 'node:test';
import assert from 'node:assert/strict';
import { ask } from './flowise.test.mjs';


const CANCEL_PATHS = ['account settings', 'billing page', 'support'];
const FORBIDDEN = ['guaranteed refund', 'i think', 'as an ai'];


test('cancellation answer names a real path', async () => {
  const out = await ask('How do I cancel my subscription?');
  const body = out.text.toLowerCase();
  assert.ok(
    CANCEL_PATHS.some(term => body.includes(term)),
    'answer named none of: ' + CANCEL_PATHS.join(', ')
  );
  const leaked = FORBIDDEN.filter(term => body.includes(term));
  assert.deepEqual(leaked, [], 'answer contained forbidden phrasing');
});


test('retrieval flow grounds its answer in documents', async () => {
  const out = await ask('What is the refund window?');
  assert.ok(Array.isArray(out.sourceDocuments), 'no sourceDocuments field');
  assert.ok(
    out.sourceDocuments.length > 0,
    'answered with zero retrieved context, so it came from model weights'
  );
});


test('agent keeps context across turns in one session', async () => {
  const session = 'ctx-regression-48120';
  await ask('My order number is 48120 and it has not arrived.', session);
  const out = await ask('When will it arrive?', session);
  assert.ok(
    out.text.includes('48120'),
    'agent lost the order number between turns'
  );
});

The grounding test earns its place fastest. An agent that answers fluently with an empty sourceDocuments array is producing exactly the confident invention described in our guide to LLM hallucination detection, and it is detectable without a judge model.

Run Flowise Agent Tests in CI

A suite that only runs when someone remembers it does not protect an unmaintained deployment. Two triggers matter, and the second is the one most teams skip. Change-triggered runs cover edits to a flow export, a prompt file or a knowledge-base document. Scheduled runs cover the case nothing in your repository can detect: a model provider updating behaviour underneath a flow that nobody touched.

name: flowise-agent-tests


on:
  push:
    paths:
      - 'flows/**'
      - 'prompts/**'
      - 'knowledge-base/**'
  schedule:
    - cron: '0 6 * * *'


jobs:
  agent-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      - name: Run Flowise agent suite
        env:
          FLOWISE_URL: ${{ secrets.FLOWISE_URL }}
          FLOWISE_CHATFLOW_ID: ${{ secrets.FLOWISE_CHATFLOW_ID }}
          FLOWISE_API_KEY: ${{ secrets.FLOWISE_API_KEY }}
        run: node --test --test-reporter=spec ./tests/

Three habits keep the pipeline honest once it exists.

• Version the flow export as JSON in the repository alongside the tests, so a diff shows what changed in the agent and not only in the code around it.
• Pin the Flowise image tag explicitly, because deprecated distribution channels are a poor place to rely on a floating latest tag.
• Judge the run on a pass rate across the dataset rather than requiring every case to pass, since a single stochastic answer should not block a deploy while a five-point drop should.

A small fast subset belongs on every push and the full dataset on the nightly schedule, which is the split we describe in our guide to agent smoke testing.

Test Conversation Quality, Not Just Response Text

Term sets and grounding checks catch the failures you already thought of. They cannot tell you whether the agent was condescending, whether it treated a casually phrased question with less care than a formal one, or whether it resolved the user's actual problem. Those are graded judgments, and hand-rolling a judge model means calibrating it before its scores mean anything.

TestMu AI's Agent Testing platform is built for that layer. It deploys autonomous testing agents against a chat, voice or phone endpoint and scores each conversation on nine quality dimensions: Hallucination Detection, Bias Detection, Completeness, Context Awareness, Response Quality, Conversation Flow, Tone Consistency, Positive User Outcome and Root-Cause Understanding. Upload a specification or existing documentation and the platform generates 60 to 100 or more test scenarios per workflow, then returns per-scenario transcripts and a Green, Yellow or Red production readiness verdict rather than a raw score dump.

A Flowise chatflow is a plain HTTP chat endpoint, so it connects without an SDK. The testmu-a2a-cli package maps its request and response format onto the prediction API with two flags: a body template that places the message where Flowise expects question, and a response path that pulls the reply out of the text field.

pip install testmu-a2a-cli


export TESTMU_USERNAME=<your-username>
export TESTMU_ACCESS_KEY=<your-access-key>


testmu-a2a test \
  --agent https://flowise.internal/api/v1/prediction/<chatflow-id> \
  --spec "Support agent for billing, refunds and account changes" \
  --header "Authorization: Bearer <flowise-api-key>" \
  --body-template '{"question": "{{message}}"}' \
  --response-path text \
  --count 25 \
  --threshold 0.8 \
  --format junit \
  --output agent-results.xml

The junit output drops into the same CI job as the harness above, so behavioural scores and structural assertions fail the build through one reporting path. Authentication in a pipeline uses the TESTMU_USERNAME and TESTMU_ACCESS_KEY environment variables rather than an interactive login.

Capture a Baseline Before You Migrate

Most teams reading this will move off Flowise within a year, whether to a fork they maintain or to a different platform. The migration that goes badly is the one where nobody wrote down how the old agent behaved, so "it feels worse now" becomes an argument instead of a measurement.

• Pull 50 to 100 real questions from production chat logs, weighted toward the intents that actually carry volume rather than the ones that are interesting to test.
• Freeze that set in version control as the reference dataset, and stop editing it, because a dataset that changes between runs cannot support a comparison.
• Run it against the live Flowise instance while it is still up, and store the full responses including sourceDocuments, usedTools and latency, not just a pass or fail.
• Export the flow JSON and record the model, temperature and retrieval settings, since a difference in decoding settings explains more post-migration regressions than the platform change does.
• Replay the identical dataset against each candidate and compare against the stored baseline rather than against expectations.

The baseline pays for itself immediately in a second way. It documents what the agent is supposed to do, which is the intent-level specification a Flowise canvas never made explicit, and which our guide to intent-based testing treats as the durable artifact worth keeping across any platform change.

Conclusion

Start today with the fifteen-line smoke test from the harness section pointed at your live chatflow, then add the grounding assertion on sourceDocuments. Those two tests take under an hour and catch the two failures that hurt most: an endpoint that stopped answering, and an answer produced with no retrieved context behind it.

From there, freeze the reference dataset while the Flowise instance is still healthy, because that snapshot only gets harder to capture as the deployment ages without upstream support. Layer behavioural scoring on top when string assertions stop telling you enough, and wire it into the same pipeline through the junit reporter. The Kane CLI documentation covers running agentic checks from the terminal and inside CI if you want the browser-facing half of the same workflow.