OpenAI CLI: Install, Core Commands, and CI Testing

What Is the OpenAI CLI?

The OpenAI CLI is a single Go binary, openai, that maps OpenAI REST API resources to shell commands. Every command follows one grammar, openai RESOURCE [SUBRESOURCE] COMMAND [FLAGS], so openai responses create calls the Responses API and openai files upload sends a file to the Files API.

The project ships often. Its changelog starts at version 0.0.1 on May 1, and versions 1.40.3, 1.40.4 and 1.40.5 all landed on October 22 and 23.

Pin a version in CI instead of tracking the latest release, and read the changelog before you bump it.

The top-level help groups commands by job. An excerpt of openai --help for v1.40.5:

$ openai --help
OpenAI CLI  v1.40.5
Use the OpenAI API from your terminal.

LOCAL TOOLS
  explore                    Search local CLI commands and examples without
                             running them.
  codex                      Show local Codex CLI instructions and
                             official links
  tokenizer                  Explore text, token IDs, and bytes locally

GENERATE CONTENT
  responses                  Generate and manage model responses.
  chat                       Create and manage Chat Completions.
  images                     Generate, edit, save, and preview images.
  audio                      Transcribe audio, generate speech, and
                             create voices.

DATA AND RETRIEVAL
  files                      Upload, inspect, and download files.
  vector-stores              Index files and search their contents.
  embeddings                 Create embedding vectors from text.

MODELS AND JOBS
  models                     List and inspect available models.
  fine-tuning                Manage training jobs, checkpoints, and graders.
  batches                    Submit and monitor batches of API requests.

SAFETY AND PROVENANCE
  moderations                Classify potentially harmful text and
                             image inputs.

The LOCAL TOOLS group runs without credentials, and every other group calls the API. Coding agents can call the CLI the same way, as a shell command, which is the CLI side of the MCP vs CLI trade-off.

Is the OpenAI CLI the Same as Codex CLI?

No. The OpenAI CLI runs one API operation per command and exits, while Codex CLI is a coding agent that reads, edits and runs code in your project.

The OpenAI CLI docs recommend the CLI for "repeatable API work you want to inspect and rerun" and point work that "still needs judgment" to Codex subagents.

The two names still get mixed up, and the CLI handles that directly. Run openai codex and it prints Codex install commands instead of starting an agent:

$ openai codex
Codex CLI is a separate command named codex.

Install with npm (choose one installed package manager):
  npm install -g @openai/codex
Install with Homebrew (choose one installed package manager):
  brew install --cask codex

Start from your project directory:
  codex
Codex CLI handles its own sign-in.

Setup, sandbox modes, and how to verify the agent's output are covered in Codex CLI.

How Do You Install and Authenticate the OpenAI CLI?

Install it with Homebrew, Go, or a release binary, confirm the version, then set an API key.

# macOS or Linux with Homebrew
brew install openai/tools/openai

# Any platform with Go (the README asks for Go 1.26 or later)
go install 'github.com/openai/openai-cli/cmd/openai@latest'

# Confirm which binary your shell resolves
openai --version
# openai version 1.40.5

• Release binaries - each GitHub release carries zip archives for macOS and Windows, Linux tarballs plus deb, rpm, apk and pkg.tar.zst packages, and a checksums file. On Windows, put openai.exe in a folder on your PATH.
• A stale openai on PATH - older versions of the Python SDK installed a legacy openai command. If openai --version prints anything other than an "openai version" line, your shell is resolving that older binary first.
• Interactive setup - openai help setup walks you through entering a key without typing it into a command line.
• Scripts and CI - export OPENAI_API_KEY from a secret store. Admin endpoints use a separate OPENAI_ADMIN_KEY.
• Update checks - interactive runs check for updates but never install them, and OPENAI_CLI_NO_UPDATE_CHECK=1 turns the check off.

The CLI reference documents the environment variables the binary reads, including OPENAI_BASE_URL for a custom API backend. Custom endpoints must use HTTPS, except localhost and loopback IP addresses such as 127.0.0.1.

That localhost exception is what makes a local stub possible, which the offline testing section below relies on.

Which OpenAI CLI Commands Do Testers Use Most?

The commands worth scripting are the ones that return output you can assert on, or that cost nothing to run.

The tokenizer is the only command in that table that runs with no key and no network.

The CLI's local utilities guide says its tokenizer commands work offline without API credentials, sets a 1 MiB input limit, and keeps o200k_base as the default encoding, which supports GPT-5.x, o1, o3, o4-mini, GPT-4o, GPT-4.1, and GPT-4.5.

$ openai tokenizer count --file prompts/checkout-test-cases.md
Encoding: o200k_base
Input bytes: 410
Tokens: 93

$ openai tokenizer count --file prompts/checkout-test-cases.md --format json
{"encoding":"o200k_base","input_bytes":410,"token_count":93}

The count covers plain text only, not request structure or images, so treat it as a floor. When you need the server's number for a full request, the tokenizer's own help text points to openai responses input-tokens count.

How Do You Get JSON Output From the OpenAI CLI?

Pass --format json or --format jsonl, or pull out one field with --transform and --raw-output. A script that omits the flag gets text meant for people.

The CLI's output guide says the CLI uses readable text by default, including pipes and redirected files, and that a filename such as answer.json "does not select JSON".

A script that pipes openai into jq without a format flag hands jq labeled text, and jq fails to parse it.

Errors go to stderr. A key whose project has reached its billing limit shows the default shape of a failed call:

$ openai responses create --model gpt-5.5 --input "Reply with the single word PASS."
HTTP 429: Too Many Requests.
Request ID: req_83f17fc5953048b8ab378c0710aa9556
API usage or billing limit reached.
Check your project's billing and limits.
API error details: --format-error json.
$ echo $?
1

• Redirection empties the file first - the shell truncates out.json before the CLI starts, and a failed request cannot restore it. Check the exit status before you read the file.
• Incomplete JSON lists - when a later page fails, --format json can leave a partial array behind along with a nonzero exit status.
• Windows PowerShell 5.1 - redirected native stderr gets wrapped in NativeCommandError records. To capture JSON errors cleanly, run the command through cmd.exe /d /c with the 2> redirect inside that command.

How Do You Test OpenAI CLI Scripts Without Calling the API?

Point the CLI at a local stub with --base-url or OPENAI_BASE_URL. Plain HTTP is allowed on localhost, so the stub needs no certificate, every run is free, and the response never changes between runs.

This tests your script rather than the model: the parsing, the assertions, and the failure handling. The stub below answers POST /v1/responses with five test cases, two test cases, or an HTTP 500, depending on its mode, and saves each request body to last-request.json.

// stub.js: stands in for POST /v1/responses so CLI scripts run offline.
// Usage: node stub.js <port> <ok|bad|fail>
const http = require('http');
const fs = require('fs');
const port = Number(process.argv[2] || 8787);
const mode = process.argv[3] || 'ok';

const testCase = (id, title) => ({
  id, title,
  preconditions: 'A product is in the cart',
  steps: ['Open the checkout page', 'Fill in the required fields', 'Confirm the order'],
  expected: 'The order confirmation page loads',
});
const cases = [
  testCase('TC-01', 'Guest checkout with a valid postcode'),
  testCase('TC-02', 'Registered-user checkout'),
  testCase('TC-03', 'Checkout with an empty cart'),
  testCase('TC-04', 'Invalid postcode is rejected'),
  testCase('TC-05', 'Expired coupon is rejected'),
];

http.createServer((req, res) => {
  let body = '';
  req.on('data', (chunk) => (body += chunk));
  req.on('end', () => {
    console.error(req.method + ' ' + req.url);
    fs.writeFileSync('last-request.json', body);
    if (mode === 'fail') {
      res.writeHead(500, { 'content-type': 'application/json' });
      return res.end(JSON.stringify({ error: { message: 'stub upstream failure', type: 'server_error' } }));
    }
    const text = JSON.stringify({ cases: mode === 'bad' ? cases.slice(0, 2) : cases });
    res.writeHead(200, { 'content-type': 'application/json' });
    res.end(JSON.stringify({
      id: 'resp_stub_001', object: 'response', status: 'completed', model: 'gpt-5.5',
      output: [{ type: 'message', role: 'assistant', content: [{ type: 'output_text', text }] }],
    }));
  });
}).listen(port, '127.0.0.1', () => console.error('stub on ' + port + ', mode=' + mode));

Start it, then run the same command you would send to the real API with --base-url added. Any non-empty key works, because the stub ignores it:

$ node stub.js 8787 ok &
$ export OPENAI_API_KEY=sk-test-stub

$ openai responses create --base-url http://127.0.0.1:8787/v1 \
    --model gpt-5.5 --input @prompts/checkout-test-cases.md \
    --transform 'output.#(type=="message").content.0.text' --raw-output
{"cases":[{"id":"TC-01","title":"Guest checkout with a valid postcode","preconditions":"A ...

$ jq -r '.model, (.input | length)' last-request.json
gpt-5.5
410

The saved body confirms that --input @prompts/checkout-test-cases.md sent the file's contents, all 410 characters. Switch the stub to fail mode and the CLI behaves the way it would during a real outage:

$ node stub.js 8788 fail &
$ openai responses create --base-url http://127.0.0.1:8788/v1 --model gpt-5.5 --input "x"
HTTP 500: Internal Server Error.
The API is temporarily unavailable.
The API may have received the request.
Check its status before trying again.
API error details: --format-error json.
$ echo $?
1

# stub log for that one command
stub on 8788, mode=fail
POST /v1/responses
POST /v1/responses
POST /v1/responses

One command produced three POST requests before the CLI exited. The retries come from openai-go, the official Go SDK the CLI is built on, whose version bumps appear in the CLI's changelog.

The OpenAI Go SDK README says certain errors are "automatically retried 2 times by default", covering connection errors, 408 Request Timeout, 409 Conflict, 429 Rate Limit, and 500 or higher.

• Exit status - 0 when the call succeeds and 1 when the API returns an error, so set -e or an explicit check stops the job.
• Extracted JSON - the --transform output must parse and match your schema. Running the gate against the stub's bad mode proves the assertion can actually fail.
• Request body - last-request.json shows what the CLI sent, such as the model, the @file input, and any --text.format schema.
• Writes under retry - a 5xx on a command that creates something can mean the API already accepted it, as the CLI's message warns. Check the resource's status before you re-run files upload or batches create.

How Do You Run LLM Regression Checks With the OpenAI CLI in CI?

Install a pinned binary, store the key as a secret, and run one script that fails when the prompt is over budget, the API call fails, or the model's JSON is malformed. The prompt under test asks for checkout test cases against the TestMu AI ecommerce playground:

You are a QA engineer. Write test cases for the checkout flow at
https://ecommerce-playground.lambdatest.io/index.php?route=checkout/checkout

Cover: guest checkout, registered-user checkout, an empty cart, an invalid
postcode, and a coupon that has expired. For each case return an id, a title,
the preconditions, numbered steps, and the expected result.
Return JSON only, matching the schema you were given.

A JSON schema in schemas/test-cases.json goes to the API through --text.format, so the model has to return an object with a cases array. The gate script:

#!/usr/bin/env bash
# ci/check-test-case-prompt.sh
# Fails the job if the prompt grows past budget or the model's test cases come back malformed.
set -euo pipefail

PROMPT=prompts/checkout-test-cases.md
mkdir -p out

# 1. Token budget gate: runs offline, no API key needed
tokens=$(openai tokenizer count --file "$PROMPT" --format json | jq '.token_count')
echo "prompt tokens: $tokens"
[ "$tokens" -le 400 ] || { echo "FAIL: prompt is over the 400-token budget"; exit 1; }

# 2. Call the model and keep only the generated text
openai responses create \
  --model gpt-5.5 \
  --input "@$PROMPT" \
  --text.format "$(cat schemas/test-cases.json)" \
  --transform 'output.#(type=="message").content.0.text' \
  --raw-output > out/test-cases.json

# 3. Assert on structure, not wording
jq -e '.cases | length >= 5' out/test-cases.json > /dev/null \
  || { echo "FAIL: expected at least 5 test cases"; exit 1; }
jq -e '[.cases[] | (.steps | length) > 0] | all' out/test-cases.json > /dev/null \
  || { echo "FAIL: a test case has no steps"; exit 1; }
echo "PASS: $(jq '.cases | length' out/test-cases.json) test cases"

Point it at the stub's ok and bad modes before it ever touches the real API. The same script passes and fails as intended:

$ OPENAI_BASE_URL=http://127.0.0.1:8787/v1 bash ci/check-test-case-prompt.sh
prompt tokens: 93
PASS: 5 test cases

$ OPENAI_BASE_URL=http://127.0.0.1:8789/v1 bash ci/check-test-case-prompt.sh
prompt tokens: 93
FAIL: expected at least 5 test cases
$ echo $?
1

The workflow installs v1.40.5 from the release tarball, verifies it against the published checksums file, and runs the script with the key from repository secrets:

# .github/workflows/prompt-checks.yml
name: LLM prompt checks
on: [pull_request]

jobs:
  prompt-checks:
    runs-on: ubuntu-latest
    env:
      OPENAI_CLI_VERSION: 1.40.5
      OPENAI_CLI_NO_UPDATE_CHECK: "1"
    steps:
      - uses: actions/checkout@v4

      - name: Install the OpenAI CLI (pinned, checksum-verified)
        run: |
          base="https://github.com/openai/openai-cli/releases/download/v${OPENAI_CLI_VERSION}"
          file="openai_${OPENAI_CLI_VERSION}_linux_amd64.tar.gz"
          curl -sSLO "$base/$file"
          curl -sSL "$base/openai_${OPENAI_CLI_VERSION}_checksums.txt" | grep -F "$file" | sha256sum -c -
          tar -xzf "$file" openai
          sudo install openai /usr/local/bin/openai
          openai --version

      - name: Run prompt checks
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: bash ci/check-test-case-prompt.sh

      - name: Keep the model output
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: prompt-check-output
          path: out/

Model output varies from run to run, so assert on structure and counts rather than exact wording. Judging whether the generated test cases are any good is an LLM evaluation problem, not a jq one.

FILE_ID=$(openai files upload --file requests.jsonl --purpose batch \
  --transform id --raw-output)

BATCH_ID=$(openai batches create --input-file-id "$FILE_ID" \
  --endpoint /v1/responses --completion-window 24h \
  --transform id --raw-output)

openai batches wait "$BATCH_ID" --output-dir ./results --wait-timeout 30m

Is the OpenAI CLI Enough to Test an LLM Feature?

No. It sends requests and reports what came back, and v1.40.5 has no evaluation command: the top-level help lists no evals group, and graders sit under fine-tuning. These gaps need another tool or your own code:

• Scoring - assertions are yours to write in jq or a test runner. Relevance, tone, and factual accuracy need an evaluator or a judge model, the methods covered in LLM testing.
• Webhook tests - the webhook testing guide states that "A completed test keeps exit status 0, including a receiver HTTP 500 response." The same guide reads the receiver's status with --transform status_code --raw-output.
• Token counts - tokenizer counts exclude request structure and are not billed usage or context-limit guarantees.
• Conversations - each command is a single request, so a chatbot that drops a policy on the third turn passes every one-shot check.

The conversation gap is where TestMu AI's Agent Testing fits. It auto-generates 60-100+ test scenarios from uploaded docs, PRDs, or connected Jira and Confluence, scores conversations on hallucination, bias, completeness, context awareness, response quality, and more, and returns a Green, Yellow, or Red production-readiness verdict.

Note: Building a chatbot on the OpenAI API? TestMu AI's Agent Testing talks to your chat endpoint the way a user would, across whole conversations, so policy slips that one-shot CLI checks miss show up before release. Create a free account

Conclusion

Install v1.40.5, run openai --version to confirm your shell finds the right binary, and copy stub.js and the gate script into your repo. Run the gate against the stub's ok and bad modes before you add the workflow and the OPENAI_API_KEY secret, so the first real CI run tests the model rather than your script.

When the generated test cases need to run against the real checkout page, install Kane CLI and follow the Kane CLI introduction to add a browser check to the same pipeline.