Uncovered

Before I got into software quality management, I studied testing. There's a field called combinatorial testing. OS × browser × resolution × authentication method × payment type — as parameters grow, the total combinations explode exponentially. Testing all of them isn't realistic. So you use pairwise methods to generate the smallest set of test cases that covers every value combination between any two parameters at least once. Two-factor coverage.

Why is two factors enough? Empirically, most software defects are triggered by the interaction of one or two parameters. Bugs that only reproduce with three or more factors in combination do exist, but their frequency drops dramatically. Pairwise hits that boundary. A pragmatic trade-off. Not a silver bullet. But when exhaustive combinations aren't feasible, this is where you get the most coverage for the cost.

The standard tool for this is Microsoft's PICT. It was the industry standard. Still is. It's been in use since the 2000s. Battle-tested. Written in C++, fast. A polished command-line tool.

Even back when I was using it, the fact that it could only generate bothered me.

On a project with hundreds of handwritten test cases, the first thing I wanted to know was how much of the parameter space our existing tests actually covered. PICT doesn't do that. It's a generator, not an analyzer. I wanted to add just the missing cases as a delta, but the only option was to dump everything into a seed file and regenerate the entire set. Output was TSV — you had to parse it before your program could use it. Constraints had to be written as DSL strings, painful to assemble dynamically.

PICT isn't the problem. It's designed as a command-line tool: feed it a model file, get test cases. Within that scope, it's close to perfect. What I needed just wasn't in PICT's scope.

So I rewrote it from scratch. I didn't want to be pulled along by PICT's internal design. PICT pre-enumerates constraints as Exclusion sets. Forbidden combinations listed upfront. Reasonable for a command-line tool, but not suited for grafting on "analysis" or "delta generation" after the fact.

coverwise's constraint engine runs on three-valued logic. True, False, and Unknown. While building test cases one parameter at a time via greedy search, some parameters don't have values yet. A constraint like "if OS is Windows then browser is not Safari" — if OS hasn't been assigned yet, the evaluation result is Unknown. PICT's Exclusion approach can only judge complete combinations. Three-valued logic lets you prune at the partial-assignment stage. The more complex the constraints, the greater the benefit should be.

That fundamental difference in design made operations possible that don't exist in PICT.

One is analyzeCoverage. Pass in parameter definitions and an array of existing test cases. You get back the coverage ratio, a list of uncovered tuples, and the reason each one is uncovered. os=Linux, browser=Safari excluded by constraint. os=Windows, arch=arm64 untested. All as structured objects. PICT doesn't have this.

The other is extendTests. Keep existing tests fixed, generate only the delta needed to reach 100% coverage. PICT has seed files, but those regenerate the entire test set containing those combinations. Not designed to return only the missing pieces while leaving existing tests untouched.

analyze to understand where you stand. extend to fill the gaps. This loop is the center of coverwise's design.

I changed how submodels work too. PICT uses a pseudo-parameter transformation approach. In coverwise, each submodel has its own independent CoverageEngine, and the greedy loop scores candidates against all engines combined. Mixed strength like "2-wise globally, 3-wise for payment parameters only" fits naturally into a single loop.

Output changed from the ground up. PICT outputs TSV — parameter names in the header row, values tab-separated. You have to parse it to use it programmatically, and uncovered tuple information isn't included. coverwise returns structured objects. An array of test cases, coverage ratio, uncovered tuples with reasons, improvement suggestions. Input is a JSON object passed directly. No DSL string assembly. A format that LLMs can generate and consume easily too.

There are npm packages that wrap PICT, and WASM-based implementations exist too. coverwise has a Pure TypeScript entry point on top of WASM. No WASM loading needed. npm install and it runs. Browser, Node.js, Deno. Zero binary dependencies means zero CI setup.

I built a constraint builder API too. when('os').eq('Windows').then(when('browser').ne('Safari')) — constraints with TypeScript type completion. You can write them as strings, but you won't catch typos, and refactoring tools won't follow when you rename parameters.

I wrote the first version in C++. With AI, it took less than a day. Too fast. Once it was running, I realized TypeScript was fast enough at this scale. Rewrote a Pure TypeScript version too. Still under a day. It would have taken months before. But the joy of building got blown away with it. What remained wasn't satisfaction. It was emptiness.

I once watched a test engineer on a project drowning in case explosion. They optimized the combinations in minutes. I admired that. coverwise grew out of that admiration.

I don't feel any closer to that engineer.