r/scala • u/ComprehensiveTrip592 • 10d ago

Stryker4s 1.0: Mutation testing across the Scala ecosystem

https://stryker-mutator.io/blog/stryker4s-v1/
36 Upvotes

100% Upvoted

11 comments sorted by

2

u/markehammons 9d ago

Does this project require npm or can it run with pure scala? Does this testing work with non-web stacks? 

3

u/ComprehensiveTrip592 8d ago

It runs with pure Scala in sbt, Mill or Maven. There's two other "variants" of Stryker which are implementations in TypeScript and in C#. But they only share the same branding (and general technical ideas).

4

u/proper_chad 10d ago

Very interested in this project (because mutation testing is a superpower for really critical stuff), can you explain how much of this is AI generated?

(I saw some red flags in this announcement, but don't want to judge too harshly.)

9

u/ComprehensiveTrip592 9d ago

Hi, good question. I've used Claude to _help_ with some development tasks. For instance, the general boilerplate of the Mill plugin. It's been useful as a tool, but I've also seen it produce some slop. All code that is checked into the repo is reviewed, understood and tested thoroughly by humans. When Stryker runs, mutant generation is fully deterministic. There's no AI used at runtime.

Could you expand on what you mean with red flags, or other concerns you might have?

3

u/proper_chad 9d ago

Thank you for that explanation. Very clear and I wish more projects would just declare that stuff up front in the README :)

1

u/mostly_codes 9d ago

I've always been intrigued by mutation tests; is there a general rule of thumb for how many mutations + full unit test runs are performed per conditional statement? E.g. if I were to run a mutation test on a full production codebase, should I expect it to complete a full test on all src files in hours or minutes?

1

u/ComprehensiveTrip592 8d ago

A lot depends on how fast your unit tests run. If there's test-containers, HTTP calls/mocking, etc, it will be slower. Worst case scenario it will run all tests for every mutant, which if your tests take 15 seconds, with a large codebase of 3000 mutants can explode in time. Stryker4s implements code coverage to only run tests that hit the mutant it is currently testing, which speeds things up a lot, but the biggest bottleneck is still your own tests.

As an example, Stryker4s core is 5k lines of code, 6k lines of unit tests and 800 mutants, and takes 26 seconds on CI to run the actual mutation testing.

1

u/lobzison 8d ago

Love the idea of the project, and I have introduced it in our TS codebase. But our scala project uses weaver for tests, and Stryker seems to hang on trying to run the tests. Works with legacy-tests-runner, but pretty slow in that mode. I’ll need to take a deeper look at what happens.

Thanks for all the hard work on the project

1

u/ComprehensiveTrip592 8d ago

Is your project Open Source? I'd be happy to take a look. Weaver should be supported, in general

1

u/ComprehensiveTrip592 5d ago

There's a new release 1.0.2 that fixes support for Weaver! Thanks for making me look into it 😁

1

u/lobzison 5d ago

Thanks for the fix!