mk: Don't run benchmarks with `make check`

The current suite of benchmarks for the standard distribution take a significant
amount of time to run, but it's unclear whether we're gaining any benefit from
running them. Some specific pain points:

* No one is looking at the data generated by the benchmarks. We have no graphs
  or analysis of what's happening, so all the data is largely being cast into
  the void.

* No benchmark has ever uncovered a bug, they have always run successfully.

* Benchmarks not only take a significant amount of time to run, but also take a
  significant amount of time to compile. I don't think we should mitigate this
  for now because it's useful to ensure that they do indeed still compile.

This commit disables --bench test runs by default as part of `make check`,
flipping the NO_BENCH environment variable to a PLEASE_BENCH variable which will
manually enable benchmarking. If and when a dedicated bot is set up for
benchmarking, this flag can be enabled on that bot.

diff --git a/Makefile.in b/Makefile.in
index c0e1973e95e59038da25441e125095d1fbc1b10f..5d37c1da8fb3cb51453324b5c76d9aca11b4f4da 100644 (file)
--- a/Makefile.in
+++ b/Makefile.in
@@ -68,7 +68,7 @@
 #
 #   * `TESTNAME=...` - Specify the name of tests to run
 #   * `CHECK_IGNORED=1` - Run normally-ignored tests
-#   * `NO_BENCH=1` - Don't run crate benchmarks (disable `--bench` flag)
+#   * `PLEASE_BENCH=1` - Run crate benchmarks (enable `--bench` flag)
 #
 #   * `CFG_ENABLE_VALGRIND=1` - Run tests under valgrind
 #   * `VALGRIND_COMPILE=1` - Run the compiler itself under valgrind

diff --git a/mk/tests.mk b/mk/tests.mk
index befd1a737f7e67a1a096b3c8f1fcf0a99e1ae028..8f20d55e3859cf997ee36e7dd75cd8f6042cd0eb 100644 (file)
--- a/mk/tests.mk
+++ b/mk/tests.mk
@@ -32,7 +32,7 @@ ifdef CHECK_IGNORED
   TESTARGS += --ignored
 endif
 
-TEST_BENCH = --bench
+TEST_BENCH =
 
 # Arguments to the cfail/rfail/rpass/bench tests
 ifdef CFG_VALGRIND
@@ -40,8 +40,8 @@ ifdef CFG_VALGRIND
   TEST_BENCH =
 endif
 
-ifdef NO_BENCH
-  TEST_BENCH =
+ifdef PLEASE_BENCH
+  TEST_BENCH = --bench
 endif
 
 # Arguments to the perf tests