Benchmarking and Testing

Have you ever had a piece of code you wanted to optimize but you also want to confirm it has the same results when you’re done?

I’ve definitely been there and if you have then this post should show you how to quickly plug Google benchmark and Google test into your code via cmake so you can use them to accomplish both types of testing needs!

Personally I get really annoyed when having to setup my own testing or benchmarking system. I just want to benchmark or test the code already and trust that the apparatus to do these things is well tested and already functions. This will show you how to do a quick integration with google test and google benchmark via cmake!


Google Test

You can find Google test here. (sometimes people refer to it as gtest instead of google test). I personally find this framework to be handy to work with and really powerful. I really recommend checking out the documentation.

Google Benchmark

You can find Google benchmark here. It has a lot of similarities to Google Test. The knowledge transfer between the two makes it a strong contender for me. Also a huge recommendation for checking out the docs.

So what are we going to benchmark and test?

I wrote some simple code here

source_code.h:

#ifndef SOURCECODE_H
#define SOURCECODE_H

#include <vector>

extern std::vector<int> do_something(void);

#endif

source_code.cpp:

#include "source_code.h"

#include <vector>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>

std::vector<int> do_something(void)
{
  int fd = open("/dev/urandom", O_RDONLY);
  if(fd < 0)
    throw "failed to open file";
  int v = 0;
  read(fd, &v, 1);
  close(fd);
  std::vector<int> ret{v};
  return ret;
}

How do we hook up benchmarking and testing?

First lets create the cpp files for benchmarking and testing our code.

Testing

For testing let’s make a simple test for this code

source_code.test.cpp:

#include "source_code.h"
#include <gtest/gtest.h>

TEST(do_something, CorrectSize)
{
  const std::vector<int> ret = do_something();
  EXPECT_EQ(ret.size(), 1);
}

Benchmarking

For benchmarking we can add another simple file.

source_code.benchmark.cpp

#include "source_code.h"
#include <benchmark/benchmark.h>

static void BM_do_something(benchmark::State & state)
{
  for(auto _ : state)
  {
    std::vector<int> foo;
    benchmark::DoNotOptimize(foo = do_something());
    benchmark::ClobberMemory();
  }
}

BENCHMARK(BM_do_something);

(4/13/2025 Note, this code is not a real benchmark that does anything meaningful. Check documentation before using benchmark::DoNotOptimize and benchmark::ClobberMemory. At the time of adding this note, the documentation for these is here)

Joining everything with cmake!

The last step is to finally join all the code together with cmake!

This is the most joyful part out of all of this. We can write a single CMakeLists.txt file to join all this code together for us AND it will pull the google test and google benchmark repos for us automatically!

Let’s check it out:

cmake_minimum_required(VERSION 3.14)
project(my_project)

set(CMAKE_CXX_STANDARD 14)
set(CMAKE_CXX_STANDARD_REQUIRED YES)

include(FetchContent)

set(BENCHMARK_ENABLE_TESTING off)

FetchContent_Declare(
  googletest
    GIT_REPOSITORY https://github.com/google/googletest.git
    GIT_TAG release-1.12.0
)

FetchContent_Declare(
  googlebenchmark
    GIT_REPOSITORY https://github.com/google/benchmark.git
    GIT_TAG v1.7.0
)

FetchContent_MakeAvailable(
  googletest
  googlebenchmark
)

add_library(
  source_code
  source_code.cpp
)
target_include_directories(
  source_code
  PUBLIC
  "${CMAKE_CURRENT_SOURCE_DIR}"
)

enable_testing()

add_executable(source_code_test)
target_sources(
  source_code_test
  PRIVATE
    source_code.test.cpp
)
target_link_libraries(
  source_code_test
  PRIVATE
    gtest_main
    source_code
)
include(GoogleTest)
gtest_discover_tests(source_code_test)

add_executable(source_code_benchmark)
target_sources(
  source_code_benchmark
  PRIVATE
    source_code.benchmark.cpp
)
target_link_libraries(
  source_code_benchmark
  PRIVATE
    source_code
    pthread
    benchmark::benchmark
    benchmark::benchmark_main
)

You can see above we have our nice source_code library and we simply link that up with two different executables: source_code_test and source_code_benchmark. Each of those is properly linked with the libraries to make their intended pieces function.

Quick aside 1: google benchmark depends on google test. So if you only care about benchmarking then you still need to have the FetchContent for google test. (You can omit the later pieces related to the actual test executable)

Quick aside 2: I did find a bug in google benchmark that appears to be a side effect from building with it. Open issue here, but it doesn’t impact the above CMakeLists.txt

Build and Run those benchmarks and tests!

This will create the executables for us:

mkdir build
cmake -DCMAKE_BUILD_TYPE=Release -S . -B build
cmake --build build -- -j

Now let’s run them!

Benchmarks:

$ ./build/source_code_benchmark
2022-11-04T00:40:14-07:00
Running ./build/source_code_benchmark
Run on (32 X 3700.01 MHz CPU s)
CPU Caches:
  L1 Data 32 KiB (x16)
  L1 Instruction 32 KiB (x16)
  L2 Unified 512 KiB (x16)
  L3 Unified 32768 KiB (x1)
Load Average: 1.18, 0.32, 0.10
----------------------------------------------------------
Benchmark                Time             CPU   Iterations
----------------------------------------------------------
BM_do_something        899 ns          899 ns       784836

Tests:

$ ./build/source_code_test
Running main() from ...snipped...
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from do_something
[ RUN      ] do_something.CorrectSize
[       OK ] do_something.CorrectSize (0 ms)
[----------] 1 test from do_something (0 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (0 ms total)
[  PASSED  ] 1 test.

And with that we’re done! As new functions get created in the source_code library we can continue to design tests and benchmarks for each of them, place them in their respective files, and rerun!

In closing I want to echo again checkout the documentation for both of these libraries: google test and google benchmark. There’s a lot more power in them than I’ve shown here, and I hope this is a quick enough introduction to encourage their use whenever you need to test or benchmark C++ code!


Thanks for your time, Tyler Sean Rau

Written on November 3, 2022