Measuring a GUI app without lying to yourself
You cannot measure the speed of a desktop app by clicking through it, because you never do the same thing twice. This post shows the small tool that I built instead, and the three reports that changed what I worked on.
Six runs each. Press a button to run again from zero.
What I learned
- A benchmark has to give the same answer twice. So I script the steps, even if the script is less realistic than a real session.
- Write down why the test does what it does. "120 thumbnails, because that is two screenfuls" is a reason that someone can question. "120" alone is only a number.
- Measure memory as well as time. No timing report showed the function that caused 47% of the memory use.
Why I could not just use the app
The usual method is to attach a profiler (a tool that records where the program spends its time) and click through the app for thirty seconds. That finds things that are badly broken. It does not tell me if a change made the app better, because I cannot do the same thing twice. On one run, I reached row 40 with an empty thumbnail cache. On the next, I reached row 31 with a full cache, because the first run filled it.
Run the code without the window
My fix was a command-line flag. --profile <folder>
runs the real code on that folder: it lists the folder and loads
photos, the catalog, and thumbnails. Then it prints timings and exits.
It never opens a window, so it does not use the UI library or the
graphics card.
The script fills thumbnails for 120 photos, which is approximately two screenfuls. Then it opens ten photos in sequence, the way you step through a shoot with the arrow keys. Each number has a comment in the code that explains it. The full mode is approximately 230 lines. It is not clever. It only gives me the same number twice.
Keep it out of the version that you download
Small markers attach the timing code to functions. Only two build settings switch it on, and both are off by default. With them off, the markers leave each function exactly as it was, so the release build contains none of the timing code and the browser build does not get any bigger. One surprise: the profiling library must be a normal dependency, not an optional one, because the markers have to compile in every build, even when they do nothing.
The first report
The report also showed a fix that I would never find by reading the code. The app prefetches the quick, low-quality preview of the next photo, but not the slow, sharp decode. So when you step through a folder of RAW files with the arrow keys, you wait the full 200 ms every time, even though the app knew half a second earlier which photo came next.
The second report
The profiler can also count the memory that each function allocates, instead of the time it takes. One function allocated 12.0 MB out of 25.3 MB, which is 47% of all the memory the app allocated in that session. It was the function that hands thumbnails to the UI library to draw.
The UI library can only build its image type by copying the pixels into fresh memory, even though the pixels were already in the loader's cache in exactly the right layout. So I wrote them directly to the graphics card instead. That took the function from 12.0 MB to 0.16 MB, and the whole session from 25.3 MB to 13.4 MB.
Nothing was slow. But the browser build runs as WebAssembly, and WebAssembly memory never shrinks once it has grown. So memory that the app needs for only a moment stays allocated for as long as the tab is open.
The third report
On 200 photos with a warm cache, decoding all 200 thumbnails across 8 workers took 39.5 ms in total. Analysing one photo took 7.39 ms, so 1.48 seconds for the batch. The analysis runs on the main thread and cannot be parallelized across the processor's cores, so it costs about fifty-five times as much as the decoding.
My plan had been to make decoding faster and to prefetch it. Neither change would have made a noticeable difference.