What ImageIO gives you free, and what it costs to leave

When I moved LightPhotos off the Mac, I expected the window, graphics and file dialogs to be hard. The hard part was reading an image file.

The one call

ImageIO is Apple's built-in image library. One of its functions does a scaled decode: "give me this image, already shrunk to about this size". You do not decode the full image and resize it yourself. The size is part of the request.

For a camera RAW file, that makes a very large difference. A RAW file holds the raw data from the sensor, and to turn that data into full colour takes a lot of work. But almost every RAW file also carries a small JPEG preview for the camera screen. If you ask ImageIO for a small version, it finds that preview and skips the sensor processing entirely. On Sony ARW files, I measured 11 ms instead of 170 to 210.

No other platform has that function

I found no ready-made equivalent on any other platform, so I built it. Most RAW files are TIFF files underneath. A TIFF file is a list of labelled entries (tags), and the preview sits at a known spot in that list, so I walk the TIFF and EXIF tags to find it. If that fails, rawler, the open source RAW library I use, falls back to a full sensor decode and demosaics the data.

Two steps in the wrong order

That 680 ms started as 1.80 seconds. My code denoised the full-size, 24-megapixel sensor image, and only shrank it afterwards.

Swapping the two steps saved 1.1 seconds per photo. Check the order of your processing steps.

The other costs

File formats were the second cost. ImageIO reads HEIC, the iPhone format. Nobody has written a HEIC reader in pure Rust, the language LightPhotos uses. My only option was to hook into a C library, which would add build tools on every platform that I want to keep simple to build. So HEIC still only works on the Mac, and you can see that as a user.

The third cost was a library that did not run in a browser. The RAW library checks the system clock to log how long each photo took to load. Under WebAssembly, no clock exists behind that call, so it crashes. My build now downloads the library and applies a four-line patch. Cargo, the Rust build tool, cannot apply a patch for only the browser build, so every build uses the patched copy.

A platform library has a price: speed, file formats, running on other systems, and bugs you cannot fix. I guessed three of those four wrong. Only the replacement, timed against the original, told me the real price.

Making the other platforms look like the Mac

The photo also had to look the same as on the Mac. That was the hardest part of the move. Bigger photo projects have camera profiles, each tuned for one camera model to describe how it records colour. I only have the basic colour data that rawler keeps for each camera. My first version looked flat and about 1.45 times darker than the Mac.

• Use what the camera recorded. I apply the saved white balance, then convert to screen colours with the numbers rawler has for that camera.
• Handle highlights the way Apple does. I pull back colours that are too bright for the screen gently instead of cutting them off. The very brightest tones go to pure white, as in ImageIO.
• Trim the edges. I did not trim all of the sensor border pixels, and those dark edges made the browser version visibly darker.
• Turn the photo the right way up. The RAW library reported every photo as upright, so I read the rotation myself.
• Reduce noise. ImageIO denoises automatically when it opens a RAW file. I add a fixed amount of noise reduction in its place.

Brightness was the biggest piece. Tuning brightness and contrast by eye got close but never looked quite right. So I wrote a small tool that opens the same RAW file with ImageIO and with my code, and compares how bright the results are. Across 148 Sony files, it aligned my output with Apple's at 17 points from black to white. I kept the middle value at each point and fitted a curve. On the ten files I checked, my version had been 1.34 to 1.51 times darker than the Mac. With the curve, it is within 2% either way.

It is still not a perfect match. I only fitted the curve on Sony files, because I shoot with Sony, so other cameras may not match as closely. The curve fixes brightness and contrast, not colour. Fujifilm cameras with X-Trans sensors also get a softer result than on the Mac.

Apple's decoder is not always right

My plan for an accuracy check was to open a test file both ways and treat Apple's result as correct. Apple's decoder could not open the file. It cannot read Linear DNG files at all.

So I made an artificial DNG file that holds a known mathematical gradient, and I calculated every pixel by hand. The Rust decoder matched exactly, with zero mismatches across 147,456 samples. If two decoders agree, that only tells me that they agree. When one matches numbers I worked out myself, I know that it is right.

I would use it again

I still use ImageIO on the Mac. It gives decades of format support, a fast shortcut I would not have thought to build, and correct results on camera formats that are mostly reverse-engineered, with quirks from each camera maker. I rebuilt a small piece of it, and now I respect it much more. Thank you, Apple.

The price is that ImageIO only works on Apple's platforms, and you cannot fix some of its bugs. I cannot patch the Linear DNG gap. The only workaround is to ship a second decoder.