Using graphite2's comparerenderer suggests that this makes hb-ft 15 times faster. No caching layer needed anymore.