Aha! Fixed it, I think. Did a little research, and found how to write the cache for the recent topics block to a specific file. However, that file gets deleted every time someone makes a new post. Then that forces a new cache file to be created.
In any case, what that means is that every now and then, the front page will still take some time to load as the server will be busy writing the new cache. But most of the time, the front page should load in a split second.
