Turns out the sleep that got removed in https://github.com/matrix-org/matrix-react-sdk/pull/12667 was useful.