You are correct assuming the aircraft was already at circuit height but what if an aircraft was overflying the active side (Circuit height + 500ft) to come in for a mid-field crosswind join? I think its a smart idea waiting until late downwind to depart/climb. Personally I would try and depart upwind where possible.