That's not true, Class E is controlled airspace - clearance is required for IFR. VFR aircraft do not require a clearance, but it is still considered controlled airspace.
CAO 95.55:
(d) the aeroplane must only be flown in:
(i) Class G airspace; or
(ii) Class E airspace in V.M.C.; or
(iii) in accordance with paragraph 7.3 — in Class A, B, C or D airspace;
It refers specifically to classes of airspace, not "controlled airspace". So if a restricted area is still class G, it appears that it is allowed.