The actor, who played Voldemort five times, admits ‘that ship has sailed’ about any potential return