The Appointment’s ‘Organizer’ field works like a second and independent ‘Owner’ field ignoring all other security configuration.

If a user is the Organizer they get access to the appointment… period.