What management feature hurts you exactly?
The fact a Switch is (Smart-/Fully-)Manageable doesn't necessarily mean that must be managed in all its aspects (well, that's very basic, at least provide it with a IP static addressing and a Default Gateway...that's for management purposes only and let it - if it supports that feature - to synchronize with an external NTP Server...or provide an internal NTP Server if your network has one...so Switch generated logs are collected with corrected date/time): it can act as a "dumb" switch too...if no management is done at all.
The only thing you should care of - at this point - is to avoid to release (Web-/SSH-/Telnet- if active) login credentials to other non administrative IT operators you don't trust (yourself excluded).