SSH logins can be limited to only certain users who need remote access. The ssh server is configured by the file /etc/ssh/sshd_config.
If you wish you can restrict remote access to specific users. For example
To restrict remote access to John and Karen to login add the following:
AllowUsers john karen
and restart the sshd service.