SSH provides a very flexible way to configure how it can be accessed.
The configuration is done from the sshd_config file. On fedora it can be located in /etc/ssh/sshd_config but for other OS it may be under /usr/local/etc/sshd_config for example. If you cannot find it there try to use locate or find commands.
Once you have located the file the rule of thumb is to create a backup. You can always delete it later on, but just in case you mess up you can easily recover. I personally do a backup everytime I change the configs so that I have the full chronology of my changes.
Now to the editing part. Open the file in your favourite editor. Now, you may already have some configurations for user access so the first thing is to look for the following directives: AllowUsers, DenyUsers, DenyGroups, and finally AllowGroups.
Each of those (as should be apparent from the name) allows or restricts users or groups from using ssh service.
The syntax is quite simple:
<Directive> <user1>[@<host1>] <user2>
So for example AllowUsers directive, provided that you have a user named johndoe may be used as following:
Note: by providing this configuration you are restricting the SSH access only to user johndoe
Another variation might be
AllowUsers email@example.com marrysmith
This configuration restricts the SSH access to user johndoe only from IP 184.108.40.206 and user marrysmith from any machine.
So you get the idea. One important thing to note: For the config changes to take place you need to restart the sshd. On fedora you may do so by issuing the following command:
service sshd restart
By the way you do not need to terminate your existing SSH session while doing so. In fact I would advise to keep the previous SSH session alive and open separate sessions to test the new configurations are correct before quiting (This might be your last chance to recover if you mess up).
Now that you had a little taste for it you can go to your console and type:
for the full documentation on the configuration directives.
Last word of advise: DO NOT EVER COPY AND PASTE COMMANDS STRAIGHT INTO TERMINAL/SHELL FROM THE ARTICLES do a man <command> to find out what it acctually does before that :-)