Mice tend to use one area of their cage as a bathroom. They will still mark other areas of their cage, however, and will also urinate or defecate when being handled. It’s difficult if not impossible to teach a mouse not to go to the bathroom when being held. Make handling sessions short to give your mouse time to relieve itself in the cage. This reduces the likelihood of it will going to the bathroom in your hand.
You can try litter box training your mouse by putting a small litter box made for small animals or a shallow dish in the corner of the cage where it normally relieves itself. Put paper-based litter inside the box, and put some of the soiled bedding from that corner of the cage inside it so the mouse gets the message. If you are lucky, it will go to the bathroom in the box most of the time. Don’t expect it to be perfect about it, however. Your small pet will definitely have accidents.
If your mouse relieves itself while you are holding it or outside the litter box, don’t punish it. This is natural behavior for mice, and something you must learn to live with.