I've worked with some people who said it's ok to use them as a reference, but I haven't spoken to them in 4 years. Is this normal?
Is it typical? Maybe. Is it optimal? No.
It's not optimal because you want your references to speak from the heart when prompted by a hiring manager. If you stay off of their radar screen for too long, your references may forget specifics about why they enjoyed working with you.
[Should you simply] send a brief e-mail next time you want to use them as a reference, making sure it's still ok?
Yes, but asking someone who you haven't spoken to in four years to act as a reference is not an ideal situation.
Let's review your original question.
How do you stay in touch with people you use as references?
By staying interested in them.
- Every now and then, send them an email.**
- Ask "How are thing since we last spoke?"
Recall what you last spoke about and ask if anything has changed.
When they respond, make a mental record of what they say.
- Track how things change/evolve in their personal and professional lives.*
- And plan to stay interested in them in the future.
** "Every now and then" should be interpreted to mean no less than twice per year. But you shouldn't be fixated on the number of communications and it shouldn't be a chore, Also, asynchronous electronic communication (i.e. email) is, by definition, an impersonal mode of communication, but it's also less intrusive/disruptive than a phone call. More personal modes of communication like a phone call or a lunch date are preferable, but not 100% necessary if circumstances or relationships don't make it possible or convenient.
* Again, your relationship with the reference may be such that you don't trade details about your personal lives. If so, that's perfectly fine, but if you do, then you should definitely incorporate these into your semi-frequent communications.