How do I explain to my coworker that I do not want to ride with him without hurting his feelings and/or our working relationship?
In order to avoid hurting his feelings or your relationship, you first need to make sure he understand that you respect him, and that your complaints are not a personal attack, but a problem you have which conflicts with his standard operating procedure.
Depending on your own personality, you might even try to downplay it as a "it's not you, it's me" but that doesn't show adequate respect for yourself, and can still damage your relationship.
There is a conflict, and in the same way you'd work with a client to resolve a project conflict, you need to work with your coworker to resolve a working conflict. This involves agreeing on the requirements, investigating the resources available, and agreeing to a solution.
You aren't going to be easily be able to say, "I'm never ever riding with you again" in a respectful manner. If you do respect him, then if he hears your complaints and promises to drive in a manner meeting your needs, then it will be disrespectful if you do not take him up on his offer. If you don't trust him, then you will have trouble telling him without also explaining that you don't trust his word, should he offer to drive better. Since you didn't speak up while he was driving (and that may have been a good choice), then you should clear things up, the sooner the better.
I'd attempt to start off similar to the following:
"John, you and I are supposed to go to the data center next week. The last time we went you gave me a ride since I don't have a car. While I appreciated the ride, I found I was very uneasy at the speed you drove, and how frequently you took your eyes off the road to interact with your phone."
You've stated the problem and why it's relevant (upcoming trip). At this point you should give him a few moments to process the information and respond. You could simply barrel on with your proposed solution if you have chosen one, but if you respect and value him and his opinion, you should give him a say. Keep in mind that this is a two way road, and he may have something to say about how he felt the trip went - for instance if you were silent the entire time and he was bored driving.
From there you can discuss potential solutions. Again, if he offers to drive more slowly and to keep his eyes on the road, if you do respect him you'll end up taking him up on that offer, and during the drive you'll feel free to speak up if he's failing to manage his speed and attention. You'll probably want to engage him in conversation so he doesn't get bored driving with a silent passenger.
Only if you two are unable to resolve the conflict should you get the boss involved.
You didn't ask for ways to avoid the conflict altogether, but if you'd like to not have to bring it up at all, then simply tell your coworker and boss that you have a few things you need to do near the data center, and as such you'll be renting a car, meeting him there, and then staying behind after everything is done. It'll cost you more than the free ride from your coworker, but if it's a once a year trip, then it's probably not a big deal.