My former co-worker keeps asking me for passwords. A few I remembered
This was your first mistake. You should have said that you didn't remember them the very first time you were asked. And you should have repeated a similar phrase every time the same request was made.
I think from now on my answer should be 'No.'
Now your answer should be: "I've given you a notebook with all my passwords, and I've told you everything I could remember."
Be a broken record if you have to. It's not that difficult. Just don't adjust your response and don't try to justify yourself to try to gain his approval, otherwise, his requests will keep on coming.
But at the same time, take him up on his offer to give you a reference before his offer evaporates. Ask for a LinkedIn reference and offer to write a LinkedIn reference in return.
Worst case scenario, his LinkedIn reference is not that good and you don't show it on your profile since you have the power to hide it if you don't like it. Best case scenario, he writes you a pretty good LinkedIn reference. Also, if any potential employer calls him, he will feel obligated to stick to what he wrote on your LinkedIn just to be consistent.
(claimed to be surprised what happened, I actually think he was in on
it but I have no proof)
So who cares if he was in on it?
Are you really telling us that if the situation had been reversed you would have told your colleague about the upcoming replacement?
And yes, he could be lying to you about his prior knowledge. But again, people tell each other white lies all the time. I'm sure you have as well.
I understand the hurt of being laid off, but do try to let go of that bitterness. It doesn't do you any good to blame him. Take him up on his offer and ask for a LinkedIn reference, but do tell him that you don't remember the passwords, which is the truth anyway. And to make the situation reciprocal, offer to give him a good LinkedIn reference in return.
But if he asks you anything else that's work related, just spell it out for him. "If this LinkedIn recommendation is contingent on me remembering a three months old password, then don't write one for me, because I can not give you what I don't remember". Or if he asks only about work, say: "If this LinkedIn recommendation is contingent on me consulting for free for a former employer three months after I was laid off from that employer, then please don't write such a recommendation for me."
And if you're still have trouble saying "No", I'd suggest you read this book When I Say No, I Feel Guilty by Manuel J. Smith. Personally, that book changed my life.