借りる vs 貸す — which one am I doing when I ask to borrow a pen?

I want to say 'can I borrow your pen' but I keep flipping 借りる and 貸す by accident. Is there a trick to remember which direction each one goes?
Post
沐學 Mizuru