Document everything. When he assigns you a task verbally, send a follow-up email saying "As per our conversation, I am taking on the responsibility of [xyz]"
If you start to get behind, send emails before it becomes an issue. Likewise with additional tasks.
"As per our conversation, I am taking on the additional responsibilities of [xyz], however, I am concerned that since my attention to [ABC] is taking up the majority of my time, this may put me behind schedule. Please advise as to which has top priority".
Similarly, if a task that has put you behind schedule is then taken away from you after you are already behind schedule, document that as well with an email.
"While I appreciate the task of [xyz] being reassigned to my coworker, as I expressed in my concerns on [date] this has already severely hindered my ability to complete task [abc] on time"
An email trail prior to things going off the rails is the best approach