How should I mention any experience I have with children that isn't
formal work experience on my CV/resume?
Should I mention this on my CV?
Your CV/resume is generally designed to showcase your relevant work experience and education. Since you "have no formal work experience in working with children", it will be tough to fit in your interactions with nephews and nieces.
You could write something in an "Other Interests" section, but this is rather weak. Wouldn't you imagine that every other candidate could write the same? Pretty much everyone has young relatives or neighbors and could write an "I like kids" paragraph.
I think your best bet is to expound on your experiences with your nephews and nieces in your cover letter, where you can be more conversational, and less formal.
Did you do something with them that mirrors almost exactly the kind of work you would be doing? If so, this is the place to explain. Did you do something unusual with your nephews and nieces that you think would make you unique among other candidates for the position? (Perhaps you babysat a bunch of them full-time for an entire summer while their parents were overseas.) If so, your cover letter would let you explain how valuable that experience would be in the position for which you are applying.
Whether you squeeze something into your CV/resume, or write about it more fully in your cover letter, make sure you tie it into the job. You want the reviewer to see that your experience would make you a good employee, not just a good Auntie or Uncle.