I'm currently doing my Bachelor at EPFL, a top rated university for computer science (top 10-20 worldwide) but also very difficult. I was pondering whether or not I am wasting my time since I am planning on doing a Master's anyways somewhere, I may aswell go to an easier uni for my Bachelor and have more fun since I am young, then do my Master's at a good university. Or do employers care whether or not I completed my computer science bachelor at a top rated university in general?
It entirely depends on the employer, many will take notice. More sophisticated employers will definitely include it as part of the data considered when looking for new team members.
You should always choose the more academically rigorous and acclaimed path if you can. Part-time programs could be a good option if you need extra time for things like caring for a family member. Search for scholarships, provided by the school and other institutions to offset the cost. Downgrading your academic experience to have more time for goofing off seems like a bad trade-off over the long term.
Your undergraduate university and major are good determinants of your first job post graduation. Employers generally have target schools for various reasons - academic rigor, regional affiliations, research collaborations, etc., so you will be exposed to better/different opportunities depending on the school you attend (you can easily research this aspect at your university's career center).
You got into the university because you had the raw intellectual ability to succeed there - working hard (early) will prepare you for your first job and in subtle ways later in your career.
I think all the other answers make a good point, however it's also worth noting that (most of the time) attending a better/more difficult university will likely also give you a better education. So even if an employer doesn't care at all about where you went, the extra technical knowledge you gain may make a difference.
If the reason the degree is more difficult is because the content is more difficult or there is more content in general. Then I'd say it does make a difference.
A degree from a highly regarded university like the The Ecole polytechnique fédérale de Lausanne serve a lifelong purpose for you when you want a job.
It gets your job application noticed by human resource people and, to some extent, hiring managers.
It makes you stand out from the pile of applications and gets you the interview. When you go to the interview you can speak for yourself.
It doesn't matter whether it's a bachelor, master's or PhD. It works. If you want a long technical career this will help you.