The questions give you a box of a set of numbers with different values in the numbers box. Then you get instructions to manipulate the numbers and after completing the instructions it asks you for one of the numbers value.
That sounds like an excerpt from an IQ test. Unlike brain teasers, where once you know the answer they are dead easy, IQ tests test logical reasoning, pattern matching and short term memory under a time constraint. This is a pretty good indicator on how you will do as a programmer. It's not the only one, but one.
What do these results mean? Did I ruin my chances for a real interview?
Well, that is in the eye of your interviewer. We cannot tell what they were looking for, how they weighted the results with all the other indicators or how the other candidates did.
Should I study more tests like these? Before taking the test they said there is no way to prepare for it.
If it was an excerpt from an IQ test then indeed you cannot prepare for the test itself. Your brain will be able to do this in the time given, or not. You can take another thousand tests and it will not change. However, the process of getting the task to your brain, reading the instructions, understanding what task you need to do, is something that does get easier with practice. So just go on the internet, google IQ test and do a few for fun. You won't get a better IQ, but you will get better at understanding the instructions to random, unknown puzzles.