They had fled the disasters on earth, or rather their great-great-grandparents had. Some of those disasters natural, some civil. The Great Portal was the best option, the mars project was too expensive, the Portal mostly just manipulated the natural instabilities in the fine fabrics of space-time. Many didn’t even get that option.
They were told that Great Portal would carry them far through space, to a new world. They were also told that it was safe, which fortunately turned out to be true, not that those who made such promises had mentioned testing it.
They had arrived in a strange place with giant plants, and slow giant animals. It was a while before they even started guessing what had happened and where they were, even once they started to half recognise some of the flora and fauna. It wasn’t until halfway through the fourth slow year that they had mostly figured out the full story, that the Great Portal had transported them through time and had changed how they fit into both space and time. They had become smaller (approximately one third of original size as was later worked out) and passed through time faster (about twice as fast), hence everything else seemed both bigger and slower.
As their population grew and spread out, they fortunately managed to preserve most of the knowledge that they had brought with them, especially histories, remembering who they were (or had been), and some of the science.
Once they started seeing humans, or “big people”, and their settlements, they could guess as to when they had been carried to. The big groups of soldiers with plumed helmets and big square shields were a good clue.
The other important thing about finding big people, especially when their own population was growing, was the need to avoid discovery. The science that they had kept was useful for that, as it had been in coping with the agricultural challenges of the different relative time. The big people’s slow reaction times were also useful in avoiding their attention.
The third generation had started to see more variety in their populations than the preserved science told them to expect, even the more exaggerated estimates. The fourth generation saw even more, and with the communities continuing to spread out, the different populations were tending in different directions, some getting slightly shorter on average, some getting slightly pointy ears.
It wasn’t until this generation that some people were now suggesting that they were, or rather their grandchildren would be, the gnomes and dwarves of old human legend.