This is a good question.
Our brains begin to form interutero and continue to develop physically through ages four to six. As we are exposed to our sensory environment, our brains form circuits of synapses and neurons to interpret what we sense. This happens for all five senses: hearing, feeling, seeing, smelling, and tasting. The receptors from all these senses are connected through our nervous system to the brain. So the brain is hardly isolated; it is in communication with the outside all the time. If it's healthy that is.
Through evolution, the brains of mankind are predisposed to interpret some things pretty much the same way. Color is one of those things. Photons of given energy levels excite the cones in the eye, which send appropriate signals to the brain where they are interpreted as given colors. This doesn't always work that way; thus some have color blindness, and certainly what's red for some people is magenta for others. So the interpretation is not uniformly consisted across beings.
Images are not as predisposed in our genetic codes; so we learn them as we grow up from birth on. And to a point, we, our brains, can be fooled by so-called optical illusions. But in the early years our brains learn to interpret images using all five senses. Watch a baby, for example, as it is predisposed to shove everything it can get its tiny little hand on into its mouth. It is correllating what it sees with what it tastes, which helps to form a more complete image of the object.
There are brain afflictions that scramble images from what most would interpret them to be. Dyslexia is one such issue. Here, a reader will see a drow backwards, for example from how it is actually written on a page. And that looks perfectly normal to the dyslexic except for the fact, over the years, he or she has learned to interpret it as "word" like normal people. As I said, interpreting images is a learned trait.
Although there is much yet to be learned on how the brain works, we are getting better and better tools for studying that. We can actually trace the electro-chemical brain patterns within a functioning brain, to see what's working and what's not for given senses and bio functions like talkiing, thinking, etc. We can even trace the emotions like hate and love.
With the advent of stem cells research, we are also learning how and why some cells grow into muscle, some into nerves, and so on. And that will help us understand how the brain forms over the formative years.
There is a saying, "God is the the gaps." The gaps are the gaps in our understanding of nature and the natural order of things. So some people tend to invoke some sort of divine intervention wherever science and engineering can't explain something. But the gaps in science and engineering are getting smaller every day. We, mankind, are able to explain more of the why and how of things by invoking science, not some sort of god, every day.