To understand why this happens, we need to know a bit about how colors work. Colors are created by the way objects reflect light. When white light hits an object, some of the light is absorbed and some is reflected. The reflected light is what we see as the color of the object.
For example, a red object appears red because it absorbs all the colors of light except for red. The red light is reflected back to our eyes, and we perceive the object as being red.
When you mix two or more colors, you are essentially mixing the reflected light from those colors. If you mix red light with green light, you will get a color that is somewhere in between red and green. This is because the reflected light from the red and green objects will combine to create a new color.
If you mix all the colors of the world, you will be mixing all the different wavelengths of light. This will result in a color that is very dark and muddy, because all the different wavelengths of light will cancel each other out.