Like with anything, there's people who like mods and people who don't. You shouldn't restrict yourself to vanilla if you think you'll have more fun with mods. However, at the same time you shouldn't force your sibling to "endure" a heavily moded experience if that's not what they enjoy.
If you're looking to play with your brother, I suggest a compromise. Instead of completely changing the experience, start off with basic mods that add a couple of things here and there (basically building on vanilla experience rather than completely changing it like some of the huge modpacks do) and see if your brother will enjoy such an experience.
