What are some signs my startup may be "too niche" to ever become mainstream?

I'm told better to build something at least a few people love, rather than something many people simply like, as it's easier to scale a loved product to the masses than it is to get even a few people to adopt one that they merely like. My concern regarding this approach is that I worry my product may not have even the potential to become mainstream (i.e. ability grow out of it's user base of early adopters to the early majority). What are some good ways to gauge whether a product that's loved (albeit by few) has the potential to grow outside of it's initial (somewhat niche) market?