That’s a fair point. That is certainly the ultimate implication of philosophical idealism. I wrote this in my book One Simple Idea, a history of the positive mind movement:

“In the early eighteenth century, the Irish bishop George Berkeley sounded a transformative note in Western philosophy when he argued that material reality had no existence outside of man’s mental-sensory perceptions. What appears in our world is a result of our observation, Berkeley reasoned. Without a sensate observer, phenomena have nothing in which to be grounded. Berkeley’s insights gave rise to the thought school later called Idealism. Yet the Anglo-Irish philosopher stopped short of anointing man as the inventor of reality: there also exists, he insisted, a rerum natura, or fixed nature of things, of which the sole author is God.”