"Potential output" is a probably harmless, but really dumb phrase. When people talk about "potential output" they're not actually talking about the output potential of the economy. They're not even really talking about the sustainable output potential of the economy.
We could not let children play, only teach them things essential to production in school, and have them work part time through their childhood and into adulthood. We could cut all higher education not relevant to production and put those people to work too. And we could make paying someone for childcare mandatory and force all parents to works.
We would produce a lot more than we do now. What's more, it's not like we'd be working people all that hard. It would be a dreary life, but it wouldn't be anywhere near subsistence living in the way Malthus envisioned it.
What we mean when we say "potential output" isn't equilibrium output either, because we know there can be high unemployment equilibrium (doubt it? google "employment to population ratio" or "NGDP").
So its certainly not a term you ought to take with a literal mindset. It's also not a term that you ought to take with a Walrasian mindset.
When we talk about "potential output" we're really taking a Keynes/Nussbaum/Sen approach to things. We're talking about macro relations but we're also making a general statement about human capabilities and the good life. What we mean is that at a particular employment rate that we feel comfortable and a particular level of leisure and pleasurable consumption that we think is reasonable, the economy could consistently produce X level, and we're going to call that "potential output".
It's a hugely value-laden concept. I only think that's interesting because it's often discussed as a technologically determined magnitude. Long run aggregate supply is often treated as a question of engineering - it's aggregate demand that's supposed to be all about hopes and dreams.
But that's really not true - even in the more circumscribed Walrasian sense.
This is not a mark against it, of course. It's just important that we restrict potential output to equations describing peoples' decisions, and keep it out of production functions, budget constraints, etc.