The most recent immigrant in my line came from Thuringia, in east central Germany, in the 1890s. They weren't particularly well off. The patriarch left first to escape conscription. He sent for his wife and kids but then decided a younger woman offered greener pastures. So when my great-great-grandmother arrived she was a single mother that didn't speak much (any?) English. That works out even less well in the late nineteenth century than it does today, so she wisely married another German immigrant (named Kuehn - I know a lot less about his origins). The Kuehns spoke progressively more English and got progressively better education, and it all worked out alright. They eventually married into a family that's been here almost four centuries (but who were no better off when they first arrived). They were always on the run from bad circumstances. The Joys running from English persecution in England. The Comeaus running from English occupation in Canada. The Luthers running from conscription in Germany. A couple branches were running from rotting potatoes. As far as I know nobody came here and contributed to American GDP by establishing a tech company (steam boats, railroads - you know). They came here and were poor and uneducated until eventually they weren't poor and uneducated.
None of them ever read: "Give me your tired, your poor, Your huddled masses yearning to breathe free, The wretched refuse of your teeming shore. Send these, the homeless, tempest-tost to me, I lift my lamp beside the golden door!"
For one thing none of them came through Ellis Island. They also were all here by 1903 which was when the plaque was added.
I worry about the discussion about immigration today. There's always tension with new immigrants who don't act like us, of course. So long as it's just "tension" and not outright hate I don't worry about this too much because although it's misguided, it's a natural reaction to new things.
But you'd hope that more liberally minded public intellectuals and even policymakers would construct a narrative (what Rorty talks about as "imagining" a new "social hope") about why we want the huddled masses.
Andrew Sullivan reports that this sentiment is not particularly common anymore. Quoting Don Hopkins: "In the corresponding paper, we show that it’s not just Democrats and Republicans who agree: it’s liberals and conservatives, those with and without higher education, the wealthy and the poor, those who report biases against other racial or ethnic groups and those who do not. When it comes to the question of the types of immigrants to be admitted, there is a hidden American immigration consensus, one that crosses party lines. From these results, it seems clear that Americans would be likely to support a more skill-based immigration system, such as the one employed at the federal level in Canada."