The more defensible complaint is that small states get a disproportional amount of votes so that it's not one man/one vote. This is a reasonable thing to complain about, in my mind. This is not a decision about the structure of an institution, after all. This is the actual election of an actual candidate. It's one thing to give a state two Senators no matter what the population is. It's another thing entirely to say that one person's vote counts more than another person's vote in the same election for the same Senator. And yet that's what goes on in the electoral college.
The less defensible complaint is that only a few states actually matter because of the winner take all system. This complaint boils down to a complaint that we don't have a popular vote for the president. The problem with a popular vote for a national office is that it would substantially threaten American federalism. Presidents would no longer need to be popular across a wide variety of states. They would cater more to interest groups not defined by states.
Who cares? - you might ask. Why should the president have to appeal to a wide variety of states if a lot of us don't live in those states? Because a lot of policy in a federation ought to be made at the state level, and if a president is going to pursue a particular program it ought to be one that a wide cross section of voters in competing jurisdictions (competing with the federal government in this case) approve of. The federal government has broad, vaguely defined, contestable powers. This creates a lot of overlap between what the federal government could do and what the states could do. If presidents are going to propose programs that might infringe on states you want people across a lot of states to be OK with that. You don't want a lot of people from a few states dictating the relative powers of all the other states.
Granted, you could flip this formula for making votes popular but still winner take all. You could also do it like the structure of the Congress and drop winner-take-all, but make the representation disproportional. This would work too. I guess I just like the idea of one-person-one vote. Explicitly telling someone from Wyoming their vote will count more than someone from New York seems a lot more problematic to me than telling everyone in Wyoming that they have to get together and agree on who they're going to pick and everyone in New York that they have to get together and agree on who they're going to pick. That second way of doing things seems like its valuing everyone equally, but structuring the election in a way that acknowledges that we are "not a nation, but a nation of nations".