There's a little trick I use to guess someone's fitness at a glance. It doesn't work with everyone at all, but I have learned you can't just go by size or visible flab. What seems to work is glancing down at the calves. You look at how big the calves are relative to the rest of the body - and in my experience, that does a remarkably good job with skinny people, fat people, and everybody in between.
Yes, skinny people can be out of shape. I know, I know - it's the appearance that matters, so often, but there's so much more to fitness than size.
Fact of the matter is that most forms of aerobic exercise - and the most intense forms of exercise, period - involve moving the whole body, and that comes down to actually using your leg muscles. Some guys work out their upper body heavily without getting into shape - but very few people, aside from dancers, work heavily on their calves specifically. If they do calf-specific exercises regularly, odds are they're doing exercises for most parts of their bodies.
And that's why it works so often. Doesn't work for someone paralyzed from the waist down, or with muscular atrophy in their legs, but for most people, overall level of physical activity is tied to calf muscle use. Most people who sit down all day and don't walk much are desk-bound and out of shape.