I don't want to harp on camera size to the point of making everyone sick of the subject (too late already?), but I do think it's a significant factor in how satisfied you'll be with any given model. A predictor of happiness, even.
To reiterate: camera size is a matter of preference. There's no right and wrong, and I don't mean to suggest or imply that there is. It's up to you to decide for yourself. I like smaller cameras myself, but I'm advocating, not arguing.
Ansel Adams's habitual rejoinder to the question "What camera do you use?" (a question he must have gotten incessantly) was, "The biggest one I can carry!" And that's probably still a good rule of thumb—buy the biggest camera, not that you can carry, but that you will carry—the biggest one you'll want to deal with, the biggest one you'll feel comfortable with—and you'll probably be on the right track.
This first comparison is almost a little silly—but it's what got me started here, because these two cameras happened to land near each other on the kitchen table one afternoon. It's John Camp's Nikon D3 and AF-S Nikkor 24–70mm ƒ/2.8 (which started its journey home today—thanks again, John) next to the Pentax ME Super and SMC Pentax-M 50mm ƒ/1.4 that my son uses for his high school photography classes.
The comparison's an unfair one, even given the disparity in vintage—the Pentax was a consumer model, and the Nikon has capabilities that the designers of the smaller camera would hardly have dreamed of. A fairer comparison would be to an LX with a motor drive attached, but my LX is currently not presentable due to a broken prism (don't ask). The ME Super was famous in its day for being miniature—Olympus had made small SLRs all the rage in the early '70s, and then Pentax did them one better. (Nikon made small cameras too, but, similar to GM making small cars at the time, its heart wasn't really in it.) The smaller camera above isn't inferior to the newer one in every single way, however. Its lens is faster, and, oddly enough, the vintage Pentax's viewfinder is considerably larger and at least as bright as the Nikon's, although its coverage is not as good.
The size differential between this D3 outfit and the D700 with the AF-S VR 24–120mm ƒ/3.5–5.6G is still considerable. The smaller camera of these two is decidedly closer to my own personal happiness threshold (again, no value judgment for others implied). Even with the largish lens, it's very ergonomic and comfortable in the hand. (Carl is sending me his AF-Nikkor 35mm ƒ/2 to try on the D700, too, which will probably make me like it even more.)
"Bragging rights" is actually a perfectly good reason to buy any particular camera, I think. I'm having a bit of trouble imagining any other rational, objective reasons why any ordinary non-professional would possibly need the D3 over the D700, but one reason is that some guys simply like to own the biggest, fanciest, fastest, most capable, and most expensive of anything—and I don't think there's a thing wrong with that. Photography's a hobby, a pasttime, a pursuit, a passion—why wouldn't you buy the camera you really want, as long as it doesn't antagonize your spouse or come out of your family's grocery budget? The fact that you might not really need it isn't in itself a reason not to get something. Even if it doesn't make the most sense. Still, the D700 is a much more logical choice for the huge majority of photography enthusiasts, and, I imagine, even some kinds of pros.
Next, here's the D700 outfit alongside a Pentax K20D with a "normal-sized" (i.e., non-pancake) prime lens, in this case the DA 35mm ƒ/2.8 Macro Limited. The K20D is my current personal standard for the highly technical designation "juuuuust right" in terms of size and weight. It's comfortable both on the shoulder and in the hand.
Finally, just for fun, here's the K20D outfit next to an old Canon Canonet QL-17. The Canon rangefinder was a "compact" in its day, but it's still quite a bit larger and heavier than the average pocket digicam of today (and, actually, not that much smaller than the ME Super in the top picture). I'm sorry I don't have one of those—a small digicam, I mean—on hand to compare to the Canonet in turn—we could keep going downward in these size comparisons like one of those nifty charts that compares the relative sizes of moons, planets, the sun, and bigger stars (I know where I come down in that size comparison too—I'll take a nice, wet, blue-green planet about 8,000 miles in diameter any day. When something's the "right" size, you just kinda know it).
Featured Comment by Jeffrey Glass: "Mike, Do you have access to a Hasselblad or Bronica SQ or similar medium-format camera? With any decent lens, it would be interesting to set it beside the D3 to add to your size comparison, especially since there is some reason to think the two are close to each other in terms of resolution and image quality."
Mike replies: Jeffrey, the D3 really did go back to John this afternoon—this is the best I can do for you (D700 and Bronica SQ-A).
Featured Comment by John Camp: "I also agree. I got the D3 two weeks after it came out, and three weeks before one of those "trips of a lifetime" things—I was embedded with an assault helicopter unit in Iraq (as a writer, but I also wanted to take the best possible high-ISO camera with me.) When the D700 came out, I felt a bit swindled. I'd paid a lot more for a much bigger, heavier, and in my estimation, less desirable camera. When I was flying in Iraq, I had to carry body armor at ~30 lbs and a helmet. The three huge Nikon zooms and the D3 didn't make life any easier. (Though they made some terrific twilight shots.)
"As Mike said, Nikon's heart might not be in a small camera—they've always gone for "the most." But I think they really need to get serious about smaller ones: they would be well-served to shrink even the D700, and then equip it with three pro-level ƒ/4 zooms. The sensor in the D3/D700 is good enough at high ISOs that it'd be worthwhile giving up one stop on the zoom to get smaller lenses. Also, I'm a fairly large guy, and if the body/lens combinations are a burden for me, they must really be a problem for all those 110-pound female PJs you see around.
"Just as gamers tended to push computer development, I think PJs push camera development—they always want the most for the most extreme conditions. But the D3, and the 1DsIII, border on the ridiculous."