The definition of ISO is based only partly in the physical sensitivity of film or sensor. ISO is a judgment about what exposure will produce the best results, on average, with a particular medium of capture. ISO is a compromise.
In 1955 Tri-X had an ASA (ISO's predecessor) of 200. In 1965 it was 400. The film hadn't changed, just the official judgement of what the best exposure was. Higher exposure of B&W film produces better shadow detail, but increased grain and poorer sharpness. Shrinking film formats increased the importance of grain and sharpness in overall quality. The manufacturers decided the emphasis was too much on shadow detail at the expense of those, so ASA values were redefined. Negatives need sufficient exposure to produce shadow detail while not overly compromising grain and sharpness. Slide films must record sufficient highlight detail without having a slide that's unacceptably dark. Many negative photographers routinely "overexpose" by around a stop*. Many slide photographers "underexpose" their photos. We're all aiming for adequate information at the end of the luminance scale where the medium is most likely to fail us. ISO doesn't require you make a certain exposure; it's merely a best guess. But when ISO consistently guesses poorly for a majority of users, it ain't doing what it's supposed to.
Digital sensors have problems similar to slide films; expose more than a certain amount and there's simply no detail. Unlike slide film, digital sensor's behavior in the highlights is not at all graceful. Once the pixel's potential well is full of photoelectrons, clipping occurs. In a purely linear device, that can happen 2–3 stops above middle gray (there are ways to improve that—many cameras do much better).
At the same time as the highlights are getting clipped, there is usually detail well into the shadows, albeit with increasing grain/noise. Cheapo digital cameras can capture a seven stop range; better ones 9–11 stops. In many cameras the luminance capture range is weighted too far in favor of shadows. Especially true as we notice loss of tonal separation in highlights a lot more readily than we notice it in shadows.
From the comments I read about digital cameras (and my own modest experience) I think the digital ISO is simply too low in many (not all) cameras, often by more than a full stop. Many of us have already learned to "underexpose" as normal practice to more usefully balance highlight and shadow detail. We just set our exposure compensation dial for –1 or –1.5 stops and leave it there most of the time. This needs to be institutionalized. Digital ISO is due for reformation to provide exposures that more usefully exploit the sensor's exposure range, balancing adequate shadow and highlight range. On my Fuji Finepix s6500, changing the current ISO of 100 to ISO 250 would be about right.
Both of these photographs were originally made in RAW mode by my Fuji FinePix s65000 camera. (Click on the images to see them larger.) The one on the left was made at ISO 100 with no exposure compensation. The one on the right was made with –1 stop exposure compensation. Both of these photographs were originally made in RAW mode by my Fuji FinePix s65000 camera.
The one on the left was made at ISO 100 with no exposure compensation. The one on the right was made with –1 stop exposure compensation. I made a curves adjustment in Photoshop to the image on the right, so that the shadows would have about the same brightness in both photographs, for comparison purposes.
The "normally" exposed photograph has blown-out highlights with little or no data in one or more channels. The "underexposed" photograph has good highlight detail and only a slight loss of clarity and separation in the shadows. So, which one would you say was correctly exposed?
There would be side effects. A good one is that you get more low-light capability. Instead of hand-holding at 1/10th of a second, you're hand-holding at 1/25th of the second. The bad is that your pictures become noisier. It's a balancing act, just as defining B&W negative ISO was a balancing act between shadow detail, grain, and sharpness.
But, digital image noise is already better than comparable speed 35 mm film in many digital cameras, and it continues to improve with each generation of sensor. Luminance capture is definitely not better than film in the highlights. I understand the current obsession among the digital newbies for low noise; I'm a fine-grain lover myself. But poor tonal placement is a worse problem that will improve less as sensors get better; it's more inherent in how the sensors map luminance values. We need a new ISO definition and we need it now.
*(By the way, this has nothing to do with the misconception that manufacturers overrate negative films by half a stop or so. That's mostly due to mistakenly metering off of 18% gray cards using ISO-standard meters calibrated for 12% reflectance.)