Content-Aware Fill in Photoshop CS5?

Adobe has provided a "sneak peek" of a future Photoshop feature called "content-aware fill," a nifty new feature that evolves the healing brush massively. Take a look:

-

(If you can't see a video window above, try going here.)

The launch of Adobe's Creative Suite 5 (including Photoshop) is slated to stream on Adobe TV at 11 a.m. Eastern Time, Monday, April 12th. If you'd like to see it, you can sign up with your Adobe ID at the CS5 launch site.

Michael R. Thompkins at Imaging-Resource cautions that Adobe hasn't explicitly promised that content-aware fill will be a feature of CS5, a warning I hereby pass along to you in a weak attempt to keep egg off my face. Will it? Won't it? I've always disliked trying to report news before it happens. Tune in on April 12th to find out.

Mike

Send this post to a friend

Note: Links in this post may be to our affiliates; sales through affiliate links may benefit this site. More...
Original contents copyright 2010 by Michael C. Johnston and/or the bylined author. All Rights Reserved.

Featured Comment by Guy Batey: "Amazing, and slightly depressing at the same time."

Mike replies: My feelings exactly. At first I thought, doesn't this belong in Illustrator?

Featured Comment by skinnyvoice: "Wow, I find the relentless shifting of responsibility for content from composition to post-processing more than a little chilling. I'm okay with darkroom tweaking, dodging, burning, spot removal and their digital equivalents, but a 'content-aware fill' is a step too far from my concept of what looking through a viewfinder and photography is all about. But I'm probably not the target market anyway (photographically speaking—I use PS almost daily though)."

Featured Comment by expiring_frog: "For those interested, the details of the algorithm ("PatchMatch") are described in this research paper [link is a PDF download —Ed.] (To the best of my knowledge, this is what's being used.) The original PatchMatch video from last year's SIGGRAPH is perhaps even more impressive than the one in Mike's post.

"Adobe has been on a roll recently transferring state-of-the-art research projects to shipping software in very short times. Two notable examples that immediately come to mind are Agarwala et al.'s Photomontage work in CS3 (see the link—while you've probably used this function for making panoramas, it's really incredibly versatile) and Avidan and Shamir's content-aware scaling [another PDF —Ed.] in CS4."

Featured Comment by John Robison: "'We have removed the lens flare.' What I find amusing is that with some of these CGI movies they add lens flare and aperture shaped ghosts such as you would see with multi-element zoom lenses on movie cameras."

Featured Comment by Robert Howell: "Amazing technology. But perhaps it's time to ask that re-touched photos now be marked unalterably in the code."

Featured Comment by David Bostedo: "I think everyone who's upset by this needs to realize that a large portion of photography today involves changes and manipulations every bit as large as those in the video. It's just that it's all commercial work, or digital art work, and not meant to be 'straight' photography. From that standpoint, this tool doesn't change anything, except making it a little easier.

"The 'easier' part, though, will likely continue to evolve the general public's expectations of photography. It used to be that most (non-commercial) photos were assumed to be pretty much documentary. I think there's a lot of doubt out there now. And we're heading toward a time when all photos are assumed to be modified, and you'll have to tag them so that people know when they haven't been greatly modified. I don't know if that's good or bad."

Comments

[erlik]

Yep. Not certain whether it will be present in CS5. Here's the official tentative statement: "an early glimpse of what could happen in the future when you press the “delete” key." (from Photoshop Facebook fan page, emphasis mine)

There was an interesting comment on Facebook: "Goodbye, watermark."

BTW, they are starting to irritate me. It's completely irrational, but couldn't they use a Windows version of Photoshop? I'm certain that Windows Photoshop users outnumber the Mac users.

[Paolo]

I think it would be simpler to be a little bit more careful when taking photos . . . I mean, if you don' t want the tree in the frame just move yourself and the camera to the left or right, if you don' t want the road in that landscape . . . go and take the photos somewhere else, if you don' t want the rubbish on the grass . . . walk around and pick it up!!!!
And then will see who has good photos because of photographic skills and who has good photos because of photoshop improvements . . . which don' t even require much skills!!
Paolo

[Jonlevi]

wow. effing wow!

[Eolake Stobblehouse]

Gaaaaawd-damn!!!

That almost must be an april-one joke. This is too good to be true. This must be the best new feature since layers.

[jon]

This tool has been available for the Gimp for a few years. Granted it doesn't seem to be as fast as PS's implementation.

http://o3.tumblr.com/post/470608946/photoshops-caf-content-aware-fill-unbelievable

[Alessandro]

Wrotniak, 2007:
http://www.wrotniak.net/photo/misc/geecam.html

[Richard Kevern]

Certainly a very nifty feature. Perhaps Adobe could augment "Content Aware" to automatically delete inferior images. Even better, perhaps they could build this into an Adobe DSLR so we would only be able to take Award Winning images. Then the mode dial could include "HCB" along with old favourites such as "Mountain" and "Sports".

[Julian]

If Adobe choose not to include it after all the build-up, they could very well have a riot on their hands... ;^)

[Bahi]

It's going to become a staple of all advertising photography.

I hope it doesn't lead to our becoming tried of simple, clean compositions but I suspect there'll be a bit of that.

[Nicolas M.]

Hum,

Technically wonderful, artistically awfull...even creepy.

You just don't need to create a great photograph,
It's another step to kill the photographer's mind and thought process. Photographers are not artists anymore even not craftmens. They even don't need to Think!

It will be a great tool for PUB and FASHION...great, there is lots of money to make with this tool.

Please we are waiting for a new age, with the artists as witnesses of their time.

If the world is awfull SHOW IT! Photoshop tools can't erase what's wrong around us. You live in this world or try to survive!

Nicolas. (in nonoptimistic mode, sorry for that)

[Till Scheel]

That realy looks amazing. This new tool would fix my biggest problems in daily work with photoshop.
Expecially since I stich a lot of panorama-shots this could be extremly usefull. I am kind of skeptical about generating large parts of the landscape, but for skies it should work pretty well.

[Dirk]

It's interesting that elements of reality in photographs have become 'problems'. Perhaps photography itself is a problem. Mirror, mirror on the wall...

[Stan Mason]

The performance of content-aware fill appears impressive in the small video clip but I'll reserve judgment until I download the trial version.

I'm not involved in commercial photography, so I'm not interested in removing/revising large sections of a photograph at the request of a client.

What I am interested in, though, is how well this will work for removing long hairline scratches from black and white film scans. How well does the filled area match the original in sharply focused areas? What does it do with the great bokeh from your favorite lens? What about grain structure, will it match the original and will it blend in naturally?

These questions are not just for pixel peepers. If you're printing your photographs large for display in a gallery setting, people will and do get close to the print and look at the details. If you plan to do folios of the type promoted by Brooks Jensen at LensWork Magazine, the prints will be viewed handheld which invites close scrutiny. Very small flaws will be noticed.

[Adam Isler]

Saw this video a few days ago. And while I hungered for content-aware fill for my own (entirely high-minded) purposes, I was in awe of the potential harm to any type of credibility for photojournalism.

[Mike T.]

I watched the demo last night and I am here to tell you it was impressive. What I liked about CS-4 was the ease of use of the Panoramic feature in either Bridge or Photoshop using Raw. I can take a series of three or four pans and get an almost perfect stitched and blended scene from those now but of course this is with some loss of foreground and sky. The use of that content aware feature will really help in doing digital darkroom fixes on those it seems. I wonder what the old Masters of Photograpy would think now...Wow!

[John Doty]

Perhaps in CS6, we won't even need to go out and take pictures. We can type in what we want and CS6 will generate a whole image for us. Perhaps it will be called the "Content Generation Tool". Hmmm, Today the Art Director want a picture of Elvis eating Sushi with Madonna at a roadside cafe in Sedona Arizona with the sun setting in the background. Click. Whirrrr. Image appears. Adjust tones. Adjust lighting. We're done!
Snarkiness aside - Content Aware fill does seem pretty interesting though.

[Luc Novovitch]

"a nifty new feature that evolves the healing brush"

Goodbye watermarks! Helloooo pictures lifted from websites!

[Jack Nelson]

That's a pretty impressive new feature. So I guess sometime around noon on April 12 the debate about what is real and what is not will heat up.

I do a lot of photo merging (stitching) and it will be nice for that, although I doubt I'll use it the way they did in the video; I'll probably crop some first. It'll also be real useful for advertising and art photos and a bane for photojournalism. I predict there will be a controversy within the year when someone uses this new feature to modify a news photo.

What I'd like to see is how this would work on portraits. Will this put Photoshop Disasters out of business or will it just increase the amount of poorly done photoshoping?

[Robert Howell]

Amazing technology. But perhaps it's time to ask that re-touched photos now be marked unalterably in the code.

[Jack Nelson]

Hey Mike, when will they come out with a tool to put trash INTO a picture? They could call it the trash tool or maybe the New Topographics tool or the Post-Modern tool. Hmm, I think this could be the start of a new project...

[Karl]

In a word, that is awesome! I'm a photoshop newbie still so I think who much time it would take me even try to do what that just did. And I'm sure it's 100% 'cleaner' than my edits too. I like it. A lot.

[Emile Sprenger]

Hope this version will finally give me image enhancement like those CSI guys have ;)

[Dennis Ng]

Just brought the few books on Jim Marshall and then you give us this.

What now. Is there any need of a photographer any more?

Image processing to this level can it be still call photography.

WIth advance in AI plus digital capture, do you think that in 10 years' time we would have people using multiple RED (3D by then that video a concert and let the computer generate prints with a Jim Marshall looks as if he is "using" a Leica M4 35/1.4 Kodachrome. You may not aware of that until you notice that it cannot be made by him as he is dead when the concert is being held and no lab can process Kodachrome any more since 2010!

[mwg]

Eh. Doesn't excite me.

What I do want to see is something that should be pretty straightforward. Adobe already allows you to run Photoshop and Lightroom on two different computers. However, you can run Lightroom on a Mac and a PC, but you're stuck with one platform or the other with Photoshop. I want to see them get rid of the cross-platform licensing restrictions for Photoshop.

[Jim Hart]

I'm holding out for CS6. With that version Rumor Control has it that you don't need a camera, you just hold the PhotoShop software box up to the scene. When you get home you lay the box on your printer and your photograph (perfectly cropped, framed, and exposed) prints out on archival paper.

[Bernie]

That is just... I'm lost for words in admiration...

[David]

If it's not included, it's misleading IMO.

1) Why else distribute a "sneak preview" video so close to the CS5 launch? "Sneak preview of CS6, but we want you to buy 5 next month!" -not.

2) This is reinforced when you visit the CS5 launch site linked above. The first sneak preview thumbnail (again, presumably of CS5 …) is of the automagical fill feature.

[John Hall]

"Any sufficiently advanced technology is indistinguishable from magic." Carl Sagan
I think that quote is appropriate here. Considering that science fiction (often) predicts a dystopian future, I wonder what all of this means for the art of photography as the future becomes the present.
Off to take some pictures.
John

[sporobolus]

it seems the main point of this video was to get people to treat Adobe's announcement of a forthcoming announcement as if it were actual news; everyone and her nephew is going on and on about this artificial news, and Adobe hopes the hype crescendo is so big that this single (unpromised) feature will punch through today's cloud cover of thriftiness.

personally, i wait until i have a rational basis to do my upgrades; my rational basis for this one will probably be that the old version i'm using (CS2) doesn't run well enough on Mac OS X 10.6 (to which i'm in no hurry to upgrade either); skipping every other version or so has saved me thousands of dollars in 20+ years using Adobe products.

[Bill Bresler]

Except for handling raw digital files, I could still use Photoshop 5.0...

[Guillermo]

GIMP has had the same feature through a plug-in since years...

http://o3.tumblr.com/post/470608946/photoshops-caf-content-aware-fill-unbelievable

[Steve G, Mendocino]

I've got a panorama done handheld where I could sure use this in one corner...

Guess I'll be paying my Photoshop tax promptly this year.

[Another Mike]

"Any sufficiently advanced technology is indistinguishable from magic." - Arthur C. Clarke

http://en.wikipedia.org/wiki/Clarke's_three_laws

[Andrew]

Put me in the doubting Thomas camp. 20+ years of seeing CAD software dog & pony shows have me just a little bit jaded. Until it works on my computer with my pictures it's vaporware.

[Mike T.]

Update from my earlier post. CS 4 has some content aware scaling in the extended version. I just tried it out. Works well when applied to a good panoramic picture set. I would guess that the upcoming CS-5 has an improved version to that technology...wow..Technology is wonderful. Sure loved reading some comments on this. Too bad we did not have these kind of tools in the old darkroom days..
Mike T.

[Henry]

This makes me want to go back to film. And to request the contact sheet for any submission.
Oh well...
E.

[David Bennett]

I am eager to try the content aware tool to build back parts of images that I have previously erased.

For example, one could take a portrait of someone and erase the top of the subject's head in Photoshop.

Then one could build it back with the content aware tool.

The possibilities are endless.

[Kurt Shoens]

I'm saddened by the "imminent death of photography as we know it" comments. If you say people will lie with this new technology, how would you lie with it? I claim that an honest person won't be tempted and a dishonest person's had the tools all along.

Personally, I've lied with Photoshop (innocent little lies, really) but I've lied more convincingly with composition, crop, and lighting.

To those who say features like this remove artistry from photography, aren't you selling the art of photography short?

[Stephen Best]

This is a potentially very useful feature and may finally warrant the "just" in client's requests to "just fix this little bit here" or "just take out this". Though it normally takes Adobe a few releases to get new features implemented properly. We'll have to see what they screw up in the release to decide if CS5 is a net gain.

[Tyler]

So...were you the people who moaned and growned, howled and ran away when Bob Dylan released "Subterranean Homesick Blues"?

Just wondering.

Did you know that the 35mm camera (and even the 120 rollfilm cameras) were condemned as 'not the tools of a serious photographer' in the way-back when?

Maybe we should just sing a refrain of Dylan's "The Times They Are A-Changin'"...and keep on truckin'.

Cheers!

[expiring_frog]

John Doty wrote:
"Perhaps in CS6, we won't even need to go out and take pictures. We can type in what we want and CS6 will generate a whole image for us. Perhaps it will be called the "Content Generation Tool". Hmmm, Today the Art Director want a picture of Elvis eating Sushi with Madonna at a roadside cafe in Sedona Arizona with the sun setting in the background. Click. Whirrrr. Image appears."

You are so going to hate this.">http://www.ece.nus.edu.sg/stfpage/eletp/Projects/Sketch2Photo/index.htm">this.

[Tom K.]

Gulp......Holy WoW!

[Paul De Zan]

I'm WAY past the handwringing stage about manipulation technology. There hasn't been anything remotely like "truth" in photography since at least 1980; the first device I used that could reasonable do things like add another person to an existing portrait scene in a totally undetectable way was a dedicated workstation the size of a very large production-volume copier that cost 150,000 much-bigger dollars . Now you can damn near do it on an iPhone.

Progress!

[Petersson]

Another reason to stick to film. The video was scary, thanks for sharing.

[PLR]

I can already see the accompanying ad:

"Create stunning images with Photoshop CS5 and Content Aware Fill (photographer optional)"

[Eolake Stobblehouse]

"What I find amusing is that with some of these CGI movies they add lens flare"

No kidding. Have you seen the new Star Trek? Not a bad movie of the kind, but there's added lens flare, often very dramatic, in virtually *every* damn shot of the whole flick. Talk about over-done.

[Eolake Stobblehouse]

I don't really get all this "it's creepy" talk. If a pole is in the way, you move a bit so it's out of the frame. If a wall is there so you can't move enough, you can remove it in Photoshop. I don't see any major difference.

[András Kalmár Nagy]

A good photograph is one that you find beautiful. Does it matter how it was made?

Although from now on, a photograph is based on reality as much as a painting is. It's filtered through the eyes and creative intentions of the artist/photographer. But problems do arise when something like this is presented as reality.

[ctein]

OK, if this works as well as I and Stan hope, I'd pay for CS5 just for this tool. I'm just gobsmacked by the uses I could put it to for photo restoration and repair.

Especially if content-aware delete will work on discontinuous selections. I've got methods for semi-speedily selecting damaged and missing areas in a photo. If I could even partially automate the task of filling them in with surrounding material, it would hugely speed up my work.

And I do enough panoramas where I have to trim a smidge more than I'd like off the margins to get it square, or else spend a very long time cloning in otherwise irrelevant minutiae, just to get an edge straight.

Yeah, the examples in the video are way over the top, by my standards, but that's good thing. If it'll do a semi-credible job on changes that outrageous, there's good chance it'll work really well on my modest alterations.

This is VERY exciting.

(And, no, it doesn't really belong in Illustrator at all.)

pax / Ctein

[David A. Goldfarb]

Pretty much all the discussion of this on the Alamy stock agency user forum is about its use as the watermark removal tool--

http://www.alamy.com/forums/Default.aspx?g=posts&t=7690

[erlik]

Hah, expiring frog, I wonder whether their Ariel Shamir is any relation to the Adi Shamir of the RSA cryptography fame. :)

[Mike Johnston]

"A good photograph is one that you find beautiful. Does it matter how it was made?"

Andras,
I'd take issue on both accounts.

Mike

[David Dyer-Bennet]

This looks really useful; could save me considerable time, and save the professionals doing ad work even more time. Remember, people, this doesn't really let people do "new" things, it just makes it easier. Advertising is ALREADY doing all this stuff, they're just doing it more slowly.

Might have to make two Photoshop upgrades in a row (I've already got CS4, so by history I probably wouldn't have bought CS5).

[sven]

How long will it take until we all have electronic glasses (with builtin cameras and direct-to-retina projection) that replace our vision by a realtime-photoshopped ideal-world representation of what's around us?

(And who decides what's ideal?)

[expiring_frog]

@erlik: Could be, but given that there must be thousands of Shamirs in Israel, what are the chances? :)

[Jon Wilson]

What I'd like to know is what was the spec of the Mac/PC that was being done on and how much would it cost?

[Nick]

"Amazing technology. But perhaps it's time to ask that re-touched photos now be marked unalterably in the code."

Heh. Yes, because DRM schemes "marked unalterably in the code" of media and software have proven so successful at preventing piracy, and professional photo houses see no need at all to watermark their pictures.

I'd expect that it would take less than a week from its release for the hacker community to put up a tool for removing any such "unalterable" markings.

[Robert Howell]

I have nothing against progress or the need for some to add or subtract elements for the images' sake. I just want a simple way of knowing if you have manipulated, and to what extent you have. A few lines in the image code could provide this and do a lot for photography and verity, the other half of the discipline.

[Mike]

"A good photograph is one that you find beautiful. Does it matter how it was made?"

Andras,
I'd take issue on both accounts.

Mike

I'm with Mike on this and have even blogged about it, I think. There is a power in being a mirror to the world that is diluted or removed by such approaches as these.

If I didn't know that I can make pictures without this amazing tool then I'd sell up now. This will have great commercial use, helping people make money more quickly by presenting even more fictive imagery - this includes the majority of current landscape imagery as well as advertising inho - and will derail thousands of earnest amateurs who (will be led to) fail to understand that they can achieve more of real value by choosing the simpler, yet harder, path of simply looking through the viewfinder and making photos.

[rant mode off]

Mike

[Richard]

I have a friend here in NY who claims to have used this new feature on a damaged photo of Mount Rushmore. He says the damage over Teddy Roosevelt's face was miraculously replaced with the face of Steve Jobs. -- Rich

[Robert Howell]

@Nick

Written as an integral part of the image-rendering code the image wouldn't render if the tagging code was removed, and it would travel with the image as it went through any permutations.

If you could find it that is. Open up an image file in a decent text editor and see if you can spot the ball.

The idea isn't to build a better watermark or DRM scheme, it's to merely have indicated whether this picture is or is not manipulated, and to what extent.

A modest proposal.

[Rana]

As someone who finds it beyond tedious to correct the pin-point spots that flaws on my sensor causes, and someone frustrated by the occasional lens flare ruining an otherwise wonderful image, I find myself intrigued by this.

The idea that you could remove whole elements from the image - trees, awkward phone poles, unwanted ex-boyfriends - is more bothersome.

But here's the thing.

The problem with photomanipulation is not with the tools. People have been altering photographs since the day they were invented, for artistic purposes, to deceive, for propaganda, whatever.

The problem is with the photographer. As an ethical photographer, I will not edit out the content of my images, only the flaws in the camera technology that ruined them. So the awkward telephone pole stays, but the specks on the sky go. They weren't in the scene I captured; they were introduced artificially by the camera, and so removing them with another tool seems acceptable to me.

If we want to combat the unethical manipulation of photographs, we need to focus on the manipulators, not on the tools they use.

But then, I'm not worried about people getting credit for doing something the "easy" way while I get ignored for doing the hard way - it's not about the public's perception of my work, but whether I can live with myself. I keep myself honest; I don't need software or other tools regulated so that I can't misbehave.

[Neuffy]

Simply because others feel that the "power in being a mirror to the world" as Mike [Note--not me, another Mike —Ed.] puts it doesn't mean that I have to hold it in the same level of respect.

To me, cameras are simply a way of getting a specific end result. If I could do it with paint, pens, pencils, a tablet or any other method (getting comparable end results with comparable effort and time investment), I would. Very simply, I cannot. That said, I'm uncertain of my personal identification as a photographer at all - I'm much more of an image maker who has simply failed to produce acceptable results (both end results and time/effort efficiency) with most other image creation methods such as 3D modeling and rendering or traditional artistic tools. Actually, if I could get the results I want from traditional or purely computer-based tools, I would. It would be much, much less expensive.

I actually truly cannot differentiate between various processes except from a technical standpoint. Sure, I'm interested in how a given image was created - but only so that I understand better (and can potentially apply similar techniques or understanding to my own image creation), not so as to change my appreciation of the image itself. The only thing I care about is the end result. If it took you 400 hours to create an image, that does not make it any better. If it took you three minutes of lazy work and the image is amazing - it's still amazing.

This tool just makes things a little bit faster and perhaps a little more blatant. It's all been happening anyway - only change is time investment and how much pre-qualification you need to be able to do it.

On the "unalterable code" issue: Anybody mentioning it here should drop it. It's effectively impossible. I'm of the opinion that the best option is the expansion and improvement of services like TinEye for finding cases of infringement.

[Robert Howell]

It may be unlikely an organization tied to the hip of marketing is going to spend time and resources on a non-profit exercise such as a transparency mechanism, but it is not impossible.

Just last week most of us here would never have thought it possible we could wipe out an entire tree with a mouse-click, or rehabilitate a desert. But there it is. Possible.

It's not technique that limits us; it's our application and incentive, and our motivation.

Given that photography is much bigger than it's commercial application, and given that most of us want to see photography flourish in all its potential, is it not time for us to demand the software manufacturers exercise some ethical behaviour by investing in checks and balances against a one-sided application of our medium?

All it takes is a little transparency.

It's happy hour out here on the coast and as I'm a willing participant in local tradition I shall now lower the podium and switch hats BUT if anyone wants to discuss this further please write me at [email protected] … this address will remain active for 3 days then it'll point to oblivion so anyone interested in rallying to the flag give me a ring by then and if I like the sound of your voice (don't say Viagra) I'll give you a real address so we can continue the conversation.

Cheers.

(Thanks Mike)