Wow, another in an ongoing series of iOS 4 cautionary tales! Here we go: Some users are reporting that in iOS 4 when you first open your Photo Library, you may have to wait for the photo library rebuild.
We've seen this happen with the music library, but never photos. It is odd.
Obviously the rebuild times will vary. They will be based on how many photos you have. In one case a library of 3k photos took 3-5 minutes to build.
We have to wonder if this has to do with the addition of location services to Photos.
A number of people have even found that these previously stored pictures appear grainy, as if a lower-quality version is being used.
The recommendation is to delete and unsync the photos from the iPhone, and reset the phone. Make sure the volume containing your photo library is mounted, and using iTunes, resync your photos to the phone.