Can see files on my USB-Stick listed, but can not read them

Try
$ file your_pictures
and see how those that cannot be displayed is different from the others. You can paste an example here.

Hi Munzir,
I post here what I do:

-----------------
root@ulap:/home/uwh1/flash# ls -al
insgesamt 1975348
drwxr-xr-x 3 uwh1 uwh1 4096 2008-09-17 22:49 .
drwxr-xr-x 72 uwh1 uwh1 12288 2008-09-26 23:06 ..
-rw-r--r-- 1 uwh1 uwh1 1020661760 2008-09-11 17:10 flash.img
-rw-r--r-- 1 uwh1 uwh1 1000085321 2008-09-17 22:53 flash.img.tar.gz
drwxr-xr-x 2 root root 4096 2008-09-11 17:14 mnt
root@ulap:/home/uwh1/flash# mount -o loop ./flash.img ./mnt/
root@ulap:/home/uwh1/flash# cd mnt/
root@ulap:/home/uwh1/flash/mnt# ls
dcim nikon001.dsc
root@ulap:/home/uwh1/flash/mnt# cd dcim/
root@ulap:/home/uwh1/flash/mnt/dcim# ls
101ncd50
root@ulap:/home/uwh1/flash/mnt/dcim# cd 101ncd50/
root@ulap:/home/uwh1/flash/mnt/dcim/101ncd50# ls
dsc_4757.jpg dsc_4758.jpg dsc_4759.jpg dsc_4760.jpg dsc_4761.jpg dsc_4762.jpg dsc_4763.jpg dsc_4764.jpg dsc_4765.jpg dsc_4766.jpg dsc_4767.jpg dsc_4768.jpg dsc_4769.jpg
root@ulap:/home/uwh1/flash/mnt/dcim/101ncd50# cp * /tmp/
cp: Lesen von „dsc_4758.jpg“: Input/output error
cp: Lesen von „dsc_4759.jpg“: Input/output error
cp: Lesen von „dsc_4760.jpg“: Input/output error
cp: Lesen von „dsc_4761.jpg“: Input/output error
cp: Lesen von „dsc_4762.jpg“: Input/output error
cp: Lesen von „dsc_4763.jpg“: Input/output error
cp: Lesen von „dsc_4764.jpg“: Input/output error
cp: Lesen von „dsc_4765.jpg“: Input/output error
cp: Lesen von „dsc_4766.jpg“: Input/output error
cp: Lesen von „dsc_4767.jpg“: Input/output error
cp: Lesen von „dsc_4768.jpg“: Input/output error
cp: Lesen von „dsc_4769.jpg“: Input/output error
root@ulap:/home/uwh1/flash/mnt/dcim/101ncd50# ls /tmp/*.jpg
/tmp/dsc_4757.jpg /tmp/dsc_4759.jpg /tmp/dsc_4761.jpg /tmp/dsc_4763.jpg /tmp/dsc_4765.jpg /tmp/dsc_4767.jpg /tmp/dsc_4769.jpg
/tmp/dsc_4758.jpg /tmp/dsc_4760.jpg /tmp/dsc_4762.jpg /tmp/dsc_4764.jpg /tmp/dsc_4766.jpg /tmp/dsc_4768.jpg
root@ulap:/home/uwh1/flash/mnt/dcim/101ncd50# ls -l /tmp/*.jpg
-rwxr-xr-x 1 root root 1273856 2008-09-27 09:24 /tmp/dsc_4757.jpg
-rwxr-xr-x 1 root root 0 2008-09-27 09:24 /tmp/dsc_4758.jpg
-rwxr-xr-x 1 root root 0 2008-09-27 09:24 /tmp/dsc_4759.jpg
-rwxr-xr-x 1 root root 0 2008-09-27 09:24 /tmp/dsc_4760.jpg
-rwxr-xr-x 1 root root 0 2008-09-27 09:24 /tmp/dsc_4761.jpg
-rwxr-xr-x 1 root root 0 2008-09-27 09:24 /tmp/dsc_4762.jpg
-rwxr-xr-x 1 root root 0 2008-09-27 09:24 /tmp/dsc_4763.jpg
-rwxr-xr-x 1 root root 0 2008-09-27 09:24 /tmp/dsc_4764.jpg
-rwxr-xr-x 1 root root 0 2008-09-27 09:24 /tmp/dsc_4765.jpg
-rwxr-xr-x 1 root root 0 2008-09-27 09:24 /tmp/dsc_4766.jpg
-rwxr-xr-x 1 root root 0 2008-09-27 09:24 /tmp/dsc_4767.jpg
-rwxr-xr-x 1 root root 0 2008-09-27 09:24 /tmp/dsc_4768.jpg
-rwxr-xr-x 1 root root 0 2008-09-27 09:24 /tmp/dsc_4769.jpg
root@ulap:/home/uwh1/flash/mnt/dcim/101ncd50#
-------------------------------------

See, the one file (dsc_4757.jpg) was able to be copied (and the contents is OK, too) and the others are all scrap. I would just accept it if it would not be the case that MSwin can just read those files without any hickup.

BTW I have an actual Ubuntu system with kernel 2.6.

here is the kernal log:
Sep 27 09:13:41 ulap kernel: [14410.912614] loop: module loaded
Sep 27 09:24:33 ulap kernel: [39117.332479] attempt to access beyond end of device
Sep 27 09:24:33 ulap kernel: [39117.332496] loop0: rw=0, want=1993578, limit=1993480
Sep 27 09:24:33 ulap kernel: [39117.338096] attempt to access beyond end of device
Sep 27 09:24:33 ulap kernel: [39117.338107] loop0: rw=0, want=1993481, limit=1993480
Sep 27 09:24:33 ulap kernel: [39117.338426] attempt to access beyond end of device
Sep 27 09:24:33 ulap kernel: [39117.338435] loop0: rw=0, want=1993737, limit=1993480
Sep 27 09:24:33 ulap kernel: [39117.338445] attempt to access beyond end of device
Sep 27 09:24:33 ulap kernel: [39117.338452] loop0: rw=0, want=1993617, limit=1993480
Sep 27 09:24:33 ulap kernel: [39117.339777] attempt to access beyond end of device
Sep 27 09:24:33 ulap kernel: [39117.339786] loop0: rw=0, want=1996361, limit=1993480
Sep 27 09:24:33 ulap kernel: [39117.339797] attempt to access beyond end of device
Sep 27 09:24:33 ulap kernel: [39117.339803] loop0: rw=0, want=1996241, limit=1993480
Sep 27 09:24:33 ulap kernel: [39117.339946] attempt to access beyond end of device
Sep 27 09:24:33 ulap kernel: [39117.339954] loop0: rw=0, want=1999369, limit=1993480
Sep 27 09:24:33 ulap kernel: [39117.339964] attempt to access beyond end of device
Sep 27 09:24:33 ulap kernel: [39117.339971] loop0: rw=0, want=1999249, limit=1993480
Sep 27 09:24:33 ulap kernel: [39117.340114] attempt to access beyond end of device
.
.
.
.
and so on...

and here is one of the first lines of the dmesg output, identifying my system:
[ 0.000000] Linux version 2.6.24-19-generic (buildd@terranova) (gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7)) #1 SMP Wed Aug 20 22:56:21 UTC 2008 (Ubuntu 2.6.24-19.41-generic)

Thanks for investigating.
See you

Uwe

Forgot to say: I can provide the image in case somebody needs this to investigate this. I just don't want to see these pictures all over the internet.

Thanks
Uwe

Try accessing the usb disk directly from ubuntu without making an image and then using loop. If you still believe the picture is viewable from windows but not from ubuntu, choose a picture or take a new snapshot that has nothing private and make it available somewhere so I can check. You can also try changing the permission of the picture to 644 instead of 755 and also change the owner and recheck. Also, make sure you have enough space in /tmp using df -h, may be you ran out of space somehow.

I can not control if a picture is not readable or readable. I did take thousands of pictures and all are readable. But these few aren't. It just happens occasionally. That's why I can not just reproduce it with some "not private" pictures. BTW the pictures are not "that" private. They are just pictures of my family in front of a house. You can have them for technical investigation.

What did I do: First, I just plugged in USB my card reader with the inserted SD-card and read all of the pictures from there. These few did fail. My wife reported a few month ago a similar problem which was "solved" by reading the pictures with the laptop of my farther in law (who is windows user). The she reformatted the drive, which made it impossible to investigate further. Now I saw the very same and was able to read the pictures with my work-computer. I re-tried after reading the pictures on my Ubuntu laptop, but here the I/O-Error problem did persist. Therefore I created the disk dump and rechecked the same behaviour there. I also do have the SD-card unmodified sitting here and use another one for my camera. I am not a file system - kernel - hacking guy, that's why I do ask for somebody who is. I can provide the image to a private mail address if you mail your address to uwe DOT hartl AT hau DOT justonead DOT de

Yes, there are a couple of GB free space on my HD. I can not change permissions on these files, this is a VFAT image. Therefore the owner is always the guy who mounts the image (as far as I understood).

Sep 27 09:24:33 ulap kernel: [39117.332479] attempt to access beyond end of device
Sep 27 09:24:33 ulap kernel: [39117.332496] loop0: rw=0, want=1993578, limit=1993480

This sounds like the formatting is broken. Did you format the card inside the camera last? Did you google yet for known issues of the D50 and card formatting?

I usually format the card inside the camera, because I do copy the files and than put the card back into the camera and format it to have a clean start. I tried Google first but didn't find anything useful.
Thanks
Uwe

Are you sure your email is working?
dig hau.justonead.de mx shows there is no mx record for your domain. Feel free to send me a sample at munzirtaha At GMAIL and I would check it.

Hi Munzir,

you made me aware of the problem:
dd if=/dev/sdd1 of=flash.img bs=8k
dd: Lesen von „/dev/sdd1“: Input/output error
124592+1 Datensätze ein
124592+1 Datensätze aus
1020661760 Bytes (1,0 GB) kopiert, 1015,87 s, 1,0 MB/s
root@ulap:~# dd if=/dev/sdd1 of=flash.img
dd: Lesen von „/dev/sdd1“: Input/output error
1993480+0 Datensätze ein
1993480+0 Datensätze aus
1020661760 Bytes (1,0 GB) kopiert, 1031,09 s, 990 kB/s

It just reads 1GB and ignores the rest. I will retry to fetch the image with
dd conv=noerror if=/dev/sdd1 of=flash.img

now. I will report the result this evening.

Sorry for the ignored failure message.

Uwe

The picture did not change with the noerror option. This is strange. It did not read any further than the 1GB. It was all I/O errors.
It looks like to be a 1GB media but the result of df is:
df -h
/dev/sdd1 2,0G 97M 1,9G 5% /media/disk

which points to a 2GB media. Funnily there is windows which can read the pictures without any problem.

Is there some restriction for the vFAT driver under linux not beeing able to access more than 1GB on USB-Flash drives?

I am tempted to try to reformat the flash with a useful file system (e.g. ext2 or so) and try to write and read 2GB of data. But this would destroy the evidence... But it would clarify, if the flash - hardware is working.

Any suggestion what should I do?

Try cat instead e.g
cat /dev/sdd1 > image
or even ddrescue or dd_rescue

By the way, what if you used windows to make another copy of one buggy file on the same media, I guess you would be able to display it from linux then, right?

This is just to confirm it's a bug related to the media or filesystem and not the file itself.

here are zillions of these messages:

dd_rescue: (info): ipos: 1993606.5k, opos: 1993606.5k, xferd: 1993606.5k
                * errs:1993733, errxfer: 996866.5k, succxfer: 996740.0k
             +curr.rate: 12821kB/s, avg.rate: 233kB/s, avg.load: 2.2%
dd_rescue: (warning): /dev/sdd1 (1993606.5k): Input/output error!

dd_rescue: (info): ipos: 1993607.0k, opos: 1993607.0k, xferd: 1993607.0k
                * errs:1993734, errxfer: 996867.0k, succxfer: 996740.0k
             +curr.rate: 12500kB/s, avg.rate: 233kB/s, avg.load: 2.2%
dd_rescue: (warning): /dev/sdd1 (1993607.0k): Input/output error!

dd_rescue: (info): ipos: 1993607.5k, opos: 1993607.5k, xferd: 1993607.5k
                * errs:1993735, errxfer: 996867.5k, succxfer: 996740.0k
             +curr.rate: 19kB/s, avg.rate: 233kB/s, avg.load: 2.2%
dd_rescue: (warning): /dev/sdd1 (1993607.5k): Input/output error!

Bad block: 3987215
dd_rescue: (info): /dev/sdd1 (1993608.0k): EOF
Summary for /dev/sdd1 -> flash.img:
dd_rescue: (info): ipos: 1993608.0k, opos: 1993608.0k, xferd: 1993608.0k
                   errs:1993736, errxfer: 996868.0k, succxfer: 996740.0k
             +curr.rate: 7353kB/s, avg.rate: 233kB/s, avg.load: 2.2%

Well, the result with cat is the same as the one with dd_rescue:
root@ulap:/home/uwh1# ls -alh flash.img
-rw-r----- 1 root root 974M 2008-10-03 00:25 flash.img

I will try next to copy one of the unreadable files in Windows onto the same disk, but this may take until noon in Germany.

Will report later.
Uwe

Also try ddrescue which is smarter than dd_rescue in your case of having Bad blocks:
ddrescue -r3 /dev/sdd1 flash.img logfile
Find a tool in windows to umount that image and see whether you can still view the pictures

root@ulap:/home/uwh1# ddrescue -r3 /dev/sdd1 flash.img logfile

Press Ctrl-C to interrupt
Initial status (read from logfile)
rescued: 0 B, errsize: 0 B, errors: 0
Current status
rescued: 1020 MB, errsize: 1020 MB, current rate: 0 B/s
   ipos: 2041 MB, errors: 2009305, average rate: 427 kB/s
   opos: 2041 MB

Logfile:
# Rescue Logfile. Created by GNU ddrescue version 1.2
# pos size status
0x00000000 0x3CD61000 +
0x3CD61000 0x3CD80200 -

root@ulap:/home/uwh1# ls -alh flash.img
-rw-r--r-- 1 root root 974M 2008-10-03 13:13 flash.img

it looks like the same result.

OK, I went to the Windows Box and plugged in the flash. I marked all the pictures and Ctrl-C Ctrl-V copied them into the same location. Now I see the originals in there and the "Copy of 'original'". And these Copies I can read without any problem on my Ubuntu-Box. The originals are still not readable.

I do not understand your last sentence: "Find a tool in windows to umount that image and see whether you can still view the pictures"

I assume you wanted to tell me to use the image I have and try to mount it under Windows using something like filedisk or HD-Workbench. I did try it (with both programs) but I am not able to mount this image. The reasom may not be that the image isn't mountable, I just may be too stupid to do it in Windows.

Well, what are yur suggestions to proceed?

Thanks
Uwe

ddrescue should be smart enough to skip the bad blocks and save an almost 2GB image. Did you remove the previous image before ddrescue?

Do this:
1. plug the SD in linux, make sure it's mounted and post the output of mount and df
2. rm flash.img; ddrescue -r3 /dev/sdd1 flash.img logfile
and post the results
3. Explain the way you used file_disk to mount the image

OK
1.
root@ulap:/home/uwh1# mount
snip
/dev/sdd1 on /media/disk type vfat (rw,nosuid,nodev,noatime,uhelper=hal,flush,uid=1000,utf8,shortname=lower)
root@ulap:/home/uwh1# df
Dateisystem 1K‐Blöcke Benutzt Verfügbar Ben% Eingehängt auf
snip
/dev/sdd1 1993344 117984 1875360 6% /media/disk

2. this is exactly what I did here the complete sequence what I did:

root@ulap:/home/uwh1# rm flash.img
root@ulap:/home/uwh1# ddrescue -r3 /dev/sdd1 flash.img logfile

Press Ctrl-C to interrupt
Initial status (read from logfile)
rescued: 0 B, errsize: 0 B, errors: 0
Current status
rescued: 1020 MB, errsize: 1020 MB, current rate: 0 B/s
   ipos: 2041 MB, errors: 2009305, average rate: 427 kB/s
   opos: 2041 MB
root@ulap:/home/uwh1# ls -alh flash.img
-rw-r--r-- 1 root root 974M 2008-10-03 13:13 flash.img

3. Sorry I did not know. Seems the program needed a reboot after install. Now it does mount the image under Windows. I can see the files as I do under Linux and I can copy these files. They do have nice file sizes, but they seem to consist mostly of 0x00 values. In the last third there are some other values (I checked two of the files). But it is not a jpg files as expected.

To sum up the information we have:
- Linux can just read approx half of the media
- Windows can see the full media (and can copy the files from the media itself)
the created Image under Linux does not contain the image data (checked with the image mounted under Windows)
- The HW (card reader) did not change. The computer running Windows is different from the Ubuntu system

Next thing I try is: creating a disk image under Windows. Will see, how this behaves under Linux.

Aha! you are using a usb card reader? Then definitely the problem would be your card reader itself.
To prove this
1. creating a disk image under windows would work under linux.
2. replacing the card reader with a better one that supports 2GB SD would also work.

I think windows read the size from the Card whereas linux read the size sent by the reader via the usb interface somehow. The first approach is easier whereas the second is safer and it's always better to be safe than sorry. I remember I've seen a kernel bug regarding this but can't find it now and don't know what decision is made regarding it but would try to look it up.

Hmm, OK, got the point. I have a second card reader. I just tried this one and I was able to read all the images without a hitch. Thanks, this clue is the solution for the problem. The hardware is crap.

I want to point out one thing: I have no clue, why the decision for reading the size was made this way but I still believe linux should be able to do what windows can do. Otherwise people will use linux as something to try out but to work with windows.

But anyways, thank you for your patience and your help. BTW, do you think, if we tell the developer of the card reader driver to change the decision if we just let him read this launchpad page here?

Thanks again,

Uwe

> but I still believe linux should be able to do what windows can do

You cannot and should not generalize this statement in neither software nor hardware. e.g In lots of cases where windows displays web pages properly, firefox fails because the bug is in the design so is it better to conform to the w3c standards and encourage people to fix their pages or just follow what crap windows does?

In your case e.g you have a crap hardware may be with a buggy or old firmware. How can the kernel tell which is reporting the false info? One is lying and believing one or the other doesn't necessarily make one smarter. The one to blame here is the hardware vendor. Most vendors don't care about Linux, don't test their equipment on it, don't conform to standards and specifications, don't open the spec so free software developers can help, and add to this the legal issues and the patents hell and you wonder how GNU/Linux come into being a success after all this crap!

We should really encourage people to check hardware before buying it which is a habit people used to Windows don't need as often. The vendors design their hardware so it works with windows e.g most the modems (winmodems) available in the market won't work with GNU/Linux. Free software developers have few resources to access and limited time and concentrate more in hardware rather than crap stuff and workarounds. Till vendors realize the need to support Linux, you cannot and should not expect Linux to be able to do what Windows can do