[Libusb-devel] usb_bulk_read() problem

[Pawel Kot]

Hi,

In gnokii project I make use of libusb when connecting via DKU2 cables
to Nokia phones. So far everything looked good. However recently it
turned out that usb_bulk_read() looses data. Code to read the data is:
retval = usb_bulk_read(DEVINSTANCE(state)->interface->dev_data,
DEVINSTANCE(state)->interface->data_endpoint_read,
(char *) bytes, size, USB_FBUS_TIMEOUT);
When size is big enough to store all response it's fine. But when the
buffer is to small, and I need to read twice, 64 bytes is being lost
between two calls.

Here's the example:

When size is set to 1024 I got:
[libusb] 1b 0c 00 14 01 be 01 78 00 13 00 05 01 58 02 12
[libusb] 00 53 00 6b 00 72 00 7a 00 79 00 6e 00 6b 00 61
[libusb] 00 20 00 6f 00 64 00 62 00 69 00 6f 00 72 00 63
[libusb] 00 7a 00 61 00 00 00 00 00 00 00 00 00 00 00 00
[libusb] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[libusb] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[libusb] 00 00 00 00 01 58 03 0f 00 57 00 69 00 61 00 64
[libusb] 00 6f 00 6d 00 2e 00 20 00 77 00 79 00 73 01 42
[libusb] 00 61 00 6e 00 65 00 00 00 00 00 00 00 00 00 00
[libusb] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[libusb] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[libusb] 00 00 00 00 00 00 00 00 00 00 00 00 01 58 04 13
[libusb] 00 5a 00 61 00 70 00 69 00 73 00 2e 00 20 00 77
[libusb] 00 69 00 61 00 64 00 2e 00 20 00 74 00 65 00 6b
[libusb] 00 73 00 74 00 2e 00 00 00 00 00 00 00 00 00 00
[libusb] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[libusb] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[libusb] 00 00 00 00 01 58 05 08 00 53 00 7a 00 61 00 62
[libusb] 00 6c 00 6f 00 6e 00 79 00 00 61 00 64 00 61 00
[libusb] 77 00 63 00 7a 00 61 00 00 00 00 00 00 00 00 00
[libusb] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[libusb] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[libusb] 00 00 00 00 00 00 00 00 00 00 00 00 01 58 1a 10
[libusb] 00 53 00 6b 00 72 00 7a 00 79 00 6e 00 2e 00 20
[libusb] 00 6e 00 61 00 64 00 61 00 77 00 63 00 7a 00 61
[libusb] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[libusb] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[libusb] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[libusb] 00 00 00 00

When size is set to 255 I got:
[libusb] 1b 0c 00 14 01 be 01 78 00 13 00 05 01 58 02 12
[libusb] 00 53 00 6b 00 72 00 7a 00 79 00 6e 00 6b 00 61
[libusb] 00 20 00 6f 00 64 00 62 00 69 00 6f 00 72 00 63
[libusb] 00 7a 00 61 00 00 00 00 00 00 00 00 00 00 00 00
[libusb] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[libusb] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[libusb] 00 00 00 00 01 58 03 0f 00 57 00 69 00 61 00 64
[libusb] 00 6f 00 6d 00 2e 00 20 00 77 00 79 00 73 01 42
[libusb] 00 61 00 6e 00 65 00 00 00 00 00 00 00 00 00 00
[libusb] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[libusb] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[libusb] 00 00 00 00 00 00 00 00 00 00 00 00 01 58 04 13




[libusb] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[libusb] 00 00 00 00 01 58 05 08 00 53 00 7a 00 61 00 62
[libusb] 00 6c 00 6f 00 6e 00 79 00 00 61 00 64 00 61 00
[libusb] 77 00 63 00 7a 00 61 00 00 00 00 00 00 00 00 00
[libusb] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[libusb] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[libusb] 00 00 00 00 00 00 00 00 00 00 00 00 01 58 1a 10
[libusb] 00 53 00 6b 00 72 00 7a 00 79 00 6e 00 2e 00 20
[libusb] 00 6e 00 61 00 64 00 61 00 77 00 63 00 7a 00 61
[libusb] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[libusb] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[libusb] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[libusb] 00 00 00 00

So as you can see 64 bytes are missing in the middle. It's with CVS
version from 0.1 branch and the one shipped by Ubuntu.

What am I doing wrong? Or is it a bug in libusb?

take care,
pkot
PS. Please Cc me on replies.

[Tim Roberts]

Post by Pawel Kot
Hi,
In gnokii project I make use of libusb when connecting via DKU2 cables
to Nokia phones. So far everything looked good. However recently it
retval = usb_bulk_read(DEVINSTANCE(state)->interface->dev_data,
DEVINSTANCE(state)->interface->data_endpoint_read,
(char *) bytes, size, USB_FBUS_TIMEOUT);
When size is big enough to store all response it's fine. But when the
buffer is to small, and I need to read twice, 64 bytes is being lost
between two calls.
...(452 byte dump deleted)...
...(192 byte dump deleted)...
...(196 byte dump deleted)...
So as you can see 64 bytes are missing in the middle. It's with CVS
version from 0.1 branch and the one shipped by Ubuntu.
What am I doing wrong? Or is it a bug in libusb?

It's probably not a bug in libusb. There just isn't that much TO
libusb. On Linux, usb_bulk_read basically sends an ioctl request to the
driver just as you have formatted it, and waits for the response.
Remember that there is NO BUFFERING, either in libusb or in the kernel
driver.

Do you have a USB monitor that shows that the data is really being
transmitted? Does the device's USB endpoint have a packet size of 64
bytes? Are these phones supposed to handle arbitrarily-sized requests?
Do the phones have buffers?

[Tim Roberts]

Post by Tim Roberts
It's probably not a bug in libusb. There just isn't that much TO
libusb. On Linux, usb_bulk_read basically sends an ioctl request to the
driver just as you have formatted it, and waits for the response.
Remember that there is NO BUFFERING, either in libusb or in the kernel
driver.

Just of the curiosity, I have printed out more buffer than it is
allowed and it seems that everything is there! Just the length is
incorrectly calculated (of the first frame in the example I gave).

What does that mean? Do you mean you asked for 255 bytes and it
returned more than that? Do you mean it told you that it returned 192
bytes but it apparently returned more? Why don't you post the entire
routine exactly as you ran it in the case where you get the confusing
results?

So it would *look* like it is buffered.

It isn't. No one in the USB stack does any buffering. No requests are
sent until you ask for data, and the only data that are returned get
copied into your buffers.

Post by Tim Roberts
Do you have a USB monitor that shows that the data is really being
transmitted?

No. Any pointer to such? What I found was only for Windows.

What I meant was a hardware USB monitor.

Post by Tim Roberts
Does the device's USB endpoint have a packet size of 64 bytes?

No idea. How do I find out?

Really? Have you never looked at the device's descriptors? How can you
be sure you're using the right communication method?

Try "cat /proc/bus/usb/devices". That will show you the basic
information of your endpoints. Post it here, if you'd like help
interpreting them.

[Pawel Kot]

Hi,

Post by Tim Roberts
It's probably not a bug in libusb. There just isn't that much TO
libusb. On Linux, usb_bulk_read basically sends an ioctl request to the
driver just as you have formatted it, and waits for the response.
Remember that there is NO BUFFERING, either in libusb or in the kernel
driver.

Just of the curiosity, I have printed out more buffer than it is
allowed and it seems that everything is there! Just the length is
incorrectly calculated (of the first frame in the example I gave). So
it would *look* like it is buffered.
I belive I should look into the kernel driver, right?

Post by Tim Roberts
Do you have a USB monitor that shows that the data is really being
transmitted?

No. Any pointer to such? What I found was only for Windows.

Post by Tim Roberts
Does the device's USB endpoint have a packet size of 64 bytes?

No idea. How do I find out?

Post by Tim Roberts
Are these phones supposed to handle arbitrarily-sized requests?

Ditto.

Post by Tim Roberts
Do the phones have buffers?

Also don't know this.

BTW: using nokia_dku2 kernel driver
(http://cvs.savannah.nongnu.org/viewvc/gnokii/patches/kernel_2.6/nokia_dku2.c?revision=1.14&root=gnokii&view=markup)
instead of libusb makes things work OK.

thanks for help,
pkot

[Pawel Kot]

Hi,

[Sorry for breaking references. Now as I am subscribed to the list,
there should be no problem anymore]

It's probably not a bug in libusb. There just isn't that much TO
libusb. On Linux, usb_bulk_read basically sends an ioctl request to the
driver just as you have formatted it, and waits for the response.
Remember that there is NO BUFFERING, either in libusb or in the kernel
driver.

Just of the curiosity, I have printed out more buffer than it is
allowed and it seems that everything is there! Just the length is
incorrectly calculated (of the first frame in the example I gave).

What does that mean? Do you mean you asked for 255 bytes and it
returned more than that? Do you mean it told you that it returned 192
bytes but it apparently returned more?

It means that I asked for 255 bytes, usb_bulk_read() told me it
returns 192 bytes, but I printed out 255 bytes. And the "overflowing"
bytes looked like the ones I have expected. I did printing out the
buffer inside usb_bulk_read() just before return from that function.

Why don't you post the entire
routine exactly as you ran it in the case where you get the confusing
results?

Because it's not a single routine. As I said it is gnokii project. You
can browse CVS at
http://cvs.savannah.nongnu.org/viewvc/gnokii/?root=gnokii
The interesging files are:
- common/devices/dku2libusb.c (these are wrappers over library functions)
- common/links/fbus-phonet.c (here phonet_loop() is the function that
is involved in reading)
- common/gsm-statemachine.c (here gn_sm_loop() and its callers)
That would be all lowlevel intrastructure involved. Further on are
phone drivers that send a request and wait for response
(common/phone/*.c files)

So it would *look* like it is buffered.

It isn't. No one in the USB stack does any buffering. No requests are
sent until you ask for data, and the only data that are returned get
copied into your buffers.

Okay.

Do you have a USB monitor that shows that the data is really being
transmitted?

No. Any pointer to such? What I found was only for Windows.

What I meant was a hardware USB monitor.

No, I don't have any.

Does the device's USB endpoint have a packet size of 64 bytes?

No idea. How do I find out?

Really? Have you never looked at the device's descriptors? How can you
be sure you're using the right communication method?

I did. I just looked over the things that looked relevant to me. And
it seemed to work smoothly so far. BTW. gnokii code is based on
openobex support written by Alex Kanavin.

Try "cat /proc/bus/usb/devices". That will show you the basic
information of your endpoints. Post it here, if you'd like help
interpreting them.

I will when get back home today. So far I have lsusb -vvv output
stored at http://tfuj.pl/gnokii/lsusb2 (it's 0421:0441 thing). And it
seems to have packet size of 64bytes indeed:
wMaxPacketSize 0x0040 1x 64 bytes

take care,
pkot

[Pawel Kot]

Hi,

Post by Tim Roberts
Does the device's USB endpoint have a packet size of 64 bytes?

It just came to my mind what did you mean with this question. Can it
be a problem that buffer size is 255 bytes and the device sends
packets of 64 bytes? So when we want 255 bytes, it reads 192 bytes
(max 64 multipication less than 255) and as libusb/kernel does not
buffer, remaining 64 bytes are lost?

take care
pkot

[Pawel Kot]

Hi,

Post by Pawel Kot
Hi,

Post by Tim Roberts
Does the device's USB endpoint have a packet size of 64 bytes?

It just came to my mind what did you mean with this question. Can it
be a problem that buffer size is 255 bytes and the device sends
packets of 64 bytes? So when we want 255 bytes, it reads 192 bytes
(max 64 multipication less than 255) and as libusb/kernel does not
buffer, remaining 64 bytes are lost?

It seems it is the case. Who do I talk (probably from linux-usb) to
provide at least log information that some data is being lost in that
case?

take care,
pkot

[Xiaofan Chen]

Post by Pawel Kot
In gnokii project I make use of libusb when connecting via DKU2 cables
to Nokia phones. So far everything looked good. However recently it
retval = usb_bulk_read(DEVINSTANCE(state)->interface->dev_data,
DEVINSTANCE(state)->interface->data_endpoint_read,
(char *) bytes, size, USB_FBUS_TIMEOUT);
When size is big enough to store all response it's fine. But when the
buffer is to small, and I need to read twice, 64 bytes is being lost
between two calls.
So as you can see 64 bytes are missing in the middle. It's with CVS
version from 0.1 branch and the one shipped by Ubuntu.
What am I doing wrong? Or is it a bug in libusb?

I am not an expert here but why do you want to use 255 as the
buffer size? Try 256 or the multiple of 64 as you mentioned in
later emails that the packet size is 64. I think you should use
multiple of endpoint packet size as the buffer size.

So maybe this is a firmware problem (failing to deal with
zero length package or short package). It is said that Nokia
phones have many buggy firmwares.

Xiaofan
http://mcuee.blogspot.com

[Pawel Kot]

Hi,

Post by Xiaofan Chen

Post by Pawel Kot
In gnokii project I make use of libusb when connecting via DKU2 cables
to Nokia phones. So far everything looked good. However recently it
retval = usb_bulk_read(DEVINSTANCE(state)->interface->dev_data,
DEVINSTANCE(state)->interface->data_endpoint_read,
(char *) bytes, size, USB_FBUS_TIMEOUT);
When size is big enough to store all response it's fine. But when the
buffer is to small, and I need to read twice, 64 bytes is being lost
between two calls.
So as you can see 64 bytes are missing in the middle. It's with CVS
version from 0.1 branch and the one shipped by Ubuntu.
What am I doing wrong? Or is it a bug in libusb?

I am not an expert here but why do you want to use 255 as the
buffer size? Try 256 or the multiple of 64 as you mentioned in
later emails that the packet size is 64. I think you should use
multiple of endpoint packet size as the buffer size.

Like I said in the other mail, setting buffer size to multiplication
of 64 indeed solves the problem. But the real problem is that
libusb/kernel looses data not saying the word! Ideally there would be
a buffer storing data the device has sent, but if it is a design
principle "no buffer at all", it would be nice to say somehow (logs?)
that some data has been lost (after looking in libusb sources I agree
that it should be done by the kernel).

take care,
pkot

[Xiaofan Chen]

Post by Pawel Kot

Post by Xiaofan Chen
I am not an expert here but why do you want to use 255 as the
buffer size? Try 256 or the multiple of 64 as you mentioned in
later emails that the packet size is 64. I think you should use
multiple of endpoint packet size as the buffer size.

Like I said in the other mail, setting buffer size to multiplication
of 64 indeed solves the problem. But the real problem is that
libusb/kernel looses data not saying the word! Ideally there would be
a buffer storing data the device has sent, but if it is a design
principle "no buffer at all", it would be nice to say somehow (logs?)
that some data has been lost (after looking in libusb sources I agree
that it should be done by the kernel).

As far as I know, you know your device and you should not
use the wrong buffer size. I believe this is not a problem
of the kernel and it is only your host software's problem.

Maybe you can ask in the linux-usb list (now combined
the old use and devel list).

Xiaofan

[Tim Roberts]

Post by Pawel Kot
Like I said in the other mail, setting buffer size to multiplication
of 64 indeed solves the problem. But the real problem is that
libusb/kernel looses data not saying the word! Ideally there would be
a buffer storing data the device has sent, but if it is a design
principle "no buffer at all", it would be nice to say somehow (logs?)
that some data has been lost (after looking in libusb sources I agree
that it should be done by the kernel).

Remember that a device is never told how much data is being requested.
It just gets a signal saying "send now!", and it sends whatever is
available, up to the maximum packet size. If a user requests 255 bytes
and the device tries to send 256, that's called "babble" and is
considered a protocol violation. The host controller or host controller
driver is within its rights to discard the erroneous packet. Windows
diagnoses this condition with an error to the driver; I don't know about
Linux.

Is this a USB 2.0 device or a USB 1.1 device? A USB 2.0 bulk pipe is
required to have a maximum packet size of 512 bytes.

[Pawel Kot]

Hi Tim.

Post by Tim Roberts

Post by Pawel Kot
Like I said in the other mail, setting buffer size to multiplication
of 64 indeed solves the problem. But the real problem is that
libusb/kernel looses data not saying the word! Ideally there would be
a buffer storing data the device has sent, but if it is a design
principle "no buffer at all", it would be nice to say somehow (logs?)
that some data has been lost (after looking in libusb sources I agree
that it should be done by the kernel).

Remember that a device is never told how much data is being requested.
It just gets a signal saying "send now!", and it sends whatever is
available, up to the maximum packet size. If a user requests 255 bytes
and the device tries to send 256, that's called "babble" and is
considered a protocol violation. The host controller or host controller
driver is within its rights to discard the erroneous packet. Windows
diagnoses this condition with an error to the driver; I don't know about
Linux.

I have changed the buffer size in my app and I'm happy with that. What
I think is dangerous is that kernel just drops 64 bytes and doesn't
say that even in some logs. Perhaps rejecting the whole packet would
tell me more about where the problem lies (I've been looking for a
while at this problem).

BTW. I'm not sure if I mentioned but the problem does not occur with
255bytes buffer and usb-serial + nokia_dku2
(http://cvs.savannah.nongnu.org/viewvc/gnokii/patches/kernel_2.6/nokia_dku2.c?revision=1.14&root=gnokii&view=markup)
which uses the same endpoints.

Post by Tim Roberts
Is this a USB 2.0 device or a USB 1.1 device? A USB 2.0 bulk pipe is
required to have a maximum packet size of 512 bytes.

USB 1.1.

take care,
pkot

[Tim Roberts]

Post by Pawel Kot
I have changed the buffer size in my app and I'm happy with that. What
I think is dangerous is that kernel just drops 64 bytes and doesn't
say that even in some logs. Perhaps rejecting the whole packet would
tell me more about where the problem lies (I've been looking for a
while at this problem).

In essence, that's what it has done. Your 255-byte request gets split
up into three 64-byte packets and a 63-byte packet, because of your
endpoint's configuration. The first three packets succeed cleanly, and
the fourth gets a babble error. The kernel is telling you that three
packets succeeded, so you have 192 bytes of data.

The root of the problem is that the API does not provide any way to
report that a request succeeded partially, and still return an error
code at the same time. This is perhaps the one advantage of the Windows
scheme, where the I/O APIs have a parameter to return the number of
bytes transferred plus a separate error code.

[Xiaofan Chen]

Post by Pawel Kot
BTW. I'm not sure if I mentioned but the problem does not occur with
255bytes buffer and usb-serial + nokia_dku2
(http://cvs.savannah.nongnu.org/viewvc/gnokii/patches/kernel_2.6/nokia_dku2.c?revision=1.14&root=gnokii&view=markup)
which uses the same endpoints.

But that driver is said to be inappropriate and was dropped from the
Linux kernel. An inappropriate driver and an inappropriate host
program may or may not work. ;-)

---------- Forwarded message ----------
From: Greg Kroah-Hartman
Date: Nov 18, 2005 1:48 AM
Subject: [linux-usb-devel] [patch 21/22] USB: delete the nokia_dku2 driver
To: Linus Torvalds , Andrew Morton
Cc: linux-kernel, linux-usb-devel


From: Greg Kroah-Hartman

It was causing too many problems, and this is not the proper type of
driver for this device.

Signed-off-by: Greg Kroah-Hartman <***@suse.de>
---
... skipped...
- * Nokia DKU2 USB driver
- *
- * Copyright (C) 2004
- * Author: C Kemp

[Pawel Kot]

Hi,

Post by Xiaofan Chen

Post by Pawel Kot
BTW. I'm not sure if I mentioned but the problem does not occur with
255bytes buffer and usb-serial + nokia_dku2
(http://cvs.savannah.nongnu.org/viewvc/gnokii/patches/kernel_2.6/nokia_dku2.c?revision=1.14&root=gnokii&view=markup)
which uses the same endpoints.

But that driver is said to be inappropriate and was dropped from the
Linux kernel. An inappropriate driver and an inappropriate host
program may or may not work. ;-)

But due to completly different reasons.

If you think there's completly no problem in libusb/linux kernel,
that's fine with me. I won't argue.

take care,
pkot

[Xiaofan Chen]

Post by Pawel Kot
If you think there's completly no problem in libusb/linux kernel,
that's fine with me. I won't argue.

I am not arguing. Actually I was hit by a very similar problem
with libusb-win32. It is the fault of my code, not libusb-win32
in the end.

http://ml.osdir.com/lib.libusb.devel.windows/2005-10/msg00012.html
http://ml.osdir.com/lib.libusb.devel.windows/2005-10/msg00013.html

Xiaofan

[Tim Roberts]

Post by Pawel Kot
BTW. I'm not sure if I mentioned but the problem does not occur with
255bytes buffer and usb-serial + nokia_dku2
(http://cvs.savannah.nongnu.org/viewvc/gnokii/patches/kernel_2.6/nokia_dku2.c?revision=1.14&root=gnokii&view=markup)
which uses the same endpoints.

Right. The USB serial driver DOES have a buffer. It submits
whole-packet-sized requests to the USB host controller, and copies the
results into your odd-sized buffer when they come back. When you use
libusb without usbserial, you are connected directly to the metal, and
so you are required to behave better.