[libusb] [RFC] Deprecations and version changes

[Chris Dickens]

Hi,

I have finally been able to get back to working on the event abstraction
effort. In the initial patch series, I made it such that
libusb_get_pollfds() would work on Windows platforms. As I was reworking
the patch series, I grew less fond of having a POSIX-specific concept being
applied to Windows, so I now have the following proposals for additions to
the API:

libusb_event_sources_handle_timeouts()
libusb_get_event_sources()
libusb_free_event_sources()
libusb_set_event_source_notifiers()

There will be a new structure as follows:

struct libusb_event_source {
libusb_os_handle source;
short events;
};

The libusb_os_handle type will be either an int or HANDLE type for
non-Windows and Windows platforms, respectively. Are there any objections
to this typedef name? I don't love it but don't have a better one.

Unless anyone disagrees, I don't think it is worthwhile to conditionally
define the 'events' field in this structure based on the platform. A
fixed-presence field, regardless of whether it is used or not, seems safer
to me.

I then would like the types and functions associated with pollfds to be
marked as deprecated, using the LIBUSB_DEPRECATED_FOR annotation that we
already have. The functions themselves will not be removed, for obvious ABI
reasons.

This brings me to library versioning. Per
http://www.gnu.org/software/libtool/manual/html_node/Updating-version-info.html,
we have not been following the correct approach in regards to adding new
functions to the API. With this event abstraction patch, I propose the
following changes:

1) Bumping the current and age in the libtool version. This will yield a
library with a soname of libusb-1.0.so.0 and a filename of
libusb-1.0.so.0.2.0
2) Changing libusb's internal version to 1.1.0. The header file directory
and library name all have 'libusb-1.0' in their name...does this matter?

Regards,

Chris

[Peter Stuge]

Hi Chris,

Post by Chris Dickens
The libusb_os_handle type will be either an int or HANDLE type for
non-Windows and Windows platforms, respectively.

This is clearly not a cross-platform API, and I consider it a mistake.

Post by Chris Dickens
1) Bumping the current and age in the libtool version. This will yield
a library with a soname of libusb-1.0.so.0 and a filename of
libusb-1.0.so.0.2.0

Indeed 1.0.9 did not bump current and age, and noone since has done
so either. I don't think there's any problem with bumping them.

Post by Chris Dickens
2) Changing libusb's internal version to 1.1.0. The header file directory
and library name all have 'libusb-1.0' in their name...does this matter?

Please don't do that, it creates quite some unneccessary confusion.

1.0 is, as you write, part of the library name and much less so of
the version, leaving really only one meaningful version atom in the
libusb-1.0 version number.

If you want to create libusb-1.1 then go for it I guess, but that's
new directory and file names, an API break and a libusb-compat-1.0 to
sit on top, emulating the 1.0 API. A lot of work, not really
worthwhile for an API extension.

If you do add some kind of event stuff then you may also want to
add a capability. The very best improvement would probably be to
start versioning symbols, instead of the library as a whole.


//Peter

[Chris Dickens]

Hi Peter,

Thanks for your input.

Post by Peter Stuge

Post by Chris Dickens
The libusb_os_handle type will be either an int or HANDLE type for
non-Windows and Windows platforms, respectively.

This is clearly not a cross-platform API, and I consider it a mistake.

I disagree. There is no cross-platform way to represent both a file
descriptor and HANDLE in a single type. Now we can get into choosing the
largest data type and type-casting, but what's the point?

This isn't the first time this problem has been encountered in a
cross-platform library. libcurl does the same thing with sockets and their
curl_socket_t (
https://github.com/curl/curl/blob/master/include/curl/curl.h#L125).

Post by Peter Stuge
2) Changing libusb's internal version to 1.1.0. The header file directory

Post by Chris Dickens
and library name all have 'libusb-1.0' in their name...does this matter?

Please don't do that, it creates quite some unneccessary confusion.
1.0 is, as you write, part of the library name and much less so of
the version, leaving really only one meaningful version atom in the
libusb-1.0 version number.
If you want to create libusb-1.1 then go for it I guess, but that's
new directory and file names, an API break and a libusb-compat-1.0 to
sit on top, emulating the 1.0 API. A lot of work, not really
worthwhile for an API extension.

Fair point, I agree that it would be unfruitful to purse changing the minor
version. I wonder if when creating libusb-1.0 this was realized.

At some point we should break API so we can fix some of the API bugs. I
don't know whether now is the opportune time or not...

Post by Peter Stuge
If you do add some kind of event stuff then you may also want to
add a capability. The very best improvement would probably be to
start versioning symbols, instead of the library as a whole.

Not sure I see the benefit of adding a capability for this. These are new
API extensions that will either be there or not. It will be a link error if
the symbols aren't available to the application.

Regards,

Chris

[Peter Stuge]

Hi,

Post by Chris Dickens

Post by Peter Stuge

Post by Chris Dickens
The libusb_os_handle type will be either an int or HANDLE type for
non-Windows and Windows platforms, respectively.

This is clearly not a cross-platform API, and I consider it a mistake.

I disagree. There is no cross-platform way to represent both a file
descriptor and HANDLE in a single type.

Right, so let's explore other ways to accomplish the same thing. The
intent of libusb APIs was always to be cross-platform, because that's
convenient to use.

I would suggest adding an event handler thread instead. And when it's
useful enable more callback registrations *into* the library, already
a core pattern in the async and hotplug APIs, rather than exporting
event sources *out of* the library.

Post by Chris Dickens
This isn't the first time this problem has been encountered in a
cross-platform library. libcurl does the same thing with sockets
and their curl_socket_t (

Many people smoke, that doesn't make smoking good for the body.

Post by Chris Dickens

Post by Peter Stuge
2) Changing libusb's internal version to 1.1.0. The header file directory

Post by Chris Dickens
and library name all have 'libusb-1.0' in their name...does this matter?

Please don't do that, it creates quite some unneccessary confusion.

..

Post by Chris Dickens

Post by Peter Stuge
If you want to create libusb-1.1 then go for it I guess, but that's
new directory and file names, an API break and a libusb-compat-1.0 to
sit on top, emulating the 1.0 API. A lot of work, not really
worthwhile for an API extension.

Fair point, I agree that it would be unfruitful to purse changing the
minor version. I wonder if when creating libusb-1.0 this was realized.

I do think it was clear to Daniel, I seem to remember him also
mentioning that libusb-1.1 would also mean a libusb-compat-1.0.

Post by Chris Dickens
At some point we should break API so we can fix some of the API bugs.

I don't disagree with that.

Post by Chris Dickens
I don't know whether now is the opportune time or not...

I guess it's a matter of how much dedicated development time is
available.

Post by Chris Dickens

Post by Peter Stuge
If you do add some kind of event stuff then you may also want to
add a capability. The very best improvement would probably be to
start versioning symbols, instead of the library as a whole.

Not sure I see the benefit of adding a capability for this. These are new
API extensions that will either be there or not. It will be a link error if
the symbols aren't available to the application.

I've seen both Linux and Windows application dynamically load the
library at runtime.


//Peter

[Orin Eman]

Post by Peter Stuge
Hi,

Post by Chris Dickens

Post by Peter Stuge

Post by Chris Dickens
The libusb_os_handle type will be either an int or HANDLE type for
non-Windows and Windows platforms, respectively.

This is clearly not a cross-platform API, and I consider it a mistake.

I disagree. There is no cross-platform way to represent both a file
descriptor and HANDLE in a single type.

Right, so let's explore other ways to accomplish the same thing. The
intent of libusb APIs was always to be cross-platform, because that's
convenient to use.
I would suggest adding an event handler thread instead. And when it's
useful enable more callback registrations *into* the library, already
a core pattern in the async and hotplug APIs, rather than exporting
event sources *out of* the library.

So the 'event handler thread' inside of libusb does the poll/wait on
events and makes a callback when something happens?

That certainly avoids the OS handle/fd problem, at the cost of the callback
being on an arbitrary thread. Personally, that wouldn't bother me. My
libusb usage has been to dedicate a thread to event handling anyway, so
I've already dealt with all the thread synchronization issues.

IMO, this is a good way to make libusb's event handling cross-platform and
much better than exposing resources that should be internal to libusb,
whether they be fds or event handles.

[Peter Stuge]

Post by Orin Eman

Post by Peter Stuge
I would suggest adding an event handler thread instead. And when it's
useful enable more callback registrations *into* the library,

So the 'event handler thread' inside of libusb does the poll/wait on
events

Right.

Post by Orin Eman
and makes a callback when something happens?

I would expect that an internal event handler (at least by default)
takes care of all things event handling and that no new callbacks
are neccessary, but there may be something - then yes.

Post by Orin Eman
That certainly avoids the OS handle/fd problem, at the cost of the
callback being on an arbitrary thread.

Yes, and that's an important point to document, especially if
transfer callbacks are then guaranteed to be called in that internal
thread. Whether or not to do full event handling could be debated, or
simply made configurable. ;)

If not automatically full event handling, then yes, there would be a
callback to tell the consumer that it should call some event handling
function. I don't think that should be the default though, because
inconvenient.

Adding event handling does mean a fair bit of documentation work though. :\


//Peter

[Chris Dickens]

Hi,

Post by Peter Stuge
If not automatically full event handling, then yes, there would be a
callback to tell the consumer that it should call some event handling
function. I don't think that should be the default though, because
inconvenient.

So a dedicated thread inside libusb just to inform the application, through
a registered callback, that something has happened and the application
should call in to handle the events? This sounds like a horrible waste of a
thread. The thread cannot be called an "event handling" thread because it
doesn't actually handle anything, it just informs the application that
something is pending. I do not see the benefit to this approach at all.

Now, as to the notion of extending libusb to be able to provide an event
handling thread internally (that actually fully handles events), this I
support. I myself have an application written that creates a thread
dedicated to calling libusb_handle_events() in a loop. I'm sure many
applications out there do the same, and this is something that can fairly
easily be provided by the library.

Regards,

Chris

[Orin Eman]

On Sun, Jul 9, 2017 at 10:03 PM, Chris Dickens <

Post by Chris Dickens
Hi,

Post by Peter Stuge
If not automatically full event handling, then yes, there would be a
callback to tell the consumer that it should call some event handling
function. I don't think that should be the default though, because
inconvenient.

So a dedicated thread inside libusb just to inform the application,
through a registered callback, that something has happened and the
application should call in to handle the events? This sounds like a
horrible waste of a thread. The thread cannot be called an "event handling"
thread because it doesn't actually handle anything, it just informs the
application that something is pending. I do not see the benefit to this
approach at all.
Now, as to the notion of extending libusb to be able to provide an event
handling thread internally (that actually fully handles events), this I
support. I myself have an application written that creates a thread
dedicated to calling libusb_handle_events() in a loop. I'm sure many
applications out there do the same, and this is something that can fairly
easily be provided by the library.

I'm fully on board with the latter - I too used a thread in a loop to
handle events, though I did use the thread to handle timeouts as well.

The former has the benefit of hiding both internal libusb resources _and_
the method used to determine something is pending, whether it be poll() or
WaitForMultipleEvents() - which is the point of the exercise - hiding OS
dependencies. It would be trivial for a callback to write to a pipe or
socket to wake up a main context via poll() - and would likely eliminate
the need for most of the 16 libusb calls to do with poll fds/event handling.

I ask the question: "What is truly OS independent?"

poll() Absolutely not (*X only)
Events Absolutely not (Windows only)
Semaphores Not really (no anonymous semaphores on some OSs)
Condition Variables Not really (recent addition on Windows)
Callbacks Yes!

So, I say go for callbacks. Either or both methods mentioned would be
acceptable to me, but I must admit, if an event handling thread was
available in libusb, it is what I would use.

[Peter Stuge]

Post by Orin Eman

Post by Chris Dickens

Post by Peter Stuge
If not automatically full event handling, then yes, there would be a
callback to tell the consumer that it should call some event handling
function. I don't think that should be the default though, because
inconvenient.

Now, as to the notion of extending libusb to be able to provide an event
handling thread internally (that actually fully handles events), this I
support.

I'm fully on board with the latter - I too used a thread in a loop to
handle events, though I did use the thread to handle timeouts as well.

..

Post by Orin Eman
if an event handling thread was available in libusb, it is what I would use.

Yep, same here. To be clear, I too prefer the thread to fully take
care of all event handling, ie. actually handle events. :)


//Peter

[David Grayson]

Hello.

I have a seen a lot of patches on this mailing list in the last few
years to fix multithreading in libusb. There were patches from Chris
Dickens in September 2015 entitled "Event handling fixes for
robustness" and patches from Hans De Goede in June 2016 entitled
"libusb locking improvements / bug-fixes". Given that the
multithreading in libsub was fixed twice in the past two years, and
threading bugs cannot easily be found by testing an application during
development, it makes me weary to use libusb in production.

Side note: Also, isn't there also an issue on Windows where libusb
cannot have more than 64 transfers running at a time because libusb
uses WaitForMultipleObjects, which only takes up to 64 objects?

Putting aside the current state of libusb and just thinking about how
a general USB abstraction library should work, I think it should be
like this: If the user wants to do a synchronous, blocking USB
transfer, the library should do it in a simple, single-threaded way,
using the corresponding blocking function provded by the OS. If the
user wants to have multiple transfers happening at once, the library
should provide some objects that correspond to the users' use cases.
For example, one use case is that you want to transfer a lot of data
from an IN endpoint to the computer as fast as possible. So there
could be an object called async_in_pipe that manages that. The object
would have functions for retrieving the data from finished transfers
and rearming those transfers to receive more data. Because of the way
events are reported on Linux and macOS, you would probably have a
separate function to handle events from the USB device handle,
updating the state of any async_in_pipe objects that are in use.

In a setup like that, you would be able to have more than 64 transfers
at once on Windows because the async_in_pipe object would know what
transfer is expected to finish next, and it would never need to call
WaitForMultipleObjects. It would just check the state of that
transfer whenever the user asks if there is more data available.

--David Grayson

Post by Peter Stuge

Post by Orin Eman

Post by Chris Dickens

Post by Peter Stuge
If not automatically full event handling, then yes, there would be a
callback to tell the consumer that it should call some event handling
function. I don't think that should be the default though, because
inconvenient.

Now, as to the notion of extending libusb to be able to provide an event
handling thread internally (that actually fully handles events), this I
support.

I'm fully on board with the latter - I too used a thread in a loop to
handle events, though I did use the thread to handle timeouts as well.

..

Post by Orin Eman
if an event handling thread was available in libusb, it is what I would use.

Yep, same here. To be clear, I too prefer the thread to fully take
care of all event handling, ie. actually handle events. :)
//Peter
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
libusb-devel mailing list
https://lists.sourceforge.net/lists/listinfo/libusb-devel

[Andreas Mohr via libusb-devel]

Post by Peter Stuge
Hi,

Post by Chris Dickens

Post by Peter Stuge

Post by Chris Dickens
The libusb_os_handle type will be either an int or HANDLE type for
non-Windows and Windows platforms, respectively.

This is clearly not a cross-platform API, and I consider it a mistake.

I disagree. There is no cross-platform way to represent both a file
descriptor and HANDLE in a single type.

Right, so let's explore other ways to accomplish the same thing. The
intent of libusb APIs was always to be cross-platform, because that's
convenient to use.
I would suggest adding an event handler thread instead. And when it's
useful enable more callback registrations *into* the library, already
a core pattern in the async and hotplug APIs, rather than exporting
event sources *out of* the library.

Drawing in a whopping threading disease dependency into the toolkit itself
might be problematic...
Some people may be using a purely single-threaded app
(with merely *one* since *global* event loop),
some a purely multi-*process* app,
some might be having issues with
potentially onerous threading-related requirements of
e.g. having some things definitely configured *prior* to
launch of first worker (non-primary) thread.


Possibly providing callbacks instead
(which get invoked *directly* in the active context, whichever that may be
in various OSes)
might be a better way of providing notifications -
that way people can (well, "should")
hook into the callbacks and do the kind of
(potentially simple) custom adaptation notification work there
which *their* environment/context requires
in order to "get the message" of these notifications
(of course hopefully without choosing the risk of
callback-type deadlocks due to
trying to take their-layer locks in those callbacks which might already be using
system-layer locks).

IOW, one does *not* throw a whole additional threading layer
into the library core code mix,
but simply
keeps dependencies (library code transitions layering!) down to
strict operation-focussed C language callbacks
as opposed to *adding*
pretty system-specific/onerous thread handling.



[Such callbacks might achieve becoming properly session-instance-distinct
either via
the relatively common void *userdata parameter capability
or via having a struct with various callback pointer members
where the struct object (address...) itself
then is the session-instance-distinct capability.]



[BTW, for an interesting
generic and thus cross-platform event loop solution, see e.g.
Pt toolkit]

Post by Peter Stuge

Post by Chris Dickens
This isn't the first time this problem has been encountered in a
cross-platform library. libcurl does the same thing with sockets
and their curl_socket_t (

Many people smoke, that doesn't make smoking good for the body.

Many people choose to employ threading disease, that doesn't make
threading disease good for the 'puter
(think e.g.
decorations of base APIs with onerous internal locking crap dependencies,
for the sole purpose of
*also* "implicitly" being "thread-safe" *WHERE NEEDED*
*)
- however nowadays fortunately usually
things instead are done via
properly reentrant since session-distinct APIs such as
strerror_r(), strtok_r()).
(which is much better since those have some kind of
properly distinctly reentrancy-storage-supplying
"session"/"instance"/"handle" parameter
rather than always acting on some internal *global* storage)

*) tell-tale sign probably: some many toolkits which
seem to be in need of
providing an "enhanced" (hah!) configuration option
worded similar to "thread-safe APIs"
(e.g. database layers etc.),
simply due to tending towards
allowing weird/questionable multi-context/-instance access
rather than
restricting to one (properly distinct) session instance.

Food for thought,

Andreas Mohr

[Peter Stuge]

Hi Andreas,

Post by Andreas Mohr via libusb-devel
threading disease dependency

Sorry to rain on your parade, but all libusb-1.0 implementations
already depend on a threading library, and hotplug already creates
a thread to catch system notifications.

Post by Andreas Mohr via libusb-devel
might be problematic...

It isn't, once you know how libusb-1.0 implementations use threads.

Post by Andreas Mohr via libusb-devel
Some people may be using a purely single-threaded app
(with merely *one* since *global* event loop),
some a purely multi-*process* app,
some might be having issues with
potentially onerous threading-related requirements of
e.g. having some things definitely configured *prior* to
launch of first worker (non-primary) thread.

I don't see any reason for a problem, so if you can provide a
concrete and detailed example of where an independent thread
(well, two, considering hotplug) for USB events would cause an
actual problem for a library consumer then that would be very
welcome in this discussion. If not, stop trolling, thanks.

Post by Andreas Mohr via libusb-devel
Possibly providing callbacks instead
(which get invoked *directly* in the active context, whichever that
may be in various OSes)

The difference is very small and IMO not relevant, but as I wrote I
think it's fine to make full event handling in the internal thread
optional but disabled by default - which then gives you the callbacks
you want in order to do your synchronization manually.

Given how USB hardware USB system software on the supported OSes and
the libusb-1.0 API works, I see no reason to do things manually, but
I think it would be fine to offer that option if only to silence trolls.

Post by Andreas Mohr via libusb-devel
one does *not* throw a whole additional threading layer
into the library core code mix,

That's nonsense. You're not a strong contributor in this project and
you obviously don't know what you are talking about - since there has
been a dependency on threading for many years now. :)

The wording "threading layer" is also nonsense.

If you can show a concrete example of a problem that's great! Please
do so. It would bring the discussion forward. Blowing smoke as you
just did does not, so let that be.


Thanks a lot!

//Peter

[Kustaa Nyholm]

Post by Peter Stuge
Thanks a lot!

Peter, why not keep the discussion civilised and calm...

cheers Kusti

[Peter Stuge]

Post by Kustaa Nyholm

Post by Peter Stuge
Thanks a lot!

Peter, why not keep the discussion civilised and calm...

I am. It's important to call out and shut down trolls, otherwise the
discussion doesn't move forward. We've seen that happen here before
and that's just a waste of time.

Also, please consider your own bias. You ask me to keep civilised,
yet you didn't react to Andreas throwing the word "diseased" around.
No good.


Let's stay on topic everyone. Thanks a lot!

//Peter

[Andreas Mohr via libusb-devel]

Post by Peter Stuge

Post by Kustaa Nyholm

Post by Peter Stuge
Thanks a lot!

Peter, why not keep the discussion civilised and calm...

I am. It's important to call out and shut down trolls, otherwise the
discussion doesn't move forward. We've seen that happen here before
and that's just a waste of time.

I simply tried to point out that
perhaps it's better to restrict library code/implementation activities to
simple callbacks (they do carry their own drawbacks though),
and then let the user flexibly interface to
his/her chosen event/threading model.

Which, admittedly, is less relevant
once a library already uses threading functionality liberally.

Post by Peter Stuge
Also, please consider your own bias. You ask me to keep civilised,
yet you didn't react to Andreas throwing the word "diseased" around.
No good.

There's an IMHO very valid argument to make that
threading *is* a disease -
it is easily, directly available for everyone to use,
yet then for everyone to shoot various feet
(race conditions left and right)
unless they know how to implement thread-related handling properly.
Threading race conditions are a very relevant developer attention item,
at least in circles where such handling is (too) common.

Andreas Mohr

[eazycnc]

Hi,

this is not really a libusb questions, rather general HID question
but since so many knowledgeable people hang around here I thought
I'd ask here.

I've got piece of code that uses a simple query/response protocol
over HID using output/input reports.

The code sends an output report and waits for an input report and
so on.

The HID device firmware always responds with an input report if
the OS requests it.

As far as I understand this means that if I send an output report
then I cannot now if the next input report (on host side) is in
response to that output report or not. It could be an earlier
input report, right?

There is no guarantee as to how the input reports are polled/
delivered versus when output reports are sent, correct?

Especially considering that I'm using HIDAPI on Mac OS, Linux
and Windows ?

To guard against this I have a byte in the output report that
I increment every time I send an output report, the device
firmware copies that byte to the input report and when I receive
an input report I only accept it if that guard byte matches
the one I sent.

AFAIK I need some mechanism like this, correct?

Now I'm thinking I might change the firmware to only send an
input report if it has previously received an output report,
this would allow me to get rid of the guard byte right?

wbr Kusti

[Peter Stuge]

Post by eazycnc
this is not really a libusb questions, rather general HID question
but since so many knowledgeable people hang around here I thought
I'd ask here.

I for one think that's fine. Please keep in mind to start a new
thread for new topics though.

Post by eazycnc
The HID device firmware always responds with an input report if
the OS requests it.
As far as I understand this means that if I send an output report
then I cannot now if the next input report (on host side) is in
response to that output report or not. It could be an earlier
input report, right?

Exactly right. Also remember that although the two endpoints (IN and
OUT) may have the same address with only the direction bit differing,
they are still independent.

Post by eazycnc
There is no guarantee as to how the input reports are polled/
delivered versus when output reports are sent, correct?

Yes, that's correct. Every endpoint has its own flow.

Post by eazycnc
Especially considering that I'm using HIDAPI on Mac OS, Linux
and Windows ?

Doesn't matter - two endpoints with same address but different direction
are per definition independent.

Post by eazycnc
To guard against this I have a byte in the output report that
I increment every time I send an output report, the device
firmware copies that byte to the input report and when I receive
an input report I only accept it if that guard byte matches
the one I sent.
AFAIK I need some mechanism like this, correct?

Again correct. It is possible and often neccessary to implement some
coupling of the two endpoints in the application, as you have.

Post by eazycnc
Now I'm thinking I might change the firmware to only send an
input report if it has previously received an output report,
this would allow me to get rid of the guard byte right?

Yes, but..

Since you're using HID maybe there is an expectation that HID IN
endpoints always deliver a report (if only in some circumstances) -
I am not sure, I don't know HID well enough to say, I would look into
that before making this change.

Besides HID expectations your plan is solid - the interrupt endpoint
can NAK until there is actual data and that will work just fine as
long as the application is aware that no transfer succeeds (ie. set
the timeout large enough, or infinite :) until the device sends data.

Utilizing NAKs allows more reliable exchange and is a good idea.


//Peter

[Kustaa Nyholm]

Hi Peter,

thanks for the thoughtful answers.

Here is a a thing (or two) that might tip the scale towards my current solution.

1) If I only respond to messages then (a hypothetical user/programmer) will not
get any input reports if he/she does not send an output report. So passive observation
of the device would not be possible. And every status update from the device
would take at least two milliseconds (one for the query and one for the response)
instead of one (on Mac OS I think HID poll period is actually two msec, IIRC).

2) And this is more weighty. I would like indicate to the user via a LED on the
device that the USB communication is working. But if I NAK the input report
'request' unless I get a output report first then how does the device know that
the host USB stack is happy with the device and is requesting those input reports?

Of course if the application is sending output reports then this is no problem
but I would like to separate (to aid diagnoses) these two cases a) host is
communication with the device and b) application is communicating with the device.

This raises one more question. If the device is enumerated and associated with
a HID driver does the host poll the device for input reports even if no application
has currently opened the device for communication?

wbr Kusti

[Peter Stuge]

This post might be inappropriate. Click to display it.

[Tim Roberts]

Post by Kustaa Nyholm
Here is a a thing (or two) that might tip the scale towards my current solution.
1) If I only respond to messages then (a hypothetical user/programmer) will not
get any input reports if he/she does not send an output report. So passive observation
of the device would not be possible.

Well, you need to decide how your device is going to work. I'm not sure
you've really settled the design in your mind. You originally said your
input reports were only sent in response to output reports. If you want
the user to get periodic reports, that's fine, but that's a different
design.

Post by Kustaa Nyholm
2) And this is more weighty. I would like indicate to the user via a LED on the
device that the USB communication is working. But if I NAK the input report
'request' unless I get a output report first then how does the device know that
the host USB stack is happy with the device and is requesting those input reports?

Your device is still going to get IN tokens. It will be up to your
firmware to decide whether to ACK or NAK those tokens, but your device
is certainly going to know that it got tickled.

Post by Kustaa Nyholm
Of course if the application is sending output reports then this is no problem
but I would like to separate (to aid diagnoses) these two cases a) host is
communication with the device and b) application is communicating with the device.

Is this a standard HID device, like a keyboard or a mouse? If this is a
custom device, then the host isn't going to communicate unless an
application is making requests. Where would the responses go? There's
no buffering in the USB stack.

Post by Kustaa Nyholm
This raises one more question. If the device is enumerated and associated with
a HID driver does the host poll the device for input reports even if no application
has currently opened the device for communication?

Not unless there is a client. If you have a keyboard or mouse device,
then there's always a system client and there will always be reads. For
a custom HID device, there is no polling unless someone asks for it.

[eazycnc]

This post might be inappropriate. Click to display it.

[Tim Roberts]

Post by eazycnc
That is what I was uncertain about. In the device side following
is what I think happens. Don't know if this is peculiar to PIC or not.
The SIE receives a package (is that a USB term) and indicates via
a flag that there the data from the transfer available

Sort of. As long as some driver has a read request pending, when your periodic time comes around, the host controller will send an IN token. That's the signal that your device can respond. It either responds with an ACK if it has data, or NAK if it does not.

Post by eazycnc
If I read
that (well that is not necessary) and tell the SIE that I've processed
it then the next time host does a transfer to that end point the
process starts again.
But if I do not tell the SIE i've processed the data in the buffer
the the SIE NAKs when the host tries next transfer.

Now we're getting in to the specifics of your device. With most of the small devices, you put data into a FIFO and "commit it" somehow. When the engine gets an IN token, it responds with ACK or NAK depending on whether the FIFO has data. But USUALLY, there's a way for you to get a callback that a packet was received, even if the SIE has responded automatically.
—
Tim Roberts, ***@probo.com
Providenza & Boekelheide, Inc.

[eazycnc]

<big snap>

Thanks Tim, that cleared things. I think I'll stick with my current
design as it does what I need, has small advantages even if it
sacrificies that one byte of payload to 'sync' input/output
reports.

cheers Kusti

[Ларионов Даниил]

Post by Peter Stuge
stop trolling, thanks.

May I kindly ask that if LibUsb actually makes it to v. 1.1, the inner polling thread will be disableable? In some of my programs I tend to operate hundreds of fd's (including dozens of usb devices), and I prefer to use a finite state machine and one thread instead of multithreading cancer.

("A Computer is a state machine. Threads are for people who can't program state machines.")
--
Даниил Ларионов

Post by Peter Stuge
Hi Andreas,

 threading disease dependency

Sorry to rain on your parade, but all libusb-1.0 implementations
already depend on a threading library, and hotplug already creates
a thread to catch system notifications.

 might be problematic...

It isn't, once you know how libusb-1.0 implementations use threads.

 Some people may be using a purely single-threaded app
 (with merely *one* since *global* event loop),
 some a purely multi-*process* app,
 some might be having issues with
 potentially onerous threading-related requirements of
 e.g. having some things definitely configured *prior* to
 launch of first worker (non-primary) thread.

I don't see any reason for a problem, so if you can provide a
concrete and detailed example of where an independent thread
(well, two, considering hotplug) for USB events would cause an
actual problem for a library consumer then that would be very
welcome in this discussion. If not, stop trolling, thanks.

 Possibly providing callbacks instead
 (which get invoked *directly* in the active context, whichever that
 may be in various OSes)

The difference is very small and IMO not relevant, but as I wrote I
think it's fine to make full event handling in the internal thread
optional but disabled by default - which then gives you the callbacks
you want in order to do your synchronization manually.
Given how USB hardware USB system software on the supported OSes and
the libusb-1.0 API works, I see no reason to do things manually, but
I think it would be fine to offer that option if only to silence trolls.

 one does *not* throw a whole additional threading layer
 into the library core code mix,

That's nonsense. You're not a strong contributor in this project and
you obviously don't know what you are talking about - since there has
been a dependency on threading for many years now. :)
The wording "threading layer" is also nonsense.
If you can show a concrete example of a problem that's great! Please
do so. It would bring the discussion forward. Blowing smoke as you
just did does not, so let that be.
Thanks a lot!
//Peter
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
libusb-devel mailing list
https://lists.sourceforge.net/lists/listinfo/libusb-devel

[Chris E]

Dont agree with the tone necessarily, but do agree with the general point.
Why use multiple threads unless you really have to?

Post by Ларионов Даниил

Post by Peter Stuge
stop trolling, thanks.

May I kindly ask that if LibUsb actually makes it to v. 1.1, the inner
polling thread will be disableable? In some of my programs I tend to
operate hundreds of fd's (including dozens of usb devices), and I prefer
to use a finite state machine and one thread instead of multithreading
cancer.
("A Computer is a state machine. Threads are for people who can't program state machines.")
--
ДаМООл ЛарОПМПв

Post by Peter Stuge
Hi Andreas,

Post by Andreas Mohr via libusb-devel
threading disease dependency

Sorry to rain on your parade, but all libusb-1.0 implementations
already depend on a threading library, and hotplug already creates
a thread to catch system notifications.

Post by Andreas Mohr via libusb-devel
might be problematic...

It isn't, once you know how libusb-1.0 implementations use threads.

Post by Andreas Mohr via libusb-devel
Some people may be using a purely single-threaded app
(with merely *one* since *global* event loop),
some a purely multi-*process* app,
some might be having issues with
potentially onerous threading-related requirements of
e.g. having some things definitely configured *prior* to
launch of first worker (non-primary) thread.

I don't see any reason for a problem, so if you can provide a
concrete and detailed example of where an independent thread
(well, two, considering hotplug) for USB events would cause an
actual problem for a library consumer then that would be very
welcome in this discussion. If not, stop trolling, thanks.

Post by Andreas Mohr via libusb-devel
Possibly providing callbacks instead
(which get invoked *directly* in the active context, whichever that
may be in various OSes)

The difference is very small and IMO not relevant, but as I wrote I
think it's fine to make full event handling in the internal thread
optional but disabled by default - which then gives you the callbacks
you want in order to do your synchronization manually.
Given how USB hardware USB system software on the supported OSes and
the libusb-1.0 API works, I see no reason to do things manually, but
I think it would be fine to offer that option if only to silence trolls.

Post by Andreas Mohr via libusb-devel
one does *not* throw a whole additional threading layer
into the library core code mix,

That's nonsense. You're not a strong contributor in this project and
you obviously don't know what you are talking about - since there has
been a dependency on threading for many years now. :)
The wording "threading layer" is also nonsense.
If you can show a concrete example of a problem that's great! Please
do so. It would bring the discussion forward. Blowing smoke as you
just did does not, so let that be.
Thanks a lot!
//Peter
------------------------------------------------------------

------------------

Post by Peter Stuge
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
libusb-devel mailing list
https://lists.sourceforge.net/lists/listinfo/libusb-devel

------------------------------------------------------------
------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
libusb-devel mailing list
https://lists.sourceforge.net/lists/listinfo/libusb-devel

[Ларионов Даниил]

Post by Chris E
Dont agree with the tone necessarily

Of course I didn't intend to offend anyone. Making a thread and calling some of the libusb_handle_events* inside it is a perfectly valid strategy and I use it myself in smaller programs, and I think moving this functionality inside the library will improve newly created code. It's just that sometimes, multithreading *does* become cancerous and you have to avoid it.

-- 
Даниил Ларионов

Post by Chris E
Dont agree with the tone necessarily, but do agree with the general point.  Why use multiple threads unless you really have to?

Post by Peter Stuge
stop trolling, thanks.

May I kindly ask that if LibUsb actually makes it to v. 1.1, the inner polling thread will be disableable? In some of my programs I tend to operate hundreds  of fd's (including dozens of usb devices), and I prefer to use a finite state machine and one thread instead of multithreading cancer.
("A Computer is a state machine. Threads are for people who can't program state machines.")
--
Даниил Ларионов

Post by Peter Stuge
Hi Andreas,

 threading disease dependency

Sorry to rain on your parade, but all libusb-1.0 implementations
already depend on a threading library, and hotplug already creates
a thread to catch system notifications.

 might be problematic...

It isn't, once you know how libusb-1.0 implementations use threads.

 Some people may be using a purely single-threaded app
 (with merely *one* since *global* event loop),
 some a purely multi-*process* app,
 some might be having issues with
 potentially onerous threading-related requirements of
 e.g. having some things definitely configured *prior* to
 launch of first worker (non-primary) thread.

I don't see any reason for a problem, so if you can provide a
concrete and detailed example of where an independent thread
(well, two, considering hotplug) for USB events would cause an
actual problem for a library consumer then that would be very
welcome in this discussion. If not, stop trolling, thanks.

 Possibly providing callbacks instead
 (which get invoked *directly* in the active context, whichever that
 may be in various OSes)

The difference is very small and IMO not relevant, but as I wrote I
think it's fine to make full event handling in the internal thread
optional but disabled by default - which then gives you the callbacks
you want in order to do your synchronization manually.
Given how USB hardware USB system software on the supported OSes and
the libusb-1.0 API works, I see no reason to do things manually, but
I think it would be fine to offer that option if only to silence trolls.

 one does *not* throw a whole additional threading layer
 into the library core code mix,

That's nonsense. You're not a strong contributor in this project and
you obviously don't know what you are talking about - since there has
been a dependency on threading for many years now. :)
The wording "threading layer" is also nonsense.
If you can show a concrete example of a problem that's great! Please
do so. It would bring the discussion forward. Blowing smoke as you
just did does not, so let that be.
Thanks a lot!
//Peter
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
libusb-devel mailing list
https://lists.sourceforge.net/lists/listinfo/libusb-devel

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
libusb-devel mailing list
https://lists.sourceforge.net/lists/listinfo/libusb-devel

[Chris Dickens]

Hi,

Post by Peter Stuge
Right, so let's explore other ways to accomplish the same thing. The
intent of libusb APIs was always to be cross-platform, because that's
convenient to use.

You are intent on arguing that a platform-specific type definition is not
cross-platform. What then is your definition of cross-platform? The
proposed change will not inhibit libusb's ability to run on any of the
currently supported platforms but on the contrary will extend some of the
functionality available on a subset of supported platforms to *all*
supported platforms. Sounds to me like it is actually becoming *more*
cross-platform :)

I would suggest adding an event handler thread instead. And when it's

Post by Peter Stuge
useful enable more callback registrations *into* the library, already
a core pattern in the async and hotplug APIs, rather than exporting
event sources *out of* the library.

Keep in mind that there are applications that truly do want this
"exporting" of event sources out of libusb, for the exact reasons
documented. There shouldn't need to be a dedicated thread in the
application (or library) for libusb's events if the library provides a way
of integrating the event sources into the application's event handling
loop. Some applications are sensitive to how and when libusb events are
handled, especially the context (thread) on which libusb's callbacks are
executed.

I've seen both Linux and Windows application dynamically load the

Post by Peter Stuge
library at runtime.

As you must know, dynamically loaded libraries require each symbol to be
looked up at runtime also, thus for the same reason as I stated earlier
this lookup will fail.

Regards,

Chris

[Ларионов Даниил]

Post by Peter Stuge
enable more callback registrations *into* the library, already
a core pattern in the async and hotplug APIs, rather than exporting
event sources *out of* the library.

Keep in mind that there are applications that truly do want this "exporting" of event sources out of libusb, for the exact reasons documented. There shouldn't need to be a dedicated thread in the application (or library) for libusb's events if the library provides a way of integrating the event sources into the application's event handling loop. Some applications are sensitive to how and when libusb events are handled, especially the context (thread) on which libusb's callbacks are executed.

Thanks a lot. This is exactly what I was trying to convey.

The former has the benefit of hiding both internal libusb resources _and_ the method used to determine something is pending, whether it be poll() or WaitForMultipleEvents() - which is the point of the exercise - hiding OS dependencies. It would be trivial for a callback to write to a pipe or socket to wake up a main context via poll() - and would likely eliminate the need for most of the 16 libusb calls to do with poll fds/event handling.

Do I understand you correctly that you are proposing to use poll() inside the library to wake up the inner thread, which will call a callback, which will write to a pipe, which will wake up *another* poll() in the user application - instead of just calling poll() in the user application - just to avoid exporting the library's event sources?..


-- 
Даниил Ларионов

Hi,

Post by Peter Stuge
Right, so let's explore other ways to accomplish the same thing. The
intent of libusb APIs was always to be cross-platform, because that's
convenient to use.

You are intent on arguing that a platform-specific type definition is not cross-platform. What then is your definition of cross-platform? The proposed change will not inhibit libusb's ability to run on any of the currently supported platforms but on the contrary will extend some of the functionality available on a subset of supported platforms to *all* supported platforms. Sounds to me like it is actually becoming *more* cross-platform :)

Post by Peter Stuge
I would suggest adding an event handler thread instead. And when it's
useful enable more callback registrations *into* the library, already
a core pattern in the async and hotplug APIs, rather than exporting
event sources *out of* the library.

Keep in mind that there are applications that truly do want this "exporting" of event sources out of libusb, for the exact reasons documented. There shouldn't need to be a dedicated thread in the application (or library) for libusb's events if the library provides a way of integrating the event sources into the application's event handling loop. Some applications are sensitive to how and when libusb events are handled, especially the context (thread) on which libusb's callbacks are executed.

Post by Peter Stuge
I've seen both Linux and Windows application dynamically load the
library at runtime.

As you must know, dynamically loaded libraries require each symbol to be looked up at runtime also, thus for the same reason as I stated earlier this lookup will fail.
Regards,
Chris
,
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
,
_______________________________________________
libusb-devel mailing list
https://lists.sourceforge.net/lists/listinfo/libusb-devel

[Ларионов Даниил]

But no-one is going to remove the existing support - the suggested internal event handling thread will be new, optional and allow platform independent coding.

This is all that I wanted to hear. Thanks.
If libusb v.1.1 will allow to export pollable fd's, then everything's perfect. Because if not, I'll have to stick to 1.0.

-- 
Даниил Ларионов

Post by Ларионов Даниил

Post by Peter Stuge
enable more callback registrations *into* the library, already
a core pattern in the async and hotplug APIs, rather than exporting
event sources *out of* the library.

Keep in mind that there are applications that truly do want this "exporting" of event sources out of libusb, for the exact reasons documented. There shouldn't need to be a dedicated thread in the application (or library) for libusb's events if the library provides a way of integrating the event sources into the application's event handling loop. Some applications are sensitive to how and when libusb events are handled, especially the context (thread) on which libusb's callbacks are executed.

Thanks a lot. This is exactly what I was trying to convey.

The former has the benefit of hiding both internal libusb resources _and_ the method used to determine something is pending, whether it be poll() or WaitForMultipleEvents() - which is the point of the exercise - hiding OS dependencies.  It would be trivial for a callback to write to a pipe or socket to wake up a main context via poll() - and would likely eliminate the need for most of the 16 libusb calls to do with poll fds/event handling.

Do I understand you correctly that you are proposing to use poll() inside the library to wake up the inner thread, which will call a callback, which will write to a pipe, which will wake up *another* poll() in the user application - instead of just calling poll() in the user application - just to avoid exporting the library's event sources?..

Yes, it's truly disgusting isn't it?
And it's about as disgusting as trying to ram poll() semantics down the throats of those that are trying to use libusb on Windows.
Remember, we are talking about platform independence and poll() simply doesn't exist on Windows, so a good cross-platform solution does not involve revealing pollable fds, or indeed any internal libusb resource.
But no-one is going to remove the existing support - the suggested internal event handling thread will be new, optional and allow platform independent coding.

[Tim Roberts]

Post by Peter Stuge

Post by Chris Dickens
The libusb_os_handle type will be either an int or HANDLE type for
non-Windows and Windows platforms, respectively.

This is clearly not a cross-platform API, and I consider it a mistake.

There are precedents. In the C++ 11 standard library, which is
obviously cross-platform, the std::thread object contains a
native_handle() method that serves exactly the same purpose.

I'm not arguing for the entire set of changes, but the concept of
providing a native handle is not inherently unreasonable.

[Vincent Pelletier]

Hello,

On Wed, 5 Jul 2017 11:48:52 -0700, Chris Dickens

Post by Chris Dickens
I don't love it but don't have a better one.

I feel this is too short and generic when taken out of context (as I
believe it will exist outside of the struct context, at least declared
if not used), so what about repeating a bit from the general
functionality name ? I'm thinking of something like
libusb_os_event_source_handle or libusb_event_source_os_handle.

I am not a libusb dev, merely author of a python wrapper, so please take
my opinion with a grain of salt.

Regards,

[Chris Dickens]

Post by Vincent Pelletier
I feel this is too short and generic when taken out of context (as I
believe it will exist outside of the struct context, at least declared
if not used), so what about repeating a bit from the general
functionality name ? I'm thinking of something like
libusb_os_event_source_handle or libusb_event_source_os_handle.

Now that I think about it, how about just libusb_event_source? Simple,
concise and it describes exactly what the data represents.

Regards,

Chris

[Andreas Mohr via libusb-devel]

On Thu, Jul 6, 2017 at 4:51 PM Vincent Pelletier
I feel this is too short and generic when taken out of context (as I
believe it will exist outside of the struct context, at least declared
if not used), so what about repeating a bit from the general
functionality name ? I'm thinking of something like
libusb_os_event_source_handle or libusb_event_source_os_handle.
Now that I think about it, how about just libusb_event_source? Simple,
concise and it describes exactly what the data represents.

As long as libusb can live with this name then being
entirely non-scoped/-specified
(result: becoming a "globally" used symbol - at least on the naming side of
things)...
(but this symbol being used in fully project-global form might exactly be
what one wants).

Not sure about the "source" part, though.
What else other than a "source" is an event handle?
I.e., the name could possibly simply be libusb_events.
OTOH without source suffix it then will be less grep:able.
Plus, a "libusb_events" name might be confused with
some sorts of event *types* enumeration.
And the symbol name isn't overly long, so there's no penalty in
adding a _source suffix.

*Not* a libusb developer (take my input with a *large* grain of salt).

Andreas Mohr