Author Topic: libhpcalcs: a toolkit for communicating with Prime calcs...  (Read 17654 times)

0 Members and 1 Guest are viewing this topic.

Offline Lionel Debroux

  • LV11 Super Veteran (Next: 3000)
  • ***********
  • Posts: 2135
  • Rating: +290/-45
    • View Profile
    • TI-Chess Team
libhpcalcs: a toolkit for communicating with Prime calcs...
« on: November 09, 2013, 04:20:26 pm »
See the end of the post for download links.

Over the past few weeks, as mentioned at http://tiplanet.org/forum/viewtopic.php?f=55&t=13240 and http://ourl.ca/19769 , I've been working on a toolkit for communicating with HP Prime calculators, strongly inspired by libti* for TI calculators, and unsurprisingly dubbed libhpcalcs. It can be downloaded from https://github.com/debrouxl/hplp .
Thanks to critor's USB dumps and tests (EDIT: and later, other persons' tests), we believe the code now works well enough for being beta-tested used by a wider public (on Windows, Linux and MacOS X) :)
While neither critor, nor I, believe that anything can make a significant dent into TI's market share because TI is too entrenched and has lobbying power, we're nevertheless trying to do something with the Prime... as it's clear that if nobody bothers, the Prime won't stand a chance to get widely used (chicken-and-egg problem).


"Toolkit for communicating with HP calculators" means that it's a library backend for people to build GUIs on top of it, but libhpcalcs itself is a GUI-less library, and its terminal-based test program is currently the only UI for using libhpcalcs...

Excerpt from the README about the library's current capabilities:
Quote
The code base does:
* support communication with a single Prime calculator connected to the computer. The communication is known to work with the "SDKV0.30" firmware version from mid-August 2013, but might not work with other older or newer versions, if HP performs backwards-incompatible changes in the protocol (like TI does);
* implement six operations (without _known_ bugs): ready check, get calculator information, receive screenshot, send file, receive file, receive backup, set date and time;
* provide a terminal-based UI: the test program "test_hpcalcs".

The code base doesn't:
* perform color conversion on PNG screenshots (libhpcalcs should depend on libpng to uncompress the files, and recompress them after mangling);
* provide a way to select one of multiple HP calculators (probing USB devices);
* provide a user-friendly GUI for the above, but a GUI can definitely build upon the library easily enough;
* implement _proper_ conversion from UTF-16 LE to other charsets on its own: that's a task for libicu, Qt or other libraries;

What's next ?
* first of all, making sure that people are interested in the work in the first place. I'm clearly not going to keep spending most of my free time (as I did over the past week) on something people do not find useful ! [EDIT a couple weeks later: seems that some people are interested :)]
* fixing bugs which are bound to be reported, none are known right now but there certainly are some :)
* implementing several extra features from the TODO list embedded into the README, starting with probing devices;
* implementing a GUI for the program (with Qt, which has become the main choice for a portable, good-looking, fast UI toolkit and software abstraction layer)... but I don't plan on doing it by myself, although I'll obviously gladly support anyone trying to do so.
* implementing 39gII support ?
* certainly more... again, iff people are interested.


Go forth and test, thanks in advance ;)


Development source code: https://github.com/debrouxl/hplp (master branch is stable, master2 contains changes for testing, which may be rewritten before integration to master).
Ready-made install script: https://raw.github.com/debrouxl/hplp/master/install_hplp.sh . Unless your distro packages hidapi (Debian and derivatives do not), you'll have to compile it yourself (autotools-based program, so configure && make && sudo make install).
Latest Windows binaries + source tarball: http://tiplanet.org/beta/libhpcalcs-0.0.1-package.zip .

Also, I'm looking for 39gII users who can tolerate command-line tools, for providing me USB dumps (usbpcap) from their Windows computers when running the HP Connectivity Kit, the way critor did for the Prime.

[latest EDIT: 2013/11/16]
« Last Edit: November 16, 2013, 03:08:21 pm by Lionel Debroux »
Member of the TI-Chess Team.
Co-maintainer of GCC4TI (GCC4TI online documentation), TILP and TIEmu.
Co-admin of TI-Planet.

Offline DJ Omnimaga

  • Former TI programmer
  • CoT Emeritus
  • LV15 Omnimagician (Next: --)
  • *
  • Posts: 55925
  • Rating: +3152/-232
  • CodeWalrus founder & retired Omnimaga founder
    • View Profile
    • DJ Omnimaga Music
Re: libhpcalcs: a toolkit for communicating with Prime calcs...
« Reply #1 on: November 09, 2013, 06:18:26 pm »
While neither critor, nor I, believe that anything can make a significant dent into TI's market share because TI is too entrenched and has lobbying power
Not to go too much off-topic nor to offend anybody, but in Critor's case, I always wondered if his negative opinion about the Prime was influenced more by the presence of a certain troublemaker in the HP community (although that person did really bad things such as real life threats towards TI-Planet admins, I feel it would be unfair if HP and the TI community that tries to expands its horizons were punished for the actions of one person). A lot of his comments about the Prime were not very positive, despite its potential, and I felt it wasn't only due to a certain bias towards TI.


Now back on-topic, I'm glad this project is coming along and that some people supports the calc. I will probably try to test it when I have some time, providing that it isn't command-line based, isn't a major PITA to install and doesn't require 2000 dependencies (as I am not very tech-savy, I don't want to spend two weeks trying to install a software like I did many times before, halting my progress on other projects x.x). I wouldn't mind giving you an USB dump of my HP 39gII at one point, though, so that you can add 39gII support.

By the way, the official 39gII connectivity kit or OS doesn't even support file transfers properly, because any program you download from the computer to your calculator will freeze the calc. <_<.
« Last Edit: November 22, 2013, 01:41:00 am by DJ Omnimaga »


Offline Lionel Debroux

  • LV11 Super Veteran (Next: 3000)
  • ***********
  • Posts: 2135
  • Rating: +290/-45
    • View Profile
    • TI-Chess Team
Re: libhpcalcs: a toolkit for communicating with Prime calcs...
« Reply #2 on: November 10, 2013, 01:57:17 am »
Quote
I will probably try to test it when I have some time, providing that it isn't command-line based,
test_hpcalcs is terminal-based, though interactive.
There's mild point in spending time developing a GUI for some code backend nobody cares about in the first place - and so far, hardly anybody cares about libhpcalcs: no reply on TI-Planet (where I posted a week ago), no reply on Cemetech, only your reply here, and a single beta-tester (besides critor) so far.

Quote
isn't a major PITA to install and doesn't require 2000 dependencies
Well, I provided ready-made Windows binaries, just unzip the ZIP file attached to my previous post ;)
libpng needs to be added as a dependency at some point... but only if the program has actual users. For now, that's not a given.
Member of the TI-Chess Team.
Co-maintainer of GCC4TI (GCC4TI online documentation), TILP and TIEmu.
Co-admin of TI-Planet.

Offline DJ Omnimaga

  • Former TI programmer
  • CoT Emeritus
  • LV15 Omnimagician (Next: --)
  • *
  • Posts: 55925
  • Rating: +3152/-232
  • CodeWalrus founder & retired Omnimaga founder
    • View Profile
    • DJ Omnimaga Music
Re: libhpcalcs: a toolkit for communicating with Prime calcs...
« Reply #3 on: November 10, 2013, 02:05:59 am »
Yeah true, and just take a look at my 3D graphics topic, it was pretty much ignored since it was posted, with very few people checking and not bothering to reply. I don't know if it's because of site navigation issues (for example, there are no links to those sub-forums on the front page), but that is kinda worrying, especially considering that some of the people who checked the thread without replying already have an HP Prime.

« Last Edit: November 10, 2013, 02:09:35 am by DJ Omnimaga »


Offline Streetwalrus

  • LV12 Extreme Poster (Next: 5000)
  • ************
  • Posts: 3821
  • Rating: +80/-8
    • View Profile
Re: libhpcalcs: a toolkit for communicating with Prime calcs...
« Reply #4 on: November 10, 2013, 03:59:20 am »
Sounds pretty good to me. Since I'm on Linux and I plan on getting a Prime, it's a no brainer for me. I'm willing to help develop a GUI for it once it's matured, but a set of CLI progs in the titools style would be cool too.

Offline Lionel Debroux

  • LV11 Super Veteran (Next: 3000)
  • ***********
  • Posts: 2135
  • Rating: +290/-45
    • View Profile
    • TI-Chess Team
Re: libhpcalcs: a toolkit for communicating with Prime calcs...
« Reply #5 on: November 10, 2013, 04:10:37 am »
Definitely, like libti*, libhpcalcs aims at being a backend for both CLI programs and GUI programs.

http://desowin.org/usbpcap/tour.html gives some information about capturing USB packets.
Member of the TI-Chess Team.
Co-maintainer of GCC4TI (GCC4TI online documentation), TILP and TIEmu.
Co-admin of TI-Planet.

Offline Lionel Debroux

  • LV11 Super Veteran (Next: 3000)
  • ***********
  • Posts: 2135
  • Rating: +290/-45
    • View Profile
    • TI-Chess Team
Re: libhpcalcs: a toolkit for communicating with Prime calcs...
« Reply #6 on: November 10, 2013, 04:18:52 pm »
I have fixed several bugs reported on TI-Planet, and performed large changes on error codes, which I had started before the bug reports. The code base doesn't seem to be completely broken, according to critor's tests (upwards propagation of errors was already mostly done).
For now, until further testing, changes were pushed to a new "master2" branch. You need to pull and explicitly checkout the branch to be able to test the new changes.

New Windows binaries + source tarball (hopefully) attached.

Thanks in advance ;)

[EDIT: removing extraneous file, I attached it twice]
« Last Edit: November 10, 2013, 04:19:22 pm by Lionel Debroux »
Member of the TI-Chess Team.
Co-maintainer of GCC4TI (GCC4TI online documentation), TILP and TIEmu.
Co-admin of TI-Planet.

Offline DJ Omnimaga

  • Former TI programmer
  • CoT Emeritus
  • LV15 Omnimagician (Next: --)
  • *
  • Posts: 55925
  • Rating: +3152/-232
  • CodeWalrus founder & retired Omnimaga founder
    • View Profile
    • DJ Omnimaga Music
Re: libhpcalcs: a toolkit for communicating with Prime calcs...
« Reply #7 on: November 13, 2013, 10:02:43 pm »
1: Check ready success

2: Get infos success

3: This one doesn't seem to work: No matter what I type as format choice ("png", "bmp","gif", or fully spelled out or with a dot at the start), It just says "Canceled" ???

4: Haven't tested yet since I don't know the syntax

5: Same

6: After one minute with nothing happening, this is what I got:
Quote
hpcalcs_calc_recv_backup failed
0 TODO human-readable error message

7: Not sure how this works, but I typed 1 as ID then it said prime_send_data succeeded


It might be good to include more documentation on how to use this, with examples, as I seemed to have absolutely no clue about what I am doing.
« Last Edit: November 13, 2013, 10:05:06 pm by DJ Omnimaga »


Offline Lionel Debroux

  • LV11 Super Veteran (Next: 3000)
  • ***********
  • Posts: 2135
  • Rating: +290/-45
    • View Profile
    • TI-Chess Team
Re: libhpcalcs: a toolkit for communicating with Prime calcs...
« Reply #8 on: November 14, 2013, 02:13:32 am »
Thanks.

3: in fact, the calculator produces only PNG images, and one needs to enter 8, 9, 10 or 11 to get screenshots in two resolutions and color depths (all of which need unimplemented color conversion). But only the source code tells that story...

6: could you send me the "trace.txt" file which was created in the same folder ?
Lack of user feedback about operations which are longer than a second or two is a major utilisability no-no, but I haven't implemented any form of progress callbacks yet. When I do, the callback system will be different from that of upstream (EDIT: I mean libticalcs).
Finally, backuping a calculator is slow, due to HID being slow, I can't do anything about that.

In fact, test_hpcalcs is only a test client (derived from test_ticalcs_2), aimed at making small tests and showing how to use the library's functions. If it were to stay the only user-facing tool, it would badly need some documentation indeed. The library aims at being the backend for higher-level GUI and CLI tools; I just hope someone will take on the task of writing a GUI for a backend which largely works...


Offline, I have started implementing error messages and i18n, but I don't think i18n actually works yet, and I think I know why.
« Last Edit: November 14, 2013, 02:28:16 am by Lionel Debroux »
Member of the TI-Chess Team.
Co-maintainer of GCC4TI (GCC4TI online documentation), TILP and TIEmu.
Co-admin of TI-Planet.

Offline DJ Omnimaga

  • Former TI programmer
  • CoT Emeritus
  • LV15 Omnimagician (Next: --)
  • *
  • Posts: 55925
  • Rating: +3152/-232
  • CodeWalrus founder & retired Omnimaga founder
    • View Profile
    • DJ Omnimaga Music
Re: libhpcalcs: a toolkit for communicating with Prime calcs...
« Reply #9 on: November 14, 2013, 02:26:33 am »
Ok thanks for the info. I'll e-mail you a copy of my trace.txt file now.

By the way, how long should it take to backup a calculator that has 400 KB of data? (Only a single program, which shows 63 KB, but I'm sure the program menu displays wrong sizes, since it's over 400 KB on the computer in unicode text form)

EDIT: Screenshots works (they are discolored as we know, but at least I got no error)
« Last Edit: November 14, 2013, 03:19:45 am by DJ Omnimaga »


Offline Lionel Debroux

  • LV11 Super Veteran (Next: 3000)
  • ***********
  • Posts: 2135
  • Rating: +290/-45
    • View Profile
    • TI-Chess Team
Re: libhpcalcs: a toolkit for communicating with Prime calcs...
« Reply #10 on: November 15, 2013, 03:29:58 am »
The trace shows that there have been a number of silent packet losses during transfer. Besides being fundamentally slow due to usage of HID and delays (even if seemingly still too fast for Windows' crappy USB stack on your computer - maybe some driver or anti-virus is interfering, as Tim Wessman doesn't witness slowness to the extent you do ?), the protocol used by HP isn't robust wrt. packet losses...
The computer sends increasing first bytes, but the calculator does not, so packet losses are harder to detect when the calculator is the sender (more precisely, it's basically impossible detect where data was lost, without more intimate knowledge of formats).

This means that I'll have to rewrite recv_backup functionality.
While triggering recv_file in a loop works most of the time, it falls apart if there are any packet losses. Receiving as much data as the calculator sends, then attempting to split it after the fact, would make it possible to salvage more data, though the heuristics might fail in very infrequent cases.
Another usability problem with the current recv_backup implementation is that in case of an error, it destroys any data received from the calculator before returning to the caller. Users would probably rather have partial, corrupted backups than nothing :)


To reduce the risks of your largest programs being lost as a result of a combination of transfer failure (induced by packet losses, statistically more likely on large programs) + calculator-side bugs (some of which require erasing stuff through the maintenance menus), you'll have no choice but to split them somehow. I don't know how.
Member of the TI-Chess Team.
Co-maintainer of GCC4TI (GCC4TI online documentation), TILP and TIEmu.
Co-admin of TI-Planet.

Offline DJ Omnimaga

  • Former TI programmer
  • CoT Emeritus
  • LV15 Omnimagician (Next: --)
  • *
  • Posts: 55925
  • Rating: +3152/-232
  • CodeWalrus founder & retired Omnimaga founder
    • View Profile
    • DJ Omnimaga Music
Re: Re: libhpcalcs: a toolkit for communicating with Prime calcs...
« Reply #11 on: November 15, 2013, 11:57:45 am »
Yeah I need to check if it's possible to use sub-programs on the Prime. I hope a solution is found, though, else if absolutely no large game can be sent, then this calc will be a failure programming-community-wise.


Offline DJ Omnimaga

  • Former TI programmer
  • CoT Emeritus
  • LV15 Omnimagician (Next: --)
  • *
  • Posts: 55925
  • Rating: +3152/-232
  • CodeWalrus founder & retired Omnimaga founder
    • View Profile
    • DJ Omnimaga Music
Re: libhpcalcs: a toolkit for communicating with Prime calcs...
« Reply #12 on: November 22, 2013, 01:38:07 am »
On a side note, I might try to get new batteries to put in my 39gII so if you need an USB dump and have good instructions on how to do it, I think I have three days off this weekend so maybe I could do one.

This is in response to http://www.hpmuseum.org/cgi-sys/cgiwrap/hpmuseum/forum.cgi?read=256650#256650 , but since I am unable to register there, I had to reply here.
« Last Edit: November 22, 2013, 01:38:48 am by DJ Omnimaga »


Offline Lionel Debroux

  • LV11 Super Veteran (Next: 3000)
  • ***********
  • Posts: 2135
  • Rating: +290/-45
    • View Profile
    • TI-Chess Team
Re: libhpcalcs: a toolkit for communicating with Prime calcs...
« Reply #13 on: November 22, 2013, 01:51:37 am »
The instructions that I know of are at http://desowin.org/usbpcap/tour.html . Only steps 1 and 2 are required on your side, step 3 is my job :)
Step 4 is an option, but then, you'd have to install Wireshark and produce files from Wireshark, which is more complicated.

Thanks.
Member of the TI-Chess Team.
Co-maintainer of GCC4TI (GCC4TI online documentation), TILP and TIEmu.
Co-admin of TI-Planet.

Offline critor

  • Editor
  • LV11 Super Veteran (Next: 3000)
  • ***********
  • Posts: 2079
  • Rating: +439/-13
    • View Profile
    • TI-Planet
Re: libhpcalcs: a toolkit for communicating with Prime calcs...
« Reply #14 on: November 22, 2013, 06:56:18 pm »
Due to Lionel being now banned from Omnimaga, further comments about HPLP adressed to the author will have to be posted on TI-Planet in the matching topic:
http://tiplanet.org/forum/viewtopic.php?f=69&p=152761#p152761

Don't worry, all the team speaks english there too ;)
« Last Edit: November 22, 2013, 07:18:36 pm by critor »
TI-Planet co-admin.