Omnimaga
Calculator Community => HP Calculators => Topic started by: DJ Omnimaga on October 20, 2013, 06:43:04 pm
-
As you might have noticed, when you connect your HP Prime to the computer then open the Connectivity Kit software, it takes about 10 minutes before the software has finished displaying the entire calculator content. It is not as bad when you have a small amount of programs (takes around 30 seconds), but when you have about 30 KB of them, 10 minutes is what it takes (with some chances of the connection dying in the meantime), meaning that if you create a game that is 500 KB large, you'll most likely have to wait several hours before the software finally detects it, pretty much rendering transfer of any large files (or small files from an overcrowded calc) impossible.
In addition to that, the connectivity kit software itself cannot transfer files. The only way is to open the program code in the software, copy its entire content, then create a new program on the calculator and paste the code in there. Once done, you hit save, but the issue is that it takes about 30 seconds per kilobytes before the updated program finally appears on the calc and there is nothing telling you it's actually sending at all!
Also, in the connectivity kit, you cannot transfer anything to the calc until the software is done 100% reading the calc content. If you want to backup often, you better wipe out your entire calc content and only keep what you're working on.
The alternate way is to use the HP Prime emulator, which sends files directly to the calc. However, it comes with the same problem about slow file transfers and 90% of the time the emulator will stop detecting the calc after one minute or so, meaning you have to disconnect and reconnect it over and over and making the transfer of any larger file impossible.
Since it is unclear yet if this software will be updated anytime soon and work better, I wonder if it would be best to have third-party tools, such as a HPLP software (in reference to the TILP name)? Anyway I saw this post on TI-Planet at http://tiplanet.org/forum/viewtopic.php?p=150590#p150689 and I think that Lionel and Critor might have started doing so, so I thought I would bring this up in case anyone with the knowledge or time to do it would be willing to help. Here is the current Github project URL (for now): https://github.com/debrouxl/hpcomm
I wonder if this software would manage to get around the long delays for file transfers too?
-
Thanks. I have just renamed the repository to hplp: https://github.com/debrouxl/hplp , which makes more sense...
This morning, I split the code across multiple files, introducing the exact same organization as libticables + libticalcs. Not that it makes the program any closer to user consumption, but it makes the code base nicer.
-
Cool to hear. Apparently Tim didn't have this slowness issue I mentionned above, so I'll send him a program that fails to send at a reasonable speed. It was just a program that scrolled through the entire Super Sonic Ball sprite sheet and it was over 10 KB large, but I had a different version with updated sprites that was much larger (around 18-28 KB IIRC). I hope it's not computer compatibility issues >.< (I use Windows 7 64 bits and a first gen i7 computer from 2010).
-
I don't remember exactly what I did, But I managed to put files onto my Prime, after I copied it and pasted into some sort of folder. Took me no time at all...
Hmmm I wish I could remember what I did...
EDIT:Oh I just put the program into folder in the files for the connectivity kit and then right click on it in connectivity and hit send to class.
(You might have just said that, but I didn't understand)
-
How large were your files btw?
Also what is the send to class function? ???
-
Here this is what I do.
As far as I can tell it's instantaneous.
-
Oh right, that part. I wonder, though... what is the speed difference between that and just sending normally via the emulator or other methods?
-
Not a clue, just works for me XD.
Turns out asteroids doesn't work.
Flashes then crashes.
-
Strange, I had no problem with it. ??? Key detection was weird but that's due to its programming.
EDIT: Btw where can I find the calc folders in the connectivity kit? I use Windows 7 and parts of your path isn't visible in the screenshot.
EDIT: By the way, I just ran another test and to read through 86 KB of memory content, the connectivity kit took me about 8 minutes (before I could finally update any program from there). Once done, I tried to send the Super Sonic Ball update that is 46 KB large and it took about 1 minute. I think the long delays I got might have been due to the connectivity kit not having been done reading my calc yet, but even 1 minute is still rather slow when you know that many games will reach hundreds of KB of memory.
-
Attached is an initial success for screenshot functionality through libhpcalcs on critor's Prime (thanks a lot for the many dumps and tests !), after manual edition of the PNG file to remove data which the lower layers of the code do not filter yet :)
Yup, the calculator's timestamp is broken. And the libhpcalcs code does not yet send or receive files to/from the calculator, which limits its practical usefulness, for now...
-
I for one am looking forward to any improvements to the software. Trying to create programs on this thing right now are a pain.
Gratitude to anyone/everyone working on doing so.
-
Great Lionel :)
And yes I agree Ryleh. Fortunately, writing the actual code on the emulator or connectivity software then testing is quite easy compared to other calcs. It's just sending it on the real calc or coding it directly on it that can be a nightmare. I managed to finally send a 46 KB program to my calc faster than mentionned above, but it still took about 8 minutes including the delay when the software was reading my calc content.
Welcome here Ryleh by the way. If you have any chance/time, would you like to share some of your creations with us in the future and introduce yourself in the intro sub-forum? :)
-
Yesterday evening, critor confirmed that my latest changes produce a directly usable PNG file. The colors are still messed up, but I'm outputting verbatim the data part of the packets produced by the calculator (or so I think, at least :D).
-
I wonder if this could eventually be used as some sort of real time screen capture if combined with Camstudio and such software or if it would be too choppy? Of course we can do it with the emulator, though, so maybe it wouldn't be a good idea to focus on such feature before getting linking to work, although the emu runs too fast.
Anyway good luck and great progress so far! :)
-
Screenshotting and backup are supposed to be implemented reliably, besides the strange colors in the screenshots (but I'm not mangling the output). Printing logging output to a file instead of the horribly Windows terminal tremendously helps for reliability...
I'm not aware critor transferred back a file created by test_hpcalcs to the calculator using the HP Connectivity Kit.
What's holding further implementation of libhpcalcs / test_hpcalcs at this point, is the integrity checking value: 16-bit value in the header of packets, which varies with the variable name and the variable contents. I can't find how it is computed, even with a brute-forcing program, which I've posted at http://tiplanet.org/forum/viewtopic.php?p=151303#p151303 : the more eyes and brains, the better !
There seem to be such integrity checking values in the packets used for sending and receiving individual files, so I assume that these operations cannot work until we get this integrity-checking computation right.
-
Could this be asked to HP devs directly ?
It's not as (well, so I hope) if it's a big copyrighted part of the OS nor hacking-related or whatnot ... ?
-
but I'm outputting verbatim the data part of the packets produced by the calculator (or so I think, at least :D).
I'm guessing then that you haven't figured out our little "hack". Basically, PNG has no definition for a 16bit per pixel (color) encoding. Since the calculator is currently running using an RGBt 5551, you have 16 bits per pixel. However, there *is* a defined 16bit per pixel greyscale in the png standard. Since PNG is lossless...
Convert your values coming out from 5551 to a 8888 or similar and I suspect it will look a bit better. :-)
As for the "8 minutes to enumerate", I've never been able to see anything like this. An empty calculator usually takes a few seconds - even over wireless which is slower. How much stuff is in your system? How long does it take an empty calc?
// CCITT CRC-16 look-up table (X^16+X^12+X^5+1)
(this looks like a good place to look at:http://www.embeddedrelated.com/groups/msp430/show/29689.php)
Initialized the crc bytes to 0 at start (yeah - dumb, caught too late to mess with). Also, looks like it might be including the first byte, but using the "length" value which does not include the first 6. (so first 6 bytes are included and last 6 are not).
unsigned short crc = 0;
while (len--)
{
crc = crc16_tab[(crc >> 8) ^ *buffer++] ^ (crc << 8);
}
-
// CCITT CRC-16 look-up table (X^16+X^12+X^5+1)
(this looks like a good place to look at: http://www.embeddedrelated.com/groups/msp430/show/29689.php)
Initialized the crc bytes to 0 at start (yeah - dumb, caught too late to mess with). Also, looks like it might be including the first byte, but using the "length" value which does not include the first 6. (so first 6 bytes are included and last 6 are not).
Works much better this way, thanks a lot :)
With the array and CRC computation function that you point:
static void crc16_block(const uint8_t * buffer, uint32_t len)
{
uint16_t crc = 0;
while (len--)
{
crc = ccitt_crc16_table[(crc >> 8) ^ *buffer++] ^ (crc << 8);
}
printf("crc16 is %" PRIX16 "\n", crc);
}
the winning call is:
crc16_block(array2+1, 0x10);
crc16_block(array3+1, 0x10);
crc16_block(array4+1, 0x10);
Output:
crc16 is 9D57
crc16 is 68B
crc16 is 703F(which shows that the CRC is written as little-endian)
I wonder why the brute-forcer didn't find it, even with 0x0000 as initial value for the CRC computation (I had already tested that instead of 0xFFFF), since it looked from 0 to 12 as start offset and 0 to 48 as end offset.
Anyhow, thanks a lot again ;)
-
So... the files seem to be nailed, but I'm still wrong for the screenshots in at least three ways...
(* as you indicated above, there's a trick for encoding the 16-bit images, so now, we know why they look like odd grayscale pictures. I didn't envision having libpng decompress the images, convert the data myself, then having libpng recompress the images, but it can be done);
* do the the colors for the paletted 4-bit color PNGs need the same kind of conversion as 16-bit grayscale PNGs ? The colors remain wrong, see (http://www.omnimaga.org/index.php?action=dlattach;topic=17172.0;attach=16224;image)
critor and others noted that this output looks like images with reversed blue and red colors. Interpreting 5551 as 565 probably wouldn't wreck the colors so much ?
* AFAICS in the test program, using the same CRC function for computing the screenshot data's CRC, and trying to match the result against the data at offset 6-7 (instead of 8-9 as in files), comes up empty, even when varying start and end (manually and programmatically). The best I could find was a match with reverse endianness, when computing between offset 8 and end of data, but that does not make sense.
In the short term, I can (and do) just ignore the CRC for screenshots, but it means that I am (and therefore, the code is) missing something.
-
As for the "8 minutes to enumerate", I've never been able to see anything like this. An empty calculator usually takes a few seconds - even over wireless which is slower. How much stuff is in your system? How long does it take an empty calc?
I have about 110 KB of stuff on my calc. With a nearly empty calc it only takes a few seconds.
Intel Core i7 (first gen) 2.80 GHz
8 GB of RAM
1 TB hard drive (50 GB free) + two externals for a total of 6 TB
USB 2.0 ports
Windows 7 Ultimate 64 bits
-
I'm not making progress on the screenshots' CRC, but making progress on other areas:
* backup is now fully implemented (it already was earlier) and gets the CRC right (which is new). In the test program, there are silly charset conversion issues (with e.g. the theta variable, whose UTF-16 LE name is 0xB8 0x03), but I don't plan on addressing them in the test program: this job is better done by clients of the library with proper Unicode handling (with libicu, Qt, whatever);
* the procedure for sending files to the calculator is nailed, thanks to critor's dumps of the HP Connectivity Kit <-> calc interaction. Sending a first virtual packet without payload, then another virtual packet with data payload, doesn't seem to be necessary, as creating a file through the "Content" tab of the connectivity kit does create+fill the file in one fell swoop - therefore, I plan on implementing only the combined create+fill. critor's dumps also show that for computer -> calc transfers, the very first byte in the packet is getting incremented (0x00 -> 0xFE -> 0x00 -> ..., that also indirectly confirms the fact that packets starting with 0xFF are sort of "ready" acknowledgments, I'm ignoring them). That's unlike calc -> computer transfers, where the first byte stays 0.
As for transfer slowness: during the calc -> computer backup process [EDIT: made with the HP Connectivity Kit], the timestamps in critor's USB packet dumps show intervals of between 5 and 20 ms between each 64-byte packet, which mechanically limits the transfer rate to between ~3 KB/s and ~12 KB/s... but still, 8 minutes for 110 KB of data indicates a problem somewhere, as even at 3 KB/s, it should last less than 40s.
Does the Windows HID stack (whose reliability seems to be pitiful, as shown by the horrible behaviour when printing traces to the horrendously slow Windows terminal) lose packets so easily that calc -> computer backups cannot be made unconditionally at 5 ms interval ?
BTW, 3-12 KB/s is a very far cry from the transfer rate in firmware upgrade (MSD-like) mode (EDIT: using the "official" reflashing tool; libhpcalcs does not tackle that), whose speed is good, faster than that of the Nspire family, despite having more data to transfer :(
[EDIT: I do not have any measurements of the performance of libhpcalcs wrt. the official connectivity kit. If the calculator is the one setting the pace for calc -> computer transfers, which I believe due to my understanding of how USB interrupt mode works, there shouldn't be a difference, as the speed will be limited by the calculator for not overwhelming poor little Windows.]
The libhpcalcs + test_hpcalcs source code (written in C11) now exceeds 2500 SLOC, as counted by `sloccount`. The raw line count is above 4000 lines, as all main headers now have some documentation for all APIs and data structures, there are some comments... and the license blurb repeated at the beginning of all files takes its toll as well.
[EDITs for clarification.]
-
Nice to see that your method transfers faster. :) Now if only there was a faster way... I know the PRIZM is about 22 KB/sec but there is a small delay when killing the connection, where the PRIZM tries to convert the received data for calc use.
-
UPDATE: The time it takes for the official software to read the calc content and display it in the nav tree seems random. Now after 5 minutes it still didn't list anything and it only just started doing so. I wonder if it has to do with the fact I am listening to music via iTunes at the same time, though. I still have 110 KB of programs.