Omnimaga

Calculator Community => TI Calculators => Lua => Topic started by: Deep Toaster on October 19, 2011, 12:19:31 am

Title: TNS and ZIP file differences
Post by: Deep Toaster on October 19, 2011, 12:19:31 am
For my BBify'r (http://omniurl.tk/9167/199763/) project, I made a short PHP snippet to open a TNS file as a ZIP archive and read the contents of Problem1.xml. Following the rough description on Hackspire's page on the [hackspire]TNS File Format[/hackspire], I replaced the first ten bytes (*TIMLP followed by four version bytes) with \x50\x4B\x03\x04 and then used ZipArchive (http://www.php.net/manual/en/class.ziparchive.php) to open it. Unfortunately, it didn't work, and ZipArchive didn't even give me an error.

What surprised me was that I had been able to do that with 7-zip before with no problems (using XVI32 to change the first few bytes). But since GUI apps all have a habit of pretending nothing happened when an error occurs, I decided to try it with the command-line unzip command, and here's what it gave me:
Quote from: unzip on bash on Debian
warning [cube.tns.txt]:  362 extra bytes at beginning or within zipfile
  (attempting to process anyway)
  inflating: Problem1.xml
So apparently there is an error, and it's insignificant enough that unzip and 7-zip pass by with no problem, but large enough that ZipArchive stumbles.

Does anyone know what it is? Hackspire mentions nothing of the sort, but it seems outdated anyway (it says the only version numbers known are 0120, 0200, and 0300, while I'm working with a file with version number 0500). For now, I'm cheating in my script and using exec (http://www.php.net/manual/en/function.exec.php) to run unzip on the uploaded file, but if possible I'd like to take that bit out and replace it with native PHP/ZipArchive. Thanks for any help!
Title: Re: TNS and ZIP file differences
Post by: AngelFish on October 19, 2011, 02:23:06 am
Is the tns file 512 byte aligned? That might be your error right there.
Title: Re: TNS and ZIP file differences
Post by: Jim Bauwens on October 19, 2011, 03:13:42 am
I had the same problem yesterday, but I only tried it with new documents. I managed to list the content, but it error'ed when I tried to extract some stuff.
Maybe because of encryption? (although kinda weird)
Title: Re: TNS and ZIP file differences
Post by: Netham45 on October 19, 2011, 03:16:55 am
I'm not familiar with the tns's, but try cutting everything out before the first 0x50,0x4B, as that should be the start of the zip. The central directory might need its offsets updated, too. Check Wikipedia's article on zip files, it's actually pretty damn good.

Edit: If you want to post a small tns, I can check what is off standard. I've worked quite a bit with zips.
Title: Re: TNS and ZIP file differences
Post by: Jim Bauwens on October 19, 2011, 03:46:51 am
Code: [Select]

 Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
--------  ------  ------- ---- ---------- ----- --------  ----
     659  Unk:013     320  51% 2001-01-08 21:17 44684573  Document.xml
   61206  Unk:013    3465  94% 2001-01-08 21:17 2941b95f  Problem1.xml
--------          -------  ---                            -------
Code: [Select]
  skipping: Document.xml            unsupported compression method 13
   skipping: Problem1.xml            unsupported compression method 13
It looks like the compression method is not supported (or wrong marked?) Maybe because of the encryption.

Edit: I didn't see the error you gave. Try using the script on hackspire to add the correct header.
Title: Re: TNS and ZIP file differences
Post by: Deep Toaster on October 19, 2011, 09:37:05 am
Is the tns file 512 byte aligned? That might be your error right there.
No idea what that means :/
I'm not familiar with the tns's, but try cutting everything out before the first 0x50,0x4B, as that should be the start of the zip. The central directory might need its offsets updated, too. Check Wikipedia's article on zip files, it's actually pretty damn good.

Edit: If you want to post a small tns, I can check what is off standard. I've worked quite a bit with zips.
Thanks. I'll try that when I get home today. The TNS file I tested with was jimbauwens's unencrypted build (http://omniurl.tk/10924/199529/) of Chockasta's CubeField.
It looks like the compression method is not supported (or wrong marked?) Maybe because of the encryption.
Hackspire does say TNS files use a "compression method 13" that doesn't really exist, but 7-zip and unzip run it fine ???
Title: Re: TNS and ZIP file differences
Post by: Adriweb on October 19, 2011, 10:12:26 am
Extended has worked on that unknown 13 (0D field) compression algorithm (a part of the encryption has been patented and is thus public).

http://ndlessly.wordpress.com/2011/05/22/d-method-encryption-and-lua-hopes-for-os-3-0-2/ (http://ndlessly.wordpress.com/2011/05/22/d-method-encryption-and-lua-hopes-for-os-3-0-2/)
and
http://ndlessly.wordpress.com/2011/07/31/lua-to-tns-converter-a-little-bit-further/ (http://ndlessly.wordpress.com/2011/07/31/lua-to-tns-converter-a-little-bit-further/)

He said that since he made a program to compress correctly (bypassing (and not breaking) the patented stuff, that's why it's not illegal), it wouldn't be so hard technically to build a de-compress tool.
Title: Re: TNS and ZIP file differences
Post by: Deep Toaster on October 19, 2011, 10:32:18 am
Why did 7-zip and unzip work then? I don't know much about encryption and compression :/
Title: Re: TNS and ZIP file differences
Post by: Jim Bauwens on October 19, 2011, 10:36:43 am
They don't work with encrypted tns files, they just list the content.
Title: Re: TNS and ZIP file differences
Post by: Netham45 on October 19, 2011, 02:03:39 pm
As far as the zip goes (I'm no good with compression) if you remove the first 367 bytes, it looks to be a normal zip that even opens with Windows' built-in zip tool, which is extraordinarily bitchy.

The document.xml at the top appears to be a valid zip entry, once you patch the header. The rest of the zip seems to be running off of the assumption that it's not there (the central directory start is offset by -367 bytes, problem1.xml starts at 0x00, etc...) It's also compressed with the 0x0D compression type.

I repaired the central directory (Added an entry for document, corrected the offset of problem, added one to the file counters, updated the start of directory offset, updated the size of directory, searched for stupid typo for 5 minutes; I didn't update the timestamps on document.xml) in a copy of it and properly placed everything. Here's my resultant zip. Other than that 0x0D compression type, everything opens it fine.

Edit: The -367 byte offset assumes that you haven't fixed the entry for document.zip (remove first 10 bytes, add 0x50,0x4B,0x03,0x04).
Title: Re: TNS and ZIP file differences
Post by: Lionel Debroux on October 19, 2011, 02:07:34 pm
In addition to the TIMLPxxxx at the beginning, where xxxx should be replaced by PK\x03\x04, at the bottom of the file, TIPD needs to be replaced by PK\x05\x06.
Title: Re: TNS and ZIP file differences
Post by: Netham45 on October 19, 2011, 02:08:39 pm
In addition to the TIMLPxxxx at the beginning, where xxxx should be replaced by PK\x03\x04, at the bottom of the file, TIPD needs to be replaced by PK\x05\x06.

I didn't have to fix those on mine. o.O
Title: Re: TNS and ZIP file differences
Post by: Lionel Debroux on October 19, 2011, 02:15:28 pm
Then it was probably not generated by TI's software (unless it was a file suitable for OS 1.1) :)
I've checked that the 8 obfuscated files I generated for a user of the tinspire Google group have TIPD instead of PK\x05\x06.
Title: Re: TNS and ZIP file differences
Post by: Deep Toaster on October 19, 2011, 03:39:15 pm
Thanks Netham45, I'll try that.

Lionel, jimbauwens built it with a compside script, so it might not have made the output in perfect TNS format.

EDIT: Does anyone have more unencrypted TNS files they could send me so I can play around with them? ;D
Title: Re: TNS and ZIP file differences
Post by: Jim Bauwens on October 19, 2011, 04:16:27 pm
Depends
Spoiler For On What?:
On how big your pocketbook is

http://bwns.be/jim/term.tns should be unencrypted.
Title: Re: TNS and ZIP file differences
Post by: Deep Toaster on October 19, 2011, 04:45:03 pm
Was that made with a script or is that a true TNS file?
Title: Re: TNS and ZIP file differences
Post by: Jim Bauwens on October 19, 2011, 04:46:17 pm
Oh, it was made with original lua2tns, so not true true.
Title: Re: TNS and ZIP file differences
Post by: ExtendeD on October 19, 2011, 04:48:28 pm
Luna is open source and may help. I had to patch the minizip library to make it compatible with the format, the changelog is provided at the beginning of the source file.
Title: Re: TNS and ZIP file differences
Post by: Deep Toaster on October 19, 2011, 05:19:49 pm
Did it :w00t:

I basically just followed Netham45's instructions to remove everything before the PK\x03\x04, and somehow it works. I still don't understand lol