Omnimaga

Calculator Community => Other Calc-Related Projects and Ideas => TI Z80 => Topic started by: Michael_Lee on February 24, 2011, 07:47:53 pm

Title: TIConvert
Post by: Michael_Lee on February 24, 2011, 07:47:53 pm
EDIT:
This project has been renamed to Croquette:
Visit http://ourl.ca/12272 (http://ourl.ca/122725) instead for the latest updates.

You can download the latest versions at https://code.google.com/p/croquette/p/ (https://code.google.com/p/croquette/p/)



I made a python script that turns an Axe source file (8xp) into a text file!  It can add indenting and everything!

Later, I want to try and enhance it so it can convert TI-Basic to text, and text to either TI-Basic or Axe (although it wouldn't probably be able to compile).

...

It's a command-line sort of thing, and it's sort of redundant given Tokens and Runer's mod, but I always wanted to make one.

Also, does anybody know if it's possible in any way to turn a Python 3.2 script into an executable?

Edit:
The attached doesn't even work, so I'm going to remove it before more people download.
Title: Re: TIConvert
Post by: ruler501 on February 24, 2011, 10:08:51 pm
yes it is but it would be easier if you could convert it into a python 2.6 script. That way py2exe could run on it
Title: Re: TIConvert
Post by: Michael_Lee on February 24, 2011, 10:40:49 pm
Yeah, it was only after I was about halfway done that I learned that Python 3.X has compatibility issues.  I'll probably start converting it tomorrow.
Title: Re: TIConvert
Post by: DJ Omnimaga on February 24, 2011, 10:46:26 pm
Interesting, I don't have Python installed so I can't try it now but I will probably try it later.

What features does it have that most other softwares don't, by the way?
Title: Re: TIConvert
Post by: Michael_Lee on February 24, 2011, 11:02:43 pm
Well, the main thing is that it automatically adds indents to the code.  It's always irritated me that no other program did that for me (at least, I think) because I find it really helpful for understanding my code. 
Title: Re: TIConvert
Post by: DJ Omnimaga on February 24, 2011, 11:05:27 pm
I think SourceCoder does, but I do not remember if it does Axe token conversion yet. Also some people might like an offline program from time to time (no internet access at home and such things)
Title: Re: TIConvert
Post by: ztrumpet on February 24, 2011, 11:05:36 pm
This sounds pretty cool.  I like the auto indent idea. :D Nice job! ;D
Title: Re: TIConvert
Post by: Michael_Lee on February 25, 2011, 12:46:00 pm
I just realized that I uploaded an incorrect version - I had set it up to detokenize a certain file automatically for testing purposes and forgot to change it back.

So to the one person who downloaded this, here's a fix.
Title: Re: TIConvert
Post by: Munchor on February 25, 2011, 12:50:05 pm
Wa-a-a-i-i-i-t? Does this read Assembly Axe programs and turns them to .txt? Or is it Axe Source Files?
Title: Re: TIConvert
Post by: Michael_Lee on February 25, 2011, 12:57:56 pm
It reads Axe source files and turns it into .txt

And I also discovered a bunch of minor bugs, so it might not function as expected.  (Fixing time!)
I should have tested more, I guess.
Title: Re: TIConvert
Post by: Munchor on February 25, 2011, 01:14:00 pm
It reads Axe source files and turns it into .txt

And I also discovered a bunch of minor bugs, so it might not function as expected.  (Fixing time!)
I should have tested more, I guess.


It'd be awesome otherwise, but still nice. I use Tokens for this matter.
/me <3 TokenIDE.
Title: Re: TIConvert
Post by: Builderboy on February 25, 2011, 01:16:49 pm
nice :D Its cool that we have an offline version that can do this for us ^^ Might there be a way to turn off indenting?  Some people might not like it, its always good to have options ^^
Title: Re: TIConvert
Post by: Michael_Lee on February 25, 2011, 01:27:02 pm
Indenting is off by default.  (I was learning about command line options, and threw in a bunch to practice :))

And @Scout:
Yeah, it would be pretty cool if this could disassemble assembly into Axe.  But I'd have to learn assembly first  ;D
Title: Re: TIConvert
Post by: Munchor on February 25, 2011, 03:01:00 pm
Indenting is off by default.  (I was learning about command line options, and threw in a bunch to practice :))

And @Scout:
Yeah, it would be pretty cool if this could disassemble assembly into Axe.  But I'd have to learn assembly first  ;D

I think only Quigibo could do it, in fact, I think it'd be really really hard, I'd say impossible.
Title: Re: TIConvert
Post by: Michael_Lee on February 26, 2011, 12:52:43 pm
*Sigh*

After much consternation and experimentation, I finally managed to turn it into an exe.

I tested this on two other computers, so I think it should work, at least on Windows computers (although the included batch file failed to run on my admittedly antiquated XP laptop).

I fixed some issues with ElseIf and ReturnIf, and rewrote the parsing part because I didn't like my older design.
Title: Re: TIConvert
Post by: DJ Omnimaga on February 28, 2011, 04:36:49 pm
Hey I just tried this and it seems to work pretty well, also I'M glad you included the dll file so we don't need to install Python. :D

I tried on some programs and the code seems fine. Here's the source:

Code: [Select]
.DJTUNNEL Axe Parser Tunnel clone
"appvDJTUNNHS"->GDB1
UnArchive GDB1
[80C0E0F0E0C08000->Pic1
"AXE TUNNEL"->Str2
"2010, by DJ Omnimaga"->Str3
"SPEED:"->Str4
0->D->P+1->S+1->Q->K
5->J
8->C+1->R
29->I+1->A
40->W->T
Full
DiagnosticOff
ClrHome
ClrDraw
For(Z,9,21
    Line(0,Z,95,Z
End
Line(0,56,95,56
StorePic
ClrDraw
For(Z,8,22
    Line(0,Z,95,Z
End
For(Z,56,63
    Line(0,Z,95,Z
End
DrawInv
Fix 5
Fix 1
Text(20,12,Str2
Fix 0
Text(14,57,Str3
Fix 3
Text(5,35,Str4
Repeat Z=54
    DS>(K,12)
        DispGraph^^r
    End
    DS>(J,11-(S*2))
        Pt-Change(I,35,Pic1)
        I+1->I
        If I=86
            29->I
        End
    End
    getKey->Z
    If Z=15
        1->S
        Goto GO
    End
    Z=3-(Z=2)+S->S
    If S=6
        5->S
    End
    !If S
        1->S
    End
End
ClrHome
ClrDraw
For(Z,10,50
    Line(0,Z,95,Z)
End
Repeat W=6 or getKey(15)
    P+S->P
    DS>(C,8)
        rand^2*2-1->D
    End
    R+D->R
    !If R
        1->R
    End
    If 62-W<R
        62-W->R
    End
    A-getKey(4)+getKey(1)->A
    Pt-Change(0,A,Pic1)
    Line(95,R,95,R+W)
    DS>(Q,S)
        DispGraph
        If pxl-Test(0,A) or pxl-Test(0,A+6)
            Goto GO
        End
    End
    Pt-Change(0,A,Pic1
    Horizontal -
    DS>(T,99)
        W-1->W
    End
End
Lbl GO
getKey
Fix 4
Fix 2
ClrHome
If P
    Output(0,0,"Score:",P>Dec
End
GetCalc(GDB1)->M
If M
    If P>{S*2+M-2}^^r
        P->{S*2+M-2}^^r
    End
Else
    GetCalc(GDB1,10
    GetCalc(GDB1)->M
    P->{S*2+M-2}^^r
End
Output(0,1,"SPEED HIGHSCORE
For(S,1,5
    Output(0,S+1,S>Dec,"     ",{S*2+M-2}^^r>Dec
End
Output(0,0

Darn this code is ancient...
Title: Re: TIConvert
Post by: Michael_Lee on February 28, 2011, 09:15:38 pm
It's nice to hear that it works :)


Currently, I'm trying to convert text to 8xp, but it runs outrageously slow and has the most vexing habit of occasionally chopping off characters at the start of a new line. /frustration
Title: Re: TIConvert
Post by: DJ Omnimaga on March 03, 2011, 01:09:50 am
Sorry to hear. Hopefully you can solve the issue soon. It seems like it might be a tedious/hard task because words made of individual letters got to be converted to tokens and it must screw up with games using lowercases X.x
Title: Re: TIConvert
Post by: Michael_Lee on March 07, 2011, 08:32:49 pm
Update:
I managed to transform a 1 minute tokenizing time to about 5 seconds - hooray!  What I was doing before was reading each line, seeing if it was a token, and if not, taking a character off of the end of the line, seeing if it was a token, etc., ad nauseum...  In retrospect, given that a token can only be about 14 characters long, starting by examining an entire line might not have been the most efficient method X.X

Wabbitemu and my calc both finally accepts the 8xp, but for some reason, the file is either truncated short, or contains loads of garbage characters at the end.  Hopefully, this should be the last thing I have to fix.
Title: Re: TIConvert
Post by: DJ Omnimaga on March 07, 2011, 10:34:36 pm
Nice to see you got it to be faster. :D I personally had no troubles with the speed, but I got a recent computer so that might be why. I hope you get the bugs fixed without too much hassle.
Title: Re: TIConvert
Post by: Michael_Lee on March 07, 2011, 11:46:48 pm
Nah, I meant the speed for txt to 8xp (which is something I haven't released yet)

I don't foresee any speed problems with 8xp to txt :)
Title: Re: TIConvert
Post by: DJ Omnimaga on March 07, 2011, 11:48:28 pm
Oh ok, lol, I was confused. X.x
Title: Re: TIConvert
Post by: XVicarious on March 08, 2011, 07:55:25 pm
Can't wait for the txt -> 8xp :D
And when that comes out, would it be okay for me to use this in a text editor made in Python? Specifically for Axe/TIBasic? You would obviously get credit.
Title: Re: TIConvert
Post by: Michael_Lee on March 08, 2011, 09:11:33 pm
Yeah, sure, go ahead.  My projects are open-source unless otherwise noted.

I'm also going to be releasing TI-Basic to 8xp, fyi (although 8xp to TI-Basic would be a little trickier.)
Title: Re: TIConvert
Post by: DJ Omnimaga on March 09, 2011, 03:32:48 pm
Cool, are you planning to upload it in the archives and ticalc.org?
Title: Re: TIConvert
Post by: Michael_Lee on March 11, 2011, 08:56:12 pm
@ DJ: Yep, once I have something that's pretty polished.
Title: Re: TIConvert
Post by: Michael_Lee on March 21, 2011, 07:41:51 pm
Update:  Hooray!  I finally finished txt to 8xp!

Now all I have left is calculating the checksum, which shouldn't be too hard...
Title: Re: TIConvert
Post by: Munchor on March 23, 2011, 11:06:33 am
Update:  Hooray!  I finally finished txt to 8xp!

Now all I have left is calculating the checksum, which shouldn't be too hard...

Here is the Python code I once used to calculate checksum for Assembly programs, not sure if it'll work fine, but you can always give it a try:

Code: [Select]
sumo=0
for i in range(0,len(hexArray[76:len(hexArray)-2])):
sumo += int(str(hexArray[76+i]),16)

I know it lacks stuff, but that's pretty much the way it works, I guess.
Title: Re: TIConvert
Post by: DJ Omnimaga on March 23, 2011, 02:26:24 pm
Glad to hear. :D Does it detect lowercase strings easily? (Like in Illusiat 12)
Title: Re: TIConvert
Post by: alberthrocks on April 23, 2011, 05:11:49 pm
Bumpity bumpity bump! :D

How's progress? The tokenizer/detokenizer in our project, Axe GUI Library, is a fail, so we're looking for a new one to take its place, and we'd love to use yours! :) Our project is GPL v3, with the exception of the library itself, which is LGPL so those who don't want to release source don't have to.

(Our project website: http://code.google.com/p/axe-gui-libraries/)
Title: Re: TIConvert
Post by: Michael_Lee on April 23, 2011, 05:53:56 pm
Well, not much progress.

I've been really busy with a bunch of projects lately (large essay, piano recital, trying to learn javascript (currently fail), trying to organize + make website for robotics club, somehow taking over said robotics club, large math assignment, mandatory school project (unless I don't want to graduate), etc..., all nearly at once)

The detokenizer that I've posted already should be pretty good to use, you should just be able to add it in without much modification (although it's really ugly...).

I'm going to go ahead and give you guys what I have right now, with the understanding that it doesn't work yet + might be overly complicated + is extremely different from previous versions + is written in Python 3.1 and might not work in older Python versions due to differences in how bytes are treated.

I think tokenizing in theory works, but I haven't had time to add in making checksums + generating metadata, and never really completed it.  Detokenizing should also work.

I'll probably resume development in about two weeks to a month (or I might just have to wait until summer).

I'm not too worried about licenses -- this project informally uses the Academic Free License 3.0, but really, I don't care too much about licensing as long as you don't go around claiming you wrote my code, or doing other weird stuff.
Title: Re: TIConvert
Post by: Munchor on April 24, 2011, 08:14:31 am
Nice to see this is still being worked on. I'll suggest alberthrocks to use this as our tokenizer for Axe GUI Libraries (http://www.code.google.com/p/axe-gui-libraries/).
Title: Re: TIConvert
Post by: Munchor on May 19, 2011, 04:27:07 pm
I have now the need to use this program since it's cross-platform and I need to write Axe files in Linux.

The version 0.2 seems to be in .exe and I'm not sure if I can use it the same way as the last one.

I got this when running:

Code: [Select]
david@David-PC ~/Documents/Calculators/TIConvert $ python main_v0_3.py
Traceback (most recent call last):
  File "main_v0_3.py", line 107, in <module>
    (input_path, target_path) = command_info.ReturnPaths(suffix = "_out.txt")
  File "main_v0_3.py", line 67, in ReturnPaths
    input_path = self.arg[0]
IndexError: list index out of range
david@David-PC ~/Documents/Calculators/TIConvert $

Python 2.6.6, Linux Mint 10
Title: Re: TIConvert
Post by: Michael_Lee on May 19, 2011, 06:26:51 pm
You probably never provided an input.

You have to drag the file you want to decompile over the python or exe.  If you don't want to do that, then here's a manual method:

Change
Code: [Select]
# Step 1: Reading arguments from the command line.
command_info = CommandLine()
options = command_info.RunParser()
(input_path, target_path) = command_info.ReturnPaths(suffix = "_out.txt")
into this:
Code: [Select]
# Step 1: Manually providing arguments.
input_path = "directory/folder/THEGAME.8xp"
target_path = "directory/folder/THEGAME_out.txt"'
options = {'tabs': 4, 'quiet': False', 'colons': False}

Basically, that single line runs a command line parser that takes arguments inputted in from the command line, then returns the path of the 8xp you want to decompile, then path you want your output to be.
Alter those two strings to suit your needs.

Alter the 'options' dictionary to suit your needs.  If you want indenting, set 'tabs' equal the the number of spaces you want per indent.  If 'quiet' is set to True, then it won't display any info on the command line/console.  If 'colons' is set to True, then it appends a colon to the start of each line.
Title: Re: TIConvert
Post by: Munchor on May 19, 2011, 06:28:14 pm
This goes from Axe source to ASM files and other way around?
Title: Re: TIConvert
Post by: Michael_Lee on May 19, 2011, 06:33:30 pm
I know for a fact it can do 8xp source to text file.

I don't think I finished text file to 8xp source, though you could give it a shot.

And I don't even want to try text or source to compiled 8xp (that's what Quigbo's working on, after all :P)
Title: Re: TIConvert
Post by: Munchor on May 19, 2011, 06:40:28 pm
I know for a fact it can do 8xp source to text file.

I don't think I finished text file to 8xp source, though you could give it a shot.

And I don't even want to try text or source to compiled 8xp (that's what Quigbo's working on, after all :P)

I know the latter doesn't work obviously. I need the second one.

Michael, I really think you should try and finish this :D
Title: Re: TIConvert
Post by: Michael_Lee on July 15, 2011, 11:57:52 am
A quick update:

I've rewritten it all from scratch (again  <_<) and managed to get both detokenizing and tokenizing working nearly perfect.

All I have left is figuring out how to wrap the metadata around the tokenized part, which shouldn't be too hard (I suppose...)
Title: Re: TIConvert
Post by: Michael_Lee on July 15, 2011, 06:17:27 pm
I finished!

Changes:
> You can convert .txt to .8xp now!
> Switched from Python 3.1 to Python 2.7
> Rewrote everything
> Switched to MIT license
> Fixed some bugs.


Edit:
I don't feel like making another post, so here's a short update.
Changes:
> A bunch of minor code fixes
> The parser can handle broken code a little better
> You can now have the .txt files have fancy output!  (characters like → and λ)
> TI-Basic tokens in .txt files will still compile normally into .8xp files.
> All .dll and other similar files have been compressed into a single .exe file for portability

I'm going to work on support for TI-Basic tomorrow.
Title: Re: TIConvert
Post by: BlakPilar on July 18, 2011, 01:01:17 am
Sorry, I didn't take the time to read any other posts, but what do you mean by "it probably wouldn't be able to compile" in the OP?

--EDIT--
I have "fancy output" in TBEXE implemented, too (a pain in the butt to find which characters were accounted for, though)! Great minds think alike, I guess ;)
Title: Re: TIConvert
Post by: Michael_Lee on July 18, 2011, 03:03:06 pm
Sorry, I didn't take the time to read any other posts, but what do you mean by "it probably wouldn't be able to compile" in the OP?

--EDIT--
I have "fancy output" in TBEXE implemented, too (a pain in the butt to find which characters were accounted for, though)! Great minds think alike, I guess ;)

Ah.  Back then, tokenizing text was very mysterious to me, and I wasn't sure if I was capable of programming such a thing.  Converted text files should have no problem compiling now :D
Title: Re: TIConvert
Post by: BlakPilar on July 18, 2011, 08:27:29 pm
Sorry, I didn't take the time to read any other posts, but what do you mean by "it probably wouldn't be able to compile" in the OP?

--EDIT--
I have "fancy output" in TBEXE implemented, too (a pain in the butt to find which characters were accounted for, though)! Great minds think alike, I guess ;)

Ah.  Back then, tokenizing text was very mysterious to me, and I wasn't sure if I was capable of programming such a thing.  Converted text files should have no problem compiling now :D


Psh, nahhh. (De)Tokenizing is actually pretty easy once you know which byte belongs to which command lol. But good to hear that it's working on here!
Title: Re: TIConvert
Post by: lunixbochs on July 18, 2011, 10:24:11 pm
I just started a Python TI-BASIC implementation a few weeks back. It's actually doing pretty well.

My parser/tokenizer implementation is here (https://github.com/lunixbochs/pitybas/blob/master/parse.py#L85)
I'm using a list of tokens and simply comparing each token (sorted from longest to shortest) against my position in the text until I find a match, then incrementing my position by the length of the match. It's worked perfectly so far with many tests, and it's extremely fast (5000 lines / 70k characters took <1s to tokenize). If you take away the cruft for expressions and bracket parsing, it might be useful for your purposes :)

This commit (https://github.com/lunixbochs/pitybas/blob/cf29725c4ba09cc74e86ac17ea72dc01ac43b2f8/parse.py#L4) has the raw token lists, the current version is all dynamic. The full parser from that commit might actually be useful, as it's the raw version with just string output.
Title: Re: TIConvert
Post by: BlakPilar on July 18, 2011, 11:23:59 pm
Hmm... Python, eh? I personally can't deal with Python because it is wwaaayyyyy too loose of a language for me (I need order) lol. That's why I use C#. calccrypto (at least that's his name over on TIBD, not sure what it is here) made an 8xp to txt Python converter that can be found here (http://calccrypto.wikidot.com/local--files/files/8xptxt.zip).

In TBEXE, my detokenizer works perfectly. I just need to stop being lazy and work on my tokenizer :\
Title: Re: TIConvert
Post by: jnesselr on July 18, 2011, 11:25:03 pm
AFAIK, calccrypto doesn't have an account on omnimaga.
Title: Re: TIConvert
Post by: BlakPilar on July 18, 2011, 11:35:33 pm
Ahh, okay.

And hi graphmastur! Remember me? :D
Title: Re: TIConvert
Post by: lunixbochs on July 18, 2011, 11:56:46 pm
my tokenizer method should be pretty easy to implement in most languages
Title: Re: TIConvert
Post by: Hayleia on July 21, 2011, 11:57:15 am
Quick questions for TI-Convert:
For "not equal to", do we have to write "=/=" ?
For "►Dec", do we have to write ">Dec" ?
Title: Re: TIConvert
Post by: Michael_Lee on July 21, 2011, 12:20:02 pm
For not-equal-to, the following tokens are accepted:
!=
=!

(=/= is a good idea, though -- I'll add support for it).

Accepted tokens for ►Dec:
>Dec
►Dec
Title: Re: TIConvert
Post by: Hayleia on July 21, 2011, 12:23:28 pm
Ok, thanks !
*Hayleia goes back to work
Title: Re: TIConvert
Post by: Munchor on July 24, 2011, 10:20:28 am
While Michael works on an online version, I'm working on an Axe IDE, called "Croquette", which is a GUI IDE for Axe (cross-platform). I'll use Michael's work and code the GUI and pack it all up.
Title: Re: TIConvert
Post by: Michael_Lee on July 26, 2011, 06:35:15 pm
Update!

Version 0.5
 > Detokenizing and tokenizing TI-Basic is supported!
 > You can now use special characters (like λ and → instead of lambda and ->)
 > More tokens supported!
 > Detokenize or tokenize all files in a directory!  (Drag a folder over the .bat or .exe)
 > Better documentation -- see zip file for a list of all possible tokens.

And just for ephan...
 > Made ticonverter.py functions easier to use.

Coming up next:
 > Better support for special tokens?
 > An online version!
Title: Re: TIConvert
Post by: Hayleia on July 27, 2011, 05:54:06 am
It still works with Axe (for example when we write "appv" in a Basic program, it doesn't change into a "v") ?
Title: Re: TIConvert
Post by: Michael_Lee on July 28, 2011, 02:48:44 am
Now it does :)

Depending on the first character is a period or not, it'll automatically convert to Axe or Basic (either way).  There's also an option to force a conversion to either Axe or TI-Basic, if you need it.
Title: Re: TIConvert
Post by: Munchor on July 28, 2011, 05:09:09 am
Now it does :)

Depending on the first character is a period or not, it'll automatically convert to Axe or Basic (either way).  There's also an option to force a conversion to either Axe or TI-Basic, if you need it.

What about two dots converting Axe too?
Title: Re: TIConvert
Post by: Hayleia on July 28, 2011, 11:48:44 am
I found another "bug" in the 0.4.1 version, and I don't know if it was fixed.
To make multiple lines comments, I use "...", so 3 "." but it is converted as the "..." token, not as 3 "." ??? so the comment is broken (I guess). Is that fixed ?
Title: Re: TIConvert
Post by: Michael_Lee on July 28, 2011, 11:58:10 am
Huh, I didn't even know that there was a "..." token.

Hmm, I might have to replace the "..." token with something like "|...|" or something...  Or maybe just remove that token altogether for Axe (although I don't really want to...)

Or maybe rewrite part of the parser so that it can tell the difference between a comment '...' and a '...' in a string?

Hmmm...
Interesting bug -- I'll get started on fixing it!
Title: Re: TIConvert
Post by: Hayleia on August 01, 2011, 05:19:48 am
Me again ! Complaining again ! Finding a bug again ! ;D ;D ;D
When I do this:
[Pic0]->Pic0
the second "Pic0" is seen as the token "Pic0" but the first one is seen as "[" "P" "i" "c" "0" "]" ???
Title: Re: TIConvert
Post by: Michael_Lee on August 01, 2011, 03:21:06 pm
Both bugs have been fixed, you can download the latest version at https://code.google.com/p/croquette/ (https://code.google.com/p/croquette/).

Also, TIConvert has now been formally and completely merged with Croquette -- discussion, requests, complaints, etc... now go here: http://ourl.ca/122725 (http://ourl.ca/122725).

You can use Croquette in the same way you've always been using TIConvert -- via the .bat file or by using command prompt.  Double-clicking Croquette.exe will open a GUI, though.