Omnimaga

Calculator Community => Other Calc-Related Projects and Ideas => TI Z80 => Topic started by: KermMartian on August 06, 2010, 02:16:54 am

Title: Tokenizer DE v1.0 Released
Post by: KermMartian on August 06, 2010, 02:16:54 am
In answer to all the users that have requested a custom oncalc TI-BASIC editor, Tokenizer DE takes a first step in that direction. Just as SourceCoder (http://sc.cemetech.net (http://sc.cemetech.net)) can convert text files into .8xp programs online, Tokenizer can take text documents typed in Document DE 6 or Document DE 7 and convert them into fully-tokenized BASIC programs editable with the TI-OS's TI-BASIC editor. Weighing in at slightly over 1KB, this small but efficient program takes advantage of the power of the DCS GUI system, and requires Doors CS 6.8 beta or higher.  It uses a naive reverse substring-match brute force algorithm that takes very little memory (roughly 300 bytes for the core tokenization algorithm) at the expense of going at a moderate rather than blazing speed.  Future improvements may integrate this with a Document DE-style editor; feel free to give it a try and post any comments or suggestions in the associated topic.  This release also includes the source code to Tokenizer DE for those wishing to learn more about using the Doors CS ASM libraries.

Download
(http://www.cemetech.net/img/icon/dl.gif) Tokenizer DE v1.0 (http://www.cemetech.net/programs/index.php?mode=file&path=/83plus/asm/programs/tokenizerde.zip)

(http://www.cemetech.net/img/ss/000277.gif) (http://www.cemetech.net/img/ss/000278.gif) (http://www.cemetech.net/img/ss/000279.gif) (http://www.cemetech.net/img/ss/000280.gif)
Title: Re: Tokenizer DE v1.0 Released
Post by: bwang on August 06, 2010, 05:20:40 am
Excellent! This is certainly useful for people who want to edit more than 7 lines of text at a time :)
Title: Re: Tokenizer DE v1.0 Released
Post by: qazz42 on August 06, 2010, 09:04:10 am
Hmmm, very interesting, I see I will have 2 ways to write programs now :D
Title: Re: Tokenizer DE v1.0 Released
Post by: meishe91 on August 06, 2010, 09:16:50 am
This is very cool! I don't know how much I'd use it but I'm sure some will find it very helpful. Thanks, Kerm :)
Title: Re: Tokenizer DE v1.0 Released
Post by: ztrumpet on August 06, 2010, 02:23:19 pm
This is indeed cool.  Nice work Kerm! ;D
Title: Re: Tokenizer DE v1.0 Released
Post by: Silver Shadow on August 06, 2010, 03:09:51 pm
Very nice! :)
Title: Re: Tokenizer DE v1.0 Released
Post by: qazz42 on August 06, 2010, 03:56:25 pm
People with nspires, it seems that nspire 2.0 will crash in 84 keypad mode when using tokenizer
Title: Re: Tokenizer DE v1.0 Released
Post by: TIfanx1999 on August 06, 2010, 05:00:47 pm
Wow, that's pretty awesome. I can't believe how much functionality Doors has. Great work!
Title: Re: Tokenizer DE v1.0 Released
Post by: DJ Omnimaga on August 06, 2010, 07:44:41 pm
Wow pretty nice indeed. Nice tool again Kerm. Pretty small too :)
Title: Re: Tokenizer DE v1.0 Released
Post by: qazz42 on August 06, 2010, 08:03:03 pm
Ohh, check that, the latest build has nspire compat, it is not released yet..
Title: Re: Tokenizer DE v1.0 Released
Post by: DJ Omnimaga on August 06, 2010, 08:04:17 pm
You have access to a secret Cemetech download folder? Tell us tell us! j/k ;D

Or do you mean he posted about the compatibility on his forums? I should check more often x.x
Title: Re: Tokenizer DE v1.0 Released
Post by: qazz42 on August 06, 2010, 10:34:06 pm
No, I mean he sent me an email with the latest build for testing :)
Title: Re: Tokenizer DE v1.0 Released
Post by: KermMartian on August 06, 2010, 10:37:47 pm
No, I mean he sent me an email with the latest build for testing :)
Both you and DJ are correct.  I posted about fixing the Nspire compatibility, and will shortly release the version I emailed Qazz.
Title: Re: Tokenizer DE v1.0 Released
Post by: DJ Omnimaga on August 06, 2010, 11:07:01 pm
Cool to hear :)

The TI-Nspire is usually the first calc I have that gets used when it comes to testing/using stuff on-calc, because it's my only z80-compatible calc that I can connect through USB (I do not have a Silverlink and my computer got no serial port)