Author Topic: tok8x: a very simple on-computer tokeniser/detokeniser  (Read 5679 times)

0 Members and 1 Guest are viewing this topic.

Offline shmibs

  • しらす丼
  • Administrator
  • LV11 Super Veteran (Next: 3000)
  • ***********
  • Posts: 2132
  • Rating: +281/-3
  • try to be ok, ok?
    • View Profile
    • shmibbles.me
Re: tok8x: a very simple on-computer tokeniser/detokeniser
« Reply #15 on: April 08, 2013, 05:01:39 pm »
coolio, i'll take a look at that, then =). the rest of your xml files have already been a big help, by the way.

as for adding in support for xml files, it sounds nice, but i'm still not convinced that it's worth the trouble. at present, all it takes for someone to add in a new token is to write a single line in the form {first_byte, second_byte, "string" }, and then run make. compilation takes less than a second, and the result is a binary that can be dragged and dropped anywhere without the need of additional files, whereas adding in xml support would increase the size considerably and either require me to include some non-standard library or write a lot more code.

EDIT: both the default BASIC and Axe token sets are now complete =D
« Last Edit: April 08, 2013, 07:29:22 pm by shmibs »

Offline Lionel Debroux

  • LV11 Super Veteran (Next: 3000)
  • ***********
  • Posts: 2135
  • Rating: +290/-45
    • View Profile
    • TI-Chess Team
Re: tok8x: a very simple on-computer tokeniser/detokeniser
« Reply #16 on: April 09, 2013, 02:06:30 am »
I'm not really convinced either by the necessity or urgency of adding XML support, even if:
* it isn't hard to make users clone some external simple existing StAX parser, and use it in your program. Handling JSON would be easier, and more efficient, than handling XML - but Tokens IDE uses XML;
* with a bit more (easy) code, it would enable detecting mismatches between the internal, compiled-in data and the data extracted from XML files.
« Last Edit: April 09, 2013, 02:06:50 am by Lionel Debroux »
Member of the TI-Chess Team.
Co-maintainer of GCC4TI (GCC4TI online documentation), TILP and TIEmu.
Co-admin of TI-Planet.

Offline shmibs

  • しらす丼
  • Administrator
  • LV11 Super Veteran (Next: 3000)
  • ***********
  • Posts: 2132
  • Rating: +281/-3
  • try to be ok, ok?
    • View Profile
    • shmibbles.me
Re: tok8x: a very simple on-computer tokeniser/detokeniser
« Reply #17 on: April 10, 2013, 01:25:21 pm »
apart from the preprocessor and another token set option for pretty printing (in case you want to display "λ" rather than "lambda", etc), i'm fairly certain this is done!

new stuff:
* full pipeline support, so things like
Code: [Select]
cat <some.8xp> | tok8x | grep -n <search term>are perfectly valid =D

* an option to strip excess whitespace (outside of strings) and comments (the format of which is specific to the language) from a generated program, so a file containing:
Code: [Select]
.ABCD
"hey there! i'm a string"->Str1

.one line comment

...
multiple
line
comment
...

For(A,0,23)
Text(0,A*6,{Str1+A})
End
run through:
Code: [Select]
tok8x -s -t axe -i <infile> | tok8x -t axewould generate:
Code: [Select]
.ABCD
"hey there! i'm a string"->Str1
For(A,0,23)
Text(0,A*6,{Str1+A})
End

* searching for individual tokens matching strings on the command line (so something like
Code: [Select]
tok8x Get azvd DiagnosticOffwould return:
Code: [Select]
"Get":Axe:E8
"DiagnosticOff":BASIC:BB67

* expanding leading spaces on a line to tabs for easier reading when converting from an 8xp
« Last Edit: April 10, 2013, 01:59:32 pm by shmibs »

Offline DJ Omnimaga

  • Former TI programmer
  • CoT Emeritus
  • LV15 Omnimagician (Next: --)
  • *
  • Posts: 55882
  • Rating: +3151/-232
  • CodeWalrus founder & retired Omnimaga founder
    • View Profile
    • DJ Omnimaga Music
Re: tok8x: a very simple on-computer tokeniser/detokeniser
« Reply #18 on: April 10, 2013, 01:34:19 pm »
Will this editor support image/picture importing/exporting by the way? Even though the calc has been out for months, no existing software other than the Mac version of TI-Connect supports them and it will be annoying for BASIC programmers who use pics to store data and title screens or whatever...

Offline shmibs

  • しらす丼
  • Administrator
  • LV11 Super Veteran (Next: 3000)
  • ***********
  • Posts: 2132
  • Rating: +281/-3
  • try to be ok, ok?
    • View Profile
    • shmibbles.me
Re: tok8x: a very simple on-computer tokeniser/detokeniser
« Reply #19 on: April 11, 2013, 11:06:59 pm »
this isn't an editor; it's just a tool for converting between file formats. someone could build an editor that uses it as a backend, though.

i added an include preprocessor directive! the format is:
##include path/to/file
and the contents of file will be inserted verbatim into the file at that point.

Offline Streetwalrus

  • LV12 Extreme Poster (Next: 5000)
  • ************
  • Posts: 3821
  • Rating: +80/-8
    • View Profile
Re: tok8x: a very simple on-computer tokeniser/detokeniser
« Reply #20 on: July 25, 2013, 08:40:38 am »
This looks pretty awesome Shmibs, just the tool I've been looking for.

Offline shmibs

  • しらす丼
  • Administrator
  • LV11 Super Veteran (Next: 3000)
  • ***********
  • Posts: 2132
  • Rating: +281/-3
  • try to be ok, ok?
    • View Profile
    • shmibbles.me
Re: tok8x: a very simple on-computer tokeniser/detokeniser
« Reply #21 on: July 25, 2013, 02:18:34 pm »
well, apparently i exploded something or other at some point, because it isn't working now. this version was more of a proof of concept than anything else, really, though, so i guess i'll try to redo it properly over this weekend. the long, tedious bit of writing out all the tokens is done, though, so there's nothing to worry about there.

Offline shmibs

  • しらす丼
  • Administrator
  • LV11 Super Veteran (Next: 3000)
  • ***********
  • Posts: 2132
  • Rating: +281/-3
  • try to be ok, ok?
    • View Profile
    • shmibbles.me
Re: tok8x: a very simple on-computer tokeniser/detokeniser
« Reply #22 on: July 28, 2013, 02:28:53 pm »
/\ignore that entirely. it's a problem of me having no memory. right now, programs generated without a specified name are always named "A", but i had completely forgotten that and thus was looking for something that wasn't there.
* shmibs goes to change it so that it defaults to the first 8 letters of the filename instead.

Offline Streetwalrus

  • LV12 Extreme Poster (Next: 5000)
  • ************
  • Posts: 3821
  • Rating: +80/-8
    • View Profile
Re: tok8x: a very simple on-computer tokeniser/detokeniser
« Reply #23 on: August 04, 2013, 02:56:47 am »
I made a little pull request for you. Not much since I just fixed a missing character but... :P
This is the first time I make a pull request so I hope I did it right.
« Last Edit: August 04, 2013, 02:57:15 am by Streetwalker »

Offline shmibs

  • しらす丼
  • Administrator
  • LV11 Super Veteran (Next: 3000)
  • ***********
  • Posts: 2132
  • Rating: +281/-3
  • try to be ok, ok?
    • View Profile
    • shmibbles.me
Re: tok8x: a very simple on-computer tokeniser/detokeniser
« Reply #24 on: August 04, 2013, 04:50:08 pm »
yup, that's the way it works =)
thanks!

Offline Streetwalrus

  • LV12 Extreme Poster (Next: 5000)
  • ************
  • Posts: 3821
  • Rating: +80/-8
    • View Profile
Re: tok8x: a very simple on-computer tokeniser/detokeniser
« Reply #25 on: August 05, 2013, 09:27:17 am »
You welcome, I'm glad I could help. :)

Offline Hayleia

  • Programming Absol
  • Coder Of Tomorrow
  • LV12 Extreme Poster (Next: 5000)
  • ************
  • Posts: 3367
  • Rating: +393/-7
    • View Profile
Re: tok8x: a very simple on-computer tokeniser/detokeniser
« Reply #26 on: July 25, 2014, 08:36:24 am »
I am having problems with this (sorry for necroposting btw).

I can't compile what I get from GitHub:
─┐  ╔[ asuka @ Luna : ~/Bureau/tok8x-master ]
 ╘══╩═[ make
ghc --make tok8x
target `tok8x' is not a module name or a source file
make: *** [all] Erreur 1


And when I download the old version, it is a bit better since I can compile it and it works in one way, but not the other way (unfortunately, it's the less interesting way that works):
─┐  ╔[ asuka @ Luna : ~/CALC/SBO ]
 ╘══╩═[ vim TEST.8xp.txt && ./tok8x -t axe -i TEST.8xp.txt -o OUTPUT.8xp -n SBOS -f && tilem2 --help
tok8x: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 *(sizeof(size_t))) - 1)) & ~((2 *(sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed.
Abandon (core dumped)


Am I doing something wrong ? Maybe that the old version needs a bit more than just the "tok8x" in the folder to convert (edit indeed, I can get this one to work when I don't move only the executable) ? But what about the "new" version ?
« Last Edit: July 25, 2014, 08:42:52 am by Hayleia »
I own: 83+ ; 84+SE ; 76.fr ; CX CAS ; Prizm ; 84+CSE
Sorry if I answer with something that seems unrelated, English is not my primary language and I might not have understood well. Sorry if I make English mistakes too.

click here to know where you got your last +1s

Offline Streetwalrus

  • LV12 Extreme Poster (Next: 5000)
  • ************
  • Posts: 3821
  • Rating: +80/-8
    • View Profile
Re: tok8x: a very simple on-computer tokeniser/detokeniser
« Reply #27 on: July 25, 2014, 09:31:49 am »
Hmmm ghc eh ? Seems like Shmibs moved to Haskell. I don't like Haskell that much due to dependencies so I can't really help. :P

Offline shmibs

  • しらす丼
  • Administrator
  • LV11 Super Veteran (Next: 3000)
  • ***********
  • Posts: 2132
  • Rating: +281/-3
  • try to be ok, ok?
    • View Profile
    • shmibbles.me
Re: tok8x: a very simple on-computer tokeniser/detokeniser
« Reply #28 on: December 29, 2015, 10:51:36 pm »
heys, sorry about that. the previous version is in a branch labelled "old". was messing around a while ago seeing if i could make a haskell version work, but gave up and am rewriting the whole thing from scratch in C, because this was sort of the first C project thing i ever made and the old code is really gross and crufty and bug-filled. new version should be done soon, if anybody actually cares about this

Offline TIfanx1999

  • ಠ_ಠ ( ͡° ͜ʖ ͡°)
  • CoT Emeritus
  • LV13 Extreme Addict (Next: 9001)
  • *
  • Posts: 6173
  • Rating: +191/-9
    • View Profile
Re: tok8x: a very simple on-computer tokeniser/detokeniser
« Reply #29 on: December 30, 2015, 10:21:23 pm »
Oh hey, activity! Pretty nice! ^^