Author Topic: Standalone relocator - Ndless-Elfloader alternative  (Read 8309 times)

0 Members and 1 Guest are viewing this topic.

tangrs

• LV5 Advanced (Next: 300)
• Posts: 228
• Rating: +97/-0
Standalone relocator - Ndless-Elfloader alternative
« on: March 31, 2012, 12:16:28 am »
As I was using Ndless-Elfloader for one of my projects, I realized a problem.

I spent more time working out problems in the ELF loader than actually working on my problem. The ELF loader was clunky and wouldn't work on the newer OS's and it was a huge problem to maintain. That aside, prototyping a program to work using it just took way too long.

I started developing a new, lighter loader similar to Ndless's binary loader to use in my projects. It's in it's early stages right now and probably useless to most people but I have included the source in case any developers are interested in using it.

Source code

Lionel Debroux

• LV11 Super Veteran (Next: 3000)
• Posts: 2135
• Rating: +290/-45
Re: Standalone relocator - Ndless-Elfloader alternative
« Reply #1 on: March 31, 2012, 02:29:31 am »
This yields transparent relocation support, and as such, for standalone programs, it should be quite nice

Among others, the approach of making programs relocate themselves was used on TI-68k calcs, as part of platform-specific startup code, for relocation formats smarter than the one supported by the OS. It works, but as you mention on the README of this new relocator, it requires custom tools...
A bit of custom tooling is not evil, and we can work with it on the long term; but when everything is platform-specific, it becomes hard to maintain in the long term. I speak from hands-on experience: I maintain GCC4TI, which has inherited from the unmaintained TIGCC a heavily patched GCC, a heavily patched binutils, a platform-specific front-end, a platform-specific linker with lots of non-portable features, etc. Obviously, most of these platform-specific changes were made for technical reasons (including reverting functionality-reducing changes in upstream GCC, many programs depended on the functionality of older GCC versions), but it hurts maintenance.

FWIW, I had started to look at porting a bFLT loader to the Nspire platform: http://ourl.ca/14975/284471 . bFLT is simple, can easily be converted from ELF, and does the job we need, AFAICT. It can handle dynamic libraries, too.
But as I mentioned in that topic, I can't handle the port alone
« Last Edit: March 31, 2012, 02:32:58 am by Lionel Debroux »
Member of the TI-Chess Team.
Co-maintainer of GCC4TI (GCC4TI online documentation), TILP and TIEmu.

ExtendeD

• Project Author
• LV8 Addict (Next: 1000)
• Posts: 825
• Rating: +167/-2
Re: Standalone relocator - Ndless-Elfloader alternative
« Reply #2 on: March 31, 2012, 03:17:12 am »
Very good job tangrs

I did agree with Lionel, but:
- A full-featured relocator becomes essential when using or porting existing code (for instance http://ourl.ca/14975/294135 ), we can't wait any longer.
- There's a rather high-level of turnover in the TI-Nspire developer community, so any wide-used tool must be Open Source and maintainable. Looking at Ndless-standalone-relocator's code, the ELF-part might be a bit tricky, but the relocator is rather lightweight.

tangrs, how stable is the relocator? What remains to be integrated?

One thing I would like to be improved: the relocator currently replaces Ndless's crt0.S. Which means any startup feature non-related to relocation (such as __crt0_savedsp's initialization) will be lost when switching to ndless-standalone-relocator. What could be done to keep Ndless's crt0.S, and use startup.S as the program entry point?

Looking closer: is crt0.s actually kept, and startup.S added before it?

[edit2]
I'm trying to use it on my Windows computer, but I'm always getting the error:
makeself.exe: Unable to parse ELF file "testlibm.elf".
I'm using libelf v0.8.13 and GNU v2.21. How can we diagnose this?
« Last Edit: March 31, 2012, 03:36:18 am by ExtendeD »
Ndless.me with the finest TI-Nspire programs

Lionel Debroux

• LV11 Super Veteran (Next: 3000)
• Posts: 2135
• Rating: +290/-45
Re: Standalone relocator - Ndless-Elfloader alternative
« Reply #3 on: March 31, 2012, 04:13:25 am »
The bFLT loader in Linux and QEMU, and the ELF -> bFLT converters, are full-featured, open source, standard and maintainable, and explicitly support dynamic, shared libraries
But they're not adapted for the Nspire yet, indeed, which is a significant drawback compared to ndless-standalone-relocator...

BTW, if the loader code is to remain in the program itself (which may prove suboptimal for dynamic libraries ?), maybe parts of it could be handled through "constructor" and "destructor" attributes ?
Member of the TI-Chess Team.
Co-maintainer of GCC4TI (GCC4TI online documentation), TILP and TIEmu.

ExtendeD

• Project Author
• LV8 Addict (Next: 1000)
• Posts: 825
• Rating: +167/-2
Re: Standalone relocator - Ndless-Elfloader alternative
« Reply #4 on: March 31, 2012, 04:18:39 am »
The bFLT loader in Linux and QEMU, and the ELF -> bFLT converters, are full-featured, open source, standard and maintainable, and explicitly support dynamic, shared libraries

But if several TI-Nspire specific patches need to be a applied, the maintenance burden won't be lighter than a lightweight relocator written from scratch.
And I'm not sure these currently a real need for dynamic libraries support.
Ndless.me with the finest TI-Nspire programs

tangrs

• LV5 Advanced (Next: 300)
• Posts: 228
• Rating: +97/-0
Re: Standalone relocator - Ndless-Elfloader alternative
« Reply #5 on: March 31, 2012, 08:05:53 am »
FWIW, I had started to look at porting a bFLT loader to the Nspire platform: http://ourl.ca/14975/284471 . bFLT is simple, can easily be converted from ELF, and does the job we need, AFAICT. It can handle dynamic libraries, too.
But as I mentioned in that topic, I can't handle the port alone

Would it be easier just to write a bFLT loader from scratch? edit: From a brief look, a bFLT loader looks easy to write - I'll have a crack at it tomorrow

For the ELF loader, I did try to port libelf to the calculator at one point but gave up and ended up just writing the whole thing from scratch.

One thing I would like to be improved: the relocator currently replaces Ndless's crt0.S. Which means any startup feature non-related to relocation (such as __crt0_savedsp's initialization) will be lost when switching to ndless-standalone-relocator. What could be done to keep Ndless's crt0.S, and use startup.S as the program entry point?

Looking closer: is crt0.s actually kept, and startup.S added before it?

[edit2]
I'm trying to use it on my Windows computer, but I'm always getting the error:
makeself.exe: Unable to parse ELF file "testlibm.elf".
I'm using libelf v0.8.13 and GNU v2.21. How can we diagnose this?

I only finished writing it today so I haven't had time to extensively test it. It does pass the unit tests though.

One thing that did require a hacky fix was for the __got_size symbol. The linker saw that symbol as an address to something and adds it to the relocation list. Of course we know it's not an address and attempting to 'fix' it will cause a lot of problems. The temporary fix I used was just to blacklist that offset but obviously, that's no long term solution.

Most likely, this will be abandoned at one point for bFLT anyway

@edit1

Yep, startup.S is simply tacked onto the executable. The memory region looks like this:

Code: [Select]
 P R G \0 [ branch instruction ] [number of relocation entries ] [relocation code ] (startup.S finishes here) [array of relocations ] [ original tns file ]
@edit2
If you could send me the ELF file you're trying to link, I can take a look at it for you.
« Last Edit: March 31, 2012, 08:43:11 am by tangrs »

Lionel Debroux

• LV11 Super Veteran (Next: 3000)
• Posts: 2135
• Rating: +290/-45
Re: Standalone relocator - Ndless-Elfloader alternative
« Reply #6 on: March 31, 2012, 08:56:59 am »
Quote
Would it be easier just to write a bFLT loader from scratch?
I don't know.

Quote
Most likely, this will be abandoned at one point for bFLT anyway
Maybe, but for this to happen, other people than ExtendeD or I need to work on the task
Neither ExtendeD nor I have enough time to move bFLT forward at the required rate.
Member of the TI-Chess Team.
Co-maintainer of GCC4TI (GCC4TI online documentation), TILP and TIEmu.

ExtendeD

• Project Author
• LV8 Addict (Next: 1000)
• Posts: 825
• Rating: +167/-2
Re: Standalone relocator - Ndless-Elfloader alternative
« Reply #7 on: March 31, 2012, 10:53:06 am »
@edit2
If you could send me the ELF file you're trying to link, I can take a look at it for you.

Here it is, I'm just trying to troubleshoot this: http://ourl.ca/14975/294135
Ndless.me with the finest TI-Nspire programs

tangrs

• LV5 Advanced (Next: 300)
• Posts: 228
• Rating: +97/-0
Re: Standalone relocator - Ndless-Elfloader alternative
« Reply #8 on: March 31, 2012, 07:33:28 pm »
@edit2
If you could send me the ELF file you're trying to link, I can take a look at it for you.

Here it is, I'm just trying to troubleshoot this: http://ourl.ca/14975/294135

That's really weird, it works perfectly on my machine.

I added some extra debugging lines into the latest commit. Could you check it out and recompile and see where it conks out?

tangrs

• LV5 Advanced (Next: 300)
• Posts: 228
• Rating: +97/-0
Re: Standalone relocator - Ndless-Elfloader alternative
« Reply #9 on: April 01, 2012, 07:34:19 am »
Quote
Would it be easier just to write a bFLT loader from scratch?
I don't know.

Quote
Most likely, this will be abandoned at one point for bFLT anyway
Maybe, but for this to happen, other people than ExtendeD or I need to work on the task
Neither ExtendeD nor I have enough time to move bFLT forward at the required rate.

I looked at the specifications for bFLT and it's insanely simple. It's not worth trying to port a loader. I spent a few hours today and wrote a basic bFLT loader from scratch. It's really a trivial task compared to ELF loading. Spend a little extra time on it and dynamic libraries should be easily supported too.

I put up the source code on Github (https://github.com/tangrs/ndless-bflt-loader) with a README with all the information needed to get the loader and a elf2flt toolchain working.

Please tell me if this is okay to use.

@edit: if there's a place with comprehensive documentation about the structure of bFLT files, please do tell me. Most of the 'documentation' I've read are header files and comments in code of existing implementations.
« Last Edit: April 01, 2012, 07:52:54 am by tangrs »

Lionel Debroux

• LV11 Super Veteran (Next: 3000)
• Posts: 2135
• Rating: +290/-45
Re: Standalone relocator - Ndless-Elfloader alternative
« Reply #10 on: April 01, 2012, 09:05:09 am »
Quote
I looked at the specifications for bFLT and it's insanely simple.
Indeed, which is several of us had thought about it

Quote
It's not worth trying to port a loader. I spent a few hours today and wrote a basic bFLT loader from scratch.
Good job, as usual

Quote
It's really a trivial task compared to ELF loading. Spend a little extra time on it and dynamic libraries should be easily supported too.
And compressed files would be the icing on the cake.

Quote
I put up the source code on Github (https://github.com/tangrs/ndless-bflt-loader) with a README with all the information needed to get the loader and a elf2flt toolchain working.
Cloned and watched.
I'm sure this will get a lot of testing soon

Quote
@edit: if there's a place with comprehensive documentation about the structure of bFLT files, please do tell me. Most of the 'documentation' I've read are header files and comments in code of existing implementations.
The main source of information I know of is http://retired.beyondlogic.org/uClinux/bflt.htm .

EDIT: I created a news about the topic of loading programs at http://tiplanet.org/forum/viewtopic.php?t=8978
« Last Edit: April 01, 2012, 12:11:51 pm by Lionel Debroux »
Member of the TI-Chess Team.
Co-maintainer of GCC4TI (GCC4TI online documentation), TILP and TIEmu.

tangrs

• LV5 Advanced (Next: 300)
• Posts: 228
• Rating: +97/-0
Re: Standalone relocator - Ndless-Elfloader alternative
« Reply #11 on: April 01, 2012, 08:35:20 pm »
Quote
I looked at the specifications for bFLT and it's insanely simple.
Indeed, which is several of us had thought about it

Quote
It's not worth trying to port a loader. I spent a few hours today and wrote a basic bFLT loader from scratch.
Good job, as usual

Quote
It's really a trivial task compared to ELF loading. Spend a little extra time on it and dynamic libraries should be easily supported too.
And compressed files would be the icing on the cake.

Quote
I put up the source code on Github (https://github.com/tangrs/ndless-bflt-loader) with a README with all the information needed to get the loader and a elf2flt toolchain working.
Cloned and watched.
I'm sure this will get a lot of testing soon

Quote
@edit: if there's a place with comprehensive documentation about the structure of bFLT files, please do tell me. Most of the 'documentation' I've read are header files and comments in code of existing implementations.
The main source of information I know of is http://retired.beyondlogic.org/uClinux/bflt.htm .

EDIT: I created a news about the topic of loading programs at http://tiplanet.org/forum/viewtopic.php?t=8978

Thank you. I'm very honored

ruler501

• Meep
• LV11 Super Veteran (Next: 3000)
• Posts: 2475
• Rating: +66/-9
• Crazy Programmer
Re: Standalone relocator - Ndless-Elfloader alternative
« Reply #12 on: April 01, 2012, 11:14:20 pm »
Good job tangrs. You may want to create a new topic for the bFLT loader and leave this for you standalone relocator.

Also I know this is going to sound newbish but what exactly do things like bFLT and relocators let you do?
I currently don't do much, but I am a developer for a game you should totally try out called AssaultCube Reloaded download here https://assaultcuber.codeplex.com/
-----BEGIN GEEK CODE BLOCK-----
Version: 3.1
GCM/CS/M/S d- s++: a---- C++ UL++ P+ L++ E---- W++ N o? K- w-- o? !M V?
PS+ PE+ Y+ PGP++ t 5? X R tv-- b+++ DI+ D+ G++ e- h! !r y

tangrs

• LV5 Advanced (Next: 300)
• Posts: 228
• Rating: +97/-0
Re: Standalone relocator - Ndless-Elfloader alternative
« Reply #13 on: April 02, 2012, 12:19:10 am »
Good job tangrs. You may want to create a new topic for the bFLT loader and leave this for you standalone relocator.

Also I know this is going to sound newbish but what exactly do things like bFLT and relocators let you do?

There's a detailed explanation if you look in the README for the ELF loader (https://github.com/tangrs/ndless-elfloader/blob/master/README.md).

But simply speaking, not everything is adjusted correctly when you load an ordinary Ndless binary - some bits get left behind and contain incorrect values. There are Ndless functions to help solve this but they are manual methods.

By using one of these loaders, it simply patches up whatever remained automatically.

@edit: bFLT discussion now moved to http://ourl.ca/15736
« Last Edit: April 02, 2012, 01:34:24 am by tangrs »

ExtendeD

• Project Author
• LV8 Addict (Next: 1000)
• Posts: 825
• Rating: +167/-2
Re: Standalone relocator - Ndless-Elfloader alternative
« Reply #14 on: April 02, 2012, 02:04:19 pm »
@edit2
If you could send me the ELF file you're trying to link, I can take a look at it for you.

Here it is, I'm just trying to troubleshoot this: http://ourl.ca/14975/294135

That's really weird, it works perfectly on my machine.

I added some extra debugging lines into the latest commit. Could you check it out and recompile and see where it conks out?

I'm now getting:
ELF: call to elf_begin() failedmakeself.exe: Unable to parse ELF file "testlibm.elf"

Tell me if you want more tests, or maybe you now prefer to invest efforts on the bflt loader.
Ndless.me with the finest TI-Nspire programs