Omnimaga
Calculator Community => Casio Calculators => Topic started by: Kristaba on January 20, 2014, 12:42:40 pm
-
Welcome back, Omnimaga!
I'm working on a litle project from a long time ago (but as I have to deal with IRL and my personal motivation, the project progress a bit randomly ;)), but I only released informations about it on Planète Casio forum.
Now, it's enought mature to begin to introduce this project on other communities, I think.
So I will not say here all I said on french Planète Casio topic, but I will try to summarize main informations, and of course to reply to any question if something is not clear.
FiXos is a kernel, writen from scratch to work on fx9860 and similar calculators. I would like to support all the SH3/SH4 models, but for now I prefer to fully support the only test model I have (fx9860G SD).
It's a UNIX-like kernel, which will try to support most important UNIX paradigms (device management, VFS, main syscalls...), but the goal is not to be 100% compatible with any POSIX version, only to be enought close for main usages.
Once the kernel is running, no more part of the Casio's OS is executed, and the kernel is fully independant of any other software.
For now, the kernel is never executed from the EEPROM directly!
It will not replace Casio's OS definitively, and it is copied on RAM before to be executed. So a simple reset of the calculator will allow to return to Casio's OS.
Later, it may be interesting to write it on a part of the EEPROM (to avoid heavy RAM usage before any user code is running), and maybe to redirect the Calculator reset code to the bootloader, to choose on which one to boot after a reset. But for now it's too much dangerous for nothing, so RAM is our best friend! ;D
From few days ago, the project includes an independant bootloader, which take the form of a G1A file. It has its own code to access efficiently to the SMEM filesystem, and use Casio's OS "syscalls" to display a menu. The whole bootloader is configurable using a config file, to change global properties (default selected entry, quiet mode or interactive mode), and individual entries content (kernel file, kernel arguments, type).
This booloader is able to prepare and run any ELF file, so the kernel is an ELF binary (with minimum informations to keep a small binary).
All the kernel, userspace tools and the bootloader are designed to be compiled by GCC toolchain for SuperH processors. It's also why the ELF file format was chosen : it's a very powerful binary format, and is now used on almost all UNIX-like and other OS. Any modern compiler (so, not the Renesas one...) is able to output ELF binary.
The kernel itself is still young, but begins to be interesting and partialy usable.
It takes advantage of the modernity of the SuperH processor, with virtual memory, separation of adress spaces between processes, and all embeded modules like SDHI for controling SD Card, USB device controler...
Main features for now are :
- Virtual File System (VFS), which allow to use a uniform interface for any filesystem, and to mount any existing filesystem on any directory.
- Casio's SMEM filesystem implementation, in read-only (based on reverse engineering, but hundred times faster than the Casio's OS implementation...).
- ProtoFS, a minimal filesystem in RAM used as root filesystem, supporting creation of directory and device nodes.
- Virtual Memory translation, depending to the current process address space.
- Dynamic device management, using UNIX-like concepts (a device is registered dynamicaly to the kernel using a unique "major" number, and is free to implement any number of "minor", each one containing a functionality), which can be accessed in userland using filesystem device node with corresponding combination or major/minor numbers.
- Early log function (wrapped on a printf-like function), targeting the T6K11 display driver, allowing to print some informations during first step of kernel boot, before any VFS and high-level devices are initialized.
- More advanced terminal access with corresponding device, usable through /dev/console file.
- USB device protocol abstraction, enought flexible to allow writing plateform-independant USB class implementation (and of course the implementation for 7705's USB controler)
- Implementation of USB class "CDC/ACM", to use USB as a serial communication (accessed as a COM/ttyS on the host), which is used as a device, linked by /dev/serial. It's so possible to communicate in both way with a computer, without any additionnal driver (as CDC/ACM are reconized by main OSs), and for example to use hyperterminal-like (gtkterm works fine) to have a real VT100 screen and keyboard as calculator's terminal. :love:
- Ability to run userland process (with their own independant address space and kernel stack), directly from any ELF file accessible from VFS.
- Some syscalls already implemented for user->kernel context switch. open(), read(), write(), fork(), exit(), getpid(), getppid() and wait() are working, others are pending ;)
- As fork() syscall indicate, a beginning of multi-process system, allowing many process to run, and a (temporary) basic implementation of a scheduler to select active process and give him some CPU time. The multi process code have only some days, and will change quickly to have a cleaner implementation (and a real scheduler, with timers to change active process)...
- Some work was done on the SD Host Interface , to control directly SD Card, but it's a realy obscure part of the CPU (a 7720 controler used on the 7705 modifed version, and not documented at all even in 7720 documentation because of commercial secret and SD Association stupid rules). So it's a lot of reverse engineering on the Casio's OS code, and it's long to have something working. For now, it's possible to send/receive SD commands, initialize SD card, get meta-informations like size, volatages supported, manufacturer... but I probably missed something for data read/write, which is not working in my implementation. If I have some hour to spend to check everything, I plane to implement SD abstract block device, and ext3/fat32 filesystems.
If you want just to take a look on a user program code, you can read user/first_test_elf.c (https://www.gitorious.org/fixos/fixos/source/7eb91d5ceb15c063eb4b116b9bfe18bca2643fb8:user/first_test_elf.c). Of course, it's a file used for testing purpose, so do not try to look for a concrete goal in this program. ;D
The project is hosted on Gitorious (https://www.gitorious.org/fixos (https://www.gitorious.org/fixos) for the project and https://www.gitorious.org/fixos/fixos (https://www.gitorious.org/fixos/fixos) for the main repository).
Of course, it's an open source project, anybody who would like to take a look to the code, to modify it for educationnal and curiosity purpose, or to contribute to the project is welcome.
(about it, sory for two things :
- my english mistakes in my messages and in my code or comments, I probably do a lot of stupid grammar mistakes, especialy in old comments)
- the quality of the kernel code is not always as clean as I would like to, sometimes I do some black magic usage, or quick implementation of something, before to clean the code later when I have a better idea or I am motivated again. So I try to maintain beautiful and clean code, but it's possible for you to burn your eyes when reading some functions and tricks used :ninja: )
I probably forgot some things, so if you have any question, I will modify this post if the reply is relevant to take place in this introduction.
-
Oh sh*t, you're alive! WHat's up pal! It's been a very long time! Welcome back in our community!
And now, if you're wiling to provide some infos about configure GCC to compile for our cute blue calcs, I'm back on Casio C. (oh, and ping me on Skype sometimes, I'd love to talk about that!)
-
Wow, I have been wondering what happened to this project.
Do you think it could be easily ported to PRIZM? (Especially since it appears there are hardly any graphics functions yet)
-
Do you think it could be easily ported to PRIZM? (Especially since it appears there are hardly any graphics functions yet)
I added some informations about that in README (https://www.gitorious.org/fixos/fixos/source/7eb91d5ceb15c063eb4b116b9bfe18bca2643fb8:README) of the project, but in some words, it seems not so difficult to port it on Prizm (excepted for not documented at all parts, like filesystem, SD card...). I have no Prizm to work on that, and I don't remember exactly what is known or not about it's CPU.
If the documentation of the closest generic processor is correct for MMU, exception and interrupt handling, and some other registers, I think 80% of the job will be to write the CPU register definition header, and to change some constants values.
But if even for low-level things we have to deal with reverse engineering because of undocumented informations, it will be, of course, more difficult...
The arch-specific code is sometime used directly in generic code, so some changes have to be done before any portage, but if its targeting an other SuperH CPU it will not be a heavy task, I think/hope.
-
DO you have any explanations for using GCC on Windows? I'm trying to compile GCC for sh3-elf target, but it's not the domain I'm at my best.
-
Wow, this sounds pretty awesome!
-
Nice to see you still active in the Casio community. This project seems interesting,although I would probably not use it much, since I am not familiar with Linux-like stuff and mainly use my calc for games, but I hope this eventually come to fruition. Linux is now available on the TI-Nspire (although not installable as main OS) and some people managed to browse internet from it (while the calc is connected to an USB dongle)
-
It's not really Linux at the moment, it's an Unix-like kernel germ. THe terminal just takes input, no shell is done. I'll try to manage to code for this project. Do you have some tasks I could do Kris?
-
DO you have any explanations for using GCC on Windows? I'm trying to compile GCC for sh3-elf target, but it's not the domain I'm at my best.
Sorry for the delay, but in my case I am not familiar with MS Windows, so I can't help you a lot.
I think the best idea is to cross-build GCC from Linux (even using a VM only for that), or with a full Cygwin install. If I have some time I'll try to build latest GCC toolchain (binutils and GCC binaries, libraries are not needed, even libc), but I think there are some binaries available somewhere, used for some Prizm toolchain, and should be able to build for SH3 big endian target.
Nice to see you still active in the Casio community. This project seems interesting,although I would probably not use it much, since I am not familiar with Linux-like stuff and mainly use my calc for games, but I hope this eventually come to fruition. Linux is now available on the TI-Nspire (although not installable as main OS) and some people managed to browse internet from it (while the calc is connected to an USB dongle)
Nice to see you still active in both communities too ;)
To make things clear, this project is far far away to be considered as a Linux-like. Even if I finish everything I would like to do, it will probably be closed to the 1990's System V than to a modern Linux or BSD ;D
TI-Nspire has a lot of ressources availables compared to fx9860 (Prizm is more interesting, but still realy light to imagine a Linux running on it).
For example, in TI89-titanium and TI-Nspire, there is a USB "on-the-go" controler, allowing to act as a usual computer with USB devices (Keyboard, WiFi dongle, USB pen or HDD...). On Casio's calculators, it's only the device part of the USB which is available (so the calculator may act as a USB device, but there is no way for it to deal with other devices, and for exemple it's not possible to use WiFi dongle to give internet capabilities).
In the other hand, as everything is written from scratch, it's not impossible to design some part of the kernel to allow games to run at a correct speed.
It's not my main goal, of course, but basicaly what is needed is a near-realtime scheduler (or a "single process" mode to avoid useless context-switches), and some realy specific device drivers to have a fast access to display rendering and keyboard input.
And a good kernel can allow game developpers to have better (and more reliable...) bases to code their software. Imagine at least a good filesystem access, memory swaping to allow to use more memory than available RAM (even if in theory, swaping in EEPROM is either fast nor a good idea for EEPROM lifetime :ninja:).
In addition, something I realy want to do in a near future is shared library support, which can be also realy useful for games (lighter executables and possibility to update/fix libraries without compiling again).
Do you have some tasks I could do Kris?
It's not easy to say you a precise thing to do, mainly because I have no precise plan myself x.x
If you want to do some kernel functionalities, first try to read some piece of existing code to check a bit how the kernel is designed. After that, you can contact me using mail/pm, to find a better way to discuss a bit (irc, skype or something else).
A possible task should be to implement a serial device driver (using the real USART, not the USB emulating a serial device ^^).
It will be useful to both kernel (serial debugging) and user space (calc-to-calc comunication), and is a task which need to use the CPU's documentation, some plateform-specific code, and a device driver. So it's probably a good exercice to begin kernel programming :D
Now, about the project advancement, I had not a lot of free time, but I did some important things last days.
In a first time, a realy better keyboard handling. Before there were only some functions inherited from my old projects, like is_key_down(), and was only used for debuging purpose.
Now there is a real implementation of keyboard, with 3 abstraction layers. The first is the 'keymatrix" level, to check the state of physical keys, and interpret state changes as keyboard pressed/released. These informations are used by the logical-keyboard layer, able to handle key modifiers (SHIFT/ALHPA and later a character case modifier), and when I will have the time to implement it, the key repetition parameters. Finaly, the higher layer is implemented in the /dev/console driver, to act as standard input stream (with DEL used as backspace for corrections).
The keyboard state is updated with a 64Hz interrupt, and later will use some pin state detection interrupt to minimize overhead when no key are pressed, but because of hardware configuration there is no "perfect way" to handle each key press/release whithout some timer interrupt usage.
Everything is not finished, but now there is a correct stdin for interractive user process. I will have to add ioctls to be able to change the input properties (at least ICANON and a raw mode, which are realy needed for ncurses-like usage).
In a second time, the first partialy working implementation of exec-like syscall (execve), one of the most important syscall on a UNIX-like system.
So it's now possible to begin to write some useful userland software (a bash-like use mainly fork/exec, as well as a userspace init program).
-
I'm thinking about a GUI interface, but, well, it's too early to test things like this! XD
I saw some typos here and there, I'll make you a correction.
I'm plannning so to implement UART driver as long as I get a sh3-GCC here. Thanks for the suggestion.
Another suggestion : port ML in ASM. That would be very useful as a library.
-
Thanks for the clarifications. :) And yeah I could understand why Linux would be hard to run on a Casio calc. I am curious if it would work on the SD models, but I don't even know if they're still in production and even if they are, having Linux only on the SD models would cause the userbase to be so incredibly small compared to the TI-Nspire.
-
Sounds really awesome :) I hope we can test it soon ;)
-
might be out of topic, but can anyone tell me where I can find the said GCC toolchain to biuld .g1a executables?
-
might be out of topic, but can anyone tell me where I can find the said GCC toolchain to biuld .g1a executables?
I would like to know this as well
-
Same for me.
-
might be out of topic, but can anyone tell me where I can find the said GCC toolchain to biuld .g1a executables?
It's late to reply, but if anybody would like to have a sh3 GCC toolchain, as the SuperH architecture is not so common, the better way is to compile yourself GCC from sources. To avoid having to understand every details on GCC configuration, you can compile it through crosstool-NG (http://crosstool-ng.org/ (http://crosstool-ng.org/)).
The only other tool than GCC binaries themself you need is a G1A wrapper tool, and you will be able to create G1A from a GCC toolchain ;)
About the FiXos, as usual with me, I stopped a time, and I'm working again on it from one or two months.
The kernel like more and more to a usable UNIX kernel, the main changes from the last time I posted here are :
- Time management is now really better, kernel soft timers allow to defer kernel function execution, the 'real' time is maintained as time ellapsed since Epoch (read init time from RTC), and process kernel/user mode time ellapsed is counted. All this allow implementation of times() and gettimeofday() syscalls for user process.
- Functional UNIX Signals (only a small subset are implemented, but the more difficult part is done and fully working).
kill(), sigaction(), sigprocmask() syscall are usable, so it is possible to send signal to a user process from another process (or from kernel itself of course), to set signal handlers...
Some interesting unhandleable signals are also usable (SIGSTOP and SIGCONT, SIGKILL... ^^) - UNIX Pipes support, this is the first pseudo-IPC in FiXos ;)
It is now possible to create a pipe with pipe2() system call, and to use file descriptors to read/write data into the pipe. - Virtual Terminals! You may have several TTYs multiplexed on your screen (for now switching is done with F1~F6 keys, and only two TTYs are created with default settings, to minimize the need of memory).
For now, sessions, process groups and TTY owning are not implemented, but user process can use any TTY by opening /dev/tty<n> device ;) . - A proof-of-concept of direct screen rendering for userland is implemented as the "/dev/display device". This device implement only some ioctls :
- one to map dedicated VRAM into user space (mmap() is not implemented for now in FiXos :-[ )
- an other to get display information such as size in pixel, VRAM size, color format used for the VRAM (this provides a platform-independant way to manage a screen)
- a third for activating/deactivating the direct display rendering
- a last to display the content of the VRAM on the real screen
Using these ioctls allow to have graphical applications in userland, which is a first step to enable user friendly interface (but keyboard handling is only implemented in TTY, so its only the first step ^^). - Support for boot-time parameters (command line arguments for the kernel), as well in the bootloader and in the kernel itself. The first usable parameter is 'console={tty<virtual_terminal>|ttyUSB0}', to set the device to use for kernel console (a virtual terminal or the USB emulating a serial port).
The USB serial output allow to use minicom-like (gtkterm for example), running on a computer host, as the kernel debug console, which is really more powerful than the tiny calc screen ;D ). - Better kernel critical sections, with possibility to disable kernel preemption, or all interruptions (hard critical section).
A basic mutexes implementation was added, but not used for now. - Some important changes on the projet structure :
- Now, all definitions which are used by both kernel and userland are separated from the source tree (that includes definition of types and constants used in system calls, ioctl related stuff...)
- Some work done to avoid useless recursive makefiles without making a unique unmaintenable one (if you are interested about why, you can find some interesting article like this one http://aegis.sourceforge.net/auug97.pdf (http://aegis.sourceforge.net/auug97.pdf) on the internet).
- There is now a way to set configuration options both for compile-time and build-time (inside makefiles), using a "config.h" file, in which you can enable/disable/customize feature by changing some macros. This file is then transformed into a Makefile include file, so build system and compilation share the same configuration values.
I think I will try to make configuration easier with tools like Linux kernel's kconfig in the future (ncurse or GUI based tool to change configuration).
- A "generic pool allocation" implementation, to make easy to have some dynamic allocation inside the kernel (this is done for fixed-size objects, by allocating physical memory pages as needed, and using them like memory pools http://en.wikipedia.org/wiki/Memory_pool (http://en.wikipedia.org/wiki/Memory_pool) ).
I think it's more or less all the important changes. If you were patient enough to read all of them, you can figure the kernel is now close to be mature to support a consequent user space, but some crutial things still missing.
The memory management is not enough powerful and will need big improvements (to support Copy-On-Write, shared area of memory, memory mapped files with mmap() syscall...).
The other really missing feature remains the shared library support. I know on some platforms it is done in user space (using mmap() and oter stuff), but I still plan to implement it in kernel space, maybe with a couple of non-POSIX syscalls.
Finaly, userland need a working libc, even before dynamic libraries implementation, in order to be really usable. Playing with syscalls is fun, but a libc is more efficient to make real programs... [-.-]~
I'm considering to port Newlib for FiXos (Newlib is already working fine on SuperH bare-metal target, and is relatively easy to port on a new platform). I tried some month ago, but there was a problem with Newlib usage of automake, which is fixed now I hope...
I don't know if I will be active on this project during the summer, but I'll let you know if any progress is done.
And of course, if some experimented programmers are interested for working on FiXos, there is probably tons of things to do or to improve. I'm still interested to work on a port of FiXos for any other platform (Prizm or other SH4-based casio calculators, maybe m68k TI calc like TI-89, or even a x86/PC port could be fun ^^), because it will force a good kernel design to be able to separate arch/platform specific code from generic kernel code, and in the same time a portage often highlights hidden bug in a software. ;)
-
I'd be glad to help this summer, but it'll need for me to read a lot of docs, I never touched a *nix kernel, so...
But, if you can give me pointers, I'd help to port to Prizm! :p
-
Sorry I tl;dr'd the thread but I guess this is like KnightOS for the (more powerful) Casio calcs ?
-
Just make sure that the OS plays nice with the calculator hardware, because with certain PRIZM add-ins, if you run OS 2.0 and do certain things with your calc, you can permanently brick it. (there's a topic about this on Cemetech with more details)
-
Sorry I tl;dr'd the thread but I guess this is like KnightOS for the (more powerful) Casio calcs ?
In fact, for now, the name of the project is a lie (FiXos is only a kernel currently, but it should become more or less a "full" operating system someday). So it should look like KnightOS, but more focused on a "powerful" kernel, because the targeted CPU is far more complex (MMU, USB controler, SD Card, and dozen of other on-chip peripherals) and hardware is enough powerful to support a bit heavier kernel.
Just make sure that the OS plays nice with the calculator hardware, because with certain PRIZM add-ins, if you run OS 2.0 and do certain things with your calc, you can permanently brick it. (there's a topic about this on Cemetech with more details)
I read this topic, and unfortunately with the Casio boot design, with everything on the same EEPROM, I imagine there is small chance to brick the calculator when rebooting at the (very) worst time of an EEPROM write/erase operation.
For now I try to avoid playing too much with EEPROM, it will be necessary, but I prefer to have a stable kernel before to do that.
Of course, I'll do the best to avoid any problems, especially on other calcs than my ones (in fact I'm surprised to never broke my calc as I ran thousand times my kernel to debug it, with really not stables things sometimes), but I can't be 100% sure that will never happen (as Casio themselves apparently failed...).
Only thing I can say is that a buggy free software have a chance to be fixed if a such a thing ever happen, and a protected-mode kernel which run user applications in user-mode limits the possibles location of a such issue ;)
-
Yeah I figured these calculators stand closer to the 68k line than to the z80 line in comparison. But the goal is still the same : a UNIX-ish OS for calculators.
-
It's a shame that the PRIZM and 9860G don't use ARM processors, because it would have made it much easier to port TI-Nspire stuff to it. The HP Prime uses ARM too (although there isn't any real doc on how to create a third-party firmware for it nor a mod of the existing ones to allow ARM assembly and C).
-
Well on the bright side they have a good c compiler so it makes the non asm stuff portable. All the HW is incompatible across the HP, Nspire and Casio calcs though.
-
I spend several days to write many proof-of-concepts around FiXos, and I finally added some intersting things to the kernel ;)
First of all, a lot of work was done on dynamic libraries!
After some reflexion and studying existing implementations on several platforms and OS, I decided to implement shared object stuff in kernel. In most modern OS (linux for example), dynamic linker is a userland tool, and the kernel is agnostic about any dynamic linking.
As FiXos kernel already have some ELF facilities, it seemed easier and lighter to have a kernel-part implementation (so userland is simpler to understand and to maintain). I also implemented it in a way it's still possible to write a full userland dynamic linker, without breaking the kernel implementation (and it's even possible to disable any kernel dynamic linker related stuff at build time with a config option ^^).
The main problem I faced was than dynamic linking is deeply dependant of the toolchain target arch/OS, and the one I used is configured to target sh3 arch without operating system (and it makes no sense to support dynamic linking for unexisting OS).
So, the two choices I had were to do some changes in GCC's sources to target my OS, or to cheat a bit in order to make a part of the normal linker job "by hand". As changes in GCC is not a good idea for now (a lot of time waste to maintain and distribute it...), I did the second choice.
Mainly, to build a shared library for FiXos, the following is done (and automatized by some shell scripts and Makefile itself) :
- Mark each "public API" function with special GCC's visibility attribute "default".
- Compile each file with -fpic or -fPIC option (which make Position Independant Code, allowing to load the library at any arbitrary virtual address), and -fvisibility=hidden (which "hide" every symbol not explicitely marked as part of the public API in C code).
- Link all resulting object files, the -shared option is used but seems to have no effect for sh*-elf toolchains (so the resulting file is an ELF executable file, but this make no real difference with an ELF shared object).
- Extract every symbol marked as "GLOBAL" with visibility set to "DEFAULT" from the linked file. This should be the list of public API symbols, and it is used to achieve to distinct things :
- strip every non-public symbol information to make the dynamic library lighter!
- generate a "stub" assembly file for each public function
- The stubs are designed to be staticaly linked to client executable (its a set of some instructions for dynamic symbol resolution, and the symbol name as a C string), and go to the .plt section of the executable.
They are compiled and placed together in a single static library named "lib<name>-shared.a".
When a program should use a dynamic library, you just have to link it with the corresponding stubs library, and to call the functions exactly as there were normal, local ones.
The first time a dynamic-linked function is called (which, in fact, call the corresponding stub code), some complex job is done, including a specific system call to find the corresponding symbol by its name.
Then, all subsequent calls will be faster by using the cached value of this symbol (in fact the overhead is only 4 instructions, so it should not be a problem for performances).
The kernel ELF loader will only care about a special segment of the ELF (the one of type PT_SHLIB, defined by ELF specification but with no predefined semantics). It expects to find a list of zero-terminated strings, each one corresponding to the shared object name (the library name, with no path). For each name, it will try to find and load the content of the shared object. Once loaded to an arbitrary not used address range, the kernel try to "relocate" it (a vector of absolute-based addresses is located, and each element is incremented by the base address where the code was loaded).
Please note this segment is populated automaticaly (when at least one of the stubs for a given library is used, the library name is added to this section).
I'll have to write some documentation about all that in the project source tree, because it involves many complex things, and it's possible that my implementation fail for complex patterns.
In addition, it's only an early implementation (the shared memory is not implemented, only one library may be linked to a process...), but the more difficult part is done and seems to work fine for now!
An other cool thing added is a more powerful post-mortem debug information display.
A new function, kdebug_oops(), may be used when the kernel reaches an abnormal state (to not say "crash" [-.-]~ ), which should display every potentially useful information (some CPU registers, current process related stuff like a memory map, and a call trace of the kernel stack).
Unfortunately, SH3 calling convention does not allow a very good call trace without a lot of additional information (more or less all the "unwind" data needed for C++ exception handling...), so the current code only display the "guessed" calls, and may repport some false-positive :/
Even if it's usefull to have addresses of caller functions, having their names allow to don't spend much time to find it by looking at the disassembly dump of the kernel. So a bit tricky mechanism may be triggered by the config option CONFIG_DEBUG_SYMBOL_NAMES to extract all symbols located in ".text" section, and to include a table for "begin address" <-> "symbol name" conversion (it is possible to keep symbol names in ELF, but it takes more memory and its more difficult to read...).
Finally, this configuration option takes many space (currently around 30% of the kernel size...), but leads to a more understandable call traces, and so should be used for kernel development.
To make this more visual, here is an example of displayed message when an unaligned data is accessed in kernel mode :
> Address : 0x3FFFFFF3
> SPC Value = 0x880025F6
Fatal kernel oops!
(CPU Access Violation (R/W))
Following informations may help :
Running pid 1 (asid=0)
Memory Map :
DIR(0x10000000~0x10008000):
vvuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu
DIR(0x3FFF8000~0x40000000):
uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuv
(End of Memory Map)
Stack=0x880126F0
Call trace:
@0x880126F4: <_exception_handler + 272> (0x88008AC0)
@0x880126F8: <_printk + 0> (0x88006E78)
@0x88012708: <infloop + 0> (0x8800896A)
@0x88012710: <_sys_ioctl + 0> (0x88005DD8)
@0x88012714: <_interrupt_inhibit_all + 0> (0x88008680)
@0x8801271C: <_exception_handler + 0> (0x880089B0)
(End of call trace)
---- Previous Context ----
PC: <_display_ioctl + 58> (0x880025F6)
PR: <_exception_handler + 192> (0x88008A70)
Stack=0x8801277C
Call trace:
@0x88012780: <_exception_handler + 192> (0x88008A70)
@0x8801278C: <infloop + 0> (0x8800896A)
@0x880127A0: <_exception_handler + 0> (0x880089B0)
(End of call trace)
Of course, without knowing the kernel internals it's difficult to interpret, but in this case, the error was located in _display_ioctl(), itself called by _exception_handler(), which is displayed in the second context info.
Now, I try to add the needed syscalls to allow a fully functionnal port of Newlib.
Some work is done, and it is still possible to build Newlib for sh3-unknown-fixos target, but without a working sbrk() syscall, malloc() can't work and a large part of Newlib depends on it.
The last syscalls really needed are sbrk() and stat(), the others are either not essential for the most part of the library (symlink(), unlink(), chown()...) or implemented by using other syscalls (isatty() may be replaced by some ioctl() calls or fstat()).
-
Wow the feature lists sound very existing. Very nice work.
-
Awesome. :D
-
So many TL;DR posts so I skipped the thread :trollface:
Anyway this looks prettty interesting, definitely keeping an eye on this thread even though I don't have a fx9860.
-
Is there any functions done to start a shell? I could do one if you want!
EDIT : FXShell (consider that name already taken, guys)
-
Once again, some news of the kernel advancement ;)
Of course, the summer is not the better time to code everyday, but since my laptop has a great backlight, I can even enjoy the sun and write code at the same time! [-.-]~
Main new features are :
- Dynamic memory allocation, through sbrk() system call. This is not as powerfull as mmap() using anonymous area, but it's enough for malloc() implementation!
- New device, /dev/keyboard, for low-level keyboard handling. It's an event-based device, which deal with raw key codes, allowing efficient way to implement IsKeyDown()/GetKey() in userspace. Of course, its more designed for graphical interface and games, console applications should deal with terminal input!
It is possible to open this device with O_NONBLOCK flag to avoid waiting if no new keyboard event are present. - BSD-like sysctl() syscalls, to deal with kernel information from userspace.
For now, available sysctls are mainly kernel/architecture information (kern.ostype, kern.osrelease, kern.osbuilddate, hw.machine, hw.model), and, more interesting, process information (kern.proc.pid.<pid> and kern.proc.all).
Process related sysctls are the BSD way to look for process state from userspace (for ps/top and other utilities), instead of the /proc special filesystem present in Linux. For now, some data are still missing (process name and arguments...) but it is still possible to have a minimal top-like application ;) - Average process CPU usage is now calculated on a short period (around 1 second), in addition to total kernel ticks spent in kernel and user mode. It is never used internally, only designed to be accessed from userspace (using sysctl()).
- A special kernel process (idle process) is now scheduled when no other process are running (it is using SuperH 'sleep' assembly instruction, so the power consumption is reduced when CPU is not needed 100% of the time).
- Wait queues are now implemented, to allow a blocking syscall to put the corresponding process in a sleep state (only uninteruptible sleep is supported for now) until a specific event. For example, a process reading keyboard input will sleep until some characters are ready for it, instead of actively waiting for a character.
- nanosleep() syscall implementation, to force a process to sleep some time. For now, this sleep is not interruptible, so a signal can't be handled until timeout.
- stat() and fstat() syscalls!
- getdents() syscall, used for listing directory entries (files and sub-directories). I think this will be the only way to list a directory content, and, as Linux and every BSD, the POSIX dirent.h functions will be part of the libc.
The getdents() syscall is a bit tricky, and not really pleasant to use in a relatively high-level application. In the same time, the latest C standard do not define anything related to directory, so I can't rely on Newlib to have a nice interface.
Newlib should now be more or less functionnal now, I did some tests recently (mainly for malloc() and date/time related functions) and its working fine. I will try to make a new repository for the Newlib port, but in the same time I think Newlib is a good temporary solution (probably the fastest way to have a working libc/libm on any new system/architecture!), but it should be more interesting in the medium/long run to use Newlib sources but with a lot of custom changes in a lot of files.
As Newlib is designed to separate platform-specific code, and to never deal directly with the generic implementation, its probably better to don't keep the Newlib build system if changes are too much importants...
Some week ago, I tried to port a game from PierrotLL on FiXos (gravity duck for fx9860), mainly to see how difficult it is and to discover potential issues in the case of a "real-time" interactive application.
So I ported MonochromeLib in a first time (using the /dev/display device, for direct rendering), which work as well as a static library and as a dynamic library with acceptable performances (around 500 FPS for overwriting the whole screen and display it).
In a second time, some changes were needed in GravityDuck code itself for replacing Casio OS functions used (GetKey(), IsKeyDown() and some functions related to ticks ellapsed to limitate the frame rate). An other function (PrintMini()) was simply removed (a similar function should be added to MonochromeLib), but it is not a big deal as Gravity Duck do not use a lot of text... As the game makes use of malloc(), I also linked the Newlib libc, but I think it is the only libc function used.
The final result is really encouraging, the game run at normal speed, and everything seems to work as the original version! :P
I didn't released the ported Gravity Duck for now because I did it for testing purpose, and everything is not really clean, but I will try to do that in order to show than it is still possible and relatively easy to write real applications for FiXos (less than a day for porting MonochromeLib and Gravity Duck, and less issue than I feared).
Is there any functions done to start a shell? I could do one if you want!
I think more or less everything needed is working, if you want to do that, but you will really need a libc if you want to have some helper functions, else you will have to deal only with syscalls.
Two things missing for a such project are the current/working directory (for now processes are not associated to a working directory, and every path need to be absolute for open(), stat() and similar syscalls), and maybe some ioctls and VT100-like escape codes for the TTYs. I will also need to implement process groups, sessions and controling terminal for a better UNIX concept support.
-
Good to hear that you managed to run a Casio OS game into FiXos. It's a good step in the right direction for people interested in this OS who might still want to play older games like this. :)
-
OK I'm officially on the project. Let's see if I can make something like a shell..
EDIT : stupid echo routine
static void test_echo(int fdin, int fdout) {
char buffer[33];
ssize_t result = 0;
int index = 0;
char current_char;
while(1){
read(fdin, ¤t_char, 1);
buffer[index++] = current_char;
if(current_char == '\n' || index == 32) {
result = write(fdout, buffer, index);
if(result != index)
break;
index = 0;
}
}
}
I think I should find a way to cleanly read a whole line of text.
-
I'm on the rails again as a side side project. I started to parse arguments given from a command line to train my UNIX programming skills. That'll come quite handy for the minishell project i plan to do.
Here's the code atm for anyone who wants to dig into (only for PC as test, I haven't ported it to FiXOS, but that should be easy) :
#define _GNU_SOURCE
#include <unistd.h>
#include <string.h>
#define BUFFER_LENGTH 128
#define COMMAND_LIST_SIZE 8
#define COMMAND_SIZE 32
void next_command(char* current_command, int* current_command_index, int* command_index) {
current_command[*current_command_index] = '\0';
*current_command_index = 0;
*command_index += 1;
}
void parse_command(char* buffer, char command_list[COMMAND_LIST_SIZE][COMMAND_SIZE], int buffer_index) {
int command_index = 0;
int current_command_index = 0;
char in_quotes = 0;
char in_double_quotes = 0;
char c;
int i;
for(i = 0; i < buffer_index; i++) {
char *current_command = command_list[command_index];
c = buffer[i];
if(in_quotes) {
if(c == '\'') {
in_quotes = 0;
}
else {
current_command[current_command_index] = c;
current_command_index++;
}
}
else if(in_double_quotes) {
if(c == '"') {
in_double_quotes = 0;
}
else {
current_command[current_command_index] = c;
current_command_index++;
}
}
else {
if(c == '"' || c == '\'') {
if(current_command_index > 0) {
next_command(current_command, ¤t_command_index, &command_index);
}
if(c == '"')
in_double_quotes = 1;
else
in_quotes = 1;
}
else if(c == ' ' && current_command_index > 0) {
next_command(current_command, ¤t_command_index, &command_index);
}
else {
current_command[current_command_index] = c;
current_command_index++;
}
}
}
next_command(command_list[command_index], ¤t_command_index, &command_index);
for(i = 0; i < command_index; i++) {
write(1, "=>{", 3);
write(1, &command_list[i], strlen(command_list[i]));
write(1, "}\n", 2);
}
}
int main()
{
char c;
int buffer_index = 0;
char buffer[BUFFER_LENGTH + 1];
char command_list[COMMAND_LIST_SIZE][COMMAND_SIZE];
while(1) {
read(0, &c, 1);
if(c == '\n') {
// TODO : manage command
parse_command(buffer, command_list, buffer_index);
buffer_index = 0;
}
else if(buffer_index <= BUFFER_LENGTH) {
buffer[buffer_index++] = c;
}
}
return 0;
}
-
Good to hear it's still progressing :)
-
(http://2.bp.blogspot.com/-s5AEA51c1jw/UfW-_UOaogI/AAAAAAAABwA/kyCLrDHJuvM/s1600/futurama-good-news-everyone.gif)
My shell project seems to work. Seems because I don't know if it actually tries to open anything but I get the reply when the program execution failed, so I consider my fork/exec/wait routine working!