Red

News

Red/System goes beta!

This is a major milestone reached on the Red roadmap. Red/System is now in a beta state, which means that it is good enough, both in design and implementation to be used for building applications.

A huge collective effort has been made by all contributors to reach that state during June, in a single month, we produced:

  • 190 new commits (total = 354 now), more in a month than in the three previous ones!
  • 3600 more unit tests (total = 6230 now)
  • 2800 more LOC (total ~ 6000 LOC now)
  • 70 issues closed on github’s tracker
  • 8 more revisions of specification document

This is what can be called an “intense” month of work!

There are still a lot of features that are lurking in the corner for inclusion. Red/System features are not set in stone, the language will evolve to match our needs, as a support tool for building Red. It will now follow a bit more organized release cycle to help users and contributors have a more stable base to work with.

The version number has been bumped to v0.2.0. From now on, only non-disruptive changes will be pushed on the main branch, other changes will go into a separate development branch until we decide to make a new release, merge new features and increment the version number. This cycle should be short, on a weekly basis (each week or two, depending on the development branch content). If this organization does not fit well the development progress, we will adapt to a more adequate one.

I cannot resist to show you the new unit testing output when running the complete tests suite (just run the %tests/run-all.r script):

Quick-Test v0.5.0
REBOL 2.7.6.3.1
Red/System Test Suite

ok - logic.............................76 / 76
ok - integer............................0 / 0
ok - byte..............................40 / 40
ok - c-string..........................20 / 20
ok - struct...........................116 / 116
ok - pointer...........................68 / 68
ok - cast..............................66 / 66
ok - alias.............................22 / 22
ok - length............................11 / 11
ok - null..............................11 / 11
ok - not...............................24 / 24
ok - size..............................36 / 36
ok - function...........................2 / 2
ok - exit...............................9 / 9
ok - return............................24 / 24
ok - modulo............................20 / 20
ok - infix..............................2 / 2
ok - conditional.......................13 / 13
ok - byte-auto.......................3492 / 3492
ok - integer-auto....................2106 / 2106
ok - alias-compile......................3 / 3
ok - cast-compile......................16 / 16
ok - comp-err...........................4 / 4
ok - exit-err...........................3 / 3
ok - int-literals-err...................1 / 1
ok - output.............................3 / 3
ok - return-err.........................5 / 5
ok - conditions-required-err...........23 / 23
ok - inference-err......................1 / 1
ok - callback-err.......................3 / 3
ok - infix-err..........................2 / 2
ok - not................................1 / 1

ok - Red/System Test Suite...........6230 / 6230
  in 0:00:16.3

A big thank to all the following contributors who are working hard each day to help me bring Red to life:

Andreas Bolka

Rudolf W.Meijer

Kaj de Vos

Peter W A Wood

Cheers!

EDIT: for those coming here for the first time and willing to test Red/System, please follow the link to Github’s repository, you will find some instructions there for getting started.

Mac OS X port released!

Red/System Mac OS X port is now completed and on par with Windows and Linux ports. The new Mach-O file emitter supports all current Red/System’s features including #syscall and #import directives.

Now that all major 3 OS are supported, I can say that the Mac OS X file ABI and function call ABI are really the most complicated one to support from scratch, the Windows one (PE/COFF) being the simplest (with its own set of oddities, but simpler) and Linux’s ELF somewhere in between. Among the issues we had to identify and deal with:

  • Stack alignment on 16 bytes constraint at the point of every library call (meet with the infamous “gate keeper” code causing segfaults and bus errors on misaligned calls)
  • Extra stack entry for each BSD system call.
  • Working around dynamically linked libraries info extraction that need to be copied from each library header to the generated executable image. This could have prevented cross-compilation.
  • The need for direct CALL opcode in the generated IA-32 code instead of indirect ones used for PE and ELF library function callings.

After a lot of efforts spent mostly in debugging, here is the result:


Image

As you can see, it is fully functional and all unit tests are passing!

A couple of notes about usage:

  • The library you declare in #import definition needs to be located in /usr/lib/ or else, you need to provide a full path to the library.
  • The file emitter should support up to 255 imported libraries (Mach-O limit), but it has been tested only with one so far (libc.dylib), so if you have anything odd happening with more than one imported library, please fill a bug report in github’s tracker, or post a message to the mailing-list.

Enjoy!

ARM support released

We are pleased to announce that Red/System v0.2.3 is out, extending Red to the mobile world with a complete support for Linux on ARMv5+ processors. The new port supports 100% of Red/System features and passes successfully the 8537 unit tests.

For those of you interested in more details, our ARM port targets the v5 family, with 32-bit instructions (no Thumb support) and optional literals pools (use -l or –literal-pool command-line option when compiling) that implement a global allocation strategy (unlike gcc’s function-local allocations).

New compilation targets have been provided for Linux and derivative OS running on ARM:

  • Linux-ARM
  • Android

Android support is using the same code as Linux-ARM, only differing in libc and dynamic linker names.

Currently, as Red/System only works on command-line in Android, you need a special loader to download the executable and run it. This can be achieved using the NativeExe app. You will need to allow temporary installing apps from non market sources (Settings > Applications > Unknown sources). Also, your local 3G provider might be filtering out executables downloaded this way, you can workaround that by either manually loading the NativeExe-0.2.apk file with adb, or share a wired Internet connection with your mobile device.

You can easily install NativeExe app by just typing the following URL in your Android web browser:

http://gimite.net/archive/NativeExe-0.2.apk

or by scanning this QR-code instead:

Image

Once done, input in the second field: http://sidl.fr/red/hello and hit [Run].

Here are a few screenshots of HelloWorld tests:

Image
hello.reds script on Android 2.2
Image
hello.reds script on Linux Debian 6.0 in QEMU

Andreas has also reported that it’s working fine on Nokia N9 devices.

Supporting Android and iOS API

The next steps to enable building full apps on Android and iOS are:

  1. Support PIC compilation mode: indirect access to all global variables and data. This is a requirement for building shared libraries on UNIX (but optional on Windows).
  2. Add shared library generation to file format emitters. This will require some new compilation directives to mark the exported code parts.
  3. Build a bridge with Java for Android and Objective-C for iOS/OSX. This generic bridge would allow accessing all the objects and methods of the host and send back to Red all I/O events.

Such approach will allow us to build easily Android or iOS apps without having to write a single line of Java or Objective-C code, while providing the full power of Red. At least, that’s the theory, we’ll see in practice if it’s up to our expectations. Also, cross-compilation should be fully available for Android (producing Android apps from any OS), but code signing and app publishing requirements might make it impossible for iOS and require a MacOS X with Xcode for producing apps (if you know workarounds, let us know).

The PIC support should be doable in a few days, the support for shared library generation might take a bit more time. Anyway, theses tasks will need to be multiplexed with Red runtime & compiler implementation, so don’t expect significant progress before a month.

In the meantime, you are welcome to test the ARM port of Red/System and hack Android and upcoming Raspberry Pi devices using it. ;-)

Cheers!

Red alpha release

We have all waited long for this to happen, but it is finally there: first Red alpha release!

This is the biggest milestone since the initial release of Red System, as it marks the entrance of the higher layer of the Red software stack. There’s still another layer pending, the JIT compiler, but that’s for 2013, when Red will be self-hosted.

What Red can do so far’

So what happened since the first Red compiler release a little more than a month ago’ We have added quite a lot of features actually:

  • 15 datatypes: block!, string!, char!, integer!, logic!, none!, word!, lit-word!, get-word!, set-word!, refinement!, action!, native!, op!, unset!.
  • 19 actions: make, form, mold, add, divide, multiply, subtract, append, at, back, clear, head, index', length', next, pick, poke, skip, tail.
  • 21 natives: if, unless, either, any, all, while, until, loop, repeat, foreach, forall, print, prin, equal', not-equal', strict-equal', lesser', greater', lesser-or-equal', greater-or-equal', not.
  • 10 infix operators: +, - , *, /, =, ==, <, >, <=, >=.

String! datatype fully supports Unicode from end to end and in a cross-platform way, according to our plan (see the hello.red demo script).

We also have some unit tests (223 tests right now), but that number should rapidly increase once we will be able to fully port QuickTest framework to Red (in a couple of weeks, once Red will gain functions).

All those features were mostly coded in the last month, some parts are still lacking (e.g. refinements support for actions/natives) but that gives you an idea of the pace at which Red will be further developed.

However, we haven’t started yet writting the documentation for Red language, we should decide soon on the kind of documentation we want (hint: you can make propositions about that on the Red mailing-list or FB page). Red API documentation will most probably be generated automatically from source code by extracting doc-strings.

What’s not yet there’

A lot of things, but they will be added progressively until the end of the year, where we should have a pretty complete Red (bootstrap) implementation. So, the main pieces to implement are:

  • Red runtime lexer (LOAD native)
  • Functions and objects support
  • I/O support (including networking)
  • More complete memory allocator and a garbage collector.
  • Concurrency support
  • Additional datatypes

Can I try it already’

Red codebase is available at github, just clone it locally or download a packaged version. Some simple instructions to setup and run it can be found here. As we are still at the bootstrap stage, the installation process and usage is more complex than it should be, once self-hosted. Anyway, we should be able to deliver some test binaries soon and simplify the usage for those who want to play with the alpha releases.

How does Red work’

Currently Red scripts are statically compiled using the %red.r front-end wrapper. The Red lexer loads the UTF-8 encoded Red scripts in memory in form of blocks of values. Those blocks are then compiled to a Red/System script, that gets itself compiled to native code and outputs an executable binary. The Red/System compiler has been enhanced to be able to compile source code directly from memory.

Some compilation options are already available, the famous -t option from Red System is also present, allowing Red to cross-compile any script as easily as with Red/System.

This is just the beginning, so for now, whole Red runtime library is compiled with user scripts each time. We will add modular compilation support at some point to get the full compilation speed back.

You can test the few example Red scripts from the repository. Here is what the demo.red script outputs on some of the platforms Red can already run on:

Image
Image
Image
Image

The missing characters in some of the screenshots are Chinese ones as I only have font support for them on Ubuntu.

Red was originally conceived to be statically typed. However, during the early stages of Red implementation, it appeared that an hybrid typing system will be more adequate for a language that will support high-level scripting, without the burden of a sophisticated, but slow, type inference engine. The overhead of such inference engine can be very high (cf Scala slow compilation speed ). The hybrid type system works very simply, typing is optional, but if local variables and function returned values are typed by user, the compiler will be able to generate faster code (more than an order of magnitude faster for, e.g., pure math expressions) and catch more programming errors at compile time.

Red compiler does not do any optimization so far, it basically outputs the equivalent of an unrolled interpreter loop. Some optimizations will be added along the development of the bootstrap version of Red, but the full extent of possible optimizations will be left for the self-hosted Red. Firstly because the current compiler code is disposable, so we don’t want to invest too much time in it, secondly because the final JIT compiler will allow even deeper optimizations thanks to reflection and profiling information collected at runtime.

Red/System changes

Red/System has been improved significantly during the Red early development:

  • function pointer support has been extended and improved, dereferencing is now possible.
  • 20 issues have been fixed.
  • some long-standing bugs with GTK+ and other bindings on ARM platforms have been fixed.
Image

So, what’s coming next’

Red will now mature very quickly, as functions and objects will make their entrance in the next days/weeks. A new Red alpha release is planned very soon, probably for mid-november with a lot of new features.

The work on shared library generation for MacOS and Linux platforms is on-going. Once available, we’ll be able to literally “plug” Red in a lot of third-party apps, including other programming languages. Also, this will enable (or make easier) the bridging with some external systems like Java, .Net and objective-c, in order to fully support the main mobile platforms.

As you can see, next months will see some important parts of Red come to life, be sure not to miss them, follow us on our mailing-list, on Twitter, IRC and Facebook.

Cheers!

Red is born

Image

Yesterday, I finally got my first real Red program compiling and running:

Red []
print 1

This doesn’t look like much but it proves that the whole current Red stack is working properly. Red compiler generates Red/System code in memory, that is then compiled to native code and linked in a 14KB executable file (contains the full Red runtime with currently 9 datatypes partially implemented).

The baby needs a few more days in the nursery before I commit the new code to the v0.3.0 branch on Github. Once the existing datatypes will be more complete and once we choose how to deal internally with Unicode strings, we should be able to release the first Red alpha.

The current Red compiler is pretty simple and should remain light in the future. I have finally chosen an hybrid dynamic/static type system, to avoid diving into a complex type inference engine now, as I realized that once we get out of the bootstrap and have the final Red JIT-compiler, it will be much more easier to achieve. Also, I want to pass that bootstrap stage as soon as possible, because it is really limiting the full Red potential.

Stay tuned, the next months will be fun! ;-)

The Journey Begins!

I’ve announced yesterday at ReBorCon 2011 conference my intention to build a new programming language, strongly inspired by REBOL. See the slides presented during the conference below.

If you’re experiencing issues seeing the slides in flash format, here’s a PDF version.

More to come soon, stay tuned!

Posts:

Tags: