Red

News

Red/System v0.2.1 released

This new release is bringing a lot of important bug fixes and some new features for Red/System. The new PRINT function might break some existing scripts, but should be easily fixable.

New features

  • Variable-arguments function support
  • RTTI function limited support
  • Polymorphic and variadic PRINT function
  • Command-line arguments access
  • Added stack low-level manipulation native functions: push, pop
  • Access to stack pointer and stack frame pointer (reading/writing)
  • Subtracting two pointers is now possible
  • Preprocessor improved to solve macros recursively
  • New built-in hexdump debugging functions
  • External library access unit tests (thanks to Peter)

PRINT function examples:

print 123 
123 

print "hello" 
hello 

print [123 "hello"] 
123hello 

print [123 tab "hello" lf "world" lf] 
123    hello 
world 

a: 1 < 2 
b: "hello" 
print [a tab b/2] 
true    e

Bugfixes

  • Complex arithmetic expressions involving pointers, path access and type casting are now much more reliable. 89 new unit tests were written to help fix the bugs and track regressions.
  • Complete list of fixed issues available in Github’s tracker.

Specification document

  • Updated to match all new and changed features

Red runtime

  • Memory allocator implemented in Red/System, documentation is pending.

As you can see, no vacation this summer for Red project!

Red/System v0.2.2 released

This release is mainly a bugfix release that solves several old issues. It is also now correctly synchronized with all the current bindings done for Red/System. The main changes are:

  • Internal compiler refactoring of: expressions compilation, type casting and ANY/ALL support.
  • Greatly improved runtime error reporting: now it reports both source line number and source file name where the error occured. It works in debug mode only (-g command-line option).
  • Aliased struct names can now be tested separately in typed (RTTI) functions.
  • Callback function attribute removed. It is no more needed and any function can now be used as a callback. In addition a new cdecl attribute is now accepted to allow the switch to C calling convention, when passing a function as argument to an imported C function.
  • 21 issue reports closed.
  • More than 2000 new unit tests were added (mostly generated using scripts wrote by Peter WA Wood) for a total of now 8613 tests.

ARM support released

We are pleased to announce that Red/System v0.2.3 is out, extending Red to the mobile world with a complete support for Linux on ARMv5+ processors. The new port supports 100% of Red/System features and passes successfully the 8537 unit tests.

For those of you interested in more details, our ARM port targets the v5 family, with 32-bit instructions (no Thumb support) and optional literals pools (use -l or –literal-pool command-line option when compiling) that implement a global allocation strategy (unlike gcc’s function-local allocations).

New compilation targets have been provided for Linux and derivative OS running on ARM:

  • Linux-ARM
  • Android

Android support is using the same code as Linux-ARM, only differing in libc and dynamic linker names.

Currently, as Red/System only works on command-line in Android, you need a special loader to download the executable and run it. This can be achieved using the NativeExe app. You will need to allow temporary installing apps from non market sources (Settings > Applications > Unknown sources). Also, your local 3G provider might be filtering out executables downloaded this way, you can workaround that by either manually loading the NativeExe-0.2.apk file with adb, or share a wired Internet connection with your mobile device.

You can easily install NativeExe app by just typing the following URL in your Android web browser:

http://gimite.net/archive/NativeExe-0.2.apk

or by scanning this QR-code instead:

Image

Once done, input in the second field: http://sidl.fr/red/hello and hit [Run].

Here are a few screenshots of HelloWorld tests:

Image
hello.reds script on Android 2.2
Image
hello.reds script on Linux Debian 6.0 in QEMU

Andreas has also reported that it’s working fine on Nokia N9 devices.

Supporting Android and iOS API

The next steps to enable building full apps on Android and iOS are:

  1. Support PIC compilation mode: indirect access to all global variables and data. This is a requirement for building shared libraries on UNIX (but optional on Windows).
  2. Add shared library generation to file format emitters. This will require some new compilation directives to mark the exported code parts.
  3. Build a bridge with Java for Android and Objective-C for iOS/OSX. This generic bridge would allow accessing all the objects and methods of the host and send back to Red all I/O events.

Such approach will allow us to build easily Android or iOS apps without having to write a single line of Java or Objective-C code, while providing the full power of Red. At least, that’s the theory, we’ll see in practice if it’s up to our expectations. Also, cross-compilation should be fully available for Android (producing Android apps from any OS), but code signing and app publishing requirements might make it impossible for iOS and require a MacOS X with Xcode for producing apps (if you know workarounds, let us know).

The PIC support should be doable in a few days, the support for shared library generation might take a bit more time. Anyway, theses tasks will need to be multiplexed with Red runtime & compiler implementation, so don’t expect significant progress before a month.

In the meantime, you are welcome to test the ARM port of Red/System and hack Android and upcoming Raspberry Pi devices using it. ;-)

Cheers!

0.5.0: Objects support

We are bumping the version number up higher as we are bringing a new foundational layer and important construct to Red: object! datatype and contexts support.

Supporting objects in the Red interpreter is relatively easy and straightforward. But adding those features in the compiler has proven to be more complex than expected, especially for access-path support, paths being especially tricky to process, given their highly dynamic nature. Though, I have pushed Red beyond the edges I was planning to stop at for objects support, and the result so far is really exciting!

Object model

Just a short reminder mainly intended for newcomers. Red implements the same object concept as Rebol, called prototype-based objects. Creating new objects is done by cloning existing objects or the base object! value. During the creation process, existing field values can be modified and new fields can be added. It is a very simple and efficient model to encapsulate your Red code. There is also a lot to say about words binding and contexts, but that topic is too long for this blog entry, we will address that in the future documentation.

Object creation

The syntax for creating a new object is:

make object! <spec>

<spec>: specification block

Shorter alternative syntaxes (just handy shortcuts):

object  <spec>
context <spec>

The specification block can contain any valid Red code. Words set at the root level of that block will be collected and will constitute the new object’s fields.

Example:

make object! [a: 123]

object [a: 123 b: "hello"]

c: context [
   list: []     
   push: func [value][append list value]
]

You can put any valid code into a specification block, it will be evaluated during the object construction, and only then.

Example:

probe obj: object [
    a: 123
    print b: "hello"
    c: mold 3 + 4
]

will output:

hello
make object! [
    a: 123
    b: "hello"
    c: "7"
]

Objects can also be nested easily:

obj: object [
    a: 123
    b: object [
        c: "hello"
        d: object [
            data: none
        ]
    ]
]

Another way to create an object is to use the copy action which does not require a specification block, so does just a simple cloning of the object. Existing functions will be re-bound to the new object.

Syntax:

copy <object>

Object access paths

In order to access object fields, the common path syntax is used (words separated by a slash character). Each word (or expression) in a path is evaluated in the context given by the left side of the path. Evaluation of a word referring to a function will result in invoking the function, with its optional refinements.

Example:

book: object [
    title: author: none
    show: does [print [mold title "was written by" author]]
]

book/title: "The Time Machine"
book/author: "H.G.Wells"
print book/title
book/show

will output:

The Time Machine
"The Time Machine" was written by H.G.Wells

SELF reference

A special keyword named self has been reserved when self-referencing the object is required.

Example:

book: object [
    title: author: none
    list-fields: does [words-of self]
]
book/list-fields

will output:

[title author list-fields]

Object inheritance

While cloning produces exact replicas of the prototype object, it is also possible to extend it in the process, using make action.

Syntax:

make <prototype> <spec>

<prototype> : object that will be cloned and extended
<spec>      : specification block

Example:

a: object [value: 123]

c: make a [
    increment: does [value: value + 1]
]

print c/increment
print c/increment

will output:

124
125

It is also possible to use another object as <spec> argument. In such case, both objects are merged to form a new one. The second object takes priority in case both objects share same field names.

Example:

a: object [
    value: 123
    show: does [print value]
]
b: object [value: 99]

c: make a b
c/show

will output:

99

Detecting changes in objects

Sometimes, it can be very useful to detect changes in an object. Red allows you to achieve that by defining a function in the object that will be called just after a word is set. This event is generated only when words are set using a path access (so inside the object, you can set words safely). This is just a first incursion in the realm of metaobject protocols, we will extend that support in the future.

In order to catch the changes, you just need to implement the following function in your object:

on-change*: func [word [word!] old new][...]

word : field name that was just affected by a change
old  : value referred by the word just before the change
new  : new value referred by the word

It is allowed to overwrite the word just changed if required. You can directly set the field name or use set:

set word <value>

Example:

book: object [
    title: author: year: none

    on-change*: func [word old new /local msg][
        if all [
            word = 'year
            msg: case [
                new >  2014 ["space-time anomaly detected!"]
                new < -3000 ["papyrus scrolls not allowed!"]
             ]
        ][
           print ["Error:" msg]
        ]
    ]
]

book/title: "Moby-Dick"
book/year: -4000

will output:

Error: papyrus scrolls not allowed!

Extended actions and natives for objects

You can use set on an object to set all fields at the same time. get on an object will return a block of all the fields values. get can also be used on a get-path!.

Example:

obj: object [a: 123 b: "hello"]
probe get obj
set obj none
'' obj
set obj [hello 0]
'' obj
probe :obj/a

will output:

[123 "hello"]
obj: make object! [
    a: none
    b: none
]
obj: make object! [
    a: 'hello
    b: 0
]
hello

Find action gives you a simple way to check for a field name in an object. If found it will return true, else none.

Select action does the same check as find, but returns the field value for matched word.

obj: object [a: 123]
probe find obj 'a
probe select obj 'a
probe find obj 'hello

will output:

true
123
none

The in native will allow you to bind a word to a target context:

a: 0
obj: object [a: 123]
probe a
probe get in obj 'a

will output:

0
123

Bind native is also available, but not completly finished nor tested.

Reflectors

Some reflective functions are provided to more easily access objects internal structure.

  • words-of returns a block of field names.
  • values-of returns a block of field values.
  • body-of returns the object’s content in a block form.

Example:

a: object [a: 123 b: "hello"]
probe words-of a
probe values-of a
probe body-of a

will output:

[a b]
[123 "hello"]
[a: 123 b: "hello"]

SYSTEM object

The system object is a special object used to hold many values required by the runtime library. You can explore it using the new extended help function, that now accepts object paths.

red>> help system
`system` is an object! of value:
    version          string!   0.5.0
    build            string!   21-Dec-2014/19:27:05+8:00
    words            function! Return a block of global words available
    platform         function! Return a word identifying the operating system
    catalog          object!   [datatypes actions natives errors]
    state            object!   [interpreted' last-error]
    modules          block!    []
    codecs           object!   []
    schemes          object!   []
    ports            object!   []
    locale           object!   [language language* locale locale* months da...
    options          object!   [boot home path script args do-arg debug sec...
    script           object!   [title header parent path args]
    standard         object!   [header]
    view             object!   [screen event-port metrics]
    lexer            object!   [make-number make-float make-hexa make-char ...

Note: not all system fields are yet defined or used.

Future evolutions

As this release already took a lot of time, some of the planned features are postponed to future releases. Here are a few of them.

Sometimes, it is convenient to be able to add fields to an object in-place, without having to recreate it, losing lexical binding information in the process. To achieve that, a new extend native will be added, working like originaly intended in Rebol3.

In order to help the Red compiler produce shorter and faster code, a new #alias compilation directive will be introduced. This directive will allow users to turn an object definition into a “virtual” type that can be used in type spec blocks. For example:

#alias book!: object [
    title: author: year: none
    banner: does [form reduce [author "wrote" title "in" year]]
]

display: func [b [book!]][
    print b/banner
]

This addition would not only permit finer-grained type checking for arguments, but also help the user better document their code.

Another possible change will be in the output mold produces for an object. Currently such output will start with “make object!”, this might be changed to just “object”, in order to be shorter and easier to read in addition to be more consistent to the way function! values are molded.

Fixed issues

In order to make this release happen as quickly as possible, we have not fixed all the open tickets that were planned to be fixed in this release, but we still managed to fix a few of them. The other pending tickets will be fixed in the upcoming minor releases.

I should also mention that 537 new tests were added to cover objects features. The coverage is already good, but we probably need more of them to cover edge cases.

That’s all for this blog article! :-)

I will publish another blog entry about additional information regarding the implementation strategy used by the compiler for supporting contexts and object paths.

As we have almost completed other significant features during the last months, you should expect new minor releases happening very quickly in the next weeks. They will include:

  • New cross-platform console engine written entirely in Red (no dependencies).
  • New Android toolchain for creating APK files 100% ported to Red (no dependencies).
  • Full error and exceptions support at Red level.
  • Redbin initial implementation (not started yet).

Also, the work for 0.6.0 has started already (GUI support), even if its at prototype stage right now. I plan to release a first minimal version in the next few weeks (we will extend it step by step until 1.0).

Hope the waiting for the new release was worth it. ;-)

0.4.1: Introducing Parse

One of the greatest feature of the Rebol language has always been its parsing engine, simply called Parse. It is an amazing piece of design from Carl Sassenrath, that spared all Rebol users for the last 15 years, from the pain of having to use the famously unmaintainable regexps. Now, Parse is also available to Red users, in an enhanced version!

So, in short, what is Parse’ It is an embedded DSL (we call them “dialects” in the Rebol world) for parsing input series using grammar rules. The Parse dialect is an enhanced member of the TDPL family. Parse’s common usages are for checking, validating, extracting, modifying input data or even implementing embedded and external DSLs.

The parse function call syntax is straightforward:

parse <input> <rules>  

<input>: any series value (string, file, block, path, ...)
<rules>: a block! value with valid Parse dialect content

Here are a few examples, even if you don’t know Red and Parse dialect, you can still “get” most of them, unlike regexps. You can copy/paste them directly into the Red console.

Some simple examples of string or block input validation using grammar rules:

parse "a plane" [["a" | "the"] space "plane"]
parse "the car" [["a" | "the"] space ["plane" | "car"]]

parse "123" ["1" "2" ["4" | "3"]]
parse "abbccc" ["a" 2 "b" 3 "c"]
parse "aaabbb" [copy letters some "a" (n: length' letters) n "b"]

parse [a] ['b | 'a | 'c]
parse [hello nice world] [3 word!]
parse [a a a b b b] [copy words some 'a (n: length' words) n 'b]

How to parse an IPv4 address accurately:

four:     charset "01234"
half:     charset "012345"
non-zero: charset "123456789"
digit:    union non-zero charset "0"

byte: [
      "25" half
    | "2" four digit
    | "1" digit digit
    | non-zero digit
    | digit
]
ipv4: [byte dot byte dot byte dot byte]

parse "192.168.10.1" ipv4
parse "127.0.0.1"    ipv4
parse "99.1234"      ipv4
parse "10.12.260.1"  ipv4

data: {
    ID: 121.34
    Version: 1.2.3-5.6
    Your IP address is: 85.94.114.88.
    NOTE: Your IP Address could be different tomorrow.
}
parse data [some [copy value ipv4 | skip]]
probe value                      ; will ouput: "85.94.114.88"

A crude, but practical email address validator:

digit:   charset "0123456789"
letters: charset [#"a" - #"z" #"A" - #"Z"]
special: charset "-"
chars:   union union letters special digit
word:    [some chars]
host:    [word]
domain:  [word some [dot word]]
email:   [host "@" domain]

parse "john@doe.com" email
parse "n00b@lost.island.org" email
parse "h4x0r-l33t@domain.net" email

Validating math expressions in string form (from Rebol/Core manual):

expr:    [term ["+" | "-"] expr | term]
term:    [factor ["*" | "/"] term | factor]
factor:  [primary "**" factor | primary]
primary: [some digit | "(" expr ")"]
digit:   charset "0123456789"

parse "1+2*(3-2)/4" expr        ; will return true
parse "1-(3/)+2" expr           ; will return false

Creating a simple parser for a subset of HTML:

html: {
    <html>
        <head><title>Test</title></head>
        <body><div><u>Hello</u> <b>World</b></div></body>
    </html>
}

ws: charset reduce [space tab cr lf]

parse html tags: [
    collect [any [
        ws
        | "</" thru ">" break
        | "<" copy name to ">" skip keep (load name) opt tags
        | keep to "<"
    ]]
]

; will produce the following tree of blocks as output of parse:
 [
     html [
         head [
             title ["Test"]
         ]
         body [
             div [
                 u ["Hello"]
                 b ["World"]
             ]
         ]
     ]
 ]

The Parse dialect

Parse’s core principles are:

  • Advance input series by matching grammar rules until top-level rule failure (returning false) or input exhaustion (returning true). (*)
  • Ordered choices (e.g. in [“a” | “ab”] rule, the second one will never succeed).
  • Rules composability (unlimited).
  • Limited backtracking: only input and rules positions are backtracked, other changes remain.
  • Two modes: string-parsing (for example: external DSL) or block-parsing (for example: embedded DSL).

(*) If collect keyword is used in any rule in its simplest form, a block will be returned by parse no matter if the root rule succeeded or not.

The Parse rules can be made from:

  • keyword : a dialect reserved word (see the tables below).
  • word : word will be evaluated and its value used as a rule.
  • word: : set the word to the current input position.
  • :word : resume input at the position referenced by the word.
  • integer value : specify an iterated rule with a fixed number or a range of iterations.
  • value : match the input to a value
  • | : backtrack and match next alternate rule
  • [rules] : a block of sub-rules
  • (expression) : escape the Parse dialect, evaluate a Red expression and return to the Parse dialect.

The following keywords are currently available in Red’s Parse implementation. They can be composed together freely.

Matching

ahead rule look-ahead rule, match the rule, but do not advance input.
endreturn success if current input position is at end.
nonealways return success (catch-all rule).
not ruleinvert the result of the sub-rule.
opt rulelook-ahead rule, optionally match the rule.
quote valuematch next value literally (for dialect escaping needs).
skipadvance the input by one element (a character or a value).
thru ruleadvance input until rule matches, input is set past the match.
to ruleadvance input until rule matches, input is set at head of the match.

Control flow

breakbreak out of a matching loop, returning success.
if (expr)evaluate the Red expression, if false or none, fail and backtrack.
into ruleswitch input to matched series (string or block) and parse it with rule.
failforce current rule to fail and backtrack.
thenregardless of failure or success of what follows, skip the next alternate rule.
rejectbreak out of a matching loop, returning failure.

Iteration

any rulerepeat rule zero or more times until failure or if input does not advance.
some rulerepeat rule one or more times until failure or if input does not advance.
while rulerepeat rule zero or more times until failure regardless of input advancing.

Extraction

collect [rule]return a block of collected values from the matched rule.
collect set word [rule]collect values from the matched rule, in a block and set the word to it.
collect into word [rule]collect values from the matched rule and insert them in the block referred by word.
copy word ruleset the word to a copy of the matched input.
keep ruleappend a copy of the matched input to the collecting block.
keep (expr)append the last value from the Red expression to the collecting block.
set word ruleset the word to the first value of the matched input.

Modification

insert only valueinsert[/only] a value at the current input position and advance input after the value.
remove ruleremove the matched input.

The core principles mention two modes for parsing. This is necessary in Red (as in Rebol) because of the two basic series datatypes we have: string! and block!. The string! datatype is currently an array of Unicode codepoints (Red will support an array of characters in a future version) while the block! datatype is an array of arbitrary Red values (including other blocks).

In practice, this results in some minor differences in Parse dialect usage. For example, it is possible to define arbitrary sets of characters using the new bitset! datatype, which are useful only for string! parsing in order to match with a large number of characters in one time. Here is an example using only bitsets matching and iterators:

letter: charset [#"a" - #"z"]
digit:  charset "0123456789"

parse "hello 123 world" [5 letter space 3 digit space some letter]

The Bitset! datatype

A bitset value is an array of bits that is used to store boolean values. In the context of Parse, bitsets are very useful to represent arbitrary sets of characters across the whole Unicode range, that can be matched against an input character, in a single operation. As bitset! is introduced in this 0.4.1 release, it is worth having an overview of the features supported. Basically, it is on par with Rebol3 bitset! implementation.

In order to create a bitset, you need to provide one or several characters as base specification. They can be provided in different forms: codepoint integer values, char! values, string! values, a range or a group of previous elements. The creation of a new bitset is done using the following syntax:

make bitset! <spec>

<spec>: char!, integer!, string! or block!

For example:

; create an empty bitset with places at least for 50 bits
make bitset! 50

; create a bitset with bit 65 set
make bitset! #"A"

; create a bitset with bits 104 and 105 set
make bitset! "hi"

; create and set bits using different values representations
make bitset! [120 "hello" #"A"]

; create a bitset using ranges of values
make bitset! [#"0" - #"9" #"a" - #"z"]

Ranges are defined using two values (char! or integer! allowed) separate by a dash word.

Bitsets are auto-sized to fit the specification value(s) provided. The size is rounded to the upper byte bound.

A shortcut charset function is also provided for practical usage, so you can write:

charset "ABCDEF"
charset [120 "hello" #"A"]
charset [#"0" - #"9" #"a" - #"z"]

For reading and writing single bits, the path notation is the simplest way to go:

bs: charset [#"a" - #"z"]
bs/97     ; will return true
bs/40     ; will return false
bs/97: false
bs/97     ; will return false

(Note: bitset indexing is zero-based.)

Additionally, the following actions are supported by bitset! datatype:
pick, poke, find, insert, append, copy, remove, clear, length', mold, form

See the Rebol3 bitset documentation for more info about usage of these actions.

In order to cope with the wide range of Unicode characters, bits outside of the bitsets are treated as virtual bits, so they can be tested and set without errors, the bitset size will auto-expand according to the needs. But that is still not enough to deal with big ranges, like for example a bitset for all Unicode characters except digits. For such cases, it is possible to define a complemented bitset that represents the complement range of the specified bits. This makes possible to have large bitsets while using only a tiny memory portion.

Complemented bitsets are created the same way as normal bitsets, but they need to start with the not word and always use a block! for their specification:

; all characters except digits
charset [not "0123456789"]

; all characters but hexadecimal symbols
charset [not "ABCDEF" #"0" - #"9"]

; all characters except whitespaces
charset reduce ['not space tab cr lf]

Set operations are also possible, but only union is currently implemented in Red (it is the most used anyway for bitsets). With union, you can merge two bitsets together to form a new one, which is very useful in practice:

digit: charset "0123456789"
lower: charset [#"a" - #"z"]
upper: charset [#"A" - #"Z"]

letters:  union lower upper
hexa:     union upper digit
alphanum: union letters digit

Parse implementation

Parse dialect has been implemented as a FSM which differs from the Rebol3 implementation that relies on recursive function calls. The FSM approach makes possible several interesting features, like the ability to stop the parsing and resume it later, or even serialize the parsing state, send it remotely and reload it to continue the parsing. Such features could now be implemented with minimal efforts.

Red Parse implementation is about 1300 lines of Red/System code, with a significant part of it spent on optimized iteration loops for common cases. About 770 unit tests have been hand-written to cover the basic Parse features.

The current Parse runs as an interpreter, which is fast enough for most use-cases you will have. For cases where maximum performance is required, work has started on a Parse static compiler to soon provide the fastest possible speed to Parse-intensive Red apps. The generated code is pure Red/System code and should be about a magnitude faster on average than the interpreted version. When Red will be self-hosted, a Parse JIT compiler will be provided to deal with the cases that the static compiler cannot process.

As Red gets more features, Parse will continue to be improved to take advantage of them. Among other future improvements, binary! parsing will be added as soon as binary! datatype is available, and stream parsing will be possible when port! datatype will be there.

The Red Parse also exposes a public event-oriented API in form of an optional callback function that can be passed to parse using the /trace refinement.

parse/trace <input> <rules> <callback>

<callback> specification:

func [
    event   [word!]   ; Trace events
    match'  [logic!]  ; Result of last matching operation
    rule    [block!]  ; Current rule at current position
    input   [series!] ; Input series at next position to match
    stack   [block!]  ; Internal parse rules stack
    return: [logic!]  ; TRUE: continue parsing, FALSE: exit
]

Events list:
- push    : once a rule or block has been pushed on stack
- pop     : before a rule or block is popped from stack
- fetch   : before a new rule is fetched
- match   : after a value matching occurs
- iterate : after a new iteration pass begins (ANY, SOME, ...)
- paren   : after a paren expression was evaluated
- end     : after end of input has been reached

This API will be extended in the future to get more fine-grained events. This API could be used to provide Parse with tracing, stats, debugging, … Let’s see what Red users will come up with! ;-)

A default callback has been implemented for tracing purposes. It can be accessed using the handy parse-trace function wrapper:

parse-trace <input> <rules>

You can try it with simple parsing rules to see the resulting output.

What about DSL support’

Parse is a powerful tool for implementing DSL parsers (both embedded and externals), thanks to its ability to inline Red expressions directly into the rules, allowing to easily link the DSL syntax with its corresponding semantics. To illustrate that, here is a simple interpreter for a famous obfuscated language, written using Parse:

bf: function [prog [string!]][
    size: 30000
    cells: make string! size
    append/dup cells null size

    parse prog [
        some [
              ">" (cells: next cells)
            | "<" (cells: back cells)
            | "+" (cells/1: cells/1 + 1)
            | "-" (cells/1: cells/1 - 1)
            | "." (prin cells/1)
            | "," (cells/1: first input "")
            | "[" [if (cells/1 = null) thru "]" | none]
            | "]" [
               pos: if (cells/1 <> null)
               (pos: find/reverse pos #"[") :pos
               | none
              ]
            | skip
        ]
    ]
]

; This code will print a Hello World! message
bf {
    ++++++++++[>+++++++>++++++++++>+++>+<<<<-]>++.>+.+++++++..+++.
    >++.<<+++++++++++++++.>.+++.------.--------.>+.>.
}

; This one will print a famous quote
bf {
    ++++++++[>+>++>+++>++++>+++++>++++++>+++++++>++++++++>
    +++++++++>++++++++++>+++++++++++>++++++++++++>++++++++
    +++++>++++++++++++++>+++++++++++++++>++++++++++++++++<
    <<<<<<<<<<<<<<<-]>>>>>>>>>>>----.++++<<<<<<<<<<<>>>>>>>
    >>>>>>.<<<<<<<<<<<<<>>>>>>>>>>>>>---.+++<<<<<<<<<<<<<>>
    >>>>>>>>>>>>++.--<<<<<<<<<<<<<<>>>>>>>>>>>>>---.+++<<<<
    <<<<<<<<<>>>>.<<<<>>>>>>>>>>>>>+.-<<<<<<<<<<<<<>>>>>>>>
    >>>>>>+++.---<<<<<<<<<<<<<<>>>>.<<<<>>>>>>>>>>>>>>--.++
    <<<<<<<<<<<<<<>>>>>>>>>>>>>>-.+<<<<<<<<<<<<<<>>>>.<<<<>
    >>>>>>>>>>>>>+++.---<<<<<<<<<<<<<<>>>>>>>>>>>>>>.<<<<<<
    <<<<<<<<>>>>>>>>>>>>>>-.+<<<<<<<<<<<<<<>>>>>>>>>>>>>>-.
    +<<<<<<<<<<<<<<>>>>>>>>>>>>>>--.++<<<<<<<<<<<<<<>>>>+.-
    <<<<.
}</-></->

Note: this implementation is limited to one-level of “[…]” nesting to keep it as short as possible. A complete, but a bit more longer and complex implementation using Parse only, is availaible here.

So, such approach works incredibly well for small DSLs. For more sophisticated ones, Parse still works fine, but it might be less helpful as the DSL semantics get more complex. Implementing an interpreter or compiler for a more advanced DSL is not a trivial task for most users. Red will address that in the future by providing a meta-DSL wrapper on top of Parse, exposing a higher-level API to build more sophisticated DSL by abstracting away core parts of an interpreter or compiler. A DSL for building other DSLs is not a crazy idea, it already exists in a very powerful form as the Rascal language for example. What Red will provide, will be just one step in that direction, but nothing as sophisticated (and complex) as Rascal.

Other changes in this release

Just to mention other changes in this release, now that we got rid of the 800 pound gorilla in the room. This release brings a significant number of bug fixes, both for Red and Red/System. Also, thanks to qtxie, the ELF file emitter is now on par with the other ones when producing shared libraries.

Thanks to all the people involved in helping for this big release, including design fixes and improvements, testing, bug reporting and test writing.

Enjoy! :-)

0.4.0: Red goes binary!

What’s that’!

As we are getting closer to the end of the alpha period, we are now moving to a more convenient way to use and distribute the Red toolchain. So far, you needed to download a Rebol interpreter and the sources from Github separately, and run it using, a bit verbose, command-lines. This is fine for developping Red with contributors that are interested in the inner workings of the toolchain, but for the end users, the goal has always been to provide a simpler and much more convenient way, like Rebol teached us in the past.

So, from now, you can get Red as a single binary (< 1 MB) from the new Download page. Yes, all of Red toolchain and runtime is packed in that small binary, even the REPL is built-in!

The Red repo landing page has been reworked to show Red usage in binary form, all previous command-line options are present, a new one (-c) has been introduced. Here is a short overview of the main options:

Launch the REPL:

$ red

Run a Red script directly (interpreted):

$ red <script>

Compile a script as executable with default name in working path:

$ red -c <script>

Compile a script as shared library with default name in working path:

$ red -dlib <script>

Compile a script as executable with alternative name:

$ red -o <new> <script>

Compile a script as executable with default name in other folder:

$ red -o <path/> <script>

Compile a script as executable with new name in other folder:

$ red  -o <path/new> <script>

Cross-compile to another platform:

$ red -t Windows <script>

Display a description of all possible options:

$ red -h

Notice that -c option is implied when -o or -t are used. It is designed to keep command-lines as simple and short as possible.

Moreover, for standalone Red/System programs, the Red binary will be able to compile them directly, no special option needed, they will be recognized automatically.

Thanks very much to Tamás Herman for helping with setting up the build farm and providing the Mac OSX machine, and thanks to the HackJam hackerspace group from Hong-Kong for the hosting!

Other changes

  • In addition to that new binary form, 17 issues have been fixed since the 0.3.3 release about a month ago (not counting regression tickets).
  • The work on objects support is half-done, objects are working fine with advanced semantics on the interpreter (see object branch), now the focus will be to support them at the Red compiler level.

What’s next’

As we are moving closer to the beta state, version numbers will increase faster, e.g., once objects will be done, the release will be the 0.5.0, while 0.6.0 will bring full I/O support. Between these major versions, smaller versions should be still possible, this means that the release cycle should accelerate with at least one release each month from now on. So, what you should expect in the next ones’

0.4.x

  • Simple I/O support: (just read, write and exists’ on files)
  • PARSE support
  • Pre-compiled runtime (much faster compilation times)

0.5.0

  • Full object support

0.5.x

  • VID-like cross-platform dialect binding to native widgets.
  • Mezzanine functions additions
  • Redbin (accurate Red values serialization in binary format)
  • Full errors management
  • Red-level exceptions handling

Enjoy!

0.3.2: REPL release

Time has finally come to release the new REPL for Red. A preview version has been pushed last Christmas, since then a lot of work has been done on several fronts (about 310 new commits, excluding merges): interpreter (DO), runtime lexer (LOAD) and console. But before going into details for the REPL, here is an overview of the changes in this new release:

  • New datatypes: routine!, issue!, file!
  • New natives: do, load, reduce, compose, switch, case, stats, platform’, quit
  • New natives: remove
  • New mezzanines functions: '', empty'
  • 38 issues from previous releases fixed.
  • Unit tests for Red raised to 5602 for a total of 17671 tests (including Red/System ones).
  • Notable improvements:
    • Paren! expressions in paths now supported.
    • Mold output for string! and char! deeply improved, special characters are now correctly escaped.
    • Improved and more accurate support for unset! values processing.
    • Prin and print now reduce their arguments.

Red REPL

This is the biggest part of this new release. The REPL has several components:

  • The interpreter: it is a full Red interpreter supporting all Red language features, except Red/System code. In the current version, though, exit and return are not yet implemented, they need some special low-level support from Red/System, so couldn’t make it for this release. The interpreter can be invoked from compiled code using the do native. It has been developped in Red/System and is about 700 LOC long. All Red compiler tests are run against the interpreter too, all are passing except for the unimplemented yet exit and return (6 tests).
  • The runtime lexer: it is the runtime counterpart to the compiler’s lexer and, is in charge of loading input string source code into Red and converting it into blocks of values. It can be invoked from compiled code using the load native. The runtime lexer current version only support Latin-1 encoding, a full Unicode lexer will replace it soon (it is a work in progress).
  • The console: it is the visible part of the REPL for most users. The current version is minimal but works on most of supported platforms (including the RaspberryPi). It has limited editing abilities, and history doesn’t work on Mac OS X, but it supports a Rebol-like multi-line input mode for blocks and strings. We will provide a much better console in the next release, with a cross-platform abstraction layer that feature-wise, will put all platforms on par.

The interpreter and runtime lexer and now part of Red’s standard library, so they are bundled with every compiled Red script. The overhead is about 30KB in the final binary, making it almost unnoticeable. The console is a separate script, that can be compiled easily producing a small standalone binary.

An important feature of the Red interpreter is that it is not only meant for the REPL support, but is actually used by the compiler to solve some code patterns that are too dynamic to be statically analyzed and compiled. Moreover, the interpreter has the ability to call pre-compiled code, so as soon as possible, it can drop back to native code execution. Both compiled and interpreted code are deeply collaborating to provide the most flexible language semantics while ensuring the maximum performances. With the future addition of a JIT-compiler, we will be able to reach the optimal internal architecture.

Image
Red Collaborative Execution Model

On the more practical side, to compile the console, from Red root folder, just do:

do/args %red.r "red/tests/console.red"

That will give you a console binary in the working folder. When you run it, you should see:

-=== Red Console alpha version ===- 
(only Latin-1 input supported)  

red>>

This is the Red prompt where you can input any valid Red expression, they will be evaluated on Enter key pressed. Use q or quit to exit the console. Remember that there is no yet proper error management, current error messages are hardcoded, and in some cases, you will be thrown out of the console (that is why it is still an alpha version as stated by the console banner). Note that it is possible to copy/paste Red scripts directly into the console.

Anyway, feel free to experiment.

Routines

In order to more easily interface Red and Red/System, a new function datatype has been added: routine!. A routine is Red/System function defined in a Red program. The routine specification takes Red datatypes as arguments and return value, and the routine will automatically convert them to appropriate Red/System types when called. For example:

increment: routine [
    n       [integer!]
    return: [integer!]
][
    n + 1
]

Here you can see how the Red integer! argument get marshalled forth and back to Red/System integer! datatype for you. For now, routines automatically converts integer! and logic! datatypes this way. Other Red datatypes are passed as their Red/System counterparts as defined in Red’s runtime (see %red/runtime/datatypes/structures.reds). Optional arguments are not supported by routines, as such feature does not exist in Red/System for now.

You can now very easily extend Red with routines using the full power of Red/System! Compiled routines can be run from interpreter too.

Project management

We are now moving to Trello for tracking the tasks about Red development. The short-term ToDo list is pretty accurate and updated regularly, you can now follow us more closely. We only started using it a month ago, so not all the tasks (especially the mid-term/long-term ones) are inserted there yet.

Thanks for all the support received for getting this major release out!
Cheers!

Red v0.3.1: functions support added

This new milestone brings (finally) functions to Red along with other new features and fixes. Let’s first have a quick overview of 0.3.1 release content:

  • function support
  • path get/set notation support
  • refinement support for native functions
  • expressions in parentheses compilation
  • new datatypes: function!, paren!, path!, lit-path!, get-path!, set-path!
  • new actions and natives: find, select, copy, reflect, type’, halt,…
  • extended mold, form and comparison operators to all new and existing datatypes
  • many new mezzanines functions
  • modulo and remainder operators on floats implemented for ARM backend
  • Quick-Test testing framework ported to Red
  • a truckload of new unit tests
  • many bugfixes
  • about 200 new commits to the Github repository
  • updated Red/System formal description document

Functions

They are declared using a similar syntax to REBOL, with some extensions. The specification block looks like this:

[arg1 [type1] arg2 [type2]...return: [type] /local var1 [type-v1]...]
  • Arguments and local variables are declared in the same way as in REBOL
  • A return value type can be optionally specified
  • Local words can have a type specifier

All type specifiers are optional, adding them will not only allow the compiler to make additional type checkings but also generate faster code in some cases (once the optimizations added to the Red compiler).

Note: argument and return value type checking have not been implemented yet, they need typeset! and error! datatypes to be implemented first.

Functions can be built using several constructors:

  • function: automatically collects local variables, like funct in REBOL.
twice: function [a [integer!] /one return: [integer!]][
    c: 2
    a: a * c
    either one [a + 1][a]
]
  • func: low-level function constructor, everything needs to be manually specified.
twice: func [a [integer!] /one return: [integer!] /local c][
   c: 2
   a: a * c
   either one [a + 1][a]
]
  • has: no argument, just lists local words.
globals: [1 2 3 _]

foobar: has [list][
    if list: find globals '_ [clear list]
]
  • does: no argument, no local words.
quit: does [
    print "Goodbye cruel world!"
    halt
]

Early exit points are also there in form of exit and return functions.

foobar: func [a [integer!] b [integer! none!]][
    if none' b [return none]
    if b = 0 [
        print "Error: division by zero"
        exit
    ]
    a / b
]

Refinements are also fully supported, when not used, their value is set to false (while in REBOL it is set to none).

Still some features are not yet implemented:

  • building a function at runtime (requires the ability to JIT-compile source code)
  • passing a function as argument (pending)
  • tail recursion optimization

Currently functions are using a local context on stack, so their life-time is very short. For indefinite extent support, a closure! type will be added in the next months.

Path support

Path datatypes were added (path!, lit-path!, get-path!, set-path!) to the runtime library and compiler, allowing the use of path notation as syntactic sugar for series access. Getting and setting values using path notation are supported.

Some usage examples:

list: [a 5 b [c 8] d #"e" name: "John"]

probe list/2        ;-- outputs 5
probe list/7        ;-- outputs name:
probe list/a        ;-- outputs 5
probe list/4/2      ;-- outputs 8
probe list/b/c      ;-- outputs 8
probe list/name     ;-- outputs "John"

list/b/c: 0
probe list/b/c      ;-- outputs 0

index: 3
probe list/:index   ;-- outputs b
list/:index: 'z
probe list/:index   ;-- outputs z

Note: notice how words lookups in blocks can work with any word datatypes.

Paren expressions

Paren! datatype and expressions in parentheses compilation support has been added too. They are mainly useful with infix operators in order to force a given execution priority. Example:

print (1 + 2) * (3 + 4) ;-- outputs 21

New built-in functions

  • actions: find, select, copy, reflect
  • natives: type', halt, function, func, does, has, exit, return
  • mezzanines: probe, first, second, third, fourth, fifth, last, spec-of, body-of and all <datatype>’ type testing functions.

Existing native or action functions were extended with refinements (like mold and form). See the boot.red script for more details.

Testing

The Quick-Test framework from Peter WA Wood was ported to Red and now powers all the current 1800+ unit tests written in Red itself, mostly added by Peter.

About 32 issues/bugs in both Red and Red/System were fixed in this new release.

Red/System formal description

Last but not least, Rudolf updated his Red/System formal description document [PDF, 162KB] to include the additional features like namespaces. Many thanks to him for his remarkable work and support to Red project!

What’s next’

Objects support is next priority along with new natives like switch and case. Typesets and error handling will probably be added very soon too. Once that part done, the next focus will be to add I/O support!

Many thanks to all the people that have made significant donations in last months to Red project, allowing me to continue to work full time on it! It is really making a big difference, as shown by Red increased progress speed since this summer. Thanks again!

Cheers!

Red alpha release

We have all waited long for this to happen, but it is finally there: first Red alpha release!

This is the biggest milestone since the initial release of Red System, as it marks the entrance of the higher layer of the Red software stack. There’s still another layer pending, the JIT compiler, but that’s for 2013, when Red will be self-hosted.

What Red can do so far’

So what happened since the first Red compiler release a little more than a month ago’ We have added quite a lot of features actually:

  • 15 datatypes: block!, string!, char!, integer!, logic!, none!, word!, lit-word!, get-word!, set-word!, refinement!, action!, native!, op!, unset!.
  • 19 actions: make, form, mold, add, divide, multiply, subtract, append, at, back, clear, head, index', length', next, pick, poke, skip, tail.
  • 21 natives: if, unless, either, any, all, while, until, loop, repeat, foreach, forall, print, prin, equal', not-equal', strict-equal', lesser', greater', lesser-or-equal', greater-or-equal', not.
  • 10 infix operators: +, - , *, /, =, ==, <, >, <=, >=.

String! datatype fully supports Unicode from end to end and in a cross-platform way, according to our plan (see the hello.red demo script).

We also have some unit tests (223 tests right now), but that number should rapidly increase once we will be able to fully port QuickTest framework to Red (in a couple of weeks, once Red will gain functions).

All those features were mostly coded in the last month, some parts are still lacking (e.g. refinements support for actions/natives) but that gives you an idea of the pace at which Red will be further developed.

However, we haven’t started yet writting the documentation for Red language, we should decide soon on the kind of documentation we want (hint: you can make propositions about that on the Red mailing-list or FB page). Red API documentation will most probably be generated automatically from source code by extracting doc-strings.

What’s not yet there’

A lot of things, but they will be added progressively until the end of the year, where we should have a pretty complete Red (bootstrap) implementation. So, the main pieces to implement are:

  • Red runtime lexer (LOAD native)
  • Functions and objects support
  • I/O support (including networking)
  • More complete memory allocator and a garbage collector.
  • Concurrency support
  • Additional datatypes

Can I try it already’

Red codebase is available at github, just clone it locally or download a packaged version. Some simple instructions to setup and run it can be found here. As we are still at the bootstrap stage, the installation process and usage is more complex than it should be, once self-hosted. Anyway, we should be able to deliver some test binaries soon and simplify the usage for those who want to play with the alpha releases.

How does Red work’

Currently Red scripts are statically compiled using the %red.r front-end wrapper. The Red lexer loads the UTF-8 encoded Red scripts in memory in form of blocks of values. Those blocks are then compiled to a Red/System script, that gets itself compiled to native code and outputs an executable binary. The Red/System compiler has been enhanced to be able to compile source code directly from memory.

Some compilation options are already available, the famous -t option from Red System is also present, allowing Red to cross-compile any script as easily as with Red/System.

This is just the beginning, so for now, whole Red runtime library is compiled with user scripts each time. We will add modular compilation support at some point to get the full compilation speed back.

You can test the few example Red scripts from the repository. Here is what the demo.red script outputs on some of the platforms Red can already run on:

Image
Image
Image
Image

The missing characters in some of the screenshots are Chinese ones as I only have font support for them on Ubuntu.

Red was originally conceived to be statically typed. However, during the early stages of Red implementation, it appeared that an hybrid typing system will be more adequate for a language that will support high-level scripting, without the burden of a sophisticated, but slow, type inference engine. The overhead of such inference engine can be very high (cf Scala slow compilation speed ). The hybrid type system works very simply, typing is optional, but if local variables and function returned values are typed by user, the compiler will be able to generate faster code (more than an order of magnitude faster for, e.g., pure math expressions) and catch more programming errors at compile time.

Red compiler does not do any optimization so far, it basically outputs the equivalent of an unrolled interpreter loop. Some optimizations will be added along the development of the bootstrap version of Red, but the full extent of possible optimizations will be left for the self-hosted Red. Firstly because the current compiler code is disposable, so we don’t want to invest too much time in it, secondly because the final JIT compiler will allow even deeper optimizations thanks to reflection and profiling information collected at runtime.

Red/System changes

Red/System has been improved significantly during the Red early development:

  • function pointer support has been extended and improved, dereferencing is now possible.
  • 20 issues have been fixed.
  • some long-standing bugs with GTK+ and other bindings on ARM platforms have been fixed.
Image

So, what’s coming next’

Red will now mature very quickly, as functions and objects will make their entrance in the next days/weeks. A new Red alpha release is planned very soon, probably for mid-november with a lot of new features.

The work on shared library generation for MacOS and Linux platforms is on-going. Once available, we’ll be able to literally “plug” Red in a lot of third-party apps, including other programming languages. Also, this will enable (or make easier) the bridging with some external systems like Java, .Net and objective-c, in order to fully support the main mobile platforms.

As you can see, next months will see some important parts of Red come to life, be sure not to miss them, follow us on our mailing-list, on Twitter, IRC and Facebook.

Cheers!

Red/System v0.2.6 released

The need for more structured code support has arisen with the addition of bigger and more sophisticated bindings like GTK one or the (soon to be released) work on alpha version of Red compiler (which emits Red/System code). So, v0.2.6 introduces namespaces support using a very simple, but efficient model.

For example, to encapsulate some variables and functions in a local context:

b: 1
a: context [
   b: 2
   inc: func [i [integer!] return: [integer!]][
       i + b
   ]
]
a/inc 5                            ;-- will return 7

Local variables take precedence other global ones with same name. This simple rule also applies to nested contexts, the nearest one has priority, e.g.:

a: context [
   b: 123

   c: context [
       #enum colors! [red green blue]
       b: "hello"
       foo: func [][print-line b]
   ]

   print-line b                    ;-- will output 123
   c/foo                           ;-- will output hello
]

print-line a/b                     ;-- will output 123
a/c/foo                            ;-- will output hello
print-line a/c/b/2                 ;-- will output e
print-line a/c/blue                ;-- will output 2

As you can see from this example, enumerations can also be defined locally to a context, but not only. Aliases and imported functions can also be defined locally too! With that powerful new feature, you can now modularize your Red/System programs simply and efficiently.

See the namespaces documentation for more features and info.

In this release you will also find several bugfixes, mainly for floats support. Thanks to Rebolek for his aggressive floats testing. ;-)

Enjoy!

Taking the Red/System pill

Here we go, this is the first release of Red/System dialect. Of course, it’s an early alpha, don’t expect too much from it for now (not crashing would be a success). I know that most of you are eager to see what this looks like, so, you can download right now an archive from github, or better clone the repository from:

https://github.com/red/Red

See the README file for instructions on running the little demo script (tests/hello.reds).

Design & Implementation notes

Red/System is a low-level dialect (or DSL) of Red. Its main purpose is to allow implementing a small runtime for Red, basically covering three domains:

  • Memory management (MAKE primitive)
  • Lexical analyzer (LOAD primitive)
  • Executable container

So, the feature set of Red/System has been chosen with regards to the goals to reach. Extending them beyond the main purpose is not in the current plan, but could become an interesting side-project for anyone interested in implementation of programming languages.

Red/System relies on a static compiler and a linker in order to produce working code. Here’s a short list of currently supported features:

  • types: integer!, string! (partial), struct!, pointer! (partial).
  • literal strings only for now (can’t build or change one at runtime)
  • math ops, boolean ops, comparison ops (all as infix operators)
  • control flow: IF, EITHER, UNTIL, WHILE, ANY, ALL
  • size’ function (== sizeof( ) in C)
  • length’ function (get the size of a string at runtime)
  • user functions (with local variables and return value support)
  • OS functions importing

Main features not yet implemented:

  • function calls and return value type-checking
  • no duplicate variables name checking
  • no reserved words protection
  • int8! and int16! datatypes (maybe int64! too if required)
  • pointer values arithmetic
  • string’s bytes read/write access
  • arrays support
  • proper minimal runtime for Red/System scripts

The compiler is a naïve one (non-optimizing) using a straightforward approach. It works in 2 passes:

  • first pass: lexical analysis producing nested s-expressions (== LOAD)
  • second pass: direct compilation from high-level to machine code

There’s no intermediary representation, nor assembly mnemonics generations, just direct machine code output. I’ve used this compiler as an opportunity to prototype several ideas, mostly seeking the simpliest ways to reach to goals, with the best possible performances. Also, I expect to re-use as much code as possible from this compiler for Red’s compiler itself. That’s the main reason for not choosing C to build the runtime.

So far, the result is quite good, there are a few limited parts that I’m not very happy with, but will probably wait for the re-implementation of Red/System in Red before fixing them:

  • Code generation for control flow function: that’s the most complicated part in the compiler, mainly caused by the need to generate code in chunks with mutual dependencies to resolve (mixed forward branching). I managed to reduce/hide the complexity somehow, but I’m still not satisfied with the result (in terms of code readability).
  • Machine code emitter is too verbose: the emitter (IA32 currently only) is too coarse-grained, this makes it easier to work with, but the CPU-dependent code is much larger than it could be, making the porting effort to other targets, bigger.

In the next days, I’ll work on :

  • adding missing features
  • start writing unit tests and fixing bugs
  • documenting Red/System specifications (on github’s wiki’)

The goal is to be ready as soon as possible for implementing Red’s runtime.

Posts:

Tags: