All of these things could be added into a syntax, I believe, with a carefully constructed extensible parser. Each extension would become a libraries (modules) adding or replacing function points in the existing parser later reusing some of them as part of the extended syntax. (The actual parser would be based on Parsec and other similar combinator parsers. OMeta is also related to this parser architecture.)

The problem with adding new syntaxes to a language isn’t just the parser implementation however. Most of the really tricky problems actually come from the rest of the tool chain. Things like the compiler, debugger, profilers, editor, and a bunch of other potential tools. It had become clear to me that I wasn’t just working on an extensible parser, but rather an extensible language, and that meant an extensible compiler architecture.

Basically I thinking about a completely extensible intermediate language, which would power a set of semi-extensible libraries for implementing the bulk of the tool-chain. These libraries will take the leg-work out of processing the source code, and will allow the tools to manipulate or interact with the code in its intermediate form (this includes editors and analytical tools). The only problem I saw was that I had absolutely no idea how to construct such an intermediate language, little own how to write tools and libraries to interact with it.

Anyway, recently I run into the latest (at the time) Steve Yegge post, with a somewhat inspiring discussion about the Prototype Pattern (The Universal Design Pattern). After musing about pattern (again) for a couple of days, it finally hit me, I had the architecture all wrong. This isn’t just some extensible compiler or parser…this was the Self programming language…redesigned, implemented and tailored for the task of compiler writing. The AST is simply a specialized variation of the Prototype pattern, or more succinctly, a compiler is just a constrained dialect of Self, designed for compiler tasks.

So this my proposal is this, use the prototype pattern for Mentions AST and build a framework of libraries and tools designed to aid in the maintenance and development of an AST and the compilers, parsers, debugger, et cetera, which depend on it.

There are already a few pitfalls that I can imagine coming out of this system. For one, a good namespace or typing system will probably be required to prevent name conflicts. That is, each extension or collection of extensions would add only private collections of attributes, while a smaller group of shared public attribute interfaces would be used to facilitate cooperation between extensions. And then there’s still the issue of expected behavior, which may be complicated by the potential for different combinations of compiler extensions to inadvertently interact in unexpected ways.

In any case, I believe that this will be one of the key features of the core Mention programming language and it may even make some of the more complicated features like type systems easier to implement and experiment with later on. At the very least, you can expect to see some follow up material on this when I start to get further into the implementation of the Mention.

– Lorenz

One of the requirements I keep stumbling across is the need for referentially transparent functions that are capable of interacting directly with mutable data structures. Yeah, the kind that, by definition, are not referentially transparent.

At first I considered using compiler attributes to mark a piece of code that interacts with side effects vs. code that doesn’t. Something like the IO/ST monads from Haskell but general enough any function can interact with mutable variables, not just ones returning a monadic value. Unfortunately any code that interacts with mutable variables may behave radically differently when not constrained by the full monadic infrastructure or some equivalent like strictness (Scheme and ML).

I believe the only useful solution is to define a new property of the code–read transparency–which allows us to recognize code that is referentially transparent on immutable data while also referentially transparent towards unchanging mutable variables.

The above definition depends on the equivalence property of values, which I define along the lines of “any two data structures are equivalent if they are functionally the same value, or they only contain mutable variables containing equivalent state.” Given a good equivalence for all primitives this allows us to easily check for all functions which could safely be called in our new semi-functional programs.

The original motivating example for this concept was to allow functions like show in Haskell to work with any arbitrary data structure, mutable or not. This would greatly simplify programs like interpreters, which could use many of the same functions as pure code, without having to use somewhat cumbersome combinators like the lifting functions and unboxing functions (readIORef, etc.).

An important thing to remember about this property is that, while many previously referentially transparent functions would now only be read transparent, when used only with pure functional data, they are in fact promoted automatically to fully referentially transparent functions. That is, they remain effectively referentially transparent if you don’t care about mutable state, or to put it another way–the functions become more flexible, not less so. Only when the implementation of a function relies on language features that prevent mutable data from being used would the more restrictive referential transparency be required (like lazily evaluated results, incorporating the read operation).

We don’t actually have to stop there either. Just because a piece of code isn’t read transparent, that doesn’t mean that it suffers from all variations of side-effects. There are still yet more shades of grey which we can use to limit the damage caused by bugs and side-effects in our software. A property like write transparency for code that reads and writes to mutable data structures but performs no IO could be equally useful, just as the ST monad in Haskell, which cannot perform IO is useful for working with mutable state.

Unfortunately while our little definition is a rather interesting take on the mutable state issue, there are some rather serious caveats we still have to consider. The biggest problem we face here is that, while a read transparent function can read from mutable variables without any reprimands, it cannot use the unread mutable variable in the result, unless it uses it, still boxed. That is, if we returned a result lazily, the result might be different depending on when the result was used, definitely not something we want, and the primary reason that mutable state in Haskell is limited to the ST monad et cetera.

More concretely, we can return new mutable variables (since the result would always be equivalent), use the results from reading a mutable variable so long as we read it strictly before the function returns, and finally we can return the mutable variable verbatim as part of the result (something Haskell can do also). If any language implemented mutable variables unboxed however, they would not be usable in last context however, and you would always have to read from them strictly, leading to somewhat less flexibility in the language.

One last thing I’d like to add. The above definition is strikingly similar to something that might be rigorously defined and explored, but for some reason I haven’t done so in this post. Well, for all intents and purposes, I wouldn’t know where to start, at least not yet anyway. If and when I apply this property in an actual language implementation however, you can be sure I’ll have a rigorous writeup of it and it will likely be part of the languages spec too.

– Lorenz

Over the years I’ve spend quite a bit of time learning the gamedev concepts and recently I was able to apply some of them in a project, intended to test Haskell’s gamedev capabilities for hobbyist or small team game development (related post). Specifically I was interested to see how the difference between Haskell and Object-Oriented languages changed the design and implementation of the game code and engine, hopefully lending itself to the eventual development of languages tailored towards game development.

I set out to develop one of my favorite arcade remakes–yet another variation of Geometry Wars. The premise would be to keep the game as simple as possible, only the basic gameplay would be present with a couple of levels of progression. Enough to test all of the simple elements of gameplay ready for a more substantial project. As you may have guessed from the absence of the obligatory screenshot it didn’t work out as well as I expected.

Luckily I did pick up a couple of things while working on the game, and better yet I may even be able to rewrite the code and find that screenshot yet. It all comes down to a couple of quirks in Haskell’s syntax which didn’t blend well with my usual game architecture. Obviously the architecture would need to be modified to suite the idiomatic style of Haskell, but I was reasonably certain (and still are) that the architecture is the simplest implementation for both Object-Oriented languages as well as functional languages, based almost solely on the requirements of the high level game data structures.

The Haskell code for these structures was essentially based directly on records. The records originally contained only functional values, but I quickly realized how crazy that was, which lead me to the current design which uses records full of IORefs, at least for the mutable slots. (If its not immediately obvious why the IORefs are needed, it might help to know that game objects quite often need to reference each other simply because the operations are related between the two…bullets from the player ship for example.)

Finally, after working with this approach for a while it eventually sunk in that I could probably write the same code in Python in not only half the time, but also quicker for the same number of overall bugs. Not because the bugs are easier to catch in Python…they really, really aren’t. But because Python allows me to test many more variations of the game. At least more variations than Haskell seems to, all of which makes me wonder if I haven’t quite tapped into the idiomatic code still.

So I’m not quite out of ideas yet. I believe I can tweak my accessors to reduce the namespace damage caused by them (at the expense of some verbosity, however…why the hell hasn’t this been fixed yet anyway). And there’s even some clever usage of typeclasses which could simplify the use of common algorithms between game objects. If all goes to plan I’ll have some code to post next time…

– Lorenz

The other change is not particularly obvious, but basically the site is now under the Git DVCS. This will make it significantly simpler to manage any updates of the site while concurrently maintaining the code I write for the plugins and visual style. Basically the site is maintained through a collections of branches which are rebased into the deployed branch which is finally published to the server. The files on the server are then reset to the latest version of the deployed branch. The side affect of this being, I can update any single piece using its corresponding feature branch and simply rebase the dependent branches to the last patch on that branch, while any changes after the branch or merge point are simply re-applied afterwards (by Git, not me). Update…fix conflicts…push…done!

So I promised more actual posts I believe, I am getting to it, albeit slowly.

For some background into the lack of actual language related work I’ve produced lately it helps to understand, arguably the most important, project that I’m working on right now. The Mention Programming Language is a new language that I’ve had in the works since the very beginning of this blog. At one point I even had some pages dedicated to the language. Don’t bother looking, there are no posts about the language yet… I decided very early on to develop the language in the dark, at least until I was sure that it would lead to an actual release of some sort (I’ve worked on quite a few dead language designs over the years).

So Mention is a functional programming language with strong influences from Haskell, and Smalltalk. Yup, you heard right, the design of the language is a dynamic one, which despite the Haskell based syntax will provide a myriad of dynamic features like delegation and accessors more, much more reminiscent of Smalltalk than Haskell. There are a couple of other minor influences that creep in there every so often, due to other languages I’ve worked with or investigated (the design), but those are the main ones.

There’s only one problem, however. I don’t understand the math (lambda calculus and type theory) needed to implement the language yet. I’m getting there but it is taking a little while. The good news is that this wont stop me from posting for much longer. Some of the posts might be about specific things relating the the theory, but the real problem was my lack of a mathematical background. Thanks to my undergrad math text that problem is much less of an issue now and it the Mention project should be getting underway properly very soon, expect some posts on the non-implementation issues like the syntax high level semantics.

On another issue might now be obvious why I haven’t finished the Scheme parsing library and the corresponding series about its design and implementation… I’m not working in Scheme anymore. There is still plenty I could say about the design of Scheme and Lisp dialects, but I’ll leave that for later, but what I would like to add in post in the near future is the relationship between laziness + monads and strictness + mutable-state. For this comparison I’ll be using Scheme and Haskell (don’t worry…the code will only be for examples in both cases). What won’t be happening is any more posts or work on the Scheme parser itself.

– Lorenz

What does this mean?

From now on the proper URL for A Lexical mistake is simply alexicalmistake.com

The configuration for the site seems to properly redirect pretty much everything to the new URL (including permalinks… I’ll keep the old URL pointing to the site for this very reason).

On another issue, the blog very much isn’t dead, it just seems that way because I’ve been somewhat preoccupied lately. The main reason for this is my move to Linux, or more specifically Ubuntu. I’ve also been transitioning most of my mac dependencies to their web equivalents (Google reader, etc).

The point is I should be back to posting biweekly again (or close too), very soon now.

– Lorenz

The real problem wasn’t that I couldn’t use Emacs for basic editing, rather I didn’t really have any idea how to use any of the extra features. Essentially it was like working with a semi-decent TextEdit.app, with code highlighting and unfamiliar keyboard shortcuts. Even after reading some of Steve Yegge’s essays I still didn’t really have any idea what I was doing.

Ok, so basically I’m now writing this in Emacs’ SGML mode and carefully trying to get around the perverse limitations of Wordpress. (Where the heck did the allow raw HTML checkbox go for author text?) Ironically I don’t think I’ve yet gained the productivity advantages from my efforts yet, but honestly that can probably just be attributed to a slow down in my typing speed and the rather unexpected work schedule I’ve been (you never realize just how much you appreciate sleep until you don’t get enough).

If I haven’t gained any real productivity, the question becomes why then am I still using Emacs to write this post? Well, as it turns out its not actually too bad for even when you don’t know all of the keyboard commands (all commands come from the keyboard, and most important commands really never migrate to them menu or at least don’t indicate they’re keyboard equivelent).

Even more importantly perhaps is the gains that I’ve achieved in other tasks…writing blog posts in Emacs may not be any faster for me (although I do seem to see much more of the post at any one time)…but writing code sure is. I seriously doubt that I’d change from using things like XCode for Cocoa applications or DrScheme for PLT Scheme applications (although I may change my mind on Scheme), but at least I have direct access to the REPL in Haskell and Erlang.

The way most of the editing modes in Emacs actually work, it essentially will even load up a file your working directly into a REPL running side by side in another text buffer. From there you can test smaller parts of your program or even just run the whole thing from the top if you needed to. Anyone who’s worked with a Lisp dialect or even Python or Erlang (among others) should be able to apreciate that.

There are a couple of things that have been bugging me so far. The first is the lack of a spell checker. Emacs does have built-in support for ispell, but as you probably guessed, it doesn’t work on my machine (actually it isn’t even installed on my cygwin install). Too make matters worse, I still need to teach Emacs how to find my Cygwin install and use programs from it before the windows paths…ugh…

And the other thing doesn’t even have anything to do with Emacs, but rather, its a limitation with the Haskell code I’m working on. Essentially I don’t have a framework for OpenGL work that allows me to substitute the code on my side of the framework after the application has been started…or more to the point I can’t stop the application after it has started, forcing me to quit the entire REPL every time I test the Game.

Of course the simple workaround is to just keep restarting the REPL or move out of Emacs into a Cygwin shell, but its still very inconvenient (damn you GLUT).

All in all I can only really see how it goes, but at the very least it should make life just that little bit easier. At worst, I’ll just have to resort to my aging Mac Powerbook (for the BSD layer).

Until next time…

– Lorenz

White this might make sense for a syntax based DSL (imho), I wasn’t sure this was absolutely necessary under OpenGL since the entire library is already just a bunch of monadic actions!?

Ok… that’s cool, maybe it was just overlooked, and I thus devised a test. Write a small piece of monadic code that re-evaluated an action over and over again…and it worked!!!

w00t!!!1one

uh… yeah… maybe you should just read this instead… save yourself some time and embarrassment ;)

IO Inside

One the upside… that game is still slowly progressing…

– Lorenz

In fact, I’ve even got an object system that I’m planning to implement just to find out how well this style of concurrency works in an Object Oriented programming language. The design of the language is even more post-Self/Slate than Newspeak (although I believe that this language will just complement the Smalltalk family of languages in all honesty).

Well then, Haskell…Haskell is actually very interesting, in this case by being one of the very few to push the limits of locking methods for concurrency rather than using the larger granularity involved in processes and message passing (not by much, but its still something).

The problem is that its impossible to write locking code by hand that doesn’t have at least one race condition hiding away in the corner. Not the ones you’ve already tested for, but the ones that would cause transactions to restart…like multiple threads locking a few items and then deadlocking…or even just managing the code required to restart deadlocks…yeah…lol…

Haskell provides a novel approach (its actually pretty old now, but it’s really only just starting to catch on, outside of Haskell anyway)… Software Transactional Memory.

STM is sometimes used in other areas of computing (OODBs, et ceteral), but the real gem here is that the Haskell type system nicely compartmentalized the side effects involved into a clever transaction form, within our imperative outer program. This makes working with transactions very much like the database stuff, only we’re only working with individual variables, so no complex table lookups.

Anyway, enough of my rambling, check it out:
Programming in the Age of Concurrency Software Transactional Memory

And if you haven’t already, remember to look at Erlang too…after all concurrency isn’t going away, and there are only two ways to get it right!

– Lorenz

To get things going, last time we looked at a simple character facility which returns a parser that will only accept that particular character. We can also use this technique to select from a range of characters.

(define (char-range start end)
  (λ (state)
    (let ([rc (read-char state)])
      (cond ((eof-object? rc) fail)
            ((and (char>=? rc start)
                  (char<=? rc end))
             rc)
            (else fail)))))

We might also wish to repeat the actions of a parser, which involves writing a proper combinator. Actually, this is the first real utility parser we’ll see and it simply allows us to infinitely repeat the parser provided.

(define (many p)
  (letrec ([mp (λ (state acc)
                 (aif result fail? ((with-mark p) state)
                      acc
                      (mp state (cons result acc))))])
    (λ (state)
      (reverse (mp state '())))))

This defines a small function that accumulates a list (backwards as usual) by repeatedly applying the parser (p). The with-mark stuff is the second combinator we’ll look at, which is used here to prevent the stream position from moving beyond the last match.

The description of many would be a combinator that continues to apply a parser until it fails, reverting the parser’s state to the state prior to running the final failing application of the parser.

The definition of with-mark is even simpler (or should this be with-state?).

(define (with-mark parser)
  (λ (state)
    (let ([mark (file-position state)])
      (aif v fail? (parser state)
           (begin (file-position state mark)
                  v)
           v))))

Our parser state is of course still non-functional, based simply on the port capabilities built into Scheme. We check to make sure the parser doesn’t fail, and if it does we reset the parsers state by recording it before hand. Non-functional user state can be stored in a larger closure/environment, and this could be changed to support functional state as a second variable (in a data structure or list), but I’ll leave to a later post, or the reader of course.

Running our parser is actually this simple (from the repl).

(define digit
  (char-range #\0 #\9))

(define state (open-input-string "12345"))

((many digit) state)

There is one slight problem with this… It accepts any number of digits… including zero digits! I’ve provided a second version called many1, but you could actually define it like the following snipped. (Note: See the bottom of this post to find the source code as it currently appears).

Ok, we still need a way to parse whitespace, and to abstract the tokens of whatever language we’re implementing. To help us here we can write a whitespace parser and a token combinator which will help us.

(define ws-char
  (λ (state)
    (let ([rc (read-char state)])
      (cond ((eof-object? rc) fail)
            ((char-whitespace? rc) rc)
            (else fail)))))

(define whitespace
  (many1 ws-char))

(define (wstok tparser)
  (mand (opt whitespace)
        tparser))

I’d just like to point out before we more on… the ws-char parser here actually recognizes any Scheme whitespace character. I could have written this parser to recognize some other whitespace character set but I felt that this would serve us better for now, since we might eventually want to test our method against the existing Scheme language, or at least a sizable subset.

Again, using this is quite simple… to recognize a number token we can simple define the number token parser as (wstok (many1 digit)).

Now we still have one minor issue here… what if we don’t want to simply recognize the language, but also produce some direct result. For example, our parser so far can recognize number tokens, but it doesn’t return those numbers… just the lists of characters that make them up! (Without the whitespace of course.)

(define t-number
  (mlet nstr (wstok (many1 digit))
    (! (string->number (list->string nstr)))))

This is where the magic begins… mlet and ! are just macros which expand to scheme let and raw expressions wrapped in lambdas which require the current parser state. mlet then also takes a symbol to bind, and two parsers, which is why we need the ! macro magic. Remember that parsers are just lambdas that take a single parameter… it doesn’t matter if they don’t use that parameter, or just pass it on (like mlet).

It might also be interesting to note that we could write a combinator that took a function that accepted the results of the parser, and produced a replacement result. Technically this parser is actually very much like the one we just wrote, except that it can only filter the results of a parser, and not the results of several parsers which can only be achieved though nested mlet expressions. Usage might look similar to the following.

(define t-number
  (return nstr (wstok (many1 digit))
    (string->number (list->string nstr))))

Pretty close right.

By this point I thing most of you are starting to see what we’re getting to… the canonical example of an expression parser, but this time I’m actually going to add a slight twist to the story instead. We’re actually going to generate Scheme code instead, and then use the standard eval function instead. I know this might seem like a strange thing to do, especially since we don’t normally use eval in most Scheme code, but think forward a little. Normally you wouldn’t directly evaluate the code you just parsed either… think of eval as our interpreter, operating on an abstract syntax tree… conveniently Scheme code is already very much like an AST so we just use that.

(define t-add
  (return _ (wstok-char #\+)
    '+))

This gives us our addition token returning, obviously, the Scheme + operator. And finally two more pieces to put it all together.

(define (left-assoc term op)
  (letrec ([f (λ (state acc)
                (aif lst fail? ((with-mark (all op term)) state)
                     acc
                     (f state
                        (list (car lst) acc (cadr lst)))))]) ;; (op lhs rhs)
    (mlet result term
      (λ (state)
        (f state result)))))

(define expr
  (left-assoc t-number t-add))

Finally, to end this… rather lengthy… example. Here is a finished REPL using our parser…

(define (expr-repl)
  (display ">> ")
  (let ([line (read-line)])
    (if (string-ci=? "exit" line)
        '()
        (begin
          (display " = ")
          (display (eval (expr (open-input-string line))))
          (newline)
          (expr-repl)))))

Calling it is pretty simple, and we can get results like so…

> (expr-repl)
>> 1
 = 1
>> 1+2+3+4
 = 10
>> 1 + 2 + 3 + 4
 = 10
>> exit
()

The purpose of this simple example is to show you how our combinators and point out what, in my opinion, is possibly the most important observation to make about parser combinators… all our combinators return parsers and all our parsers are not combinators. Yup, you always have to realize when you’re looking at a combinator, and when your looking at a parser. Hell, you could even be looking at a combinator that returns a combinator! (currying et cetera)

Also, in the code I’ve used above, I made a point of following a coding convention that separates top-level parser definitions by using lambdas, and combinators or simple functions which use the define syntax instead. I found that this makes it easier to visually distinguish between the two, without having to resort to an explicit naming convention.

Next time we’ll move beyond this simple stuff and I’ll try to investigate more complicated combinators that don’t take parsers but data structures to construct their resulting parser.

– Lorenz

There are two main elements involved in writing parsers… The Why… and the How (to some extent what and when also fit here… these are technical aspect that is).

Why do we write parsers?

There are actually a few reasons, and this isn’t really a particularly interesting question for us (after all, we’re already writing a parsers aren’t we?), but I would like to briefly recap what I believe are important points to keep in mind when designing a parser combinator framework.

There are two main varieties of parsers, which come in the form of prototypes and production parsers. Regardless of what the parser is being used for, the purpose of the parser is either to support another process, or it is simply to test some linguistic nature of a computer language (concrete syntax typically).

Of these two categories, we also stumble over a hybrid variation, where a parser written for production use may end up serving a prototype role and, of course, prototype parsers may end up evolving into production parsers.

The importance of this categorization is the implications of generally knee-jerk decisions associated with each approach. The main decisions we need to worry about ourselves when designing a parser framework are of course the error reporting and recovery mechanisms, basic performance concerns, and to some extent the language and tool support.

This isn’t just limited to our intended audience either, but also to ourselves as parser implementors… its all to common to consider why we are writing a parser and select a methodology based purely on practical concerns, and yet, quite often the very notation we use to write the parser is more important. The notation can have very severe effects on the cognitive processes involved in writing a parser or computer language, and so too can the tools that support that notation.

How do we implement parsers?

This part is a little simpler to describe at a high level… that and we’ll be delving into the issues more closely later on anyway.

I personally like to break the implementation concerns into three major categories… recognition, state, and output.

Each of these can become extraordinarily complex and intertwined, but they always tend to follow the same basic trade-offs which we’ll discuss briefly here.

Recognizing input is possibly the most obvious element of parser construction, since it tends to follow the design of the language anyway. More specifically it tends to follow the formal definition of a language, and most conventional notations use the same form as the BNF or PEG formal notation.

While it might seem useful to apply the same notation as the formal definition of a language, imo its not particularly friendly in practice. The main issue with this trade off is the difficulty in implementing non-trivial concerns like left-factoring, lookahead, error reporting/recovery or simply the fact that they don’t contain any proper form of abstraction (no state information or context sensitivity).

This the actually the reason we’re using a combinator approach… it allows us to capture the conceptual elements involved rather than directly writing a whole lot of repetitive rules, simply because the parser generator tool we selected, didn’t realize that scanners aren’t the only type of repetitive task in language writing (expression/operator grammars for example)… or even the lowly delimited list based on non-conventional parameter parsers.

This might actually be a good time to point out the ANTLR parser generator as an example of the kind of trade of we’re trying to avoid. I can’t stress enough how often I’ve wished I could refactor part of a parser involving delimited lists, enclosed parsers, associativity and precedence rules, and it just gets worse from there. The problem? No higher-level abstraction of rules (its like programming a user interface with 1st order logic… well… maybe not quite that bad but you get the idea).

Collecting state is the second important consideration when parsing a computer language. It might be interesting to consider what kind of state you’re accumulation however. In a calculator for example, the state is actually the current number and possibly a bindings table for variables if you allow them, but for a C parser you also need the symbol table and possibly even some other annotations for the preprocessor phase.

The point I’m trying to make here is not that parsers must return something… we’ll get to that in a second… the point is that many parser have other state that is needed by the other software components that receive the output form. This state obviously cannot be collected directly as a result, and yet it also has to be collected directly as a result of parsing.

Why do I care so much about these side effects? Its a perfect opportunity to reflect on Parsec’s usage of monads… and why we don’t need them!

Yup… that’s right, monads primarily capture a few things, but the main elements are timely evaluation (not necessarily eager), imperative ordering of evaluation, and propagation of out-of-band state information. To paraphrase… side effects and imperative code blocks. (Monads are actually capable of quite a bit more than this, but in our case, none of that behaviour is relevant.)

Actually we do still need to pass state, but whether we do it globally (uh…), or through an accumulator doesn’t matter… we have a different way of writing side effects, and we’ll use that for them. Scheme also provides the begin notation for us, so we have a perfectly viable means of writing imperative code if we need it, but even in Parsec it’s rarely a necessity.

Finally we come to the Output form, which is almost exclusively broken down into Abstract Syntax Trees (ASTs), Compilers (intermediate or machine code output), Simple Recognizers, Translators (Scheme to C for example), and Interpreters. Of these only ASTs are used when we have reasonable levels of computation power available, while the other forms are typically relegated to AST parsers or in our case, the native pattern matching and program code of the host language.

The big challenge we’re facing here however is actually related somewhat to the difference between prototype and production parsers. If we wish to provide good support for prototyping, we have to realize that its not entirely uncommon for interpreters to be written without an accompanying AST representation. Unless we can make the AST representation so inexpensive to generate and extraordinarily useful, we may wish to provide support for both ASTs and interpreters.

Whatever trade-off we make however, as long as we realize that we need to effectively support all of the possible applications unless we expect programmers to also use another parser framework or generator. Not necessarily perfectly, but at least adequately (we need to at least make generating ASTs cheap so that programmers can work with them even if the other forms are not simple to directly implement).

Before I finish I’d also like to add a small note about how parsers are actually used… or more specifically that the tools we provide for testing, debugging and maintaining our parsers are possibly more important than the ability to write the parsers in the first place.

For those who have used ANTLR before, you probably realize what I’m talking about here… yes the ANTLR grammars are limited, and painful to effectively refactor, when half the time you’re only even generating ASTs anyway. But the grammar debuggers and generally the ANTLR Works IDE are exceptional tools for constructing parsers. Also… I don’t mean to pay out the ANTLR parser generator… it really is the best tool I’ve used with Java or C++… its just not as powerful as I believe we can get in Scheme

While we can simplify the construction of most parsers greatly, it would be even better if we could also provide dedicated tooling for profiling, debugging, inspecting or even just testing our parsers. To this end, I’ll be adding various tooling issues interspersed with the construction of various parser facilities as we construct hopefully the canonical Scheme parser framework… or at least make the framework as useful as possible.

I don’t intend on covering any of this again in future posts in the series. Instead we’ll only be investigating the trade offs relevant for a Scheme parser combinator framework

– Lorenz