+++ /dev/null
-README.TXT
-==========
-For installation matters, cf. INSTALL.TXT
-
-Metalua 0.5
-===========
-
-Metalua is a static metaprogramming system for Lua: a set of tools
-that let you alter the compilation process in arbitrary, powerful and
-maintainable ways. For the potential first-time users of such a
-system, a description of these tools, as implemented by Metalua,
-follows.
-
-Dynamic Parsers
----------------
-
-One of the tools is the dynamic parser, which allows a source file to
-change the grammar recognized by the parser while it's being
-parsed. Taken alone, this feature lets you make superficial syntax
-tweaks on the language. The parser is based on a parser combinator
-library called 'gg'; you should know the half dozen functions in gg
-API to do advanced things:
-
-- There are a couple of very simple combinators like gg.list,
- gg.sequence, qq.multisequence, gg.optkeyword etc. that offer a level
- of expressiveness comparable to Yacc-like parsers. For instance, if
- mlp.expr parses Lua expressions, gg.list{ mlp.expr } creates a
- parser which handles lists of Lua expressions.
-
-- Since you can create all the combinators you can think of (they're
- regular, higher-order functions), there also are combinators
- specialized for typical language tasks. In Yacc-like systems, the
- language definition quickly becomes unreadable, because all
- non-native features have to be encoded in clumsy and brittle ways.
- So if your parser won't natively let you specify infix operator
- precedence and associativity easily, tough luck for you and your
- code maintainers. With combinators OTOH, most of such useful
- functions already exist, and you can write your own without
- rewriting the parser itself. For instance, adding an infix operator
- would just look like:
-
- > mlp.expr.infix:add{ "xor", prec=40, assoc='left', builder=xor_builder }
-
- Moreover, combinators tend to produce usable error messages when fed
- with syntactically incorrect inputs. It matters, because clearly
- explaining why an invalid input is invalid is almost as important as
- compiling a valid one, for a use=able compiler.
-
-Yacc-like systems might seem simpler to adopt than combinators, as
-long as they're used on extremely simple problems. However, if you
-either try to write something non trivial, or to write a simple macro
-in a robust way, you'll need to use lots of messy tricks and hacks,
-and spend much more time getting them (approximately) right than
-that 1/2 hour required to master the regular features of gg.
-
-
-Real meta-programming
----------------------
-
-If you plan to go beyond trivial keyword-for-keyword syntax tweaks,
-what will limit you is not syntax definition, but the ability to
-manipulate source code conveniently: without the proper tools and
-abstractions, even the simplest tasks will turn into a dirty hacks
-fest, then either into a maintenance nightmare, or simply into
-abandonware. Providing an empowering framework so that you don't get
-stuck in such predicaments is Metalua's whole purpose. The central
-concept is that programs prefer to manipulate code as trees, whereas
-most developers prefer ASCII sources, so both representations must be
-freely interchangeable. The make-or-break deal is then:
-
-- To easily let users see sources as trees, as sources, or as
- combination thereof, and switch representations seamlessly.
-
-- To offer the proper libraries that won't force you to reinvent a
- square wheel will take care of the most common pitfalls and won't
- force you to resort to brittle hacks.
-
-On the former point, Lisps are at a huge advantage, their user syntax
-already being trees. But languages with casual syntax can also offer
-interchangeable tree/source views; Metalua has some quoting +{ ... }
-and anti-quoting -{ ... } operators which let you switch between both
-representations at will: internally it works on trees, but you always
-have the option to see them as quoted sources. Metalua also supports a
-slightly improved syntax for syntax trees, to improve their
-readability.
-
-Library-wise, Metalua offers a set of syntax tree manipulation tools:
-
-- Structural pattern matching, a feature traditionally found in
- compiler-writing specialized languages (and which has nothing to do
- with string regular expressions BTW), which lets you express
- advanced tree analysis operations in a compact, readable and
- efficient way. If you have to work with advanced data structures
- and you try it, you'll never go back.
-
-- The walker library allows you to perform transformations on big
- portions of programs. It lets you easily express things like:
- "replace all return statements which aren't in a nested function by
- error statements", "rename all local variables and their instances
- into unique fresh names", "list the variables which escape this
- chunk's scope", "insert a type-checking instruction into every
- assignment to variable X", etc. Most of non-trivial macros will
- require some of those global code transformations, if you really want
- them to behave correctly.
-
-- Macro hygiene, although not perfect yet in Metalua, is required if
- you want to make macro writing reasonably usable (and contrary to a
- popular belief, renaming local variables into fresh names only
- address the easiest part of the hygiene issue; cf. changelog below
- for more details).
-
-- The existing extensions are progressively refactored in more modular
- ways, so that their features can be effectively reused in other
- extensions.
-
-
-Noteworthy changes from 0.4.1 to 0.5
-====================================
-
-Simplification of the install and structure:
-
-- This release is included in Lua for Windows, so now it couldn't get simpler
- for MS-Windows users!
-
-- Metalua is written in pure Lua again, thus making it platform-independant.
- No more mandatory C libraries. Pluto interface might be back, as an option,
- in a future version, but it's not worth the install trouble involved by
- DLL dependencies.
-
-- Simpler build process, just run make.sh or make.bat depending on your OS.
-
-- Metalua libraries are now in a separate metalua/* package. This allows to
- mix them with other Lua libraries, and to use them from plain Lua programs
- if you FIXME
-
-
-Other changes:
-
-- new option -S in metalua: prints sources re-generated from AST, after macro
- expansion.
-
-- compatible with more Lua VMs: 64 bits numbers, integral numbers, big endians...
-
-- some new extensions: xloop, xmatch, improved match.
-
-- ASTs now keep track of the source that generated them (API is not
- mature though, it will be changed and broken).
-
-- improved table printer: support of a plain-Lua mode, alternative indentation
- mode for deeply-nested tables.
-
-- added a generic table serializer, which handles shared and recursive
- sub-tables correctly.
-
-- gg API has been made slightly more flexible, as a first step towards a
- comprehensive syntax support for gg grammar definition. Follow the gg-syntax
- branch on github for ongoing work.
-
-
-Noteworthy changes from 0.4 to 0.4.1
-====================================
-
-- Proper reporting of runtime errors
-- Interactive REPL loop
-- Support for 64 bits architectures
-- Update to Pluto 2.2 and Lua 5.1.3
-- Build for Visual Studio .NET
-
-Noteworthy changes from 0.3 to 0.4
-=================================
-
-- A significantly bigger code base, mostly due to more libraries:
- about 2.5KLoC for libs, 4KLoC for the compiler. However, this remains
- tiny in today's desktop computers standards. You don't have to know
- all of the system to do useful stuff with it, and since compiled
- files are Lua 5.1 compatible, you can keep the "big" system on a
- development platform, and keep a lightweight runtime for embedded or
- otherwise underpowered targets.
-
-
-- The compiler/interpreter front-end is completely rewritten. The new
- frontend program, aptly named 'Metalua', supports proper passing of
- arguments to programs, and is generally speaking much more user
- friendly than the mlc from the previous version.
-
-
-- Metalua source libraries are searched for in environmemt variable
- LUA_MPATH, distinct from LUA_PATH. This way, in an application
- that's part Lua part Metalua, you keep a natural access to the
- native Lua compiler.
-
- By convention, Metalua source files should have extension .mlua. By
- default, bytecode and plain lua files have higher precedence than
- Metalua sources, which lets you easily precompile your libraries.
-
-
-- Compilation of files are separated in different Lua Rings: this
- prevents unwanted side-effects when several files are compiled
- (This can be turned off, but shouldn't be IMO).
-
-
-- Metalua features are accessible programmatically. Library
- 'Metalua.runtime' loads only the libraries necessary to run an
- already compiled file; 'Metalua.compile' loads everything useful at
- compile-time.
-
- Transformation functions are available in a library 'mlc' that
- contains all meaningful transformation functions in the form
- 'mlc.destformat_of_sourceformat()', such as 'mlc.luacfile_of_ast()',
- 'mlc.function_of_luastring()' etc. This library has been
- significantly completed and rewritten (in Metalua) since v0.3.
-
-
-- Helper libraries have been added. For now they're in the
- distribution, at some point they should be luarocked in. These
- include:
- - Lua Rings and Pluto, duct-taped together into Springs, an improved
- Rings that lets states exchange arbitrary data instead of just
- scalars and strings. Since Pluto requires a (minor) patch to the
- VM, it can be disabled.
- - Lua bits for bytecode dumping.
- - As always, very large amounts of code borrowed from Yueliang.
- - As a commodity, I've also packaged Lua sources in.
-
-
-- Extensions to Lua standard libraries: many more features in table
- and the baselib, a couple of string features, and a package system
- which correctly handles Metalua source files.
-
-
-- Builds on Linux, OSX, Microsoft Visual Studio. Might build on mingw
- (not tested recently, patches welcome). It's easily ported to all
- systems with a full support for lua, and if possible dynamic
- libraries.
-
- The MS-windows building is based on a dirty .bat script, because
- that's pretty much the only thing you're sure to find on a win32
- computer. It uses Microsoft Visual Studio as a compiler (tested with
- VC++ 6).
-
- Notice that parts of the compiler itself are now written in Metalua,
- which means that its building now goes through a bootstrapping
- stage.
-
-
-- Structural pattern matching improvements:
- - now also handles string regular expressions: 'someregexp'/pattern
- will match if the tested term is a string accepted by the regexp,
- and on success, the list of captures done by the regexp is matched
- against pattern.
- - Matching of multiple values has been optimized
- - the default behavior when no case match is no to raise an error,
- it's the most commonly expected case in practice. Trivial to
- cancel with a final catch-all pattern.
- - generated calls to type() are now hygienic (it's been the cause of
- a puzzling bug report; again, hygiene is hard).
-
-
-- AST grammar overhaul:
- The whole point of being alpha is to fix APIs with a more relaxed
- attitude towards backward compatibility. I think and hope it's the
- last AST revision, so here is it:
- - `Let{...} is now called `Set{...}
- (Functional programmers would expect 'Let' to introduce an
- immutable binding, and assignment isn't immutable in Lua)
- - `Key{ key, value } in table literals is now written `Pair{ key, value }
- (it contained a key *and* its associated value; besides, 'Pair' is
- consistent with the name of the for-loop iterator)
- - `Method{...} is now `Invoke{...}
- (because it's a method invocation, not a method declaration)
- - `One{...} is now `Paren{...} and is properly documented
- (it's the node representing parentheses: it's necessary, since
- parentheses are sometimes meaningful in Lua)
- - Operator are simplified: `Op{ 'add', +{2}, +{2} } instead of
- `Op{ `Add, +{2}, +{2} }. Operator names match the corresponding
- metatable entries, without the leading double-underscore.
- - The operators which haven't a metatable counterpart are
- deprecated: 'ne', 'ge', 'gt'.
-
-
-- Overhaul of the code walking library:
- - the API has been simplified: the fancy predicates proved more
- cumbersome to use than a bit of pattern matching in the visitors.
- - binding identifiers are handled as a distinct AST class
- - walk.id is scope-aware, handles free and bound variables in a
- sensible way.
- - the currified API proved useless and sometimes cumbersome, it's
- been removed.
-
-
-- Hygiene: I originally planned to release a full-featured hygienic
- macro system with v0.4, but what exists remains a work in
- progress. Lua is a Lisp-1, which means unhygienic macros are very
- dangerous, and hygiene a la Scheme pretty much limits macro writing
- to a term rewriting subset of the language, which would be crippling
- to use.
-
- Note: inside hygiene, i.e. preventing macro code from capturing
- variables in user code, is trivial to address through alpha
- conversion, it's not the issue. The trickier part is outside
- hygiene, when user's binders capture globals required by the
- macro-generated code. That's the cause of pretty puzzling and hard
- to find bugs. And the *really* tricky part, which is still an open
- problem in Metalua, is when you have several levels of nesting
- between user code and macro code. For now this case has to be
- hygienized by hand.
-
- Note 2: Converge has a pretty powerful approach to hygienic macros
- in a Lisp-1 language; for reasons that would be too long to expose
- here, I don't think its approach would be the best suited to Metalua.
- But I might well be proved wrong eventually.
-
- Note 3: Redittors must have read that Paul Graham has released Arc,
- which is also a Lisp-1 with Common Lisp style macros; I expect this
- to create a bit of buzz, out of which might emerge proper solutions
- the macro hygiene problem.
-
-
-- No more need to create custom syntax for macros when you don't want
- to. Extension 'dollar' will let you declare macros in the dollar
- table, as in +{block: function dollar.MYMACRO(a, b, c) ... end},
- and use it as $MYMACRO(1, 2, 3) in your code.
-
- With this extension, you can write macros without knowing anything
- about the Metalua parser. Together with quasi-quotes and automatic
- hygiene, this will probably be the closest we can go to "macros for
- dummies" without creating an unmaintainable mess generator.
-
- Besides, it's consistent with my official position that focusing on
- superficial syntax issues is counter-productive most of the time :)
-
-
-- Lexers can be switched on the fly. This lets you change the set of
- keywords temporarily, with the new gg.with_lexer() combinator. You
- can also handle radically different syntaxes in a single file (think
- multiple-languages systems such as LuaTeX, or programs+goo as PHP).
-
-
-- Incorporation of the bug fixes reported to the mailing list and on
- the blog.
-
-
-- New samples and extensions, in various states of completion:
-
- * lists by comprehension, a la python/haskell. It includes lists
- chunking, e.g. mylist[1 ... 3, 5 ... 7]
-
- * anaphoric macros for 'if' and 'while' statements: with this
- extension, the condition of the 'if'/'while' is bound to variable
- 'it' in the body; it lets you write things like:
-
- > while file:read '*l' do print(it) end.
-
- No runtime overhead when 'it' isn't used in the body. An anaphoric
- variable should also be made accessible for functions, to let
- easily write anonymous recursive functions.
-
- * Try ... catch ... finally extension. Syntax is less than ideal,
- but the proper way to fix that is to refactor the match extension
- to improve code reuse. There would be many other great ways to
- leverage a refactored match extension, e.g. destructuring binds or
- multiple dispatch methods. To be done in the next version.
-
- * with ... do extension: it uses try/finally to make sure that
- resources will be properly closed. The only constraint on
- resources is that they have to support a :close() releasing method.
- For instance, he following code guarantees that file1 and file2
- will be closed, even if a return or an error occurs in the body.
-
- > with file1, file2 = io.open "f1.txt", io.open "f2.txt" do
- > contents = file1:read'*a' .. file2:read ;*a'
- > end
-
- * continue statement, logging facilities, ternary "?:" choice
- operator, assignments as expressions, and a couple of similarly
- tiny syntax sugar extensions.
-
-
-You might expect in next versions
-=================================
-The next versions of Metalua will provide some of the following
-improvements, in no particular order: better error reporting,
-especially at runtime (there's a patch I've been too lazy to test
-yet), support for 64 bits CPUs, better support for macro hygiene, more
-samples and extensions, an adequate test suite, refactored libraries.
-
-
-Credits
-=======
-
-I'd like to thank the people who wrote the open source code which
-makes Metalua run: the Lua team, the authors of Yueliang, Pluto, Lua
-Rings, Bitlib; and the people whose bug reports, patches and
-insightful discussions dramatically improved the global design,
-including John Belmonte, Vyacheslav Egorov, David Manura, Olivier
-Gournet, Eric Raible, Laurence Tratt, Alexander Gladysh, Ryan
-Pusztai...
projects / metalua.git / blobdiff