1 .HTML "Plumbing and Other Utilities
3 Plumbing and Other Utilities
10 Plumbing is a new mechanism for inter-process communication in Plan 9,
11 specifically the passing of messages between interactive programs as part of
13 Although plumbing shares some properties with familiar notions
14 such as cut and paste,
15 it offers a more general data exchange mechanism without imposing
16 a particular user interface.
18 The core of the plumbing system is a program called the
20 which handles all messages and dispatches and reformats them
21 according to configuration rules written in a special-purpose language.
22 This approach allows the contents and context of a piece of data to define how
24 Unlike with drag and drop or cut and paste,
25 the user doesn't need to deliver the data;
26 the contents of a plumbing message, as interpreted by the plumbing rules,
27 determine its destination.
29 The plumber has an unusual architecture: it is a language-driven file server.
30 This design has distinct advantages.
31 It makes plumbing easy to add to an existing, Unix-like command environment;
32 it guarantees uniform handling of inter-application messages;
33 it off-loads from those applications most of the work of extracting and dispatching messages;
34 and it works transparently across a network.
39 Data moves from program to program in myriad ways.
40 Command-line arguments,
43 drag and drop, and other user interface techniques all provide some form
44 of interprocess communication.
45 Then there are tricks associated with special domains,
46 such as HTML hyperlinks or the heuristics mail readers
47 use to highlight URLs embedded in mail messages.
48 Some systems provide implicit ways to automate the attachment of program to data\(emthe
49 best known examples are probably the resource forks in MacOS and the
50 file name extension `associations' in Microsoft Windows\(embut in practice
51 humans must too often carry their data from program to program.
53 Why should a human do the work?
54 Usually there is one obvious thing to do with a piece of data,
55 and the data itself suggests what this is.
56 Resource forks and associations speak to this issue directly, but statically and narrowly and with
57 little opportunity to control the behavior.
58 Mechanisms with more generality,
59 such as cut and paste or drag and drop, demand too much manipulation by
60 the user and are (therefore) too error-prone.
62 We want a system that, given a piece of data,
63 hands it to the appropriate application by default with little or no human intervention,
64 while still permitting the user to override the defaults if desired.
66 The plumbing system is an attempt to address some of these issues in a single,
67 coherent, central way.
68 It provides a mechanism for
69 formatting and sending arbitrary messages between applications,
70 typically interactive programs such as text editors, web browsers, and the window system,
71 under the control of a central message-handling server called the
73 Interactive programs provide application-specific connections to the plumber,
74 triggering with minimal user action the transfer of data or control to other programs.
75 The result is similar to a hypertext system in which all the links are implicit,
76 extracted automatically by examining the data and the user's actions.
78 cut and paste and other such hand-driven interprocess communication mechanisms.
79 Plumbing delivers the goods to the right place automatically.
83 The plumber is implemented as a Plan 9 file server [Pike93];
84 programs send messages by writing them to the plumber's file
86 and receive messages by reading them from
88 which are other plumber files in
92 is by convention the file from which a text editor reads messages requesting it to
93 open and display a file for editing.
99 P1: ellipse "ProgramA"
101 P2: ellipse "ProgramB"
103 P3: ellipse "ProgramC"
105 INVIS: box wid 1.3 invis at P2.e
106 SEND: arrow from INVIS.e "\f(CWsend \fP" ""
107 arrow -> right 0.2 from P1.e; spline -> right 0.2 then down 1 to SEND.w
108 arrow -> right 0.2 from P2.e; arrow -> to SEND.w
109 arrow -> right 0.2 from P3.e; spline -> right 0.2 then up 1 to SEND.w
111 PL: box height 1 "plumber" with .w at SEND.e
112 A3: arrow 0.8 -> "\f(CWimage\fP" ""; arrow ->
114 O2: ellipse "Browser" with .s at O3.n + (0, 0.1)
115 O1: ellipse "Editor" with .s at O2.n + (0, 0.1)
116 O4: ellipse "Faces" with .n at O3.s + (0, -0.1)
117 O5: ellipse "..." with .n at O4.s + (0, -0.1)
119 A1: arrow 0.8 -> "\f(CWedit\fP" "" from PL.e + (0, .4); spline -> right 0.15 then up 0.7 then to O1.w
121 A2: arrow 0.8 -> "\f(CWweb\fP" "" from PL.e + (0, .2); spline -> right 0.3 then up 0.3 then to O2.w
123 A4: arrow 0.8 -> "\f(CWnewmail\fP" "" from PL.e + (0, -.2); spline -> right 0.3 then down 0.3 then to O4.w
125 A5: arrow 0.8 -> "\f(CW...\fP" "" from PL.e + (0, -.4); spline -> right 0.15 then down 0.7 then to O5.w
129 Figure 1. The plumber controls the flow of messages between applications.
130 Programs write to the file
132 and receive on `ports' of various names representing services such as
136 Although the figure doesn't illustrate it, some programs may both send and receive messages,
137 and some ports are read by multiple applications.
142 The plumber takes messages from the
144 file and interprets their contents using rules defined by
145 a special-purpose pattern-action language.
146 The language specifies any rewriting of the message that is to be done by the plumber
147 and defines how to dispose of a message, such as by sending it to a port or
148 starting a new process to handle it.
150 The behavior is best described by example.
151 Imagine that the user has, in a terminal emulator window,
152 just run a compilation that has failed:
156 rmstar.c:32: syntax error
159 The user points the typing cursor somewhere in the string
164 This causes the terminal emulator to format a plumbing message
165 containing the entire string surrounding the cursor,
168 .CW /mnt/plumb/send .
169 The plumber receives this message and compares it sequentially to the various
170 patterns in its configuration.
171 Eventually, it will find one that breaks the string into pieces,
176 Other associated patterns verify that
178 is a file in the current directory of the program generating
179 the message, and that
181 looks like a line number within it.
182 The plumber rewrites the message,
183 setting the data to the string
185 and attaching an indication that
187 is a line number to display.
188 Finally, it sends the resulting message to the
191 The text editor picks up the message, opens
193 (if it's not already open) and highlights line 32, the location of the syntax error.
195 From the user's point of view, this process is simple: the error message appears,
196 it is `plumbed', and the editor jumps to the problem.
198 Of course, there are many different ways to cause compiler messages to
199 pop up the source of an error,
200 but the design of the plumber addresses more general issues than the specific
201 goal of shortening the compile/debug/edit cycle.
202 It facilitates the general exchange of data among programs, interactive or otherwise,
203 throughout the environment, and its
204 architecture\(ema central, language-driven file server\(emalthough
205 unusual, has distinct advantages.
206 It makes plumbing easy to add to an existing, Unix-like command environment;
207 it guarantees uniform handling of inter-application messages;
208 it off-loads from those applications most of the work of extracting and dispatching messages;
209 and it works transparently and effortlessly across a network.
211 This paper is organized bottom-up, beginning with the format of the messages
212 and proceeding through the plumbing language, the handling of messages,
213 and the interactive user interface.
214 The last sections discuss the implications of the design
215 and compare the plumbing system to other environments that
216 provide similar services.
220 Since the language that controls the plumber is defined in terms of the
221 contents of plumbing messages, we begin by describing their layout.
223 Plumbing messages have a fixed-format textual
224 header followed by a free-format data section.
225 The header consists of six lines of text, in set order,
226 each specifying a property of the message.
227 Any line may be blank except the last, which is the length of the data portion of the
228 message, as a decimal string.
229 The lines are, in order:
231 The source application, the name of the program generating the message.
233 The destination port, the name of the port to which the messages should be sent.
235 The working directory in which the message was generated.
237 The type of the data, analogous to a MIME type, such as
242 Attributes of the message, given as blank-separated
243 .I name\f(CW=\fPvalue
245 The values may be quoted to protect
246 blanks or quotes; values may not contain newlines.
248 The length of the data section, in bytes.
250 Here is a sample message, one that (conventionally) tells the editor to open the file
251 .CW /usr/rob/src/mem.c
263 Because in general it need not be text, the data section of the message has no terminating newline.
265 A library interface simplifies the processing of messages by translating them
266 to and from a data structure,
270 .ta 4n +4n +4n +4n +4n +4n +4n +4n +4n +4n +4n +4n +4n
271 typedef struct Plumbattr Plumbattr;
272 typedef struct Plumbmsg Plumbmsg;
276 char *src; /* source application */
277 char *dst; /* destination port */
278 char *wdir; /* working directory */
279 char *type; /* type of data */
280 Plumbattr *attr; /* attribute list */
281 int ndata; /* #bytes of data */
292 The library also includes routines to send a message, receive a message,
293 manipulate the attribute list, and so on.
297 An instance of the plumber runs for each user on each terminal or workstation.
299 begins by reading its rules from the file
301 in the user's home directory,
302 which in turn may use
304 statements to interpolate macro definitions and
305 rules from standard plumbing rule libraries stored in
308 The rules control the processing of messages.
310 a pattern-action language comprising a sequence of blank-line-separated
313 each of which contains one or more
315 followed by one or more
317 Each incoming message is compared against the rule sets in order.
318 If all the patterns within a rule set succeed,
319 one of the associated actions is taken and processing completes.
321 The syntax of the language is straightforward.
322 Each rule (pattern or action) has three components, separated by white space:
330 identifies a part of the message, such as
331 the source application
334 portion of the message
336 or the rule's own arguments
340 which introduces an action.
341 The verb specifies an operation to perform on the object, such as the word
343 to require precise equality between the object and the argument, or
345 to require that the object be the name of a directory.
347 For instance, this rule set sends messages containing the names of files
353 to display them; it is analogous to a Windows association rule:
355 # image files go to page
357 data matches '[a-zA-Z0-9_\e-./]+'
358 data matches '([a-zA-Z0-9_\e-./]+)\e.(jpe?g|gif|bit|tiff|ppm)'
361 plumb client page -wi
363 (Lines beginning with
366 Consider how this rule handles the following message, annotated down the left column for clarity:
371 \f2wdir\fP /usr/rob/pics
379 verb specifies a precise match, and the
381 field of the message is the string
383 so the first pattern succeeds.
386 verb invokes a regular expression pattern match of the object (here
388 against the argument pattern.
391 patterns in this rule set will succeed, and in the process set the variables
393 to the matched string,
395 to the first parenthesized submatch, and so on (analogous to
400 regular expressions).
405 verifies that the named file,
407 is an actual file in the directory
409 If all the patterns succeed, one of the actions will be executed.
411 There are two actions in this rule set.
417 as the destination port of the message.
418 By convention, the plumber mounts its services in the directory
420 so in this case if the file
422 has been opened, the message will be made available to the program reading from it.
423 Note that the message does not name a port, but the rule set that matches
424 the message does, and that is sufficient to dispatch the message.
425 If on the other hand a message matches no rule but has an explicit port mentioned,
426 that too is sufficient.
428 If no client has opened the
431 that is, if the program
433 is not already running, the
436 action gives the execution script to start the application
437 and send the message on its way; the
441 to create a window and to receive its initial arguments from the plumbing port.
442 The process by which the plumber starts a program is described in more detail in the next section.
444 It may seem odd that there are two
446 rules in this example.
447 The reason is related to the way the plumber can use the rules themselves
450 in the message, somewhat in the manner of Structural Regular Expressions [Pike87a].
451 For example, consider what happens if the cursor is at the last character of
453 % make nightmare>horse.gif
455 and the user asks to plumb what the cursor is pointing at.
456 The program creating the plumbing
457 message\(emin this case the terminal emulator running the window\(emcan send the
458 entire white-space-delimited string
459 .CW nightmare>horse.gif
460 or even the entire line, and the combination of
462 rules can determine that the user was referring to the string
464 The user could of course select the entire string
466 but it's more convenient just to point in the general location and let the machine
467 figure out what should be done.
468 The process is as follows.
470 The application generating the message adds a special attribute to the message, named
472 whose numerical value is the offset of the cursor\(emthe selection point\(emwithin the data string.
473 This attribute tells the plumber two things:
474 first, that the regular expressions in
476 rules should be used to identify the relevant data;
477 and second, approximately where the relevant data lies.
479 will then use the first
481 pattern to identify the longest leftmost match that touches the cursor, which will extract the string
483 and the second pattern will then verify that that names a picture file.
484 The rule set succeeds and the data is winnowed to the matching substring
485 before being sent to its destination.
489 pattern within a given rule set must match the same portion of the string, which
490 guarantees that the rule set fails to match a string for which the
491 second pattern matches only a portion.
492 For instance, our example rule set should not execute if the data is the string
494 and although the first pattern will match
496 the second will match only
498 and the rule set will fail.
500 The same approach of multiple
502 rules can be used to exclude, for instance, a terminal period from
503 a file name or URL, so a file name or URL at the end of a sentence is recognized properly.
507 attribute is not specified, all patterns must match the entire string,
508 so the user has an option:
509 he or she may select exactly what data to send,
510 or may instead indicate where the data is by clicking the selection button on the mouse
511 and letting the machine locate the URL or image file name within the text.
513 the user can control the contents of the message precisely when required,
514 but the default, simplest action in the user interface does the right thing most of the time.
516 How Messages are Handled in the Plumber
518 An application creates a message header, fills in whatever fields it wishes to define,
519 attaches the data, and writes the result to the file
521 in the plumber's service directory,
523 The plumber receives the message and applies the plumbing rules successively to it.
524 When a rule set matches, the message is dispatched as indicated by that rule set
525 and processing continues with the next message.
526 If no rule set matches the message, the plumber indicates this by returning a write
527 error to the application, that is, the write to
529 fails, with the resulting error string
530 describing the failure.
531 (Plan 9 uses strings rather than pre-defined numbers to describe error conditions.)
532 Thus a program can discover whether a plumbing message has been sent successfully.
534 After a matching rule set has been identified, the plumber applies a series of rewriting
535 steps to the message. Some rewritings are defined by the rule set; others are implicit.
536 For example, if the message does not specify a destination port, the outgoing message
537 will be rewritten to identify it.
538 If the message does specify the port, the rule set will only match if any
541 action in the rule set names the same port.
542 (If it matches no rule sets, but mentions a port, it will be sent there unmodified.)
544 The rule set may contain actions that explicitly rewrite components of the message.
545 These may modify the attribute list or replace the data section of the message.
546 Here is a sample rule set that does both.
547 It matches strings of the form
551 If that string identifies a file in the standard C include directory,
553 perhaps with an optional line number, the outgoing message
554 is rewritten to contain the full path name and an attribute,
556 to hold the line number:
558 # .h files are looked up in /sys/include and passed to edit
560 data matches '([a-zA-Z0-9]+\e.h)(:([0-9]+))?'
561 arg isfile /sys/include/$1
562 data set /sys/include/$1
569 rule replaces the contents of the data, and the
572 rule adds a new attribute to the message.
573 The intent of this rule is to permit one to plumb an include file name in a C program
574 to trigger the opening of that file, perhaps at a specified line, in the text editor.
575 A variant of this rule, discussed below,
576 tells the editor how to interpret syntax errors from the compiler,
580 both of which use a fixed syntax
582 to identify a line of source.
584 The Plan 9 text editors interpret the
586 attribute as the definition of which portion of the file to display.
587 In fact, the real rule includes a richer definition of the address syntax,
588 so one may plumb strings such as
589 .CW plumb.h:/plumbsend
590 (using a regular expression after the
592 to pop up the declaration of a function in a C header file.
594 Another form of rewriting is that the plumber may modify the attribute list of
595 the message to clarify how to handle the message.
596 The primary example of this involves the treatment of the
598 attribute, described in the previous section.
599 If the message contains a
601 attribute and the matching rule set uses it to extract the matching substring from the data,
605 attribute and replaces the data with the matching substring.
607 Once the message is rewritten, the actions of the matching rule set are examined.
608 If the rule set contains a
611 action and the corresponding port is open\(emthat is, if a program is already reading
612 from that port\(emthe message is delivered to the port.
613 The application will receive the message and handle it as it sees fit.
614 If the port is not open, a
620 action will start a new program to handle the message.
625 action is the simpler: its argument specifies a command to run
626 instead of passing on the message; the message is discarded.
627 Here for instance is a rule that, given the process id (pid) of an existing process,
630 debugger [Wint94] in a new window to examine that process:
632 # processes go to acid (assuming strlen(pid) >= 2)
634 data matches '[a-zA-Z0-9.:_\e-/]+'
635 data matches '[0-9][0-9]+'
637 plumb start window acid $0
639 (Note the use of multiple
641 rules to avoid misfires from strings like
646 rule checks that the pid represents a running process (or broken one; Plan 9 does not create
648 files but leaves broken processes around for debugging) by checking that the process file
649 system has a directory for that pid [Kill84].
650 Using this rule, one may plumb the pid string printed by the
652 command or by the operating system when the program breaks;
653 the debugger will then start automatically.
655 The other startup action,
658 is used when a program will read messages from the plumbing port.
660 text editors can read files specified as command arguments, so one could use a
663 rule to begin editing a file.
664 If, however, the editor will read messages from the
666 plumbing port, letting it read the message
667 from the port insures that it uses other information in the message,
668 such as the line number to display.
672 action is therefore like
675 but keeps the message around for delivery when the application opens the port.
676 Here is the full rule set to pass a regular file to the text editor:
678 # existing files, possibly tagged by address, go to editor
680 data matches '([.a-zA-Z0-9_/\e-]*[a-zA-Z0-9_/\e-])('$addr')?'
685 plumb client window $editor
687 If the editor is already running, the
690 rule causes it to receive the message on the port.
695 will create a new window (using the Plan 9 program
697 to run the editor, and once that starts it will open the
699 plumbing port as usual and discover this first message already waiting.
706 are macros defined in the plumbing rules file; they specify the name of the user's favorite text editor
707 and a regular expression
708 that matches that editor's address syntax, such as line numbers and patterns.
709 This rule set lives in a library of shared plumbing rules that
710 users' private rules can build on,
711 so the rule set needs to be adaptable to different editors and their address syntax.
712 The macro definitions for Acme and Sam [Pike94,Pike87b] look like this:
716 addrelem='((#?[0-9]+)|(/[A-Za-z0-9_\e^]+/?)|[.$])'
717 addr=:($addrelem([,;+\e-]$addrelem)*)
720 Finally, the application reads the message from the appropriate port, such as
721 .CW /mnt/plumb/edit ,
722 unpacks it, and goes to work.
726 In summary, a message is delivered by writing it to the
728 file and having the plumber, perhaps after some rewriting, send it to the destination
729 port or start a new application to handle it.
730 If no destination can be found by the plumber, the original write to the
732 file will fail, and the application will know the message could not be delivered.
734 If multiple applications are reading from the destination port, each will receive
735 an identical copy of the message; that is, the plumber implements fan-out.
736 The number of messages delivered is equal to the number of clients that have
737 opened the destination port.
738 The plumber queues the messages and makes sure that each application that opened
739 the port before the message was written gets exactly one copy.
741 This design minimizes blocking in the sending applications, since the write to the
743 file can complete as soon as the message has been queued for the appropriate port.
744 If the plumber waited for the message to be read by the recipient, the sender could
746 Unfortunately, this design also means that there is no way for a sender to know when
747 the message has been handled; in fact, there are cases when
748 the message will not be delivered at all, such as if the recipient exits while there are
749 still messages in the queue.
750 Since the plumber is part of a user interface, and not
751 an autonomous message delivery system,
752 the decision was made to give the
753 non-blocking property priority over reliability of message delivery.
754 In practice, this tradeoff has worked out well:
755 applications almost always know when a message has failed to be delivered (the
757 fails because no destination could be found),
758 and those occasions when the sender believes incorrectly that the message has been delivered
759 are both extremely rare and easily recognized by the user\(emusually because the recipient
760 application has exited.
764 The plumber begins execution by reading the user's startup plumbing rules file,
766 Since the plumber is implemented as a file server, it can also present its current rules
767 as a dynamic file, a design that provides an easily understood way to maintain the rules.
771 is the text of the rule set the plumber is currently using,
772 and it may be edited like a regular file to update those rules.
773 To clear the rules, truncate that file;
774 to add a new rule set, append to it:
777 data is self-destruct
778 plumb start rm -rf $HOME' >> /mnt/plumb/rules
780 This rule set will take effect immediately.
781 If it has a syntax error, the write will fail with an error message from the plumber,
782 such as `malformed rule' or 'undefined verb'.
784 To restore the plumber to its startup configuration,
786 % cp /usr/$user/lib/plumbing /mnt/plumb/rules
788 For more sophisticated changes,
789 one can of course use a regular text editor to modify
790 .CW /mnt/plumb/rules .
792 This simple way of maintaining an active service could profitably be adopted by other systems.
793 It avoids the need to reboot, to update registries with special tools, or to send asynchronous signals
794 to critical programs.
798 One unusual property of the plumbing system is that
799 the user interface that programs provide to access it can vary considerably, yet
800 the result is nonetheless a unifying force in the environment.
801 Shells talk to editors, image viewers, and web browsers; debuggers talk to editors;
802 editors talk to themselves; and the window system talks to everybody.
804 The plumber grew out of some of the ideas of the Acme editor/window-system/user interface [Pike94],
805 in particular its `acquisition' feature.
806 With a three-button mouse, clicking the right button in Acme on a piece of text tells Acme to
807 get the thing being pointed to.
808 If it is a file name, open the file;
809 if it is a directory, open a viewer for its contents;
810 if a line number, go to that line;
811 if a regular expression, search for it.
812 This one-click access to anything describable textually was very powerful but had several
813 limitations, of which the most important were that Acme's rules for interpreting the
814 text (that is, the implicit hyperlinks) were hard-wired and inflexible, and
815 that they only applied to and within Acme itself.
816 One could not, for example, use Acme's power to open an image file, since Acme is
819 The plumber addresses these limitations, even with Acme itself:
820 Acme now uses the plumber to interpret the right button clicks for it.
821 When the right button is clicked on some text,
822 Acme constructs a plumbing message much as described above,
825 attribute and the white-space-delimited text surrounding the click.
826 It then writes the message to the plumber; if the write succeeds, all is well.
827 If not, it falls back to its original, internal rules, which will result in a context search
828 for the word within the current document.
830 If the message is sent successfully, the recipient is likely to be Acme itself, of course:
831 the request may be to open a file, for example.
832 Thus Acme has turned the plumber into an external component of its own operation,
833 while expanding the possibilities; the operation might be to start an image viewer to
834 open a picture file, something Acme cannot do itself.
835 The plumber expands the power of Acme's original user interface.
837 Traditional menu-driven programs such as the text editor Sam [Pike87b] and the default
838 shell window of the window
841 [Pike91] cannot dedicate a mouse button solely to plumbing, but they can certainly
842 dedicate a menu entry.
843 The editing menu for such programs now contains an entry,
845 that creates a plumbing message using the current selection.
846 (Acme manages to send a message by clicking on the text with one button;
847 other programs require a click with the select button and then a menu operation.)
848 For example, after this happens in a shell window:
852 shaney.c:232: i undefined
855 one can click anywhere on the string
859 menu entry, and have line 232 appear in the text editor, be it Sam or Acme\(emwhichever has the
862 (If this were an Acme shell window, it would be sufficient to right-click on the string.)
864 [An interesting side line is how the window system knows what directory the
865 shell is running in; in other words, what value to place in the
867 field of the plumb message.
870 is, like many Plan 9 programs, a file server.
871 It now serves a new file,
873 that is private to each window.
874 Programs, in particular the
877 can write that file to inform the window system of its current directory.
880 command is executed in an interactive shell,
882 updates the contents of
884 and plumbing can proceed with local file names.]
886 Of course, users can plumb image file names, process ids, URLs, and other items\(emany string
887 whose syntax and disposition are defined in the plumbing rules file.
888 An example of how the pieces fit together is the way Plan 9 now handles mail, particularly
889 MIME-encoded messages.
891 When a new mail message arrives, the mail receiver process sends a plumbing message to the
893 port, which notifies any interested process that new mail is here.
894 The plumbing message contains information about the mail, including
895 its sender, date, and current location in the file system.
896 The interested processes include a program,
898 that gives a graphical display of the mail box using
899 faces to represent the senders of messages [PiPr85],
900 as well as interactive mail programs such as the Acme mail viewer [Pike94].
901 The user can then click on the face that appears, and the
903 program will send another plumbing message, this time to the
906 Here is the rule for that port:
908 # faces -> new mail window for message
910 data matches '[a-zA-Z0-9_\e-./]+'
911 data matches '/mail/fs/[a-zA-Z0-9/]+/[0-9]+'
913 plumb start window edmail -s $0
915 If a program, such as the Acme mail reader, is reading that port, it will open a new window
916 in which to display the message.
920 rule will create a new window and run
922 a conventional mail reading process, to examine it.
923 Notice how the plumbing connects the components of the interface together the same way
924 regardless of which components are actually being used to view mail.
926 There is more to the mail story.
927 Naturally, mail boxes in Plan 9 are treated as little file systems, which are synthesized
928 on demand by a special-purpose file server that takes a flat mail box file and converts
929 it into a set of directories, one per message, with component files containing the header,
930 body, MIME information, and so on.
931 Multi-part MIME messages are unpacked into multi-level directories, like this:
933 % ls -l /mail/fs/mbox/25
934 d-r-xr-xr-x M 20 rob rob 0 Nov 21 13:06 /mail/fs/mbox/25/1
935 d-r-xr-xr-x M 20 rob rob 0 Nov 21 13:06 /mail/fs/mbox/25/2
936 --r--r--r-- M 20 rob rob 28678 Nov 21 13:06 /mail/fs/mbox/25/body
937 --r--r--r-- M 20 rob rob 0 Nov 21 13:06 /mail/fs/mbox/25/cc
943 Date: Sun Nov 21 13:05:51 EST 1999
948 ===> 2/ (image/jpeg) [inline]
949 /mail/fs/mbox/25/2/fabio.jpg
952 Since the components are all (synthetic) files, the user can plumb the pieces
953 to view embedded pictures, URLs, and so on.
954 Note that the mail program can plumb the contents of
956 attachments automatically, without user interaction;
957 in other words, plumbing lets the mailer handle multimedia data
958 without itself interpreting it.
960 At a more mundane level, a shell command,
962 can be used to send messages:
967 will send the appropriate message to the
970 A surprising use of the
972 command is in actions within the plumbing rules file.
973 In our lab, we commonly receive Microsoft Word documents by mail,
974 but we do not run Microsoft operating systems on our machines so we cannot
975 view them without at least rebooting.
976 Therefore, when a Word document arrives in mail, we could plumb the
978 file but the text editor could not decode it.
979 However, we have a program,
981 that decodes the Word file format to extract and format the embedded text.
982 The solution is to use
991 files and synthesize a plain text file:
993 # rule set for microsoft word documents
995 data matches '[a-zA-Z0-9_\e-./]+'
996 data matches '([a-zA-Z0-9_\e-./]+)\e.doc'
998 plumb start doc2txt $data | \e
999 plumb -i -d edit -a action=showdata -a filename=$0
1003 tell it to take standard input as its data rather than the text of the arguments
1005 define the destination port
1008 and set a conventional attribute so the editor knows to show the message data
1009 itself rather than interpret it as a file name
1011 .CW action=showdata )
1012 and provide the original file name
1015 Now when a user plumbs a
1017 file the plumbing rules run a process to extract the text and send it as a
1018 temporary file to the editor for viewing.
1019 It's imperfect, but it's easy and it beats rebooting.
1021 Another simple example is a rule that turns man pages into hypertext.
1022 Manual page entries of the form
1024 can be clicked on to pop up a window containing the formatted `man page'.
1025 That man page will in turn contain more such citations, which will also be clickable.
1026 The rule is a little like that for Word documents:
1028 # man index entries are synthesized
1030 data matches '([a-zA-Z0-9_\e-./]+)\e(([0-9])\e)'
1031 plumb start man $2 $1 | \e
1032 plumb -i -d edit -a action=showdata -a filename=/man/$1($2)
1035 There are many other inventive uses of plumbing.
1036 One more should give some of the flavor.
1037 We have a shell script,
1039 that takes as argument the name of an executable binary file.
1040 It examines the symbol table of the binary to find the source file
1041 from which it was compiled.
1042 Since the Plan 9 compilers place full source path names in the symbol table,
1044 can discover the complete file name.
1045 That is then passed to
1047 complete with the line number to find the
1054 is all it takes to pop up an editor window on the
1058 command, beginning at line 39 of
1059 .CW /sys/src/cmd/plumb/plumb.c .
1060 Like most uses of plumbing,
1061 this is not a breakthrough in functionality, but it is a great convenience.
1063 Why This Architecture?
1065 The design of the plumbing system is peculiar:
1066 a centralized language-based file server does most of the work,
1067 while compared to other systems the applications themselves
1068 contribute relatively little.
1069 This architecture is deliberate, of course.
1071 That the plumber's behavior is derived from a linguistic description
1072 gives the system great flexibility and dynamism\(emrules can be added
1073 and changed at will, without rebooting\(embut the existence of a central library of rules
1074 ensures that, for most users, the environment behaves in well-established ways.
1076 That the plumber is a file server is perhaps the most unusual aspect of its design,
1077 but is also one of the most important.
1078 Messages are passed by regular I/O operations on files, so no extra technology
1079 such as remote procedure call or request brokers needs to be provided;
1080 messages are transmitted by familiar means.
1081 Almost every service in Plan 9 is a file server, so services can be exported
1082 trivially using the system's remote file system operations [Pike93].
1083 The plumber is no exception;
1084 plumbing messages pass routinely across the network to remote applications without
1085 any special provision,
1086 in contrast to some commercial IPC mechanisms that become
1087 significantly more complex when they involve multiple machines.
1088 As I write this, my window system is talking to applications running on three
1089 different machines, but they all share a single instance of the plumber and so
1090 can interoperate to integrate my environment.
1091 Plan 9 uses a shared file name space
1092 to combine multiple networked machines\(emcompute servers,
1093 file servers, and interactive workstations\(eminto a single
1094 computing environment; plumbing's design as a file server
1095 is a natural by-product of, and contributor to, the overall system architecture
1098 The centrality of the plumber is also unusual.
1099 Other systems tend to let the applications determine where messages will go;
1100 consider mail readers that recognize and highlight URLs in the messages.
1101 Why should just the mail readers do this, and why should they just do it for URLs?
1102 (Acme was guilty of similar crimes.)
1103 The plumber, by removing such decisions to a central authority,
1104 guarantees that all applications behave the same and simultaneously
1105 frees them all from figuring out what's important.
1106 The ability for the plumber to excerpt useful data from within a message
1107 is critical to the success of this model.
1109 The entire system is remarkably small.
1110 The plumber itself is only about two thousand lines of C code.
1111 Most applications work fine in a plumbing environment without knowing about it at all;
1112 some need trivial changes such as to standardize their error output;
1113 a few need to generate and receive plumbing messages.
1114 But even to add the ability to send and receive messages in a program such as text editor is short work,
1115 involving typically a few dozen lines of code.
1116 Plumbing fits well into the existing environment.
1118 But plumbing is new and it hasn't been pushed far enough yet.
1119 Most of the work so far has been with textual messages, although
1120 the underlying system is capable of handling general data.
1121 We plan to reimplement some of the existing data movement operations,
1122 such as cut and paste or drag and drop, to use plumbing as their exchange mechanism.
1123 Since the plumber is a central message handler, it is an obvious place to store the `clipboard'.
1124 The clipboard could be built as a special port that holds onto messages rather than
1125 deleting them after delivery.
1126 Since the clipboard would then be holding a plumbing
1127 message rather than plain text, as in the current Plan 9 environment,
1128 it would become possible to cut and paste arbitrary data without
1129 providing new mechanism.
1130 In effect, we would be providing a new user interface to the existing plumbing facilities.
1132 Another possible extension is the ability to override plumbing operations interactively.
1133 Originally, the plan was to provide a mechanism, perhaps a pop-up menu, that one could
1134 use to direct messages, for example to send a PostScript file to the editor rather than the
1135 PostScript viewer by naming an explicit destination in the message.
1136 Although this deficiency should one day be addressed, it should be done without
1137 complicating the interface for invoking the default behavior.
1138 Meanwhile, in practice the default behavior seems to work very well in practice\(emas it
1139 must if plumbing is to be successful\(emso the lack of
1140 overrides is not keenly felt.
1142 Comparison with Other Systems
1144 The ideas of the plumbing system grew from an
1145 attempt to generalize the way Acme acquires files and data.
1146 Systems further from that lineage also share some properties with plumbing.
1147 Most, however, require explicit linking or message passing rather than
1148 plumbing's implicit, context-based pattern matching, and none
1149 has the plumber's design of a language-based file server.
1151 Reiss's FIELD system [Reis95] probably comes the closest to providing the facilities of the plumber.
1152 It has a central message-passing mechanism that connects applications together through
1153 a combination of a library and a pattern-matching central message dispatcher that handles
1154 message send and reply.
1155 The main differences between FIELD's message dispatcher and the plumber are first
1156 that the plumber is based on a special-purpose language while the FIELD
1157 system uses an object-oriented library, second that the plumber has no concept
1158 of a reply to a message, and finally that the FIELD system
1159 has no concept of port.
1160 But the key distinction is probably in the level of use.
1161 In FIELD, the message dispatcher is a critical integrating force of the underlying
1162 programming environment, handling everything from debugging events to
1163 changing the working directory of a program.
1164 Plumbing, by contrast, is intended primarily for integrating the user interface
1165 of existing tools; it is more modest and very much simpler.
1166 The central advantage of the plumber is its convenience and dynamism;
1167 the FIELD system does not share the ease with which
1168 message dispatch rules can be added or modified.
1170 The inspiration for Acme was
1171 the user interface to the object-oriented Oberon system [WiGu92].
1172 Oberon's user interface interprets mouse clicks on strings such as
1174 to invoke calls to the method
1178 This was the starting point for Acme's middle-button execution [Pike94],
1179 but nothing in Oberon is much like Acme's right-button `acquisition',
1180 which was the starting point for the plumber.
1181 Oberon's implicit method-based linking is not nearly as general as the pattern-matched
1182 linking of the plumber, nor does its style of user-triggered method call
1183 correspond well to the more general idea of inter-application communication
1184 of plumbing messages.
1186 Microsoft's OLE interface is another relative.
1187 It allows one application to
1189 its own data within another's,
1190 for example to place an Excel spreadsheet within a Frame document;
1191 when Frame needs to format the page, it will start Excel itself, or at least some of its
1192 DLLs, to format the spreadsheet.
1193 OLE data can only be understood by the application that created it;
1194 plumbing messages, by contrast, contain arbitrary data with a rigidly formatted header
1195 that will be interpreted by the pattern matcher and the destination application.
1196 The plumber's simplified message format may limit its
1197 flexibility but makes messages easy and efficient to dispatch and to interpret.
1198 At least for the cut-and-paste style of exchange OLE encourages,
1199 plumbing gives up some power in return for simplicity, while avoiding
1200 the need to invoke a vestigial program (if Excel can be called a vestige) every time
1201 the pasted data is examined.
1202 Plumbing is also better suited to
1203 other styles of data exchange, such as connecting compiler errors to the
1206 The Hyperbole [Wein] package for Emacs adds hypertext facilities to existing documents.
1207 It includes explicit links and, like plumbing, a rule-driven way to form implicit links.
1208 Since Emacs is purely textual, like Acme, Hyperbole does not easily extend to driving
1209 graphical applications, nor does it provide a general interprocess communication method.
1210 For instance, although Hyperbole provides some integration for mail applications,
1211 it cannot provide the glue that allows a click on a face icon in an external program to open a
1212 mail message within the viewer.
1213 Moreover, since it is not implemented as a file server,
1214 Hyperbole does not share the advantages of that architecture.
1218 program in 4BSD echoes a small but common use of plumbing.
1219 It takes the error messages produced by a compiler and drives a text editor
1220 through the steps of looking at each one in turn; the notion is to quicken the
1221 compile/edit/debug cycle.
1222 Similar results are achieved in EMACS by writing special M-LISP
1223 macros to parse the error messages from various compilers.
1224 Although for this particular purpose they may be more convenient than plumbing,
1225 these are specific solutions to a specific problem and lack plumbing's generality.
1227 Of course, the resource forks in MacOS and the association rules for
1228 file name extensions in Windows also provide some of the functionality of
1229 the plumber, although again without the generality or dynamic nature.
1231 Closer to home, Ousterhout's Tcl (Tool Command Language) [Oust90]
1232 was originally designed to embed a little command interpreter
1233 in each application to control interprocess communication and
1234 provide a level of integration.
1235 Plumbing, on the other hand, provides minimal support within
1236 the application, offloading most of the message handling and all the
1237 command execution to the central plumber.
1239 The most obvious relative to plumbing is perhaps the hypertext links of a web browser.
1240 Plumbing differs by synthesizing
1241 the links on demand.
1242 Rather than constructing links within a document as in HTML,
1243 plumbing uses the context of a button click to derive what it should link to.
1244 That the rules for this decision can be modified dynamically gives it a more
1245 fluid feel than a standard web browsing world.
1246 One possibility for future work is to adapt a web browser to use
1247 plumbing as its link-following engine, much as Acme used plumbing to offload
1248 its acquisition rules.
1249 This would connect the web browser to the existing tools, rather than the
1250 current trend in most systems of replacing the tools by a browser.
1252 Each of these prior systems\(emand there are others, e.g. [Pasa93, Free93]\(emaddresses
1253 a particular need or subset of the
1254 issues of system integration.
1255 Plumbing differs because its particular choices were different.
1256 It focuses on two key issues:
1257 centralizing and automating the handling of interprocess communication
1258 among interactive programs,
1259 and maximizing the convenience (or minimizing the trouble) for the human user
1261 Moreover, the plumber's implementation as a file server, with messages
1262 passed over files it controls,
1263 permits the architecture to work transparently across a network.
1264 None of the other systems discussed here integrates distributed systems
1265 as smoothly as local ones without the addition of significant extra technology.
1269 There were a few surprises during the development of plumbing.
1270 The first version of plumbing was done for the Inferno system [Dorw97a,Dorw97b],
1271 using its file-to-channel mechanism to mediate the IPC.
1272 Although it was very simple to build, it encountered difficulties because
1273 the plumber was too disconnected from its clients; in particular, there was
1274 no way to discover whether a port was in use.
1275 When plumbing was implemented afresh for Plan 9, it was provided through a true file server.
1276 Although this was much more work, it paid off handsomely.
1277 The plumber now knows whether a port is open, which makes it easy to decide whether
1278 a new program must be started to handle a message,
1279 and the ability to edit the rules file dynamically is a major advantage.
1280 Other advantages arise from the file-server design,
1282 the ease of exporting plumbing ports across the network to remote machines
1283 and the implicit security model a file-based interface provides: no one has
1284 permission to open my private plumbing files.
1286 On the other hand, Inferno was an all-new environment and the user interface for plumbing was
1287 able to be made uniform for all applications.
1288 This was impractical for Plan 9, so more
1290 interfaces had to be provided for that environment.
1291 Yet even in Plan 9 the advantages of efficient,
1292 convenient, dynamic interprocess communication outweigh the variability of
1294 In fact, it is perhaps a telling point that the system works well for a variety of interfaces;
1295 the provision of a central, convenient message-passing
1296 service is a good idea regardless of how the programs use it.
1298 Plumbing's rule language uses only regular expressions and a few special
1302 There is much more that could be done. For example, in the current system a JPEG
1303 file can be recognized by a
1305 suffix but not by its contents, since the plumbing language has no facility
1308 of files named in its messages.
1309 To address this issue without adding more special rules requires rethinking
1310 the language itself.
1311 Although the current system seems a good balance of complexity
1313 perhaps a richer, more general-purpose language would
1314 permit more exotic applications of the plumbing model.
1316 In conclusion, plumbing adds an effective, easy-to-use inter-application
1317 communication mechanism to the Plan 9
1319 Its unusual design as a language-driven file server makes it easy to add
1320 context-dependent, dynamically interpreted, general-purpose hyperlinks
1321 to the desktop, for both existing tools and new ones.
1325 Dave Presotto wrote the mail file system and
1327 He, Russ Cox, Sape Mullender, and Cliff Young influenced the design, offered useful suggestions,
1328 and suffered early versions of the software.
1329 They also made helpful comments on this paper, as did Dennis Ritchie and Brian Kernighan.
1334 Sean Dorward, Rob Pike, David Leo Presotto, Dennis M. Ritchie,
1335 Howard W. Trickey, and Philip Winterbottom,
1337 .I "Proceedings of the IEEE Compcon 97 Conference" ,
1338 San Jose, 1997, pp. 241-244.
1341 Sean Dorward, Rob Pike, David Leo Presotto, Dennis M. Ritchie,
1342 Howard W. Trickey, and Philip Winterbottom,
1343 ``The Inferno Operating System'',
1344 .I "Bell Labs Technical Journal" ,
1350 Syslog configuration file manual
1355 ``Processes as Files'',
1356 .I "Proceedings of the Summer 1984 USENIX Conference" ,
1357 Salt Lake City, 1984, pp. 203-207.
1361 ``Tcl: An Embeddable Command Languages'',
1362 .I "Proceedings of the Winter 1990 USENIX Conference" ,
1363 Washington, 1990, pp. 133-146.
1366 Vern Paxson and Chris Saltmarsh,
1367 "Glish: A User-Level Software Bus for Loosely-Coupled Distributed Systems" ,
1368 .I "Proceedings of the Winter 1993 USENIX Conference" ,
1369 San Diego, 1993, pp. 141-155.
1373 ``Structural Regular Expressions'',
1374 .I "EUUG Spring 1987 Conference Proceedings" ,
1375 Helsinki, May 1987, pp. 21-28.
1379 ``The Text Editor sam'',
1380 .I "Software - Practice and Experience" ,
1382 5, Nov. 1987, pp. 813-845.
1386 ``8½, the Plan 9 Window System'',
1387 .I "Proceedings of the Summer 1991 USENIX Conference" ,
1388 Nashville, 1991, pp. 257-265.
1391 Rob Pike, Dave Presotto, Ken Thompson, Howard Trickey, and Phil Winterbottom,
1392 ``The Use of Name Spaces in Plan 9'',
1393 .I "Operating Systems Review" ,
1395 2, April 1993, pp. 72-76.
1399 ``Acme: A User Interface for Programmers'',
1400 .I "Proceedings of the Winter 1994 USENIX Conference",
1401 San Francisco, 1994, pp. 223-234.
1404 Rob Pike and Dave Presotto,
1405 ``Face the Nation'',
1406 .I "Proceedings of the USENIX Summer 1985 Conference" ,
1407 Portland, 1985, pg. 81.
1411 .I "The FIELD Programming Environment: A Friendly Integrated Environment for Learning and Development" ,
1412 Kluwer, Boston, 1995.
1416 .I "Hyperbole User Manual" ,
1417 .CW http://www.cs.indiana.edu/elisp/hyperbole/hyperbole_1.html
1420 Philip Winterbottom,
1421 ``ACID: A Debugger based on a Language'',
1422 .I "Proceedings of the USENIX Winter Conference" ,
1423 San Francisco, CA, 1994.
1426 Niklaus Wirth and Jurg Gutknecht,
1427 .I "Project Oberon: The Design of an Operating System and Compilers" ,
1428 Addison-Wesley, Reading, 1992.
projects / plan9front.git / blob