3 venti \- archival storage server
5 Venti is a block storage server intended for archival data.
6 In a Venti server, the SHA1 hash of a block's contents acts
7 as the block identifier for read and write operations.
8 This approach enforces a write-once policy, preventing
9 accidental or malicious destruction of data. In addition,
10 duplicate copies of a block are coalesced, reducing the
11 consumption of storage and simplifying the implementation
14 This manual page documents the basic concepts of
15 block storage using Venti as well as the Venti network protocol.
18 documents some simple clients.
22 are more complex clients.
25 describes a C library interface for accessing
26 Venti servers and manipulating Venti data structures.
29 describes the programs used to run a Venti server.
32 The SHA1 hash that identifies a block is called its
34 The score of the zero-length block is called the
37 Scores may have an optional
39 prefix, typically used to
40 describe the format of the data.
45 prefix, while vbackup uses prefixes corresponding to the file system
50 .SS "Files and Directories
51 Venti accepts blocks up to 56 kilobytes in size.
52 By convention, Venti clients use hash trees of blocks to
53 represent arbitrary-size data
55 The data to be stored is split into fixed-size
56 blocks and written to the server, producing a list
58 The resulting list of scores is split into fixed-size pointer
59 blocks (using only an integral number of scores per block)
60 and written to the server, producing a smaller list
62 The process continues, eventually ending with the
63 score for the hash tree's top-most block.
64 Each file stored this way is summarized by
67 structure recording the top-most score, the depth
68 of the tree, the data block size, and the pointer block size.
71 structures can be concatenated
72 and stored as a special file called a
75 manner, arbitrary trees of files can be constructed
78 Scores passed between programs conventionally refer
81 blocks, which contain descriptive information
82 as well as the score of a directory block containing a small number
85 Conventionally, programs do not mix data and directory entries
86 in the same file. Instead, they keep two separate files, one with
87 directory entries and one with metadata referencing those
89 Keeping this parallel representation is a minor annoyance
90 but makes it possible for general programs like
94 to traverse the block tree without knowing the specific details
95 of any particular program's data.
97 To allow programs to traverse these structures without
98 needing to understand their higher-level meanings,
99 Venti tags each block with a type. The types are:
103 VtDataType 000 \f1data\fL
104 VtDataType+1 001 \fRscores of \fPVtDataType\fR blocks\fL
105 VtDataType+2 002 \fRscores of \fPVtDataType+1\fR blocks\fL
107 VtDirType 010 VtEntry\fR structures\fL
108 VtDirType+1 011 \fRscores of \fLVtDirType\fR blocks\fL
109 VtDirType+2 012 \fRscores of \fLVtDirType+1\fR blocks\fL
111 VtRootType 020 VtRoot\fR structure\fL
114 The octal numbers listed are the type numbers used
115 by the commands below.
116 (For historical reasons, the type numbers used on
117 disk and on the wire are different from the above.
118 They do not distinguish
124 To avoid storing the same short data blocks padded with
125 differing numbers of zeros, Venti clients working with fixed-size
126 blocks conventionally
127 `zero truncate' the blocks before writing them to the server.
128 For example, if a 1024-byte data block contains the
130 .RB ` hello " " world '
131 followed by 1013 zero bytes,
132 a client would store only the 11-byte block.
133 When the client later read the block from the server,
134 it would append zero bytes to the end as necessary to
135 reach the expected size.
137 When truncating pointer blocks
138 .RB ( VtDataType+ \fIn
142 trailing zero scores are removed
143 instead of trailing zero bytes.
145 Because of the truncation convention,
146 any file consisting entirely of zero bytes,
147 no matter what its length, will be represented by the zero score:
148 the data blocks contain all zeros and are thus truncated
149 to the empty block, and the pointer blocks contain all zero scores
150 and are thus also truncated to the empty block,
151 and so on up the hash tree.
153 A Venti session begins when a
155 connects to the network address served by a Venti
157 the conventional address is
158 .BI tcp! server !venti
162 Both client and server begin by sending a version
164 .BI venti- versions - comment \en \fR.
167 field is a list of acceptable versions separated by
169 The protocol described here is version
171 The client is responsible for choosing a common
172 version and sending it in the
174 message, described below.
176 After the initial version exchange, the client transmits
179 to the server, which subsequently returns
183 The combined act of transmitting (receiving) a request
184 of a particular type, and receiving (transmitting) its reply
189 Each message consists of a sequence of bytes.
190 Two-byte fields hold unsigned integers represented
191 in big-endian order (most significant byte first).
192 Data items of variable lengths are represented by
193 a one-byte field specifying a count,
198 Text strings are represented similarly,
199 using a two-byte count with
200 the text itself stored as a UTF-encoded sequence
201 of Unicode characters (see
207 counts the bytes of UTF data, which include no final
211 character is illegal in text strings in the Venti protocol.
212 The maximum string length in Venti is 1024 bytes.
214 Each Venti message begins with a two-byte size field
215 specifying the length in bytes of the message,
216 not including the length field itself.
217 The next byte is the message type, one of the constants
218 in the enumeration in the include file
220 The next byte is an identifying
222 used to match responses to requests.
223 The remaining bytes are parameters of different sizes.
224 In the message descriptions, the number of bytes in a field
225 is given in brackets after the field name.
230 is not a constant represents a variable-length parameter:
234 bytes of data forming the
250 is the last field in the message represents a
251 variable-length field that comprises all remaining
252 bytes in the message.
254 All Venti RPC messages are prefixed with a field
256 giving the length of the message that follows
260 The message bodies are:
261 .ta \w'\fLVtTgoodbye 'u
324 Each T-message has a one-byte
326 field, chosen and used by the client to identify the message.
327 The server will echo the request's
330 Clients should arrange that no two outstanding
331 messages have the same tag field so that responses
332 can be distinguished.
334 The type of an R-message will either be one greater than
335 the type of the corresponding T-message or
337 indicating that the request failed.
338 In the latter case, the
340 field contains a string describing the reason for failure.
342 Venti connections must begin with a
347 message contains the protocol
349 that the client has chosen to use.
355 could be used to add authentication, encryption,
356 and compression to the Venti session
357 but are currently ignored.
364 response are similarly ignored.
369 fields are intended to be the identity
370 of the client and server but, given the lack of
371 authentication, should be treated only as advisory.
376 transaction during the session.
380 message has no effect and
381 is used mainly for debugging.
382 Servers should respond immediately to pings.
386 message requests a block with the given
396 to convert a block type enumeration value
401 used on disk and in the protocol.
404 field specifies the maximum expected size
408 in the reply is the block's contents.
412 message writes a new block of the given
417 The response includes the
419 to use to read the block,
420 which should be the SHA1 hash of
423 The Venti server may buffer written blocks in memory,
424 waiting until after responding to the
426 message before writing them to
428 The server will delay the response to a
430 message until after all blocks in earlier
432 messages have been written to permanent storage.
436 message ends a session. There is no
440 message, the server terminates up the connection.
446 Sean Quinlan and Sean Dorward,
447 ``Venti: a new approach to archival storage'',
448 .I "Usenix Conference on File and Storage Technologies" ,