1 // Copyright 2016 The Rust Project Developers. See the COPYRIGHT
2 // file at the top-level directory of this distribution and at
3 // http://rust-lang.org/COPYRIGHT.
5 // Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
6 // http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
7 // <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your
8 // option. This file may not be copied, modified, or distributed
9 // except according to those terms.
12 //! This module manages how the incremental compilation cache is represented in
15 //! Incremental compilation caches are managed according to a copy-on-write
16 //! strategy: Once a complete, consistent cache version is finalized, it is
17 //! never modified. Instead, when a subsequent compilation session is started,
18 //! the compiler will allocate a new version of the cache that starts out as
19 //! a copy of the previous version. Then only this new copy is modified and it
20 //! will not be visible to other processes until it is finalized. This ensures
21 //! that multiple compiler processes can be executed concurrently for the same
22 //! crate without interfering with each other or blocking each other.
24 //! More concretely this is implemented via the following protocol:
26 //! 1. For a newly started compilation session, the compiler allocates a
27 //! new `session` directory within the incremental compilation directory.
28 //! This session directory will have a unique name that ends with the suffix
29 //! "-working" and that contains a creation timestamp.
30 //! 2. Next, the compiler looks for the newest finalized session directory,
31 //! that is, a session directory from a previous compilation session that
32 //! has been marked as valid and consistent. A session directory is
33 //! considered finalized if the "-working" suffix in the directory name has
34 //! been replaced by the SVH of the crate.
35 //! 3. Once the compiler has found a valid, finalized session directory, it will
36 //! hard-link/copy its contents into the new "-working" directory. If all
37 //! goes well, it will have its own, private copy of the source directory and
38 //! subsequently not have to worry about synchronizing with other compiler
40 //! 4. Now the compiler can do its normal compilation process, which involves
41 //! reading and updating its private session directory.
42 //! 5. When compilation finishes without errors, the private session directory
43 //! will be in a state where it can be used as input for other compilation
44 //! sessions. That is, it will contain a dependency graph and cache artifacts
45 //! that are consistent with the state of the source code it was compiled
46 //! from, with no need to change them ever again. At this point, the compiler
47 //! finalizes and "publishes" its private session directory by renaming it
48 //! from "s-{timestamp}-{random}-working" to "s-{timestamp}-{SVH}".
49 //! 6. At this point the "old" session directory that we copied our data from
50 //! at the beginning of the session has become obsolete because we have just
51 //! published a more current version. Thus the compiler will delete it.
53 //! ## Garbage Collection
55 //! Naively following the above protocol might lead to old session directories
56 //! piling up if a compiler instance crashes for some reason before its able to
57 //! remove its private session directory. In order to avoid wasting disk space,
58 //! the compiler also does some garbage collection each time it is started in
59 //! incremental compilation mode. Specifically, it will scan the incremental
60 //! compilation directory for private session directories that are not in use
61 //! any more and will delete those. It will also delete any finalized session
62 //! directories for a given crate except for the most recent one.
64 //! ## Synchronization
66 //! There is some synchronization needed in order for the compiler to be able to
67 //! determine whether a given private session directory is not in used any more.
68 //! This is done by creating a lock file for each session directory and
69 //! locking it while the directory is still being used. Since file locks have
70 //! operating system support, we can rely on the lock being released if the
71 //! compiler process dies for some unexpected reason. Thus, when garbage
72 //! collecting private session directories, the collecting process can determine
73 //! whether the directory is still in use by trying to acquire a lock on the
74 //! file. If locking the file fails, the original process must still be alive.
75 //! If locking the file succeeds, we know that the owning process is not alive
76 //! any more and we can safely delete the directory.
77 //! There is still a small time window between the original process creating the
78 //! lock file and actually locking it. In order to minimize the chance that
79 //! another process tries to acquire the lock in just that instance, only
80 //! session directories that are older than a few seconds are considered for
81 //! garbage collection.
83 //! Another case that has to be considered is what happens if one process
84 //! deletes a finalized session directory that another process is currently
85 //! trying to copy from. This case is also handled via the lock file. Before
86 //! a process starts copying a finalized session directory, it will acquire a
87 //! shared lock on the directory's lock file. Any garbage collecting process,
88 //! on the other hand, will acquire an exclusive lock on the lock file.
89 //! Thus, if a directory is being collected, any reader process will fail
90 //! acquiring the shared lock and will leave the directory alone. Conversely,
91 //! if a collecting process can't acquire the exclusive lock because the
92 //! directory is currently being read from, it will leave collecting that
93 //! directory to another process at a later point in time.
94 //! The exact same scheme is also used when reading the metadata hashes file
95 //! from an extern crate. When a crate is compiled, the hash values of its
96 //! metadata are stored in a file in its session directory. When the
97 //! compilation session of another crate imports the first crate's metadata,
98 //! it also has to read in the accompanying metadata hashes. It thus will access
99 //! the finalized session directory of all crates it links to and while doing
100 //! so, it will also place a read lock on that the respective session directory
101 //! so that it won't be deleted while the metadata hashes are loaded.
105 //! This system relies on two features being available in the file system in
106 //! order to work really well: file locking and hard linking.
107 //! If hard linking is not available (like on FAT) the data in the cache
108 //! actually has to be copied at the beginning of each session.
109 //! If file locking does not work reliably (like on NFS), some of the
110 //! synchronization will go haywire.
111 //! In both cases we recommend to locate the incremental compilation directory
112 //! on a file system that supports these things.
113 //! It might be a good idea though to try and detect whether we are on an
114 //! unsupported file system and emit a warning in that case. This is not yet
117 use rustc::hir::def_id::CrateNum;
118 use rustc::hir::svh::Svh;
119 use rustc::session::Session;
120 use rustc::ty::TyCtxt;
121 use rustc::util::fs as fs_util;
122 use rustc_data_structures::{flock, base_n};
123 use rustc_data_structures::fx::{FxHashSet, FxHashMap};
125 use std::ffi::OsString;
126 use std::fs as std_fs;
129 use std::path::{Path, PathBuf};
130 use std::time::{UNIX_EPOCH, SystemTime, Duration};
131 use std::__rand::{thread_rng, Rng};
133 const LOCK_FILE_EXT: &'static str = ".lock";
134 const DEP_GRAPH_FILENAME: &'static str = "dep-graph.bin";
135 const WORK_PRODUCTS_FILENAME: &'static str = "work-products.bin";
136 const METADATA_HASHES_FILENAME: &'static str = "metadata.bin";
138 // We encode integers using the following base, so they are shorter than decimal
139 // or hexadecimal numbers (we want short file and directory names). Since these
140 // numbers will be used in file names, we choose an encoding that is not
141 // case-sensitive (as opposed to base64, for example).
142 const INT_ENCODE_BASE: u64 = 36;
144 pub fn dep_graph_path(sess: &Session) -> PathBuf {
145 in_incr_comp_dir_sess(sess, DEP_GRAPH_FILENAME)
148 pub fn work_products_path(sess: &Session) -> PathBuf {
149 in_incr_comp_dir_sess(sess, WORK_PRODUCTS_FILENAME)
152 pub fn metadata_hash_export_path(sess: &Session) -> PathBuf {
153 in_incr_comp_dir_sess(sess, METADATA_HASHES_FILENAME)
156 pub fn metadata_hash_import_path(import_session_dir: &Path) -> PathBuf {
157 import_session_dir.join(METADATA_HASHES_FILENAME)
160 pub fn lock_file_path(session_dir: &Path) -> PathBuf {
161 let crate_dir = session_dir.parent().unwrap();
163 let directory_name = session_dir.file_name().unwrap().to_string_lossy();
164 assert_no_characters_lost(&directory_name);
166 let dash_indices: Vec<_> = directory_name.match_indices("-")
169 if dash_indices.len() != 3 {
170 bug!("Encountered incremental compilation session directory with \
172 session_dir.display())
175 crate_dir.join(&directory_name[0 .. dash_indices[2]])
176 .with_extension(&LOCK_FILE_EXT[1..])
179 pub fn in_incr_comp_dir_sess(sess: &Session, file_name: &str) -> PathBuf {
180 in_incr_comp_dir(&sess.incr_comp_session_dir(), file_name)
183 pub fn in_incr_comp_dir(incr_comp_session_dir: &Path, file_name: &str) -> PathBuf {
184 incr_comp_session_dir.join(file_name)
187 /// Allocates the private session directory. The boolean in the Ok() result
188 /// indicates whether we should try loading a dep graph from the successfully
189 /// initialized directory, or not.
190 /// The post-condition of this fn is that we have a valid incremental
191 /// compilation session directory, if the result is `Ok`. A valid session
192 /// directory is one that contains a locked lock file. It may or may not contain
193 /// a dep-graph and work products from a previous session.
194 /// If the call fails, the fn may leave behind an invalid session directory.
195 /// The garbage collection will take care of it.
196 pub fn prepare_session_directory(sess: &Session,
198 crate_disambiguator: &str) {
199 if sess.opts.incremental.is_none() {
203 debug!("prepare_session_directory");
205 // {incr-comp-dir}/{crate-name-and-disambiguator}
206 let crate_dir = crate_path(sess, crate_name, crate_disambiguator);
207 debug!("crate-dir: {}", crate_dir.display());
208 if create_dir(sess, &crate_dir, "crate").is_err() {
212 // Hack: canonicalize the path *after creating the directory*
213 // because, on windows, long paths can cause problems;
214 // canonicalization inserts this weird prefix that makes windows
215 // tolerate long paths.
216 let crate_dir = match crate_dir.canonicalize() {
219 sess.err(&format!("incremental compilation: error canonicalizing path `{}`: {}",
220 crate_dir.display(), err));
225 let mut source_directories_already_tried = FxHashSet();
228 // Generate a session directory of the form:
230 // {incr-comp-dir}/{crate-name-and-disambiguator}/s-{timestamp}-{random}-working
231 let session_dir = generate_session_dir_path(&crate_dir);
232 debug!("session-dir: {}", session_dir.display());
234 // Lock the new session directory. If this fails, return an
235 // error without retrying
236 let (directory_lock, lock_file_path) = match lock_directory(sess, &session_dir) {
241 // Now that we have the lock, we can actually create the session
243 if create_dir(sess, &session_dir, "session").is_err() {
247 // Find a suitable source directory to copy from. Ignore those that we
248 // have already tried before.
249 let source_directory = find_source_directory(&crate_dir,
250 &source_directories_already_tried);
252 let source_directory = if let Some(dir) = source_directory {
255 // There's nowhere to copy from, we're done
256 debug!("no source directory found. Continuing with empty session \
259 sess.init_incr_comp_session(session_dir, directory_lock, false);
263 debug!("attempting to copy data from source: {}",
264 source_directory.display());
266 let print_file_copy_stats = sess.opts.debugging_opts.incremental_info;
268 // Try copying over all files from the source directory
269 if let Ok(allows_links) = copy_files(&session_dir, &source_directory,
270 print_file_copy_stats) {
271 debug!("successfully copied data from: {}",
272 source_directory.display());
275 sess.warn(&format!("Hard linking files in the incremental \
276 compilation cache failed. Copying files \
277 instead. Consider moving the cache \
278 directory to a file system which supports \
279 hard linking in session dir `{}`",
280 session_dir.display())
284 sess.init_incr_comp_session(session_dir, directory_lock, true);
287 debug!("copying failed - trying next directory");
289 // Something went wrong while trying to copy/link files from the
290 // source directory. Try again with a different one.
291 source_directories_already_tried.insert(source_directory);
293 // Try to remove the session directory we just allocated. We don't
294 // know if there's any garbage in it from the failed copy action.
295 if let Err(err) = safe_remove_dir_all(&session_dir) {
296 sess.warn(&format!("Failed to delete partly initialized \
297 session dir `{}`: {}",
298 session_dir.display(),
302 delete_session_dir_lock_file(sess, &lock_file_path);
303 mem::drop(directory_lock);
309 /// This function finalizes and thus 'publishes' the session directory by
310 /// renaming it to `s-{timestamp}-{svh}` and releasing the file lock.
311 /// If there have been compilation errors, however, this function will just
312 /// delete the presumably invalid session directory.
313 pub fn finalize_session_directory(sess: &Session, svh: Svh) {
314 if sess.opts.incremental.is_none() {
318 let incr_comp_session_dir: PathBuf = sess.incr_comp_session_dir().clone();
320 if sess.has_errors() {
321 // If there have been any errors during compilation, we don't want to
322 // publish this session directory. Rather, we'll just delete it.
324 debug!("finalize_session_directory() - invalidating session directory: {}",
325 incr_comp_session_dir.display());
327 if let Err(err) = safe_remove_dir_all(&*incr_comp_session_dir) {
328 sess.warn(&format!("Error deleting incremental compilation \
329 session directory `{}`: {}",
330 incr_comp_session_dir.display(),
334 let lock_file_path = lock_file_path(&*incr_comp_session_dir);
335 delete_session_dir_lock_file(sess, &lock_file_path);
336 sess.mark_incr_comp_session_as_invalid();
339 debug!("finalize_session_directory() - session directory: {}",
340 incr_comp_session_dir.display());
342 let old_sub_dir_name = incr_comp_session_dir.file_name()
345 assert_no_characters_lost(&old_sub_dir_name);
347 // Keep the 's-{timestamp}-{random-number}' prefix, but replace the
348 // '-working' part with the SVH of the crate
349 let dash_indices: Vec<_> = old_sub_dir_name.match_indices("-")
352 if dash_indices.len() != 3 {
353 bug!("Encountered incremental compilation session directory with \
355 incr_comp_session_dir.display())
358 // State: "s-{timestamp}-{random-number}-"
359 let mut new_sub_dir_name = String::from(&old_sub_dir_name[.. dash_indices[2] + 1]);
362 base_n::push_str(svh.as_u64(), INT_ENCODE_BASE, &mut new_sub_dir_name);
364 // Create the full path
365 let new_path = incr_comp_session_dir.parent().unwrap().join(new_sub_dir_name);
366 debug!("finalize_session_directory() - new path: {}", new_path.display());
368 match std_fs::rename(&*incr_comp_session_dir, &new_path) {
370 debug!("finalize_session_directory() - directory renamed successfully");
372 // This unlocks the directory
373 sess.finalize_incr_comp_session(new_path);
376 // Warn about the error. However, no need to abort compilation now.
377 sess.warn(&format!("Error finalizing incremental compilation \
378 session directory `{}`: {}",
379 incr_comp_session_dir.display(),
382 debug!("finalize_session_directory() - error, marking as invalid");
383 // Drop the file lock, so we can garage collect
384 sess.mark_incr_comp_session_as_invalid();
388 let _ = garbage_collect_session_directories(sess);
391 pub fn delete_all_session_dir_contents(sess: &Session) -> io::Result<()> {
392 let sess_dir_iterator = sess.incr_comp_session_dir().read_dir()?;
393 for entry in sess_dir_iterator {
395 safe_remove_file(&entry.path())?
400 fn copy_files(target_dir: &Path,
402 print_stats_on_success: bool)
403 -> Result<bool, ()> {
404 // We acquire a shared lock on the lock file of the directory, so that
405 // nobody deletes it out from under us while we are reading from it.
406 let lock_file_path = lock_file_path(source_dir);
407 let _lock = if let Ok(lock) = flock::Lock::new(&lock_file_path,
408 false, // don't wait,
409 false, // don't create
410 false) { // not exclusive
413 // Could not acquire the lock, don't try to copy from here
417 let source_dir_iterator = match source_dir.read_dir() {
419 Err(_) => return Err(())
422 let mut files_linked = 0;
423 let mut files_copied = 0;
425 for entry in source_dir_iterator {
428 let file_name = entry.file_name();
430 let target_file_path = target_dir.join(file_name);
431 let source_path = entry.path();
433 debug!("copying into session dir: {}", source_path.display());
434 match fs_util::link_or_copy(source_path, target_file_path) {
435 Ok(fs_util::LinkOrCopy::Link) => {
438 Ok(fs_util::LinkOrCopy::Copy) => {
441 Err(_) => return Err(())
450 if print_stats_on_success {
451 eprintln!("incremental: session directory: {} files hard-linked", files_linked);
452 eprintln!("incremental: session directory: {} files copied", files_copied);
455 Ok(files_linked > 0 || files_copied == 0)
458 /// Generate unique directory path of the form:
459 /// {crate_dir}/s-{timestamp}-{random-number}-working
460 fn generate_session_dir_path(crate_dir: &Path) -> PathBuf {
461 let timestamp = timestamp_to_string(SystemTime::now());
462 debug!("generate_session_dir_path: timestamp = {}", timestamp);
463 let random_number = thread_rng().next_u32();
464 debug!("generate_session_dir_path: random_number = {}", random_number);
466 let directory_name = format!("s-{}-{}-working",
468 base_n::encode(random_number as u64,
470 debug!("generate_session_dir_path: directory_name = {}", directory_name);
471 let directory_path = crate_dir.join(directory_name);
472 debug!("generate_session_dir_path: directory_path = {}", directory_path.display());
476 fn create_dir(sess: &Session, path: &Path, dir_tag: &str) -> Result<(),()> {
477 match std_fs::create_dir_all(path) {
479 debug!("{} directory created successfully", dir_tag);
483 sess.err(&format!("Could not create incremental compilation {} \
493 /// Allocate a the lock-file and lock it.
494 fn lock_directory(sess: &Session,
496 -> Result<(flock::Lock, PathBuf), ()> {
497 let lock_file_path = lock_file_path(session_dir);
498 debug!("lock_directory() - lock_file: {}", lock_file_path.display());
500 match flock::Lock::new(&lock_file_path,
502 true, // create the lock file
503 true) { // the lock should be exclusive
504 Ok(lock) => Ok((lock, lock_file_path)),
506 sess.err(&format!("incremental compilation: could not create \
507 session directory lock file: {}", err));
513 fn delete_session_dir_lock_file(sess: &Session,
514 lock_file_path: &Path) {
515 if let Err(err) = safe_remove_file(&lock_file_path) {
516 sess.warn(&format!("Error deleting lock file for incremental \
517 compilation session directory `{}`: {}",
518 lock_file_path.display(),
523 /// Find the most recent published session directory that is not in the
525 fn find_source_directory(crate_dir: &Path,
526 source_directories_already_tried: &FxHashSet<PathBuf>)
528 let iter = crate_dir.read_dir()
530 .filter_map(|e| e.ok().map(|e| e.path()));
532 find_source_directory_in_iter(iter, source_directories_already_tried)
535 fn find_source_directory_in_iter<I>(iter: I,
536 source_directories_already_tried: &FxHashSet<PathBuf>)
538 where I: Iterator<Item=PathBuf>
540 let mut best_candidate = (UNIX_EPOCH, None);
542 for session_dir in iter {
543 debug!("find_source_directory_in_iter - inspecting `{}`",
544 session_dir.display());
546 let directory_name = session_dir.file_name().unwrap().to_string_lossy();
547 assert_no_characters_lost(&directory_name);
549 if source_directories_already_tried.contains(&session_dir) ||
550 !is_session_directory(&directory_name) ||
551 !is_finalized(&directory_name) {
552 debug!("find_source_directory_in_iter - ignoring.");
556 let timestamp = extract_timestamp_from_session_dir(&directory_name)
557 .unwrap_or_else(|_| {
558 bug!("unexpected incr-comp session dir: {}", session_dir.display())
561 if timestamp > best_candidate.0 {
562 best_candidate = (timestamp, Some(session_dir.clone()));
569 fn is_finalized(directory_name: &str) -> bool {
570 !directory_name.ends_with("-working")
573 fn is_session_directory(directory_name: &str) -> bool {
574 directory_name.starts_with("s-") &&
575 !directory_name.ends_with(LOCK_FILE_EXT)
578 fn is_session_directory_lock_file(file_name: &str) -> bool {
579 file_name.starts_with("s-") && file_name.ends_with(LOCK_FILE_EXT)
582 fn extract_timestamp_from_session_dir(directory_name: &str)
583 -> Result<SystemTime, ()> {
584 if !is_session_directory(directory_name) {
588 let dash_indices: Vec<_> = directory_name.match_indices("-")
591 if dash_indices.len() != 3 {
595 string_to_timestamp(&directory_name[dash_indices[0]+1 .. dash_indices[1]])
598 fn timestamp_to_string(timestamp: SystemTime) -> String {
599 let duration = timestamp.duration_since(UNIX_EPOCH).unwrap();
600 let micros = duration.as_secs() * 1_000_000 +
601 (duration.subsec_nanos() as u64) / 1000;
602 base_n::encode(micros, INT_ENCODE_BASE)
605 fn string_to_timestamp(s: &str) -> Result<SystemTime, ()> {
606 let micros_since_unix_epoch = u64::from_str_radix(s, 36);
608 if micros_since_unix_epoch.is_err() {
612 let micros_since_unix_epoch = micros_since_unix_epoch.unwrap();
614 let duration = Duration::new(micros_since_unix_epoch / 1_000_000,
615 1000 * (micros_since_unix_epoch % 1_000_000) as u32);
616 Ok(UNIX_EPOCH + duration)
619 fn crate_path_tcx(tcx: TyCtxt, cnum: CrateNum) -> PathBuf {
620 crate_path(tcx.sess, &tcx.crate_name(cnum).as_str(), &tcx.crate_disambiguator(cnum).as_str())
623 /// Finds the session directory containing the correct metadata hashes file for
624 /// the given crate. In order to do that it has to compute the crate directory
625 /// of the given crate, and in there, look for the session directory with the
626 /// correct SVH in it.
627 /// Note that we have to match on the exact SVH here, not just the
628 /// crate's (name, disambiguator) pair. The metadata hashes are only valid for
629 /// the exact version of the binary we are reading from now (i.e. the hashes
630 /// are part of the dependency graph of a specific compilation session).
631 pub fn find_metadata_hashes_for(tcx: TyCtxt, cnum: CrateNum) -> Option<PathBuf> {
632 let crate_directory = crate_path_tcx(tcx, cnum);
634 if !crate_directory.exists() {
638 let dir_entries = match crate_directory.read_dir() {
639 Ok(dir_entries) => dir_entries,
642 .err(&format!("incremental compilation: Could not read crate directory `{}`: {}",
643 crate_directory.display(), e));
648 let target_svh = tcx.crate_hash(cnum);
649 let target_svh = base_n::encode(target_svh.as_u64(), INT_ENCODE_BASE);
651 let sub_dir = find_metadata_hashes_iter(&target_svh, dir_entries.filter_map(|e| {
652 e.ok().map(|e| e.file_name().to_string_lossy().into_owned())
655 sub_dir.map(|sub_dir_name| crate_directory.join(&sub_dir_name))
658 fn find_metadata_hashes_iter<'a, I>(target_svh: &str, iter: I) -> Option<OsString>
659 where I: Iterator<Item=String>
661 for sub_dir_name in iter {
662 if !is_session_directory(&sub_dir_name) || !is_finalized(&sub_dir_name) {
663 // This is not a usable session directory
667 let is_match = if let Some(last_dash_pos) = sub_dir_name.rfind("-") {
668 let candidate_svh = &sub_dir_name[last_dash_pos + 1 .. ];
669 target_svh == candidate_svh
671 // some kind of invalid directory name
676 return Some(OsString::from(sub_dir_name))
683 fn crate_path(sess: &Session,
685 crate_disambiguator: &str)
687 use std::hash::{Hasher, Hash};
688 use std::collections::hash_map::DefaultHasher;
690 let incr_dir = sess.opts.incremental.as_ref().unwrap().clone();
692 // The full crate disambiguator is really long. A hash of it should be
694 let mut hasher = DefaultHasher::new();
695 crate_disambiguator.hash(&mut hasher);
697 let crate_name = format!("{}-{}",
699 base_n::encode(hasher.finish(), INT_ENCODE_BASE));
700 incr_dir.join(crate_name)
703 fn assert_no_characters_lost(s: &str) {
704 if s.contains('\u{FFFD}') {
705 bug!("Could not losslessly convert '{}'.", s)
709 fn is_old_enough_to_be_collected(timestamp: SystemTime) -> bool {
710 timestamp < SystemTime::now() - Duration::from_secs(10)
713 pub fn garbage_collect_session_directories(sess: &Session) -> io::Result<()> {
714 debug!("garbage_collect_session_directories() - begin");
716 let session_directory = sess.incr_comp_session_dir();
717 debug!("garbage_collect_session_directories() - session directory: {}",
718 session_directory.display());
720 let crate_directory = session_directory.parent().unwrap();
721 debug!("garbage_collect_session_directories() - crate directory: {}",
722 crate_directory.display());
724 // First do a pass over the crate directory, collecting lock files and
725 // session directories
726 let mut session_directories = FxHashSet();
727 let mut lock_files = FxHashSet();
729 for dir_entry in try!(crate_directory.read_dir()) {
730 let dir_entry = match dir_entry {
731 Ok(dir_entry) => dir_entry,
738 let entry_name = dir_entry.file_name();
739 let entry_name = entry_name.to_string_lossy();
741 if is_session_directory_lock_file(&entry_name) {
742 assert_no_characters_lost(&entry_name);
743 lock_files.insert(entry_name.into_owned());
744 } else if is_session_directory(&entry_name) {
745 assert_no_characters_lost(&entry_name);
746 session_directories.insert(entry_name.into_owned());
748 // This is something we don't know, leave it alone
752 // Now map from lock files to session directories
753 let lock_file_to_session_dir: FxHashMap<String, Option<String>> =
754 lock_files.into_iter()
755 .map(|lock_file_name| {
756 assert!(lock_file_name.ends_with(LOCK_FILE_EXT));
757 let dir_prefix_end = lock_file_name.len() - LOCK_FILE_EXT.len();
759 let dir_prefix = &lock_file_name[0 .. dir_prefix_end];
760 session_directories.iter()
761 .find(|dir_name| dir_name.starts_with(dir_prefix))
763 (lock_file_name, session_dir.map(String::clone))
767 // Delete all lock files, that don't have an associated directory. They must
768 // be some kind of leftover
769 for (lock_file_name, directory_name) in &lock_file_to_session_dir {
770 if directory_name.is_none() {
771 let timestamp = match extract_timestamp_from_session_dir(lock_file_name) {
772 Ok(timestamp) => timestamp,
774 debug!("Found lock-file with malformed timestamp: {}",
775 crate_directory.join(&lock_file_name).display());
781 let lock_file_path = crate_directory.join(&**lock_file_name);
783 if is_old_enough_to_be_collected(timestamp) {
784 debug!("garbage_collect_session_directories() - deleting \
785 garbage lock file: {}", lock_file_path.display());
786 delete_session_dir_lock_file(sess, &lock_file_path);
788 debug!("garbage_collect_session_directories() - lock file with \
789 no session dir not old enough to be collected: {}",
790 lock_file_path.display());
795 // Filter out `None` directories
796 let lock_file_to_session_dir: FxHashMap<String, String> =
797 lock_file_to_session_dir.into_iter()
798 .filter_map(|(lock_file_name, directory_name)| {
799 directory_name.map(|n| (lock_file_name, n))
803 let mut deletion_candidates = vec![];
804 let mut definitely_delete = vec![];
806 for (lock_file_name, directory_name) in &lock_file_to_session_dir {
807 debug!("garbage_collect_session_directories() - inspecting: {}",
810 let timestamp = match extract_timestamp_from_session_dir(directory_name) {
811 Ok(timestamp) => timestamp,
813 debug!("Found session-dir with malformed timestamp: {}",
814 crate_directory.join(directory_name).display());
820 if is_finalized(directory_name) {
821 let lock_file_path = crate_directory.join(lock_file_name);
822 match flock::Lock::new(&lock_file_path,
824 false, // don't create the lock-file
825 true) { // get an exclusive lock
827 debug!("garbage_collect_session_directories() - \
828 successfully acquired lock");
829 debug!("garbage_collect_session_directories() - adding \
830 deletion candidate: {}", directory_name);
832 // Note that we are holding on to the lock
833 deletion_candidates.push((timestamp,
834 crate_directory.join(directory_name),
838 debug!("garbage_collect_session_directories() - \
839 not collecting, still in use");
842 } else if is_old_enough_to_be_collected(timestamp) {
843 // When cleaning out "-working" session directories, i.e.
844 // session directories that might still be in use by another
845 // compiler instance, we only look a directories that are
846 // at least ten seconds old. This is supposed to reduce the
847 // chance of deleting a directory in the time window where
848 // the process has allocated the directory but has not yet
849 // acquired the file-lock on it.
851 // Try to acquire the directory lock. If we can't, it
852 // means that the owning process is still alive and we
853 // leave this directory alone.
854 let lock_file_path = crate_directory.join(lock_file_name);
855 match flock::Lock::new(&lock_file_path,
857 false, // don't create the lock-file
858 true) { // get an exclusive lock
860 debug!("garbage_collect_session_directories() - \
861 successfully acquired lock");
863 // Note that we are holding on to the lock
864 definitely_delete.push((crate_directory.join(directory_name),
868 debug!("garbage_collect_session_directories() - \
869 not collecting, still in use");
873 debug!("garbage_collect_session_directories() - not finalized, not \
878 // Delete all but the most recent of the candidates
879 for (path, lock) in all_except_most_recent(deletion_candidates) {
880 debug!("garbage_collect_session_directories() - deleting `{}`",
883 if let Err(err) = safe_remove_dir_all(&path) {
884 sess.warn(&format!("Failed to garbage collect finalized incremental \
885 compilation session directory `{}`: {}",
889 delete_session_dir_lock_file(sess, &lock_file_path(&path));
893 // Let's make it explicit that the file lock is released at this point,
894 // or rather, that we held on to it until here
898 for (path, lock) in definitely_delete {
899 debug!("garbage_collect_session_directories() - deleting `{}`",
902 if let Err(err) = safe_remove_dir_all(&path) {
903 sess.warn(&format!("Failed to garbage collect incremental \
904 compilation session directory `{}`: {}",
908 delete_session_dir_lock_file(sess, &lock_file_path(&path));
911 // Let's make it explicit that the file lock is released at this point,
912 // or rather, that we held on to it until here
919 fn all_except_most_recent(deletion_candidates: Vec<(SystemTime, PathBuf, Option<flock::Lock>)>)
920 -> FxHashMap<PathBuf, Option<flock::Lock>> {
921 let most_recent = deletion_candidates.iter()
922 .map(|&(timestamp, ..)| timestamp)
925 if let Some(most_recent) = most_recent {
926 deletion_candidates.into_iter()
927 .filter(|&(timestamp, ..)| timestamp != most_recent)
928 .map(|(_, path, lock)| (path, lock))
935 /// Since paths of artifacts within session directories can get quite long, we
936 /// need to support deleting files with very long paths. The regular
937 /// WinApi functions only support paths up to 260 characters, however. In order
938 /// to circumvent this limitation, we canonicalize the path of the directory
939 /// before passing it to std::fs::remove_dir_all(). This will convert the path
940 /// into the '\\?\' format, which supports much longer paths.
941 fn safe_remove_dir_all(p: &Path) -> io::Result<()> {
943 let canonicalized = try!(p.canonicalize());
944 std_fs::remove_dir_all(canonicalized)
950 fn safe_remove_file(p: &Path) -> io::Result<()> {
952 let canonicalized = try!(p.canonicalize());
953 std_fs::remove_file(canonicalized)
960 fn test_all_except_most_recent() {
961 assert_eq!(all_except_most_recent(
963 (UNIX_EPOCH + Duration::new(4, 0), PathBuf::from("4"), None),
964 (UNIX_EPOCH + Duration::new(1, 0), PathBuf::from("1"), None),
965 (UNIX_EPOCH + Duration::new(5, 0), PathBuf::from("5"), None),
966 (UNIX_EPOCH + Duration::new(3, 0), PathBuf::from("3"), None),
967 (UNIX_EPOCH + Duration::new(2, 0), PathBuf::from("2"), None),
968 ]).keys().cloned().collect::<FxHashSet<PathBuf>>(),
974 ].into_iter().collect::<FxHashSet<PathBuf>>()
977 assert_eq!(all_except_most_recent(
979 ]).keys().cloned().collect::<FxHashSet<PathBuf>>(),
985 fn test_timestamp_serialization() {
986 for i in 0 .. 1_000u64 {
987 let time = UNIX_EPOCH + Duration::new(i * 1_434_578, (i as u32) * 239_000);
988 let s = timestamp_to_string(time);
989 assert_eq!(Ok(time), string_to_timestamp(&s));
994 fn test_find_source_directory_in_iter() {
995 let already_visited = FxHashSet();
998 assert_eq!(find_source_directory_in_iter(
999 vec![PathBuf::from("crate-dir/s-3234-0000-svh"),
1000 PathBuf::from("crate-dir/s-2234-0000-svh"),
1001 PathBuf::from("crate-dir/s-1234-0000-svh")].into_iter(), &already_visited),
1002 Some(PathBuf::from("crate-dir/s-3234-0000-svh")));
1004 // Filter out "-working"
1005 assert_eq!(find_source_directory_in_iter(
1006 vec![PathBuf::from("crate-dir/s-3234-0000-working"),
1007 PathBuf::from("crate-dir/s-2234-0000-svh"),
1008 PathBuf::from("crate-dir/s-1234-0000-svh")].into_iter(), &already_visited),
1009 Some(PathBuf::from("crate-dir/s-2234-0000-svh")));
1012 assert_eq!(find_source_directory_in_iter(vec![].into_iter(), &already_visited),
1015 // Handle only working
1016 assert_eq!(find_source_directory_in_iter(
1017 vec![PathBuf::from("crate-dir/s-3234-0000-working"),
1018 PathBuf::from("crate-dir/s-2234-0000-working"),
1019 PathBuf::from("crate-dir/s-1234-0000-working")].into_iter(), &already_visited),
1024 fn test_find_metadata_hashes_iter()
1026 assert_eq!(find_metadata_hashes_iter("testsvh2",
1028 String::from("s-timestamp1-testsvh1"),
1029 String::from("s-timestamp2-testsvh2"),
1030 String::from("s-timestamp3-testsvh3"),
1032 Some(OsString::from("s-timestamp2-testsvh2"))
1035 assert_eq!(find_metadata_hashes_iter("testsvh2",
1037 String::from("s-timestamp1-testsvh1"),
1038 String::from("s-timestamp2-testsvh2"),
1039 String::from("invalid-name"),
1041 Some(OsString::from("s-timestamp2-testsvh2"))
1044 assert_eq!(find_metadata_hashes_iter("testsvh2",
1046 String::from("s-timestamp1-testsvh1"),
1047 String::from("s-timestamp2-testsvh2-working"),
1048 String::from("s-timestamp3-testsvh3"),
1053 assert_eq!(find_metadata_hashes_iter("testsvh1",
1055 String::from("s-timestamp1-random1-working"),
1056 String::from("s-timestamp2-random2-working"),
1057 String::from("s-timestamp3-random3-working"),
1062 assert_eq!(find_metadata_hashes_iter("testsvh2",
1064 String::from("timestamp1-testsvh2"),
1065 String::from("timestamp2-testsvh2"),
1066 String::from("timestamp3-testsvh2"),