1 //! This module manages how the incremental compilation cache is represented in
4 //! Incremental compilation caches are managed according to a copy-on-write
5 //! strategy: Once a complete, consistent cache version is finalized, it is
6 //! never modified. Instead, when a subsequent compilation session is started,
7 //! the compiler will allocate a new version of the cache that starts out as
8 //! a copy of the previous version. Then only this new copy is modified and it
9 //! will not be visible to other processes until it is finalized. This ensures
10 //! that multiple compiler processes can be executed concurrently for the same
11 //! crate without interfering with each other or blocking each other.
13 //! More concretely this is implemented via the following protocol:
15 //! 1. For a newly started compilation session, the compiler allocates a
16 //! new `session` directory within the incremental compilation directory.
17 //! This session directory will have a unique name that ends with the suffix
18 //! "-working" and that contains a creation timestamp.
19 //! 2. Next, the compiler looks for the newest finalized session directory,
20 //! that is, a session directory from a previous compilation session that
21 //! has been marked as valid and consistent. A session directory is
22 //! considered finalized if the "-working" suffix in the directory name has
23 //! been replaced by the SVH of the crate.
24 //! 3. Once the compiler has found a valid, finalized session directory, it will
25 //! hard-link/copy its contents into the new "-working" directory. If all
26 //! goes well, it will have its own, private copy of the source directory and
27 //! subsequently not have to worry about synchronizing with other compiler
29 //! 4. Now the compiler can do its normal compilation process, which involves
30 //! reading and updating its private session directory.
31 //! 5. When compilation finishes without errors, the private session directory
32 //! will be in a state where it can be used as input for other compilation
33 //! sessions. That is, it will contain a dependency graph and cache artifacts
34 //! that are consistent with the state of the source code it was compiled
35 //! from, with no need to change them ever again. At this point, the compiler
36 //! finalizes and "publishes" its private session directory by renaming it
37 //! from "s-{timestamp}-{random}-working" to "s-{timestamp}-{SVH}".
38 //! 6. At this point the "old" session directory that we copied our data from
39 //! at the beginning of the session has become obsolete because we have just
40 //! published a more current version. Thus the compiler will delete it.
42 //! ## Garbage Collection
44 //! Naively following the above protocol might lead to old session directories
45 //! piling up if a compiler instance crashes for some reason before its able to
46 //! remove its private session directory. In order to avoid wasting disk space,
47 //! the compiler also does some garbage collection each time it is started in
48 //! incremental compilation mode. Specifically, it will scan the incremental
49 //! compilation directory for private session directories that are not in use
50 //! any more and will delete those. It will also delete any finalized session
51 //! directories for a given crate except for the most recent one.
53 //! ## Synchronization
55 //! There is some synchronization needed in order for the compiler to be able to
56 //! determine whether a given private session directory is not in used any more.
57 //! This is done by creating a lock file for each session directory and
58 //! locking it while the directory is still being used. Since file locks have
59 //! operating system support, we can rely on the lock being released if the
60 //! compiler process dies for some unexpected reason. Thus, when garbage
61 //! collecting private session directories, the collecting process can determine
62 //! whether the directory is still in use by trying to acquire a lock on the
63 //! file. If locking the file fails, the original process must still be alive.
64 //! If locking the file succeeds, we know that the owning process is not alive
65 //! any more and we can safely delete the directory.
66 //! There is still a small time window between the original process creating the
67 //! lock file and actually locking it. In order to minimize the chance that
68 //! another process tries to acquire the lock in just that instance, only
69 //! session directories that are older than a few seconds are considered for
70 //! garbage collection.
72 //! Another case that has to be considered is what happens if one process
73 //! deletes a finalized session directory that another process is currently
74 //! trying to copy from. This case is also handled via the lock file. Before
75 //! a process starts copying a finalized session directory, it will acquire a
76 //! shared lock on the directory's lock file. Any garbage collecting process,
77 //! on the other hand, will acquire an exclusive lock on the lock file.
78 //! Thus, if a directory is being collected, any reader process will fail
79 //! acquiring the shared lock and will leave the directory alone. Conversely,
80 //! if a collecting process can't acquire the exclusive lock because the
81 //! directory is currently being read from, it will leave collecting that
82 //! directory to another process at a later point in time.
83 //! The exact same scheme is also used when reading the metadata hashes file
84 //! from an extern crate. When a crate is compiled, the hash values of its
85 //! metadata are stored in a file in its session directory. When the
86 //! compilation session of another crate imports the first crate's metadata,
87 //! it also has to read in the accompanying metadata hashes. It thus will access
88 //! the finalized session directory of all crates it links to and while doing
89 //! so, it will also place a read lock on that the respective session directory
90 //! so that it won't be deleted while the metadata hashes are loaded.
94 //! This system relies on two features being available in the file system in
95 //! order to work really well: file locking and hard linking.
96 //! If hard linking is not available (like on FAT) the data in the cache
97 //! actually has to be copied at the beginning of each session.
98 //! If file locking does not work reliably (like on NFS), some of the
99 //! synchronization will go haywire.
100 //! In both cases we recommend to locate the incremental compilation directory
101 //! on a file system that supports these things.
102 //! It might be a good idea though to try and detect whether we are on an
103 //! unsupported file system and emit a warning in that case. This is not yet
106 use rustc_data_structures::fx::{FxHashMap, FxHashSet};
107 use rustc_data_structures::svh::Svh;
108 use rustc_data_structures::{base_n, flock};
109 use rustc_errors::ErrorReported;
110 use rustc_fs_util::{link_or_copy, LinkOrCopy};
111 use rustc_session::{CrateDisambiguator, Session};
113 use std::fs as std_fs;
116 use std::path::{Path, PathBuf};
117 use std::time::{Duration, SystemTime, UNIX_EPOCH};
119 use rand::{thread_rng, RngCore};
124 const LOCK_FILE_EXT: &str = ".lock";
125 const DEP_GRAPH_FILENAME: &str = "dep-graph.bin";
126 const STAGING_DEP_GRAPH_FILENAME: &str = "dep-graph.part.bin";
127 const WORK_PRODUCTS_FILENAME: &str = "work-products.bin";
128 const QUERY_CACHE_FILENAME: &str = "query-cache.bin";
130 // We encode integers using the following base, so they are shorter than decimal
131 // or hexadecimal numbers (we want short file and directory names). Since these
132 // numbers will be used in file names, we choose an encoding that is not
133 // case-sensitive (as opposed to base64, for example).
134 const INT_ENCODE_BASE: usize = base_n::CASE_INSENSITIVE;
136 pub fn dep_graph_path(sess: &Session) -> PathBuf {
137 in_incr_comp_dir_sess(sess, DEP_GRAPH_FILENAME)
139 pub fn staging_dep_graph_path(sess: &Session) -> PathBuf {
140 in_incr_comp_dir_sess(sess, STAGING_DEP_GRAPH_FILENAME)
142 pub fn dep_graph_path_from(incr_comp_session_dir: &Path) -> PathBuf {
143 in_incr_comp_dir(incr_comp_session_dir, DEP_GRAPH_FILENAME)
146 pub fn work_products_path(sess: &Session) -> PathBuf {
147 in_incr_comp_dir_sess(sess, WORK_PRODUCTS_FILENAME)
150 pub fn query_cache_path(sess: &Session) -> PathBuf {
151 in_incr_comp_dir_sess(sess, QUERY_CACHE_FILENAME)
154 pub fn lock_file_path(session_dir: &Path) -> PathBuf {
155 let crate_dir = session_dir.parent().unwrap();
157 let directory_name = session_dir.file_name().unwrap().to_string_lossy();
158 assert_no_characters_lost(&directory_name);
160 let dash_indices: Vec<_> = directory_name.match_indices('-').map(|(idx, _)| idx).collect();
161 if dash_indices.len() != 3 {
163 "Encountered incremental compilation session directory with \
165 session_dir.display()
169 crate_dir.join(&directory_name[0..dash_indices[2]]).with_extension(&LOCK_FILE_EXT[1..])
172 pub fn in_incr_comp_dir_sess(sess: &Session, file_name: &str) -> PathBuf {
173 in_incr_comp_dir(&sess.incr_comp_session_dir(), file_name)
176 pub fn in_incr_comp_dir(incr_comp_session_dir: &Path, file_name: &str) -> PathBuf {
177 incr_comp_session_dir.join(file_name)
180 /// Allocates the private session directory. The boolean in the Ok() result
181 /// indicates whether we should try loading a dep graph from the successfully
182 /// initialized directory, or not.
183 /// The post-condition of this fn is that we have a valid incremental
184 /// compilation session directory, if the result is `Ok`. A valid session
185 /// directory is one that contains a locked lock file. It may or may not contain
186 /// a dep-graph and work products from a previous session.
187 /// If the call fails, the fn may leave behind an invalid session directory.
188 /// The garbage collection will take care of it.
189 pub fn prepare_session_directory(
192 crate_disambiguator: CrateDisambiguator,
193 ) -> Result<(), ErrorReported> {
194 if sess.opts.incremental.is_none() {
198 let _timer = sess.timer("incr_comp_prepare_session_directory");
200 debug!("prepare_session_directory");
202 // {incr-comp-dir}/{crate-name-and-disambiguator}
203 let crate_dir = crate_path(sess, crate_name, crate_disambiguator);
204 debug!("crate-dir: {}", crate_dir.display());
205 create_dir(sess, &crate_dir, "crate")?;
207 // Hack: canonicalize the path *after creating the directory*
208 // because, on windows, long paths can cause problems;
209 // canonicalization inserts this weird prefix that makes windows
210 // tolerate long paths.
211 let crate_dir = match crate_dir.canonicalize() {
215 "incremental compilation: error canonicalizing path `{}`: {}",
219 return Err(ErrorReported);
223 let mut source_directories_already_tried = FxHashSet::default();
226 // Generate a session directory of the form:
228 // {incr-comp-dir}/{crate-name-and-disambiguator}/s-{timestamp}-{random}-working
229 let session_dir = generate_session_dir_path(&crate_dir);
230 debug!("session-dir: {}", session_dir.display());
232 // Lock the new session directory. If this fails, return an
233 // error without retrying
234 let (directory_lock, lock_file_path) = lock_directory(sess, &session_dir)?;
236 // Now that we have the lock, we can actually create the session
238 create_dir(sess, &session_dir, "session")?;
240 // Find a suitable source directory to copy from. Ignore those that we
241 // have already tried before.
242 let source_directory = find_source_directory(&crate_dir, &source_directories_already_tried);
244 let source_directory = if let Some(dir) = source_directory {
247 // There's nowhere to copy from, we're done
249 "no source directory found. Continuing with empty session \
253 sess.init_incr_comp_session(session_dir, directory_lock, false);
257 debug!("attempting to copy data from source: {}", source_directory.display());
259 // Try copying over all files from the source directory
260 if let Ok(allows_links) = copy_files(sess, &session_dir, &source_directory) {
261 debug!("successfully copied data from: {}", source_directory.display());
265 "Hard linking files in the incremental \
266 compilation cache failed. Copying files \
267 instead. Consider moving the cache \
268 directory to a file system which supports \
269 hard linking in session dir `{}`",
270 session_dir.display()
274 sess.init_incr_comp_session(session_dir, directory_lock, true);
277 debug!("copying failed - trying next directory");
279 // Something went wrong while trying to copy/link files from the
280 // source directory. Try again with a different one.
281 source_directories_already_tried.insert(source_directory);
283 // Try to remove the session directory we just allocated. We don't
284 // know if there's any garbage in it from the failed copy action.
285 if let Err(err) = safe_remove_dir_all(&session_dir) {
287 "Failed to delete partly initialized \
288 session dir `{}`: {}",
289 session_dir.display(),
294 delete_session_dir_lock_file(sess, &lock_file_path);
295 mem::drop(directory_lock);
300 /// This function finalizes and thus 'publishes' the session directory by
301 /// renaming it to `s-{timestamp}-{svh}` and releasing the file lock.
302 /// If there have been compilation errors, however, this function will just
303 /// delete the presumably invalid session directory.
304 pub fn finalize_session_directory(sess: &Session, svh: Svh) {
305 if sess.opts.incremental.is_none() {
309 let _timer = sess.timer("incr_comp_finalize_session_directory");
311 let incr_comp_session_dir: PathBuf = sess.incr_comp_session_dir().clone();
313 if sess.has_errors_or_delayed_span_bugs() {
314 // If there have been any errors during compilation, we don't want to
315 // publish this session directory. Rather, we'll just delete it.
318 "finalize_session_directory() - invalidating session directory: {}",
319 incr_comp_session_dir.display()
322 if let Err(err) = safe_remove_dir_all(&*incr_comp_session_dir) {
324 "Error deleting incremental compilation \
325 session directory `{}`: {}",
326 incr_comp_session_dir.display(),
331 let lock_file_path = lock_file_path(&*incr_comp_session_dir);
332 delete_session_dir_lock_file(sess, &lock_file_path);
333 sess.mark_incr_comp_session_as_invalid();
336 debug!("finalize_session_directory() - session directory: {}", incr_comp_session_dir.display());
338 let old_sub_dir_name = incr_comp_session_dir.file_name().unwrap().to_string_lossy();
339 assert_no_characters_lost(&old_sub_dir_name);
341 // Keep the 's-{timestamp}-{random-number}' prefix, but replace the
342 // '-working' part with the SVH of the crate
343 let dash_indices: Vec<_> = old_sub_dir_name.match_indices('-').map(|(idx, _)| idx).collect();
344 if dash_indices.len() != 3 {
346 "Encountered incremental compilation session directory with \
348 incr_comp_session_dir.display()
352 // State: "s-{timestamp}-{random-number}-"
353 let mut new_sub_dir_name = String::from(&old_sub_dir_name[..=dash_indices[2]]);
356 base_n::push_str(svh.as_u64() as u128, INT_ENCODE_BASE, &mut new_sub_dir_name);
358 // Create the full path
359 let new_path = incr_comp_session_dir.parent().unwrap().join(new_sub_dir_name);
360 debug!("finalize_session_directory() - new path: {}", new_path.display());
362 match std_fs::rename(&*incr_comp_session_dir, &new_path) {
364 debug!("finalize_session_directory() - directory renamed successfully");
366 // This unlocks the directory
367 sess.finalize_incr_comp_session(new_path);
370 // Warn about the error. However, no need to abort compilation now.
372 "Error finalizing incremental compilation \
373 session directory `{}`: {}",
374 incr_comp_session_dir.display(),
378 debug!("finalize_session_directory() - error, marking as invalid");
379 // Drop the file lock, so we can garage collect
380 sess.mark_incr_comp_session_as_invalid();
384 let _ = garbage_collect_session_directories(sess);
387 pub fn delete_all_session_dir_contents(sess: &Session) -> io::Result<()> {
388 let sess_dir_iterator = sess.incr_comp_session_dir().read_dir()?;
389 for entry in sess_dir_iterator {
391 safe_remove_file(&entry.path())?
396 fn copy_files(sess: &Session, target_dir: &Path, source_dir: &Path) -> Result<bool, ()> {
397 // We acquire a shared lock on the lock file of the directory, so that
398 // nobody deletes it out from under us while we are reading from it.
399 let lock_file_path = lock_file_path(source_dir);
400 let _lock = if let Ok(lock) = flock::Lock::new(
402 false, // don't wait,
403 false, // don't create
409 // Could not acquire the lock, don't try to copy from here
413 let source_dir_iterator = match source_dir.read_dir() {
415 Err(_) => return Err(()),
418 let mut files_linked = 0;
419 let mut files_copied = 0;
421 for entry in source_dir_iterator {
424 let file_name = entry.file_name();
426 let target_file_path = target_dir.join(file_name);
427 let source_path = entry.path();
429 debug!("copying into session dir: {}", source_path.display());
430 match link_or_copy(source_path, target_file_path) {
431 Ok(LinkOrCopy::Link) => files_linked += 1,
432 Ok(LinkOrCopy::Copy) => files_copied += 1,
433 Err(_) => return Err(()),
436 Err(_) => return Err(()),
440 if sess.opts.debugging_opts.incremental_info {
442 "[incremental] session directory: \
443 {} files hard-linked",
447 "[incremental] session directory: \
453 Ok(files_linked > 0 || files_copied == 0)
456 /// Generates unique directory path of the form:
457 /// {crate_dir}/s-{timestamp}-{random-number}-working
458 fn generate_session_dir_path(crate_dir: &Path) -> PathBuf {
459 let timestamp = timestamp_to_string(SystemTime::now());
460 debug!("generate_session_dir_path: timestamp = {}", timestamp);
461 let random_number = thread_rng().next_u32();
462 debug!("generate_session_dir_path: random_number = {}", random_number);
464 let directory_name = format!(
467 base_n::encode(random_number as u128, INT_ENCODE_BASE)
469 debug!("generate_session_dir_path: directory_name = {}", directory_name);
470 let directory_path = crate_dir.join(directory_name);
471 debug!("generate_session_dir_path: directory_path = {}", directory_path.display());
475 fn create_dir(sess: &Session, path: &Path, dir_tag: &str) -> Result<(), ErrorReported> {
476 match std_fs::create_dir_all(path) {
478 debug!("{} directory created successfully", dir_tag);
483 "Could not create incremental compilation {} \
494 /// Allocate the lock-file and lock it.
498 ) -> Result<(flock::Lock, PathBuf), ErrorReported> {
499 let lock_file_path = lock_file_path(session_dir);
500 debug!("lock_directory() - lock_file: {}", lock_file_path.display());
502 match flock::Lock::new(
505 true, // create the lock file
508 // the lock should be exclusive
509 Ok(lock) => Ok((lock, lock_file_path)),
511 let mut err = sess.struct_err(&format!(
512 "incremental compilation: could not create \
513 session directory lock file: {}",
516 if flock::Lock::error_unsupported(&lock_err) {
518 "the filesystem for the incremental path at {} \
519 does not appear to support locking, consider changing the \
520 incremental path to a filesystem that supports locking \
521 or disable incremental compilation",
522 session_dir.display()
524 if std::env::var_os("CARGO").is_some() {
526 "incremental compilation can be disabled by setting the \
527 environment variable CARGO_INCREMENTAL=0 (see \
528 https://doc.rust-lang.org/cargo/reference/profiles.html#incremental)",
531 "the entire build directory can be changed to a different \
532 filesystem by setting the environment variable CARGO_TARGET_DIR \
533 to a different path (see \
534 https://doc.rust-lang.org/cargo/reference/config.html#buildtarget-dir)",
544 fn delete_session_dir_lock_file(sess: &Session, lock_file_path: &Path) {
545 if let Err(err) = safe_remove_file(&lock_file_path) {
547 "Error deleting lock file for incremental \
548 compilation session directory `{}`: {}",
549 lock_file_path.display(),
555 /// Finds the most recent published session directory that is not in the
557 fn find_source_directory(
559 source_directories_already_tried: &FxHashSet<PathBuf>,
560 ) -> Option<PathBuf> {
564 .filter_map(|e| e.ok().map(|e| e.path()));
566 find_source_directory_in_iter(iter, source_directories_already_tried)
569 fn find_source_directory_in_iter<I>(
571 source_directories_already_tried: &FxHashSet<PathBuf>,
574 I: Iterator<Item = PathBuf>,
576 let mut best_candidate = (UNIX_EPOCH, None);
578 for session_dir in iter {
579 debug!("find_source_directory_in_iter - inspecting `{}`", session_dir.display());
581 let directory_name = session_dir.file_name().unwrap().to_string_lossy();
582 assert_no_characters_lost(&directory_name);
584 if source_directories_already_tried.contains(&session_dir)
585 || !is_session_directory(&directory_name)
586 || !is_finalized(&directory_name)
588 debug!("find_source_directory_in_iter - ignoring");
592 let timestamp = extract_timestamp_from_session_dir(&directory_name).unwrap_or_else(|_| {
593 bug!("unexpected incr-comp session dir: {}", session_dir.display())
596 if timestamp > best_candidate.0 {
597 best_candidate = (timestamp, Some(session_dir.clone()));
604 fn is_finalized(directory_name: &str) -> bool {
605 !directory_name.ends_with("-working")
608 fn is_session_directory(directory_name: &str) -> bool {
609 directory_name.starts_with("s-") && !directory_name.ends_with(LOCK_FILE_EXT)
612 fn is_session_directory_lock_file(file_name: &str) -> bool {
613 file_name.starts_with("s-") && file_name.ends_with(LOCK_FILE_EXT)
616 fn extract_timestamp_from_session_dir(directory_name: &str) -> Result<SystemTime, ()> {
617 if !is_session_directory(directory_name) {
621 let dash_indices: Vec<_> = directory_name.match_indices('-').map(|(idx, _)| idx).collect();
622 if dash_indices.len() != 3 {
626 string_to_timestamp(&directory_name[dash_indices[0] + 1..dash_indices[1]])
629 fn timestamp_to_string(timestamp: SystemTime) -> String {
630 let duration = timestamp.duration_since(UNIX_EPOCH).unwrap();
631 let micros = duration.as_secs() * 1_000_000 + (duration.subsec_nanos() as u64) / 1000;
632 base_n::encode(micros as u128, INT_ENCODE_BASE)
635 fn string_to_timestamp(s: &str) -> Result<SystemTime, ()> {
636 let micros_since_unix_epoch = u64::from_str_radix(s, INT_ENCODE_BASE as u32);
638 if micros_since_unix_epoch.is_err() {
642 let micros_since_unix_epoch = micros_since_unix_epoch.unwrap();
644 let duration = Duration::new(
645 micros_since_unix_epoch / 1_000_000,
646 1000 * (micros_since_unix_epoch % 1_000_000) as u32,
648 Ok(UNIX_EPOCH + duration)
654 crate_disambiguator: CrateDisambiguator,
656 let incr_dir = sess.opts.incremental.as_ref().unwrap().clone();
658 // The full crate disambiguator is really long. 64 bits of it should be
660 let crate_disambiguator = crate_disambiguator.to_fingerprint().to_smaller_hash();
661 let crate_disambiguator = base_n::encode(crate_disambiguator as u128, INT_ENCODE_BASE);
663 let crate_name = format!("{}-{}", crate_name, crate_disambiguator);
664 incr_dir.join(crate_name)
667 fn assert_no_characters_lost(s: &str) {
668 if s.contains('\u{FFFD}') {
669 bug!("Could not losslessly convert '{}'.", s)
673 fn is_old_enough_to_be_collected(timestamp: SystemTime) -> bool {
674 timestamp < SystemTime::now() - Duration::from_secs(10)
677 pub fn garbage_collect_session_directories(sess: &Session) -> io::Result<()> {
678 debug!("garbage_collect_session_directories() - begin");
680 let session_directory = sess.incr_comp_session_dir();
682 "garbage_collect_session_directories() - session directory: {}",
683 session_directory.display()
686 let crate_directory = session_directory.parent().unwrap();
688 "garbage_collect_session_directories() - crate directory: {}",
689 crate_directory.display()
692 // First do a pass over the crate directory, collecting lock files and
693 // session directories
694 let mut session_directories = FxHashSet::default();
695 let mut lock_files = FxHashSet::default();
697 for dir_entry in crate_directory.read_dir()? {
698 let dir_entry = match dir_entry {
699 Ok(dir_entry) => dir_entry,
706 let entry_name = dir_entry.file_name();
707 let entry_name = entry_name.to_string_lossy();
709 if is_session_directory_lock_file(&entry_name) {
710 assert_no_characters_lost(&entry_name);
711 lock_files.insert(entry_name.into_owned());
712 } else if is_session_directory(&entry_name) {
713 assert_no_characters_lost(&entry_name);
714 session_directories.insert(entry_name.into_owned());
716 // This is something we don't know, leave it alone
720 // Now map from lock files to session directories
721 let lock_file_to_session_dir: FxHashMap<String, Option<String>> = lock_files
723 .map(|lock_file_name| {
724 assert!(lock_file_name.ends_with(LOCK_FILE_EXT));
725 let dir_prefix_end = lock_file_name.len() - LOCK_FILE_EXT.len();
727 let dir_prefix = &lock_file_name[0..dir_prefix_end];
728 session_directories.iter().find(|dir_name| dir_name.starts_with(dir_prefix))
730 (lock_file_name, session_dir.map(String::clone))
734 // Delete all lock files, that don't have an associated directory. They must
735 // be some kind of leftover
736 for (lock_file_name, directory_name) in &lock_file_to_session_dir {
737 if directory_name.is_none() {
738 let timestamp = match extract_timestamp_from_session_dir(lock_file_name) {
739 Ok(timestamp) => timestamp,
742 "found lock-file with malformed timestamp: {}",
743 crate_directory.join(&lock_file_name).display()
750 let lock_file_path = crate_directory.join(&**lock_file_name);
752 if is_old_enough_to_be_collected(timestamp) {
754 "garbage_collect_session_directories() - deleting \
755 garbage lock file: {}",
756 lock_file_path.display()
758 delete_session_dir_lock_file(sess, &lock_file_path);
761 "garbage_collect_session_directories() - lock file with \
762 no session dir not old enough to be collected: {}",
763 lock_file_path.display()
769 // Filter out `None` directories
770 let lock_file_to_session_dir: FxHashMap<String, String> = lock_file_to_session_dir
772 .filter_map(|(lock_file_name, directory_name)| directory_name.map(|n| (lock_file_name, n)))
775 // Delete all session directories that don't have a lock file.
776 for directory_name in session_directories {
777 if !lock_file_to_session_dir.values().any(|dir| *dir == directory_name) {
778 let path = crate_directory.join(directory_name);
779 if let Err(err) = safe_remove_dir_all(&path) {
781 "Failed to garbage collect invalid incremental \
782 compilation session directory `{}`: {}",
790 // Now garbage collect the valid session directories.
791 let mut deletion_candidates = vec![];
793 for (lock_file_name, directory_name) in &lock_file_to_session_dir {
794 debug!("garbage_collect_session_directories() - inspecting: {}", directory_name);
796 let timestamp = match extract_timestamp_from_session_dir(directory_name) {
797 Ok(timestamp) => timestamp,
800 "found session-dir with malformed timestamp: {}",
801 crate_directory.join(directory_name).display()
808 if is_finalized(directory_name) {
809 let lock_file_path = crate_directory.join(lock_file_name);
810 match flock::Lock::new(
813 false, // don't create the lock-file
816 // get an exclusive lock
819 "garbage_collect_session_directories() - \
820 successfully acquired lock"
823 "garbage_collect_session_directories() - adding \
824 deletion candidate: {}",
828 // Note that we are holding on to the lock
829 deletion_candidates.push((
831 crate_directory.join(directory_name),
837 "garbage_collect_session_directories() - \
838 not collecting, still in use"
842 } else if is_old_enough_to_be_collected(timestamp) {
843 // When cleaning out "-working" session directories, i.e.
844 // session directories that might still be in use by another
845 // compiler instance, we only look a directories that are
846 // at least ten seconds old. This is supposed to reduce the
847 // chance of deleting a directory in the time window where
848 // the process has allocated the directory but has not yet
849 // acquired the file-lock on it.
851 // Try to acquire the directory lock. If we can't, it
852 // means that the owning process is still alive and we
853 // leave this directory alone.
854 let lock_file_path = crate_directory.join(lock_file_name);
855 match flock::Lock::new(
858 false, // don't create the lock-file
861 // get an exclusive lock
864 "garbage_collect_session_directories() - \
865 successfully acquired lock"
868 delete_old(sess, &crate_directory.join(directory_name));
870 // Let's make it explicit that the file lock is released at this point,
871 // or rather, that we held on to it until here
876 "garbage_collect_session_directories() - \
877 not collecting, still in use"
883 "garbage_collect_session_directories() - not finalized, not \
889 // Delete all but the most recent of the candidates
890 for (path, lock) in all_except_most_recent(deletion_candidates) {
891 debug!("garbage_collect_session_directories() - deleting `{}`", path.display());
893 if let Err(err) = safe_remove_dir_all(&path) {
895 "Failed to garbage collect finalized incremental \
896 compilation session directory `{}`: {}",
901 delete_session_dir_lock_file(sess, &lock_file_path(&path));
904 // Let's make it explicit that the file lock is released at this point,
905 // or rather, that we held on to it until here
912 fn delete_old(sess: &Session, path: &Path) {
913 debug!("garbage_collect_session_directories() - deleting `{}`", path.display());
915 if let Err(err) = safe_remove_dir_all(&path) {
917 "Failed to garbage collect incremental compilation session directory `{}`: {}",
922 delete_session_dir_lock_file(sess, &lock_file_path(&path));
926 fn all_except_most_recent(
927 deletion_candidates: Vec<(SystemTime, PathBuf, Option<flock::Lock>)>,
928 ) -> FxHashMap<PathBuf, Option<flock::Lock>> {
929 let most_recent = deletion_candidates.iter().map(|&(timestamp, ..)| timestamp).max();
931 if let Some(most_recent) = most_recent {
934 .filter(|&(timestamp, ..)| timestamp != most_recent)
935 .map(|(_, path, lock)| (path, lock))
942 /// Since paths of artifacts within session directories can get quite long, we
943 /// need to support deleting files with very long paths. The regular
944 /// WinApi functions only support paths up to 260 characters, however. In order
945 /// to circumvent this limitation, we canonicalize the path of the directory
946 /// before passing it to std::fs::remove_dir_all(). This will convert the path
947 /// into the '\\?\' format, which supports much longer paths.
948 fn safe_remove_dir_all(p: &Path) -> io::Result<()> {
949 let canonicalized = match std_fs::canonicalize(p) {
950 Ok(canonicalized) => canonicalized,
951 Err(err) if err.kind() == io::ErrorKind::NotFound => return Ok(()),
952 Err(err) => return Err(err),
955 std_fs::remove_dir_all(canonicalized)
958 fn safe_remove_file(p: &Path) -> io::Result<()> {
959 let canonicalized = match std_fs::canonicalize(p) {
960 Ok(canonicalized) => canonicalized,
961 Err(err) if err.kind() == io::ErrorKind::NotFound => return Ok(()),
962 Err(err) => return Err(err),
965 match std_fs::remove_file(canonicalized) {
966 Err(err) if err.kind() == io::ErrorKind::NotFound => Ok(()),