htsjdk.samtools.util.IOUtil.assertFileIsWritable on certain NFS environments fails to correctly assess permissions

[jd-daniels]

Description of the issue:

When calling assertFilesIsWritable through htsjdk.samtools, through Picard mark duplicates, we find that IOUtil.java fails to correctly assert writable on NFS file shares that have special backend permissions. This NFS location is on Dell Powerscale and has ACL permissions to allow for multi protocol access. The user can write to the parent directory.

Exception in thread "main" htsjdk.samtools.SAMException: Cannot write file: . File does not exist and parent directory is not writable..
at htsjdk.samtools.util.IOUtil.assertFileIsWritable(IOUtil.java:562)
at picard.sam.markduplicates.MarkDuplicates.doWork(MarkDuplicates.java:251)
at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:308)
at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:103)
at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:113)

Your environment:

• version of htsjdk
  2.24.1

• version of java
  Java 15.0.1

• which OS
  RHEL 9.1
  5.14.0-162.6.1.el9_1.x86_64

Steps to reproduce

Run Picard mark duplicates on an NFS location that has multi protocol ACLs or probably any server side permission modifications like that.
There is probably a simpler test case using that specific htsjdk.samtools.util.IOUtil.assertFileIsWritable.

Expected behaviour

The directory is writable to the user, so it should not be telling us that it isn’t writable.
We use a lot of different software suites on this same NFS storage location that don’t seem to have an issue, so while a workaround is to use a NFS-only standard permission NFS share, it’s not ideal for many of our researchers..

Actual behaviour

File does not exist and parent directory is not writable..

[lbergelson]

@jd-daniels This is tricky for me to debug because I don't have access to a similar system.

It's strange that the reported file path is blank. (Cannot write file: . ). It means that for some reason file.getAbsolutePath() is returning an empty string. Can you include an example command line that fails here? Does providing an absolute path vs a relative one make any difference? It looks like it's failing with the metrics file path specifically so we should check if there is anything strange about that in particular.

I doubt it will fix this problem, but you might want to try upgrading your picard version since that one is pretty old.

[michaeljmetzger]

We are having this exact error. Was there ever a fix for it?

[andvon]

Hi all,

Our team is running into this error a well. Similar to @jd-daniels , we started to encounter issues when switching to rhel9 (with a modified permissions structure of some sort).

The error reads:

 /usr/local/bin/picard: line 5: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8): No such file or directory
  Aug 15, 2024 2:11:11 PM com.intel.gkl.NativeLibraryLoader load
  INFO: Loading libgkl_compression.so from jar:file:/usr/local/share/picard-3.1.1-0/picard.jar!/com/intel/gkl/native/libgkl_compression.so
  [Thu Aug 15 14:11:11 GMT 2024] MarkDuplicates --INPUT ENCFF869TVW_subset.lowq_filt.bam --OUTPUT ENCFF869TVW_subset.dupmarked.bam --METRICS_FILE ENCFF869TVW_subset.dupmarked.MarkDuplicates.metrics.txt --REMOVE_DUPLICATES false --ASSUME_SORTED true --VALIDATION_STRINGENCY LENIENT --MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP 50000 --MAX_FILE_HANDLES_FOR_READ_ENDS_MAP 8000 --SORTING_COLLECTION_SIZE_RATIO 0.25 --TAG_DUPLICATE_SET_MEMBERS false --REMOVE_SEQUENCING_DUPLICATES false --TAGGING_POLICY DontTag --CLEAR_DT true --DUPLEX_UMI false --FLOW_MODE false --FLOW_QUALITY_SUM_STRATEGY false --USE_END_IN_UNPAIRED_READS false --USE_UNPAIRED_CLIPPED_END false --UNPAIRED_END_UNCERTAINTY 0 --FLOW_SKIP_FIRST_N_FLOWS 0 --FLOW_Q_IS_KNOWN_END false --FLOW_EFFECTIVE_QUALITY_THRESHOLD 15 --ADD_PG_TAG_TO_READS true --DUPLICATE_SCORING_STRATEGY SUM_OF_BASE_QUALITIES --PROGRAM_RECORD_ID MarkDuplicates --PROGRAM_GROUP_NAME MarkDuplicates --READ_NAME_REGEX <optimized capture of last three ':' separated fields as numeric values> --OPTICAL_DUPLICATE_PIXEL_DISTANCE 100 --MAX_OPTICAL_DUPLICATE_SET_SIZE 300000 --VERBOSITY INFO --QUIET false --COMPRESSION_LEVEL 5 --MAX_RECORDS_IN_RAM 500000 --CREATE_INDEX false --CREATE_MD5_FILE false --help false --version false --showHidden false --USE_JDK_DEFLATER false --USE_JDK_INFLATER false
  [Thu Aug 15 14:11:11 GMT 2024] Executing as <##REDACTED##> on Linux 5.14.0-427.16.1.el9_4.x86_64 amd64; OpenJDK 64-Bit Server VM 21.0.1-internal-adhoc.conda.src; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: Version:3.1.1
  [Thu Aug 15 14:11:11 GMT 2024] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.01 minutes.
  Runtime.totalMemory()=2151677952
  To get help, see http://broadinstitute.github.io/picard/index.html#GettingHelp
  Exception in thread "main" htsjdk.samtools.SAMException: Cannot write file: ENCFF869TVW_subset.dupmarked.bam. File does not exist and parent directory is not writable..
        at htsjdk.samtools.util.IOUtil.assertFileIsWritable(IOUtil.java:562)
        at picard.sam.markduplicates.MarkDuplicates.doWork(MarkDuplicates.java:258)
        at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:280)
        at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:105)
        at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:115)

I tried setting --TMP_DIR, which didn't work. This CAN run successfully in /scratch, but these errors typically occur in the middle of workflows where we can't necessarily shift everything to a /scratch dir.

[andvon]

@michaeljmetzger @jd-daniels @lbergelson

Not sure if you are still looking for an answer, but I tested out a fix that seems to be working. Here are the steps I took to make it work:

the fix

1. clone the Picard repository
2. clone the htsjdk repository
3. make the following modification to this file in the htsjdk repository: src/main/java/htsjdk/samtools/util/IOUtil.java

diff --git a/src/main/java/htsjdk/samtools/util/IOUtil.java b/src/main/java/htsjdk/samtools/util/IOUtil.java
index 730506b2e..f8edf16f2 100644
--- a/src/main/java/htsjdk/samtools/util/IOUtil.java
+++ b/src/main/java/htsjdk/samtools/util/IOUtil.java
@@ -559,8 +559,14 @@ public class IOUtil {
                         "File does not exist and parent is not a directory.");
             }
             else if (!parent.canWrite()) {
-                throw new SAMException("Cannot write file: " + file.getAbsolutePath() + ". " +
-                        "File does not exist and parent directory is not writable..");
+                System.err.println("Initial write method failed, trying alternative...");
+                try {
+                    File tempFile = File.createTempFile(".writeTest", null, parent);
+                    tempFile.delete();
+                } catch (IOException e) {
+                    throw new SAMException("Cannot write file: " + file.getAbsolutePath() + ". " +
+                            "File does not exist and parent directory is not writable..");
+                }
             }
         }
         else if (file.isDirectory()) {

Now you can follow the remaining steps from Picard's documentation to compile with a custom htsjdk version.

1. In the htsjdk directory, run ./gradlew printVersion. Record the version that it returns
2. switch to the picard repo and run ./gradlew shadowJar -Dhtsjdk.version=<INSERT VERSION FROM BEFORE HERE>

If everything worked, you should now have a build/libs/picard.jar file there. I tested this on our rhel9 / NFS where we have been hitting this problem and it worked!

What does this do?

This performs a try-catch when the initial parent.canWrite() fails. I have no clue why this fails since it seems like a standard operation, but I am also not very familiar with the deep internals of java. Upon that initial failure, it will write to stderr that it is trying an alternative method. This method tries to write a temporary file in the parent. If that works, it then removes the temporary file. If it fails, then it throws the exception that has been causing issues.

This tripped us up for awhile; I hope it works for you guys, let me know if you have questions / improvements!