Centralize schema-safe writable ROOT I/O (preserve StreamerInfo on ROOT ops)

CONTRIBUTING.md

@@ -22,6 +22,12 @@ It is recommended that new contributions and functionalities added to REST have
 
 TODO : Explain doxygen formatting, tutorials, where official doc is located. ETC.
 
+### ROOT file I/O changes
+
+Code that creates, updates, replaces, or merges ROOT files must follow the
+[safe writable ROOT I/O guide](doc/developer/Safe%20writable%20ROOT%20IO.md). It documents the required
+schema preflight, ownership, transactional replacement, and local/remote path rules.
+
 ### Pipeline validation tests
 
 TODO : Explain how pipeline validation tests should be implemented

doc/developer/Safe writable ROOT IO.md

@@ -0,0 +1,118 @@
+# Safe writable ROOT I/O
+
+REST ROOT files may contain several historical `TStreamerInfo` entries and embedded schema rules. Opening such
+a file directly with `TFile::Open(..., "UPDATE")` bypasses REST's schema preflight and can let ROOT rewrite
+schema metadata before REST has established that every historical class layout is usable. Framework code that
+creates or mutates ROOT files must therefore use `TRestRootFileHandle`.
+
+## Opening files
+
+Use `TRestRootFileHandle::Open` for new code:
+
+```cpp
+#include "TRestTools.h"
+
+auto file = TRestRootFileHandle::Open(filename, TRestRootFileMode::Update);
+if (!file) {
+    ReportError(file.Error());
+    return false;
+}
+
+file->cd();
+WriteObjects();
+
+if (!file.Close()) {
+    ReportError(file.Error());
+    return false;
+}
+```
+
+The available modes are `Read`, `Recreate`, and `Update`. `Recreate` intentionally replaces an existing file
+and must not be used as a shortcut for `Update`. An update is initially opened read-only. REST inventories the
+exact class-name, class-version, and checksum tuples stored in the file, collects the embedded schema rules,
+prepares loaded or emulated ROOT classes, and preflights the historical entries. It then transitions the same
+`TFile` to update mode and verifies the local file identity and ROOT UUID before marking the exact historical
+class-index entries required for writing.
+
+`PrepareBorrowedUpdate(TFile&, std::string*)` provides the same update preparation when legacy code already
+owns a `TFile`. The supplied file must be valid, open in `READ` mode, and not writable:
+
+```cpp
+std::unique_ptr<TFile> file(TFile::Open(filename.c_str(), "READ"));
+std::string error;
+if (!file || !TRestRootFileHandle::PrepareBorrowedUpdate(*file, &error)) {
+    ReportError(error);
+    return false;
+}
+```
+
+Prefer `TRestRootFileHandle` whenever ownership can be changed. A borrowed file remains the caller's
+responsibility, including checking its close/write status. If preparation fails, propagate the error and do not
+attempt to write through that file.
+
+The preflight preserves the historical StreamerInfo entries and schema rules it found on disk; it does not make
+an incompatible class change or an incorrect schema rule valid. Class authors must still increment class
+versions as required, write correct evolution rules, and test representative old files both before and after a
+writable open.
+
+## Ownership and error handling
+
+`TRestRootFileHandle` is move-only. Pass it by reference while it remains owned by a component, or transfer it
+with `std::move`. Pointers returned by `Get()` and `operator->` are non-owning and must not be deleted or retained
+beyond the handle's lifetime.
+
+The destructor closes an open file, but cannot report failure. Code that writes must call `Close()` explicitly
+and handle a false result using `Error()`. Close a live destination before move-assigning another handle to it,
+because move assignment cannot return a close error for the previous file.
+
+## Replacing or merging files
+
+Use `TRestTools::MergeRootFilesTransactionally` instead of merging directly into the destination or manually
+renaming a partially written file:
+
+```cpp
+std::string error;
+const std::string existing = outputAlreadyExists ? output : "";
+if (!TRestTools::MergeRootFilesTransactionally(output, newInputs, existing, true, &error)) {
+    ReportError(error);
+    return false;
+}
+```
+
+When non-empty, `existingTarget` is included as the first merge input. Callers updating an existing output must
+pass it explicitly; otherwise its existing contents are not part of the merge.
+
+The helper inventories every input, rejects incompatible classes at the same key path, and constructs the
+result in a temporary sibling of the local destination. Before replacement it validates the expected
+StreamerInfo and schema rules, recursive key paths and classes, and summed `TTree` entries. It validates the
+installed file again and attempts to restore the previous destination from a rollback backup on failure.
+Local input files are removed only after successful replacement and validation when
+`removeInputsOnSuccess` is true.
+
+A false return can also mean that the merged output is valid but a backup or input could not be removed.
+Always inspect the returned error before deciding how to recover. Replacement uses platform filesystem
+operations on sibling paths, but this is not a promise of power-loss durability or atomic behavior on every
+mounted filesystem. Rollback can itself fail; preserve and report the detailed error, including any retained
+backup path.
+
+## Local and remote paths
+
+Remote ROOT files may be read and may be merge inputs if the installed ROOT transports can open them. Writable
+opens and transactional merge destinations must resolve to local paths; remote URLs are rejected before
+mutation. Local `file://` URLs are accepted. Use `TRestTools::IsRemoteRootPath` when a caller needs to validate
+or explain this policy before opening a file.
+
+Only local entries in `inputFiles` are candidates for removal after a successful merge. Remote inputs are not
+deleted by the helper.
+
+## `TRestRun` is intentionally non-copyable
+
+`TRestRun` owns live input and output handles and also holds raw aliases to file-owned objects. The previous
+implicit copy would have shallow-copied that state, making ownership and lifetime unsafe. Its copy constructor
+and copy assignment operator are therefore deleted.
+
+APIs should pass runs as `TRestRun&`, `const TRestRun&`, or pointers rather than by value. Use
+`std::unique_ptr<TRestRun>` when ownership of a run object itself must be transferred, or construct a separate
+`TRestRun` from the appropriate filename/configuration when an independent instance is required. `TRestRun`
+does not currently expose move construction or move assignment, so do not rely on `std::move` to transfer the
+object directly.

macros/REST_AddComponentDataSet.C

@@ -1,5 +1,6 @@
 #include "TRestComponent.h"
 #include "TRestTask.h"
+#include "TRestTools.h"
 
 #ifndef RestTask_AddComponent
 #define RestTask_AddComponent
@@ -29,18 +30,17 @@ Int_t REST_AddComponentDataSet(std::string cfgFile, std::string sectionName,
     TRestComponentDataSet comp(cfgFile.c_str(), sectionName.c_str());
     comp.Initialize();
 
-    TFile* f;
-    if (update)
-        f = TFile::Open(outputFile.c_str(), "UPDATE");
-    else
-        f = TFile::Open(outputFile.c_str(), "RECREATE");
+    auto file = TRestRootFileHandle::Open(outputFile,
+                                          update ? TRestRootFileMode::Update : TRestRootFileMode::Recreate);
+    if (!file) {
+        RESTError << file.Error() << RESTendl;
+        return -1;
+    }
 
     if (componentName == "") componentName = sectionName;
 
     comp.Write(componentName.c_str());
 
-    f->Close();
-
-    return 0;
+    return file.Close() ? 0 : -1;
 }
 #endif

macros/REST_AddComponentFormula.C

@@ -1,5 +1,6 @@
 #include "TRestComponent.h"
 #include "TRestTask.h"
+#include "TRestTools.h"
 
 #ifndef RestTask_AddComponentFormula
 #define RestTask_AddComponentFormula
@@ -29,18 +30,17 @@ Int_t REST_AddComponentFormula(std::string cfgFile, std::string sectionName,
     TRestComponentFormula comp(cfgFile.c_str(), sectionName.c_str());
     comp.Initialize();
 
-    TFile* f;
-    if (update)
-        f = TFile::Open(outputFile.c_str(), "UPDATE");
-    else
-        f = TFile::Open(outputFile.c_str(), "RECREATE");
+    auto file = TRestRootFileHandle::Open(outputFile,
+                                          update ? TRestRootFileMode::Update : TRestRootFileMode::Recreate);
+    if (!file) {
+        RESTError << file.Error() << RESTendl;
+        return -1;
+    }
 
     if (componentName == "") componentName = sectionName;
 
     comp.Write(componentName.c_str());
 
-    f->Close();
-
-    return 0;
+    return file.Close() ? 0 : -1;
 }
 #endif

macros/REST_CreateHisto.C

@@ -5,6 +5,7 @@
 #include <TMath.h>
 #include <TRestRun.h>
 #include <TRestTask.h>
+#include <TRestTools.h>
 #include <TSystem.h>
 
 #ifndef RestTask_CreateHisto
@@ -54,9 +55,13 @@ Int_t REST_CreateHisto(string varName, string rootFileName, TString histoName, i
 
     h->Scale(normFactor);
 
-    TFile* f = new TFile((TString)rootFileName, "update");
+    auto file = TRestRootFileHandle::Open(rootFileName, TRestRootFileMode::Update);
+    if (!file) {
+        RESTLog << file.Error() << RESTendl;
+        return -1;
+    }
     h->Write(histoName);
-    f->Close();
+    if (!file.Close()) return -1;
 
     RESTLog << "Written histogram " << histoName << " into " << rootFileName << RESTendl;
 

macros/REST_MergeFiles.C

@@ -1,5 +1,5 @@
-#include "TFileMerger.h"
 #include "TRestTask.h"
+#include "TRestTools.h"
 
 #ifndef RESTTask_MergeFiles
 #define RESTTask_MergeFiles
@@ -12,14 +12,13 @@
 //*******************************************************************************************************
 Int_t REST_MergeFiles(TString pathAndPattern, TString outputFilename) {
     vector<string> files = TRestTools::GetFilesMatchingPattern((string)pathAndPattern);
-    TFileMerger* m = new TFileMerger(false);
-    m->OutputFile(outputFilename);
-    for (auto f : files) {
-        m->AddFile(f.c_str());
+    std::string error;
+    const bool success =
+        TRestTools::MergeRootFilesTransactionally(outputFilename.Data(), files, "", false, &error);
+    if (!success) {
+        RESTError << error << RESTendl;
     }
-    int a = m->Merge();
-    delete m;
-    return a;
+    return success;
 
     // TRestRunMerger *runMerger = new TRestRunMerger( pathAndPattern );
 

source/framework/CMakeLists.txt

@@ -27,3 +27,6 @@ endif (CMAKE_SYSTEM_NAME MATCHES "Windows")
 compiledir(RestFramework)
 
 add_library_test()
+if (TEST)
+    add_subdirectory(test/io)
+endif ()

source/framework/analysis/src/TRestDataSetCalibration.cxx

@@ -100,6 +100,7 @@
 #include "TRestDataSetCalibration.h"
 
 #include "TRestDataSet.h"
+#include "TRestTools.h"
 
 ClassImp(TRestDataSetCalibration);
 
@@ -257,14 +258,18 @@ void TRestDataSetCalibration::Calibrate() {
     if (!fOutputFileName.empty()) {
         if (TRestTools::GetFileNameExtension(fOutputFileName) == "root") {
             dataSet.Export(fOutputFileName);
-            TFile* f = TFile::Open(fOutputFileName.c_str(), "UPDATE");
+            auto file = TRestRootFileHandle::Open(fOutputFileName, TRestRootFileMode::Update);
+            if (!file) {
+                RESTError << file.Error() << RESTendl;
+                return;
+            }
             this->Write();
             if (gr) gr->Write();
             if (linearFit) linearFit->Write();
             // if(lFit)lFit->Write();
             // spectrumFit->Write();
             if (spectrum) spectrum->Write();
-            f->Close();
+            if (!file.Close()) RESTError << file.Error() << RESTendl;
         }
     }
 }

source/framework/analysis/src/TRestDataSetGainMap.cxx

@@ -139,6 +139,8 @@
 
 #include "TRestDataSetGainMap.h"
 
+#include "TRestTools.h"
+
 ClassImp(TRestDataSetGainMap);
 ///////////////////////////////////////////////
 /// \brief Default constructor
@@ -320,10 +322,13 @@ void TRestDataSetGainMap::CalibrateDataSet(const std::string& dataSetFileName, s
     dataSet.Export(outputFileName, std::vector<std::string>(excludeCol.begin(), excludeCol.end()));
 
     // Add this TRestDataSetGainMap metadata to the output file
-    TFile* f = TFile::Open(outputFileName.c_str(), "UPDATE");
+    auto file = TRestRootFileHandle::Open(outputFileName, TRestRootFileMode::Update);
+    if (!file) {
+        RESTError << file.Error() << RESTendl;
+        return;
+    }
     this->Write();
-    f->Close();
-    delete f;
+    if (!file.Close()) RESTError << file.Error() << RESTendl;
 }
 
 /////////////////////////////////////////////
@@ -505,10 +510,16 @@ void TRestDataSetGainMap::Export(const std::string& fileName) {
     }
 
     if (TRestTools::GetFileNameExtension(fOutputFileName) == "root") {
-        TFile* f = TFile::Open(fOutputFileName.c_str(), "UPDATE");
+        auto file = TRestRootFileHandle::Open(fOutputFileName, TRestRootFileMode::Update);
+        if (!file) {
+            RESTError << file.Error() << RESTendl;
+            return;
+        }
         this->Write(GetName());
-        f->Close();
-        delete f;
+        if (!file.Close()) {
+            RESTError << file.Error() << RESTendl;
+            return;
+        }
         RESTInfo << "Calibration saved to " << fOutputFileName << RESTendl;
     } else
         RESTError << "File extension for " << fOutputFileName << "is not supported." << RESTendl;

source/framework/analysis/src/TRestDataSetOdds.cxx

@@ -104,6 +104,7 @@
 #include "TRestDataSetOdds.h"
 
 #include "TRestDataSet.h"
+#include "TRestTools.h"
 
 ClassImp(TRestDataSetOdds);
 
@@ -273,11 +274,15 @@ void TRestDataSetOdds::ComputeLogOdds() {
         if (TRestTools::GetFileNameExtension(fOutputFileName) == "root") {
             RESTDebug << "Exporting dataset to " << fOutputFileName << RESTendl;
             dataSet.Export(fOutputFileName);
-            TFile* f = TFile::Open(fOutputFileName.c_str(), "UPDATE");
+            auto file = TRestRootFileHandle::Open(fOutputFileName, TRestRootFileMode::Update);
+            if (!file) {
+                RESTError << file.Error() << RESTendl;
+                return;
+            }
             this->Write();
             RESTDebug << "Writing histograms to " << fOutputFileName << RESTendl;
             for (const auto& [obsName, histo] : fHistos) histo->Write();
-            f->Close();
+            if (!file.Close()) RESTError << file.Error() << RESTendl;
         }
     }
 }

source/framework/core/inc/TRestProcessRunner.h

@@ -9,6 +9,7 @@
 #include "TRestEventProcess.h"
 #include "TRestMetadata.h"
 #include "TRestRun.h"
+#include "TRestTools.h"
 
 #define TIME_MEASUREMENT
 
@@ -35,8 +36,9 @@ class TRestProcessRunner : public TRestMetadata {
     TRestEvent* fOutputEvent;  //!
 
     // self variables for processing
-    std::vector<TRestThread*> fThreads;  //!
-    TFile* fOutputDataFile;              //! the TFile pointer being used
+    std::vector<TRestThread*> fThreads;        //!
+    TRestRootFileHandle fOutputDataFileOwner;  //!
+    TFile* fOutputDataFile;                    //! the TFile pointer being used
     TString fOutputDataFileName;  //! indicates the name of the first file created as output data file. The
                                   //! actual output file maybe changed if tree is too large
     TTree* fEventTree;            //!
@@ -99,7 +101,7 @@ class TRestProcessRunner : public TRestMetadata {
     void FillThreadEventFunc(TRestThread* t);
     void ConfigOutputFile();
     void MergeOutputFile();
-    void WriteProcessesMetadata();
+    void WriteProcessesMetadata(TFile* destination = nullptr);
 
     // tools
     void ResetRunTimes();