| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
| Name | Name | Last commit date | ||
|---|---|---|---|---|
Recursive Extractor is a Cross-Platform .NET Standard 2.0 Library and Command Line Program for parsing archive files and disk images, including nested archives and disk images.
| 7zip+ | ar | bzip2 |
| deb | dmg** | gzip |
| iso | rar^ | tar |
| vhd | vhdx | vmdk |
| wim* | xzip | zip+ |
This adds RecursiveExtractor to your path so you can run it directly from your shell.
Basic usage is: RecursiveExtractor --input archive.ext --output outputDirectory
Detailed UsageFor example, to extract only ".cs" files:
RecursiveExtractor --input archive.ext --output outputDirectory --allow-globs **/*.cs
Run RecursiveExtractor --help for more details.
Recursive Extractor is available on NuGet as Microsoft.CST.RecursiveExtractor. Recursive Extractor targets netstandard2.0+ and the latest .NET, currently .NET 6.0, .NET 7.0 and .NET 8.0.
The most basic usage is to enumerate through all the files in the archive provided and do something with their contents as a Stream.
using Microsoft.CST.RecursiveExtractor;
var path = "path/to/file";
var extractor = new Extractor();
foreach(var file in extractor.Extract(path))
{
doSomething(file.Content); //Do Something with the file contents (a Stream)
}using Microsoft.CST.RecursiveExtractor;
var extractor = new Extractor();
var extractorOptions = new ExtractorOptions()
{
ExtractSelfOnFail = true,
};
extractor.ExtractToDirectory(options.Output, options.Input, extractorOptions);var path = "/Path/To/Your/Archive"
var extractor = new Extractor();
try {
IEnumerable<FileEntry> results = extractor.ExtractFileAsync(path);
await foreach(var found in results)
{
Console.WriteLine(found.FullPath);
}
}
catch(OverflowException)
{
// This means Recursive Extractor has detected a Quine or Zip Bomb
}public Stream Content { get; }
public string FullPath { get; }
public string Name { get; }
public FileEntry? Parent { get; }
public string? ParentPath { get; }
public DateTime CreateTime { get; }
public DateTime ModifyTime { get; }
public DateTime AccessTime { get; }var path = "/Path/To/Your/Archive"
var directory
var extractor = new Extractor();
try {
IEnumerable<FileEntry> results = extractor.ExtractFile(path, new ExtractorOptions()
{
Passwords = new Dictionary<Regex, List<string>>()
{
{ new Regex("\.zip"), new List<string>(){ "PasswordForZipFiles" } },
{ new Regex("\.7z"), new List<string>(){ "PasswordFor7zFiles" } },
{ new Regex(".*"), new List<string>(){ "PasswordForAllFiles" } }
}
});
foreach(var found in results)
{
Console.WriteLine(found.FullPath);
}
}
catch(OverflowException)
{
// This means Recursive Extractor has detected a Quine or Zip Bomb
}To create a custom extractor, implement the ICustomAsyncExtractor interface and register it with the extractor:
using Microsoft.CST.RecursiveExtractor;
using Microsoft.CST.RecursiveExtractor.Extractors;
using System.IO;
using System.Collections.Generic;
using System.Linq;
// Example: Custom extractor for a hypothetical archive format with magic bytes "MYARC"
public class MyCustomExtractor : ICustomAsyncExtractor
{
private readonly Extractor context;
private static readonly byte[] MAGIC_BYTES = System.Text.Encoding.ASCII.GetBytes("MYARC");
public MyCustomExtractor(Extractor ctx)
{
context = ctx;
}
// Check if this extractor can handle the file based on binary signatures
public bool CanExtract(Stream stream)
{
if (stream == null || !stream.CanRead || !stream.CanSeek || stream.Length < MAGIC_BYTES.Length)
{
return false;
}
var initialPosition = stream.Position;
try
{
stream.Position = 0;
var buffer = new byte[MAGIC_BYTES.Length];
var bytesRead = stream.Read(buffer, 0, MAGIC_BYTES.Length);
return bytesRead == MAGIC_BYTES.Length && buffer.SequenceEqual(MAGIC_BYTES);
}
finally
{
// Always restore the original position
stream.Position = initialPosition;
}
}
// Implement extraction logic
public IEnumerable<FileEntry> Extract(FileEntry fileEntry, ExtractorOptions options, ResourceGovernor governor, bool topLevel = true)
{
// Your extraction logic here
// For example, parse the archive and yield FileEntry objects for each contained file
yield break;
}
public async IAsyncEnumerable<FileEntry> ExtractAsync(FileEntry fileEntry, ExtractorOptions options, ResourceGovernor governor, bool topLevel = true)
{
// Your async extraction logic here
yield break;
}
}
// Register the custom extractor via constructor
var customExtractor = new MyCustomExtractor(null);
var extractor = new Extractor(new[] { customExtractor });
// Now the extractor will use your custom extractor for files matching your CanExtract criteria
var results = extractor.Extract("path/to/custom/archive.myarc");Key points:
RecursiveExtractor protects against ZipSlip, Quines, and Zip Bombs. Calls to Extract will throw an OverflowException when a Quine or Zip bomb is detected and a TimeOutException if EnableTiming is set and the specified time period has elapsed before completion.
Otherwise, invalid files found while crawling will emit a logger message and be skipped. You can also enable ExtractSelfOnFail to return the original archive file on an extraction failure.
You should not iterate the Enumeration returned from the Extract and ExtractAsync interfaces multiple times, if you need to do so, convert the Enumeration to an in memory collection first.
If you want to enumerate the output with parallelization you should use a batching mechanism, for example:
var extractedEnumeration = Extract(fileEntry, opts);
using var enumerator = extractedEnumeration.GetEnumerator();
ConcurrentBag<FileEntry> entryBatch = new();
bool moreAvailable = enumerator.MoveNext();
while (moreAvailable)
{
entryBatch = new();
for (int i = 0; i < BatchSize; i++)
{
entryBatch.Add(enumerator.Current);
moreAvailable = enumerator.MoveNext();
if (!moreAvailable)
{
break;
}
}
if (entryBatch.Count == 0)
{
break;
}
// Run your parallel processing on the batch
Parallel.ForEach(entryBatch, new ParallelOptions() { CancellationToken = cts.Token }, entry =>
{
// Do something with each FileEntry
}
}If you are working with a very large archive or in particularly constrained environment you can reduce memory and file handle usage for the Content streams in each FileEntry by disposing as you iterate.
var results = extractor.Extract(path);
foreach(var file in results)
{
using var theStream = file.Content;
// Do something with the stream.
_ = theStream.ReadByte();
// The stream is disposed here by the using statement
} The Content stream of each FileEntry is held in memory while it is smaller than ExtractorOptions.MemoryStreamCutoff (100MB by default) and backed by a temporary file once it grows past it. Entries whose size is not known before extraction begins start in memory and move to disk only if they actually exceed the cutoff.
Because entries are held concurrently when ExtractorOptions.Parallel is set, peak memory is bounded by BatchSize * MemoryStreamCutoff. Lower MemoryStreamCutoff if you are extracting in a memory constrained environment, or raise it to trade memory for fewer temporary files.
If you have any issues or feature requests (for example, supporting other formats) you can open a new Issue.
If you are having trouble parsing a specific archive of one of the supported formats, it is helpful if you can include an sample archive with your report that demonstrates the issue.
Recursive Extractor aims to provide a unified interface to extract arbitrary archives and relies on a number of libraries to parse the archives.
This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.
When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.
This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.
| Back | FazBrowse Home | New Git URL |