| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
1 parent 174faa9 commit 55c30a3
7 files changed
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
@@ -0,0 +1,172 @@ | |||
| 1 | + # Fuzzing GitPython | ||
| 2 | + | ||
| 3 | + [][oss-fuzz-issue-tracker] | ||
| 4 | + | ||
| 5 | + This directory contains files related to GitPython's suite of fuzz tests that are executed daily on automated | ||
| 6 | + infrastructure provided by [OSS-Fuzz][oss-fuzz-repo]. This document aims to provide necessary information for working | ||
| 7 | + with fuzzing in GitPython. | ||
| 8 | + | ||
| 9 | + The details about the latest OSS-Fuzz test status, including build logs and coverage reports, is made available | ||
| 10 | + at [this link](https://introspector.oss-fuzz.com/project-profile?project=gitpython). | ||
| 11 | + | ||
| 12 | + ## How to Contribute | ||
| 13 | + | ||
| 14 | + There are many ways to contribute to GitPython's fuzzing efforts! Contributions are welcomed through issues, | ||
| 15 | + discussions, or pull requests on this repository. | ||
| 16 | + | ||
| 17 | + Areas that are particularly appreciated include: | ||
| 18 | + | ||
| 19 | + - **Tackling the existing backlog of open issues**. While fuzzing is an effective way to identify bugs, that information | ||
| 20 | + isn't useful unless they are fixed. If you are not sure where to start, the issues tab is a great place to get ideas! | ||
| 21 | + - **Improvements to this (or other) documentation** make it easier for new contributors to get involved, so even small | ||
| 22 | + improvements can have a large impact over time. If you see something that could be made easier by a documentation | ||
| 23 | + update of any size, please consider suggesting it! | ||
| 24 | + | ||
| 25 | + For everything else, such as expanding test coverage, optimizing test performance, or enhancing error detection | ||
| 26 | + capabilities, jump in to the "Getting Started" section below. | ||
| 27 | + | ||
| 28 | + ## Getting Started with Fuzzing GitPython | ||
| 29 | + | ||
| 30 | + > [!TIP] | ||
| 31 | + > **New to fuzzing or unfamiliar with OSS-Fuzz?** | ||
| 32 | + > | ||
| 33 | + > These resources are an excellent place to start: | ||
| 34 | + > | ||
| 35 | + > - [OSS-Fuzz documentation][oss-fuzz-docs] - Continuous fuzzing service for open source software. | ||
| 36 | + > - [Google/fuzzing][google-fuzzing-repo] - Tutorials, examples, discussions, research proposals, and other resources | ||
| 37 | + related to fuzzing. | ||
| 38 | + > - [CNCF Fuzzing Handbook](https://github.com/cncf/tag-security/blob/main/security-fuzzing-handbook/handbook-fuzzing.pdf) - | ||
| 39 | + A comprehensive guide for fuzzing open source software. | ||
| 40 | + > - [Efficient Fuzzing Guide by The Chromium Project](https://chromium.googlesource.com/chromium/src/+/main/testing/libfuzzer/efficient_fuzzing.md) - | ||
| 41 | + Explores strategies to enhance the effectiveness of your fuzz tests, recommended for those looking to optimize their | ||
| 42 | + testing efforts. | ||
| 43 | + | ||
| 44 | + ### Setting Up Your Local Environment | ||
| 45 | + | ||
| 46 | + Before contributing to fuzzing efforts, ensure Python and Docker are installed on your machine. Docker is required for | ||
| 47 | + running fuzzers in containers provided by OSS-Fuzz. [Install Docker](https://docs.docker.com/get-docker/) following the | ||
| 48 | + official guide if you do not already have it. | ||
| 49 | + | ||
| 50 | + ### Understanding Existing Fuzz Targets | ||
| 51 | + | ||
| 52 | + Review the `fuzz-targets/` directory to familiarize yourself with how existing tests are implemented. See | ||
| 53 | + the [Files & Directories Overview](#files--directories-overview) for more details on the directory structure. | ||
| 54 | + | ||
| 55 | + ### Contributing to Fuzz Tests | ||
| 56 | + | ||
| 57 | + Start by reviewing the [Atheris documentation][atheris-repo] and the section | ||
| 58 | + on [Running Fuzzers Locally](#running-fuzzers-locally) to begin writing or improving fuzz tests. | ||
| 59 | + | ||
| 60 | + ## Files & Directories Overview | ||
| 61 | + | ||
| 62 | + The `fuzzing/` directory is organized into three key areas: | ||
| 63 | + | ||
| 64 | + ### Fuzz Targets (`fuzz-targets/`) | ||
| 65 | + | ||
| 66 | + Contains Python files for each fuzz test, targeting specific functionalities of GitPython. | ||
| 67 | + | ||
| 68 | + **Things to Know**: | ||
| 69 | + | ||
| 70 | + - Each fuzz test targets a specific part of GitPython's functionality. | ||
| 71 | + - Test files adhere to the naming convention: `fuzz_<API Under Test>.py`, where `<API Under Test>` indicates the | ||
| 72 | + functionality targeted by the test. | ||
| 73 | + - Any functionality that involves performing operations on input data is a possible candidate for fuzz testing, but | ||
| 74 | + features that involve processing untrusted user input or parsing operations are typically going to be the most | ||
| 75 | + interesting. | ||
| 76 | + - The goal of these tests is to identify previously unknown or unexpected error cases caused by a given input. For that | ||
| 77 | + reason, fuzz tests should gracefully handle anticipated exception cases with a `try`/`except` block to avoid false | ||
| 78 | + positives that halt the fuzzing engine. | ||
| 79 | + | ||
| 80 | + ### Dictionaries (`dictionaries/`) | ||
| 81 | + | ||
| 82 | + Provides hints to the fuzzing engine about inputs that might trigger unique code paths. Each fuzz target may have a | ||
| 83 | + corresponding `.dict` file. For details on how these are used, refer | ||
| 84 | + to [LibFuzzer documentation](https://llvm.org/docs/LibFuzzer.html#dictionaries). | ||
| 85 | + | ||
| 86 | + **Things to Know**: | ||
| 87 | + | ||
| 88 | + - OSS-Fuzz loads dictionary files per fuzz target if one exists with the same name, all others are ignored. | ||
| 89 | + - Most entries in the dictionary files found here are escaped hex or Unicode values that were recommended by the fuzzing | ||
| 90 | + engine after previous runs. | ||
| 91 | + - A default set of dictionary entries are created for all fuzz targets as part of the build process, regardless of an | ||
| 92 | + existing file here. | ||
| 93 | + - Development or updates to dictionaries should reflect the varied formats and edge cases relevant to the | ||
| 94 | + functionalities under test. | ||
| 95 | + - Example dictionaries (some of which are used to build the default dictionaries mentioned above) are can be found here: | ||
| 96 | + - [AFL++ dictionary repository](https://github.com/AFLplusplus/AFLplusplus/tree/stable/dictionaries#readme) | ||
| 97 | + - [Google/fuzzing dictionary repository](https://github.com/google/fuzzing/tree/master/dictionaries) | ||
| 98 | + | ||
| 99 | + ### OSS-Fuzz Scripts (`oss-fuzz-scripts/`) | ||
| 100 | + | ||
| 101 | + Includes scripts for building and integrating fuzz targets with OSS-Fuzz: | ||
| 102 | + | ||
| 103 | + - **`container-environment-bootstrap.sh`** - Sets up the execution environment. It is responsible for fetching default | ||
| 104 | + dictionary entries and ensuring all required build dependencies are installed and up-to-date. | ||
| 105 | + - **`build.sh`** - Executed within the Docker container, this script builds fuzz targets with necessary instrumentation | ||
| 106 | + and prepares seed corpora and dictionaries for use. | ||
| 107 | + | ||
| 108 | + ## Running Fuzzers Locally | ||
| 109 | + | ||
| 110 | + ### Direct Execution of Fuzz Targets | ||
| 111 | + | ||
| 112 | + For quick testing of changes, [Atheris][atheris-repo] makes it possible to execute a fuzz target directly: | ||
| 113 | + | ||
| 114 | + 1. Install Atheris following the [installation guide][atheris-repo] for your operating system. | ||
| 115 | + 2. Execute a fuzz target, for example: | ||
| 116 | + | ||
| 117 | + ```shell | ||
| 118 | + python fuzzing/fuzz-targets/fuzz_config.py | ||
| 119 | + ``` | ||
| 120 | + | ||
| 121 | + ### Running OSS-Fuzz Locally | ||
| 122 | + | ||
| 123 | + This approach uses Docker images provided by OSS-Fuzz for building and running fuzz tests locally. It offers | ||
| 124 | + comprehensive features but requires a local clone of the OSS-Fuzz repository and sufficient disk space for Docker | ||
| 125 | + containers. | ||
| 126 | + | ||
| 127 | + #### Preparation | ||
| 128 | + | ||
| 129 | + Set environment variables to simplify command usage: | ||
| 130 | + | ||
| 131 | + ```shell | ||
| 132 | + export SANITIZER=address # Can be either 'address' or 'undefined'. | ||
| 133 | + export FUZZ_TARGET=fuzz_config # specify the fuzz target without the .py extension. | ||
| 134 | + ``` | ||
| 135 | + | ||
| 136 | + #### Build and Run | ||
| 137 | + | ||
| 138 | + Clone the OSS-Fuzz repository and prepare the Docker environment: | ||
| 139 | + | ||
| 140 | + ```shell | ||
| 141 | + git clone --depth 1 https://github.com/google/oss-fuzz.git oss-fuzz | ||
| 142 | + cd oss-fuzz | ||
| 143 | + python infra/helper.py build_image gitpython | ||
| 144 | + python infra/helper.py build_fuzzers --sanitizer $SANITIZER gitpython | ||
| 145 | + ``` | ||
| 146 | + | ||
| 147 | + Verify the build of your fuzzers with the optional `check_build` command: | ||
| 148 | + | ||
| 149 | + ```shell | ||
| 150 | + python infra/helper.py check_build gitpython | ||
| 151 | + ``` | ||
| 152 | + | ||
| 153 | + Execute the desired fuzz target: | ||
| 154 | + | ||
| 155 | + ```shell | ||
| 156 | + python infra/helper.py run_fuzzer gitpython $FUZZ_TARGET | ||
| 157 | + ``` | ||
| 158 | + | ||
| 159 | + #### Next Steps | ||
| 160 | + | ||
| 161 | + For detailed instructions on advanced features like reproducing OSS-Fuzz issues or using the Fuzz Introspector, refer | ||
| 162 | + to [the official OSS-Fuzz documentation][oss-fuzz-docs]. | ||
| 163 | + | ||
| 164 | + [oss-fuzz-repo]: https://github.com/google/oss-fuzz | ||
| 165 | + | ||
| 166 | + [oss-fuzz-docs]: https://google.github.io/oss-fuzz | ||
| 167 | + | ||
| 168 | + [oss-fuzz-issue-tracker]: https://bugs.chromium.org/p/oss-fuzz/issues/list?sort=-opened&can=1&q=proj:gitpython | ||
| 169 | + | ||
| 170 | + [google-fuzzing-repo]: https://github.com/google/fuzzing | ||
| 171 | + | ||
| 172 | + [atheris-repo]: https://github.com/google/atheris | ||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
@@ -0,0 +1,56 @@ | |||
| 1 | + "\\004\\000\\000\\000\\000\\000\\000\\000" | ||
| 2 | + "\\006\\000\\000\\000\\000\\000\\000\\000" | ||
| 3 | + "_validate_value_" | ||
| 4 | + "\\000\\000\\000\\000\\000\\000\\000\\000" | ||
| 5 | + "rem" | ||
| 6 | + "__eq__" | ||
| 7 | + "\\001\\000\\000\\000" | ||
| 8 | + "__abstrac" | ||
| 9 | + "_mutating_methods_" | ||
| 10 | + "items" | ||
| 11 | + "\\0021\\"" | ||
| 12 | + "\\001\\000" | ||
| 13 | + "\\000\\000\\000\\000" | ||
| 14 | + "DEFAULT" | ||
| 15 | + "getfloat" | ||
| 16 | + "\\004\\000\\000\\000\\000\\000\\000\\000" | ||
| 17 | + "news" | ||
| 18 | + "\\037\\000\\000\\000\\000\\000\\000\\000" | ||
| 19 | + "\\001\\000\\000\\000\\000\\000\\000\\037" | ||
| 20 | + "\\000\\000\\000\\000\\000\\000\\000\\014" | ||
| 21 | + "list" | ||
| 22 | + "\\376\\377\\377\\377\\377\\377\\377\\377" | ||
| 23 | + "items_all" | ||
| 24 | + "\\004\\000\\000\\000\\000\\000\\000\\000" | ||
| 25 | + "\\377\\377\\377\\377\\377\\377\\377\\014" | ||
| 26 | + "\\001\\000\\000\\000" | ||
| 27 | + "_acqui" | ||
| 28 | + "\\000\\000\\000\\000\\000\\000\\000\\000" | ||
| 29 | + "__ne__" | ||
| 30 | + "__exit__" | ||
| 31 | + "__modu" | ||
| 32 | + "uucp" | ||
| 33 | + "__str__" | ||
| 34 | + "\\001\\000\\000\\000" | ||
| 35 | + "\\017\\000\\000\\000\\000\\000\\000\\000" | ||
| 36 | + "_has_incl" | ||
| 37 | + "update" | ||
| 38 | + "\\377\\377\\377\\377\\377\\377\\377\\023" | ||
| 39 | + "setdef" | ||
| 40 | + "setdefaul" | ||
| 41 | + "\\000\\000\\000\\000" | ||
| 42 | + "\\001\\000\\000\\000" | ||
| 43 | + "\\001\\000" | ||
| 44 | + "\\022\\000\\000\\000\\000\\000\\000\\000" | ||
| 45 | + "_value_to_string" | ||
| 46 | + "__abstr" | ||
| 47 | + "\\001\\000\\000\\000\\000\\000\\000\\000" | ||
| 48 | + "\\000\\000\\000\\000\\000\\000\\000\\022" | ||
| 49 | + "\\377\\377\\377\\377" | ||
| 50 | + "\\004\\000\\000\\000\\000\\000\\000\\000" | ||
| 51 | + "\\000\\000\\000\\000\\000\\000\\000\\000" | ||
| 52 | + "\\000\\000\\000\\000\\000\\000\\000\\037" | ||
| 53 | + "\\001\\000\\000\\000\\000\\000\\000\\013" | ||
| 54 | + "_OPT_TM" | ||
| 55 | + "__name__" | ||
| 56 | + "_get_conv" | ||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
@@ -0,0 +1,13 @@ | |||
| 1 | + "\\001\\000\\000\\000" | ||
| 2 | + "_join_multiline_va" | ||
| 3 | + "setdef" | ||
| 4 | + "1\\000\\000\\000\\000\\000\\000\\000" | ||
| 5 | + "\\000\\000\\000\\000\\000\\000\\000\\020" | ||
| 6 | + "\\377\\377\\377\\377\\377\\377\\377r" | ||
| 7 | + "\\001\\000\\000\\000\\000\\000\\000\\001" | ||
| 8 | + "\\000\\000\\000\\000\\000\\000\\000\\014" | ||
| 9 | + "\\000\\000\\000\\000\\000\\000\\000\\003" | ||
| 10 | + "\\001\\000" | ||
| 11 | + "\\032\\000\\000\\000\\000\\000\\000\\000" | ||
| 12 | + "-\\000\\000\\000\\000\\000\\000\\000" | ||
| 13 | + "__format" | ||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
@@ -0,0 +1,51 @@ | |||
| 1 | + #!/usr/bin/python3 | ||
| 2 | + # Copyright 2023 Google LLC | ||
| 3 | + # | ||
| 4 | + # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| 5 | + # you may not use this file except in compliance with the License. | ||
| 6 | + # You may obtain a copy of the License at | ||
| 7 | + # | ||
| 8 | + # http://www.apache.org/licenses/LICENSE-2.0 | ||
| 9 | + # | ||
| 10 | + # Unless required by applicable law or agreed to in writing, software | ||
| 11 | + # distributed under the License is distributed on an "AS IS" BASIS, | ||
| 12 | + # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| 13 | + # See the License for the specific language governing permissions and | ||
| 14 | + # limitations under the License. | ||
| 15 | + import atheris | ||
| 16 | + import sys | ||
| 17 | + import io | ||
| 18 | + from configparser import MissingSectionHeaderError, ParsingError | ||
| 19 | + | ||
| 20 | + with atheris.instrument_imports(): | ||
| 21 | + from git import GitConfigParser | ||
| 22 | + | ||
| 23 | + | ||
| 24 | + def TestOneInput(data): | ||
| 25 | + sio = io.BytesIO(data) | ||
| 26 | + sio.name = "/tmp/fuzzconfig.config" | ||
| 27 | + git_config = GitConfigParser(sio) | ||
| 28 | + try: | ||
| 29 | + git_config.read() | ||
| 30 | + except (MissingSectionHeaderError, ParsingError, UnicodeDecodeError): | ||
| 31 | + return -1 # Reject inputs raising expected exceptions | ||
| 32 | + except (IndexError, ValueError) as e: | ||
| 33 | + if isinstance(e, IndexError) and "string index out of range" in str(e): | ||
| 34 | + # Known possibility that might be patched | ||
| 35 | + # See: https://github.com/gitpython-developers/GitPython/issues/1887 | ||
| 36 | + pass | ||
| 37 | + elif isinstance(e, ValueError) and "embedded null byte" in str(e): | ||
| 38 | + # The `os.path.expanduser` function, which does not accept strings | ||
| 39 | + # containing null bytes might raise this. | ||
| 40 | + return -1 | ||
| 41 | + else: | ||
| 42 | + raise e # Raise unanticipated exceptions as they might be bugs | ||
| 43 | + | ||
| 44 | + | ||
| 45 | + def main(): | ||
| 46 | + atheris.Setup(sys.argv, TestOneInput) | ||
| 47 | + atheris.Fuzz() | ||
| 48 | + | ||
| 49 | + | ||
| 50 | + if __name__ == "__main__": | ||
| 51 | + main() | ||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
@@ -0,0 +1,59 @@ | |||
| 1 | + #!/usr/bin/python3 | ||
| 2 | + # Copyright 2023 Google LLC | ||
| 3 | + # | ||
| 4 | + # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| 5 | + # you may not use this file except in compliance with the License. | ||
| 6 | + # You may obtain a copy of the License at | ||
| 7 | + # | ||
| 8 | + # http://www.apache.org/licenses/LICENSE-2.0 | ||
| 9 | + # | ||
| 10 | + # Unless required by applicable law or agreed to in writing, software | ||
| 11 | + # distributed under the License is distributed on an "AS IS" BASIS, | ||
| 12 | + # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| 13 | + # See the License for the specific language governing permissions and | ||
| 14 | + # limitations under the License. | ||
| 15 | + import atheris | ||
| 16 | + import io | ||
| 17 | + import sys | ||
| 18 | + import os | ||
| 19 | + import shutil | ||
| 20 | + | ||
| 21 | + with atheris.instrument_imports(): | ||
| 22 | + from git.objects import Tree | ||
| 23 | + from git.repo import Repo | ||
| 24 | + | ||
| 25 | + | ||
| 26 | + def TestOneInput(data): | ||
| 27 | + fdp = atheris.FuzzedDataProvider(data) | ||
| 28 | + git_dir = "/tmp/.git" | ||
| 29 | + head_file = os.path.join(git_dir, "HEAD") | ||
| 30 | + refs_dir = os.path.join(git_dir, "refs") | ||
| 31 | + common_dir = os.path.join(git_dir, "commondir") | ||
| 32 | + objects_dir = os.path.join(git_dir, "objects") | ||
| 33 | + | ||
| 34 | + if os.path.isdir(git_dir): | ||
| 35 | + shutil.rmtree(git_dir) | ||
| 36 | + | ||
| 37 | + os.mkdir(git_dir) | ||
| 38 | + with open(head_file, "w") as f: | ||
| 39 | + f.write(fdp.ConsumeUnicodeNoSurrogates(1024)) | ||
| 40 | + os.mkdir(refs_dir) | ||
| 41 | + os.mkdir(common_dir) | ||
| 42 | + os.mkdir(objects_dir) | ||
| 43 | + | ||
| 44 | + _repo = Repo("/tmp/") | ||
| 45 | + | ||
| 46 | + fuzz_tree = Tree(_repo, Tree.NULL_BIN_SHA, 0, "") | ||
| 47 | + try: | ||
| 48 | + fuzz_tree._deserialize(io.BytesIO(data)) | ||
| 49 | + except IndexError: | ||
| 50 | + return -1 | ||
| 51 | + | ||
| 52 | + | ||
| 53 | + def main(): | ||
| 54 | + atheris.Setup(sys.argv, TestOneInput) | ||
| 55 | + atheris.Fuzz() | ||
| 56 | + | ||
| 57 | + | ||
| 58 | + if __name__ == "__main__": | ||
| 59 | + main() | ||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
@@ -0,0 +1,37 @@ | |||
| 1 | + #!/usr/bin/env bash | ||
| 2 | + | ||
| 3 | + set -euo pipefail | ||
| 4 | + | ||
| 5 | + python3 -m pip install . | ||
| 6 | + | ||
| 7 | + # Directory to look in for dictionaries, options files, and seed corpa: | ||
| 8 | + SEED_DATA_DIR="$SRC/seed_data" | ||
| 9 | + | ||
| 10 | + find "$SEED_DATA_DIR" \( -name '*_seed_corpus.zip' -o -name '*.options' -o -name '*.dict' \) \ | ||
| 11 | + ! \( -name '__base.*' \) -exec printf 'Copying: %s\n' {} \; \ | ||
| 12 | + -exec chmod a-x {} \; \ | ||
| 13 | + -exec cp {} "$OUT" \; | ||
| 14 | + | ||
| 15 | + # Build fuzzers in $OUT. | ||
| 16 | + find "$SRC" -name 'fuzz_*.py' -print0 | while IFS= read -r -d $'\0' fuzz_harness; do | ||
| 17 | + compile_python_fuzzer "$fuzz_harness" | ||
| 18 | + | ||
| 19 | + common_base_dictionary_filename="$SEED_DATA_DIR/__base.dict" | ||
| 20 | + if [[ -r "$common_base_dictionary_filename" ]]; then | ||
| 21 | + # Strip the `.py` extension from the filename and replace it with `.dict`. | ||
| 22 | + fuzz_harness_dictionary_filename="$(basename "$fuzz_harness" .py).dict" | ||
| 23 | + output_file="$OUT/$fuzz_harness_dictionary_filename" | ||
| 24 | + | ||
| 25 | + printf 'Appending %s to %s\n' "$common_base_dictionary_filename" "$output_file" | ||
| 26 | + if [[ -s "$output_file" ]]; then | ||
| 27 | + # If a dictionary file for this fuzzer already exists and is not empty, | ||
| 28 | + # we append a new line to the end of it before appending any new entries. | ||
| 29 | + # | ||
| 30 | + # libfuzzer will happily ignore multiple empty lines in a dictionary but crash | ||
| 31 | + # if any single line has incorrect syntax (e.g., if we accidentally add two entries to the same line.) | ||
| 32 | + # See docs for valid syntax: https://llvm.org/docs/LibFuzzer.html#id32 | ||
| 33 | + echo >>"$output_file" | ||
| 34 | + fi | ||
| 35 | + cat "$common_base_dictionary_filename" >>"$output_file" | ||
| 36 | + fi | ||
| 37 | + done | ||
| Back | FazBrowse Home | New Git URL |
0 commit comments