FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

chore: add fast crc32c concat implementation by BenWhitehead · Pull Request #3148 · googleapis/java-storage · GitHub

chore: add fast crc32c concat implementation - #3148

Merged
BenWhitehead merged 1 commit into
mainfrom
checksumming/01/fast-crc32c-concat
Jun 24, 2025
Merged

chore: add fast crc32c concat implementation#3148
BenWhitehead merged 1 commit into
mainfrom
checksumming/01/fast-crc32c-concat

Conversation

Copy link
Copy Markdown
Collaborator

This new implementation provides a constant time computation regardless of the length of B, rather than the previous implementation which took linear time relative to the length of B.

Benchmark summary

CPU: AMD Ryzen Threadripper PRO 3945WX 12-Cores
OS: Debian Testing

# JMH version: 1.37
# VM version: JDK 11.0.17, OpenJDK 64-Bit Server VM, 11.0.17+8
# VM invoker: /home/benwhitehead/opt/java/jdk-11.0.17+8/bin/java
# VM options: -Xms1g -Xmx1g -XX:+AlwaysPreTouch
# Blackhole mode: full + dont-inline hint (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
# Warmup: 4 iterations, 30 s each
# Measurement: 8 iterations, 30 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Sampling time
Benchmark                                  (objectSize)    Mode      Cnt     Score    Error  Units
Crc32cBenchmark.concatCrc32c                     131072  sample  9243752     0.074 ±  0.001  us/op
Crc32cBenchmark.concatCrc32c                     262144  sample  9015106     0.078 ±  0.001  us/op
Crc32cBenchmark.concatCrc32c                     524288  sample  9010257     0.074 ±  0.001  us/op
Crc32cBenchmark.concatCrc32c                    1048576  sample  9165519     0.074 ±  0.001  us/op
Crc32cBenchmark.concatCrc32c                    2097152  sample  8993038     0.075 ±  0.001  us/op
Crc32cBenchmark.crc32cCombineGoogle              131072  sample  2070639   115.806 ±  0.018  us/op
Crc32cBenchmark.crc32cCombineGoogle              262144  sample  1029209   233.019 ±  0.045  us/op
Crc32cBenchmark.crc32cCombineGoogle              524288  sample   516517   464.360 ±  0.106  us/op
Crc32cBenchmark.crc32cCombineGoogle             1048576  sample   262118   915.067 ±  0.354  us/op
Crc32cBenchmark.crc32cCombineGoogle             2097152  sample   131385  1825.685 ±  0.834  us/op

This new implementation provides a constant time computation regardless of the length of B, rather than the previous implementation which took linear time relative to the length of B.

#### Benchmark summary

CPU: AMD Ryzen Threadripper PRO 3945WX 12-Cores
OS: Debian Testing
```
# JMH version: 1.37
# VM version: JDK 11.0.17, OpenJDK 64-Bit Server VM, 11.0.17+8
# VM invoker: /home/benwhitehead/opt/java/jdk-11.0.17+8/bin/java
# VM options: -Xms1g -Xmx1g -XX:+AlwaysPreTouch
# Blackhole mode: full + dont-inline hint (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
# Warmup: 4 iterations, 30 s each
# Measurement: 8 iterations, 30 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Sampling time
```

```
Benchmark                                  (objectSize)    Mode      Cnt     Score    Error  Units
Crc32cBenchmark.concatCrc32c                     131072  sample  9243752     0.074 ±  0.001  us/op
Crc32cBenchmark.concatCrc32c                     262144  sample  9015106     0.078 ±  0.001  us/op
Crc32cBenchmark.concatCrc32c                     524288  sample  9010257     0.074 ±  0.001  us/op
Crc32cBenchmark.concatCrc32c                    1048576  sample  9165519     0.074 ±  0.001  us/op
Crc32cBenchmark.concatCrc32c                    2097152  sample  8993038     0.075 ±  0.001  us/op
Crc32cBenchmark.crc32cCombineGoogle              131072  sample  2070639   115.806 ±  0.018  us/op
Crc32cBenchmark.crc32cCombineGoogle              262144  sample  1029209   233.019 ±  0.045  us/op
Crc32cBenchmark.crc32cCombineGoogle              524288  sample   516517   464.360 ±  0.106  us/op
Crc32cBenchmark.crc32cCombineGoogle             1048576  sample   262118   915.067 ±  0.354  us/op
Crc32cBenchmark.crc32cCombineGoogle             2097152  sample   131385  1825.685 ±  0.834  us/op
```
BenWhitehead requested a review from a team June 13, 2025 21:23
product-auto-label Bot added size: l Pull request size is large. api: storage Issues related to the googleapis/java-storage API. labels Jun 13, 2025
BenWhitehead requested a review from BrandonY June 13, 2025 21:27
BenWhitehead merged commit b956eb9 into main Jun 24, 2025
BenWhitehead deleted the checksumming/01/fast-crc32c-concat branch June 24, 2025 19:35
BenWhitehead added this to the e2e-crc32c-media milestone Jun 30, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: storage Issues related to the googleapis/java-storage API. size: l Pull request size is large.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants


Back | FazBrowse Home | New Git URL