FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

Add capability to generate anonymous commercial averages as part of s… · datanerd/BenchmarkJava@5ec064e · GitHub

  1. datanerd
  2. BenchmarkJava

Repository navigation

Commit 5ec064e

Browse files
davewichers
committed
Add capability to generate anonymous commercial averages as part of scorecard generation.
And include such results for 6 commercial SAST tools against v1.1 of the Benchmark.
1 parent 2e71611 commit 5ec064e

47 files changed

Lines changed: 1149 additions & 403 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

scorecard/Benchmark_v1.2beta_Scorecard_for_Command_Injection.html renamed to scorecard/Benchmark_v1.1,1.2beta_Scorecard_for_Command_Injection.html

Lines changed: 36 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -33,36 +33,37 @@
3333
<nav class="navbar navbar-inverse navbar-fixed-top">
3434
<div class="container-fluid">
3535
<div class="navbar-header">
36-
<a class="navbar-brand" href="OWASP_Benchmark_Home.html">OWASP Benchmark v1.2beta</a>
36+
<a class="navbar-brand" href="OWASP_Benchmark_Home.html">OWASP Benchmark v1.1,1.2beta</a>
3737
</div>
3838
<div id="navbar" class="navbar-collapse collapse">
3939
<ul class="nav navbar-nav navbar-right">
4040
<li><a href="OWASP_Benchmark_Home.html">Home</a></li>
4141
<li class="dropdown">
4242
<a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Tools<span class="caret"></span></a>
4343
<ul class="dropdown-menu">
44-
<li><a href="Benchmark_v1.2beta_Scorecard_for_FBwFindSecBugs.html">FBwFindSecBugs</a></li>
45-
<li><a href="Benchmark_v1.2beta_Scorecard_for_FindBugs.html">FindBugs</a></li>
46-
<li><a href="Benchmark_v1.2beta_Scorecard_for_OWASP_ZAP.html">OWASP ZAP</a></li>
47-
<li><a href="Benchmark_v1.2beta_Scorecard_for_PMD.html">PMD</a></li>
48-
<li><a href="Benchmark_v1.2beta_Scorecard_for_SonarQube.html">SonarQube</a></li>
44+
<li><a href="Benchmark_v1.2beta_Scorecard_for_FBwFindSecBugs.html">FBwFindSecBugs</a></li>
45+
<li><a href="Benchmark_v1.2beta_Scorecard_for_FindBugs.html">FindBugs</a></li>
46+
<li><a href="Benchmark_v1.2beta_Scorecard_for_OWASP_ZAP.html">OWASP ZAP</a></li>
47+
<li><a href="Benchmark_v1.2beta_Scorecard_for_PMD.html">PMD</a></li>
48+
<li><a href="Benchmark_v1.2beta_Scorecard_for_SonarQube.html">SonarQube</a></li>
49+
<li><a href="Benchmark_v1.1,1.2beta_Scorecard_for_Commercial_Tools.html">Commercial Average</a></li>
4950

5051
</ul>
5152
</li>
5253
<li class="dropdown">
5354
<a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Vulnerabilities<span class="caret"></span></a>
5455
<ul class="dropdown-menu">
55-
<li><a href="Benchmark_v1.2beta_Scorecard_for_Command_Injection.html">Command Injection</a></li>
56-
<li><a href="Benchmark_v1.2beta_Scorecard_for_Cross-Site_Scripting.html">Cross-Site Scripting</a></li>
57-
<li><a href="Benchmark_v1.2beta_Scorecard_for_Insecure_Cookie.html">Insecure Cookie</a></li>
58-
<li><a href="Benchmark_v1.2beta_Scorecard_for_LDAP_Injection.html">LDAP Injection</a></li>
59-
<li><a href="Benchmark_v1.2beta_Scorecard_for_Path_Traversal.html">Path Traversal</a></li>
60-
<li><a href="Benchmark_v1.2beta_Scorecard_for_SQL_Injection.html">SQL Injection</a></li>
61-
<li><a href="Benchmark_v1.2beta_Scorecard_for_Trust_Boundary_Violation.html">Trust Boundary Violation</a></li>
62-
<li><a href="Benchmark_v1.2beta_Scorecard_for_Weak_Encryption_Algorithm.html">Weak Encryption Algorithm</a></li>
63-
<li><a href="Benchmark_v1.2beta_Scorecard_for_Weak_Hash_Algorithm.html">Weak Hash Algorithm</a></li>
64-
<li><a href="Benchmark_v1.2beta_Scorecard_for_Weak_Random_Number.html">Weak Random Number</a></li>
65-
<li><a href="Benchmark_v1.2beta_Scorecard_for_XPath_Injection.html">XPath Injection</a></li>
56+
<li><a href="Benchmark_v1.1,1.2beta_Scorecard_for_Command_Injection.html">Command Injection</a></li>
57+
<li><a href="Benchmark_v1.1,1.2beta_Scorecard_for_Cross-Site_Scripting.html">Cross-Site Scripting</a></li>
58+
<li><a href="Benchmark_v1.1,1.2beta_Scorecard_for_Insecure_Cookie.html">Insecure Cookie</a></li>
59+
<li><a href="Benchmark_v1.1,1.2beta_Scorecard_for_LDAP_Injection.html">LDAP Injection</a></li>
60+
<li><a href="Benchmark_v1.1,1.2beta_Scorecard_for_Path_Traversal.html">Path Traversal</a></li>
61+
<li><a href="Benchmark_v1.1,1.2beta_Scorecard_for_SQL_Injection.html">SQL Injection</a></li>
62+
<li><a href="Benchmark_v1.1,1.2beta_Scorecard_for_Trust_Boundary_Violation.html">Trust Boundary Violation</a></li>
63+
<li><a href="Benchmark_v1.1,1.2beta_Scorecard_for_Weak_Encryption_Algorithm.html">Weak Encryption Algorithm</a></li>
64+
<li><a href="Benchmark_v1.1,1.2beta_Scorecard_for_Weak_Hash_Algorithm.html">Weak Hash Algorithm</a></li>
65+
<li><a href="Benchmark_v1.1,1.2beta_Scorecard_for_Weak_Random_Number.html">Weak Random Number</a></li>
66+
<li><a href="Benchmark_v1.1,1.2beta_Scorecard_for_XPath_Injection.html">XPath Injection</a></li>
6667

6768
</ul>
6869
</li>
@@ -82,13 +83,27 @@
8283
<h3>OWASP Benchmark Scorecard for Command Injection</h3>
8384

8485
<p>The OWASP Benchmark is a test suite designed to evaluate the speed, coverage, and accuracy of automated vulnerability detection tools. Without the ability to measure these tools,
85-
it is difficult to understand their strengths and weaknesses, and compare them to each other. The Benchmark contains over 20,000 test cases that are fully runnable and exploitable.
86-
The following is the scorecard showing how well all the tools perform against Command Injection in version 1.2beta of the Benchmark. It shows how well each tool finds true positives and
86+
it is difficult to understand their strengths and weaknesses, and compare them to each other. The Benchmark contains thousands of test cases that are fully runnable and exploitable.
87+
The following is the scorecard showing how well all the tools perform against Command Injection in version 1.1,1.2beta of the Benchmark. It shows how well each tool finds true positives and
8788
avoids false positives for that type of vulnerability in the Benchmark test cases.</p>
8889

8990
<p>For more information, please visit the <a href="https://www.owasp.org/index.php/Benchmark">OWASP Benchmark Project Site</a>.
9091

91-
<img src='Benchmark_v1.2beta_Scorecard_for_Command_Injection.png' />
92+
<img src='Benchmark_v1.1,1.2beta_Scorecard_for_Command_Injection.png' />
93+
94+
<p>
95+
<p>
96+
97+
<h2>Detailed Results Per Tool for Command Injection</h2>
98+
<table class="table">
99+
<tr><th>Tool</th><th>Benchmark Version</th><th>TP</th><th>FN</th><th>TN</th><th>FP</th><th>Total</th><th>TPR</th><th>FPR</th><th>Score</th></tr>
100+
<tr class="danger"><td>FBwFindSecBugs</td><td>1.2beta</td><td>93</td><td>33</td><td>30</td><td>95</td><td>251</td><td>73.81%</td><td>76.00%</td><td>-2.19%</td></tr>
101+
<tr class="danger"><td>FindBugs</td><td>1.2beta</td><td>0</td><td>126</td><td>125</td><td>0</td><td>251</td><td>0.00%</td><td>0.00%</td><td>0.00%</td></tr>
102+
<tr ><td>OWASP ZAP</td><td>1.2beta</td><td>44</td><td>82</td><td>125</td><td>0</td><td>251</td><td>34.92%</td><td>0.00%</td><td>34.92%</td></tr>
103+
<tr class="danger"><td>PMD</td><td>1.2beta</td><td>0</td><td>126</td><td>125</td><td>0</td><td>251</td><td>0.00%</td><td>0.00%</td><td>0.00%</td></tr>
104+
<tr class="danger"><td>SonarQube</td><td>1.2beta</td><td>107</td><td>19</td><td>16</td><td>109</td><td>251</td><td>84.92%</td><td>87.20%</td><td>-2.28%</td></tr>
105+
</tr>
106+
</table>
92107

93108
<p>
94109
<p>
@@ -117,7 +132,7 @@ <h2>Key</h2>
117132

118133
<tr>
119134
<th>True Positive Rate (TPR) = TP / ( TP + FN )</th>
120-
<td>The rate at which the tool correctly reports real vulnerabilities. Also referred to as Precision, as defined at
135+
<td>The rate at which the tool correctly reports real vulnerabilities. Also referred to as Recall, as defined at
121136
<a href="https://en.wikipedia.org/wiki/Precision_and_recall">Wikipedia</a>.</td>
122137
</tr>
123138

‎scorecard/Benchmark_v1.1,1.2beta_Scorecard_for_Commercial_Tools.html‎

Lines changed: 173 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,173 @@
1+
<!DOCTYPE html>
2+
<html lang="en">
3+
<head>
4+
<meta charset="utf-8">
5+
<meta http-equiv="X-UA-Compatible" content="IE=edge">
6+
<meta name="viewport" content="width=device-width, initial-scale=1">
7+
<!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
8+
<meta name="description" content="">
9+
<meta name="author" content="">
10+
<link rel="icon" href="../../favicon.ico">
11+
12+
<title>Commercial Tools Scorecard</title>
13+
14+
<!-- Bootstrap core CSS -->
15+
<link href="content/css/bootstrap.min.css" rel="stylesheet">
16+
17+
<!-- Custom styles for this template -->
18+
<link href="content/dashboard.css" rel="stylesheet">
19+
20+
<!-- Just for debugging purposes. Don't actually copy these 2 lines! -->
21+
<!--[if lt IE 9]><script src="../../assets/js/ie8-responsive-file-warning.js"></script><![endif]-->
22+
<script src="content/js/ie-emulation-modes-warning.js"></script>
23+
24+
<!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
25+
<!--[if lt IE 9]>
26+
<script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
27+
<script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
28+
<![endif]-->
29+
</head>
30+
31+
<body>
32+
33+
<nav class="navbar navbar-inverse navbar-fixed-top">
34+
<div class="container-fluid">
35+
<div class="navbar-header">
36+
<a class="navbar-brand" href="OWASP_Benchmark_Home.html">OWASP Benchmark v1.1,1.2beta</a>
37+
</div>
38+
<div id="navbar" class="navbar-collapse collapse">
39+
<ul class="nav navbar-nav navbar-right">
40+
<li><a href="OWASP_Benchmark_Home.html">Home</a></li>
41+
<li class="dropdown">
42+
<a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Tools<span class="caret"></span></a>
43+
<ul class="dropdown-menu">
44+
<li><a href="Benchmark_v1.2beta_Scorecard_for_FBwFindSecBugs.html">FBwFindSecBugs</a></li>
45+
<li><a href="Benchmark_v1.2beta_Scorecard_for_FindBugs.html">FindBugs</a></li>
46+
<li><a href="Benchmark_v1.2beta_Scorecard_for_OWASP_ZAP.html">OWASP ZAP</a></li>
47+
<li><a href="Benchmark_v1.2beta_Scorecard_for_PMD.html">PMD</a></li>
48+
<li><a href="Benchmark_v1.2beta_Scorecard_for_SonarQube.html">SonarQube</a></li>
49+
<li><a href="Benchmark_v1.1,1.2beta_Scorecard_for_Commercial_Tools.html">Commercial Average</a></li>
50+
51+
</ul>
52+
</li>
53+
<li class="dropdown">
54+
<a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Vulnerabilities<span class="caret"></span></a>
55+
<ul class="dropdown-menu">
56+
<li><a href="Benchmark_v1.1,1.2beta_Scorecard_for_Command_Injection.html">Command Injection</a></li>
57+
<li><a href="Benchmark_v1.1,1.2beta_Scorecard_for_Cross-Site_Scripting.html">Cross-Site Scripting</a></li>
58+
<li><a href="Benchmark_v1.1,1.2beta_Scorecard_for_Insecure_Cookie.html">Insecure Cookie</a></li>
59+
<li><a href="Benchmark_v1.1,1.2beta_Scorecard_for_LDAP_Injection.html">LDAP Injection</a></li>
60+
<li><a href="Benchmark_v1.1,1.2beta_Scorecard_for_Path_Traversal.html">Path Traversal</a></li>
61+
<li><a href="Benchmark_v1.1,1.2beta_Scorecard_for_SQL_Injection.html">SQL Injection</a></li>
62+
<li><a href="Benchmark_v1.1,1.2beta_Scorecard_for_Trust_Boundary_Violation.html">Trust Boundary Violation</a></li>
63+
<li><a href="Benchmark_v1.1,1.2beta_Scorecard_for_Weak_Encryption_Algorithm.html">Weak Encryption Algorithm</a></li>
64+
<li><a href="Benchmark_v1.1,1.2beta_Scorecard_for_Weak_Hash_Algorithm.html">Weak Hash Algorithm</a></li>
65+
<li><a href="Benchmark_v1.1,1.2beta_Scorecard_for_Weak_Random_Number.html">Weak Random Number</a></li>
66+
<li><a href="Benchmark_v1.1,1.2beta_Scorecard_for_XPath_Injection.html">XPath Injection</a></li>
67+
68+
</ul>
69+
</li>
70+
<li><a href="OWASP_Benchmark_Guide.html">Guide</a></li>
71+
</ul>
72+
</div>
73+
</div>
74+
</nav>
75+
76+
<div class="container">
77+
78+
<div class="starter-template">
79+
80+
<div>empty</div>
81+
<div>empty</div>
82+
83+
<h3>OWASP Benchmark Scorecard for Commercial Tools</h3>
84+
85+
<p>The OWASP Benchmark is a test suite designed to evaluate the speed, coverage, and accuracy of automated vulnerability detection tools. Without the ability to measure these tools,
86+
it is difficult to understand their strengths and weaknesses, and compare them to each other. The Benchmark contains thousands of test cases that are fully runnable and exploitable.
87+
The following is the scorecard showing how well the commercial tools collectively performed against version 1.1,1.2beta of the Benchmark. For each vulnerability
88+
it shows the lowest, average, and highest scores across all the commercial tools included in this scorecard calculation.</p>
89+
90+
<p>For more information, please visit the <a href="https://www.owasp.org/index.php/Benchmark">OWASP Benchmark Project Site</a>.
91+
92+
<p>
93+
<p>
94+
95+
<h2>Average Scores Per Vulnerability for Commercial Tools</h2>
96+
<table class="table">
97+
<tr><th>Vulnerability Category</th><th>Low Tool Type</th><th>Low Score</th><th>Ave Score</th><th>High Score</th><th>High Tool Type</th></tr>
98+
<tr><td>Command Injection</td><td>SAST</td><td class="danger">0</td><td>16</td><td class="danger">25</td><td>SAST</td></tr>
99+
<tr><td>Cross-Site Scripting</td><td>SAST</td><td class="danger">9</td><td>22</td><td class="danger">39</td><td>SAST</td></tr>
100+
<tr><td>Insecure Cookie</td><td>SAST</td><td class="danger">0</td><td>33</td><td class="success">100</td><td>SAST</td></tr>
101+
<tr><td>LDAP Injection</td><td>SAST</td><td class="danger">0</td><td>17</td><td class="success">54</td><td>SAST</td></tr>
102+
<tr><td>Path Traversal</td><td>SAST</td><td class="danger">1</td><td>18</td><td class="danger">30</td><td>SAST</td></tr>
103+
<tr><td>SQL Injection</td><td>SAST</td><td class="danger">9</td><td>24</td><td class="danger">34</td><td>SAST</td></tr>
104+
<tr><td>Trust Boundary Violation</td><td>SAST</td><td class="danger">0</td><td>8</td><td class="danger">16</td><td>SAST</td></tr>
105+
<tr><td>Weak Encryption Algorithm</td><td>SAST</td><td class="danger">0</td><td>39</td><td class="success">74</td><td>SAST</td></tr>
106+
<tr><td>Weak Hash Algorithm</td><td>SAST</td><td class="danger">0</td><td>42</td><td class="success">77</td><td>SAST</td></tr>
107+
<tr><td>Weak Random Number</td><td>SAST</td><td class="danger">0</td><td>36</td><td class="success">90</td><td>SAST</td></tr>
108+
<tr><td>XPath Injection</td><td>SAST</td><td class="danger">0</td><td>27</td><td class="success">59</td><td>SAST</td></tr>
109+
<tr><td>Average across all categories for 6 tools</td><td></td><td>1.7</td><td>25.6</td><td>54.4</td><td></td></tr>
110+
111+
112+
<p>
113+
<p>
114+
115+
<h2>Key</h2>
116+
<table class="table">
117+
<tr>
118+
<th>Tool Type</th>
119+
<td>SAST - Static Application Security Testing. DAST - Dynamic Application Security Testing.
120+
IAST - Interactive Application Security Testing. These terms were coined by Gartner.</td>
121+
</tr>
122+
123+
<tr>
124+
<th>True Positive (TP)</th>
125+
<td>Tests with real vulnerabilities that were correctly reported as vulnerable by the tool.</td>
126+
</tr>
127+
128+
<tr>
129+
<th>False Negative (FN)</th>
130+
<td>Tests with real vulnerabilities that were not correctly reported as vulnerable by the tool.</td>
131+
</tr>
132+
133+
<tr>
134+
<th>True Negative (TN)</th>
135+
<td>Tests with fake vulnerabilities that were correctly not reported as vulnerable by the tool.</td>
136+
</tr>
137+
138+
<tr>
139+
<th>False Positive (FP)</th>
140+
<td>Tests with fake vulnerabilities that were incorrectly reported as vulnerable by the tool.</td>
141+
</tr>
142+
143+
<tr>
144+
<th>True Positive Rate (TPR) = TP / ( TP + FN )</th>
145+
<td>The rate at which the tool correctly reports real vulnerabilities. Also referred to as Recall, as defined at
146+
<a href="https://en.wikipedia.org/wiki/Precision_and_recall">Wikipedia</a>.</td>
147+
</tr>
148+
149+
<tr>
150+
<th>False Positive Rate (FPR) = FP / ( FP + TN )</th>
151+
<td>The rate at which the tool incorrectly reports fake vulnerabilities as real.</td>
152+
</tr>
153+
154+
<tr>
155+
<th>Score = TPR - FPR</th>
156+
<td>Normalized distance from the random guess line.</td>
157+
</tr>
158+
159+
</table>
160+
161+
</div>
162+
163+
</div><!-- /.container -->
164+
165+
<!-- Bootstrap core JavaScript
166+
================================================== -->
167+
<!-- jQuery (necessary for Bootstrap's JavaScript plugins) -->
168+
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>
169+
<!-- Include all compiled plugins (below), or include individual files as needed -->
170+
<script src="content/js/bootstrap.min.js"></script>
171+
</body>
172+
</html>
173+

0 commit comments

Comments
 (0)
Back | FazBrowse Home | New Git URL