Benchmarks

In this document you will find benchmark results for different endpoints of Ory Hydra. All benchmarks are executed using rakyll/hey. Please note that these benchmarks run against the in-memory storage adapter of Ory Hydra. These benchmarks represent what performance you would get with a zero-overhead database implementation.

We don't include benchmarks against databases (for example MySQL, PostgreSQL, SQLite or CockroachDB) as the performance greatly differs between deployments (for example request latency, database configuration) and tweaking individual things may greatly improve performance. We believe, for that reason, that benchmark results for these database adapters are difficult to generalize and potentially deceiving. They're thus not included.

This file is updated on every push to master. It thus represents the benchmark data for the latest version.

All benchmarks run 10.000 requests in total, with 100 concurrent requests. All benchmarks run on Circle-CI with a "2 CPU cores and 4GB RAM" configuration.

BCrypt

Ory Hydra uses BCrypt to obfuscate secrets of OAuth 2.0 Clients. When using flows such as the OAuth 2.0 Client Credentials Grant, Ory Hydra validates the client credentials using BCrypt which causes (by design) CPU load. CPU load and performance depend on the BCrypt cost which can be set using the environment variable BCRYPT_COST. For these benchmarks, we've set BCRYPT_COST=8.

OAuth 2.0

This section contains various benchmarks against OAuth 2.0 endpoints

Token introspection

Summary:
  Total:    1.9573 secs
  Slowest:  0.1461 secs
  Fastest:  0.0002 secs
  Average:  0.0186 secs
  Requests/sec: 5108.9654

  Total data:   1550000 bytes
  Size/request: 155 bytes

Response time histogram:
  0.000 [1] |
  0.015 [5258]  |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.029 [2339]  |■■■■■■■■■■■■■■■■■■
  0.044 [1325]  |■■■■■■■■■■
  0.059 [651]   |■■■■■
  0.073 [261]   |■■
  0.088 [117]   |■
  0.102 [33]    |
  0.117 [11]    |
  0.131 [2] |
  0.146 [2] |


Latency distribution:
  10% in 0.0006 secs
  25% in 0.0015 secs
  50% in 0.0133 secs
  75% in 0.0284 secs
  90% in 0.0454 secs
  95% in 0.0563 secs
  99% in 0.0799 secs

Details (average, fastest, slowest):
  DNS+dialup:   0.0000 secs, 0.0002 secs, 0.1461 secs
  DNS-lookup:   0.0000 secs, 0.0000 secs, 0.0049 secs
  req write:    0.0000 secs, 0.0000 secs, 0.0052 secs
  resp wait:    0.0185 secs, 0.0002 secs, 0.1460 secs
  resp read:    0.0000 secs, 0.0000 secs, 0.0048 secs

Status code distribution:
  [200] 10000 responses

Client credentials grant

This endpoint uses BCrypt.

Summary:
  Total:    19.4831 secs
  Slowest:  0.9063 secs
  Fastest:  0.0171 secs
  Average:  0.1861 secs
  Requests/sec: 513.2648

  Total data:   1570000 bytes
  Size/request: 157 bytes

Response time histogram:
  0.017 [1] |
  0.106 [3566]  |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.195 [2358]  |■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.284 [1850]  |■■■■■■■■■■■■■■■■■■■■■
  0.373 [1146]  |■■■■■■■■■■■■■
  0.462 [650]   |■■■■■■■
  0.551 [288]   |■■■
  0.640 [101]   |■
  0.728 [27]    |
  0.817 [11]    |
  0.906 [2] |


Latency distribution:
  10% in 0.0318 secs
  25% in 0.0922 secs
  50% in 0.1736 secs
  75% in 0.2728 secs
  90% in 0.3815 secs
  95% in 0.4191 secs
  99% in 0.5863 secs

Details (average, fastest, slowest):
  DNS+dialup:   0.0001 secs, 0.0171 secs, 0.9063 secs
  DNS-lookup:   0.0000 secs, 0.0000 secs, 0.0432 secs
  req write:    0.0001 secs, 0.0000 secs, 0.0615 secs
  resp wait:    0.1857 secs, 0.0171 secs, 0.9062 secs
  resp read:    0.0001 secs, 0.0000 secs, 0.0302 secs

Status code distribution:
  [200] 10000 responses

OAuth 2.0 client management

Creating OAuth 2.0 clients

This endpoint uses BCrypt and generates IDs and secrets by reading from which negatively impacts performance. Performance will be better if IDs and secrets are set in the request as opposed to generated by Ory Hydra.

This test is disabled due to issues with /dev/urandom being inaccessible in the CI.

Listing OAuth 2.0 clients

Summary:
  Total:    0.6655 secs
  Slowest:  0.0337 secs
  Fastest:  0.0001 secs
  Average:  0.0062 secs
  Requests/sec: 15025.1907

  Total data:   4880000 bytes
  Size/request: 488 bytes

Response time histogram:
  0.000 [1] |
  0.003 [4995]  |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.007 [960]   |■■■■■■■■
  0.010 [1250]  |■■■■■■■■■■
  0.014 [843]   |■■■■■■■
  0.017 [988]   |■■■■■■■■
  0.020 [619]   |■■■■■
  0.024 [243]   |■■
  0.027 [80]    |■
  0.030 [18]    |
  0.034 [3] |


Latency distribution:
  10% in 0.0002 secs
  25% in 0.0002 secs
  50% in 0.0035 secs
  75% in 0.0115 secs
  90% in 0.0167 secs
  95% in 0.0190 secs
  99% in 0.0236 secs

Details (average, fastest, slowest):
  DNS+dialup:   0.0000 secs, 0.0001 secs, 0.0337 secs
  DNS-lookup:   0.0000 secs, 0.0000 secs, 0.0064 secs
  req write:    0.0000 secs, 0.0000 secs, 0.0057 secs
  resp wait:    0.0061 secs, 0.0001 secs, 0.0306 secs
  resp read:    0.0001 secs, 0.0000 secs, 0.0038 secs

Status code distribution:
  [200] 10000 responses

Fetching a specific OAuth 2.0 Client

Summary:
  Total:    0.5859 secs
  Slowest:  0.0398 secs
  Fastest:  0.0001 secs
  Average:  0.0055 secs
  Requests/sec: 17067.5827

  Total data:   4860000 bytes
  Size/request: 486 bytes

Response time histogram:
  0.000 [1] |
  0.004 [5492]  |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.008 [1579]  |■■■■■■■■■■■■
  0.012 [1105]  |■■■■■■■■
  0.016 [923]   |■■■■■■■
  0.020 [595]   |■■■■
  0.024 [178]   |■
  0.028 [87]    |■
  0.032 [29]    |
  0.036 [7] |
  0.040 [4] |


Latency distribution:
  10% in 0.0002 secs
  25% in 0.0002 secs
  50% in 0.0018 secs
  75% in 0.0093 secs
  90% in 0.0155 secs
  95% in 0.0184 secs
  99% in 0.0250 secs

Details (average, fastest, slowest):
  DNS+dialup:   0.0000 secs, 0.0001 secs, 0.0398 secs
  DNS-lookup:   0.0000 secs, 0.0000 secs, 0.0063 secs
  req write:    0.0000 secs, 0.0000 secs, 0.0066 secs
  resp wait:    0.0053 secs, 0.0001 secs, 0.0398 secs
  resp read:    0.0001 secs, 0.0000 secs, 0.0101 secs

Status code distribution:
  [200] 10000 responses