FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

fix: make WrappedExpiringStorage finalizable and jitter expiry sweeps by jeswr · Pull Request #2224 · CommunitySolidServer/CommunitySolidServer · GitHub

Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension .json  (4) .md  (1) .ts  (2) All 3 file types selected
Viewed files
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Unified
Split
Hide whitespace
Diff view
Unified
Split
Hide whitespace
5 changes: 5 additions & 0 deletions RELEASE_NOTES.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,9 @@ The following changes pertain to the imports in the default configs:
- There is a new import option for `storage/middleware`: `cache.json`, which adds caching for backend resources.
All default configurations have been changed to use that option.
Servers using worker threads can not use this option.
- Default identity configurations now register their `WrappedExpiringStorage` instances with the `Finalizer`,
ensuring the cleanup timers are stopped during shutdown.
Custom configurations should register such instances as well.

The following changes are relevant for v7 custom configs that replaced certain features.

Expand All @@ -54,6 +57,8 @@ These changes are relevant if you wrote custom modules for the server that depen
- Due to several features being extracted, all related classes are impacted.
- The handler changes discussed above also impact the code if you depended on these.
- The `DeleteParentExtractor` class has been removed.
- `WrappedExpiringStorage` now implements `Finalizable` and accepts an optional jitter fraction as its third argument.
Cleanup intervals use up to 15% jitter by default; pass `0` to retain a fixed interval.

## v7.0.0

Expand Down
17 changes: 16 additions & 1 deletion config/identity/handler/base/adapter-factory.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
{
"@context": "https://linkedsoftwaredependencies.org/bundles/npm/@solid/community-server/^8.0.0/components/context.jsonld",
"@context": [
"https://linkedsoftwaredependencies.org/bundles/npm/@solid/community-server/^8.0.0/components/context.jsonld",
"https://linkedsoftwaredependencies.org/bundles/npm/asynchronous-handlers/^1.0.0/components/context.jsonld"
],
"@graph": [
{
"comment": "An adapter is responsible for storing all interaction metadata.",
Expand All @@ -13,6 +16,7 @@
"source": {
"@type": "ExpiringAdapterFactory",
"storage": {
"@id": "urn:solid-server:default:IdpAdapterExpiringStorage",
"@type": "WrappedExpiringStorage",
"source": {
"@type": "ContainerPathStorage",
Expand All @@ -24,6 +28,17 @@
}
}
}
},
{
"comment": "Makes sure the expiring storage cleanup timer is stopped when the application needs to stop.",
"@id": "urn:solid-server:default:Finalizer",
"@type": "ParallelHandler",
"handlers": [
{
"@type": "FinalizableHandler",
"finalizable": { "@id": "urn:solid-server:default:IdpAdapterExpiringStorage" }
}
]
}
]
}
11 changes: 11 additions & 0 deletions config/identity/handler/storage/default.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,17 @@
}
}
},
{
"comment": "Makes sure the expiring storage cleanup timer is stopped when the application needs to stop.",
"@id": "urn:solid-server:default:Finalizer",
"@type": "ParallelHandler",
"handlers": [
{
"@type": "FinalizableHandler",
"finalizable": { "@id": "urn:solid-server:default:CookieStorage" }
}
]
},

{
"@id": "urn:solid-server:default:PodStore",
Expand Down
11 changes: 11 additions & 0 deletions config/identity/handler/storage/password.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,17 @@
"source": { "@id": "urn:solid-server:default:KeyValueStorage" }
}
}
},
{
"comment": "Makes sure the expiring storage cleanup timer is stopped when the application needs to stop.",
"@id": "urn:solid-server:default:Finalizer",
"@type": "ParallelHandler",
"handlers": [
{
"@type": "FinalizableHandler",
"finalizable": { "@id": "urn:solid-server:default:ForgotPasswordStorage" }
}
]
}
]
}
16 changes: 15 additions & 1 deletion config/identity/ownership/token.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
{
"@context": "https://linkedsoftwaredependencies.org/bundles/npm/@solid/community-server/^8.0.0/components/context.jsonld",
"@context": [
"https://linkedsoftwaredependencies.org/bundles/npm/@solid/community-server/^8.0.0/components/context.jsonld",
"https://linkedsoftwaredependencies.org/bundles/npm/asynchronous-handlers/^1.0.0/components/context.jsonld"
],
"@graph": [
{
"comment": "Determines WebID ownership by requesting a specific value to be added to the WebID document",
Expand All @@ -17,6 +20,17 @@
"relativePath": "/idp/tokens/",
"source": { "@id": "urn:solid-server:default:KeyValueStorage" }
}
},
{
"comment": "Makes sure the expiring storage cleanup timer is stopped when the application needs to stop.",
"@id": "urn:solid-server:default:Finalizer",
"@type": "ParallelHandler",
"handlers": [
{
"@type": "FinalizableHandler",
"finalizable": { "@id": "urn:solid-server:default:ExpiringTokenStorage" }
}
]
}
]
}
16 changes: 13 additions & 3 deletions src/storage/keyvalue/WrappedExpiringStorage.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import { getLoggerFor } from 'global-logger-factory';
import type { Finalizable } from '../../init/final/Finalizable';
import { InternalServerError } from '../../util/errors/InternalServerError';
import { setSafeInterval } from '../../util/TimerUtil';
import type { ExpiringStorage } from './ExpiringStorage';
Expand All @@ -11,23 +12,28 @@ export type Expires<T> = { expires?: string; payload: T };
* A storage that wraps around another storage and expires resources based on the given (optional) expiry date.
* Will delete expired entries when trying to get their value.
* Has a timer that will delete all expired data every hour (default value).
* A random jitter is added to this interval so instances created together do not all sweep at the same time.
* The timer is cleared when the storage is finalized.
*/
export class WrappedExpiringStorage<TKey, TValue> implements ExpiringStorage<TKey, TValue> {
export class WrappedExpiringStorage<TKey, TValue> implements ExpiringStorage<TKey, TValue>, Finalizable {
protected readonly logger = getLoggerFor(this);
private readonly source: KeyValueStorage<TKey, Expires<TValue>>;
private readonly timer: NodeJS.Timeout;

/**
* @param source - KeyValueStorage to actually store the data.
* @param timeout - How often the expired data needs to be checked in minutes.
* @param jitter - Maximum fraction of the timeout that is randomly added to the interval. `0` disables jitter.
*/
public constructor(source: KeyValueStorage<TKey, Expires<TValue>>, timeout = 60) {
public constructor(source: KeyValueStorage<TKey, Expires<TValue>>, timeout = 60, jitter = 0.15) {
this.source = source;
const period = timeout * 60 * 1000;
const jitterMs = Math.floor(Math.random() * period * jitter);
Comment thread
jeswr marked this conversation as resolved.
this.timer = setSafeInterval(
this.logger,
'Failed to remove expired entries',
this.removeExpiredEntries.bind(this),
timeout * 60 * 1000,
period + jitterMs,
);
this.timer.unref();
}
Expand Down Expand Up @@ -123,4 +129,8 @@ export class WrappedExpiringStorage<TKey, TValue> implements ExpiringStorage<TKe
}
return result;
}

public async finalize(): Promise<void> {
clearInterval(this.timer);
}
}
95 changes: 63 additions & 32 deletions test/unit/storage/keyvalue/WrappedExpiringStorage.test.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -117,41 +117,72 @@ describe('A WrappedExpiringStorage', (): void => {
);
});

it('removes expired entries after a given time.', async(): Promise<void> => {
// Disable interval function and simply check it was called with the correct parameters
// Otherwise it gets quite difficult to verify the async interval function gets executed
const mockInterval = jest.spyOn(globalThis, 'setInterval');

// We only need to call the timer.unref() once when the object is created
const mockTimer = { unref: jest.fn() };
const mockFn = jest.fn().mockReturnValueOnce(mockTimer);
mockInterval.mockImplementationOnce(mockFn);

// Timeout of 1 minute
storage = new WrappedExpiringStorage(source, 1);
const data = [
[ 'key1', createExpires('data1', tomorrow) ],
[ 'key2', createExpires('data2', yesterday) ],
[ 'key3', createExpires('data3') ],
];
source.entries.mockImplementationOnce(function* (): any {
yield* data;
describe('scheduling the cleanup timer', (): void => {
// Disable the actual interval and simply check it was created with the correct parameters.
// Otherwise it gets quite difficult to verify the async interval function gets executed.
let mockInterval: jest.SpyInstance;
let mockClear: jest.SpyInstance;
let mockRandom: jest.SpyInstance;
// We only need a stub timer with an `unref` function since we never let it fire on its own.
let mockTimer: { unref: jest.Mock };

beforeEach((): void => {
mockTimer = { unref: jest.fn() };
mockInterval = jest.spyOn(globalThis, 'setInterval')
.mockImplementation(jest.fn().mockReturnValue(mockTimer));
mockClear = jest.spyOn(globalThis, 'clearInterval').mockImplementation(jest.fn());
// Fixed jitter source so the scheduled delay is deterministic.
mockRandom = jest.spyOn(globalThis.Math, 'random').mockReturnValue(0.5);
});

// Make sure interval is created correctly
expect(mockInterval.mock.calls).toHaveLength(1);
expect(mockInterval.mock.calls[0]).toHaveLength(2);
expect(mockInterval.mock.calls[0][1]).toBe(60 * 1000);
afterEach((): void => {
mockInterval.mockRestore();
mockClear.mockRestore();
mockRandom.mockRestore();
});

// Await the function that should have been executed by the interval
await (mockInterval.mock.calls[0][0] as () => Promise<void>)();
it('schedules the sweep on the configured timeout when jitter is disabled.', (): void => {
storage = new WrappedExpiringStorage(source, 1, 0);
expect(mockInterval).toHaveBeenCalledTimes(1);
expect(mockInterval.mock.calls[0]).toHaveLength(2);
expect(mockInterval.mock.calls[0][1]).toBe(60 * 1000);
});

// Make sure timer.unref() is called on initialization
expect(mockTimer.unref).toHaveBeenCalledTimes(1);
// Make sure setSafeInterval has been called once as well
expect(mockFn).toHaveBeenCalledTimes(1);
expect(source.delete).toHaveBeenCalledTimes(1);
expect(source.delete).toHaveBeenLastCalledWith('key2');
mockInterval.mockRestore();
it('adds a jitter fraction to the scheduled sweep interval.', (): void => {
// Math.random is 0.5 and jitter is 0.2, so floor(0.5 * 60000 * 0.2) = 6000 is added.
storage = new WrappedExpiringStorage(source, 1, 0.2);
expect(mockInterval).toHaveBeenCalledTimes(1);
expect(mockInterval.mock.calls[0][1]).toBe(60 * 1000 + 6000);
});

it('unrefs the timer so it does not keep the event loop alive.', (): void => {
storage = new WrappedExpiringStorage(source, 1, 0);
expect(mockTimer.unref).toHaveBeenCalledTimes(1);
});

it('removes expired entries when the scheduled sweep fires.', async(): Promise<void> => {
storage = new WrappedExpiringStorage(source, 1, 0);
const data = [
[ 'key1', createExpires('data1', tomorrow) ],
[ 'key2', createExpires('data2', yesterday) ],
[ 'key3', createExpires('data3') ],
];
source.entries.mockImplementationOnce(function* (): any {
yield* data;
});

// Await the function that should have been executed by the interval.
await (mockInterval.mock.calls[0][0] as () => Promise<void>)();

expect(source.delete).toHaveBeenCalledTimes(1);
expect(source.delete).toHaveBeenLastCalledWith('key2');
});

it('clears the timer on finalize.', async(): Promise<void> => {
storage = new WrappedExpiringStorage(source, 1, 0);
await expect(storage.finalize()).resolves.toBeUndefined();
expect(mockClear).toHaveBeenCalledTimes(1);
expect(mockClear).toHaveBeenLastCalledWith(mockTimer);
});
});
});
Loading

Back | FazBrowse Home | New Git URL