213 Commits

Author SHA1 Message Date
Wout Slabbinck
ca62511d12
feat: Edit metadata resources (#1188)
* feat: only allow metadata to be edited with PATCH request & only allow metadata files to be edited when a resource is available

* fix: remove unnecesary log at POST

* feat: PUT resets metadata contents + not possible to add metadata with PUT to container

* feat: add metadataStrategy (auxiliaryStrategy) + use that strategy in operationhandlers

* feat: PUT request on existing LDPC is not allowed as it would be possible to edit (read reset) metadata

* test: add unit tests to operationhandlers to handle metadata editing

* test: add unit tests to representationPatchHandler to handle metadata editing

* fix: update dependency of meta.json to version 3.0.0

* fix: lint and dependency still v2

* fix: replaced file references to resource references + moved Patch check to new patchhandler which is more generic

* fix: moved checking metadata resources checking from DELETE and POST handler down to DABS

* fix: remove PATCH message about metadata extension

* fix: move PATCH message about metadata extension

* WIP: adding writeMetadata + getMetadata in DABS and add writeMetadata to DataAccessors (part 1)

* WIP: implement writeMetadata in memorybackend + change resourceExists

* WIP: implement writeMetadata in SparqlDataAccessor.ts

* test: fix test interfaces

* test: InMemoryDataAccessor.ts resulted into changing identifier for writeMetadata in DataAccessor.ts (now taking subject identifier instead of metadata resource identifier)

* test: accessor tests implemented for metadata

* test: add RdfImmutableCheckPatcher.test.ts

* test: add tests in DataAccessorBasedStore.test.ts

* test: fix template config for DynamicPods test

* test: add integration tests for metadata

* fix: change metaStrategy to metadataStrategy

* refactor: comments updated to new location CSS on github + some alphabetical edits

* refactor: remove getMetadata function in DABS as it is only used once

* refactor: add DataAccessorBasedStoreArgs to DataAccessorBasedStore.ts

* docs: modify documentation for writeMetadata function in DataAccessor.ts

* feat: ldp:contains is also part of the metadata resource of a container

* refactor: change function name and move check to DataAccessorBasedStore

* fix: fix tests for DABS and PutOperationHandler

* feat: avoid cloneRepresentation by introducing RdfPatcher, RdfStorePatcher and modifying ImmutableMetadataPatcher, N3Patcher, patching.json and SparqlUpdatePatcher

* test: fix patcher tests

* feat: create sparqlInsertMetadata in SparqlDataAccessor.ts

* fix: move check during put on container if it exists already back to PutOperationHandler.ts after discussion in PR

* test: update tests PutOperationHandler.ts and DataAccessorBasedStore.ts regarding previous commit

* test: add converter to DABS and replace rejection on data during container creation to warning

* test: implemented RdfPatcher test

* feat: remove ContainerPatcher

* fix: fix lint

* fix: fix integration tests

* refactor: fix minor issues mentioned in the PR

* WIP: problem with removeResponseMetadata

* refactor: remove responseMetadata in QuadToRdfConverter.ts

* feat: handle ResponeMetadata when writing to the store via a patch

* refactor: refactor based on comments in PR

* feat: make ImmutableMetadataPatcher.ts instantiation more clear

* test: achieve 100% coverage again

* fix: fix lint

* refactor: return to explicit arguments for the DABS

* fix: return to explicit arguments for the DABS (missed one)

* feat: optimise immutable checker

* fix: fix, enhance docs + optimise config files

* fix: DABS + QuadToRdfConverter feedback implemented

* fix: patching feedback implemented

* test: update operationhandler tests

* test: update integration tests after feedback

* test: update DABS tests after feedback

* test: update ImmutableMetadataPatcher.test.ts after feedback

* test: update patch tests after feedback

* docs: add documentation about editing metadata

* fix: config: intendation + name change + extra filters | filter pattern

* docs: tsdoc added to RdfStorePatcher.ts

* fix: DABS split implemented for getRepresentation + comment refactoring

* docs: further documentation on removing response data on serialization

* fix: DABS getRepresentation method

* docs: apply feedback from Joachim on the documentation of metadata-editing.md

* fix: indentation fix + fix metadata-editing.md documentation after feedback from Joachim

* docs: small fix in metadata-editing.md documentation after feedback from Joachim

* fix: fix metadata-editing.md documentation after feedback from Joachim

* fix: fix tests meta-editing after feedback Joachim

* feat: first attempt at RELEASE_NOTES.md

* docs: update release notes based on feedback

* docs: fix newline

* fix: patching config changes after feedback

* docs: metadata editing documentation changes after feedback

* docs: metadata editing documentation changes after feedback

* docs: metadata editing documentation changes after feedback

* feat: optimisation on ImmutableMetadataPatcher.ts algorithm

* feat: remove converter from DABS and add conversion for metadata resources in the RCS

* fix: Fix documentation RepresentationPatchHandler.ts + fix response graph not being stored due to convertingstore

* feat: make RepresentationPatcher generic

* test: generic RepresentationPatcher tests

* test: 100% coverage for patchers again

* feat: containers can be created with POST with no content-type

* feat: Immutable checks always with subject identifier

* feat: create AuxiliaryLinkMetadataWriter for adding description resources Link Header

* test: add tests for AuxiliaryLinkMetadataWriter and update them for ImmutableMetadataPatcher

* feat: remove metadataGenerator from acl.json and fix tests accordingly

* WIP: preserve metadata on PUT

* feat: preserve metadata on PUT

* fix: keep metadata on PATCHes

* test: add unit tests for preserving metadata on PUT

* fix: remove inConverter from sparql endpoint as that is already the default in the (converting.json)

* fix: add metadatastrategy to RepresentationConvertingStore in regex.json

* test: add integration tests for preserving metadata on PUT

* docs: update release notes and adding documentation about preserving metadata on PUT

* WIP: Template create setRepresentation

* fix: Move container exists and not allowed check to setRepresentation

* test: fix lint

* fix: update configs and documentation

* refactor: update and add documentation + small refactoring

* refactor: update and add documentation + small refactoring + fix tests

* fix: Dynamic pod config + tests

* fix: TemplatedResourcesGenerator does not create containers when they already exist

* fix: metadata preservation now deals with complex content types

* docs: explain the case when there is no content-type

* fix: minor comments
2022-08-04 15:24:36 +02:00
Joachim Van Herwegen
f609f1a9c5 feat: Accept both Settings and VariableBindings to create an App 2022-07-27 10:00:43 +02:00
Joachim Van Herwegen
7996fe5c3b feat: Split WebAclReader behaviour over multiple classes 2022-07-11 14:07:43 +02:00
Joachim Van Herwegen
0ff05fd420 feat: Update PermissionReaders to support new permission interface 2022-07-11 14:07:43 +02:00
Joachim Van Herwegen
18391ec414 feat: Introduce ModesExtractor for intermediate containers 2022-07-11 14:07:43 +02:00
Joachim Van Herwegen
7085252b3f feat: Update ModesExtractors to support new permission interface 2022-07-11 14:07:43 +02:00
Joachim Van Herwegen
23f0b37c28 feat: Change permission interface to store identifiers 2022-07-11 14:07:43 +02:00
Joachim Van Herwegen
b5d5071403 feat: Add a SetMultiMap interface and implementation 2022-07-11 14:07:43 +02:00
Joachim Van Herwegen
c35cd599a3 feat: Add a map that can check equality between object keys 2022-07-11 14:07:43 +02:00
Joachim Van Herwegen
45f8aa157d feat: Add utilities for Iterables 2022-07-11 14:07:43 +02:00
Thomas Dupont
311f8756ec fix: return contenttype header value string with parameters 2022-07-07 10:02:24 +02:00
Wannes Kerckhove
1c65b06392 fix: Introducing initializers for handling lock cleanup on start 2022-07-06 10:00:15 +02:00
Thomas Dupont
236bbc6e5d feat: initial proposal for multithreaded execution 2022-05-16 10:15:37 +02:00
Wannes Kerckhove
fbbccb0cf1 fix: Stop creating meta files for each new resource #1217 2022-05-06 08:20:55 +02:00
Thomas Dupont
fa78bc6856 feat: add a process-/thread-safe file-based ResourceLocker
test: unit test succeeds

fix: not quiting loop when releasing unexisting lock

refactor: pull wait() function into TimerUtils

feat: store all locks inside a single lock folder

feat: use md5 hashing for filepath hashes

test: coverage back to 100%

fix: store locks in proper .internal/locks folder
feat: reworked tryfn

test: coverage back to 100%

buidl: package json types next to lib

style: linting

dos: add more documentation to Locker classes

refactor: SingleThreadedResourceLocker -> MemoryResourceLocker

refactor: MultiThreadedResourceLocker -> FileSystemResourceLocker

feat: update all file-based backend configs to use the new FileSystemResourceLocker

feat: add warning on starting the MemoryResourceLocker in a worker process

test: coverage back to 100%

fix: finalizer of file.json was configured wrong

docs: updated release notes for 5.0.0

refactor: incorporated changes so far

refactor: retryFunctions are less complex now

test: jitter fix
2022-04-28 14:12:30 +02:00
Joachim Van Herwegen
d2bc995272 refactor: Replace RedirectAllHttpHandler usage with RedirectingHttpHandler 2022-04-25 09:09:39 +02:00
Thomas Dupont
e2e2d0864f feat: add a Redis based Read-Write Locker
refactor: more elegant way of providing default attemptSettings to constructor

style(jsdoc): rewording of jsdoc comment

fix: RegExp(/regex/) => /regex/

fix: Replace Error with InternalServerError

docs: jsdoc for RedisReadWriteLocker class

feat: make RedisReadWriteLocker a ResourceLocker too

test: coverage back to 100%

refactor: linting fix

style(jsdoc): Add explanation to tryRedisFn() method

refactor: remove RedisResourceLocker

fix: bug in lua script

chore(deps): update ioredis, remove redlock

refactor: removed RedisResourceLocker in favor of generic RedisLocker class

test: add redis lua scripts tests and integrate all 3 redis integration tests in 1

refactor: remove .vscode folder from index

refactor: Add some typing and  change redis references to Redis in comments

refactor: more changes after PR review

refactor: remove redis.json

refactor: rename redis-rw.json to redis.json

docs: added readme and release notes
2022-04-19 13:52:31 +02:00
Joachim Van Herwegen
2ec8fabd54 feat: Add support for client_credentials authentication 2022-04-15 11:20:26 +02:00
Wannes Kerckhove
283c301f08 feat: new helper functions to replace regexes #807
Implemented new StringUtil helper functions: splitCommaSeparated, sanitizeUrlPart, isValidFileName.
Added helper functions to HeaderUtil: matchesAuthorizationScheme, hasScheme.
Added unit tests for the new helper functions.
Refactored codebase to use helper functions instead of regexes if applicable.
2022-04-13 11:26:07 +02:00
Joachim Van Herwegen
1b7cc1ea3a Merge branch 'main' into versions/4.0.0
# Conflicts:
#	test/unit/util/errors/RedirectHttpError.test.ts
2022-04-11 10:21:49 +02:00
Wouter Termont
468e11d906
feat: Add RedirectingHttpHandler
* feat: redirect handler

Signed-off-by: Wouter Termont <woutermont@gmail.com>

* chore: rewrite reduction as loop

Signed-off-by: Wouter Termont <woutermont@gmail.com>

* chore: remove example import

Signed-off-by: Wouter Termont <woutermont@gmail.com>

* feat: add more redirect flavours

Signed-off-by: Wouter Termont <woutermont@gmail.com>

* chore: RedirectingHttpHandler with RedirectAllHttpHandler

Signed-off-by: Wouter Termont <woutermont@gmail.com>

* chore: replace RedirectAllHttpHandler with RedirectingHttpHandler

* chore: revert 5956385 (chore: replace RedirectAllHttpHandler with RedirectingHttpHandler)

This reverts commit 5956385c4180e8e8914b9199c4ed6ca8ae7d39fb.

* docs: complete constructor params

Signed-off-by: Wouter Termont <woutermont@gmail.com>
2022-04-11 09:49:36 +02:00
Joachim Van Herwegen
97e600bf4f fix: Undo authorization on OPTIONS requests 2022-04-04 13:35:15 +02:00
Thomas Dupont
48efc6fae1
feat: Create MetadataParser that detects JSON with Context link and throws an error
* feat: add PlainJsonLdFilter to reject JSON with context link

* refactor: abstract parseLinkHeader into HeaderUtils

* docs: typo in comment field

Co-authored-by: Ruben Verborgh <ruben@verborgh.org>

* refactor: Replace BadRequestHttpError with NotImplementedError

Co-authored-by: Ruben Verborgh <ruben@verborgh.org>

* refactor: incorporate requested changes

* refactor: requested changes incorporated

* refactor: remove obsolete code lines

Co-authored-by: Ruben Verborgh <ruben@verborgh.org>
2022-04-01 14:25:09 +02:00
Ruben Verborgh
238570b3d2 feat: Make LazyLoggerFactory buffer messages. 2022-04-01 14:18:58 +02:00
Joachim Van Herwegen
f08cdf75f7 fix: Prevent expired storage cleanup from crashing the server 2022-03-31 17:00:09 +02:00
Joachim Van Herwegen
6e98c6aae4 feat: Dynamically generate Allow and Accept-* headers 2022-03-29 14:52:36 +02:00
Joachim Van Herwegen
a1a6ce01fa feat: Create separate storage to generate keys 2022-03-28 11:43:14 +02:00
Joachim Van Herwegen
ad3edcf1a8 feat: Handle OPTIONS requests in OperationHandler 2022-03-18 14:15:12 +01:00
Joachim Van Herwegen
0e4d012086 feat: Add CachedResourceSet
Uses a WeakMap on the identifier to cache resource existence.
2022-03-18 14:15:12 +01:00
Joachim Van Herwegen
4404fa07d9 refactor: Rename resourceExists to hasResource
The function was also moved to the smaller interface ResourceSet.
2022-03-18 14:15:12 +01:00
Adler Faulkner
c8d4bfec39
feat: Support seeding pods and accounts
* feat(seeding): seed accounts and pods with seeded-pod-config.json

* feat(seeding): dry up SeededPodInitializer by using RegistrationManager directly and make compatible with version/3.0.0

* feat(seeding): update seed config files to version 3.0.0 context

* feat(seeding): simplify seeded-root config by importing pre-existing prefilled-root config

* feat(seeding): Add seeding as a default initializer, update seeded pod copy and guide, change seeded pod config to array

* feat(seeding): remove template info from seeded pod guide, use mockFs, code style nit, fix redlock test

* feat(seeding): remove old config file
2022-03-10 10:16:05 +01:00
Joachim Van Herwegen
2dc20fe3bc feat: Store the server version on start 2022-02-22 09:31:24 +01:00
Joachim Van Herwegen
62e2210023 feat: Warn users when they change the base URL 2022-02-22 09:31:24 +01:00
Joachim Van Herwegen
c9ed90aeeb fix: Update OIDC provider dependency to v7
The biggest resulting change is that the consent page always appears
after logging in.
Some minor fixes to be closer to the spec are included
together with some minor structural refactors.
2022-02-21 10:04:57 +01:00
Joachim Van Herwegen
1769b799df fix: Make IDP routes independent of handlers 2022-02-16 12:47:45 +01:00
Joachim Van Herwegen
9577791472 feat: Simplify setup to be more in line with IDP behaviour 2022-02-11 10:52:45 +01:00
Joachim Van Herwegen
a684b2ead7 feat: Update IDP templates to work with new API format 2022-02-11 10:52:45 +01:00
Joachim Van Herwegen
bc0eeb1012 feat: Split up IDP HTML, routing, and handler behaviour 2022-02-11 10:52:45 +01:00
Joachim Van Herwegen
c216efd62f
feat: Allow for custom CLI and variable options
* feat: (AppRunner) Mechanism to configure cli args and derive componentsjs vars from them implemented

* fix: (AppRunner) tidying

* fix: (AppRunner) tidying up

* fix: (AppRunner) runCli method made sync

* fix; (VarResolver) refactored to multiple files, and other stylistic fixes.

* chore: (AppRunner) Uses builder pattern for yargs base arguments setup to enable better typescript inference

* fix(AppRunner): refactoring AppRunner and VarResolver

* fix(AppRunner): refactoring AppRunner promise handling

* fix(AppRunner): verror dependency removal

* fix: Simplify CLI error handling

* feat: Use same config for both CLI and app instantiation

* fix: Update typings and imports

* feat: Split VariableResolver behaviour to 2 classes

* feat: Move default value behaviour from CLI to ValueComputers

* test: Add unit tests for new CLI classes

* feat: Integrate new CLI configuration with all default configurations

* feat: Add createApp function to AppRunner

* docs: Update comments in CLI-related classes

* fix: Various fixes and refactors

Co-authored-by: damooo <damodara@protonmail.com>
2022-02-11 10:00:12 +01:00
Joachim Van Herwegen
a9941ebe78 feat: Add support for N3 Patch 2022-01-25 11:30:42 +01:00
Joachim Van Herwegen
1afed65368 feat: Return correct status codes for invalid requests 2022-01-21 17:10:38 +01:00
Arthur Joppart
0cb4d7b161
feat: Add support for quota limits
* feat: implemented SizeReporter and FileSizeReporter

* test: FileSizeReporter tests

* feat: added QuotedDataAccessor

* test: added extra test to check recursiveness of filesizereporter

* feat: added QuotaStrategy interface

* feat: further progress in different files

* feat: wrote doc, tests and improved code

* feat: fixed bugs and code is now runnable and buildable

* feat: finished implementation

* fix: revert accidental chanegs

* fix: fileSizeReported did not count container size

* fix: bug calculating container sizes fixed

* test: FileSizeReporter tests

* test: QuotaDataValidator tests

* test: QuotaError tests

* fix: removed console.log

* doc: added doc to several files

* doc: changed doc for QuotaStrategy to new implementation

* fix: improved content length regex

* feat: improved GlobalQuotaStrategy code

* fix: made FileSizeReported readonly

* feat: added comments to quota-file.json

* fix: changed default tempFilePath variable

* test: included new tempFilePath variable in testing

* chore: created seperate command for start:file:quota to pass tests

* feat: removed all sync fs calls from FileSizeReporter

* feat: minor changes in multple files

* fix: changed function signatures to be in line with others

* feat: optimized quota data validation

* feat: improved FileSizeReporter code

* fix: corrected calculation of containersizes and fixed erroring edgecase

* feat: save content-length as number in metadata

* feat: added comments and changed GlobalQuotaStrategy constructor

* feat: changed file names and added small comment

* test: AtomicFileDataAccessor tests

* test: completed FileSizeReporter tests

* fix: content-length is now saved correctly in RepresentationMetadata

* feat: adapted content length metadata + tests

* fix: removed tempFilePath variable

* fix: reverted .gitignore

* fix: forgot to remove tempFilePath variable from componentsjs config

* test: GlobalQuotaStrategy tests

* feat: replaced DataValidator with Validator

* feat: reworked DataValidator

* feat: added calcultateChunkSize() to SizeReporter

* test: updated FileSizeReporter tests

* fix: tempFile location now relative to rootFilePath

* test: QuotaDataValidator tests

* fix: corrected FileSizeReporter tests

* fix: adapted FileSizeReporter tests

* fix: FileSizeReporter bug on Windows

* fix: regex linting error

* feat: changed Validator class

* feat: added PodQuotaStrategy to enable suota on a per pod basis

* chore: bump context versions

* fix: Capitalized comments in json file

* chore: renamed ValidatorArgs to ValidatorInput

* chore: order all exports

* fix: made TODO comment clearer

* chore: added seperated config files for global and pod based quota + fixed comments

* chore: made minor changes to comments

* feat: added PassthroughDataAccessor

* feat: added PasstroughtDataAccessor + tests

* fix: added invalid header check to ContentLengthParser

* chore: improved mocks

* chore: move quota limit higher up in config

* fix: atomicity issue in AtomicFileDataAccessor

* chore: moved .internal folder to config from FileSizeReporter

* fix: improved algorithm to ignore folders while calculating file size in FileSizeReporter

* fix: changes to support containers in the future

* fix: added error handling to prevent reading of unexistent files

* feat: added generic type to SizeReporter to calculate chunk sizes

* test: use mocked DataAccessor

* chore: added some comments to test and made minor improvement

* fix: fs mock rename

* chore: QuotaStrategy.estimateSize refactor

* chore: move trackAvailableSpace to abstract class QuotaStrategy

* fix: improved test case

* test: quota integration tests

* chore: edited some comments

* chore: change lstat to stat

* feat: moved estimateSize to SizeReporter to be consistent with calcultateChunkSize

* test: finish up tests to reach coverage

* fix: basic config

* fix: minor changes to test CI run

* fix: small fix for windows

* fix: improved writing to file

* chore: linting errors

* chore: rename trackAvailableSpace

* test: improved integration tests

* test: logging info for test debugging

* test: extra logging for debugging

* test: logging for debugging

* test: logging for debugging

* test: logging for debugging

* test: improved Quota integration test setup

* test: improve quota tests for CI run

* test: debugging Quota test

* test: uncommented global quota test

* test: changed global quota parameters

* test: logging for debugging

* test: logging cleanup

* chore: minor changes, mostly typo fixes

* chore: remove console.log

* fix: getting inconsistent results

* chore: try fix index.ts CI error

* chore: try fix CI error

* chore: try fix CI error

* chore: revert last commits

* chore: fix inconsistent files with origin

* test: minor test improvements

* chore: minor refactors and improvements

* fix: added extra try catch for breaking bug

* chore: improve config

* chore: minor code improvements

* test: use mockFs

* feat: add extra check in podQuotaStrategy

* chore: replace handle by handleSafe in ValidatingDataAccessor

* chore: typo

* test: improved Quota integration tests

* test: made comment in test more correct

* fix: rm -> rmdir for backwards compatibility

* fix: fsPromises issue

* chore: leave out irrelevant config

* chore: removed start script from package.json

* fix: Small fixes

Co-authored-by: Joachim Van Herwegen <joachimvh@gmail.com>
2022-01-21 10:49:05 +01:00
Lina
9a1f324685
feat: Create VoidLocker to disable locking resources
* add: Add VoidLocker and unittest

* Update src/util/locking/VoidLocker.ts

Co-authored-by: Ruben Verborgh <ruben@verborgh.org>

* Update src/util/locking/VoidLocker.ts

Co-authored-by: Joachim Van Herwegen <joachimvh@gmail.com>

* update: noop function and add debug void config

* add: debug-void in readme

* Update RELEASE_NOTES.md

Co-authored-by: Joachim Van Herwegen <joachimvh@gmail.com>

* Update config/util/README.md

Co-authored-by: Joachim Van Herwegen <joachimvh@gmail.com>

* add: missing line

Co-authored-by: lina <lina7906@gmail.com>
Co-authored-by: Ruben Verborgh <ruben@verborgh.org>
Co-authored-by: Joachim Van Herwegen <joachimvh@gmail.com>
2021-12-16 15:09:58 +01:00
Joachim Van Herwegen
4241c5348d feat: Move redirect support from IDP handler to specific handlers 2021-12-09 14:10:42 +01:00
Joachim Van Herwegen
7163a0317b feat: Support redirection through errors 2021-12-09 14:10:42 +01:00
Joachim Van Herwegen
520e4fe42f feat: Move OIDC library behaviour to separate path 2021-12-09 13:15:05 +01:00
Joachim Van Herwegen
11192ed4df Merge branch 'main' into versions/3.0.0 2021-11-22 09:20:41 +01:00
Ruben Verborgh
c13456c225 fix: Make UnionCredentialsExtractor tolerate failures.
Fixes https://github.com/solid/community-server/issues/1031
2021-11-11 12:48:39 +00:00
Joachim Van Herwegen
d52aa94e53 feat: Add "no conversion" as possible path in ChainedConverter 2021-11-08 10:16:28 +01:00
Joachim Van Herwegen
27306d6e3f refactor: Create BaseTypedRepresentationConverter 2021-11-08 10:16:28 +01:00