170 Commits

Author SHA1 Message Date
Arthur Joppart
4a37f8c879 chore: replace redis library with ioredis 2022-03-29 14:41:31 +02:00
Joachim Van Herwegen
027e3707fd feat: Parse content-type more strictly 2022-03-28 10:45:05 +02:00
Thomas Dupont
a8602055e6
feat: Store content type parameters
* feat: support storage and retrievel of content-type parameters

* test: extra unit tests for parseContentTypeWithParameters

* refactor: simplify set contentType()

Co-authored-by: Joachim Van Herwegen <joachimvh@gmail.com>

* refactor: simplify for loop because of unique blankNodes

Co-authored-by: Joachim Van Herwegen <joachimvh@gmail.com>

* refactor: ContentTypeParameter should be contentTypeParameter

Co-authored-by: Joachim Van Herwegen <joachimvh@gmail.com>

* refactor: remove undefined type in favor of var? syntax

Co-authored-by: Joachim Van Herwegen <joachimvh@gmail.com>

* refactor: use new parseContentType internally

* chore: remove commented code

* docs: code documentation line changed

Co-authored-by: Joachim Van Herwegen <joachimvh@gmail.com>

* refactor: Check for faulty metadata in contentType rdf structure

Co-authored-by: Joachim Van Herwegen <joachimvh@gmail.com>

* refactor: remove all instances of blanknodes

Co-authored-by: Joachim Van Herwegen <joachimvh@gmail.com>

* refactor: use full contentType when parsing header

Co-authored-by: Joachim Van Herwegen <joachimvh@gmail.com>

* refactor: use quads() method instead of store.getQuads()

* refactor: .value needed for type correctness

* feat: ReprMetadata constructor now supports full content-type string

Co-authored-by: Joachim Van Herwegen <joachimvh@gmail.com>
2022-03-14 10:27:34 +01:00
Ruben Verborgh
6306090114 refactor: Add resolveModulePath. 2022-02-22 09:31:24 +01:00
Thomas Dupont
ce754c119f
fix: Add content-negotiation when fetching dataset from url
* Solution works but tests don't

* refactor(FetchUtil): use arrayifyStream

* refactor(FetchUtil): split fetchDataset into 2 separate functions

* style(FetchUtil): onelining instead of declaring new local var

* test: trying to mock rdfDereferencer

* refactor: promise can't have async function as arg

* test(FetchUtil): pass Quad array to mockDereference instead

* test: all tests should pass now and coverage is back to 100%

* style: comment typo

* chore: make package.json and package-lock.json compatible with main

* chore: fix package.json double entries

* chore: updated package.json to be alfabetical again

* refactor(AgentGroupAccessChecker): Remove converter from contructor and config

* refactor(TokenOwnerShipValidator): Remove converter from constructor and config

* refactor(FetchUtil): Return BadRequestHttpError instead of generic Error

* test(FetchUtil): return Response object instead of mocking fetch

* style: typos and newlines
2022-02-15 13:44:03 +01:00
Joachim Van Herwegen
c5052625d1 Merge branch 'main' into versions/3.0.0
# Conflicts:
#	package-lock.json
#	package.json
#	src/storage/patch/SparqlUpdatePatcher.ts
#	test/unit/init/AppRunner.test.ts
#	test/unit/util/QuadUtil.test.ts
2022-02-11 15:30:45 +01:00
Joachim Van Herwegen
d2870e5c8b fix(deps): Update dependencies 2022-02-11 15:28:50 +01:00
Joachim Van Herwegen
9ceb62b8b6 refactor: Remove @rdfjs/data-model dependency
We already have N3.js for term generation.
No need to have two solutions there.
2022-02-11 15:28:50 +01:00
Joachim Van Herwegen
90a6460c8d Merge branch 'main' into versions/3.0.0
# Conflicts:
#	package-lock.json
#	test/integration/Identity.test.ts
#	test/integration/RepresentationConverter.test.ts
2022-01-25 11:44:24 +01:00
Joachim Van Herwegen
a9941ebe78 feat: Add support for N3 Patch 2022-01-25 11:30:42 +01:00
Joachim Van Herwegen
af049124bf chore: Update eslint dependencies 2022-01-24 09:37:50 +01:00
Joachim Van Herwegen
1afed65368 feat: Return correct status codes for invalid requests 2022-01-21 17:10:38 +01:00
Arthur Joppart
0cb4d7b161
feat: Add support for quota limits
* feat: implemented SizeReporter and FileSizeReporter

* test: FileSizeReporter tests

* feat: added QuotedDataAccessor

* test: added extra test to check recursiveness of filesizereporter

* feat: added QuotaStrategy interface

* feat: further progress in different files

* feat: wrote doc, tests and improved code

* feat: fixed bugs and code is now runnable and buildable

* feat: finished implementation

* fix: revert accidental chanegs

* fix: fileSizeReported did not count container size

* fix: bug calculating container sizes fixed

* test: FileSizeReporter tests

* test: QuotaDataValidator tests

* test: QuotaError tests

* fix: removed console.log

* doc: added doc to several files

* doc: changed doc for QuotaStrategy to new implementation

* fix: improved content length regex

* feat: improved GlobalQuotaStrategy code

* fix: made FileSizeReported readonly

* feat: added comments to quota-file.json

* fix: changed default tempFilePath variable

* test: included new tempFilePath variable in testing

* chore: created seperate command for start:file:quota to pass tests

* feat: removed all sync fs calls from FileSizeReporter

* feat: minor changes in multple files

* fix: changed function signatures to be in line with others

* feat: optimized quota data validation

* feat: improved FileSizeReporter code

* fix: corrected calculation of containersizes and fixed erroring edgecase

* feat: save content-length as number in metadata

* feat: added comments and changed GlobalQuotaStrategy constructor

* feat: changed file names and added small comment

* test: AtomicFileDataAccessor tests

* test: completed FileSizeReporter tests

* fix: content-length is now saved correctly in RepresentationMetadata

* feat: adapted content length metadata + tests

* fix: removed tempFilePath variable

* fix: reverted .gitignore

* fix: forgot to remove tempFilePath variable from componentsjs config

* test: GlobalQuotaStrategy tests

* feat: replaced DataValidator with Validator

* feat: reworked DataValidator

* feat: added calcultateChunkSize() to SizeReporter

* test: updated FileSizeReporter tests

* fix: tempFile location now relative to rootFilePath

* test: QuotaDataValidator tests

* fix: corrected FileSizeReporter tests

* fix: adapted FileSizeReporter tests

* fix: FileSizeReporter bug on Windows

* fix: regex linting error

* feat: changed Validator class

* feat: added PodQuotaStrategy to enable suota on a per pod basis

* chore: bump context versions

* fix: Capitalized comments in json file

* chore: renamed ValidatorArgs to ValidatorInput

* chore: order all exports

* fix: made TODO comment clearer

* chore: added seperated config files for global and pod based quota + fixed comments

* chore: made minor changes to comments

* feat: added PassthroughDataAccessor

* feat: added PasstroughtDataAccessor + tests

* fix: added invalid header check to ContentLengthParser

* chore: improved mocks

* chore: move quota limit higher up in config

* fix: atomicity issue in AtomicFileDataAccessor

* chore: moved .internal folder to config from FileSizeReporter

* fix: improved algorithm to ignore folders while calculating file size in FileSizeReporter

* fix: changes to support containers in the future

* fix: added error handling to prevent reading of unexistent files

* feat: added generic type to SizeReporter to calculate chunk sizes

* test: use mocked DataAccessor

* chore: added some comments to test and made minor improvement

* fix: fs mock rename

* chore: QuotaStrategy.estimateSize refactor

* chore: move trackAvailableSpace to abstract class QuotaStrategy

* fix: improved test case

* test: quota integration tests

* chore: edited some comments

* chore: change lstat to stat

* feat: moved estimateSize to SizeReporter to be consistent with calcultateChunkSize

* test: finish up tests to reach coverage

* fix: basic config

* fix: minor changes to test CI run

* fix: small fix for windows

* fix: improved writing to file

* chore: linting errors

* chore: rename trackAvailableSpace

* test: improved integration tests

* test: logging info for test debugging

* test: extra logging for debugging

* test: logging for debugging

* test: logging for debugging

* test: logging for debugging

* test: improved Quota integration test setup

* test: improve quota tests for CI run

* test: debugging Quota test

* test: uncommented global quota test

* test: changed global quota parameters

* test: logging for debugging

* test: logging cleanup

* chore: minor changes, mostly typo fixes

* chore: remove console.log

* fix: getting inconsistent results

* chore: try fix index.ts CI error

* chore: try fix CI error

* chore: try fix CI error

* chore: revert last commits

* chore: fix inconsistent files with origin

* test: minor test improvements

* chore: minor refactors and improvements

* fix: added extra try catch for breaking bug

* chore: improve config

* chore: minor code improvements

* test: use mockFs

* feat: add extra check in podQuotaStrategy

* chore: replace handle by handleSafe in ValidatingDataAccessor

* chore: typo

* test: improved Quota integration tests

* test: made comment in test more correct

* fix: rm -> rmdir for backwards compatibility

* fix: fsPromises issue

* chore: leave out irrelevant config

* chore: removed start script from package.json

* fix: Small fixes

Co-authored-by: Joachim Van Herwegen <joachimvh@gmail.com>
2022-01-21 10:49:05 +01:00
Lina
9a1f324685
feat: Create VoidLocker to disable locking resources
* add: Add VoidLocker and unittest

* Update src/util/locking/VoidLocker.ts

Co-authored-by: Ruben Verborgh <ruben@verborgh.org>

* Update src/util/locking/VoidLocker.ts

Co-authored-by: Joachim Van Herwegen <joachimvh@gmail.com>

* update: noop function and add debug void config

* add: debug-void in readme

* Update RELEASE_NOTES.md

Co-authored-by: Joachim Van Herwegen <joachimvh@gmail.com>

* Update config/util/README.md

Co-authored-by: Joachim Van Herwegen <joachimvh@gmail.com>

* add: missing line

Co-authored-by: lina <lina7906@gmail.com>
Co-authored-by: Ruben Verborgh <ruben@verborgh.org>
Co-authored-by: Joachim Van Herwegen <joachimvh@gmail.com>
2021-12-16 15:09:58 +01:00
Joachim Van Herwegen
7163a0317b feat: Support redirection through errors 2021-12-09 14:10:42 +01:00
Ruben Verborgh
c13456c225 fix: Make UnionCredentialsExtractor tolerate failures.
Fixes https://github.com/solid/community-server/issues/1031
2021-11-11 12:48:39 +00:00
Wout Slabbinck
54ccbed48d fix: Added a content-type parser to HeaderUtil
* fixes #995

* fixed the remarks and added a test for parseContentType

* refactor: remove unnecessary comment

Co-authored-by: woutslabbinck <wout.slabbinck@ugent.com>
2021-10-15 13:13:37 +02:00
Joachim Van Herwegen
b3da9c9fcf refactor: Restructure source code folder
This way the location of certain classes should make more sense
2021-10-12 12:51:02 +02:00
Joachim Van Herwegen
62f026f2bc feat: Create UnionHandler to combine AsyncHandler results 2021-09-28 13:06:38 +02:00
Joachim Van Herwegen
b592d449eb feat: Integrate setup behaviour
This adds options for enabling setup to the config folder.
All default configs with permanent storage (file/sparql)
are configured to require setup at server start.
Memory-based configs merely have it as an option.
2021-09-27 10:32:34 +02:00
Joachim Van Herwegen
facf691e86 feat: Create ConditionalHandler
This handler rejects all inputs once a certain condition is met
2021-09-27 10:32:34 +02:00
Joachim Van Herwegen
fb0b50c997 fix: Make the RouterHandler more robust
It now extracts paths based on the base URL
and allows catching all methods.
2021-09-27 10:32:34 +02:00
Joachim Van Herwegen
42d3ab0a4c fix: Add filename parameter for EJS templates
This is required if we want to include partial templates
2021-09-27 10:32:34 +02:00
Joachim Van Herwegen
cc1c3d9223 feat: Support JSON errors
The IDP behaviour has been changed to move all error related knowledge
to the IdentityProviderHttpHandler instead of managing it
in the Interactionhandlers.
2021-09-08 13:55:52 +02:00
Joachim Van Herwegen
4f1a86dfa0 feat: Convert IDP input data to JSON 2021-09-08 13:55:52 +02:00
Simone Persiani
73867f0827 feat: Add BooleanHandler
Co-Authored-By: Ludovico Granata <Ludogranata@gmail.com>
2021-08-23 16:39:25 +02:00
Simone Persiani
0355673a0f feat: Add function promiseSome
Co-Authored-By: Ludovico Granata <Ludogranata@gmail.com>
2021-08-23 16:39:25 +02:00
Simone Persiani
c13c03ef54 feat: Add function readableToQuads
Co-Authored-By: Ludovico Granata <Ludogranata@gmail.com>
2021-08-23 16:39:25 +02:00
Joachim Van Herwegen
80ebd02cc4 feat: Support content negotiation for IDP requests 2021-08-18 15:39:02 +02:00
Joachim Van Herwegen
0d42987bbd feat: Verify conditions in DataAccessorBasedStore 2021-08-18 13:16:08 +02:00
Joachim Van Herwegen
97c534b2bf feat: Keep track of last modified date of resources 2021-08-18 13:16:08 +02:00
Ruben Verborgh
1719857e4b change: Use @css: instead of $PACKAGE_ROOT/
This makes usage from the command line easier,
since $ is typically used to mark variables.
2021-08-03 13:19:38 +01:00
Joachim Van Herwegen
18a71032c0 feat: Create ChainedTemplateEngine for combining engines 2021-08-03 12:52:30 +02:00
Joachim Van Herwegen
2ae95bd167 refactor: Remove streamify array dependency 2021-08-02 14:28:43 +02:00
Joachim Van Herwegen
63e88578c3 feat: Rewrite TokenOwnershipValidator behaviour to remove @rdfjs/fetch dependency 2021-08-02 12:02:10 +02:00
Joachim Van Herwegen
5f8ec5dd15 fix: Prevent warnings for expected errors 2021-07-29 16:57:50 +02:00
Ruben Verborgh
9628fe98b8 refactor: Align EJS engine with Handlebars. 2021-07-22 11:12:21 +02:00
Ruben Verborgh
19624dc729 refactor: Allow default template in template engines. 2021-07-22 11:12:21 +02:00
Ruben Verborgh
31dcfbf51e chore: Move FetchUtil to util folder. 2021-07-20 15:33:28 +02:00
Ruben Verborgh
b9295f00c2 feat: Default error code to HTTP status code. 2021-07-15 10:22:28 +01:00
Ruben Verborgh
649f7a9a78 refactor: Assign properties directly to error. 2021-07-13 13:25:30 +01:00
Ruben Verborgh
8a62938c18 test: Use individual error assertions. 2021-07-13 13:25:30 +01:00
Joachim Van Herwegen
f2f967ff8a feat: Convert errorCodes using markdown 2021-07-06 16:42:06 +02:00
Joachim Van Herwegen
35a7cf988c feat: Support metadata in multiple graphs 2021-07-05 14:33:20 +02:00
Joachim Van Herwegen
7cfb87e516 feat: Add better support for non-native errors 2021-06-14 15:29:47 +02:00
Joachim Van Herwegen
e1f95877da feat: Create ErrorHandler to convert errors to Representations 2021-06-07 13:54:38 +02:00
renovate[bot]
afc662ca9a
chore: Update (ts-)jest and use modern fake timers where possible
* chore(deps): update dependency jest to v27

* chore: Update (ts-)jest and use modern fake timers where possible

Co-authored-by: Renovate Bot <bot@renovateapp.com>
Co-authored-by: Joachim Van Herwegen <joachimvh@gmail.com>
2021-05-31 15:29:14 +02:00
Joachim Van Herwegen
96a07e4853 refactor: Simplify resource metadata generation 2021-05-12 15:27:51 +02:00
jaxoncreed
1d65143e89
feat: Add identity provider (#455)
* Add identity provider handler as a dependency

* Temp Identity

* Figured out how to get koa to work

* Hooked up idp to networking

* Feat/idp architecture refactor (#430)

* Logs in with solid oidc

* Refactored Provider

* Attempt to hook up dependencies

* Partial wiring of oidc provider components

* IdP networking now works with architecture

* Interaction Handlers Set Up

* fix: Rename & adapt to CSS

* Included Login Interaction

* Refactored architecture to bind Interaction Policy to HttpHandlers

Co-authored-by: Matthieu Bosquet <matthieubosquet@gmail.com>

* fix: Rebase on master

* fix: DI after rebase

* Reimplemented Routing

* Renamed modules and removed ProviderFactory (#450)

* refactor: Solid IdP DI

* refactor: IdP interaction handler DI

* refactor: IdP interaction waterfall

* refactor: Remove unnecessary legacy URL parse

* fix: Add legacy parse back in

* feat: adapter & fix: handlers

* Removed adapter factory

* fix: refactor IdP

* fix: refactor IdP

* fix: refactor IdP

* feat: Add IdP to file storage config

* fix: Unintended commit

* fix: Components ignore

* feat: Basic resource store adapter

* Partially complete idp routing

* Set up initial routing injection graph

* Clean up ResourceStorageAdapter

* Refactored configuration architecture

* Hooked up Login UIs (#518)

* feat: Use template path & run fileserver

* feat: Use util function to read resource

* Fixed DI JSON-LD context

* fixed rendering

* WebId validator

* Set up persistent storage for loing and register

* Fixed ejs template routing

* Refactored StorageAdapters

* NSS login successful

* Forgot password infrastructure

* Can send email (#557)

* Can send email

* fix: IdP crashes if interaction ID doesn't exist (#587)

* feat: Require an issuer registration token

* fix: Issuer registration token typo in error

* fix: Remove dummy IdP storage adapter

* fix: Remove unused library lodash

* fix: Remove unused library lru-cache

* Production ready keystore

* Ruben comments before clownface removal

* Removed clownface

* Change key value store

* Completed Ruben's comments

* Added comments to each class

* Fixed errors on login

* Ruben feedback

* Refactored out getPostRenderHandler

* Identity provider tests (#622)

* corrected tests lacking <void> on promises

* Added files for all idp tests

* Added unfinished tests for all added files

* ErrorHandlingWaterfallHandler

* RenderEjsHandler and RouterHandler tests

* GetPostRouterHandler and BasicOnErrorHandler tests

* Corrected tests for updates to Idp

* fix: missing export

* fix: audience claim

* Client Id Support (#630)

* Added client_id for the auth challenge

* Update src/identity/storage/ClientWebIdFetchingStorageAdapterFactory.ts

Co-authored-by: Matthieu Bosquet <matthieubosquet@gmail.com>

Co-authored-by: Matthieu Bosquet <matthieubosquet@gmail.com>

* fix: Rebase fixes

* Several minor Idp changes/refactors (#656)

* fix: Minor changes

* refactor: Split EmailPasswordInteractionPolicy

* refactor: Remove ErrorHandlingWaterfallHandler

* refactor: Clean up dependencies

* fix: Add dummy IdentityProviderHandler to fix integration tests

* Replace KeyValueStore with KeyValueStorage (#663)

* feat: Create WrappedExpiringStorage

* refactor: Update ResourceStoreEmailPasswordStore to use KeyValueStorage

* refactor: Update KeyGeneratingIdpConfigurationGenerator to use KeyValueStorage

* refactor: Update ResourceStoreStorageAdapterFactory to use ExpiringStorage

* refactor: Removed KeyValueStore

* refactor: Simplify EmailPassword handlers (#664)

* refactor: Order index.ts

* test: Add EmailPasswordForgotPasswordHandler unit tests

* test: Add EmailPasswordGetResetPasswordHandler unit tests

* test: Add EmailPasswordLoginHandler unit tests

* test: Add EmailPasswordRegistrationHandler unit tests

* test: Add EmailPasswordResetPasswordHandler unit tests

* test: Remove unnecessary test file

* feat: Basic instructions for using the IdP

* fix: IdP instructions and add example WebID

* fix: IdP registration copy

* fix: IdP instruction editorial

* Update README.md

Co-authored-by: Joachim Van Herwegen <joachimvh@gmail.com>

* Update README.md

Co-authored-by: Joachim Van Herwegen <joachimvh@gmail.com>

* test: Add KeyGeneratingIdpConfigurationGenerator unit tests

* test: Add KeyValueEmailPasswordStore unit tests

* test: Create IdP integration test

* test: Add EmailPasswordInteractionPolicy unit tests

* test: Add BasicIssuerReferenceWebIdOwnershipValidator unit tests

* test: Add ChooseInitialInteractionHandler unit tests

Also fixes the config warning.

* test: Add EjsTemplateRenderer unit tests

* test: Add EmailSender unit tests

* test: Add FormDataUtil unit tests

* test: Add IdpRouteController unit tests

* test: Add OidcInteractionCompleter unit tests

* refactor: Simplify ClientWebIdFetchingStorageAdapterFactory

* test: Add ClientWebIdFetchingStorageAdapterFactory unit tests

* refactor: Fix ejs html warnings

* test: Add step to test logging in again

Included are updates to handle cookies more correctly.

* feat: Add IdpConfirmHttpHandler

This way there's a handler for the confirm page.

* test: Add ExpiringStorageAdapterFactory unit tests

* test: Add IdentityProviderFactory unit tests

* test: Add IdentityProviderHttpHandler unit tests

* refactor: Minor refactors

* refactor: Use jose instead of node-jose

* refactor: Use jose instead of node-jose

Reduces the number of dependencies since other libraries
also depend on jose.

* Update src/identity/configuration/KeyGeneratingIdpConfigurationGenerator.ts

Co-authored-by: Matthieu Bosquet <matthieubosquet@gmail.com>

* refactor: Use interfaces instead of abstract classes

* refactor: Make WebIdOwnershipValidator an AsyncHandler

* refactor: Make TemplateRenderer an AsyncHandler

* fix: Fix typing issue

* fix: Convert JWK to plain object for node 15

* feat: Update CI configuration

--ignore-scripts was removed because it also stopped dependency scripts,
which was a requirement for bcrypt.
15.0 was removed since that version doesn't run the required scripts after install.
14.0 was removed since the somehow it caused the solid-authn client to do the wrong calls.

* test: Run integration tests on Node 14.2

This is the lowest 14.x version where the IdP integration tests succeed.

* feat: Use ErrorResponseWriter for handling oidc errors

* test: Mock Date in OidcInteractionCompleter tests

* fix: Correctly generate new identifiers

Previously there could be double slashes if the base URL ended in slash.

* fix: Correctly handle storagePathName in ExpiringStorageAdapterFactory

* fix: Fix issue with new CliRunner test in rebase

* fix: Handle unknown errors more consistently

* feat: Make idp parameter dynamic

* feat: Add more logging

* refactor: Link css instead of injecting

* fix: Fix redis integration tests with idp

* refactor: Shorten idp class names

* refactor: Remove e-mail configuration from default config

* feat: Store JsonResourceStorage data in a single container

* feat: Make sure expired data gets removed at some point

* feat: Only accept strings as keys in KeyValueStorage

* fix: Various minor fixes based on review

Co-authored-by: Matthieu Bosquet <matthieubosquet@gmail.com>
Co-authored-by: Joachim Van Herwegen <joachimvh@gmail.com>
2021-05-04 13:17:43 +02:00
Ruben Verborgh
2d89350ed5 fix: Improve identifier error messages. 2021-04-27 09:02:11 +02:00