feat: Add support for quota limits

* feat: implemented SizeReporter and FileSizeReporter

* test: FileSizeReporter tests

* feat: added QuotedDataAccessor

* test: added extra test to check recursiveness of filesizereporter

* feat: added QuotaStrategy interface

* feat: further progress in different files

* feat: wrote doc, tests and improved code

* feat: fixed bugs and code is now runnable and buildable

* feat: finished implementation

* fix: revert accidental chanegs

* fix: fileSizeReported did not count container size

* fix: bug calculating container sizes fixed

* test: FileSizeReporter tests

* test: QuotaDataValidator tests

* test: QuotaError tests

* fix: removed console.log

* doc: added doc to several files

* doc: changed doc for QuotaStrategy to new implementation

* fix: improved content length regex

* feat: improved GlobalQuotaStrategy code

* fix: made FileSizeReported readonly

* feat: added comments to quota-file.json

* fix: changed default tempFilePath variable

* test: included new tempFilePath variable in testing

* chore: created seperate command for start:file:quota to pass tests

* feat: removed all sync fs calls from FileSizeReporter

* feat: minor changes in multple files

* fix: changed function signatures to be in line with others

* feat: optimized quota data validation

* feat: improved FileSizeReporter code

* fix: corrected calculation of containersizes and fixed erroring edgecase

* feat: save content-length as number in metadata

* feat: added comments and changed GlobalQuotaStrategy constructor

* feat: changed file names and added small comment

* test: AtomicFileDataAccessor tests

* test: completed FileSizeReporter tests

* fix: content-length is now saved correctly in RepresentationMetadata

* feat: adapted content length metadata + tests

* fix: removed tempFilePath variable

* fix: reverted .gitignore

* fix: forgot to remove tempFilePath variable from componentsjs config

* test: GlobalQuotaStrategy tests

* feat: replaced DataValidator with Validator

* feat: reworked DataValidator

* feat: added calcultateChunkSize() to SizeReporter

* test: updated FileSizeReporter tests

* fix: tempFile location now relative to rootFilePath

* test: QuotaDataValidator tests

* fix: corrected FileSizeReporter tests

* fix: adapted FileSizeReporter tests

* fix: FileSizeReporter bug on Windows

* fix: regex linting error

* feat: changed Validator class

* feat: added PodQuotaStrategy to enable suota on a per pod basis

* chore: bump context versions

* fix: Capitalized comments in json file

* chore: renamed ValidatorArgs to ValidatorInput

* chore: order all exports

* fix: made TODO comment clearer

* chore: added seperated config files for global and pod based quota + fixed comments

* chore: made minor changes to comments

* feat: added PassthroughDataAccessor

* feat: added PasstroughtDataAccessor + tests

* fix: added invalid header check to ContentLengthParser

* chore: improved mocks

* chore: move quota limit higher up in config

* fix: atomicity issue in AtomicFileDataAccessor

* chore: moved .internal folder to config from FileSizeReporter

* fix: improved algorithm to ignore folders while calculating file size in FileSizeReporter

* fix: changes to support containers in the future

* fix: added error handling to prevent reading of unexistent files

* feat: added generic type to SizeReporter to calculate chunk sizes

* test: use mocked DataAccessor

* chore: added some comments to test and made minor improvement

* fix: fs mock rename

* chore: QuotaStrategy.estimateSize refactor

* chore: move trackAvailableSpace to abstract class QuotaStrategy

* fix: improved test case

* test: quota integration tests

* chore: edited some comments

* chore: change lstat to stat

* feat: moved estimateSize to SizeReporter to be consistent with calcultateChunkSize

* test: finish up tests to reach coverage

* fix: basic config

* fix: minor changes to test CI run

* fix: small fix for windows

* fix: improved writing to file

* chore: linting errors

* chore: rename trackAvailableSpace

* test: improved integration tests

* test: logging info for test debugging

* test: extra logging for debugging

* test: logging for debugging

* test: logging for debugging

* test: logging for debugging

* test: improved Quota integration test setup

* test: improve quota tests for CI run

* test: debugging Quota test

* test: uncommented global quota test

* test: changed global quota parameters

* test: logging for debugging

* test: logging cleanup

* chore: minor changes, mostly typo fixes

* chore: remove console.log

* fix: getting inconsistent results

* chore: try fix index.ts CI error

* chore: try fix CI error

* chore: try fix CI error

* chore: revert last commits

* chore: fix inconsistent files with origin

* test: minor test improvements

* chore: minor refactors and improvements

* fix: added extra try catch for breaking bug

* chore: improve config

* chore: minor code improvements

* test: use mockFs

* feat: add extra check in podQuotaStrategy

* chore: replace handle by handleSafe in ValidatingDataAccessor

* chore: typo

* test: improved Quota integration tests

* test: made comment in test more correct

* fix: rm -> rmdir for backwards compatibility

* fix: fsPromises issue

* chore: leave out irrelevant config

* chore: removed start script from package.json

* fix: Small fixes

Co-authored-by: Joachim Van Herwegen <joachimvh@gmail.com>

config/ldp/metadata-parser/default.json

@@ -2,6 +2,7 @@
   "@context": "https://linkedsoftwaredependencies.org/bundles/npm/@solid/community-server/^2.0.0/components/context.jsonld",
   "import": [
     "files-scs:config/ldp/metadata-parser/parsers/content-type.json",
+    "files-scs:config/ldp/metadata-parser/parsers/content-length.json",
     "files-scs:config/ldp/metadata-parser/parsers/slug.json",
     "files-scs:config/ldp/metadata-parser/parsers/link.json"
   ],
@@ -12,6 +13,7 @@
       "@type": "ParallelHandler",
       "handlers": [
         { "@id": "urn:solid-server:default:ContentTypeParser" },
+        { "@id": "urn:solid-server:default:ContentLengthParser" },
         { "@id": "urn:solid-server:default:SlugParser" },
         { "@id": "urn:solid-server:default:LinkRelParser" }
       ]

config/ldp/metadata-parser/parsers/content-length.json

@@ -0,0 +1,10 @@
+{
+  "@context": "https://linkedsoftwaredependencies.org/bundles/npm/@solid/community-server/^2.0.0/components/context.jsonld",
+  "@graph": [
+    {
+      "comment": "Converts content-length headers into RDF metadata.",
+      "@id": "urn:solid-server:default:ContentLengthParser",
+      "@type": "ContentLengthParser"
+    }
+  ]
+}

config/quota-file.json

@@ -0,0 +1,48 @@
+{
+  "@context": "https://linkedsoftwaredependencies.org/bundles/npm/@solid/community-server/^2.0.0/components/context.jsonld",
+  "import": [
+    "files-scs:config/app/main/default.json",
+    "files-scs:config/app/init/default.json",
+    "files-scs:config/app/setup/required.json",
+    "files-scs:config/http/handler/default.json",
+    "files-scs:config/http/middleware/websockets.json",
+    "files-scs:config/http/server-factory/websockets.json",
+    "files-scs:config/http/static/default.json",
+    "files-scs:config/identity/access/public.json",
+    "files-scs:config/identity/email/default.json",
+    "files-scs:config/identity/handler/default.json",
+    "files-scs:config/identity/ownership/token.json",
+    "files-scs:config/identity/pod/static.json",
+    "files-scs:config/identity/registration/enabled.json",
+    "files-scs:config/ldp/authentication/dpop-bearer.json",
+    "files-scs:config/ldp/authorization/allow-all.json",
+    "files-scs:config/ldp/handler/default.json",
+    "files-scs:config/ldp/metadata-parser/default.json",
+    "files-scs:config/ldp/metadata-writer/default.json",
+    "files-scs:config/ldp/modes/default.json",
+    "files-scs:config/storage/backend/pod-quota-file.json",
+    "files-scs:config/storage/key-value/resource-store.json",
+    "files-scs:config/storage/middleware/default.json",
+    "files-scs:config/util/auxiliary/acl.json",
+    "files-scs:config/util/identifiers/suffix.json",
+    "files-scs:config/util/index/default.json",
+    "files-scs:config/util/logging/winston.json",
+    "files-scs:config/util/representation-conversion/default.json",
+    "files-scs:config/util/resource-locker/memory.json",
+    "files-scs:config/util/variables/default.json"
+  ],
+  "@graph": [
+    {
+      "comment": "A server that stores its resources on disk while enforcing quota."
+    },
+    {
+      "@id": "urn:solid-server:default:QuotaStrategy",
+      "PodQuotaStrategy:_limit_amount": 7000,
+      "PodQuotaStrategy:_limit_unit": "bytes"
+    },
+    {
+      "@id": "urn:solid-server:default:SizeReporter",
+      "FileSizeReporter:_ignoreFolders": [ "^/\\.internal$" ]
+    }
+  ]
+}

config/storage/README.md

@@ -5,7 +5,9 @@ Options related to how data and resources are stored.
 The final part of the ResourceStore chain that handles data access.
 * *dynamic*: The routing store used here is needed when using dynamic pod creation.
 * *file*: Default setup with a file backend.
+* *global-quota-file*: File backend with a global quota over the entire server.
 * *memory*: Default setup with a memory backend.
+* *pod-quota-file*: File backend with a max quota per pod.
 * *regex*: Uses a different backend based on the container that is being used.
 * *sparql*: Default setup with a SPARQL endpoint backend.
   Also updates the converting store so all incoming data is transformed into quads.

config/storage/backend/global-quota-file.json

@@ -0,0 +1,17 @@
+{
+  "@context": "https://linkedsoftwaredependencies.org/bundles/npm/@solid/community-server/^2.0.0/components/context.jsonld",
+  "import": [
+    "files-scs:config/storage/backend/quota/global-quota-file.json",
+    "files-scs:config/storage/backend/quota/quota-file.json"
+  ],
+  "@graph": [
+    {
+      "comment": "A global quota store setup with a file system backend.",
+      "@id": "urn:solid-server:default:ResourceStore_Backend",
+      "@type": "DataAccessorBasedStore",
+      "identifierStrategy": { "@id": "urn:solid-server:default:IdentifierStrategy" },
+      "auxiliaryStrategy": { "@id": "urn:solid-server:default:AuxiliaryStrategy" },
+      "accessor": { "@id": "urn:solid-server:default:FileDataAccessor" }
+    }
+  ]
+}

config/storage/backend/pod-quota-file.json

@@ -0,0 +1,17 @@
+{
+  "@context": "https://linkedsoftwaredependencies.org/bundles/npm/@solid/community-server/^2.0.0/components/context.jsonld",
+  "import": [
+    "files-scs:config/storage/backend/quota/pod-quota-file.json",
+    "files-scs:config/storage/backend/quota/quota-file.json"
+  ],
+  "@graph": [
+    {
+      "comment": "A pod quota store setup with a file system backend.",
+      "@id": "urn:solid-server:default:ResourceStore_Backend",
+      "@type": "DataAccessorBasedStore",
+      "identifierStrategy": { "@id": "urn:solid-server:default:IdentifierStrategy" },
+      "auxiliaryStrategy": { "@id": "urn:solid-server:default:AuxiliaryStrategy" },
+      "accessor": { "@id": "urn:solid-server:default:FileDataAccessor" }
+    }
+  ]
+}

config/storage/backend/quota/global-quota-file.json

@@ -0,0 +1,13 @@
+{
+  "@context": "https://linkedsoftwaredependencies.org/bundles/npm/@solid/community-server/^2.0.0/components/context.jsonld",
+  "comment": "Configuration of a GlobalQuotaStrategy to enforce quota globally on the server.",
+  "@graph": [
+    {
+      "comment": "Enforces quota globally for all data on the server",
+      "@id": "urn:solid-server:default:QuotaStrategy",
+      "@type": "GlobalQuotaStrategy",
+      "reporter": { "@id": "urn:solid-server:default:SizeReporter" },
+      "base": { "@id": "urn:solid-server:default:variable:baseUrl" }
+    }
+  ]
+}

config/storage/backend/quota/pod-quota-file.json

@@ -0,0 +1,14 @@
+{
+  "@context": "https://linkedsoftwaredependencies.org/bundles/npm/@solid/community-server/^2.0.0/components/context.jsonld",
+  "comment": "Configuration of a PodQuotaStrategy to enforce pod quotas on the server.",
+  "@graph": [
+    {
+      "comment": "Enforces quota for all data per pod on the server",
+      "@id": "urn:solid-server:default:QuotaStrategy",
+      "@type": "PodQuotaStrategy",
+      "reporter": { "@id": "urn:solid-server:default:SizeReporter" },
+      "accessor": { "@id": "urn:solid-server:default:AtomicFileDataAccessor" },
+      "identifierStrategy": { "@id": "urn:solid-server:default:IdentifierStrategy" }
+    }
+  ]
+}

config/storage/backend/quota/quota-file.json

@@ -0,0 +1,37 @@
+{
+  "@context": "https://linkedsoftwaredependencies.org/bundles/npm/@solid/community-server/^2.0.0/components/context.jsonld",
+  "comment": "DataAccessor configuration using a QuotaStrategy to enforce quota on the server.",
+  "@graph": [
+    {
+      "comment": "DataAccessor that writes data to the disk with atomicity in mind",
+      "@id": "urn:solid-server:default:AtomicFileDataAccessor",
+      "@type": "AtomicFileDataAccessor",
+      "resourceMapper": { "@id": "urn:solid-server:default:FileIdentifierMapper" },
+      "rootFilePath": { "@id": "urn:solid-server:default:variable:rootFilePath" },
+      "tempFilePath": "/.internal/tempFiles/"
+    },
+
+    {
+      "comment": "Calculates the space already taken up by a resource",
+      "@id": "urn:solid-server:default:SizeReporter",
+      "@type": "FileSizeReporter",
+      "fileIdentifierMapper": { "@id": "urn:solid-server:default:FileIdentifierMapper" },
+      "rootFilePath": { "@id": "urn:solid-server:default:variable:rootFilePath" }
+    },
+
+    {
+      "comment": "Validates the data being written to the server",
+      "@id": "urn:solid-server:default:QuotaValidator",
+      "@type": "QuotaValidator",
+      "strategy": { "@id": "urn:solid-server:default:QuotaStrategy" }
+    },
+
+    {
+      "comment": "Simple wrapper for another DataAccessor but adds validation",
+      "@id": "urn:solid-server:default:FileDataAccessor",
+      "@type": "ValidatingDataAccessor",
+      "accessor": { "@id": "urn:solid-server:default:AtomicFileDataAccessor" },
+      "validator": { "@id": "urn:solid-server:default:QuotaValidator" }
+    }
+  ]
+}