What would you like to be added?
Add new compactor based revision count, instead of fixed interval time.
In order to make it happen, the mvcc store needs to export
`CompactNotify` function to notify the compactor that configured number of
write transactions have occured since previsious compaction. The
new compactor can get the revision change and delete out-of-date data in time,
instead of waiting with fixed interval time. The underly bbolt db can
reuse the free pages as soon as possible.
Why is this needed?
In the kubernetes cluster, for instance, argo workflow, there will be batch
requests to create pods , and then there are also a lot of pod status's PATCH
requests, especially when the pod has more than 3 containers. If the burst
requests increase the db size in short time, it will be easy to exceed the max
quota size. And then the cluster admin get involved to defrag, which may casue
long downtime. So, we hope the ETCD can delete the out-of-date data as
soon as possible and slow down the grow of total db size.
Currently, both revision and periodic are based on time. It's not easy
to use fixed interval time to face the unexpected burst update requests.
The new compactor based on revision count can make the admin life easier.
For instance, let's say that average of object size is 50 KiB. The new
compactor will compact based on 10,000 revisions. It's like that ETCD can
compact after new 500 MiB data in, no matter how long ETCD takes to get
new 10,000 revisions. It can handle the burst update requests well.
There are some test results:
* Fixed value size: 10 KiB, Update Rate: 100/s, Total key space: 3,000
```
enchmark put --rate=100 --total=300000 --compact-interval=0 \
--key-space-size=3000 --key-size=256 --val-size=10240
```
| Compactor | DB Total Size | DB InUse Size |
| -- | -- | -- |
| Revision(5min,retension:10000) | 570 MiB | 208 MiB |
| Periodic(1m) | 232 MiB | 165 MiB |
| Periodic(30s) | 151 MiB | 127 MiB |
| NewRevision(retension:10000) | 195 MiB | 187 MiB |
* Random value size: [9 KiB, 11 KiB], Update Rate: 150/s, Total key space: 3,000
```
bnchmark put --rate=150 --total=300000 --compact-interval=0 \
--key-space-size=3000 --key-size=256 --val-size=10240 \
--delta-val-size=1024
```
| Compactor | DB Total Size | DB InUse Size |
| -- | -- | -- |
| Revision(5min,retension:10000) | 718 MiB | 554 MiB |
| Periodic(1m) | 297 MiB | 246 MiB |
| Periodic(30s) | 185 MiB | 146 MiB |
| NewRevision(retension:10000) | 186 MiB | 178 MiB |
* Random value size: [6 KiB, 14 KiB], Update Rate: 200/s, Total key space: 3,000
```
bnchmark put --rate=200 --total=300000 --compact-interval=0 \
--key-space-size=3000 --key-size=256 --val-size=10240 \
--delta-val-size=4096
```
| Compactor | DB Total Size | DB InUse Size |
| -- | -- | -- |
| Revision(5min,retension:10000) | 874 MiB | 221 MiB |
| Periodic(1m) | 357 MiB | 260 MiB |
| Periodic(30s) | 215 MiB | 151 MiB |
| NewRevision(retension:10000) | 182 MiB | 176 MiB |
For the burst requests, we needs to use short periodic interval.
Otherwise, the total size will be large. I think the new compactor can
handle it well.
Additional Change:
Currently, the quota system only checks DB total size. However, there
could be a lot of free pages which can be reused to upcoming requests.
Based on this proposal, I also want to extend current quota system with DB's
InUse size.
If the InUse size is less than max quota size, we should allow requests to
update. Since the bbolt might be resized if there is no available
continuous pages, we should setup a hard limit for the overflow, like 1
GiB.
```diff
// Quota represents an arbitrary quota against arbitrary requests. Each request
@@ -130,7 +134,17 @@ func (b *BackendQuota) Available(v interface{}) bool {
return true
}
// TODO: maybe optimize Backend.Size()
- return b.be.Size()+int64(cost) < b.maxBackendBytes
+
+ // Since the compact comes with allocatable pages, we should check the
+ // SizeInUse first. If there is no continuous pages for key/value and
+ // the boltdb continues to resize, it should not increase more than 1
+ // GiB. It's hard limitation.
+ //
+ // TODO: It should be enabled by flag.
+ if b.be.Size()+int64(cost)-b.maxBackendBytes >= maxAllowedOverflowBytes(b.maxBackendBytes) {
+ return false
+ }
+ return b.be.SizeInUse()+int64(cost) < b.maxBackendBytes
}
```
And it's likely to disable NOSPACE alarm if the compact can get much
more free pages. It can reduce downtime.
Signed-off-by: Wei Fu <fuweid89@gmail.com>
Two rules have been specified for yamlfmt here:
* Automatically add the header `---` to YAML files.
* Preserve line breaks.
ref:
https://github.com/google/yamlfmt/blob/main/docs/config-file.md#configuration-1
Signed-off-by: Jintao Zhang <zhangjintao9020@gmail.com>
Co-authored-by: Marek Siarkowicz <siarkowicz@google.com>
The schwag was introduced to generate swagger with authorization support
[1][1] in 2017. And in 2018, the grpc-gateway supports to render
security fields by protoc-gen-swagger [2][2]. After several years, I
think it's good to use upstream protoc supports.
NOTE:
The json's key in `rpc.swagger.json` has been reordered so that it seems
that there's a lot of changes. How to verify it:
```bash
$ # use jq -S to sort the key
$ latest_commit="https://raw.githubusercontent.com/etcd-io/etcd/228f493c7697ce3e9d3a1d831bcffad175846c75/Documentation/dev-guide/apispec/swagger/rpc.swagger.json"
$ curl -s "${latest_commit}" | jq -S . > /tmp/old.json
$ cat Documentation/dev-guide/apispec/swagger/rpc.swagger.json | jq -S . > /tmp/new.json
$ diff --color -u /tmp/old.json /tmp/new.json
```
```diff
--- /tmp/old.json 2023-04-26 10:58:07.142311861 +0800
+++ /tmp/new.json 2023-04-26 10:58:12.170299194 +0800
@@ -1523,11 +1523,14 @@
"type": "object"
},
"protobufAny": {
+ "description": "`Any` contains an arbitrary serialized protocol buffer message along with a\nURL that describes the type of the serialized message.\n\nProtobuf library provides support to pack/unpack Any values in the form\nof utility functions or additional generated methods of the Any type.\n\nExample 1: Pack and unpack a message in C++.\n\n Foo foo = ...;\n Any any;\n any.PackFrom(foo);\n ...\n if (any.UnpackTo(&foo)) {\n ...\n }\n\nExample 2: Pack and unpack a message in Java.\n\n Foo foo = ...;\n Any any = Any.pack(foo);\n ...\n if (any.is(Foo.class)) {\n foo = any.unpack(Foo.class);\n }\n\n Example 3: Pack and unpack a message in Python.\n\n foo = Foo(...)\n any = Any()\n any.Pack(foo)\n ...\n if any.Is(Foo.DESCRIPTOR):\n any.Unpack(foo)\n ...\n\n Example 4: Pack and unpack a message in Go\n\n foo := &pb.Foo{...}\n any, err := ptypes.MarshalAny(foo)\n ...\n foo := &pb.Foo{}\n if err := ptypes.UnmarshalAny(any, foo); err != nil {\n ...\n }\n\nThe pack methods provided by protobuf library will by default use\n'type.googleapis.com/full.type.name' as the type URL and the unpack\nmethods only use the fully qualified type name after the last '/'\nin the type URL, for example \"foo.bar.com/x/y.z\" will yield type\nname \"y.z\".\n\n\nJSON\n====\nThe JSON representation of an `Any` value uses the regular\nrepresentation of the deserialized, embedded message, with an\nadditional field `@type` which contains the type URL. Example:\n\n package google.profile;\n message Person {\n string first_name = 1;\n string last_name = 2;\n }\n\n {\n \"@type\": \"type.googleapis.com/google.profile.Person\",\n \"firstName\": <string>,\n \"lastName\": <string>\n }\n\nIf the embedded message type is well-known and has a custom JSON\nrepresentation, that representation will be embedded adding a field\n`value` which holds the custom JSON in addition to the `@type`\nfield. Example (for message [google.protobuf.Duration][]):\n\n {\n \"@type\": \"type.googleapis.com/google.protobuf.Duration\",\n \"value\": \"1.212s\"\n }",
"properties": {
"type_url": {
+ "description": "A URL/resource name that uniquely identifies the type of the serialized\nprotocol buffer message. This string must contain at least\none \"/\" character. The last segment of the URL's path must represent\nthe fully qualified name of the type (as in\n`path/google.protobuf.Duration`). The name should be in a canonical form\n(e.g., leading \".\" is not accepted).\n\nIn practice, teams usually precompile into the binary all types that they\nexpect it to use in the context of Any. However, for URLs which use the\nscheme `http`, `https`, or no scheme, one can optionally set up a type\nserver that maps type URLs to message definitions as follows:\n\n* If no scheme is provided, `https` is assumed.\n* An HTTP GET on the URL must yield a [google.protobuf.Type][]\n value in binary format, or produce an error.\n* Applications are allowed to cache lookup results based on the\n URL, or have them precompiled into a binary to avoid any\n lookup. Therefore, binary compatibility needs to be preserved\n on changes to types. (Use versioned type names to manage\n breaking changes.)\n\nNote: this functionality is not currently available in the official\nprotobuf release, and it is not used for type URLs beginning with\ntype.googleapis.com.\n\nSchemes other than `http`, `https` (or the empty scheme) might be\nused with implementation specific semantics.",
"type": "string"
},
"value": {
+ "description": "Must be a valid serialized protocol buffer of the above specified type.",
"format": "byte",
"type": "string"
}
```
REF:
1: <https://github.com/etcd-io/etcd/pull/7999#issuecomment-307512043>
2: <https://github.com/grpc-ecosystem/grpc-gateway/pull/547>
Signed-off-by: Wei Fu <fuweid89@gmail.com>
