This commit change the type of cached permission information from the
home made thing to interval tree. It improves computational complexity
of permission checking from O(n) to O(lg n).
This commit resolves a TODO of auth store:
Current scheme of role deletion allows existing users to have the
deleted roles. Assume a case like below:
create a role r1
create a user u1 and grant r1 to u1
delete r1
After this sequence, u1 is still granted the role r1. So if admin
create a new role with the name r1, The new r1 is automatically
granted u1. In some cases, it would be confusing. So we need to
revoke the deleted role from all users.
If the context does not include auth information, get authinfo will
return a nil auth info and a nil error. This is then passed to
IsAdminPermitted, which would dereference the nil auth info.
This commit adds jwt token support in v3 auth API.
Remaining major ToDos:
- Currently token type isn't hidden from etcdserver. In the near
future the information should be completely invisible from
etcdserver package.
- Configurable expiration of token. Currently tokens can be valid
until keys are changed.
How to use:
1. generate keys for signing and verfying jwt tokens:
$ openssl genrsa -out app.rsa 1024
$ openssl rsa -in app.rsa -pubout > app.rsa.pub
2. add command line options to etcd like below:
--auth-token-type jwt \
--auth-jwt-pub-key app.rsa.pub --auth-jwt-priv-key app.rsa \
--auth-jwt-sign-method RS512
3. launch etcd cluster
Below is a performance comparison of serializable read w/ and w/o jwt
token. Every (3) etcd node is executed on a single machine. Signing
method is RS512 and key length is 1024 bit. As the results show, jwt
based token introduces a performance overhead but it would be
acceptable for a case that requires authentication.
w/o jwt token auth (no auth):
Summary:
Total: 1.6172 secs.
Slowest: 0.0125 secs.
Fastest: 0.0001 secs.
Average: 0.0002 secs.
Stddev: 0.0004 secs.
Requests/sec: 6183.5877
Response time histogram:
0.000 [1] |
0.001 [9982] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
0.003 [1] |
0.004 [1] |
0.005 [0] |
0.006 [0] |
0.008 [6] |
0.009 [0] |
0.010 [1] |
0.011 [5] |
0.013 [3] |
Latency distribution:
10% in 0.0001 secs.
25% in 0.0001 secs.
50% in 0.0001 secs.
75% in 0.0001 secs.
90% in 0.0002 secs.
95% in 0.0002 secs.
99% in 0.0003 secs.
w/ jwt token auth:
Summary:
Total: 2.5364 secs.
Slowest: 0.0182 secs.
Fastest: 0.0002 secs.
Average: 0.0003 secs.
Stddev: 0.0005 secs.
Requests/sec: 3942.5185
Response time histogram:
0.000 [1] |
0.002 [9975] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
0.004 [0] |
0.006 [1] |
0.007 [11] |
0.009 [2] |
0.011 [4] |
0.013 [5] |
0.015 [0] |
0.016 [0] |
0.018 [1] |
Latency distribution:
10% in 0.0002 secs.
25% in 0.0002 secs.
50% in 0.0002 secs.
75% in 0.0002 secs.
90% in 0.0003 secs.
95% in 0.0003 secs.
99% in 0.0004 secs.
Because of my own silly mistake, current NewAuthStore() doesn't
initialize authStore in a correct manner. For example, after recovery
from snapshot, it cannot revive the flag of enabled/disabled. This
commit fixes the problem.
Fix https://github.com/coreos/etcd/issues/7165
The cost of bcrypt password checking is quite high (almost 100ms on a
modern machine) so executing it in apply loop will be
problematic. This commit exclude the checking mechanism to the API
layer. The password checking is validated with the OCC like way
similar to the auth of serializable get.
This commit also removes a unit test of Authenticate RPC from
auth/store_test.go. It is because the RPC now accepts an auth request
unconditionally and delegates the checking functionality to
authStore.CheckPassword() (so a unit test for CheckPassword() is
added). The combination of the two functionalities can be tested by
e2e (e.g. TestCtlV3AuthWriteKey).
Fixes https://github.com/coreos/etcd/issues/6530
This commit introduces revision of authStore. The revision number
represents a version of authStore that is incremented by updating auth
related information.
The revision is required for avoiding TOCTOU problems. Currently there
are two types of the TOCTOU problems in v3 auth.
The first one is in ordinal linearizable requests with a sequence like
below ():
1. Request from client CA is processed in follower FA. FA looks up the
username (let it U) for the request from a token of the request. At
this time, the request is authorized correctly.
2. Another request from client CB is processed in follower FB. CB
is for changing U's password.
3. FB forwards the request from CB to the leader before FA. Now U's
password is updated and the request from CA should be rejected.
4. However, the request from CA is processed by the leader because
authentication is already done in FA.
For avoiding the above sequence, this commit lets
etcdserverpb.RequestHeader have a member revision. The member is
initialized during authentication by followers and checked in a
leader. If the revision in RequestHeader is lower than the leader's
authStore revision, it means a sequence like above happened. In such a
case, the state machine returns auth.ErrAuthRevisionObsolete. The
error code lets nodes retry their requests.
The second one, a case of serializable range and txn, is more
subtle. Because these requests are processed in follower directly. The
TOCTOU problem can be caused by a sequence like below:
1. Serializable request from client CA is processed in follower FA. At
first, FA looks up the username (let it U) and its permission
before actual access to KV.
2. Another request from client CB is processed in follower FB and
forwarded to the leader. The cluster including FA now commits a log
entry of the request from CB. Assume the request changed the
permission or password of U.
3. Now the serializable request from CA is accessing to KV. Even if
the access is allowed at the point of 1, now it can be invalid
because of the change introduced in 2.
For avoiding the above sequence, this commit lets the functions of
serializable requests (EtcdServer.Range() and EtcdServer.Txn())
compare the revision in the request header with the latest revision of
authStore after the actual access. If the saved revision is lower than
the latest one, it means the permission can be changed. Although it
would introduce false positives (e.g. changing other user's password),
it prevents the TOCTOU problem. This idea is an implementation of
Anthony's comment:
https://github.com/coreos/etcd/pull/5739#issuecomment-228128254
This commit expands RPCs for getting user and role and support list up
all users and roles. etcdctl v3 is now support getting all users and
roles with the newly added option --all e.g. etcdctl user get --all
Currently auth tokens are generated in the replicated state machine
layer randomly. It means one auth token generated in node A cannot be
used for node B. It is problematic for load balancing and fail
over. This commit moves the token generation logic from the state
machine to API layer (before raft) and let all nodes share a single
token.
Log index of Raft is also added to a token for ensuring uniqueness of
the token and detecting activation of the token in the cluster (some
nodes can receive the token before generating and installing the token
in its state machine).
This commit also lets authStore have simple token related things. It
is required because of unit test. The test requires cleaning of the
state of the simple token things after one test (succeeding test can
create duplicated token and it causes panic).
This commit adds a feature for granting and revoking range of keys,
not a single key.
Example:
$ ETCDCTL_API=3 bin/etcdctl role grant r1 readwrite k1 k3
Role r1 updated
$ ETCDCTL_API=3 bin/etcdctl role get r1
Role r1
KV Read:
[a, b)
[k1, k3)
[k2, k4)
KV Write:
[a, b)
[k1, k3)
[k2, k4)
$ ETCDCTL_API=3 bin/etcdctl --user u1:p get k1 k4
k1
v1
$ ETCDCTL_API=3 bin/etcdctl --user u1:p get k1 k5
Error: etcdserver: permission denied
Currently the auth mechanism doesn't support permissions of range
request. It just checks exact matching of key names even for range
queries. This commit adds a mechanism for setting permission to range
queries. Range queries are allowed if a range of the query is [begin1,
end1) and the user has a permission of reading [begin2, range2) and
[begin1, end2) is a subset of [begin2, range2). Range delete requests
will follow the same rule.
This commit implements RoleGet() RPC of etcdserver and adds a new
subcommand "role get" to etcdctl v3. It will list up permissions that
are granted to a given role.
$ ETCDCTL_API=3 bin/etcdctl role get r1
Role r1
KV Read:
b
d
KV Write:
a
c
d
This commit adds a new subcommand "user get" to etcdctl v3. It will
list up roles that are granted to a given user.
Example:
$ ETCDCTL_API=3 bin/etcdctl user get u1
User: u1
Roles: r1 r2 r3
This commit also modifies the layout of InternalRaftRequest for
frequent update of auth related members.
