From 978cb54fc7eb7b0a8600151710e9ca131b2bf3a8 Mon Sep 17 00:00:00 2001 From: Brandon Philips Date: Wed, 27 Nov 2013 14:37:31 -0800 Subject: [PATCH 01/18] fix(README): use machine instead of node Internally we call every member of an etcd cluster a "machine" so stay consistent when talking in the README. --- README.md | 46 +++++++++++++++++++++++----------------------- 1 file changed, 23 insertions(+), 23 deletions(-) diff --git a/README.md b/README.md index 3caf06fd0..b18e42b66 100644 --- a/README.md +++ b/README.md @@ -56,18 +56,18 @@ go version ``` -### Running a single node +### Running a single machine -These examples will use a single node cluster to show you the basics of the etcd REST API. +These examples will use a single machine cluster to show you the basics of the etcd REST API. Let's start etcd: ```sh -./etcd -data-dir node0 -name node0 +./etcd -data-dir machine0 -name machine0 ``` -This will bring up an etcd node listening on port 4001 for client communication and on port 7001 for server-to-server communication. -The `-data-dir node0` argument tells etcd to write node configuration, logs and snapshots to the `./node0/` directory. -The `-name node0` tells the rest of the cluster that this node is named node0. +This will bring up etcd listening on port 4001 for client communication and on port 7001 for server-to-server communication. +The `-data-dir machine0` argument tells etcd to write machine configuration, logs and snapshots to the `./machine0/` directory. +The `-name machine` tells the rest of the cluster that this machine is named machine0. @@ -75,7 +75,7 @@ The `-name node0` tells the rest of the cluster that this node is named node0. ### Setting the value to a key -Let’s set the first key-value pair to the node. +Let’s set the first key-value pair to the datastore. In this case the key is `/message` and the value is `Hello world`. ```sh @@ -164,7 +164,7 @@ Note the two new fields in response: 2. The `ttl` is the time to live for the key, in seconds. -_NOTE_: Keys can only be expired by a cluster leader so if a node gets disconnected from the cluster, its keys will not expire until it rejoins. +_NOTE_: Keys can only be expired by a cluster leader so if a machine gets disconnected from the cluster, its keys will not expire until it rejoins. Now you can try to get the key by sending a `GET` request: @@ -378,12 +378,12 @@ For testing you can use the certificates in the `fixtures/ca` directory. Let's configure etcd to use this keypair: ```sh -./etcd -f -name node0 -data-dir node0 -cert-file=./fixtures/ca/server.crt -key-file=./fixtures/ca/server.key.insecure +./etcd -f -name machine0 -data-dir machine0 -cert-file=./fixtures/ca/server.crt -key-file=./fixtures/ca/server.key.insecure ``` There are a few new options we're using: -* `-f` - forces a new node configuration, even if an existing configuration is found. (WARNING: data loss!) +* `-f` - forces a new machine configuration, even if an existing configuration is found. (WARNING: data loss!) * `-cert-file` and `-key-file` specify the location of the cert and key files to be used for for transport layer security between the client and server. You can now test the configuration using HTTPS: @@ -413,7 +413,7 @@ We can also do authentication using CA certs. The clients will provide their cert to the server and the server will check whether the cert is signed by the CA and decide whether to serve the request. ```sh -./etcd -f -name node0 -data-dir node0 -ca-file=./fixtures/ca/ca.crt -cert-file=./fixtures/ca/server.crt -key-file=./fixtures/ca/server.key.insecure +./etcd -f -name machine0 -data-dir machine0 -ca-file=./fixtures/ca/ca.crt -cert-file=./fixtures/ca/server.crt -key-file=./fixtures/ca/server.key.insecure ``` ```-ca-file``` is the path to the CA cert. @@ -463,20 +463,20 @@ We use Raft as the underlying distributed protocol which provides consistency an Let start by creating 3 new etcd instances. -We use `-peer-addr` to specify server port and `-addr` to specify client port and `-data-dir` to specify the directory to store the log and info of the node in the cluster: +We use `-peer-addr` to specify server port and `-addr` to specify client port and `-data-dir` to specify the directory to store the log and info of the machine in the cluster: ```sh -./etcd -peer-addr 127.0.0.1:7001 -addr 127.0.0.1:4001 -data-dir nodes/node1 -name node1 +./etcd -peer-addr 127.0.0.1:7001 -addr 127.0.0.1:4001 -data-dir machines/machine1 -name machine1 ``` **Note:** If you want to run etcd on an external IP address and still have access locally, you'll need to add `-bind-addr 0.0.0.0` so that it will listen on both external and localhost addresses. A similar argument `-peer-bind-addr` is used to setup the listening address for the server port. -Let's join two more nodes to this cluster using the `-peers` argument: +Let's join two more machines to this cluster using the `-peers` argument: ```sh -./etcd -peer-addr 127.0.0.1:7002 -addr 127.0.0.1:4002 -peers 127.0.0.1:7001 -data-dir nodes/node2 -name node2 -./etcd -peer-addr 127.0.0.1:7003 -addr 127.0.0.1:4003 -peers 127.0.0.1:7001 -data-dir nodes/node3 -name node3 +./etcd -peer-addr 127.0.0.1:7002 -addr 127.0.0.1:4002 -peers 127.0.0.1:7001 -data-dir machines/machine2 -name machine2 +./etcd -peer-addr 127.0.0.1:7003 -addr 127.0.0.1:4003 -peers 127.0.0.1:7001 -data-dir machines/machine3 -name machine3 ``` We can retrieve a list of machines in the cluster using the HTTP API: @@ -485,7 +485,7 @@ We can retrieve a list of machines in the cluster using the HTTP API: curl -L http://127.0.0.1:4001/v1/machines ``` -We should see there are three nodes in the cluster +We should see there are three machines in the cluster ``` http://127.0.0.1:4001, http://127.0.0.1:4002, http://127.0.0.1:4003 @@ -498,7 +498,7 @@ curl -L http://127.0.0.1:4001/v1/keys/_etcd/machines ``` ```json -[{"action":"get","key":"/_etcd/machines/node1","value":"raft=http://127.0.0.1:7001\u0026etcd=http://127.0.0.1:4001","index":1},{"action":"get","key":"/_etcd/machines/node2","value":"raft=http://127.0.0.1:7002\u0026etcd=http://127.0.0.1:4002","index":1},{"action":"get","key":"/_etcd/machines/node3","value":"raft=http://127.0.0.1:7003\u0026etcd=http://127.0.0.1:4003","index":1}] +[{"action":"get","key":"/_etcd/machines/machine1","value":"raft=http://127.0.0.1:7001\u0026etcd=http://127.0.0.1:4001","index":1},{"action":"get","key":"/_etcd/machines/machine2","value":"raft=http://127.0.0.1:7002\u0026etcd=http://127.0.0.1:4002","index":1},{"action":"get","key":"/_etcd/machines/machine3","value":"raft=http://127.0.0.1:7003\u0026etcd=http://127.0.0.1:4003","index":1}] ``` We can also get the current leader in the cluster: @@ -551,8 +551,8 @@ http://127.0.0.1:7003 ### Testing Persistence -Next we'll kill all the nodes to test persistence. -Type `CTRL-C` on each terminal and then rerun the same command you used to start each node. +Next we'll kill all the machines to test persistence. +Type `CTRL-C` on each terminal and then rerun the same command you used to start each machine. Your request for the `foo` key will return the correct value: @@ -654,8 +654,8 @@ The command is not committed until the majority of the cluster peers receive tha Because of this majority voting property, the ideal cluster should be kept small to keep speed up and be made up of an odd number of peers. Odd numbers are good because if you have 8 peers the majority will be 5 and if you have 9 peers the majority will still be 5. -The result is that an 8 peer cluster can tolerate 3 peer failures and a 9 peer cluster can tolerate 4 nodes failures. -And in the best case when all 9 peers are responding the cluster will perform at the speed of the fastest 5 nodes. +The result is that an 8 peer cluster can tolerate 3 peer failures and a 9 peer cluster can tolerate 4 machine failures. +And in the best case when all 9 peers are responding the cluster will perform at the speed of the fastest 5 machines. ### Why SSLv3 alert handshake failure when using SSL client auth? @@ -677,7 +677,7 @@ Add the following section to your openssl.cnf: When creating the cert be sure to reference it in the `-extensions` flag: ``` -openssl ca -config openssl.cnf -policy policy_anything -extensions ssl_client -out certs/node.crt -infiles node.csr +openssl ca -config openssl.cnf -policy policy_anything -extensions ssl_client -out certs/machine.crt -infiles machine.csr ``` From 08c59895b5c9ed51902abd0d5414c7fcec196fb9 Mon Sep 17 00:00:00 2001 From: Xiang Li Date: Wed, 27 Nov 2013 21:02:28 -0500 Subject: [PATCH 02/18] refactor(Node) do not expose node struct --- store/heap_test.go | 2 +- store/node.go | 50 +++++++++++++++++++++---------------------- store/store.go | 21 +++++++++--------- store/ttl_key_heap.go | 18 ++++++++-------- 4 files changed, 45 insertions(+), 46 deletions(-) diff --git a/store/heap_test.go b/store/heap_test.go index 9175feef9..3397ae99f 100644 --- a/store/heap_test.go +++ b/store/heap_test.go @@ -33,7 +33,7 @@ func TestHeapPushPop(t *testing.T) { func TestHeapUpdate(t *testing.T) { h := newTtlKeyHeap() - kvs := make([]*Node, 10) + kvs := make([]*node, 10) // add from older expire time to earlier expire time // the path is equal to ttl from now diff --git a/store/node.go b/store/node.go index a4968e1a0..534750993 100644 --- a/store/node.go +++ b/store/node.go @@ -10,21 +10,21 @@ import ( var Permanent time.Time -// Node is the basic element in the store system. +// node is the basic element in the store system. // A key-value pair will have a string value // A directory will have a children map -type Node struct { +type node struct { Path string CreateIndex uint64 ModifiedIndex uint64 - Parent *Node `json:"-"` // should not encode this field! avoid circular dependency. + Parent *node `json:"-"` // should not encode this field! avoid circular dependency. ExpireTime time.Time ACL string Value string // for key-value pair - Children map[string]*Node // for directory + Children map[string]*node // for directory // A reference to the store this node is attached to. store *store @@ -32,9 +32,9 @@ type Node struct { // newKV creates a Key-Value pair func newKV(store *store, nodePath string, value string, createIndex uint64, - parent *Node, ACL string, expireTime time.Time) *Node { + parent *node, ACL string, expireTime time.Time) *node { - return &Node{ + return &node{ Path: nodePath, CreateIndex: createIndex, ModifiedIndex: createIndex, @@ -47,17 +47,17 @@ func newKV(store *store, nodePath string, value string, createIndex uint64, } // newDir creates a directory -func newDir(store *store, nodePath string, createIndex uint64, parent *Node, - ACL string, expireTime time.Time) *Node { +func newDir(store *store, nodePath string, createIndex uint64, parent *node, + ACL string, expireTime time.Time) *node { - return &Node{ + return &node{ Path: nodePath, CreateIndex: createIndex, ModifiedIndex: createIndex, Parent: parent, ACL: ACL, ExpireTime: expireTime, - Children: make(map[string]*Node), + Children: make(map[string]*node), store: store, } } @@ -67,14 +67,14 @@ func newDir(store *store, nodePath string, createIndex uint64, parent *Node, // A hidden node will not be shown via get command under a directory // For example if we have /foo/_hidden and /foo/notHidden, get "/foo" // will only return /foo/notHidden -func (n *Node) IsHidden() bool { +func (n *node) IsHidden() bool { _, name := path.Split(n.Path) return name[0] == '_' } // IsPermanent function checks if the node is a permanent one. -func (n *Node) IsPermanent() bool { +func (n *node) IsPermanent() bool { // we use a uninitialized time.Time to indicate the node is a // permanent one. // the uninitialized time.Time should equal zero. @@ -84,13 +84,13 @@ func (n *Node) IsPermanent() bool { // IsDir function checks whether the node is a directory. // If the node is a directory, the function will return true. // Otherwise the function will return false. -func (n *Node) IsDir() bool { +func (n *node) IsDir() bool { return !(n.Children == nil) } // Read function gets the value of the node. // If the receiver node is not a key-value pair, a "Not A File" error will be returned. -func (n *Node) Read() (string, *etcdErr.Error) { +func (n *node) Read() (string, *etcdErr.Error) { if n.IsDir() { return "", etcdErr.NewError(etcdErr.EcodeNotFile, "", n.store.Index()) } @@ -100,7 +100,7 @@ func (n *Node) Read() (string, *etcdErr.Error) { // Write function set the value of the node to the given value. // If the receiver node is a directory, a "Not A File" error will be returned. -func (n *Node) Write(value string, index uint64) *etcdErr.Error { +func (n *node) Write(value string, index uint64) *etcdErr.Error { if n.IsDir() { return etcdErr.NewError(etcdErr.EcodeNotFile, "", n.store.Index()) } @@ -111,7 +111,7 @@ func (n *Node) Write(value string, index uint64) *etcdErr.Error { return nil } -func (n *Node) ExpirationAndTTL() (*time.Time, int64) { +func (n *node) ExpirationAndTTL() (*time.Time, int64) { if !n.IsPermanent() { return &n.ExpireTime, int64(n.ExpireTime.Sub(time.Now())/time.Second) + 1 } @@ -120,12 +120,12 @@ func (n *Node) ExpirationAndTTL() (*time.Time, int64) { // List function return a slice of nodes under the receiver node. // If the receiver node is not a directory, a "Not A Directory" error will be returned. -func (n *Node) List() ([]*Node, *etcdErr.Error) { +func (n *node) List() ([]*node, *etcdErr.Error) { if !n.IsDir() { return nil, etcdErr.NewError(etcdErr.EcodeNotDir, "", n.store.Index()) } - nodes := make([]*Node, len(n.Children)) + nodes := make([]*node, len(n.Children)) i := 0 for _, node := range n.Children { @@ -138,7 +138,7 @@ func (n *Node) List() ([]*Node, *etcdErr.Error) { // GetChild function returns the child node under the directory node. // On success, it returns the file node -func (n *Node) GetChild(name string) (*Node, *etcdErr.Error) { +func (n *node) GetChild(name string) (*node, *etcdErr.Error) { if !n.IsDir() { return nil, etcdErr.NewError(etcdErr.EcodeNotDir, n.Path, n.store.Index()) } @@ -156,7 +156,7 @@ func (n *Node) GetChild(name string) (*Node, *etcdErr.Error) { // If the receiver is not a directory, a "Not A Directory" error will be returned. // If there is a existing node with the same name under the directory, a "Already Exist" // error will be returned -func (n *Node) Add(child *Node) *etcdErr.Error { +func (n *node) Add(child *node) *etcdErr.Error { if !n.IsDir() { return etcdErr.NewError(etcdErr.EcodeNotDir, "", n.store.Index()) } @@ -175,7 +175,7 @@ func (n *Node) Add(child *Node) *etcdErr.Error { } // Remove function remove the node. -func (n *Node) Remove(recursive bool, callback func(path string)) *etcdErr.Error { +func (n *node) Remove(recursive bool, callback func(path string)) *etcdErr.Error { if n.IsDir() && !recursive { // cannot delete a directory without set recursive to true @@ -223,7 +223,7 @@ func (n *Node) Remove(recursive bool, callback func(path string)) *etcdErr.Error return nil } -func (n *Node) Pair(recurisive, sorted bool) KeyValuePair { +func (n *node) Pair(recurisive, sorted bool) KeyValuePair { if n.IsDir() { pair := KeyValuePair{ Key: n.Path, @@ -272,7 +272,7 @@ func (n *Node) Pair(recurisive, sorted bool) KeyValuePair { return pair } -func (n *Node) UpdateTTL(expireTime time.Time) { +func (n *node) UpdateTTL(expireTime time.Time) { if !n.IsPermanent() { if expireTime.IsZero() { @@ -299,7 +299,7 @@ func (n *Node) UpdateTTL(expireTime time.Time) { // Clone function clone the node recursively and return the new node. // If the node is a directory, it will clone all the content under this directory. // If the node is a key-value pair, it will clone the pair. -func (n *Node) Clone() *Node { +func (n *node) Clone() *node { if !n.IsDir() { return newKV(n.store, n.Path, n.Value, n.CreateIndex, n.Parent, n.ACL, n.ExpireTime) } @@ -320,7 +320,7 @@ func (n *Node) Clone() *Node { // call this function on its children. // We check the expire last since we need to recover the whole structure first and add all the // notifications into the event history. -func (n *Node) recoverAndclean() { +func (n *node) recoverAndclean() { if n.IsDir() { for _, child := range n.Children { child.Parent = n diff --git a/store/store.go b/store/store.go index 80f9622af..78f8d227d 100644 --- a/store/store.go +++ b/store/store.go @@ -59,7 +59,7 @@ type Store interface { } type store struct { - Root *Node + Root *node WatcherHub *watcherHub CurrentIndex uint64 Stats *Stats @@ -152,7 +152,7 @@ func (s *store) Get(nodePath string, recursive, sorted bool) (*Event, error) { return e, nil } -// Create function creates the Node at nodePath. Create will help to create intermediate directories with no ttl. +// Create function creates the node at nodePath. Create will help to create intermediate directories with no ttl. // If the node has already existed, create will fail. // If any node on the path is a file, create will fail. func (s *store) Create(nodePath string, value string, unique bool, expireTime time.Time) (*Event, error) { @@ -171,7 +171,7 @@ func (s *store) Create(nodePath string, value string, unique bool, expireTime ti return e, err } -// Set function creates or replace the Node at nodePath. +// Set function creates or replace the node at nodePath. func (s *store) Set(nodePath string, value string, expireTime time.Time) (*Event, error) { nodePath = path.Clean(path.Join("/", nodePath)) @@ -309,7 +309,7 @@ func (s *store) Watch(prefix string, recursive bool, sinceIndex uint64) (<-chan } // walk function walks all the nodePath and apply the walkFunc on each directory -func (s *store) walk(nodePath string, walkFunc func(prev *Node, component string) (*Node, *etcdErr.Error)) (*Node, *etcdErr.Error) { +func (s *store) walk(nodePath string, walkFunc func(prev *node, component string) (*node, *etcdErr.Error)) (*node, *etcdErr.Error) { components := strings.Split(nodePath, "/") curr := s.Root @@ -396,8 +396,7 @@ func (s *store) internalCreate(nodePath string, value string, unique bool, repla expireTime = Permanent } - - dir, newNodeName := path.Split(nodePath) + dir, newnodeName := path.Split(nodePath) // walk through the nodePath, create dirs and get the last directory node d, err := s.walk(dir, s.checkDir) @@ -410,7 +409,7 @@ func (s *store) internalCreate(nodePath string, value string, unique bool, repla e := newEvent(action, nodePath, nextIndex) - n, _ := d.GetChild(newNodeName) + n, _ := d.GetChild(newnodeName) // force will try to replace a existing file if n != nil { @@ -441,7 +440,7 @@ func (s *store) internalCreate(nodePath string, value string, unique bool, repla // we are sure d is a directory and does not have the children with name n.Name d.Add(n) - // Node with TTL + // node with TTL if !n.IsPermanent() { s.ttlKeyHeap.push(n) @@ -455,10 +454,10 @@ func (s *store) internalCreate(nodePath string, value string, unique bool, repla } // InternalGet function get the node of the given nodePath. -func (s *store) internalGet(nodePath string) (*Node, *etcdErr.Error) { +func (s *store) internalGet(nodePath string) (*node, *etcdErr.Error) { nodePath = path.Clean(path.Join("/", nodePath)) - walkFunc := func(parent *Node, name string) (*Node, *etcdErr.Error) { + walkFunc := func(parent *node, name string) (*node, *etcdErr.Error) { if !parent.IsDir() { err := etcdErr.NewError(etcdErr.EcodeNotDir, parent.Path, s.CurrentIndex) @@ -507,7 +506,7 @@ func (s *store) DeleteExpiredKeys(cutoff time.Time) { // If it is a directory, this function will return the pointer to that node. // If it does not exist, this function will create a new directory and return the pointer to that node. // If it is a file, this function will return error. -func (s *store) checkDir(parent *Node, dirName string) (*Node, *etcdErr.Error) { +func (s *store) checkDir(parent *node, dirName string) (*node, *etcdErr.Error) { node, ok := parent.Children[dirName] if ok { diff --git a/store/ttl_key_heap.go b/store/ttl_key_heap.go index 0cda91d8d..0b4eb11ba 100644 --- a/store/ttl_key_heap.go +++ b/store/ttl_key_heap.go @@ -6,12 +6,12 @@ import ( // An TTLKeyHeap is a min-heap of TTLKeys order by expiration time type ttlKeyHeap struct { - array []*Node - keyMap map[*Node]int + array []*node + keyMap map[*node]int } func newTtlKeyHeap() *ttlKeyHeap { - h := &ttlKeyHeap{keyMap: make(map[*Node]int)} + h := &ttlKeyHeap{keyMap: make(map[*node]int)} heap.Init(h) return h } @@ -34,7 +34,7 @@ func (h ttlKeyHeap) Swap(i, j int) { } func (h *ttlKeyHeap) Push(x interface{}) { - n, _ := x.(*Node) + n, _ := x.(*node) h.keyMap[n] = len(h.array) h.array = append(h.array, n) } @@ -48,16 +48,16 @@ func (h *ttlKeyHeap) Pop() interface{} { return x } -func (h *ttlKeyHeap) top() *Node { +func (h *ttlKeyHeap) top() *node { if h.Len() != 0 { return h.array[0] } return nil } -func (h *ttlKeyHeap) pop() *Node { +func (h *ttlKeyHeap) pop() *node { x := heap.Pop(h) - n, _ := x.(*Node) + n, _ := x.(*node) return n } @@ -65,7 +65,7 @@ func (h *ttlKeyHeap) push(x interface{}) { heap.Push(h, x) } -func (h *ttlKeyHeap) update(n *Node) { +func (h *ttlKeyHeap) update(n *node) { index, ok := h.keyMap[n] if ok { heap.Remove(h, index) @@ -73,7 +73,7 @@ func (h *ttlKeyHeap) update(n *Node) { } } -func (h *ttlKeyHeap) remove(n *Node) { +func (h *ttlKeyHeap) remove(n *node) { index, ok := h.keyMap[n] if ok { heap.Remove(h, index) From b7d07ea5c86f54b0b4a336640abb3ed0abbc08cc Mon Sep 17 00:00:00 2001 From: Xiang Li Date: Wed, 27 Nov 2013 23:04:52 -0500 Subject: [PATCH 03/18] refactor event --- server/registry.go | 6 +- server/v2/tests/delete_handler_test.go | 2 +- server/v2/tests/get_handler_test.go | 54 ++++++----- server/v2/tests/post_handler_test.go | 14 ++- server/v2/tests/put_handler_test.go | 30 +++--- store/event.go | 58 ++++++------ store/event_history.go | 4 +- store/kv_pairs.go | 31 ------- store/node.go | 42 ++++----- store/node_repr.go | 35 +++++++ store/store.go | 59 ++++++------ store/store_test.go | 124 ++++++++++++------------- store/watcher_hub.go | 4 +- 13 files changed, 244 insertions(+), 219 deletions(-) delete mode 100644 store/kv_pairs.go create mode 100644 store/node_repr.go diff --git a/server/registry.go b/server/registry.go index d1d98d9ed..298e10a2f 100644 --- a/server/registry.go +++ b/server/registry.go @@ -70,7 +70,7 @@ func (r *Registry) Count() int { if err != nil { return 0 } - return len(e.KVPairs) + return len(e.Node.Nodes) } // Retrieves the client URL for a given node by name. @@ -135,7 +135,7 @@ func (r *Registry) urls(leaderName, selfName string, url func(name string) (stri // Retrieve a list of all nodes. if e, _ := r.store.Get(RegistryKey, false, false); e != nil { // Lookup the URL for each one. - for _, pair := range e.KVPairs { + for _, pair := range e.Node.Nodes { _, name := filepath.Split(pair.Key) if url, _ := url(name); len(url) > 0 && name != leaderName { urls = append(urls, url) @@ -166,7 +166,7 @@ func (r *Registry) load(name string) { } // Parse as a query string. - m, err := url.ParseQuery(e.Value) + m, err := url.ParseQuery(e.Node.Value) if err != nil { panic(fmt.Sprintf("Failed to parse peers entry: %s", name)) } diff --git a/server/v2/tests/delete_handler_test.go b/server/v2/tests/delete_handler_test.go index 997127a9e..f2f6decbc 100644 --- a/server/v2/tests/delete_handler_test.go +++ b/server/v2/tests/delete_handler_test.go @@ -24,6 +24,6 @@ func TestV2DeleteKey(t *testing.T) { resp, err = tests.DeleteForm(fmt.Sprintf("http://%s%s", s.URL(), "/v2/keys/foo/bar"), url.Values{}) body := tests.ReadBody(resp) assert.Nil(t, err, "") - assert.Equal(t, string(body), `{"action":"delete","key":"/foo/bar","prevValue":"XXX","modifiedIndex":2}`, "") + assert.Equal(t, string(body), `{"action":"delete","node":{"key":"/foo/bar","modifiedIndex":2}}`, "") }) } diff --git a/server/v2/tests/get_handler_test.go b/server/v2/tests/get_handler_test.go index b15195873..ea0ec1189 100644 --- a/server/v2/tests/get_handler_test.go +++ b/server/v2/tests/get_handler_test.go @@ -25,9 +25,11 @@ func TestV2GetKey(t *testing.T) { resp, _ = tests.Get(fmt.Sprintf("http://%s%s", s.URL(), "/v2/keys/foo/bar")) body := tests.ReadBodyJSON(resp) assert.Equal(t, body["action"], "get", "") - assert.Equal(t, body["key"], "/foo/bar", "") - assert.Equal(t, body["value"], "XXX", "") - assert.Equal(t, body["modifiedIndex"], 1, "") + + node := body["node"].(map[string]interface{}) + assert.Equal(t, node["key"], "/foo/bar", "") + assert.Equal(t, node["value"], "XXX", "") + assert.Equal(t, node["modifiedIndex"], 1, "") }) } @@ -52,23 +54,25 @@ func TestV2GetKeyRecursively(t *testing.T) { resp, _ = tests.Get(fmt.Sprintf("http://%s%s", s.URL(), "/v2/keys/foo?recursive=true")) body := tests.ReadBodyJSON(resp) assert.Equal(t, body["action"], "get", "") - assert.Equal(t, body["key"], "/foo", "") - assert.Equal(t, body["dir"], true, "") - assert.Equal(t, body["modifiedIndex"], 1, "") - assert.Equal(t, len(body["kvs"].([]interface{})), 2, "") - kv0 := body["kvs"].([]interface{})[0].(map[string]interface{}) - assert.Equal(t, kv0["key"], "/foo/x", "") - assert.Equal(t, kv0["value"], "XXX", "") - assert.Equal(t, kv0["ttl"], 10, "") + node := body["node"].(map[string]interface{}) + assert.Equal(t, node["key"], "/foo", "") + assert.Equal(t, node["dir"], true, "") + assert.Equal(t, node["modifiedIndex"], 1, "") + assert.Equal(t, len(node["nodes"].([]interface{})), 2, "") - kv1 := body["kvs"].([]interface{})[1].(map[string]interface{}) - assert.Equal(t, kv1["key"], "/foo/y", "") - assert.Equal(t, kv1["dir"], true, "") + node0 := node["nodes"].([]interface{})[0].(map[string]interface{}) + assert.Equal(t, node0["key"], "/foo/x", "") + assert.Equal(t, node0["value"], "XXX", "") + assert.Equal(t, node0["ttl"], 10, "") - kvs2 := kv1["kvs"].([]interface{})[0].(map[string]interface{}) - assert.Equal(t, kvs2["key"], "/foo/y/z", "") - assert.Equal(t, kvs2["value"], "YYY", "") + node1 := node["nodes"].([]interface{})[1].(map[string]interface{}) + assert.Equal(t, node1["key"], "/foo/y", "") + assert.Equal(t, node1["dir"], true, "") + + node2 := node1["nodes"].([]interface{})[0].(map[string]interface{}) + assert.Equal(t, node2["key"], "/foo/y/z", "") + assert.Equal(t, node2["value"], "YYY", "") }) } @@ -109,9 +113,11 @@ func TestV2WatchKey(t *testing.T) { assert.NotNil(t, body, "") assert.Equal(t, body["action"], "set", "") - assert.Equal(t, body["key"], "/foo/bar", "") - assert.Equal(t, body["value"], "XXX", "") - assert.Equal(t, body["modifiedIndex"], 1, "") + + node := body["node"].(map[string]interface{}) + assert.Equal(t, node["key"], "/foo/bar", "") + assert.Equal(t, node["value"], "XXX", "") + assert.Equal(t, node["modifiedIndex"], 1, "") }) } @@ -162,8 +168,10 @@ func TestV2WatchKeyWithIndex(t *testing.T) { assert.NotNil(t, body, "") assert.Equal(t, body["action"], "set", "") - assert.Equal(t, body["key"], "/foo/bar", "") - assert.Equal(t, body["value"], "YYY", "") - assert.Equal(t, body["modifiedIndex"], 2, "") + + node := body["node"].(map[string]interface{}) + assert.Equal(t, node["key"], "/foo/bar", "") + assert.Equal(t, node["value"], "YYY", "") + assert.Equal(t, node["modifiedIndex"], 2, "") }) } diff --git a/server/v2/tests/post_handler_test.go b/server/v2/tests/post_handler_test.go index 856633ef0..c0cb23078 100644 --- a/server/v2/tests/post_handler_test.go +++ b/server/v2/tests/post_handler_test.go @@ -21,18 +21,22 @@ func TestV2CreateUnique(t *testing.T) { resp, _ := tests.PostForm(fmt.Sprintf("http://%s%s", s.URL(), "/v2/keys/foo/bar"), nil) body := tests.ReadBodyJSON(resp) assert.Equal(t, body["action"], "create", "") - assert.Equal(t, body["key"], "/foo/bar/1", "") - assert.Equal(t, body["dir"], true, "") - assert.Equal(t, body["modifiedIndex"], 1, "") + + node := body["node"].(map[string]interface{}) + assert.Equal(t, node["key"], "/foo/bar/1", "") + assert.Equal(t, node["dir"], true, "") + assert.Equal(t, node["modifiedIndex"], 1, "") // Second POST should add next index to list. resp, _ = tests.PostForm(fmt.Sprintf("http://%s%s", s.URL(), "/v2/keys/foo/bar"), nil) body = tests.ReadBodyJSON(resp) - assert.Equal(t, body["key"], "/foo/bar/2", "") + node = body["node"].(map[string]interface{}) + assert.Equal(t, node["key"], "/foo/bar/2", "") // POST to a different key should add index to that list. resp, _ = tests.PostForm(fmt.Sprintf("http://%s%s", s.URL(), "/v2/keys/foo/baz"), nil) body = tests.ReadBodyJSON(resp) - assert.Equal(t, body["key"], "/foo/baz/3", "") + node = body["node"].(map[string]interface{}) + assert.Equal(t, node["key"], "/foo/baz/3", "") }) } diff --git a/server/v2/tests/put_handler_test.go b/server/v2/tests/put_handler_test.go index 3ee642604..dd98b065f 100644 --- a/server/v2/tests/put_handler_test.go +++ b/server/v2/tests/put_handler_test.go @@ -22,7 +22,7 @@ func TestV2SetKey(t *testing.T) { resp, err := tests.PutForm(fmt.Sprintf("http://%s%s", s.URL(), "/v2/keys/foo/bar"), v) body := tests.ReadBody(resp) assert.Nil(t, err, "") - assert.Equal(t, string(body), `{"action":"set","key":"/foo/bar","value":"XXX","modifiedIndex":1}`, "") + assert.Equal(t, string(body), `{"action":"set","node":{"key":"/foo/bar","value":"XXX","modifiedIndex":1}}`, "") }) } @@ -38,10 +38,11 @@ func TestV2SetKeyWithTTL(t *testing.T) { v.Set("ttl", "20") resp, _ := tests.PutForm(fmt.Sprintf("http://%s%s", s.URL(), "/v2/keys/foo/bar"), v) body := tests.ReadBodyJSON(resp) - assert.Equal(t, body["ttl"], 20, "") + node := body["node"].(map[string]interface{}) + assert.Equal(t, node["ttl"], 20, "") // Make sure the expiration date is correct. - expiration, _ := time.Parse(time.RFC3339Nano, body["expiration"].(string)) + expiration, _ := time.Parse(time.RFC3339Nano, node["expiration"].(string)) assert.Equal(t, expiration.Sub(t0)/time.Second, 20, "") }) } @@ -74,7 +75,8 @@ func TestV2CreateKeySuccess(t *testing.T) { v.Set("prevExist", "false") resp, _ := tests.PutForm(fmt.Sprintf("http://%s%s", s.URL(), "/v2/keys/foo/bar"), v) body := tests.ReadBodyJSON(resp) - assert.Equal(t, body["value"], "XXX", "") + node := body["node"].(map[string]interface{}) + assert.Equal(t, node["value"], "XXX", "") }) } @@ -116,7 +118,9 @@ func TestV2UpdateKeySuccess(t *testing.T) { resp, _ = tests.PutForm(fmt.Sprintf("http://%s%s", s.URL(), "/v2/keys/foo/bar"), v) body := tests.ReadBodyJSON(resp) assert.Equal(t, body["action"], "update", "") - assert.Equal(t, body["prevValue"], "XXX", "") + + node := body["node"].(map[string]interface{}) + assert.Equal(t, node["prevValue"], "XXX", "") }) } @@ -173,9 +177,11 @@ func TestV2SetKeyCASOnIndexSuccess(t *testing.T) { resp, _ = tests.PutForm(fmt.Sprintf("http://%s%s", s.URL(), "/v2/keys/foo/bar"), v) body := tests.ReadBodyJSON(resp) assert.Equal(t, body["action"], "compareAndSwap", "") - assert.Equal(t, body["prevValue"], "XXX", "") - assert.Equal(t, body["value"], "YYY", "") - assert.Equal(t, body["modifiedIndex"], 2, "") + + node := body["node"].(map[string]interface{}) + assert.Equal(t, node["prevValue"], "XXX", "") + assert.Equal(t, node["value"], "YYY", "") + assert.Equal(t, node["modifiedIndex"], 2, "") }) } @@ -234,9 +240,11 @@ func TestV2SetKeyCASOnValueSuccess(t *testing.T) { resp, _ = tests.PutForm(fmt.Sprintf("http://%s%s", s.URL(), "/v2/keys/foo/bar"), v) body := tests.ReadBodyJSON(resp) assert.Equal(t, body["action"], "compareAndSwap", "") - assert.Equal(t, body["prevValue"], "XXX", "") - assert.Equal(t, body["value"], "YYY", "") - assert.Equal(t, body["modifiedIndex"], 2, "") + + node := body["node"].(map[string]interface{}) + assert.Equal(t, node["prevValue"], "XXX", "") + assert.Equal(t, node["value"], "YYY", "") + assert.Equal(t, node["modifiedIndex"], 2, "") }) } diff --git a/store/event.go b/store/event.go index f3d607e0b..9a5a835c0 100644 --- a/store/event.go +++ b/store/event.go @@ -1,9 +1,5 @@ package store -import ( - "time" -) - const ( Get = "get" Create = "create" @@ -15,22 +11,20 @@ const ( ) type Event struct { - Action string `json:"action"` - Key string `json:"key, omitempty"` - Dir bool `json:"dir,omitempty"` - PrevValue string `json:"prevValue,omitempty"` - Value string `json:"value,omitempty"` - KVPairs kvPairs `json:"kvs,omitempty"` - Expiration *time.Time `json:"expiration,omitempty"` - TTL int64 `json:"ttl,omitempty"` // Time to live in second - ModifiedIndex uint64 `json:"modifiedIndex"` + Action string `json:"action"` + Node *Node `json:"node,omitempty"` } -func newEvent(action string, key string, index uint64) *Event { - return &Event{ - Action: action, +func newEvent(action string, key string, modifiedIndex, createdIndex uint64) *Event { + n := &Node{ Key: key, - ModifiedIndex: index, + ModifiedIndex: modifiedIndex, + CreatedIndex: createdIndex, + } + + return &Event{ + Action: action, + Node: n, } } @@ -39,7 +33,7 @@ func (e *Event) IsCreated() bool { return true } - if e.Action == Set && e.PrevValue == "" { + if e.Action == Set && e.Node.PrevValue == "" { return true } @@ -47,20 +41,20 @@ func (e *Event) IsCreated() bool { } func (e *Event) Index() uint64 { - return e.ModifiedIndex + return e.Node.ModifiedIndex } // Converts an event object into a response object. func (event *Event) Response() interface{} { - if !event.Dir { + if !event.Node.Dir { response := &Response{ Action: event.Action, - Key: event.Key, - Value: event.Value, - PrevValue: event.PrevValue, - Index: event.ModifiedIndex, - TTL: event.TTL, - Expiration: event.Expiration, + Key: event.Node.Key, + Value: event.Node.Value, + PrevValue: event.Node.PrevValue, + Index: event.Node.ModifiedIndex, + TTL: event.Node.TTL, + Expiration: event.Node.Expiration, } if response.Action == Set { @@ -75,15 +69,15 @@ func (event *Event) Response() interface{} { return response } else { - responses := make([]*Response, len(event.KVPairs)) + responses := make([]*Response, len(event.Node.Nodes)) - for i, kv := range event.KVPairs { + for i, node := range event.Node.Nodes { responses[i] = &Response{ Action: event.Action, - Key: kv.Key, - Value: kv.Value, - Dir: kv.Dir, - Index: event.ModifiedIndex, + Key: node.Key, + Value: node.Value, + Dir: node.Dir, + Index: node.ModifiedIndex, } } return responses diff --git a/store/event_history.go b/store/event_history.go index 4fd077184..ae29ad6ed 100644 --- a/store/event_history.go +++ b/store/event_history.go @@ -33,7 +33,7 @@ func (eh *EventHistory) addEvent(e *Event) *Event { eh.LastIndex = e.Index() - eh.StartIndex = eh.Queue.Events[eh.Queue.Front].ModifiedIndex + eh.StartIndex = eh.Queue.Events[eh.Queue.Front].Index() return e } @@ -62,7 +62,7 @@ func (eh *EventHistory) scan(prefix string, index uint64) (*Event, *etcdErr.Erro for { e := eh.Queue.Events[i] - if strings.HasPrefix(e.Key, prefix) && index <= e.Index() { // make sure we bypass the smaller one + if strings.HasPrefix(e.Node.Key, prefix) && index <= e.Index() { // make sure we bypass the smaller one return e, nil } diff --git a/store/kv_pairs.go b/store/kv_pairs.go deleted file mode 100644 index be6c01fb4..000000000 --- a/store/kv_pairs.go +++ /dev/null @@ -1,31 +0,0 @@ -package store - -import ( - "time" -) - -// When user list a directory, we add all the node into key-value pair slice -type KeyValuePair struct { - Key string `json:"key, omitempty"` - Value string `json:"value,omitempty"` - Dir bool `json:"dir,omitempty"` - Expiration *time.Time `json:"expiration,omitempty"` - TTL int64 `json:"ttl,omitempty"` // Time to live in second - KVPairs kvPairs `json:"kvs,omitempty"` - ModifiedIndex uint64 `json:"modifiedIndex,omitempty"` -} - -type kvPairs []KeyValuePair - -// interfaces for sorting -func (kvs kvPairs) Len() int { - return len(kvs) -} - -func (kvs kvPairs) Less(i, j int) bool { - return kvs[i].Key < kvs[j].Key -} - -func (kvs kvPairs) Swap(i, j int) { - kvs[i], kvs[j] = kvs[j], kvs[i] -} diff --git a/store/node.go b/store/node.go index 534750993..809b4cadb 100644 --- a/store/node.go +++ b/store/node.go @@ -16,7 +16,7 @@ var Permanent time.Time type node struct { Path string - CreateIndex uint64 + CreatedIndex uint64 ModifiedIndex uint64 Parent *node `json:"-"` // should not encode this field! avoid circular dependency. @@ -31,13 +31,13 @@ type node struct { } // newKV creates a Key-Value pair -func newKV(store *store, nodePath string, value string, createIndex uint64, +func newKV(store *store, nodePath string, value string, createdIndex uint64, parent *node, ACL string, expireTime time.Time) *node { return &node{ Path: nodePath, - CreateIndex: createIndex, - ModifiedIndex: createIndex, + CreatedIndex: createdIndex, + ModifiedIndex: createdIndex, Parent: parent, ACL: ACL, store: store, @@ -47,13 +47,13 @@ func newKV(store *store, nodePath string, value string, createIndex uint64, } // newDir creates a directory -func newDir(store *store, nodePath string, createIndex uint64, parent *node, +func newDir(store *store, nodePath string, createdIndex uint64, parent *node, ACL string, expireTime time.Time) *node { return &node{ Path: nodePath, - CreateIndex: createIndex, - ModifiedIndex: createIndex, + CreatedIndex: createdIndex, + ModifiedIndex: createdIndex, Parent: parent, ACL: ACL, ExpireTime: expireTime, @@ -223,21 +223,21 @@ func (n *node) Remove(recursive bool, callback func(path string)) *etcdErr.Error return nil } -func (n *node) Pair(recurisive, sorted bool) KeyValuePair { +func (n *node) Repr(recurisive, sorted bool) Node { if n.IsDir() { - pair := KeyValuePair{ + node := Node{ Key: n.Path, Dir: true, ModifiedIndex: n.ModifiedIndex, } - pair.Expiration, pair.TTL = n.ExpirationAndTTL() + node.Expiration, node.TTL = n.ExpirationAndTTL() if !recurisive { - return pair + return node } children, _ := n.List() - pair.KVPairs = make([]KeyValuePair, len(children)) + node.Nodes = make(Nodes, len(children)) // we do not use the index in the children slice directly // we need to skip the hidden one @@ -249,27 +249,27 @@ func (n *node) Pair(recurisive, sorted bool) KeyValuePair { continue } - pair.KVPairs[i] = child.Pair(recurisive, sorted) + node.Nodes[i] = child.Repr(recurisive, sorted) i++ } // eliminate hidden nodes - pair.KVPairs = pair.KVPairs[:i] + node.Nodes = node.Nodes[:i] if sorted { - sort.Sort(pair.KVPairs) + sort.Sort(node.Nodes) } - return pair + return node } - pair := KeyValuePair{ + node := Node{ Key: n.Path, Value: n.Value, ModifiedIndex: n.ModifiedIndex, } - pair.Expiration, pair.TTL = n.ExpirationAndTTL() - return pair + node.Expiration, node.TTL = n.ExpirationAndTTL() + return node } func (n *node) UpdateTTL(expireTime time.Time) { @@ -301,10 +301,10 @@ func (n *node) UpdateTTL(expireTime time.Time) { // If the node is a key-value pair, it will clone the pair. func (n *node) Clone() *node { if !n.IsDir() { - return newKV(n.store, n.Path, n.Value, n.CreateIndex, n.Parent, n.ACL, n.ExpireTime) + return newKV(n.store, n.Path, n.Value, n.CreatedIndex, n.Parent, n.ACL, n.ExpireTime) } - clone := newDir(n.store, n.Path, n.CreateIndex, n.Parent, n.ACL, n.ExpireTime) + clone := newDir(n.store, n.Path, n.CreatedIndex, n.Parent, n.ACL, n.ExpireTime) for key, child := range n.Children { clone.Children[key] = child.Clone() diff --git a/store/node_repr.go b/store/node_repr.go new file mode 100644 index 000000000..2dabf95be --- /dev/null +++ b/store/node_repr.go @@ -0,0 +1,35 @@ +package store + +import ( + "time" +) + +// Node is the representation of the internal node with additional fields +// PrevValue is the previous value of the node +// TTL is time to live in second +type Node struct { + Key string `json:"key, omitempty"` + PrevValue string `json:"prevValue,omitempty"` + Value string `json:"value,omitempty"` + Dir bool `json:"dir,omitempty"` + Expiration *time.Time `json:"expiration,omitempty"` + TTL int64 `json:"ttl,omitempty"` + Nodes Nodes `json:"nodes,omitempty"` + ModifiedIndex uint64 `json:"modifiedIndex,omitempty"` + CreatedIndex uint64 `json:"createdIndex,omitempty"` +} + +type Nodes []Node + +// interfaces for sorting +func (ns Nodes) Len() int { + return len(ns) +} + +func (ns Nodes) Less(i, j int) bool { + return ns[i].Key < ns[j].Key +} + +func (ns Nodes) Swap(i, j int) { + ns[i], ns[j] = ns[j], ns[i] +} diff --git a/store/store.go b/store/store.go index 78f8d227d..95db10037 100644 --- a/store/store.go +++ b/store/store.go @@ -113,13 +113,14 @@ func (s *store) Get(nodePath string, recursive, sorted bool) (*Event, error) { return nil, err } - e := newEvent(Get, nodePath, n.ModifiedIndex) + e := newEvent(Get, nodePath, n.ModifiedIndex, n.CreatedIndex) + eNode := e.Node if n.IsDir() { // node is a directory - e.Dir = true + eNode.Dir = true children, _ := n.List() - e.KVPairs = make([]KeyValuePair, len(children)) + eNode.Nodes = make(Nodes, len(children)) // we do not use the index in the children slice directly // we need to skip the hidden one @@ -130,22 +131,22 @@ func (s *store) Get(nodePath string, recursive, sorted bool) (*Event, error) { continue } - e.KVPairs[i] = child.Pair(recursive, sorted) + eNode.Nodes[i] = child.Repr(recursive, sorted) i++ } // eliminate hidden nodes - e.KVPairs = e.KVPairs[:i] + eNode.Nodes = eNode.Nodes[:i] if sorted { - sort.Sort(e.KVPairs) + sort.Sort(eNode.Nodes) } } else { // node is a file - e.Value, _ = n.Read() + eNode.Value, _ = n.Read() } - e.Expiration, e.TTL = n.ExpirationAndTTL() + eNode.Expiration, eNode.TTL = n.ExpirationAndTTL() s.Stats.Inc(GetSuccess) @@ -214,15 +215,17 @@ func (s *store) CompareAndSwap(nodePath string, prevValue string, prevIndex uint // update etcd index s.CurrentIndex++ - e := newEvent(CompareAndSwap, nodePath, s.CurrentIndex) - e.PrevValue = n.Value + e := newEvent(CompareAndSwap, nodePath, s.CurrentIndex, n.CreatedIndex) + eNode := e.Node + + eNode.PrevValue = n.Value // if test succeed, write the value n.Write(value, s.CurrentIndex) n.UpdateTTL(expireTime) - e.Value = value - e.Expiration, e.TTL = n.ExpirationAndTTL() + eNode.Value = value + eNode.Expiration, eNode.TTL = n.ExpirationAndTTL() s.WatcherHub.notify(e) s.Stats.Inc(CompareAndSwapSuccess) @@ -251,12 +254,13 @@ func (s *store) Delete(nodePath string, recursive bool) (*Event, error) { return nil, err } - e := newEvent(Delete, nodePath, nextIndex) + e := newEvent(Delete, nodePath, nextIndex, n.CreatedIndex) + eNode := e.Node if n.IsDir() { - e.Dir = true + eNode.Dir = true } else { - e.PrevValue = n.Value + eNode.PrevValue = eNode.Value } callback := func(path string) { // notify function @@ -348,7 +352,8 @@ func (s *store) Update(nodePath string, newValue string, expireTime time.Time) ( return nil, err } - e := newEvent(Update, nodePath, nextIndex) + e := newEvent(Update, nodePath, nextIndex, n.CreatedIndex) + eNode := e.Node if len(newValue) != 0 { if n.IsDir() { @@ -357,18 +362,19 @@ func (s *store) Update(nodePath string, newValue string, expireTime time.Time) ( return nil, etcdErr.NewError(etcdErr.EcodeNotFile, nodePath, currIndex) } - e.PrevValue = n.Value + eNode.PrevValue = n.Value n.Write(newValue, nextIndex) - e.Value = newValue + eNode.Value = newValue + } else { // do not update value - e.Value = n.Value + eNode.Value = n.Value } // update ttl n.UpdateTTL(expireTime) - e.Expiration, e.TTL = n.ExpirationAndTTL() + eNode.Expiration, eNode.TTL = n.ExpirationAndTTL() s.WatcherHub.notify(e) @@ -407,7 +413,8 @@ func (s *store) internalCreate(nodePath string, value string, unique bool, repla return nil, err } - e := newEvent(action, nodePath, nextIndex) + e := newEvent(action, nodePath, nextIndex, nextIndex) + eNode := e.Node n, _ := d.GetChild(newnodeName) @@ -417,7 +424,7 @@ func (s *store) internalCreate(nodePath string, value string, unique bool, repla if n.IsDir() { return nil, etcdErr.NewError(etcdErr.EcodeNotFile, nodePath, currIndex) } - e.PrevValue, _ = n.Read() + eNode.PrevValue, _ = n.Read() n.Remove(false, nil) } else { @@ -426,12 +433,12 @@ func (s *store) internalCreate(nodePath string, value string, unique bool, repla } if len(value) != 0 { // create file - e.Value = value + eNode.Value = value n = newKV(s, nodePath, value, nextIndex, d, "", expireTime) } else { // create directory - e.Dir = true + eNode.Dir = true n = newDir(s, nodePath, nextIndex, d, "", expireTime) @@ -444,7 +451,7 @@ func (s *store) internalCreate(nodePath string, value string, unique bool, repla if !n.IsPermanent() { s.ttlKeyHeap.push(n) - e.Expiration, e.TTL = n.ExpirationAndTTL() + eNode.Expiration, eNode.TTL = n.ExpirationAndTTL() } s.CurrentIndex = nextIndex @@ -497,7 +504,7 @@ func (s *store) DeleteExpiredKeys(cutoff time.Time) { s.CurrentIndex++ s.Stats.Inc(ExpireCount) - s.WatcherHub.notify(newEvent(Expire, node.Path, s.CurrentIndex)) + s.WatcherHub.notify(newEvent(Expire, node.Path, s.CurrentIndex, node.CreatedIndex)) } } diff --git a/store/store_test.go b/store/store_test.go index 29b2986b9..602dc7174 100644 --- a/store/store_test.go +++ b/store/store_test.go @@ -31,8 +31,8 @@ func TestStoreGetValue(t *testing.T) { e, err := s.Get("/foo", false, false) assert.Nil(t, err, "") assert.Equal(t, e.Action, "get", "") - assert.Equal(t, e.Key, "/foo", "") - assert.Equal(t, e.Value, "bar", "") + assert.Equal(t, e.Node.Key, "/foo", "") + assert.Equal(t, e.Node.Value, "bar", "") } // Ensure that the store can recrusively retrieve a directory listing. @@ -49,21 +49,21 @@ func TestStoreGetDirectory(t *testing.T) { e, err := s.Get("/foo", true, false) assert.Nil(t, err, "") assert.Equal(t, e.Action, "get", "") - assert.Equal(t, e.Key, "/foo", "") - assert.Equal(t, len(e.KVPairs), 2, "") - assert.Equal(t, e.KVPairs[0].Key, "/foo/bar", "") - assert.Equal(t, e.KVPairs[0].Value, "X", "") - assert.Equal(t, e.KVPairs[0].Dir, false, "") - assert.Equal(t, e.KVPairs[1].Key, "/foo/baz", "") - assert.Equal(t, e.KVPairs[1].Dir, true, "") - assert.Equal(t, len(e.KVPairs[1].KVPairs), 2, "") - assert.Equal(t, e.KVPairs[1].KVPairs[0].Key, "/foo/baz/bat", "") - assert.Equal(t, e.KVPairs[1].KVPairs[0].Value, "Y", "") - assert.Equal(t, e.KVPairs[1].KVPairs[0].Dir, false, "") - assert.Equal(t, e.KVPairs[1].KVPairs[1].Key, "/foo/baz/ttl", "") - assert.Equal(t, e.KVPairs[1].KVPairs[1].Value, "Y", "") - assert.Equal(t, e.KVPairs[1].KVPairs[1].Dir, false, "") - assert.Equal(t, e.KVPairs[1].KVPairs[1].TTL, 3, "") + assert.Equal(t, e.Node.Key, "/foo", "") + assert.Equal(t, len(e.Node.Nodes), 2, "") + assert.Equal(t, e.Node.Nodes[0].Key, "/foo/bar", "") + assert.Equal(t, e.Node.Nodes[0].Value, "X", "") + assert.Equal(t, e.Node.Nodes[0].Dir, false, "") + assert.Equal(t, e.Node.Nodes[1].Key, "/foo/baz", "") + assert.Equal(t, e.Node.Nodes[1].Dir, true, "") + assert.Equal(t, len(e.Node.Nodes[1].Nodes), 2, "") + assert.Equal(t, e.Node.Nodes[1].Nodes[0].Key, "/foo/baz/bat", "") + assert.Equal(t, e.Node.Nodes[1].Nodes[0].Value, "Y", "") + assert.Equal(t, e.Node.Nodes[1].Nodes[0].Dir, false, "") + assert.Equal(t, e.Node.Nodes[1].Nodes[1].Key, "/foo/baz/ttl", "") + assert.Equal(t, e.Node.Nodes[1].Nodes[1].Value, "Y", "") + assert.Equal(t, e.Node.Nodes[1].Nodes[1].Dir, false, "") + assert.Equal(t, e.Node.Nodes[1].Nodes[1].TTL, 3, "") } // Ensure that the store can retrieve a directory in sorted order. @@ -77,11 +77,11 @@ func TestStoreGetSorted(t *testing.T) { s.Create("/foo/y/b", "0", false, Permanent) e, err := s.Get("/foo", true, true) assert.Nil(t, err, "") - assert.Equal(t, e.KVPairs[0].Key, "/foo/x", "") - assert.Equal(t, e.KVPairs[1].Key, "/foo/y", "") - assert.Equal(t, e.KVPairs[1].KVPairs[0].Key, "/foo/y/a", "") - assert.Equal(t, e.KVPairs[1].KVPairs[1].Key, "/foo/y/b", "") - assert.Equal(t, e.KVPairs[2].Key, "/foo/z", "") + assert.Equal(t, e.Node.Nodes[0].Key, "/foo/x", "") + assert.Equal(t, e.Node.Nodes[1].Key, "/foo/y", "") + assert.Equal(t, e.Node.Nodes[1].Nodes[0].Key, "/foo/y/a", "") + assert.Equal(t, e.Node.Nodes[1].Nodes[1].Key, "/foo/y/b", "") + assert.Equal(t, e.Node.Nodes[2].Key, "/foo/z", "") } // Ensure that the store can create a new key if it doesn't already exist. @@ -90,14 +90,14 @@ func TestStoreCreateValue(t *testing.T) { e, err := s.Create("/foo", "bar", false, Permanent) assert.Nil(t, err, "") assert.Equal(t, e.Action, "create", "") - assert.Equal(t, e.Key, "/foo", "") - assert.False(t, e.Dir, "") - assert.Equal(t, e.PrevValue, "", "") - assert.Equal(t, e.Value, "bar", "") - assert.Nil(t, e.KVPairs, "") - assert.Nil(t, e.Expiration, "") - assert.Equal(t, e.TTL, 0, "") - assert.Equal(t, e.ModifiedIndex, uint64(1), "") + assert.Equal(t, e.Node.Key, "/foo", "") + assert.False(t, e.Node.Dir, "") + assert.Equal(t, e.Node.PrevValue, "", "") + assert.Equal(t, e.Node.Value, "bar", "") + assert.Nil(t, e.Node.Nodes, "") + assert.Nil(t, e.Node.Expiration, "") + assert.Equal(t, e.Node.TTL, 0, "") + assert.Equal(t, e.Node.ModifiedIndex, uint64(1), "") } // Ensure that the store can create a new directory if it doesn't already exist. @@ -106,8 +106,8 @@ func TestStoreCreateDirectory(t *testing.T) { e, err := s.Create("/foo", "", false, Permanent) assert.Nil(t, err, "") assert.Equal(t, e.Action, "create", "") - assert.Equal(t, e.Key, "/foo", "") - assert.True(t, e.Dir, "") + assert.Equal(t, e.Node.Key, "/foo", "") + assert.True(t, e.Node.Dir, "") } // Ensure that the store fails to create a key if it already exists. @@ -130,14 +130,14 @@ func TestStoreUpdateValue(t *testing.T) { e, err := s.Update("/foo", "baz", Permanent) assert.Nil(t, err, "") assert.Equal(t, e.Action, "update", "") - assert.Equal(t, e.Key, "/foo", "") - assert.False(t, e.Dir, "") - assert.Equal(t, e.PrevValue, "bar", "") - assert.Equal(t, e.Value, "baz", "") - assert.Equal(t, e.TTL, 0, "") - assert.Equal(t, e.ModifiedIndex, uint64(2), "") + assert.Equal(t, e.Node.Key, "/foo", "") + assert.False(t, e.Node.Dir, "") + assert.Equal(t, e.Node.PrevValue, "bar", "") + assert.Equal(t, e.Node.Value, "baz", "") + assert.Equal(t, e.Node.TTL, 0, "") + assert.Equal(t, e.Node.ModifiedIndex, uint64(2), "") e, _ = s.Get("/foo", false, false) - assert.Equal(t, e.Value, "baz", "") + assert.Equal(t, e.Node.Value, "baz", "") } // Ensure that the store cannot update a directory. @@ -165,7 +165,7 @@ func TestStoreUpdateValueTTL(t *testing.T) { s.Create("/foo", "bar", false, Permanent) _, err := s.Update("/foo", "baz", time.Now().Add(500*time.Millisecond)) e, _ := s.Get("/foo", false, false) - assert.Equal(t, e.Value, "baz", "") + assert.Equal(t, e.Node.Value, "baz", "") time.Sleep(600 * time.Millisecond) e, err = s.Get("/foo", false, false) @@ -187,7 +187,7 @@ func TestStoreUpdateDirTTL(t *testing.T) { s.Create("/foo/bar", "baz", false, Permanent) _, err := s.Update("/foo", "", time.Now().Add(500*time.Millisecond)) e, _ := s.Get("/foo/bar", false, false) - assert.Equal(t, e.Value, "baz", "") + assert.Equal(t, e.Node.Value, "baz", "") time.Sleep(600 * time.Millisecond) e, err = s.Get("/foo/bar", false, false) @@ -231,10 +231,10 @@ func TestStoreCompareAndSwapPrevValue(t *testing.T) { e, err := s.CompareAndSwap("/foo", "bar", 0, "baz", Permanent) assert.Nil(t, err, "") assert.Equal(t, e.Action, "compareAndSwap", "") - assert.Equal(t, e.PrevValue, "bar", "") - assert.Equal(t, e.Value, "baz", "") + assert.Equal(t, e.Node.PrevValue, "bar", "") + assert.Equal(t, e.Node.Value, "baz", "") e, _ = s.Get("/foo", false, false) - assert.Equal(t, e.Value, "baz", "") + assert.Equal(t, e.Node.Value, "baz", "") } // Ensure that the store cannot conditionally update a key if it has the wrong previous value. @@ -247,7 +247,7 @@ func TestStoreCompareAndSwapPrevValueFailsIfNotMatch(t *testing.T) { assert.Equal(t, err.Message, "Test Failed", "") assert.Nil(t, e, "") e, _ = s.Get("/foo", false, false) - assert.Equal(t, e.Value, "bar", "") + assert.Equal(t, e.Node.Value, "bar", "") } // Ensure that the store can conditionally update a key if it has a previous index. @@ -257,10 +257,10 @@ func TestStoreCompareAndSwapPrevIndex(t *testing.T) { e, err := s.CompareAndSwap("/foo", "", 1, "baz", Permanent) assert.Nil(t, err, "") assert.Equal(t, e.Action, "compareAndSwap", "") - assert.Equal(t, e.PrevValue, "bar", "") - assert.Equal(t, e.Value, "baz", "") + assert.Equal(t, e.Node.PrevValue, "bar", "") + assert.Equal(t, e.Node.Value, "baz", "") e, _ = s.Get("/foo", false, false) - assert.Equal(t, e.Value, "baz", "") + assert.Equal(t, e.Node.Value, "baz", "") } // Ensure that the store cannot conditionally update a key if it has the wrong previous index. @@ -273,7 +273,7 @@ func TestStoreCompareAndSwapPrevIndexFailsIfNotMatch(t *testing.T) { assert.Equal(t, err.Message, "Test Failed", "") assert.Nil(t, e, "") e, _ = s.Get("/foo", false, false) - assert.Equal(t, e.Value, "bar", "") + assert.Equal(t, e.Node.Value, "bar", "") } // Ensure that the store can watch for key creation. @@ -283,7 +283,7 @@ func TestStoreWatchCreate(t *testing.T) { s.Create("/foo", "bar", false, Permanent) e := nbselect(c) assert.Equal(t, e.Action, "create", "") - assert.Equal(t, e.Key, "/foo", "") + assert.Equal(t, e.Node.Key, "/foo", "") e = nbselect(c) assert.Nil(t, e, "") } @@ -295,7 +295,7 @@ func TestStoreWatchRecursiveCreate(t *testing.T) { s.Create("/foo/bar", "baz", false, Permanent) e := nbselect(c) assert.Equal(t, e.Action, "create", "") - assert.Equal(t, e.Key, "/foo/bar", "") + assert.Equal(t, e.Node.Key, "/foo/bar", "") } // Ensure that the store can watch for key updates. @@ -306,7 +306,7 @@ func TestStoreWatchUpdate(t *testing.T) { s.Update("/foo", "baz", Permanent) e := nbselect(c) assert.Equal(t, e.Action, "update", "") - assert.Equal(t, e.Key, "/foo", "") + assert.Equal(t, e.Node.Key, "/foo", "") } // Ensure that the store can watch for recursive key updates. @@ -317,7 +317,7 @@ func TestStoreWatchRecursiveUpdate(t *testing.T) { s.Update("/foo/bar", "baz", Permanent) e := nbselect(c) assert.Equal(t, e.Action, "update", "") - assert.Equal(t, e.Key, "/foo/bar", "") + assert.Equal(t, e.Node.Key, "/foo/bar", "") } // Ensure that the store can watch for key deletions. @@ -328,7 +328,7 @@ func TestStoreWatchDelete(t *testing.T) { s.Delete("/foo", false) e := nbselect(c) assert.Equal(t, e.Action, "delete", "") - assert.Equal(t, e.Key, "/foo", "") + assert.Equal(t, e.Node.Key, "/foo", "") } // Ensure that the store can watch for recursive key deletions. @@ -339,7 +339,7 @@ func TestStoreWatchRecursiveDelete(t *testing.T) { s.Delete("/foo/bar", false) e := nbselect(c) assert.Equal(t, e.Action, "delete", "") - assert.Equal(t, e.Key, "/foo/bar", "") + assert.Equal(t, e.Node.Key, "/foo/bar", "") } // Ensure that the store can watch for CAS updates. @@ -350,7 +350,7 @@ func TestStoreWatchCompareAndSwap(t *testing.T) { s.CompareAndSwap("/foo", "bar", 0, "baz", Permanent) e := nbselect(c) assert.Equal(t, e.Action, "compareAndSwap", "") - assert.Equal(t, e.Key, "/foo", "") + assert.Equal(t, e.Node.Key, "/foo", "") } // Ensure that the store can watch for recursive CAS updates. @@ -361,7 +361,7 @@ func TestStoreWatchRecursiveCompareAndSwap(t *testing.T) { s.CompareAndSwap("/foo/bar", "baz", 0, "bat", Permanent) e := nbselect(c) assert.Equal(t, e.Action, "compareAndSwap", "") - assert.Equal(t, e.Key, "/foo/bar", "") + assert.Equal(t, e.Node.Key, "/foo/bar", "") } // Ensure that the store can watch for key expiration. @@ -383,11 +383,11 @@ func TestStoreWatchExpire(t *testing.T) { time.Sleep(600 * time.Millisecond) e = nbselect(c) assert.Equal(t, e.Action, "expire", "") - assert.Equal(t, e.Key, "/foo", "") + assert.Equal(t, e.Node.Key, "/foo", "") c, _ = s.Watch("/", true, 4) e = nbselect(c) assert.Equal(t, e.Action, "expire", "") - assert.Equal(t, e.Key, "/foofoo", "") + assert.Equal(t, e.Node.Key, "/foofoo", "") } // Ensure that the store can recover from a previously saved state. @@ -403,11 +403,11 @@ func TestStoreRecover(t *testing.T) { e, err := s.Get("/foo/x", false, false) assert.Nil(t, err, "") - assert.Equal(t, e.Value, "bar", "") + assert.Equal(t, e.Node.Value, "bar", "") e, err = s.Get("/foo/y", false, false) assert.Nil(t, err, "") - assert.Equal(t, e.Value, "baz", "") + assert.Equal(t, e.Node.Value, "baz", "") } // Ensure that the store can recover from a previously saved state that includes an expiring key. @@ -441,7 +441,7 @@ func TestStoreRecoverWithExpiration(t *testing.T) { e, err := s.Get("/foo/x", false, false) assert.Nil(t, err, "") - assert.Equal(t, e.Value, "bar", "") + assert.Equal(t, e.Node.Value, "bar", "") e, err = s.Get("/foo/y", false, false) assert.NotNil(t, err, "") diff --git a/store/watcher_hub.go b/store/watcher_hub.go index b952aec6d..81127b924 100644 --- a/store/watcher_hub.go +++ b/store/watcher_hub.go @@ -77,7 +77,7 @@ func (wh *watcherHub) watch(prefix string, recursive bool, index uint64) (<-chan func (wh *watcherHub) notify(e *Event) { e = wh.EventHistory.addEvent(e) // add event into the eventHistory - segments := strings.Split(e.Key, "/") + segments := strings.Split(e.Node.Key, "/") currPath := "/" @@ -111,7 +111,7 @@ func (wh *watcherHub) notifyWatchers(e *Event, path string, deleted bool) { w, _ := curr.Value.(*watcher) - if w.notify(e, e.Key == path, deleted) { + if w.notify(e, e.Node.Key == path, deleted) { // if we successfully notify a watcher // we need to remove the watcher from the list From 67b4c27d5dae314b99558eb823435490bc1a65d2 Mon Sep 17 00:00:00 2001 From: Xiang Li Date: Thu, 28 Nov 2013 21:34:38 -0500 Subject: [PATCH 04/18] refactor change node_repr to node_extern --- server/v2/tests/delete_handler_test.go | 2 +- server/v2/tests/put_handler_test.go | 2 +- store/event.go | 6 ++--- store/event_test.go | 12 ++++----- store/node.go | 10 ++++--- store/node_extern.go | 36 ++++++++++++++++++++++++++ store/node_repr.go | 35 ------------------------- store/store.go | 4 +-- store/watcher_test.go | 6 ++--- 9 files changed, 58 insertions(+), 55 deletions(-) create mode 100644 store/node_extern.go delete mode 100644 store/node_repr.go diff --git a/server/v2/tests/delete_handler_test.go b/server/v2/tests/delete_handler_test.go index f2f6decbc..4907e102d 100644 --- a/server/v2/tests/delete_handler_test.go +++ b/server/v2/tests/delete_handler_test.go @@ -24,6 +24,6 @@ func TestV2DeleteKey(t *testing.T) { resp, err = tests.DeleteForm(fmt.Sprintf("http://%s%s", s.URL(), "/v2/keys/foo/bar"), url.Values{}) body := tests.ReadBody(resp) assert.Nil(t, err, "") - assert.Equal(t, string(body), `{"action":"delete","node":{"key":"/foo/bar","modifiedIndex":2}}`, "") + assert.Equal(t, string(body), `{"action":"delete","node":{"key":"/foo/bar","modifiedIndex":2,"createdIndex":1}}`, "") }) } diff --git a/server/v2/tests/put_handler_test.go b/server/v2/tests/put_handler_test.go index dd98b065f..3a89790dd 100644 --- a/server/v2/tests/put_handler_test.go +++ b/server/v2/tests/put_handler_test.go @@ -22,7 +22,7 @@ func TestV2SetKey(t *testing.T) { resp, err := tests.PutForm(fmt.Sprintf("http://%s%s", s.URL(), "/v2/keys/foo/bar"), v) body := tests.ReadBody(resp) assert.Nil(t, err, "") - assert.Equal(t, string(body), `{"action":"set","node":{"key":"/foo/bar","value":"XXX","modifiedIndex":1}}`, "") + assert.Equal(t, string(body), `{"action":"set","node":{"key":"/foo/bar","value":"XXX","modifiedIndex":1,"createdIndex":1}}`, "") }) } diff --git a/store/event.go b/store/event.go index 9a5a835c0..6a379985b 100644 --- a/store/event.go +++ b/store/event.go @@ -11,12 +11,12 @@ const ( ) type Event struct { - Action string `json:"action"` - Node *Node `json:"node,omitempty"` + Action string `json:"action"` + Node *NodeExtern `json:"node,omitempty"` } func newEvent(action string, key string, modifiedIndex, createdIndex uint64) *Event { - n := &Node{ + n := &NodeExtern{ Key: key, ModifiedIndex: modifiedIndex, CreatedIndex: createdIndex, diff --git a/store/event_test.go b/store/event_test.go index dc30ce44d..39dc7fee1 100644 --- a/store/event_test.go +++ b/store/event_test.go @@ -13,7 +13,7 @@ func TestEventQueue(t *testing.T) { // Add for i := 0; i < 200; i++ { - e := newEvent(Create, "/foo", uint64(i)) + e := newEvent(Create, "/foo", uint64(i), uint64(i)) eh.addEvent(e) } @@ -35,11 +35,11 @@ func TestScanHistory(t *testing.T) { eh := newEventHistory(100) // Add - eh.addEvent(newEvent(Create, "/foo", 1)) - eh.addEvent(newEvent(Create, "/foo/bar", 2)) - eh.addEvent(newEvent(Create, "/foo/foo", 3)) - eh.addEvent(newEvent(Create, "/foo/bar/bar", 4)) - eh.addEvent(newEvent(Create, "/foo/foo/foo", 5)) + eh.addEvent(newEvent(Create, "/foo", 1, 1)) + eh.addEvent(newEvent(Create, "/foo/bar", 2, 2)) + eh.addEvent(newEvent(Create, "/foo/foo", 3, 3)) + eh.addEvent(newEvent(Create, "/foo/bar/bar", 4, 4)) + eh.addEvent(newEvent(Create, "/foo/foo/foo", 5, 5)) e, err := eh.scan("/foo", 1) if err != nil || e.Index() != 1 { diff --git a/store/node.go b/store/node.go index 809b4cadb..a165add3b 100644 --- a/store/node.go +++ b/store/node.go @@ -223,12 +223,13 @@ func (n *node) Remove(recursive bool, callback func(path string)) *etcdErr.Error return nil } -func (n *node) Repr(recurisive, sorted bool) Node { +func (n *node) Repr(recurisive, sorted bool) NodeExtern { if n.IsDir() { - node := Node{ + node := NodeExtern{ Key: n.Path, Dir: true, ModifiedIndex: n.ModifiedIndex, + CreatedIndex: n.CreatedIndex, } node.Expiration, node.TTL = n.ExpirationAndTTL() @@ -237,7 +238,7 @@ func (n *node) Repr(recurisive, sorted bool) Node { } children, _ := n.List() - node.Nodes = make(Nodes, len(children)) + node.Nodes = make(NodeExterns, len(children)) // we do not use the index in the children slice directly // we need to skip the hidden one @@ -263,10 +264,11 @@ func (n *node) Repr(recurisive, sorted bool) Node { return node } - node := Node{ + node := NodeExtern{ Key: n.Path, Value: n.Value, ModifiedIndex: n.ModifiedIndex, + CreatedIndex: n.CreatedIndex, } node.Expiration, node.TTL = n.ExpirationAndTTL() return node diff --git a/store/node_extern.go b/store/node_extern.go new file mode 100644 index 000000000..319378f74 --- /dev/null +++ b/store/node_extern.go @@ -0,0 +1,36 @@ +package store + +import ( + "time" +) + +// NodeExtern is the external representation of the +// internal node with additional fields +// PrevValue is the previous value of the node +// TTL is time to live in second +type NodeExtern struct { + Key string `json:"key, omitempty"` + PrevValue string `json:"prevValue,omitempty"` + Value string `json:"value,omitempty"` + Dir bool `json:"dir,omitempty"` + Expiration *time.Time `json:"expiration,omitempty"` + TTL int64 `json:"ttl,omitempty"` + Nodes NodeExterns `json:"nodes,omitempty"` + ModifiedIndex uint64 `json:"modifiedIndex,omitempty"` + CreatedIndex uint64 `json:"createdIndex,omitempty"` +} + +type NodeExterns []NodeExtern + +// interfaces for sorting +func (ns NodeExterns) Len() int { + return len(ns) +} + +func (ns NodeExterns) Less(i, j int) bool { + return ns[i].Key < ns[j].Key +} + +func (ns NodeExterns) Swap(i, j int) { + ns[i], ns[j] = ns[j], ns[i] +} diff --git a/store/node_repr.go b/store/node_repr.go deleted file mode 100644 index 2dabf95be..000000000 --- a/store/node_repr.go +++ /dev/null @@ -1,35 +0,0 @@ -package store - -import ( - "time" -) - -// Node is the representation of the internal node with additional fields -// PrevValue is the previous value of the node -// TTL is time to live in second -type Node struct { - Key string `json:"key, omitempty"` - PrevValue string `json:"prevValue,omitempty"` - Value string `json:"value,omitempty"` - Dir bool `json:"dir,omitempty"` - Expiration *time.Time `json:"expiration,omitempty"` - TTL int64 `json:"ttl,omitempty"` - Nodes Nodes `json:"nodes,omitempty"` - ModifiedIndex uint64 `json:"modifiedIndex,omitempty"` - CreatedIndex uint64 `json:"createdIndex,omitempty"` -} - -type Nodes []Node - -// interfaces for sorting -func (ns Nodes) Len() int { - return len(ns) -} - -func (ns Nodes) Less(i, j int) bool { - return ns[i].Key < ns[j].Key -} - -func (ns Nodes) Swap(i, j int) { - ns[i], ns[j] = ns[j], ns[i] -} diff --git a/store/store.go b/store/store.go index 95db10037..8f17d5a3c 100644 --- a/store/store.go +++ b/store/store.go @@ -120,7 +120,7 @@ func (s *store) Get(nodePath string, recursive, sorted bool) (*Event, error) { eNode.Dir = true children, _ := n.List() - eNode.Nodes = make(Nodes, len(children)) + eNode.Nodes = make(NodeExterns, len(children)) // we do not use the index in the children slice directly // we need to skip the hidden one @@ -260,7 +260,7 @@ func (s *store) Delete(nodePath string, recursive bool) (*Event, error) { if n.IsDir() { eNode.Dir = true } else { - eNode.PrevValue = eNode.Value + eNode.PrevValue = n.Value } callback := func(path string) { // notify function diff --git a/store/watcher_test.go b/store/watcher_test.go index 386fec440..ae36eda28 100644 --- a/store/watcher_test.go +++ b/store/watcher_test.go @@ -35,7 +35,7 @@ func TestWatcher(t *testing.T) { // do nothing } - e := newEvent(Create, "/foo/bar", 1) + e := newEvent(Create, "/foo/bar", 1, 1) wh.notify(e) @@ -47,7 +47,7 @@ func TestWatcher(t *testing.T) { c, _ = wh.watch("/foo", false, 2) - e = newEvent(Create, "/foo/bar", 2) + e = newEvent(Create, "/foo/bar", 2, 2) wh.notify(e) @@ -58,7 +58,7 @@ func TestWatcher(t *testing.T) { // do nothing } - e = newEvent(Create, "/foo", 3) + e = newEvent(Create, "/foo", 3, 3) wh.notify(e) From 2a5030b3ca02d650ea7b6899c1056445678b0782 Mon Sep 17 00:00:00 2001 From: Brandon Philips Date: Tue, 26 Nov 2013 10:29:03 -0600 Subject: [PATCH 05/18] fix(README): fix 9 instances of v1 remove references to the v1 API and consistently use the v2 API. --- README.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index b18e42b66..9d38186bd 100644 --- a/README.md +++ b/README.md @@ -121,7 +121,7 @@ curl -L http://127.0.0.1:4001/v2/keys/message You can change the value of `/message` from `Hello world` to `Hello etcd` with another `PUT` request to the key: ```sh -curl -L http://127.0.0.1:4001/v1/keys/message -XPUT -d value="Hello etcd" +curl -L http://127.0.0.1:4001/v2/keys/message -XPUT -d value="Hello etcd" ``` ```json @@ -235,14 +235,14 @@ Here is a simple example. Let's create a key-value pair first: `foo=one`. ```sh -curl -L http://127.0.0.1:4001/v1/keys/foo -XPUT -d value=one +curl -L http://127.0.0.1:4001/v2/keys/foo -XPUT -d value=one ``` Let's try an invalid `CompareAndSwap` command first. We can provide the `prevValue` parameter to the set command to make it a `CompareAndSwap` command. ```sh -curl -L http://127.0.0.1:4001/v1/keys/foo?prevValue=two -XPUT -d value=three +curl -L http://127.0.0.1:4001/v2/keys/foo?prevValue=two -XPUT -d value=three ``` This will try to compare the previous value of the key and the previous value we provided. If they are equal, the value of the key will change to three. @@ -435,7 +435,7 @@ routines:SSL3_READ_BYTES:sslv3 alert bad certificate We need to give the CA signed cert to the server. ```sh -curl --key ./fixtures/ca/server2.key.insecure --cert ./fixtures/ca/server2.crt --cacert ./fixtures/ca/server-chain.pem -L https://127.0.0.1:4001/v1/keys/foo -XPUT -d value=bar -v +curl --key ./fixtures/ca/server2.key.insecure --cert ./fixtures/ca/server2.crt --cacert ./fixtures/ca/server-chain.pem -L https://127.0.0.1:4001/v2/keys/foo -XPUT -d value=bar -v ``` You should able to see: @@ -482,7 +482,7 @@ Let's join two more machines to this cluster using the `-peers` argument: We can retrieve a list of machines in the cluster using the HTTP API: ```sh -curl -L http://127.0.0.1:4001/v1/machines +curl -L http://127.0.0.1:4001/v2/machines ``` We should see there are three machines in the cluster @@ -494,7 +494,7 @@ http://127.0.0.1:4001, http://127.0.0.1:4002, http://127.0.0.1:4003 The machine list is also available via the main key API: ```sh -curl -L http://127.0.0.1:4001/v1/keys/_etcd/machines +curl -L http://127.0.0.1:4001/v2/keys/_etcd/machines ``` ```json @@ -529,13 +529,13 @@ curl -L http://127.0.0.1:4001/v2/keys/foo -XPUT -d value=bar Now if we kill the leader of the cluster, we can get the value from one of the other two machines: ```sh -curl -L http://127.0.0.1:4002/v1/keys/foo +curl -L http://127.0.0.1:4002/v2/keys/foo ``` We can also see that a new leader has been elected: ``` -curl -L http://127.0.0.1:4002/v1/leader +curl -L http://127.0.0.1:4002/v2/leader ``` ``` @@ -557,7 +557,7 @@ Type `CTRL-C` on each terminal and then rerun the same command you used to start Your request for the `foo` key will return the correct value: ```sh -curl -L http://127.0.0.1:4002/v1/keys/foo +curl -L http://127.0.0.1:4002/v2/keys/foo ``` ```json From 5097a2adee0ca27d3af6a2225e62b1d2aba93626 Mon Sep 17 00:00:00 2001 From: Xiang Li Date: Sun, 1 Dec 2013 00:47:23 -0500 Subject: [PATCH 06/18] fix(event_history.go) should not scan prefix --- store/event_history.go | 13 ++++++++++--- store/event_test.go | 8 ++++---- store/store.go | 9 ++++----- store/watcher_hub.go | 12 ++++++------ 4 files changed, 24 insertions(+), 18 deletions(-) diff --git a/store/event_history.go b/store/event_history.go index 4fd077184..d61d54d60 100644 --- a/store/event_history.go +++ b/store/event_history.go @@ -2,6 +2,7 @@ package store import ( "fmt" + "path" "strings" "sync" @@ -39,8 +40,8 @@ func (eh *EventHistory) addEvent(e *Event) *Event { } // scan function is enumerating events from the index in history and -// stops till the first point where the key has identified prefix -func (eh *EventHistory) scan(prefix string, index uint64) (*Event, *etcdErr.Error) { +// stops till the first point where the key has identified key +func (eh *EventHistory) scan(key string, recursive bool, index uint64) (*Event, *etcdErr.Error) { eh.rwl.RLock() defer eh.rwl.RUnlock() @@ -62,7 +63,13 @@ func (eh *EventHistory) scan(prefix string, index uint64) (*Event, *etcdErr.Erro for { e := eh.Queue.Events[i] - if strings.HasPrefix(e.Key, prefix) && index <= e.Index() { // make sure we bypass the smaller one + ok := (e.Key == key) + + if recursive { + ok = ok || strings.HasPrefix(e.Key, path.Join(key, "/")) + } + + if ok && index <= e.Index() { // make sure we bypass the smaller one return e, nil } diff --git a/store/event_test.go b/store/event_test.go index dc30ce44d..1d82ebf74 100644 --- a/store/event_test.go +++ b/store/event_test.go @@ -41,24 +41,24 @@ func TestScanHistory(t *testing.T) { eh.addEvent(newEvent(Create, "/foo/bar/bar", 4)) eh.addEvent(newEvent(Create, "/foo/foo/foo", 5)) - e, err := eh.scan("/foo", 1) + e, err := eh.scan("/foo", false, 1) if err != nil || e.Index() != 1 { t.Fatalf("scan error [/foo] [1] %v", e.Index) } - e, err = eh.scan("/foo/bar", 1) + e, err = eh.scan("/foo/bar", false, 1) if err != nil || e.Index() != 2 { t.Fatalf("scan error [/foo/bar] [2] %v", e.Index) } - e, err = eh.scan("/foo/bar", 3) + e, err = eh.scan("/foo/bar", true, 3) if err != nil || e.Index() != 4 { t.Fatalf("scan error [/foo/bar/bar] [4] %v", e.Index) } - e, err = eh.scan("/foo/bar", 6) + e, err = eh.scan("/foo/bar", true, 6) if e != nil { t.Fatalf("bad index shoud reuturn nil") diff --git a/store/store.go b/store/store.go index 80f9622af..1edb02f29 100644 --- a/store/store.go +++ b/store/store.go @@ -280,8 +280,8 @@ func (s *store) Delete(nodePath string, recursive bool) (*Event, error) { return e, nil } -func (s *store) Watch(prefix string, recursive bool, sinceIndex uint64) (<-chan *Event, error) { - prefix = path.Clean(path.Join("/", prefix)) +func (s *store) Watch(key string, recursive bool, sinceIndex uint64) (<-chan *Event, error) { + key = path.Clean(path.Join("/", key)) nextIndex := s.CurrentIndex + 1 @@ -292,10 +292,10 @@ func (s *store) Watch(prefix string, recursive bool, sinceIndex uint64) (<-chan var err *etcdErr.Error if sinceIndex == 0 { - c, err = s.WatcherHub.watch(prefix, recursive, nextIndex) + c, err = s.WatcherHub.watch(key, recursive, nextIndex) } else { - c, err = s.WatcherHub.watch(prefix, recursive, sinceIndex) + c, err = s.WatcherHub.watch(key, recursive, sinceIndex) } if err != nil { @@ -396,7 +396,6 @@ func (s *store) internalCreate(nodePath string, value string, unique bool, repla expireTime = Permanent } - dir, newNodeName := path.Split(nodePath) // walk through the nodePath, create dirs and get the last directory node diff --git a/store/watcher_hub.go b/store/watcher_hub.go index b952aec6d..19eef7a57 100644 --- a/store/watcher_hub.go +++ b/store/watcher_hub.go @@ -33,11 +33,11 @@ func newWatchHub(capacity int) *watcherHub { } // watch function returns an Event channel. -// If recursive is true, the first change after index under prefix will be sent to the event channel. -// If recursive is false, the first change after index at prefix will be sent to the event channel. +// If recursive is true, the first change after index under key will be sent to the event channel. +// If recursive is false, the first change after index at key will be sent to the event channel. // If index is zero, watch will start from the current index + 1. -func (wh *watcherHub) watch(prefix string, recursive bool, index uint64) (<-chan *Event, *etcdErr.Error) { - event, err := wh.EventHistory.scan(prefix, index) +func (wh *watcherHub) watch(key string, recursive bool, index uint64) (<-chan *Event, *etcdErr.Error) { + event, err := wh.EventHistory.scan(key, recursive, index) if err != nil { return nil, err @@ -57,7 +57,7 @@ func (wh *watcherHub) watch(prefix string, recursive bool, index uint64) (<-chan sinceIndex: index, } - l, ok := wh.watchers[prefix] + l, ok := wh.watchers[key] if ok { // add the new watcher to the back of the list l.PushBack(w) @@ -65,7 +65,7 @@ func (wh *watcherHub) watch(prefix string, recursive bool, index uint64) (<-chan } else { // create a new list and add the new watcher l := list.New() l.PushBack(w) - wh.watchers[prefix] = l + wh.watchers[key] = l } atomic.AddInt64(&wh.count, 1) From 08dffe85d6227d055c6226e8228c63db445d463f Mon Sep 17 00:00:00 2001 From: Baruch Even Date: Sun, 1 Dec 2013 08:35:56 +0200 Subject: [PATCH 07/18] Fix spelling mistakes in internal-protocol-versioning.md --- Documentation/internal-protocol-versioning.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Documentation/internal-protocol-versioning.md b/Documentation/internal-protocol-versioning.md index 4602f91b6..ec3874d25 100644 --- a/Documentation/internal-protocol-versioning.md +++ b/Documentation/internal-protocol-versioning.md @@ -4,9 +4,9 @@ Goal: We want to be able to upgrade an individual peer in an etcd cluster to a n The process will take the form of individual followers upgrading to the latest version until the entire cluster is on the new version. Immediate need: etcd is moving too fast to version the internal API right now. -But, we need to keep mixed version clusters from being started by a rollowing upgrade process (e.g. the CoreOS developer alpha). +But, we need to keep mixed version clusters from being started by a rolling upgrade process (e.g. the CoreOS developer alpha). -Longer term need: Having a mixed version cluster where all peers are not be running the exact same version of etcd itself but are able to speak one version of the internal protocol. +Longer term need: Having a mixed version cluster where all peers are not running the exact same version of etcd itself but are able to speak one version of the internal protocol. Solution: The internal protocol needs to be versioned just as the client protocol is. Initially during the 0.\*.\* series of etcd releases we won't allow mixed versions at all. From cc9c75acbee32b042ebf8af0e1ef455eae35832c Mon Sep 17 00:00:00 2001 From: Brandon Philips Date: Sat, 30 Nov 2013 22:46:12 -0800 Subject: [PATCH 08/18] fix(README): repair some noisy spaces --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 9d38186bd..0cc358d61 100644 --- a/README.md +++ b/README.md @@ -219,9 +219,9 @@ The watch command returns immediately with the same response as previous. ### Atomic Compare-and-Swap (CAS) -Etcd can be used as a centralized coordination service in a cluster and `CompareAndSwap` is the most basic operation to build distributed lock service. +Etcd can be used as a centralized coordination service in a cluster and `CompareAndSwap` is the most basic operation to build distributed lock service. -This command will set the value of a key only if the client-provided conditions are equal to the current conditions. +This command will set the value of a key only if the client-provided conditions are equal to the current conditions. The current comparable conditions are: From 1d02a70802637b4940938314e0058763ea6b18d8 Mon Sep 17 00:00:00 2001 From: Brandon Philips Date: Sun, 1 Dec 2013 13:28:08 -0800 Subject: [PATCH 09/18] feat(README): add a prevExist example On the mailing list Dustin Oprea suggested adding a prevExist concrete example would make it more clear that the value must be true or false. --- README.md | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 0cc358d61..3db3f8642 100644 --- a/README.md +++ b/README.md @@ -238,8 +238,20 @@ Let's create a key-value pair first: `foo=one`. curl -L http://127.0.0.1:4001/v2/keys/foo -XPUT -d value=one ``` -Let's try an invalid `CompareAndSwap` command first. -We can provide the `prevValue` parameter to the set command to make it a `CompareAndSwap` command. +Let's try some invalid `CompareAndSwap` commands first. + +Trying to set this existing key with `prevExist=false` fails as expected: +```sh +curl -L http://127.0.0.1:4001/v2/keys/foo?prevExist=false -XPUT -d value=three +``` + +The error code explains the problem: + +```json +{"errorCode":105,"message":"Already exists","cause":"/foo","index":39776} +``` + +Now lets provide a `prevValue` parameter: ```sh curl -L http://127.0.0.1:4001/v2/keys/foo?prevValue=two -XPUT -d value=three From fc562bd62537d6ad0e22841e18413dbc860166cf Mon Sep 17 00:00:00 2001 From: Xiang Li Date: Sun, 1 Dec 2013 17:24:30 -0500 Subject: [PATCH 10/18] fix tests bump deps --- server/v2/tests/delete_handler_test.go | 2 +- .../multi_node_kill_all_and_recovery_test.go | 4 +- tests/functional/remove_node_test.go | 20 +- tests/functional/simple_multi_node_test.go | 10 +- tests/functional/simple_snapshot_test.go | 10 +- tests/functional/single_node_recovery_test.go | 13 +- tests/functional/single_node_test.go | 15 +- tests/functional/util.go | 2 +- tests/functional/v1_migration_test.go | 2 +- .../coreos/go-etcd/etcd/add_child.go | 16 +- .../coreos/go-etcd/etcd/add_child_test.go | 16 +- .../github.com/coreos/go-etcd/etcd/client.go | 45 ++- .../coreos/go-etcd/etcd/compare_and_swap.go | 9 +- .../go-etcd/etcd/compare_and_swap_test.go | 12 +- .../github.com/coreos/go-etcd/etcd/debug.go | 5 +- .../github.com/coreos/go-etcd/etcd/delete.go | 31 +- .../coreos/go-etcd/etcd/delete_test.go | 26 +- .../github.com/coreos/go-etcd/etcd/get.go | 38 ++- .../coreos/go-etcd/etcd/get_test.go | 71 +++-- .../github.com/coreos/go-etcd/etcd/options.go | 68 ++++ .../coreos/go-etcd/etcd/requests.go | 291 ++++++++---------- .../coreos/go-etcd/etcd/response.go | 83 +++-- .../coreos/go-etcd/etcd/set_curl_chan_test.go | 17 +- .../coreos/go-etcd/etcd/set_update_create.go | 96 +++++- .../go-etcd/etcd/set_update_create_test.go | 55 ++-- .../github.com/coreos/go-etcd/etcd/watch.go | 99 ++++-- .../coreos/go-etcd/etcd/watch_test.go | 24 +- .../github.com/gorilla/context/README.md | 1 + .../github.com/gorilla/context/context.go | 2 +- third_party/github.com/gorilla/mux/README.md | 1 + third_party/github.com/gorilla/mux/mux.go | 8 + .../github.com/gorilla/mux/mux_test.go | 53 +++- .../github.com/gorilla/mux/old_test.go | 8 +- 33 files changed, 722 insertions(+), 431 deletions(-) create mode 100644 third_party/github.com/coreos/go-etcd/etcd/options.go diff --git a/server/v2/tests/delete_handler_test.go b/server/v2/tests/delete_handler_test.go index 4907e102d..c18b402f5 100644 --- a/server/v2/tests/delete_handler_test.go +++ b/server/v2/tests/delete_handler_test.go @@ -24,6 +24,6 @@ func TestV2DeleteKey(t *testing.T) { resp, err = tests.DeleteForm(fmt.Sprintf("http://%s%s", s.URL(), "/v2/keys/foo/bar"), url.Values{}) body := tests.ReadBody(resp) assert.Nil(t, err, "") - assert.Equal(t, string(body), `{"action":"delete","node":{"key":"/foo/bar","modifiedIndex":2,"createdIndex":1}}`, "") + assert.Equal(t, string(body), `{"action":"delete","node":{"key":"/foo/bar","prevValue":"XXX","modifiedIndex":2,"createdIndex":1}}`, "") }) } diff --git a/tests/functional/multi_node_kill_all_and_recovery_test.go b/tests/functional/multi_node_kill_all_and_recovery_test.go index 75fe22974..c7e710636 100644 --- a/tests/functional/multi_node_kill_all_and_recovery_test.go +++ b/tests/functional/multi_node_kill_all_and_recovery_test.go @@ -65,7 +65,7 @@ func TestMultiNodeKillAllAndRecovery(t *testing.T) { t.Fatalf("Recovery error: %s", err) } - if result.ModifiedIndex != 16 { - t.Fatalf("recovery failed! [%d/16]", result.ModifiedIndex) + if result.Node.ModifiedIndex != 16 { + t.Fatalf("recovery failed! [%d/16]", result.Node.ModifiedIndex) } } diff --git a/tests/functional/remove_node_test.go b/tests/functional/remove_node_test.go index dbd9ebd71..9793b22de 100644 --- a/tests/functional/remove_node_test.go +++ b/tests/functional/remove_node_test.go @@ -35,13 +35,13 @@ func TestRemoveNode(t *testing.T) { fmt.Println("send remove to node3 and wait for its exiting") etcds[2].Wait() - resp, err := c.Get("_etcd/machines", false) + resp, err := c.Get("_etcd/machines", false, false) if err != nil { panic(err) } - if len(resp.Kvs) != 2 { + if len(resp.Node.Nodes) != 2 { t.Fatal("cannot remove peer") } @@ -59,14 +59,14 @@ func TestRemoveNode(t *testing.T) { time.Sleep(time.Second) - resp, err = c.Get("_etcd/machines", false) + resp, err = c.Get("_etcd/machines", false, false) if err != nil { panic(err) } - if len(resp.Kvs) != 3 { - t.Fatalf("add peer fails #1 (%d != 3)", len(resp.Kvs)) + if len(resp.Node.Nodes) != 3 { + t.Fatalf("add peer fails #1 (%d != 3)", len(resp.Node.Nodes)) } } @@ -78,13 +78,13 @@ func TestRemoveNode(t *testing.T) { client.Do(rmReq) - resp, err := c.Get("_etcd/machines", false) + resp, err := c.Get("_etcd/machines", false, false) if err != nil { panic(err) } - if len(resp.Kvs) != 2 { + if len(resp.Node.Nodes) != 2 { t.Fatal("cannot remove peer") } @@ -102,14 +102,14 @@ func TestRemoveNode(t *testing.T) { time.Sleep(time.Second) - resp, err = c.Get("_etcd/machines", false) + resp, err = c.Get("_etcd/machines", false, false) if err != nil { panic(err) } - if len(resp.Kvs) != 3 { - t.Fatalf("add peer fails #2 (%d != 3)", len(resp.Kvs)) + if len(resp.Node.Nodes) != 3 { + t.Fatalf("add peer fails #2 (%d != 3)", len(resp.Node.Nodes)) } } } diff --git a/tests/functional/simple_multi_node_test.go b/tests/functional/simple_multi_node_test.go index 7afe0a35f..0506b95aa 100644 --- a/tests/functional/simple_multi_node_test.go +++ b/tests/functional/simple_multi_node_test.go @@ -39,24 +39,26 @@ func templateTestSimpleMultiNode(t *testing.T, tls bool) { // Test Set result, err := c.Set("foo", "bar", 100) + node := result.Node - if err != nil || result.Key != "/foo" || result.Value != "bar" || result.TTL < 95 { + if err != nil || node.Key != "/foo" || node.Value != "bar" || node.TTL < 95 { if err != nil { t.Fatal(err) } - t.Fatalf("Set 1 failed with %s %s %v", result.Key, result.Value, result.TTL) + t.Fatalf("Set 1 failed with %s %s %v", node.Key, node.Value, node.TTL) } time.Sleep(time.Second) result, err = c.Set("foo", "bar", 100) + node = result.Node - if err != nil || result.Key != "/foo" || result.Value != "bar" || result.PrevValue != "bar" || result.TTL < 95 { + if err != nil || node.Key != "/foo" || node.Value != "bar" || node.PrevValue != "bar" || node.TTL < 95 { if err != nil { t.Fatal(err) } - t.Fatalf("Set 2 failed with %s %s %v", result.Key, result.Value, result.TTL) + t.Fatalf("Set 2 failed with %s %s %v", node.Key, node.Value, node.TTL) } } diff --git a/tests/functional/simple_snapshot_test.go b/tests/functional/simple_snapshot_test.go index 2ca14cdfe..29872d648 100644 --- a/tests/functional/simple_snapshot_test.go +++ b/tests/functional/simple_snapshot_test.go @@ -30,13 +30,14 @@ func TestSimpleSnapshot(t *testing.T) { // issue first 501 commands for i := 0; i < 501; i++ { result, err := c.Set("foo", "bar", 100) + node := result.Node - if err != nil || result.Key != "/foo" || result.Value != "bar" || result.TTL < 95 { + if err != nil || node.Key != "/foo" || node.Value != "bar" || node.TTL < 95 { if err != nil { t.Fatal(err) } - t.Fatalf("Set failed with %s %s %v", result.Key, result.Value, result.TTL) + t.Fatalf("Set failed with %s %s %v", node.Key, node.Value, node.TTL) } } @@ -62,13 +63,14 @@ func TestSimpleSnapshot(t *testing.T) { // issue second 501 commands for i := 0; i < 501; i++ { result, err := c.Set("foo", "bar", 100) + node := result.Node - if err != nil || result.Key != "/foo" || result.Value != "bar" || result.TTL < 95 { + if err != nil || node.Key != "/foo" || node.Value != "bar" || node.TTL < 95 { if err != nil { t.Fatal(err) } - t.Fatalf("Set failed with %s %s %v", result.Key, result.Value, result.TTL) + t.Fatalf("Set failed with %s %s %v", node.Key, node.Value, node.TTL) } } diff --git a/tests/functional/single_node_recovery_test.go b/tests/functional/single_node_recovery_test.go index dde9e16ca..632b767de 100644 --- a/tests/functional/single_node_recovery_test.go +++ b/tests/functional/single_node_recovery_test.go @@ -28,13 +28,14 @@ func TestSingleNodeRecovery(t *testing.T) { c.SyncCluster() // Test Set result, err := c.Set("foo", "bar", 100) + node := result.Node - if err != nil || result.Key != "/foo" || result.Value != "bar" || result.TTL < 95 { + if err != nil || node.Key != "/foo" || node.Value != "bar" || node.TTL < 95 { if err != nil { t.Fatal(err) } - t.Fatalf("Set 1 failed with %s %s %v", result.Key, result.Value, result.TTL) + t.Fatalf("Set 1 failed with %s %s %v", node.Key, node.Value, node.TTL) } time.Sleep(time.Second) @@ -50,16 +51,18 @@ func TestSingleNodeRecovery(t *testing.T) { time.Sleep(time.Second) - result, err = c.Get("foo", false) + result, err = c.Get("foo", false, false) + node = result.Node + if err != nil { t.Fatal("get fail: " + err.Error()) return } - if err != nil || result.Key != "/foo" || result.Value != "bar" || result.TTL > 99 { + if err != nil || node.Key != "/foo" || node.Value != "bar" || node.TTL > 99 { if err != nil { t.Fatal(err) } - t.Fatalf("Recovery Get failed with %s %s %v", result.Key, result.Value, result.TTL) + t.Fatalf("Recovery Get failed with %s %s %v", node.Key, node.Value, node.TTL) } } diff --git a/tests/functional/single_node_test.go b/tests/functional/single_node_test.go index 87d297eea..9294e2f96 100644 --- a/tests/functional/single_node_test.go +++ b/tests/functional/single_node_test.go @@ -28,36 +28,39 @@ func TestSingleNode(t *testing.T) { c.SyncCluster() // Test Set result, err := c.Set("foo", "bar", 100) + node := result.Node - if err != nil || result.Key != "/foo" || result.Value != "bar" || result.TTL < 95 { + if err != nil || node.Key != "/foo" || node.Value != "bar" || node.TTL < 95 { if err != nil { t.Fatal("Set 1: ", err) } - t.Fatalf("Set 1 failed with %s %s %v", result.Key, result.Value, result.TTL) + t.Fatalf("Set 1 failed with %s %s %v", node.Key, node.Value, node.TTL) } time.Sleep(time.Second) result, err = c.Set("foo", "bar", 100) + node = result.Node - if err != nil || result.Key != "/foo" || result.Value != "bar" || result.PrevValue != "bar" || result.TTL != 100 { + if err != nil || node.Key != "/foo" || node.Value != "bar" || node.PrevValue != "bar" || node.TTL != 100 { if err != nil { t.Fatal("Set 2: ", err) } - t.Fatalf("Set 2 failed with %s %s %v", result.Key, result.Value, result.TTL) + t.Fatalf("Set 2 failed with %s %s %v", node.Key, node.Value, node.TTL) } // Add a test-and-set test // First, we'll test we can change the value if we get it write result, err = c.CompareAndSwap("foo", "foobar", 100, "bar", 0) + node = result.Node - if err != nil || result.Key != "/foo" || result.Value != "foobar" || result.PrevValue != "bar" || result.TTL != 100 { + if err != nil || node.Key != "/foo" || node.Value != "foobar" || node.PrevValue != "bar" || node.TTL != 100 { if err != nil { t.Fatal(err) } - t.Fatalf("Set 3 failed with %s %s %v", result.Key, result.Value, result.TTL) + t.Fatalf("Set 3 failed with %s %s %v", node.Key, node.Value, node.TTL) } // Next, we'll make sure we can't set it without the correct prior value diff --git a/tests/functional/util.go b/tests/functional/util.go index eb0173c76..9ceff0e70 100644 --- a/tests/functional/util.go +++ b/tests/functional/util.go @@ -44,7 +44,7 @@ func Set(stop chan bool) { result, err := c.Set(key, "bar", 0) - if err != nil || result.Key != "/"+key || result.Value != "bar" { + if err != nil || result.Node.Key != "/"+key || result.Node.Value != "bar" { select { case <-stop: stopSet = true diff --git a/tests/functional/v1_migration_test.go b/tests/functional/v1_migration_test.go index 506f48cc7..9f4fbeeea 100644 --- a/tests/functional/v1_migration_test.go +++ b/tests/functional/v1_migration_test.go @@ -99,5 +99,5 @@ func TestV1ClusterMigration(t *testing.T) { body = tests.ReadBody(resp) assert.Nil(t, err, "") assert.Equal(t, resp.StatusCode, 200, "") - assert.Equal(t, string(body), `{"action":"get","key":"/foo","value":"one","modifiedIndex":9}`) + assert.Equal(t, string(body), `{"action":"get","node":{"key":"/foo","value":"one","modifiedIndex":9,"createdIndex":9}}`) } diff --git a/third_party/github.com/coreos/go-etcd/etcd/add_child.go b/third_party/github.com/coreos/go-etcd/etcd/add_child.go index f275599c5..53af6d285 100644 --- a/third_party/github.com/coreos/go-etcd/etcd/add_child.go +++ b/third_party/github.com/coreos/go-etcd/etcd/add_child.go @@ -2,10 +2,22 @@ package etcd // Add a new directory with a random etcd-generated key under the given path. func (c *Client) AddChildDir(key string, ttl uint64) (*Response, error) { - return c.post(key, "", ttl) + raw, err := c.post(key, "", ttl) + + if err != nil { + return nil, err + } + + return raw.toResponse() } // Add a new file with a random etcd-generated key under the given path. func (c *Client) AddChild(key string, value string, ttl uint64) (*Response, error) { - return c.post(key, value, ttl) + raw, err := c.post(key, value, ttl) + + if err != nil { + return nil, err + } + + return raw.toResponse() } diff --git a/third_party/github.com/coreos/go-etcd/etcd/add_child_test.go b/third_party/github.com/coreos/go-etcd/etcd/add_child_test.go index efe155467..b5a5bd971 100644 --- a/third_party/github.com/coreos/go-etcd/etcd/add_child_test.go +++ b/third_party/github.com/coreos/go-etcd/etcd/add_child_test.go @@ -5,8 +5,8 @@ import "testing" func TestAddChild(t *testing.T) { c := NewClient(nil) defer func() { - c.DeleteAll("fooDir") - c.DeleteAll("nonexistentDir") + c.Delete("fooDir", true) + c.Delete("nonexistentDir", true) }() c.SetDir("fooDir", 5) @@ -21,10 +21,10 @@ func TestAddChild(t *testing.T) { t.Fatal(err) } - resp, err := c.Get("fooDir", true) + resp, err := c.Get("fooDir", true, false) // The child with v0 should proceed the child with v1 because it's added // earlier, so it should have a lower key. - if !(len(resp.Kvs) == 2 && (resp.Kvs[0].Value == "v0" && resp.Kvs[1].Value == "v1")) { + if !(len(resp.Node.Nodes) == 2 && (resp.Node.Nodes[0].Value == "v0" && resp.Node.Nodes[1].Value == "v1")) { t.Fatalf("AddChild 1 failed. There should be two chlidren whose values are v0 and v1, respectively."+ " The response was: %#v", resp) } @@ -40,8 +40,8 @@ func TestAddChild(t *testing.T) { func TestAddChildDir(t *testing.T) { c := NewClient(nil) defer func() { - c.DeleteAll("fooDir") - c.DeleteAll("nonexistentDir") + c.Delete("fooDir", true) + c.Delete("nonexistentDir", true) }() c.SetDir("fooDir", 5) @@ -56,10 +56,10 @@ func TestAddChildDir(t *testing.T) { t.Fatal(err) } - resp, err := c.Get("fooDir", true) + resp, err := c.Get("fooDir", true, false) // The child with v0 should proceed the child with v1 because it's added // earlier, so it should have a lower key. - if !(len(resp.Kvs) == 2 && (len(resp.Kvs[0].KVPairs) == 0 && len(resp.Kvs[1].KVPairs) == 0)) { + if !(len(resp.Node.Nodes) == 2 && (len(resp.Node.Nodes[0].Nodes) == 0 && len(resp.Node.Nodes[1].Nodes) == 0)) { t.Fatalf("AddChildDir 1 failed. There should be two chlidren whose values are v0 and v1, respectively."+ " The response was: %#v", resp) } diff --git a/third_party/github.com/coreos/go-etcd/etcd/client.go b/third_party/github.com/coreos/go-etcd/etcd/client.go index 9b9efce0a..e03719e72 100644 --- a/third_party/github.com/coreos/go-etcd/etcd/client.go +++ b/third_party/github.com/coreos/go-etcd/etcd/client.go @@ -11,7 +11,6 @@ import ( "net/url" "os" "path" - "reflect" "strings" "time" ) @@ -30,6 +29,10 @@ const ( WEAK_CONSISTENCY = "WEAK" ) +const ( + defaultBufferSize = 10 +) + type Cluster struct { Leader string `json:"leader"` Machines []string `json:"machines"` @@ -48,14 +51,9 @@ type Client struct { config Config `json:"config"` httpClient *http.Client persistence io.Writer + cURLch chan string } -type options map[string]interface{} - -// An internally-used data structure that represents a mapping -// between valid options and their kinds -type validOptions map[string]reflect.Kind - // NewClient create a basic client that is configured to be used // with the given machine list. func NewClient(machines []string) *Client { @@ -333,9 +331,7 @@ func dialTimeout(network, addr string) (net.Conn, error) { return net.DialTimeout(network, addr, time.Second) } -func (c *Client) updateLeader(httpPath string) { - u, _ := url.Parse(httpPath) - +func (c *Client) updateLeader(u *url.URL) { var leader string if u.Scheme == "" { leader = "http://" + u.Host @@ -347,3 +343,32 @@ func (c *Client) updateLeader(httpPath string) { c.cluster.Leader = leader c.saveConfig() } + +// switchLeader switch the current leader to machines[num] +func (c *Client) switchLeader(num int) { + logger.Debugf("switch.leader[from %v to %v]", + c.cluster.Leader, c.cluster.Machines[num]) + + c.cluster.Leader = c.cluster.Machines[num] +} + +func (c *Client) OpenCURL() { + c.cURLch = make(chan string, defaultBufferSize) +} + +func (c *Client) CloseCURL() { + c.cURLch = nil +} + +func (c *Client) sendCURL(command string) { + go func() { + select { + case c.cURLch <- command: + default: + } + }() +} + +func (c *Client) RecvCURL() string { + return <-c.cURLch +} diff --git a/third_party/github.com/coreos/go-etcd/etcd/compare_and_swap.go b/third_party/github.com/coreos/go-etcd/etcd/compare_and_swap.go index 565a03ef1..4099d1348 100644 --- a/third_party/github.com/coreos/go-etcd/etcd/compare_and_swap.go +++ b/third_party/github.com/coreos/go-etcd/etcd/compare_and_swap.go @@ -14,5 +14,12 @@ func (c *Client) CompareAndSwap(key string, value string, ttl uint64, prevValue if prevIndex != 0 { options["prevIndex"] = prevIndex } - return c.put(key, value, ttl, options) + + raw, err := c.put(key, value, ttl, options) + + if err != nil { + return nil, err + } + + return raw.toResponse() } diff --git a/third_party/github.com/coreos/go-etcd/etcd/compare_and_swap_test.go b/third_party/github.com/coreos/go-etcd/etcd/compare_and_swap_test.go index bc452a910..56f8e4a8e 100644 --- a/third_party/github.com/coreos/go-etcd/etcd/compare_and_swap_test.go +++ b/third_party/github.com/coreos/go-etcd/etcd/compare_and_swap_test.go @@ -7,7 +7,7 @@ import ( func TestCompareAndSwap(t *testing.T) { c := NewClient(nil) defer func() { - c.DeleteAll("foo") + c.Delete("foo", true) }() c.Set("foo", "bar", 5) @@ -17,8 +17,8 @@ func TestCompareAndSwap(t *testing.T) { if err != nil { t.Fatal(err) } - if !(resp.Value == "bar2" && resp.PrevValue == "bar" && - resp.Key == "/foo" && resp.TTL == 5) { + if !(resp.Node.Value == "bar2" && resp.Node.PrevValue == "bar" && + resp.Node.Key == "/foo" && resp.Node.TTL == 5) { t.Fatalf("CompareAndSwap 1 failed: %#v", resp) } @@ -34,12 +34,12 @@ func TestCompareAndSwap(t *testing.T) { } // This should succeed - resp, err = c.CompareAndSwap("foo", "bar2", 5, "", resp.ModifiedIndex) + resp, err = c.CompareAndSwap("foo", "bar2", 5, "", resp.Node.ModifiedIndex) if err != nil { t.Fatal(err) } - if !(resp.Value == "bar2" && resp.PrevValue == "bar" && - resp.Key == "/foo" && resp.TTL == 5) { + if !(resp.Node.Value == "bar2" && resp.Node.PrevValue == "bar" && + resp.Node.Key == "/foo" && resp.Node.TTL == 5) { t.Fatalf("CompareAndSwap 1 failed: %#v", resp) } diff --git a/third_party/github.com/coreos/go-etcd/etcd/debug.go b/third_party/github.com/coreos/go-etcd/etcd/debug.go index bd6739881..44e59ef9e 100644 --- a/third_party/github.com/coreos/go-etcd/etcd/debug.go +++ b/third_party/github.com/coreos/go-etcd/etcd/debug.go @@ -1,16 +1,15 @@ package etcd import ( - "github.com/coreos/go-log/log" "os" + + "github.com/coreos/go-log/log" ) var logger *log.Logger func init() { setLogger(log.PriErr) - // Uncomment the following line if you want to see lots of logs - // OpenDebug() } func OpenDebug() { diff --git a/third_party/github.com/coreos/go-etcd/etcd/delete.go b/third_party/github.com/coreos/go-etcd/etcd/delete.go index 00348f6ba..869b88e34 100644 --- a/third_party/github.com/coreos/go-etcd/etcd/delete.go +++ b/third_party/github.com/coreos/go-etcd/etcd/delete.go @@ -1,17 +1,26 @@ package etcd -// DeleteAll deletes everything under the given key. If the key -// points to a file, the file will be deleted. If the key points -// to a directory, then everything under the directory, include +// Delete deletes the given key. +// When recursive set to false If the key points to a +// directory, the method will fail. +// When recursive set to true, if the key points to a file, +// the file will be deleted. If the key points +// to a directory, then everything under the directory, including // all child directories, will be deleted. -func (c *Client) DeleteAll(key string) (*Response, error) { - return c.delete(key, options{ - "recursive": true, - }) +func (c *Client) Delete(key string, recursive bool) (*Response, error) { + raw, err := c.DeleteRaw(key, recursive) + + if err != nil { + return nil, err + } + + return raw.toResponse() } -// Delete deletes the given key. If the key points to a -// directory, the method will fail. -func (c *Client) Delete(key string) (*Response, error) { - return c.delete(key, nil) +func (c *Client) DeleteRaw(key string, recursive bool) (*RawResponse, error) { + ops := options{ + "recursive": recursive, + } + + return c.delete(key, ops) } diff --git a/third_party/github.com/coreos/go-etcd/etcd/delete_test.go b/third_party/github.com/coreos/go-etcd/etcd/delete_test.go index 0f8475a23..30089aac4 100644 --- a/third_party/github.com/coreos/go-etcd/etcd/delete_test.go +++ b/third_party/github.com/coreos/go-etcd/etcd/delete_test.go @@ -7,21 +7,21 @@ import ( func TestDelete(t *testing.T) { c := NewClient(nil) defer func() { - c.DeleteAll("foo") + c.Delete("foo", true) }() c.Set("foo", "bar", 5) - resp, err := c.Delete("foo") + resp, err := c.Delete("foo", false) if err != nil { t.Fatal(err) } - if !(resp.PrevValue == "bar" && resp.Value == "") { - t.Fatalf("Delete failed with %s %s", resp.PrevValue, - resp.Value) + if !(resp.Node.PrevValue == "bar" && resp.Node.Value == "") { + t.Fatalf("Delete failed with %s %s", resp.Node.PrevValue, + resp.Node.Value) } - resp, err = c.Delete("foo") + resp, err = c.Delete("foo", false) if err == nil { t.Fatalf("Delete should have failed because the key foo did not exist. "+ "The response was: %v", resp) @@ -31,32 +31,32 @@ func TestDelete(t *testing.T) { func TestDeleteAll(t *testing.T) { c := NewClient(nil) defer func() { - c.DeleteAll("foo") - c.DeleteAll("fooDir") + c.Delete("foo", true) + c.Delete("fooDir", true) }() c.Set("foo", "bar", 5) - resp, err := c.DeleteAll("foo") + resp, err := c.Delete("foo", true) if err != nil { t.Fatal(err) } - if !(resp.PrevValue == "bar" && resp.Value == "") { + if !(resp.Node.PrevValue == "bar" && resp.Node.Value == "") { t.Fatalf("DeleteAll 1 failed: %#v", resp) } c.SetDir("fooDir", 5) c.Set("fooDir/foo", "bar", 5) - resp, err = c.DeleteAll("fooDir") + resp, err = c.Delete("fooDir", true) if err != nil { t.Fatal(err) } - if !(resp.PrevValue == "" && resp.Value == "") { + if !(resp.Node.PrevValue == "" && resp.Node.Value == "") { t.Fatalf("DeleteAll 2 failed: %#v", resp) } - resp, err = c.DeleteAll("foo") + resp, err = c.Delete("foo", true) if err == nil { t.Fatalf("DeleteAll should have failed because the key foo did not exist. "+ "The response was: %v", resp) diff --git a/third_party/github.com/coreos/go-etcd/etcd/get.go b/third_party/github.com/coreos/go-etcd/etcd/get.go index d42a83c7d..7988f1a80 100644 --- a/third_party/github.com/coreos/go-etcd/etcd/get.go +++ b/third_party/github.com/coreos/go-etcd/etcd/get.go @@ -1,23 +1,27 @@ package etcd -// GetDir gets the all contents under the given key. -// If the key points to a file, the file is returned. -// If the key points to a directory, everything under it is returnd, -// including all contents under all child directories. -func (c *Client) GetAll(key string, sort bool) (*Response, error) { - return c.get(key, options{ - "recursive": true, - "sorted": sort, - }) -} - // Get gets the file or directory associated with the given key. // If the key points to a directory, files and directories under // it will be returned in sorted or unsorted order, depending on -// the sort flag. Note that contents under child directories -// will not be returned. To get those contents, use GetAll. -func (c *Client) Get(key string, sort bool) (*Response, error) { - return c.get(key, options{ - "sorted": sort, - }) +// the sort flag. +// If recursive is set to false, contents under child directories +// will not be returned. +// If recursive is set to true, all the contents will be returned. +func (c *Client) Get(key string, sort, recursive bool) (*Response, error) { + raw, err := c.RawGet(key, sort, recursive) + + if err != nil { + return nil, err + } + + return raw.toResponse() +} + +func (c *Client) RawGet(key string, sort, recursive bool) (*RawResponse, error) { + ops := options{ + "recursive": recursive, + "sorted": sort, + } + + return c.get(key, ops) } diff --git a/third_party/github.com/coreos/go-etcd/etcd/get_test.go b/third_party/github.com/coreos/go-etcd/etcd/get_test.go index a34946c7e..1f56d4ab5 100644 --- a/third_party/github.com/coreos/go-etcd/etcd/get_test.go +++ b/third_party/github.com/coreos/go-etcd/etcd/get_test.go @@ -8,22 +8,22 @@ import ( func TestGet(t *testing.T) { c := NewClient(nil) defer func() { - c.DeleteAll("foo") + c.Delete("foo", true) }() c.Set("foo", "bar", 5) - result, err := c.Get("foo", false) + result, err := c.Get("foo", false, false) if err != nil { t.Fatal(err) } - if result.Key != "/foo" || result.Value != "bar" { - t.Fatalf("Get failed with %s %s %v", result.Key, result.Value, result.TTL) + if result.Node.Key != "/foo" || result.Node.Value != "bar" { + t.Fatalf("Get failed with %s %s %v", result.Node.Key, result.Node.Value, result.Node.TTL) } - result, err = c.Get("goo", false) + result, err = c.Get("goo", false, false) if err == nil { t.Fatalf("should not be able to get non-exist key") } @@ -32,7 +32,7 @@ func TestGet(t *testing.T) { func TestGetAll(t *testing.T) { c := NewClient(nil) defer func() { - c.DeleteAll("fooDir") + c.Delete("fooDir", true) }() c.SetDir("fooDir", 5) @@ -40,25 +40,36 @@ func TestGetAll(t *testing.T) { c.Set("fooDir/k1", "v1", 5) // Return kv-pairs in sorted order - result, err := c.Get("fooDir", true) + result, err := c.Get("fooDir", true, false) if err != nil { t.Fatal(err) } - expected := kvPairs{ - KeyValuePair{ - Key: "/fooDir/k0", - Value: "v0", + expected := Nodes{ + Node{ + Key: "/fooDir/k0", + Value: "v0", + TTL: 5, + ModifiedIndex: 31, + CreatedIndex: 31, }, - KeyValuePair{ - Key: "/fooDir/k1", - Value: "v1", + Node{ + Key: "/fooDir/k1", + Value: "v1", + TTL: 5, + ModifiedIndex: 32, + CreatedIndex: 32, }, } - if !reflect.DeepEqual(result.Kvs, expected) { - t.Fatalf("(actual) %v != (expected) %v", result.Kvs, expected) + // do not check expiration time, too hard to fake + for i, _ := range result.Node.Nodes { + result.Node.Nodes[i].Expiration = nil + } + + if !reflect.DeepEqual(result.Node.Nodes, expected) { + t.Fatalf("(actual) %v != (expected) %v", result.Node.Nodes, expected) } // Test the `recursive` option @@ -66,34 +77,44 @@ func TestGetAll(t *testing.T) { c.Set("fooDir/childDir/k2", "v2", 5) // Return kv-pairs in sorted order - result, err = c.GetAll("fooDir", true) + result, err = c.Get("fooDir", true, true) + + // do not check expiration time, too hard to fake + result.Node.Expiration = nil + for i, _ := range result.Node.Nodes { + result.Node.Nodes[i].Expiration = nil + } if err != nil { t.Fatal(err) } - expected = kvPairs{ - KeyValuePair{ + expected = Nodes{ + Node{ Key: "/fooDir/childDir", Dir: true, - KVPairs: kvPairs{ - KeyValuePair{ + Nodes: Nodes{ + Node{ Key: "/fooDir/childDir/k2", Value: "v2", + TTL: 5, }, }, + TTL: 5, }, - KeyValuePair{ + Node{ Key: "/fooDir/k0", Value: "v0", + TTL: 5, }, - KeyValuePair{ + Node{ Key: "/fooDir/k1", Value: "v1", + TTL: 5, }, } - if !reflect.DeepEqual(result.Kvs, expected) { - t.Fatalf("(actual) %v != (expected) %v", result.Kvs) + if !reflect.DeepEqual(result.Node.Nodes, expected) { + t.Fatalf("(actual) %v != (expected) %v", result.Node.Nodes, expected) } } diff --git a/third_party/github.com/coreos/go-etcd/etcd/options.go b/third_party/github.com/coreos/go-etcd/etcd/options.go new file mode 100644 index 000000000..31fe80288 --- /dev/null +++ b/third_party/github.com/coreos/go-etcd/etcd/options.go @@ -0,0 +1,68 @@ +package etcd + +import ( + "fmt" + "net/url" + "reflect" +) + +type options map[string]interface{} + +// An internally-used data structure that represents a mapping +// between valid options and their kinds +type validOptions map[string]reflect.Kind + +// Valid options for GET, PUT, POST, DELETE +// Using CAPITALIZED_UNDERSCORE to emphasize that these +// values are meant to be used as constants. +var ( + VALID_GET_OPTIONS = validOptions{ + "recursive": reflect.Bool, + "consistent": reflect.Bool, + "sorted": reflect.Bool, + "wait": reflect.Bool, + "waitIndex": reflect.Uint64, + } + + VALID_PUT_OPTIONS = validOptions{ + "prevValue": reflect.String, + "prevIndex": reflect.Uint64, + "prevExist": reflect.Bool, + } + + VALID_POST_OPTIONS = validOptions{} + + VALID_DELETE_OPTIONS = validOptions{ + "recursive": reflect.Bool, + } +) + +// Convert options to a string of HTML parameters +func (ops options) toParameters(validOps validOptions) (string, error) { + p := "?" + values := url.Values{} + + if ops == nil { + return "", nil + } + + for k, v := range ops { + // Check if the given option is valid (that it exists) + kind := validOps[k] + if kind == reflect.Invalid { + return "", fmt.Errorf("Invalid option: %v", k) + } + + // Check if the given option is of the valid type + t := reflect.TypeOf(v) + if kind != t.Kind() { + return "", fmt.Errorf("Option %s should be of %v kind, not of %v kind.", + k, kind, t.Kind()) + } + + values.Set(k, fmt.Sprintf("%v", v)) + } + + p += values.Encode() + return p, nil +} diff --git a/third_party/github.com/coreos/go-etcd/etcd/requests.go b/third_party/github.com/coreos/go-etcd/etcd/requests.go index 83e3b519e..7385bfacd 100644 --- a/third_party/github.com/coreos/go-etcd/etcd/requests.go +++ b/third_party/github.com/coreos/go-etcd/etcd/requests.go @@ -1,54 +1,18 @@ package etcd import ( - "encoding/json" - "errors" "fmt" "io/ioutil" "math/rand" "net/http" "net/url" "path" - "reflect" "strings" "time" ) -// Valid options for GET, PUT, POST, DELETE -// Using CAPITALIZED_UNDERSCORE to emphasize that these -// values are meant to be used as constants. -var ( - VALID_GET_OPTIONS = validOptions{ - "recursive": reflect.Bool, - "consistent": reflect.Bool, - "sorted": reflect.Bool, - "wait": reflect.Bool, - "waitIndex": reflect.Uint64, - } - - VALID_PUT_OPTIONS = validOptions{ - "prevValue": reflect.String, - "prevIndex": reflect.Uint64, - "prevExist": reflect.Bool, - } - - VALID_POST_OPTIONS = validOptions{} - - VALID_DELETE_OPTIONS = validOptions{ - "recursive": reflect.Bool, - } - - curlChan chan string -) - -// SetCurlChan sets a channel to which cURL commands which can be used to -// re-produce requests are sent. This is useful for debugging. -func SetCurlChan(c chan string) { - curlChan = c -} - // get issues a GET request -func (c *Client) get(key string, options options) (*Response, error) { +func (c *Client) get(key string, options options) (*RawResponse, error) { logger.Debugf("get %s [%s]", key, c.cluster.Leader) p := path.Join("keys", key) @@ -57,15 +21,14 @@ func (c *Client) get(key string, options options) (*Response, error) { if c.config.Consistency == STRONG_CONSISTENCY { options["consistent"] = true } - if options != nil { - str, err := optionsToString(options, VALID_GET_OPTIONS) - if err != nil { - return nil, err - } - p += str - } - resp, err := c.sendRequest("GET", p, url.Values{}) + str, err := options.toParameters(VALID_GET_OPTIONS) + if err != nil { + return nil, err + } + p += str + + resp, err := c.sendRequest("GET", p, nil) if err != nil { return nil, err @@ -75,28 +38,19 @@ func (c *Client) get(key string, options options) (*Response, error) { } // put issues a PUT request -func (c *Client) put(key string, value string, ttl uint64, options options) (*Response, error) { +func (c *Client) put(key string, value string, ttl uint64, + options options) (*RawResponse, error) { + logger.Debugf("put %s, %s, ttl: %d, [%s]", key, value, ttl, c.cluster.Leader) - v := url.Values{} - - if value != "" { - v.Set("value", value) - } - - if ttl > 0 { - v.Set("ttl", fmt.Sprintf("%v", ttl)) - } - p := path.Join("keys", key) - if options != nil { - str, err := optionsToString(options, VALID_PUT_OPTIONS) - if err != nil { - return nil, err - } - p += str - } - resp, err := c.sendRequest("PUT", p, v) + str, err := options.toParameters(VALID_PUT_OPTIONS) + if err != nil { + return nil, err + } + p += str + + resp, err := c.sendRequest("PUT", p, buildValues(value, ttl)) if err != nil { return nil, err @@ -106,19 +60,11 @@ func (c *Client) put(key string, value string, ttl uint64, options options) (*Re } // post issues a POST request -func (c *Client) post(key string, value string, ttl uint64) (*Response, error) { +func (c *Client) post(key string, value string, ttl uint64) (*RawResponse, error) { logger.Debugf("post %s, %s, ttl: %d, [%s]", key, value, ttl, c.cluster.Leader) - v := url.Values{} + p := path.Join("keys", key) - if value != "" { - v.Set("value", value) - } - - if ttl > 0 { - v.Set("ttl", fmt.Sprintf("%v", ttl)) - } - - resp, err := c.sendRequest("POST", path.Join("keys", key), v) + resp, err := c.sendRequest("POST", p, buildValues(value, ttl)) if err != nil { return nil, err @@ -128,20 +74,18 @@ func (c *Client) post(key string, value string, ttl uint64) (*Response, error) { } // delete issues a DELETE request -func (c *Client) delete(key string, options options) (*Response, error) { +func (c *Client) delete(key string, options options) (*RawResponse, error) { logger.Debugf("delete %s [%s]", key, c.cluster.Leader) - v := url.Values{} p := path.Join("keys", key) - if options != nil { - str, err := optionsToString(options, VALID_DELETE_OPTIONS) - if err != nil { - return nil, err - } - p += str - } - resp, err := c.sendRequest("DELETE", p, v) + str, err := options.toParameters(VALID_DELETE_OPTIONS) + if err != nil { + return nil, err + } + p += str + + resp, err := c.sendRequest("DELETE", p, nil) if err != nil { return nil, err @@ -151,126 +95,128 @@ func (c *Client) delete(key string, options options) (*Response, error) { } // sendRequest sends a HTTP request and returns a Response as defined by etcd -func (c *Client) sendRequest(method string, _path string, values url.Values) (*Response, error) { - var body string = values.Encode() - var resp *http.Response - var req *http.Request +func (c *Client) sendRequest(method string, relativePath string, + values url.Values) (*RawResponse, error) { + + var req *http.Request + var resp *http.Response + var httpPath string + var err error + var b []byte + + trial := 0 - retry := 0 // if we connect to a follower, we will retry until we found a leader for { - var httpPath string - - // If _path has schema already, then it's assumed to be - // a complete URL and therefore needs no further processing. - u, err := url.Parse(_path) - if err != nil { - return nil, err + trial++ + logger.Debug("begin trail ", trial) + if trial > 2*len(c.cluster.Machines) { + return nil, fmt.Errorf("Cannot reach servers after %v time", trial) } - if u.Scheme != "" { - httpPath = _path + if method == "GET" && c.config.Consistency == WEAK_CONSISTENCY { + // If it's a GET and consistency level is set to WEAK, + // then use a random machine. + httpPath = c.getHttpPath(true, relativePath) } else { - if method == "GET" && c.config.Consistency == WEAK_CONSISTENCY { - // If it's a GET and consistency level is set to WEAK, - // then use a random machine. - httpPath = c.getHttpPath(true, _path) - } else { - // Else use the leader. - httpPath = c.getHttpPath(false, _path) - } + // Else use the leader. + httpPath = c.getHttpPath(false, relativePath) } // Return a cURL command if curlChan is set - if curlChan != nil { + if c.cURLch != nil { command := fmt.Sprintf("curl -X %s %s", method, httpPath) for key, value := range values { command += fmt.Sprintf(" -d %s=%s", key, value[0]) } - curlChan <- command + c.sendCURL(command) } logger.Debug("send.request.to ", httpPath, " | method ", method) - if body == "" { + if values == nil { req, _ = http.NewRequest(method, httpPath, nil) - } else { - req, _ = http.NewRequest(method, httpPath, strings.NewReader(body)) - req.Header.Set("Content-Type", "application/x-www-form-urlencoded; param=value") + req, _ = http.NewRequest(method, httpPath, + strings.NewReader(values.Encode())) + + req.Header.Set("Content-Type", + "application/x-www-form-urlencoded; param=value") } - resp, err = c.httpClient.Do(req) - - logger.Debug("recv.response.from ", httpPath) // network error, change a machine! - if err != nil { - retry++ - if retry > 2*len(c.cluster.Machines) { - return nil, errors.New("Cannot reach servers") - } - num := retry % len(c.cluster.Machines) - logger.Debug("update.leader[", c.cluster.Leader, ",", c.cluster.Machines[num], "]") - c.cluster.Leader = c.cluster.Machines[num] + if resp, err = c.httpClient.Do(req); err != nil { + c.switchLeader(trial % len(c.cluster.Machines)) time.Sleep(time.Millisecond * 200) continue } if resp != nil { - if resp.StatusCode == http.StatusTemporaryRedirect { - httpPath := resp.Header.Get("Location") + logger.Debug("recv.response.from ", httpPath) - resp.Body.Close() + var ok bool + ok, b = c.handleResp(resp) - if httpPath == "" { - return nil, errors.New("Cannot get redirection location") - } - - c.updateLeader(httpPath) - logger.Debug("send.redirect") - // try to connect the leader + if !ok { continue - } else if resp.StatusCode == http.StatusInternalServerError { - resp.Body.Close() - - retry++ - if retry > 2*len(c.cluster.Machines) { - return nil, errors.New("Cannot reach servers") - } - continue - } else { - logger.Debug("send.return.response ", httpPath) - break } + logger.Debug("recv.success.", httpPath) + break } - logger.Debug("error.from ", httpPath, " ", err.Error()) + + // should not reach here + // err and resp should not be nil at the same time + logger.Debug("error.from ", httpPath) return nil, err } - // Convert HTTP response to etcd response - b, err := ioutil.ReadAll(resp.Body) - - resp.Body.Close() - - if err != nil { - return nil, err + r := &RawResponse{ + StatusCode: resp.StatusCode, + Body: b, + Header: resp.Header, } - if !(resp.StatusCode == http.StatusOK || - resp.StatusCode == http.StatusCreated) { - return nil, handleError(b) + return r, nil +} + +// handleResp handles the responses from the etcd server +// If status code is OK, read the http body and return it as byte array +// If status code is TemporaryRedirect, update leader. +// If status code is InternalServerError, sleep for 200ms. +func (c *Client) handleResp(resp *http.Response) (bool, []byte) { + defer resp.Body.Close() + + code := resp.StatusCode + + if code == http.StatusTemporaryRedirect { + u, err := resp.Location() + + if err != nil { + logger.Warning(err) + } else { + c.updateLeader(u) + } + + return false, nil + + } else if code == http.StatusInternalServerError { + time.Sleep(time.Millisecond * 200) + + } else if code == http.StatusOK || + code == http.StatusCreated || + code == http.StatusBadRequest { + b, err := ioutil.ReadAll(resp.Body) + + if err != nil { + return false, nil + } + + return true, b } - var result Response - - err = json.Unmarshal(b, &result) - - if err != nil { - return nil, err - } - - return &result, nil + logger.Warning("bad status code ", resp.StatusCode) + return false, nil } func (c *Client) getHttpPath(random bool, s ...string) string { @@ -288,3 +234,18 @@ func (c *Client) getHttpPath(random bool, s ...string) string { return fullPath } + +// buildValues builds a url.Values map according to the given value and ttl +func buildValues(value string, ttl uint64) url.Values { + v := url.Values{} + + if value != "" { + v.Set("value", value) + } + + if ttl > 0 { + v.Set("ttl", fmt.Sprintf("%v", ttl)) + } + + return v +} diff --git a/third_party/github.com/coreos/go-etcd/etcd/response.go b/third_party/github.com/coreos/go-etcd/etcd/response.go index a8bb21a68..746a7c6e5 100644 --- a/third_party/github.com/coreos/go-etcd/etcd/response.go +++ b/third_party/github.com/coreos/go-etcd/etcd/response.go @@ -1,51 +1,68 @@ package etcd import ( + "encoding/json" + "net/http" "time" ) -// The response object from the server. +const ( + rawResponse = iota + normalResponse +) + +type responseType int + +type RawResponse struct { + StatusCode int + Body []byte + Header http.Header +} + +func (rr *RawResponse) toResponse() (*Response, error) { + if rr.StatusCode == http.StatusBadRequest { + return nil, handleError(rr.Body) + } + + resp := new(Response) + + err := json.Unmarshal(rr.Body, resp) + + if err != nil { + return nil, err + } + + return resp, nil +} + type Response struct { - Action string `json:"action"` - Key string `json:"key"` - Dir bool `json:"dir,omitempty"` - PrevValue string `json:"prevValue,omitempty"` - Value string `json:"value,omitempty"` - Kvs kvPairs `json:"kvs,omitempty"` - - // If the key did not exist before the action, - // this field should be set to true - NewKey bool `json:"newKey,omitempty"` - - Expiration *time.Time `json:"expiration,omitempty"` - - // Time to live in second - TTL int64 `json:"ttl,omitempty"` - - // The command index of the raft machine when the command is executed - ModifiedIndex uint64 `json:"modifiedIndex"` + Action string `json:"action"` + Node *Node `json:"node,omitempty"` } -// When user list a directory, we add all the node into key-value pair slice -type KeyValuePair struct { - Key string `json:"key, omitempty"` - Value string `json:"value,omitempty"` - Dir bool `json:"dir,omitempty"` - KVPairs kvPairs `json:"kvs,omitempty"` - TTL int64 `json:"ttl,omitempty"` +type Node struct { + Key string `json:"key, omitempty"` + PrevValue string `json:"prevValue,omitempty"` + Value string `json:"value,omitempty"` + Dir bool `json:"dir,omitempty"` + Expiration *time.Time `json:"expiration,omitempty"` + TTL int64 `json:"ttl,omitempty"` + Nodes Nodes `json:"nodes,omitempty"` + ModifiedIndex uint64 `json:"modifiedIndex,omitempty"` + CreatedIndex uint64 `json:"createdIndex,omitempty"` } -type kvPairs []KeyValuePair +type Nodes []Node // interfaces for sorting -func (kvs kvPairs) Len() int { - return len(kvs) +func (ns Nodes) Len() int { + return len(ns) } -func (kvs kvPairs) Less(i, j int) bool { - return kvs[i].Key < kvs[j].Key +func (ns Nodes) Less(i, j int) bool { + return ns[i].Key < ns[j].Key } -func (kvs kvPairs) Swap(i, j int) { - kvs[i], kvs[j] = kvs[j], kvs[i] +func (ns Nodes) Swap(i, j int) { + ns[i], ns[j] = ns[j], ns[i] } diff --git a/third_party/github.com/coreos/go-etcd/etcd/set_curl_chan_test.go b/third_party/github.com/coreos/go-etcd/etcd/set_curl_chan_test.go index 6d0633109..756e31781 100644 --- a/third_party/github.com/coreos/go-etcd/etcd/set_curl_chan_test.go +++ b/third_party/github.com/coreos/go-etcd/etcd/set_curl_chan_test.go @@ -7,12 +7,11 @@ import ( func TestSetCurlChan(t *testing.T) { c := NewClient(nil) - defer func() { - c.DeleteAll("foo") - }() + c.OpenCURL() - curlChan := make(chan string, 1) - SetCurlChan(curlChan) + defer func() { + c.Delete("foo", true) + }() _, err := c.Set("foo", "bar", 5) if err != nil { @@ -21,21 +20,21 @@ func TestSetCurlChan(t *testing.T) { expected := fmt.Sprintf("curl -X PUT %s/v2/keys/foo -d value=bar -d ttl=5", c.cluster.Leader) - actual := <-curlChan + actual := c.RecvCURL() if expected != actual { t.Fatalf(`Command "%s" is not equal to expected value "%s"`, actual, expected) } c.SetConsistency(STRONG_CONSISTENCY) - _, err = c.Get("foo", false) + _, err = c.Get("foo", false, false) if err != nil { t.Fatal(err) } - expected = fmt.Sprintf("curl -X GET %s/v2/keys/foo?consistent=true&sorted=false", + expected = fmt.Sprintf("curl -X GET %s/v2/keys/foo?consistent=true&recursive=false&sorted=false", c.cluster.Leader) - actual = <-curlChan + actual = c.RecvCURL() if expected != actual { t.Fatalf(`Command "%s" is not equal to expected value "%s"`, actual, expected) diff --git a/third_party/github.com/coreos/go-etcd/etcd/set_update_create.go b/third_party/github.com/coreos/go-etcd/etcd/set_update_create.go index 281cd577c..8558db1ac 100644 --- a/third_party/github.com/coreos/go-etcd/etcd/set_update_create.go +++ b/third_party/github.com/coreos/go-etcd/etcd/set_update_create.go @@ -2,42 +2,110 @@ package etcd // SetDir sets the given key to a directory. func (c *Client) SetDir(key string, ttl uint64) (*Response, error) { - return c.put(key, "", ttl, nil) + raw, err := c.RawSetDir(key, ttl) + + if err != nil { + return nil, err + } + + return raw.toResponse() } // UpdateDir updates the given key to a directory. It succeeds only if the // given key already exists. func (c *Client) UpdateDir(key string, ttl uint64) (*Response, error) { - return c.put(key, "", ttl, options{ - "prevExist": true, - }) + raw, err := c.RawUpdateDir(key, ttl) + + if err != nil { + return nil, err + } + + return raw.toResponse() } // UpdateDir creates a directory under the given key. It succeeds only if // the given key does not yet exist. func (c *Client) CreateDir(key string, ttl uint64) (*Response, error) { - return c.put(key, "", ttl, options{ - "prevExist": false, - }) + raw, err := c.RawCreateDir(key, ttl) + + if err != nil { + return nil, err + } + + return raw.toResponse() } // Set sets the given key to the given value. func (c *Client) Set(key string, value string, ttl uint64) (*Response, error) { - return c.put(key, value, ttl, nil) + raw, err := c.RawSet(key, value, ttl) + + if err != nil { + return nil, err + } + + return raw.toResponse() } // Update updates the given key to the given value. It succeeds only if the // given key already exists. func (c *Client) Update(key string, value string, ttl uint64) (*Response, error) { - return c.put(key, value, ttl, options{ - "prevExist": true, - }) + raw, err := c.RawUpdate(key, value, ttl) + + if err != nil { + return nil, err + } + + return raw.toResponse() } // Create creates a file with the given value under the given key. It succeeds // only if the given key does not yet exist. func (c *Client) Create(key string, value string, ttl uint64) (*Response, error) { - return c.put(key, value, ttl, options{ - "prevExist": false, - }) + raw, err := c.RawCreate(key, value, ttl) + + if err != nil { + return nil, err + } + + return raw.toResponse() +} + +func (c *Client) RawSetDir(key string, ttl uint64) (*RawResponse, error) { + return c.put(key, "", ttl, nil) +} + +func (c *Client) RawUpdateDir(key string, ttl uint64) (*RawResponse, error) { + ops := options{ + "prevExist": true, + } + + return c.put(key, "", ttl, ops) +} + +func (c *Client) RawCreateDir(key string, ttl uint64) (*RawResponse, error) { + ops := options{ + "prevExist": false, + } + + return c.put(key, "", ttl, ops) +} + +func (c *Client) RawSet(key string, value string, ttl uint64) (*RawResponse, error) { + return c.put(key, value, ttl, nil) +} + +func (c *Client) RawUpdate(key string, value string, ttl uint64) (*RawResponse, error) { + ops := options{ + "prevExist": true, + } + + return c.put(key, value, ttl, ops) +} + +func (c *Client) RawCreate(key string, value string, ttl uint64) (*RawResponse, error) { + ops := options{ + "prevExist": false, + } + + return c.put(key, value, ttl, ops) } diff --git a/third_party/github.com/coreos/go-etcd/etcd/set_update_create_test.go b/third_party/github.com/coreos/go-etcd/etcd/set_update_create_test.go index 6f27fdfa6..215622b9e 100644 --- a/third_party/github.com/coreos/go-etcd/etcd/set_update_create_test.go +++ b/third_party/github.com/coreos/go-etcd/etcd/set_update_create_test.go @@ -7,14 +7,14 @@ import ( func TestSet(t *testing.T) { c := NewClient(nil) defer func() { - c.DeleteAll("foo") + c.Delete("foo", true) }() resp, err := c.Set("foo", "bar", 5) if err != nil { t.Fatal(err) } - if resp.Key != "/foo" || resp.Value != "bar" || resp.TTL != 5 { + if resp.Node.Key != "/foo" || resp.Node.Value != "bar" || resp.Node.TTL != 5 { t.Fatalf("Set 1 failed: %#v", resp) } @@ -22,8 +22,8 @@ func TestSet(t *testing.T) { if err != nil { t.Fatal(err) } - if !(resp.Key == "/foo" && resp.Value == "bar2" && - resp.PrevValue == "bar" && resp.TTL == 5) { + if !(resp.Node.Key == "/foo" && resp.Node.Value == "bar2" && + resp.Node.PrevValue == "bar" && resp.Node.TTL == 5) { t.Fatalf("Set 2 failed: %#v", resp) } } @@ -31,12 +31,12 @@ func TestSet(t *testing.T) { func TestUpdate(t *testing.T) { c := NewClient(nil) defer func() { - c.DeleteAll("foo") - c.DeleteAll("nonexistent") + c.Delete("foo", true) + c.Delete("nonexistent", true) }() resp, err := c.Set("foo", "bar", 5) - t.Logf("%#v", resp) + if err != nil { t.Fatal(err) } @@ -47,8 +47,8 @@ func TestUpdate(t *testing.T) { t.Fatal(err) } - if !(resp.Action == "update" && resp.Key == "/foo" && - resp.PrevValue == "bar" && resp.TTL == 5) { + if !(resp.Action == "update" && resp.Node.Key == "/foo" && + resp.Node.PrevValue == "bar" && resp.Node.TTL == 5) { t.Fatalf("Update 1 failed: %#v", resp) } @@ -56,14 +56,14 @@ func TestUpdate(t *testing.T) { resp, err = c.Update("nonexistent", "whatever", 5) if err == nil { t.Fatalf("The key %v did not exist, so the update should have failed."+ - "The response was: %#v", resp.Key, resp) + "The response was: %#v", resp.Node.Key, resp) } } func TestCreate(t *testing.T) { c := NewClient(nil) defer func() { - c.DeleteAll("newKey") + c.Delete("newKey", true) }() newKey := "/newKey" @@ -75,8 +75,8 @@ func TestCreate(t *testing.T) { t.Fatal(err) } - if !(resp.Action == "create" && resp.Key == newKey && - resp.Value == newValue && resp.PrevValue == "" && resp.TTL == 5) { + if !(resp.Action == "create" && resp.Node.Key == newKey && + resp.Node.Value == newValue && resp.Node.PrevValue == "" && resp.Node.TTL == 5) { t.Fatalf("Create 1 failed: %#v", resp) } @@ -84,22 +84,22 @@ func TestCreate(t *testing.T) { resp, err = c.Create(newKey, newValue, 5) if err == nil { t.Fatalf("The key %v did exist, so the creation should have failed."+ - "The response was: %#v", resp.Key, resp) + "The response was: %#v", resp.Node.Key, resp) } } func TestSetDir(t *testing.T) { c := NewClient(nil) defer func() { - c.DeleteAll("foo") - c.DeleteAll("fooDir") + c.Delete("foo", true) + c.Delete("fooDir", true) }() resp, err := c.SetDir("fooDir", 5) if err != nil { t.Fatal(err) } - if !(resp.Key == "/fooDir" && resp.Value == "" && resp.TTL == 5) { + if !(resp.Node.Key == "/fooDir" && resp.Node.Value == "" && resp.Node.TTL == 5) { t.Fatalf("SetDir 1 failed: %#v", resp) } @@ -120,8 +120,8 @@ func TestSetDir(t *testing.T) { if err != nil { t.Fatal(err) } - if !(resp.Key == "/foo" && resp.Value == "" && - resp.PrevValue == "bar" && resp.TTL == 5) { + if !(resp.Node.Key == "/foo" && resp.Node.Value == "" && + resp.Node.PrevValue == "bar" && resp.Node.TTL == 5) { t.Fatalf("SetDir 2 failed: %#v", resp) } } @@ -129,11 +129,10 @@ func TestSetDir(t *testing.T) { func TestUpdateDir(t *testing.T) { c := NewClient(nil) defer func() { - c.DeleteAll("fooDir") + c.Delete("fooDir", true) }() resp, err := c.SetDir("fooDir", 5) - t.Logf("%#v", resp) if err != nil { t.Fatal(err) } @@ -144,8 +143,8 @@ func TestUpdateDir(t *testing.T) { t.Fatal(err) } - if !(resp.Action == "update" && resp.Key == "/fooDir" && - resp.Value == "" && resp.PrevValue == "" && resp.TTL == 5) { + if !(resp.Action == "update" && resp.Node.Key == "/fooDir" && + resp.Node.Value == "" && resp.Node.PrevValue == "" && resp.Node.TTL == 5) { t.Fatalf("UpdateDir 1 failed: %#v", resp) } @@ -153,14 +152,14 @@ func TestUpdateDir(t *testing.T) { resp, err = c.UpdateDir("nonexistentDir", 5) if err == nil { t.Fatalf("The key %v did not exist, so the update should have failed."+ - "The response was: %#v", resp.Key, resp) + "The response was: %#v", resp.Node.Key, resp) } } func TestCreateDir(t *testing.T) { c := NewClient(nil) defer func() { - c.DeleteAll("fooDir") + c.Delete("fooDir", true) }() // This should succeed @@ -169,8 +168,8 @@ func TestCreateDir(t *testing.T) { t.Fatal(err) } - if !(resp.Action == "create" && resp.Key == "/fooDir" && - resp.Value == "" && resp.PrevValue == "" && resp.TTL == 5) { + if !(resp.Action == "create" && resp.Node.Key == "/fooDir" && + resp.Node.Value == "" && resp.Node.PrevValue == "" && resp.Node.TTL == 5) { t.Fatalf("CreateDir 1 failed: %#v", resp) } @@ -178,6 +177,6 @@ func TestCreateDir(t *testing.T) { resp, err = c.CreateDir("fooDir", 5) if err == nil { t.Fatalf("The key %v did exist, so the creation should have failed."+ - "The response was: %#v", resp.Key, resp) + "The response was: %#v", resp.Node.Key, resp) } } diff --git a/third_party/github.com/coreos/go-etcd/etcd/watch.go b/third_party/github.com/coreos/go-etcd/etcd/watch.go index bbce2039b..1e1ac74a5 100644 --- a/third_party/github.com/coreos/go-etcd/etcd/watch.go +++ b/third_party/github.com/coreos/go-etcd/etcd/watch.go @@ -9,44 +9,74 @@ var ( ErrWatchStoppedByUser = errors.New("Watch stopped by the user via stop channel") ) -// WatchAll returns the first change under the given prefix since the given index. To -// watch for the latest change, set waitIndex = 0. +// If recursive is set to true the watch returns the first change under the given +// prefix since the given index. // -// If the prefix points to a directory, any change under it, including all child directories, -// will be returned. +// If recursive is set to false the watch returns the first change to the given key +// since the given index. +// +// To watch for the latest change, set waitIndex = 0. // // If a receiver channel is given, it will be a long-term watch. Watch will block at the -// channel. And after someone receive the channel, it will go on to watch that prefix. -// If a stop channel is given, client can close long-term watch using the stop channel -func (c *Client) WatchAll(prefix string, waitIndex uint64, receiver chan *Response, stop chan bool) (*Response, error) { - return c.watch(prefix, waitIndex, true, receiver, stop) -} - -// Watch returns the first change to the given key since the given index. To -// watch for the latest change, set waitIndex = 0. -// -// If a receiver channel is given, it will be a long-term watch. Watch will block at the -// channel. And after someone receive the channel, it will go on to watch that -// prefix. If a stop channel is given, client can close long-term watch using -// the stop channel -func (c *Client) Watch(key string, waitIndex uint64, receiver chan *Response, stop chan bool) (*Response, error) { - return c.watch(key, waitIndex, false, receiver, stop) -} - -func (c *Client) watch(prefix string, waitIndex uint64, recursive bool, receiver chan *Response, stop chan bool) (*Response, error) { +//channel. After someone receives the channel, it will go on to watch that +// prefix. If a stop channel is given, the client can close long-term watch using +// the stop channel. +func (c *Client) Watch(prefix string, waitIndex uint64, recursive bool, + receiver chan *Response, stop chan bool) (*Response, error) { logger.Debugf("watch %s [%s]", prefix, c.cluster.Leader) if receiver == nil { - return c.watchOnce(prefix, waitIndex, recursive, stop) - } else { - for { - resp, err := c.watchOnce(prefix, waitIndex, recursive, stop) - if resp != nil { - waitIndex = resp.ModifiedIndex + 1 - receiver <- resp - } else { - return nil, err - } + raw, err := c.watchOnce(prefix, waitIndex, recursive, stop) + + if err != nil { + return nil, err } + + return raw.toResponse() + } + + for { + raw, err := c.watchOnce(prefix, waitIndex, recursive, stop) + + if err != nil { + return nil, err + } + + resp, err := raw.toResponse() + + if err != nil { + return nil, err + } + + waitIndex = resp.Node.ModifiedIndex + 1 + receiver <- resp + } + + return nil, nil +} + +func (c *Client) RawWatch(prefix string, waitIndex uint64, recursive bool, + receiver chan *RawResponse, stop chan bool) (*RawResponse, error) { + + logger.Debugf("rawWatch %s [%s]", prefix, c.cluster.Leader) + if receiver == nil { + return c.watchOnce(prefix, waitIndex, recursive, stop) + } + + for { + raw, err := c.watchOnce(prefix, waitIndex, recursive, stop) + + if err != nil { + return nil, err + } + + resp, err := raw.toResponse() + + if err != nil { + return nil, err + } + + waitIndex = resp.Node.ModifiedIndex + 1 + receiver <- raw } return nil, nil @@ -54,9 +84,9 @@ func (c *Client) watch(prefix string, waitIndex uint64, recursive bool, receiver // helper func // return when there is change under the given prefix -func (c *Client) watchOnce(key string, waitIndex uint64, recursive bool, stop chan bool) (*Response, error) { +func (c *Client) watchOnce(key string, waitIndex uint64, recursive bool, stop chan bool) (*RawResponse, error) { - respChan := make(chan *Response) + respChan := make(chan *RawResponse, 1) errChan := make(chan error) go func() { @@ -74,6 +104,7 @@ func (c *Client) watchOnce(key string, waitIndex uint64, recursive bool, stop ch if err != nil { errChan <- err + return } respChan <- resp diff --git a/third_party/github.com/coreos/go-etcd/etcd/watch_test.go b/third_party/github.com/coreos/go-etcd/etcd/watch_test.go index 10fc2b6b5..9b466489c 100644 --- a/third_party/github.com/coreos/go-etcd/etcd/watch_test.go +++ b/third_party/github.com/coreos/go-etcd/etcd/watch_test.go @@ -9,26 +9,26 @@ import ( func TestWatch(t *testing.T) { c := NewClient(nil) defer func() { - c.DeleteAll("watch_foo") + c.Delete("watch_foo", true) }() go setHelper("watch_foo", "bar", c) - resp, err := c.Watch("watch_foo", 0, nil, nil) + resp, err := c.Watch("watch_foo", 0, false, nil, nil) if err != nil { t.Fatal(err) } - if !(resp.Key == "/watch_foo" && resp.Value == "bar") { + if !(resp.Node.Key == "/watch_foo" && resp.Node.Value == "bar") { t.Fatalf("Watch 1 failed: %#v", resp) } go setHelper("watch_foo", "bar", c) - resp, err = c.Watch("watch_foo", resp.ModifiedIndex, nil, nil) + resp, err = c.Watch("watch_foo", resp.Node.ModifiedIndex+1, false, nil, nil) if err != nil { t.Fatal(err) } - if !(resp.Key == "/watch_foo" && resp.Value == "bar") { + if !(resp.Node.Key == "/watch_foo" && resp.Node.Value == "bar") { t.Fatalf("Watch 2 failed: %#v", resp) } @@ -39,7 +39,7 @@ func TestWatch(t *testing.T) { go receiver(ch, stop) - _, err = c.Watch("watch_foo", 0, ch, stop) + _, err = c.Watch("watch_foo", 0, false, ch, stop) if err != ErrWatchStoppedByUser { t.Fatalf("Watch returned a non-user stop error") } @@ -48,26 +48,26 @@ func TestWatch(t *testing.T) { func TestWatchAll(t *testing.T) { c := NewClient(nil) defer func() { - c.DeleteAll("watch_foo") + c.Delete("watch_foo", true) }() go setHelper("watch_foo/foo", "bar", c) - resp, err := c.WatchAll("watch_foo", 0, nil, nil) + resp, err := c.Watch("watch_foo", 0, true, nil, nil) if err != nil { t.Fatal(err) } - if !(resp.Key == "/watch_foo/foo" && resp.Value == "bar") { + if !(resp.Node.Key == "/watch_foo/foo" && resp.Node.Value == "bar") { t.Fatalf("WatchAll 1 failed: %#v", resp) } go setHelper("watch_foo/foo", "bar", c) - resp, err = c.WatchAll("watch_foo", resp.ModifiedIndex, nil, nil) + resp, err = c.Watch("watch_foo", resp.Node.ModifiedIndex+1, true, nil, nil) if err != nil { t.Fatal(err) } - if !(resp.Key == "/watch_foo/foo" && resp.Value == "bar") { + if !(resp.Node.Key == "/watch_foo/foo" && resp.Node.Value == "bar") { t.Fatalf("WatchAll 2 failed: %#v", resp) } @@ -78,7 +78,7 @@ func TestWatchAll(t *testing.T) { go receiver(ch, stop) - _, err = c.WatchAll("watch_foo", 0, ch, stop) + _, err = c.Watch("watch_foo", 0, true, ch, stop) if err != ErrWatchStoppedByUser { t.Fatalf("Watch returned a non-user stop error") } diff --git a/third_party/github.com/gorilla/context/README.md b/third_party/github.com/gorilla/context/README.md index 8ee62b426..c60a31b05 100644 --- a/third_party/github.com/gorilla/context/README.md +++ b/third_party/github.com/gorilla/context/README.md @@ -1,5 +1,6 @@ context ======= +[![Build Status](https://travis-ci.org/gorilla/context.png?branch=master)](https://travis-ci.org/gorilla/context) gorilla/context is a general purpose registry for global request variables. diff --git a/third_party/github.com/gorilla/context/context.go b/third_party/github.com/gorilla/context/context.go index 35d65561f..12accb114 100644 --- a/third_party/github.com/gorilla/context/context.go +++ b/third_party/github.com/gorilla/context/context.go @@ -92,7 +92,7 @@ func Purge(maxAge int) int { datat = make(map[*http.Request]int64) } else { min := time.Now().Unix() - int64(maxAge) - for r, _ := range data { + for r := range data { if datat[r] < min { clear(r) count++ diff --git a/third_party/github.com/gorilla/mux/README.md b/third_party/github.com/gorilla/mux/README.md index f6db41ad8..e60301b03 100644 --- a/third_party/github.com/gorilla/mux/README.md +++ b/third_party/github.com/gorilla/mux/README.md @@ -1,5 +1,6 @@ mux === +[![Build Status](https://travis-ci.org/gorilla/mux.png?branch=master)](https://travis-ci.org/gorilla/mux) gorilla/mux is a powerful URL router and dispatcher. diff --git a/third_party/github.com/gorilla/mux/mux.go b/third_party/github.com/gorilla/mux/mux.go index ddc1acc63..ca51a011d 100644 --- a/third_party/github.com/gorilla/mux/mux.go +++ b/third_party/github.com/gorilla/mux/mux.go @@ -67,6 +67,14 @@ func (r *Router) Match(req *http.Request, match *RouteMatch) bool { func (r *Router) ServeHTTP(w http.ResponseWriter, req *http.Request) { // Clean path to canonical form and redirect. if p := cleanPath(req.URL.Path); p != req.URL.Path { + + // Added 3 lines (Philip Schlump) - It was droping the query string and #whatever from query. + // This matches with fix in go 1.2 r.c. 4 for same problem. Go Issue: + // http://code.google.com/p/go/issues/detail?id=5252 + url := *req.URL + url.Path = p + p = url.String() + w.Header().Set("Location", p) w.WriteHeader(http.StatusMovedPermanently) return diff --git a/third_party/github.com/gorilla/mux/mux_test.go b/third_party/github.com/gorilla/mux/mux_test.go index 8789697f5..1a2a092df 100644 --- a/third_party/github.com/gorilla/mux/mux_test.go +++ b/third_party/github.com/gorilla/mux/mux_test.go @@ -22,6 +22,7 @@ type routeTest struct { shouldMatch bool // whether the request is expected to match the route at all } + func TestHost(t *testing.T) { // newRequestHost a new request with a method, url, and host header newRequestHost := func(method, url, host string) *http.Request { @@ -416,6 +417,15 @@ func TestQueries(t *testing.T) { path: "", shouldMatch: true, }, + { + title: "Queries route, match with a query string", + route: new(Route).Host("www.example.com").Path("/api").Queries("foo", "bar", "baz", "ding"), + request: newRequest("GET", "http://www.example.com/api?foo=bar&baz=ding"), + vars: map[string]string{}, + host: "", + path: "", + shouldMatch: true, + }, { title: "Queries route, bad query", route: new(Route).Queries("foo", "bar", "baz", "ding"), @@ -663,7 +673,7 @@ func testRoute(t *testing.T, test routeTest) { func TestKeepContext(t *testing.T) { func1 := func(w http.ResponseWriter, r *http.Request) {} - r := NewRouter() + r:= NewRouter() r.HandleFunc("/", func1).Name("func1") req, _ := http.NewRequest("GET", "http://localhost/", nil) @@ -688,6 +698,47 @@ func TestKeepContext(t *testing.T) { } + +type TestA301ResponseWriter struct { + hh http.Header + status int +} + +func (ho TestA301ResponseWriter) Header() http.Header { + return http.Header(ho.hh) +} + +func (ho TestA301ResponseWriter) Write( b []byte) (int, error) { + return 0, nil +} + +func (ho TestA301ResponseWriter) WriteHeader( code int ) { + ho.status = code +} + +func Test301Redirect(t *testing.T) { + m := make(http.Header) + + func1 := func(w http.ResponseWriter, r *http.Request) {} + func2 := func(w http.ResponseWriter, r *http.Request) {} + + r:= NewRouter() + r.HandleFunc("/api/", func2).Name("func2") + r.HandleFunc("/", func1).Name("func1") + + req, _ := http.NewRequest("GET", "http://localhost//api/?abc=def", nil) + + res := TestA301ResponseWriter{ + hh: m, + status : 0, + } + r.ServeHTTP(&res, req) + + if "http://localhost/api/?abc=def" != res.hh["Location"][0] { + t.Errorf("Should have complete URL with query string") + } +} + // https://plus.google.com/101022900381697718949/posts/eWy6DjFJ6uW func TestSubrouterHeader(t *testing.T) { expected := "func1 response" diff --git a/third_party/github.com/gorilla/mux/old_test.go b/third_party/github.com/gorilla/mux/old_test.go index 7e266bb69..42530590e 100644 --- a/third_party/github.com/gorilla/mux/old_test.go +++ b/third_party/github.com/gorilla/mux/old_test.go @@ -96,8 +96,8 @@ func TestRouteMatchers(t *testing.T) { method = "GET" headers = map[string]string{"X-Requested-With": "XMLHttpRequest"} resultVars = map[bool]map[string]string{ - true: map[string]string{"var1": "www", "var2": "product", "var3": "42"}, - false: map[string]string{}, + true: {"var1": "www", "var2": "product", "var3": "42"}, + false: {}, } } @@ -110,8 +110,8 @@ func TestRouteMatchers(t *testing.T) { method = "POST" headers = map[string]string{"Content-Type": "application/json"} resultVars = map[bool]map[string]string{ - true: map[string]string{"var4": "google", "var5": "product", "var6": "42"}, - false: map[string]string{}, + true: {"var4": "google", "var5": "product", "var6": "42"}, + false: {}, } } From 78e382cb6b687c5de1e0d0f7c090da86f904a0d3 Mon Sep 17 00:00:00 2001 From: Xiang Li Date: Sun, 1 Dec 2013 17:35:22 -0500 Subject: [PATCH 11/18] test add watcher prefix test --- store/event_history.go | 8 +++++++- store/watcher_test.go | 20 ++++++++++++++++++++ 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/store/event_history.go b/store/event_history.go index d61d54d60..8ca9e8f50 100644 --- a/store/event_history.go +++ b/store/event_history.go @@ -66,7 +66,13 @@ func (eh *EventHistory) scan(key string, recursive bool, index uint64) (*Event, ok := (e.Key == key) if recursive { - ok = ok || strings.HasPrefix(e.Key, path.Join(key, "/")) + // add tailing slash + key := path.Clean(key) + if key[len(key)-1] != '/' { + key = key + "/" + } + + ok = ok || strings.HasPrefix(e.Key, key) } if ok && index <= e.Index() { // make sure we bypass the smaller one diff --git a/store/watcher_test.go b/store/watcher_test.go index 386fec440..3afd44c75 100644 --- a/store/watcher_test.go +++ b/store/watcher_test.go @@ -68,4 +68,24 @@ func TestWatcher(t *testing.T) { t.Fatal("recv != send") } + // ensure we are doing exact matching rather than prefix matching + c, _ = wh.watch("/fo", true, 1) + + select { + case re = <-c: + t.Fatal("should not receive from channel:", re) + default: + // do nothing + } + + e = newEvent(Create, "/fo/bar", 3) + + wh.notify(e) + + re = <-c + + if e != re { + t.Fatal("recv != send") + } + } From 6252037376917169d02880954c6880ec37ccfd20 Mon Sep 17 00:00:00 2001 From: Xiang Li Date: Sun, 1 Dec 2013 18:01:24 -0500 Subject: [PATCH 12/18] fix root should be rdonly --- error/error.go | 2 ++ store/store.go | 25 +++++++++++++++++++------ 2 files changed, 21 insertions(+), 6 deletions(-) diff --git a/error/error.go b/error/error.go index dddad19b4..61eb7c3f7 100644 --- a/error/error.go +++ b/error/error.go @@ -32,6 +32,7 @@ const ( EcodeNotDir = 104 EcodeNodeExist = 105 EcodeKeyIsPreserved = 106 + EcodeRootROnly = 107 EcodeValueRequired = 200 EcodePrevValueRequired = 201 @@ -56,6 +57,7 @@ func init() { errors[EcodeNoMorePeer] = "Reached the max number of peers in the cluster" errors[EcodeNotDir] = "Not A Directory" errors[EcodeNodeExist] = "Already exists" // create + errors[EcodeRootROnly] = "Root is read only" errors[EcodeKeyIsPreserved] = "The prefix of given key is a keyword in etcd" // Post form related errors diff --git a/store/store.go b/store/store.go index 1edb02f29..688e2ccf8 100644 --- a/store/store.go +++ b/store/store.go @@ -156,8 +156,6 @@ func (s *store) Get(nodePath string, recursive, sorted bool) (*Event, error) { // If the node has already existed, create will fail. // If any node on the path is a file, create will fail. func (s *store) Create(nodePath string, value string, unique bool, expireTime time.Time) (*Event, error) { - nodePath = path.Clean(path.Join("/", nodePath)) - s.worldLock.Lock() defer s.worldLock.Unlock() e, err := s.internalCreate(nodePath, value, unique, false, expireTime, Create) @@ -173,8 +171,6 @@ func (s *store) Create(nodePath string, value string, unique bool, expireTime ti // Set function creates or replace the Node at nodePath. func (s *store) Set(nodePath string, value string, expireTime time.Time) (*Event, error) { - nodePath = path.Clean(path.Join("/", nodePath)) - s.worldLock.Lock() defer s.worldLock.Unlock() e, err := s.internalCreate(nodePath, value, false, true, expireTime, Set) @@ -192,6 +188,10 @@ func (s *store) CompareAndSwap(nodePath string, prevValue string, prevIndex uint value string, expireTime time.Time) (*Event, error) { nodePath = path.Clean(path.Join("/", nodePath)) + // we do not allow the user to change "/" + if nodePath == "/" { + return nil, etcdErr.NewError(etcdErr.EcodeRootROnly, "/", s.CurrentIndex) + } s.worldLock.Lock() defer s.worldLock.Unlock() @@ -238,6 +238,10 @@ func (s *store) CompareAndSwap(nodePath string, prevValue string, prevIndex uint // If the node is a directory, recursive must be true to delete it. func (s *store) Delete(nodePath string, recursive bool) (*Event, error) { nodePath = path.Clean(path.Join("/", nodePath)) + // we do not allow the user to change "/" + if nodePath == "/" { + return nil, etcdErr.NewError(etcdErr.EcodeRootROnly, "/", s.CurrentIndex) + } s.worldLock.Lock() defer s.worldLock.Unlock() @@ -334,13 +338,17 @@ func (s *store) walk(nodePath string, walkFunc func(prev *Node, component string // If the node is a file, the value and the ttl can be updated. // If the node is a directory, only the ttl can be updated. func (s *store) Update(nodePath string, newValue string, expireTime time.Time) (*Event, error) { + nodePath = path.Clean(path.Join("/", nodePath)) + // we do not allow the user to change "/" + if nodePath == "/" { + return nil, etcdErr.NewError(etcdErr.EcodeRootROnly, "/", s.CurrentIndex) + } + s.worldLock.Lock() defer s.worldLock.Unlock() currIndex, nextIndex := s.CurrentIndex, s.CurrentIndex+1 - nodePath = path.Clean(path.Join("/", nodePath)) - n, err := s.internalGet(nodePath) if err != nil { // if the node does not exist, return error @@ -390,6 +398,11 @@ func (s *store) internalCreate(nodePath string, value string, unique bool, repla nodePath = path.Clean(path.Join("/", nodePath)) + // we do not allow the user to change "/" + if nodePath == "/" { + return nil, etcdErr.NewError(etcdErr.EcodeRootROnly, "/", currIndex) + } + // Assume expire times that are way in the past are not valid. // This can occur when the time is serialized to JSON and read back in. if expireTime.Before(minExpireTime) { From b929e71948fbb2fcc5c960390b999fef29916162 Mon Sep 17 00:00:00 2001 From: Xiang Li Date: Sun, 1 Dec 2013 18:16:32 -0500 Subject: [PATCH 13/18] tests add root readonly test --- store/store_test.go | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/store/store_test.go b/store/store_test.go index 29b2986b9..863874fcb 100644 --- a/store/store_test.go +++ b/store/store_test.go @@ -213,6 +213,26 @@ func TestStoreDeleteDiretory(t *testing.T) { assert.Equal(t, e.Action, "delete", "") } +func TestRootRdOnly(t *testing.T) { + s := newStore() + + _, err := s.Set("/", "", Permanent) + assert.NotNil(t, err, "") + + _, err = s.Delete("/", true) + assert.NotNil(t, err, "") + + _, err = s.Create("/", "", false, Permanent) + assert.NotNil(t, err, "") + + _, err = s.Update("/", "", Permanent) + assert.NotNil(t, err, "") + + _, err = s.CompareAndSwap("/", "", 0, "", Permanent) + assert.NotNil(t, err, "") + +} + // Ensure that the store cannot delete a directory if recursive is not specified. func TestStoreDeleteDiretoryFailsIfNonRecursive(t *testing.T) { s := newStore() From 72bf216cb47504309e37f05ee94469e9073516ae Mon Sep 17 00:00:00 2001 From: Brandon Philips Date: Mon, 2 Dec 2013 18:20:11 -0800 Subject: [PATCH 14/18] fix(server/v2): redirect to ClientURL not PeerURL If consistent is set you must redirect the client to the leader's ClientURL not the PeerURL. --- server/server.go | 5 +++++ server/v2/get_handler.go | 2 +- server/v2/v2.go | 1 + tests/mock/server_v2.go | 5 +++++ 4 files changed, 12 insertions(+), 1 deletion(-) diff --git a/server/server.go b/server/server.go index 4f75df2e0..00c39227a 100644 --- a/server/server.go +++ b/server/server.go @@ -96,6 +96,11 @@ func (s *Server) PeerURL(name string) (string, bool) { return s.registry.PeerURL(name) } +// ClientURL retrieves the Client URL for a given node name. +func (s *Server) ClientURL(name string) (string, bool) { + return s.registry.ClientURL(name) +} + // Returns a reference to the Store. func (s *Server) Store() store.Store { return s.store diff --git a/server/v2/get_handler.go b/server/v2/get_handler.go index dd0bdc180..2f48fc32a 100644 --- a/server/v2/get_handler.go +++ b/server/v2/get_handler.go @@ -23,7 +23,7 @@ func GetHandler(w http.ResponseWriter, req *http.Request, s Server) error { // Help client to redirect the request to the current leader if req.FormValue("consistent") == "true" && s.State() != raft.Leader { leader := s.Leader() - hostname, _ := s.PeerURL(leader) + hostname, _ := s.ClientURL(leader) url := hostname + req.URL.Path log.Debugf("Redirect consistent get to %s", url) http.Redirect(w, req, url, http.StatusTemporaryRedirect) diff --git a/server/v2/v2.go b/server/v2/v2.go index 135479d29..0cdfb1cf7 100644 --- a/server/v2/v2.go +++ b/server/v2/v2.go @@ -13,6 +13,7 @@ type Server interface { CommitIndex() uint64 Term() uint64 PeerURL(string) (string, bool) + ClientURL(string) (string, bool) Store() store.Store Dispatch(raft.Command, http.ResponseWriter, *http.Request) error } diff --git a/tests/mock/server_v2.go b/tests/mock/server_v2.go index d48d5c0ef..a4e8821a1 100644 --- a/tests/mock/server_v2.go +++ b/tests/mock/server_v2.go @@ -45,6 +45,11 @@ func (s *ServerV2) PeerURL(name string) (string, bool) { return args.String(0), args.Bool(1) } +func (s *ServerV2) ClientURL(name string) (string, bool) { + args := s.Called(name) + return args.String(0), args.Bool(1) +} + func (s *ServerV2) Store() store.Store { return s.store } From 6729776e25c271dfd7d38d105c951538d02ee8a2 Mon Sep 17 00:00:00 2001 From: Brandon Philips Date: Mon, 2 Dec 2013 18:41:03 -0800 Subject: [PATCH 15/18] fix(server/usage): fixup the usage based on feedback People were getting confused by the lack of explanation on which port to use. Fix this. /cc @polvi --- server/usage.go | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/server/usage.go b/server/usage.go index ff23aac53..3809fb04b 100644 --- a/server/usage.go +++ b/server/usage.go @@ -26,8 +26,9 @@ Options: -vv Enabled very verbose logging. Cluster Configuration Options: - -peers= Comma-separated list of peers (ip + port) in the cluster. - -peers-file= Path to a file containing the peer list. + -peers-file= Path to a file containing the peer list. + -peers=, Comma-separated list of peers. The members + should match the peer's '-peer-addr' flag. Client Communication Options: -addr= The public host:port used for client communication. From aa4660d7801794876cb846678b4f5df8986317bd Mon Sep 17 00:00:00 2001 From: Brandon Philips Date: Sun, 1 Dec 2013 20:41:18 -0800 Subject: [PATCH 16/18] chore(README): add note about curl on OSX Discussion about this issue here: https://groups.google.com/forum/#!topic/munki-dev/oX2xUnoQEi4 --- README.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/README.md b/README.md index 9d38186bd..e8601e104 100644 --- a/README.md +++ b/README.md @@ -394,6 +394,12 @@ curl --cacert ./fixtures/ca/server-chain.pem https://127.0.0.1:4001/v2/keys/foo You should be able to see the handshake succeed. +**OSX 10.9+ Users**: curl 7.30.0 on OSX 10.9+ doesn't understand certificates passed in on the command line. +Instead you must import the dummy ca.crt directly into the keychain or add the `-k` flag to curl to ignore errors. +If you want to test without the `-k` flag run `open ./fixtures/ca/ca.crt` and follow the prompts. +Please remove this certificate after you are done testing! +If you know of a workaround let us know. + ``` ... SSLv3, TLS handshake, Finished (20): From 0dc428b5d68355c24c71f789d09b4c3856238c09 Mon Sep 17 00:00:00 2001 From: Brandon Philips Date: Wed, 4 Dec 2013 12:06:41 -0800 Subject: [PATCH 17/18] fix(README): use the new response format update all of the examples to use the new response format. --- README.md | 40 ++++++++++++++++++++-------------------- 1 file changed, 20 insertions(+), 20 deletions(-) diff --git a/README.md b/README.md index a74363eb4..207a4e7f3 100644 --- a/README.md +++ b/README.md @@ -83,7 +83,7 @@ curl -L http://127.0.0.1:4001/v2/keys/message -X PUT -d value="Hello world" ``` ```json -{"action":"set","key":"/message","value":"Hello world","modifiedIndex":2} +{"action":"set","node":{"key":"/message","value":"Hello world","modifiedIndex":2,"createdIndex":2}} ``` This response contains four fields. @@ -112,7 +112,7 @@ curl -L http://127.0.0.1:4001/v2/keys/message ``` ```json -{"action":"get","key":"/message","value":"Hello world","modifiedIndex":2} +{"action":"get","node":{"key":"/message","value":"Hello world","modifiedIndex":2,"createdIndex":2}} ``` @@ -125,10 +125,10 @@ curl -L http://127.0.0.1:4001/v2/keys/message -XPUT -d value="Hello etcd" ``` ```json -{"action":"set","key":"/message","prevValue":"Hello world","value":"Hello etcd","index":3} +{"action":"set","node":{"key":"/message","prevValue":"Hello world","value":"Hello etcd","modifiedIndex":3,"createdIndex":3}} ``` -Notice that the `prevValue` is set to the previous value of the key - `Hello world`. +Notice that `node.prevValue` is set to the previous value of the key - `Hello world`. It is useful when you want to atomically set a value to a key and get its old value. @@ -141,7 +141,7 @@ curl -L http://127.0.0.1:4001/v2/keys/message -XDELETE ``` ```json -{"action":"delete","key":"/message","prevValue":"Hello etcd","modifiedIndex":4} +{"action":"delete","node":{"key":"/message","prevValue":"Hello etcd","modifiedIndex":4,"createdIndex":3}} ``` @@ -155,7 +155,7 @@ curl -L http://127.0.0.1:4001/v2/keys/foo -XPUT -d value=bar -d ttl=5 ``` ```json -{"action":"set","key":"/foo","value":"bar","expiration":"2013-11-12T20:21:22.629352334-05:00","ttl":5,"modifiedIndex":5} +{"action":"set","node":{"key":"/foo","value":"bar","expiration":"2013-12-04T12:01:21.874888581-08:00","ttl":5,"modifiedIndex":5,"createdIndex":5}} ``` Note the two new fields in response: @@ -201,7 +201,7 @@ curl -L http://127.0.0.1:4001/v2/keys/foo -XPUT -d value=bar The first terminal should get the notification and return with the same response as the set request. ```json -{"action":"set","key":"/foo","value":"bar","modifiedIndex":7} +{"action":"set","node":{"key":"/foo","value":"bar","modifiedIndex":7,"createdIndex":7}} ``` However, the watch command can do more than this. @@ -274,10 +274,10 @@ curl -L http://127.0.0.1:4001/v2/keys/foo?prevValue=one -XPUT -d value=two The response should be ```json -{"action":"compareAndSwap","key":"/foo","prevValue":"one","value":"two","modifiedIndex":9} +{"action":"compareAndSwap","node":{"key":"/foo","prevValue":"one","value":"two","modifiedIndex":9,"createdIndex":8}} ``` -We successfully changed the value from “one” to “two” since we gave the correct previous value. +We successfully changed the value from "one" to "two" since we gave the correct previous value. ### Listing a directory @@ -295,7 +295,7 @@ curl -L http://127.0.0.1:4001/v2/keys/foo_dir/foo -XPUT -d value=bar ``` ```json -{"action":"set","key":"/foo_dir/foo","value":"bar","modifiedIndex":10} +{"action":"set","node":{"key":"/foo_dir/foo","value":"bar","modifiedIndex":2,"createdIndex":2}} ``` Now we can list the keys under root `/`: @@ -307,7 +307,7 @@ curl -L http://127.0.0.1:4001/v2/keys/ We should see the response as an array of items: ```json -{"action":"get","key":"/","dir":true,"kvs":[{"key":"/foo","value":"two","modifiedIndex":9},{"key":"/foo_dir","dir":true,"modifiedIndex":10}],"modifiedIndex":0} +{"action":"get","node":{"key":"/","dir":true,"nodes":[{"key":"/foo_dir","dir":true,"modifiedIndex":2,"createdIndex":2}]}} ``` Here we can see `/foo` is a key-value pair under `/` and `/foo_dir` is a directory. @@ -318,7 +318,7 @@ curl -L http://127.0.0.1:4001/v2/keys/?recursive=true ``` ```json -{"action":"get","key":"/","dir":true,"kvs":[{"key":"/foo","value":"two","modifiedIndex":9},{"key":"/foo_dir","dir":true,"kvs":[{"key":"/foo_dir/foo","value":"bar","modifiedIndex":10}],"modifiedIndex":10}],"modifiedIndex":0} +{"action":"get","node":{"key":"/","dir":true,"nodes":[{"key":"/foo_dir","dir":true,"nodes":[{"key":"/foo_dir/foo","value":"bar","modifiedIndex":2,"createdIndex":2}],"modifiedIndex":2,"createdIndex":2}]}} ``` @@ -333,7 +333,7 @@ curl -L http://127.0.0.1:4001/v2/keys/foo_dir?recursive=true -XDELETE ``` ```json -{"action":"delete","key":"/foo_dir","dir":true,"modifiedIndex":11} +{"action":"delete","node":{"key":"/foo_dir","dir":true,"modifiedIndex":11,"createdIndex":10}} ``` @@ -349,7 +349,7 @@ curl -L http://127.0.0.1:4001/v2/keys/_message -XPUT -d value="Hello hidden worl ``` ```json -{"action":"set","key":"/_message","value":"Hello hidden world","modifiedIndex":12} +{"action":"set","node":{"key":"/_message","value":"Hello hidden world","modifiedIndex":3,"createdIndex":3}} ``` @@ -360,7 +360,7 @@ curl -L http://127.0.0.1:4001/v2/keys/message -XPUT -d value="Hello world" ``` ```json -{"action":"set","key":"/message","value":"Hello world","modifiedIndex":13} +{"action":"set","node":{"key":"/message","value":"Hello world","modifiedIndex":4,"createdIndex":4}} ``` Now let's try to get a listing of keys under the root directory, `/`: @@ -370,7 +370,7 @@ curl -L http://127.0.0.1:4001/v2/keys/ ``` ```json -{"action":"get","key":"/","dir":true,"kvs":[{"key":"/foo","value":"two","modifiedIndex":9},{"key":"/message","value":"Hello world","modifiedIndex":13}],"modifiedIndex":0} +{"action":"get","node":{"key":"/","dir":true,"nodes":[{"key":"/foo_dir","dir":true,"modifiedIndex":2,"createdIndex":2},{"key":"/message","value":"Hello world","modifiedIndex":4,"createdIndex":4}]}} ``` Here we see the `/message` key but our hidden `/_message` key is not returned. @@ -468,7 +468,7 @@ TLS handshake, Finished (20) And also the response from the server: ```json -{"action":"set","key":"/foo","prevValue":"bar","value":"bar","modifiedIndex":3} +{"action":"set","node":{"key":"/foo","prevValue":"two","value":"bar","modifiedIndex":12,"createdIndex":12}} ``` @@ -516,7 +516,7 @@ curl -L http://127.0.0.1:4001/v2/keys/_etcd/machines ``` ```json -[{"action":"get","key":"/_etcd/machines/machine1","value":"raft=http://127.0.0.1:7001\u0026etcd=http://127.0.0.1:4001","index":1},{"action":"get","key":"/_etcd/machines/machine2","value":"raft=http://127.0.0.1:7002\u0026etcd=http://127.0.0.1:4002","index":1},{"action":"get","key":"/_etcd/machines/machine3","value":"raft=http://127.0.0.1:7003\u0026etcd=http://127.0.0.1:4003","index":1}] +{"action":"get","node":{"key":"/_etcd/machines","dir":true,"nodes":[{"key":"/_etcd/machines/machine1","value":"raft=http://127.0.0.1:7001\u0026etcd=http://127.0.0.1:4001","modifiedIndex":1,"createdIndex":1},{"key":"/_etcd/machines/machine2","value":"raft=http://127.0.0.1:7002\u0026etcd=http://127.0.0.1:4002","modifiedIndex":2,"createdIndex":2},{"key":"/_etcd/machines/machine3","value":"raft=http://127.0.0.1:7003\u0026etcd=http://127.0.0.1:4003","modifiedIndex":3,"createdIndex":3}],"modifiedIndex":1,"createdIndex":1}} ``` We can also get the current leader in the cluster: @@ -538,7 +538,7 @@ curl -L http://127.0.0.1:4001/v2/keys/foo -XPUT -d value=bar ``` ```json -{"action":"set","key":"/foo","value":"bar","modifiedIndex":4} +{"action":"set","node":{"key":"/foo","value":"bar","modifiedIndex":4,"createdIndex":4}} ``` @@ -579,7 +579,7 @@ curl -L http://127.0.0.1:4002/v2/keys/foo ``` ```json -{"action":"get","key":"/foo","value":"bar","index":4} +{"action":"get","node":{"key":"/foo","value":"bar","modifiedIndex":4,"createdIndex":4}} ``` From 7ec3f861e4aa6d40c5b06ffe6fed0f9b2d16b677 Mon Sep 17 00:00:00 2001 From: Brandon Philips Date: Wed, 4 Dec 2013 13:48:38 -0800 Subject: [PATCH 18/18] fix(README): prettify json fixed using vim and python -m json.tool --- README.md | 266 +++++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 244 insertions(+), 22 deletions(-) diff --git a/README.md b/README.md index 207a4e7f3..a95da5eec 100644 --- a/README.md +++ b/README.md @@ -83,7 +83,15 @@ curl -L http://127.0.0.1:4001/v2/keys/message -X PUT -d value="Hello world" ``` ```json -{"action":"set","node":{"key":"/message","value":"Hello world","modifiedIndex":2,"createdIndex":2}} +{ + "action": "set", + "node": { + "createdIndex": 2, + "key": "/message", + "modifiedIndex": 2, + "value": "Hello world" + } +} ``` This response contains four fields. @@ -112,7 +120,15 @@ curl -L http://127.0.0.1:4001/v2/keys/message ``` ```json -{"action":"get","node":{"key":"/message","value":"Hello world","modifiedIndex":2,"createdIndex":2}} +{ + "action": "get", + "node": { + "createdIndex": 2, + "key": "/message", + "modifiedIndex": 2, + "value": "Hello world" + } +} ``` @@ -125,7 +141,16 @@ curl -L http://127.0.0.1:4001/v2/keys/message -XPUT -d value="Hello etcd" ``` ```json -{"action":"set","node":{"key":"/message","prevValue":"Hello world","value":"Hello etcd","modifiedIndex":3,"createdIndex":3}} +{ + "action": "set", + "node": { + "createdIndex": 3, + "key": "/message", + "modifiedIndex": 3, + "prevValue": "Hello world", + "value": "Hello etcd" + } +} ``` Notice that `node.prevValue` is set to the previous value of the key - `Hello world`. @@ -141,7 +166,15 @@ curl -L http://127.0.0.1:4001/v2/keys/message -XDELETE ``` ```json -{"action":"delete","node":{"key":"/message","prevValue":"Hello etcd","modifiedIndex":4,"createdIndex":3}} +{ + "action": "delete", + "node": { + "createdIndex": 3, + "key": "/message", + "modifiedIndex": 4, + "prevValue": "Hello etcd" + } +} ``` @@ -155,7 +188,17 @@ curl -L http://127.0.0.1:4001/v2/keys/foo -XPUT -d value=bar -d ttl=5 ``` ```json -{"action":"set","node":{"key":"/foo","value":"bar","expiration":"2013-12-04T12:01:21.874888581-08:00","ttl":5,"modifiedIndex":5,"createdIndex":5}} +{ + "action": "set", + "node": { + "createdIndex": 5, + "expiration": "2013-12-04T12:01:21.874888581-08:00", + "key": "/foo", + "modifiedIndex": 5, + "ttl": 5, + "value": "bar" + } +} ``` Note the two new fields in response: @@ -175,7 +218,12 @@ curl -L http://127.0.0.1:4001/v2/keys/foo If the TTL has expired, the key will be deleted, and you will be returned a 100. ```json -{"errorCode":100,"message":"Key Not Found","cause":"/foo","index":6} +{ + "cause": "/foo", + "errorCode": 100, + "index": 6, + "message": "Key Not Found" +} ``` @@ -201,7 +249,15 @@ curl -L http://127.0.0.1:4001/v2/keys/foo -XPUT -d value=bar The first terminal should get the notification and return with the same response as the set request. ```json -{"action":"set","node":{"key":"/foo","value":"bar","modifiedIndex":7,"createdIndex":7}} +{ + "action": "set", + "node": { + "createdIndex": 7, + "key": "/foo", + "modifiedIndex": 7, + "value": "bar" + } +} ``` However, the watch command can do more than this. @@ -248,7 +304,12 @@ curl -L http://127.0.0.1:4001/v2/keys/foo?prevExist=false -XPUT -d value=three The error code explains the problem: ```json -{"errorCode":105,"message":"Already exists","cause":"/foo","index":39776} +{ + "cause": "/foo", + "errorCode": 105, + "index": 39776, + "message": "Already exists" +} ``` Now lets provide a `prevValue` parameter: @@ -260,7 +321,12 @@ curl -L http://127.0.0.1:4001/v2/keys/foo?prevValue=two -XPUT -d value=three This will try to compare the previous value of the key and the previous value we provided. If they are equal, the value of the key will change to three. ```json -{"errorCode":101,"message":"Test Failed","cause":"[two != one] [0 != 8]","index":8} +{ + "cause": "[two != one] [0 != 8]", + "errorCode": 101, + "index": 8, + "message": "Test Failed" +} ``` which means `CompareAndSwap` failed. @@ -274,7 +340,16 @@ curl -L http://127.0.0.1:4001/v2/keys/foo?prevValue=one -XPUT -d value=two The response should be ```json -{"action":"compareAndSwap","node":{"key":"/foo","prevValue":"one","value":"two","modifiedIndex":9,"createdIndex":8}} +{ + "action": "compareAndSwap", + "node": { + "createdIndex": 8, + "key": "/foo", + "modifiedIndex": 9, + "prevValue": "one", + "value": "two" + } +} ``` We successfully changed the value from "one" to "two" since we gave the correct previous value. @@ -295,7 +370,15 @@ curl -L http://127.0.0.1:4001/v2/keys/foo_dir/foo -XPUT -d value=bar ``` ```json -{"action":"set","node":{"key":"/foo_dir/foo","value":"bar","modifiedIndex":2,"createdIndex":2}} +{ + "action": "set", + "node": { + "createdIndex": 2, + "key": "/foo_dir/foo", + "modifiedIndex": 2, + "value": "bar" + } +} ``` Now we can list the keys under root `/`: @@ -307,7 +390,21 @@ curl -L http://127.0.0.1:4001/v2/keys/ We should see the response as an array of items: ```json -{"action":"get","node":{"key":"/","dir":true,"nodes":[{"key":"/foo_dir","dir":true,"modifiedIndex":2,"createdIndex":2}]}} +{ + "action": "get", + "node": { + "dir": true, + "key": "/", + "nodes": [ + { + "createdIndex": 2, + "dir": true, + "key": "/foo_dir", + "modifiedIndex": 2 + } + ] + } +} ``` Here we can see `/foo` is a key-value pair under `/` and `/foo_dir` is a directory. @@ -318,7 +415,29 @@ curl -L http://127.0.0.1:4001/v2/keys/?recursive=true ``` ```json -{"action":"get","node":{"key":"/","dir":true,"nodes":[{"key":"/foo_dir","dir":true,"nodes":[{"key":"/foo_dir/foo","value":"bar","modifiedIndex":2,"createdIndex":2}],"modifiedIndex":2,"createdIndex":2}]}} +{ + "action": "get", + "node": { + "dir": true, + "key": "/", + "nodes": [ + { + "createdIndex": 2, + "dir": true, + "key": "/foo_dir", + "modifiedIndex": 2, + "nodes": [ + { + "createdIndex": 2, + "key": "/foo_dir/foo", + "modifiedIndex": 2, + "value": "bar" + } + ] + } + ] + } +} ``` @@ -333,7 +452,15 @@ curl -L http://127.0.0.1:4001/v2/keys/foo_dir?recursive=true -XDELETE ``` ```json -{"action":"delete","node":{"key":"/foo_dir","dir":true,"modifiedIndex":11,"createdIndex":10}} +{ + "action": "delete", + "node": { + "createdIndex": 10, + "dir": true, + "key": "/foo_dir", + "modifiedIndex": 11 + } +} ``` @@ -349,7 +476,15 @@ curl -L http://127.0.0.1:4001/v2/keys/_message -XPUT -d value="Hello hidden worl ``` ```json -{"action":"set","node":{"key":"/_message","value":"Hello hidden world","modifiedIndex":3,"createdIndex":3}} +{ + "action": "set", + "node": { + "createdIndex": 3, + "key": "/_message", + "modifiedIndex": 3, + "value": "Hello hidden world" + } +} ``` @@ -360,7 +495,15 @@ curl -L http://127.0.0.1:4001/v2/keys/message -XPUT -d value="Hello world" ``` ```json -{"action":"set","node":{"key":"/message","value":"Hello world","modifiedIndex":4,"createdIndex":4}} +{ + "action": "set", + "node": { + "createdIndex": 4, + "key": "/message", + "modifiedIndex": 4, + "value": "Hello world" + } +} ``` Now let's try to get a listing of keys under the root directory, `/`: @@ -370,7 +513,27 @@ curl -L http://127.0.0.1:4001/v2/keys/ ``` ```json -{"action":"get","node":{"key":"/","dir":true,"nodes":[{"key":"/foo_dir","dir":true,"modifiedIndex":2,"createdIndex":2},{"key":"/message","value":"Hello world","modifiedIndex":4,"createdIndex":4}]}} +{ + "action": "get", + "node": { + "dir": true, + "key": "/", + "nodes": [ + { + "createdIndex": 2, + "dir": true, + "key": "/foo_dir", + "modifiedIndex": 2 + }, + { + "createdIndex": 4, + "key": "/message", + "modifiedIndex": 4, + "value": "Hello world" + } + ] + } +} ``` Here we see the `/message` key but our hidden `/_message` key is not returned. @@ -421,7 +584,13 @@ SSLv3, TLS handshake, Finished (20): And also the response from the etcd server: ```json -{"action":"set","key":"/foo","prevValue":"bar","value":"bar","modifiedIndex":3} +{ + "action": "set", + "key": "/foo", + "modifiedIndex": 3, + "prevValue": "bar", + "value": "bar" +} ``` @@ -468,7 +637,16 @@ TLS handshake, Finished (20) And also the response from the server: ```json -{"action":"set","node":{"key":"/foo","prevValue":"two","value":"bar","modifiedIndex":12,"createdIndex":12}} +{ + "action": "set", + "node": { + "createdIndex": 12, + "key": "/foo", + "modifiedIndex": 12, + "prevValue": "two", + "value": "bar" + } +} ``` @@ -516,7 +694,35 @@ curl -L http://127.0.0.1:4001/v2/keys/_etcd/machines ``` ```json -{"action":"get","node":{"key":"/_etcd/machines","dir":true,"nodes":[{"key":"/_etcd/machines/machine1","value":"raft=http://127.0.0.1:7001\u0026etcd=http://127.0.0.1:4001","modifiedIndex":1,"createdIndex":1},{"key":"/_etcd/machines/machine2","value":"raft=http://127.0.0.1:7002\u0026etcd=http://127.0.0.1:4002","modifiedIndex":2,"createdIndex":2},{"key":"/_etcd/machines/machine3","value":"raft=http://127.0.0.1:7003\u0026etcd=http://127.0.0.1:4003","modifiedIndex":3,"createdIndex":3}],"modifiedIndex":1,"createdIndex":1}} +{ + "action": "get", + "node": { + "createdIndex": 1, + "dir": true, + "key": "/_etcd/machines", + "modifiedIndex": 1, + "nodes": [ + { + "createdIndex": 1, + "key": "/_etcd/machines/machine1", + "modifiedIndex": 1, + "value": "raft=http://127.0.0.1:7001&etcd=http://127.0.0.1:4001" + }, + { + "createdIndex": 2, + "key": "/_etcd/machines/machine2", + "modifiedIndex": 2, + "value": "raft=http://127.0.0.1:7002&etcd=http://127.0.0.1:4002" + }, + { + "createdIndex": 3, + "key": "/_etcd/machines/machine3", + "modifiedIndex": 3, + "value": "raft=http://127.0.0.1:7003&etcd=http://127.0.0.1:4003" + } + ] + } +} ``` We can also get the current leader in the cluster: @@ -538,7 +744,15 @@ curl -L http://127.0.0.1:4001/v2/keys/foo -XPUT -d value=bar ``` ```json -{"action":"set","node":{"key":"/foo","value":"bar","modifiedIndex":4,"createdIndex":4}} +{ + "action": "set", + "node": { + "createdIndex": 4, + "key": "/foo", + "modifiedIndex": 4, + "value": "bar" + } +} ``` @@ -579,7 +793,15 @@ curl -L http://127.0.0.1:4002/v2/keys/foo ``` ```json -{"action":"get","node":{"key":"/foo","value":"bar","modifiedIndex":4,"createdIndex":4}} +{ + "action": "get", + "node": { + "createdIndex": 4, + "key": "/foo", + "modifiedIndex": 4, + "value": "bar" + } +} ```