cassandra rack replication fixed to (dead) nodes [RF and CL confusion] -

• 1 cluster, 3 nodes, in 2 physical locations, grouped in 2 racks
• rf 2
• propertyfilesnitch
• cl quorum

the problem is:

first node's (in rac1) replication pointed third node rac2 , not change if node down, reads , writes fails.

if start third node , shut down second node, reads , writes works.

both second , third node replicate first node, , if first node down, reads , writes fails.

the question is:

is possible make automatically detect dead nodes , point replication active detected nodes?

• if first node down, second , third node replicate data between each other
• if second or third node down, first node should detect active , replicate it

update1:

made tests:

• shut down first node - reads second , third node fails (unable complete request: 1 or more nodes unavailable.)
• shut down second node - reads first , third node works
• shut down third node - reads first , second node fails

very strange ...

update2:

i think found answer. how now: 3 nodes, rf 2, writes , reads has cl 2. if 1 replica down, reads , writes fails (i tested selecting different keys, succeeded when 1 node down, , fails when down)

now thinking this: move nodes 1 rack, change rf 3, reads , writes use cl 2 (two replications required write succeed, , third made in background). there 3 replicas, if 1 fails, cl 2 succeed anyway.

am right?

will writes succeed if there 2 nodes active, replication factor 3, , consistency level current write operation 2?

update3:

yes, think i'm on right way. same question here

from screenshot can assumed opscenter.

in opscenter there special feature called alerts. in detecting dead node.

now coming question of node down & read write fails, these thing depends in read/write consistency level. go through consistency levels, able find out solution on own.

update:

may might find blog interesting. time cassandra fail when few replicas alive when coordinator receives request. might reason behind strange situation