OpenStack Object Storage (swift)

What happens if a device is completely out of disk space?

Asked by Rob Janney

Will data destined for partitions on this device fail to be stored? Or will the data be stored to another partition on another device with available space?

Is there any manual procedure necessary for recovering from this condition? (aside from http delete calls to reclaim space before more objects can be stored)

Is there a way to set a threshold to stop storing at 1%/etc disk free or does the system already do this internally?

Question information

Language:
English Edit question
Status:
Solved
For:
OpenStack Object Storage (swift) Edit question
Assignee:
No assignee Edit question
Solved by:
clayg
Solved:
Last query:
Last reply:
Revision history for this message
Best clayg (clay-gerrard) said : 		#1

Handoffs will ensure the data is stored. An object server out of space will respond to a PUT with a 507. As one long as the object can get written to at least ONE of the primary replica nodes - the condition should be transparent to the client and data durability won't get compromised. The handoff node will keep trying to replicate the partition off of itself tho..

Ultimately you'll need to rebalance the ring (http://swift.openstack.org/admin_guide.html#managing-the-rings). The specific device(s) that are over-loaded will need to have their 'weight' dropped down and the partitions will get shuffled around. Once you push out the new ring the "full" devices will start to replicate partitions that no longer belong there off to other devices.

There is currently no mechanism for setting a threshold that would automatically make changes to the ring.

Revision history for this message
Jay Payne (letterj) said : 		#2

As a best practice, it is recommended that when your cluster gets 80% full you add additional storage. How full did your entire cluster get?

Revision history for this message
Rob Janney (rob-janney) said : 		#3

Thanks! That answers my question. We definitely will be re-balancing & trying to prevent this from happening.

@Jay It hasn't gotten over 80% full. I was just curious about the failure mode in the event of this happening.

Revision history for this message
Rob Janney (rob-janney) said : 		#4

Thanks clayg, that solved my question.