Debugging a cf push ‘Failed setting cpu shares’ Stager Error

When I was pushing an app to my CF in AWS, the following staging error occurred.

2016-08-23T16:12:13.59+0000 [DEA/0] OUT Got staging request for app with id 71e82161-f1e8-408f-bd0f-fe8701284e83
2016-08-23T16:12:13.64+0000 [API/0] OUT Updated app with guid 71e82161-f1e8-408f-bd0f-fe8701284e83 ({"state"=>"STARTED"})
2016-08-23T16:12:16.05+0000 [API/0] ERR Encountered error: Stager error: failed to stage application:
2016-08-23T16:12:16.05+0000 [API/0] ERR Failed setting cpu shares: No such file or directory @ rb_sysopen - /tmp/warden/cgroup/cpu/instance-19qntmnpnj1/cpu.shares

I ran bosh ssh to access one of my runner VMs and ran sudo su - and then ls /tmp/warden/cgroup to see what was going on there. To my surprise, nothing was listed in the output! I had no idea where to even start debugging this error since it looked like something did not go right inside warden itself which I wasn’t familiar with. I then asked for help in the CF slack #runtime-org channel, Dan Lavine and Sandy Cash were very helpful, they pointed me in the right direction to look in cloudfoundry’s [warden/root/linux/setup.h][cloudfoundry/warden/root/linux/setup.h].

After digging into the code, I figured out that the problem is due to the control group fs not being initialized as expected after a VM restart. The reason I was stopping and starting the VMs in my IaaS provider (AWS in my case) was because I was helping save costs while working by turning the VMs off at the end of the workday and back on when I started work the next day, I guess most people don’t do this 🙂

I appreciate that we can get help from the CF community and thank Dan Lavine and Sandy Cash again for their help and also for merging my pull request!

Regarding the fix for this bug, please see the [Pull Request][Pull Request] for details.

中文版请见:[调试发布APP到Cloud Foundry过程中的’Failed setting cpu shares’ Stager Error][调试发布APP到Cloud Foundry过程中的’Failed setting cpu shares’ Stager Error].
[Pull Request]:
https://github.com/cloudfoundry/warden/pull/111
[cloudfoundry/warden/root/linux/setup.h]:
https://github.com/cloudfoundry/warden/blob/master/warden/root/linux/setup.sh
[调试发布APP到Cloud Foundry过程中的’Failed setting cpu shares’ Stager Error]:
https://www.starkandwayne.com//blog/diao-shi-fa-bu-appdao-cloud-foundryguo-cheng-zhong-de-failed-setting-cpu-shares-stager-error/

Spread the word

twitter icon facebook icon linkedin icon