Wednesday, August 31, 2016

Scheduled Heroku Dyno restarts

Heroku Dynos get restarted once a day.
The cycling happens once every 24 hours (plus up to 216 random minutes, to prevent every dyno for an application from restarting at the same time). Manual restarts (heroku ps:restart) and releases (deploys or changing config vars) will reset this 24 hour period.
This is fine if you are running multiple dynos. But if you were running an app you didn't want to cycle during the day but only had one dyno (I know, I know, don't ask...) there's a quick hack you can use to instead control the time of restart so that it's done during off hours (3am or whatever works for you app's usage patterns).

You can use the Heroku API to perform a restart like below using a simple curl call. If you schedule that curl call for once-daily execution using the Heroku Scheduler add-on then you can have a reliable restart of of your app and a 24 hour runway of no unexpected cycling.

Here's the details:

# install the oauth plugin
$ heroku plugins:install heroku-cli-oauth

# create a new authorization
$ heroku authorizations:create -d "Platform API example token"
Creating OAuth Authorization... done
Client:      <none>
ID:          830d64b9-fd52-4d95-<redacted>
Description: Platform API example token
Scope:       global
Token:       1d82fc23-<redacted>

# use the new authorization token from above to interact with the API, e.g.
$ curl https://api.heroku.com/apps/ -H "Accept: application/vnd.heroku+json; version=3" -H "Authorization: Bearer 1d82fc23-<redacted>"

# so to restart dynos for an app
$ curl -X DELETE https://api.heroku.com/apps/kountdown/dynos -H "Content-Type:application/json" -H "Accept: application/vnd.heroku+json; version=3" -H "Authorization: Bearer 1d82fc23-<redacted>"

So just stick that curl command into the Heroku Scheduler for the time you want and Bob's your uncle.

More on the Heroku API can be found here.

4 comments:

  1. Thanks for the fix, I've been running an app on heroku for 8 months without realizing it was restarting every 24 hoursish - until I saw an error on a client application today caused by the minute of downtime and then a tiny little dot on the heroku metrics that says "Daily Restart".

    ReplyDelete
  2. The solution is perfectly fine in theory (e.g. to cycle a dyno at a specific time) if you just want a simple way out (be it temporarily or more permanently). However, worth pointing out is that the API key is connected to your Heroku account and has full rights to do pretty much whatever it wants with any app that account has access to. Therefore, you should most definitely not put it in a shared app (e.g. in its config vars or in any of its addons, such as Heroku Scheduler). Only in personal apps.

    ReplyDelete
  3. Do you also know how to kill all the stuck DB connections that may occur during a restart using a curl command. I can do it with the CLI using heroku pg:killall but I want to automate this.

    ReplyDelete