Backend scaling

I have my backend configured on Digital ocean and it’s pointing to a Tag. So this means that the node will redirect the traffic to any backend that matches that tag.

We have peeks on our website and we know when they happen, and when they do happen we resize our “backend” to be able to respond to the higher demand.

Since the Cloud backend configuration only allows us to do it on a TAG, then we cannot exclude the backend when we resize them.

What will happen when we resize our backend, if we don’t change the tag before shutting it down, I would imagine that if there is a connection to it they would have some error on the website?

Do we need to change the tag on the backend we want to resize before we do it? Or will the ADC quickly detect the backend is down and just redirect to the other backend?

Hi @pat74

You’re correct as far as Nova’s behaviour goes - targeting a specific tag will make no exceptions, so when scaling down your backend it’s reasonable to assume that the tag will still be used.

That said, the health checks performed on each backend server are quite frequent - expect <5 seconds per check. As a result, a failed health check will have a given server marked as stale quite quickly, and will route further requests between the remaining servers. You may wish to benchmark this yourself, but I’d feel confident scaling as necessary without much risk of failed requests.

However, if you need to be absolutely sure that no requests will be routed to a server that’s due to be scaled down, I’d recommend removing the tag in question, waiting until the server is no longer listed in the Nova Dashboard and then shutting down accordingly.

What behaviour should I expect when a backend is marked as Stale?
Over the weekend I had the servers running at 100% because of an unexpected level of traffic and the CPU usage when 100%. The server where returning 502 Gateway error as they could not respond to demand.
Would that mark the server as STALE?

With DigitalOcean load balancer the server doesn’t seem to get marked as stale in their load balancer and when this error happens you can see the page switching between 502 error and sometimes letting connection going through for some page request if you are lucky. So the server doesn’t appear completely down.

But from what I see with NOVA, the servers get marked as down and no traffic seems to get through until the ADC decides they are back online, is that what happens?

It could also be because of the way I’m using Cloudflare, because really what I saw when it was configured on the Flatten Cname record (With Nova Destination), was seeing a Cloudflare error, which looks much nicer than the default 502 Gateway.
When I’m connected with the DO load balancer I’m using an A record with IP address so the request doesn’t go through Cloudflare anymore.

Perhaps this explains the different behaviour have been experiencing with Nova service?

Yes, that’s what I decided to do. But I’m not using the Cloud service anymore. I’m now using a simple backend configuration, that way I can choose the IP being used and I can remove the IP address before rescaling servers. I prefer to make sure there is no more connection on it instead of shutting it down suddenly.

This is effectively it. Perhaps bad terminology on my part, but this is when health checks for your backend(s) fail as configured. At this point whichever backend server fails will be marked as offline and taken out of rotation.

What seems likely here is that your configured health check interval and healthy/unhealthy thresholds are at play. I’m not sure of your particular use-case, but it’s likely that only after x failures as configured, will the backend server in question be taken out of rotation. With Nova this is near-instant.

This seems possible, though I can’t say for sure. For what it’s worth though, you can also use Cloudflare’s CNAME flattening without having Cloudflare proxy your content in the middle.

Good to hear - this does seem more suitable for your use-case, potentially. In case this suits you, you can make use of Nova’s REST API to modify your backends (and therefore add/remove individual servers) in a programmatic way, before shutting down the associated instance. Documentation is available here, and effectively will consist of each individual backend server configured as part of a multipart form. This way you update the backend with a comprehensive list of backend servers at every update