Haproxy-Health checking


Before performing traffic forwarding to servers, it is recommended to tell HAProxy to check the health of the service hosted by each servers of a farm.

This technic is called health checking or health check.

NOTE

it is important to be able to set up a health check as close as possible to the application being load-balanced

Server’s operational state

From an HAProxy point of view, a server can have the different operational states.

A server state is defined by the latest check results. The operational states are:

  • UP: server is operational
  • UP - transitionally DOWN: server is currently considered as operational, but last check failed. So server is in transition to the DOWN state
  • DOWN - transitionally UP: server is currently considered as inoperational, but last check succed. So server is in transition to the UP state
  • DOWN: server is inoperational

The operational state changes are triggered by the check’s parameters.

The picture below shows the different server states and how HAProxy parameters interact with them:

https://cdn.vietnamhost.vn/images/news/haproxy_server_state_checks.png

NOTE

For more information about HAProxy‘s check parameters, read below.

Creating a health check

The minimum configuration for a health check is the check keyword on a server line.

That said, in order to be ready to run, a check needs at least an IP address and a TCP port. These information are provided by the server itself.

IP address, port and protocol

In HAProxy, all the server configured in a backend will get the same health check and must answer the same type of response.

To be performed, a check needs at least the following information:

InformationDefault valueAlternative value
IP address Server’s IP address Address provided by server’s addr parameter
TCP port Server’s port TCP port provided by server’s port parameter
Proxy protocol Server’s send-proxyparameter follow flag check-send-proxy from the server line
ciphered connection Server’s ssl parameter follow flag check-ssl from the server line
Protocol TCP handshake SSL, HTTP, LDAP, MySql, PgSQL, redis, SMTP, generic Send/Expect
valid response depends on the Protocol in use

NOTE

When set, addr and port parameters have precedence over the own server’s IP and port

Check parameters

Check Interval

All the keword below applies to the server or default-server directives and can be used to setup healch check frequency base on server’s state:

  • inter parameter sets the interval between two consecutive health checks. If not specified, the default value is 2s.
  • fastinter parameter sets the interval between two consecutive health checks when the server is any of the transition state (read above):UP - transitionally DOWN or DOWN - transitionally UP. If not set, then inter is used.
  • downinter parameter sets the interval between two consecutive health checks when the server is un the DOWN state. If not set, theninter is used.

Initial check (startup)

All the keword below applies to the global section and can be used to setup HAProxy‘s behavior when running the first check after starting up:

  • max-spread-checks <delay> : When starting up, HAProxy spread the first health checks of a farm over the inter period. When interis very long, some server may appear UP after a very long time. This parameters allows to reduce this initial period to <delay>.
  • spread-checks <0..50, in percent> : Add some randomness on the interval delay between two consecutive checks in order to avoid sending health check at a too much regular interval. Default value, if not set, is 0.

Check buffer size

Some health check may need to look for data in the response body. By default, HAProxy reads only the first 16384 bytes of the response and ignore the rest. For some reason, if you want to search an information above this value, you have to setup the tune.chksize in the globalsection.

Response validation

All the keword below applies to the server or default-server directives and can be used to teach HAProxy how many positive or negative successful checks must be considered before changing a server’s state:

  • rise <count> : number of consecutive valid health checks before considering the server as UP. Default value is 2
  • fall : number of consecutive invalid health checks before considering the server as DOWN. Default value is 3

Check timeouts

All the keword below applies to the backend or defaults sections and can be used to teach HAProxy how long to wait for a server response:

timeout check <timeout> : time let to the server to answer the check. If both inter and timeout check are set, then the smallest value of both of them is used, after the TCP connection is established

NOTE

When timeout check is not set, inter starts with the establishement of the TCP connection

The picture below describes which settings is used depending on the configuration:

../_images/haproxy_checks_timeouts.png

Logging health check

By default, HAProxy logs only health checks triggering a state change from UP to DOWN.

It is possible to log any change of the check status or the server’s health by enabling the directive option log-health-checks in the backendor defaults sections.

That way it becomes possible to know that a server was failing occasional checks before crashing, or exactly when it failed to respond a valid HTTP status, then when the port started to reject connections, then when the server stopped responding at all.

Checking a TCP port

To enable TCP handshake, you need the following parameters:

  • In the backend section:
    • option tcp-check (optional) : this setting is not needed, but it makes the TCP handshake check more obvious
  • on the server line:
    • check : enables health checking
    • a TCP port on configured on the server IP or the port parameter

The check is valid if the server answers a SYN/ACK packet.

Examples:

  1. Perform a TCP handshake on port 80, even if we load-balance port 443:

    backend be_myapp
 [...]
 option tcp-check
 server srv1 10.0.0.1:443 check port 80
 server srv2 10.0.0.2:443 check port 80

Checking a SSL port

To enable SSL handshake, you need the following parameters:

  • In the backend section:
    • option ssl-hello-chk : Sends a SSLv3 client hello message
  • on the server line:
    • check : enables health checking
    • a TCP port on configured on the server IP or the port parameter

The check is valid if the server answers a valid SSL server hello message.

Examples:

  1. Send a SSL HELLO handshake every 10s, consider 2 unsuccessful checks to set server DOWN, log any error occuring:

    backend be_myapp
 [...]
 option ssl-hello-chk
 option log-health-checks
 default-server inter 10s fall 2
 server srv1 10.0.0.1:443 check
 server srv2 10.0.0.2:443 check

Checking a HTTP service

HAProxy can send a HTTP request and analyse the response’s status and/or body to validate the status service:

  • In the backend section, the following directive are available:

    • option httpchk [<method>] [<uri>] [<version>] (mandatory)

      • [<method>] (optional): HTTP method used when building the HTTP request. If not set, then OPTIONS is used.

      • [<uri>] (optional, requires <method> ): set the HTTP request URI. If not set, then / is used. Query strings are allowed.

      • [<version>] (optional, requires <uri> ): set the HTTP version protocol of the request. If not set, HTTP/1.0 is used.

        NOTE

        It is possible to send HTTP headers after the <version> string by concatenating them using rn and backslashes spaces. This is useful to send Host headers when probing a virtual host

    • http-check disable-on-404 : turn the server in administrative status Maintenance when the server answers a 404 to the health check. This option has precedence over the http-check expect one. It is useful to turn a server in soft-stop mode and can be triggered at the server layer, without changing anything in HAProxy.

    • http-check expect [!] <match> <pattern> : change the response validation behavior by setting up a match rule:

      • [!] : negate the match result
      • <match> : matching method used to look for <pattern> in the response. Accepted match method are:
        • status : raw status code comparison
        • rstatus : regex match on status code
        • string : raw sting match in the response’s body
        • rstring : regex match in the response’s body
      • <pattern> : raw information or regex used by the <match> method

    • http-check send-state : add a X-Haproxy-Server-State HTTP header in the request. This header contains the following information, separated by semi-colons:

      1. Server’s operational status (UP, DOWN, NOLB) followed by transitional status (number of failed or successfull checks)
      2. string name followed by the backend and server names separated by a slash ‘/’
      3. string node followed by the name of the HAProxy node
      4. string weight followed by server’s weight, then ‘/’, then sum of weight of availble servers in the farm
      5. string scur followed by the number of connections on the server, then ‘/’ then the total number of connections on the backend
      6. string qcur followed by the number of requests currently on the server’s queue at HAProxy layer

      Example of a send-state line:

      X-Haproxy-Server-State: UP 2/3; name=bck/srv2; node=lb1; weight=1/2; scur=13/22; qcur=0

  • on the server line:

    • check : enables health checking
    • a TCP port on configured on the server IP or the port parameter

The check is valid if the server answers with a status code of 2xx or 3xx, unless this behavior is changed by the directive http-check expect.

Examples:

  1. Simplest HTTP health check on a web server:

    backend bk_myapp
 [...]
 option httpchk
 server srv1 10.0.0.1:80 check
 server srv1 10.0.0.1:80 check

  2. Equivalent of the configuration above, but showing all default options (configuration more obvious):

    backend bk_myapp
 [...]
 option httpchk OPTIONS / HTTP/1.0
 http-check expect rstatus (2|3)[0-9][0-9]
 default-server inter 3s fall 3 rise 2
 server srv1 10.0.0.1:80 check
 server srv1 10.0.0.1:80 check

  3. send the request “get /check” and consider only status code 200 as valid:

    backend bk_myapp
 [...]
 option httpchk get /check
 http-check expect status 200
 default-server inter 3s fall 3 rise 2
 server srv1 10.0.0.1:80 check
 server srv1 10.0.0.1:80 check

  4. send the request “get /check” and consider all statuses as valid bu 5xx:

    backend bk_myapp
 [...]
 option httpchk get /check
 http-check expect ! rstatus ^5
 default-server inter 3s fall 3 rise 2
 server srv1 10.0.0.1:80 check
 server srv1 10.0.0.1:80 check

  5. Send the request “get /check” and look for the keywork OK in the response’s body. It may happen after 20K bytes:

    global
 tune.chksize 32768

[...]

backend bk_myapp
 [...]
 option httpchk get /check
 http-check expect string OK
 default-server inter 3s fall 3 rise 2
 server srv1 10.0.0.1:80 check
 server srv1 10.0.0.1:80 check

  6. send the request “get /check” to the virtual host www.domain.com:

    backend bk_myapp
 [...]
 option httpchk get /check HTTP/1.0\r\nHost:\ www.domain.com
 default-server inter 3s fall 3 rise 2
 server srv1 10.0.0.1:80 check
 server srv1 10.0.0.1:80 check

  7. send the request “get /check” to the virtual host www.domain.com and consider 404 as maintenance mode:

    backend bk_myapp
 [...]
 option httpchk get /check HTTP/1.0\r\nHost:\ www.domain.com
 http-check disable-on-404
 default-server inter 3s fall 3 rise 2
 server srv1 10.0.0.1:80 check
 server srv1 10.0.0.1:80 check

  8. send the request “get /check” to the virtual host www.domain.com of a TCP based farm load-balancing TLS traffic (HAProxy does not handle SSL in this mode):

    backend bk_myapp
 mode tcp
 [...]
 option httpchk get /check HTTP/1.0\r\nHost:\ www.domain.com
 default-server inter 3s fall 3 rise 2
 server srv1 10.0.0.1:443 check check-ssl
 server srv1 10.0.0.1:443 check check-ssl

  9. send the request “get /check” to the virtual host www.domain.com of a HTTP farm (HAProxy ciphers the traffic to the server in this mode):

    backend bk_myapp
 mode http
 [...]
 option httpchk get /check HTTP/1.0\r\nHost:\ www.domain.com
 default-server inter 3s fall 3 rise 2
 server srv1 10.0.0.1:443 ssl check
 server srv1 10.0.0.1:443 ssl check

Checking a LDAP service

HAProxy can perform an anonymous simple LDAPv3 bind checks

  • In the backend section:
    • option ldap-check : Turns on LDAPv3 check
  • on the server line:
    • check : enables health checking
    • a TCP port on configured on the server IP or the port parameter

The check is valid if the server response contains a successful resultCode.

NOTE

The LDAP servers must be configured accordingly to this check (allow anonymous binding). This can be done an IP alias on the server side and allowing only HAProxy IP addresses to bind to it.

NOTE

For more complicated and advanced LDAP checks, please read the the section generic Send/Expect below.

Examples:

  1. Check a LDAP service on a server’s alternative IP address:

    backend be_myapp
 [...]
 option ldap-check
 server srv1 10.0.0.1:389 check addr 10.0.0.11
 server srv2 10.0.0.2:389 check addr 10.0.0.12

  2. Check a LDAPs service on a server’s alternative IP address:

    backend be_myapp
 [...]
 option ldap-check
 server srv1 10.0.0.1:636 check check-ssl addr 10.0.0.11
 server srv2 10.0.0.2:636 check check-ssl addr 10.0.0.12

Checking a MySql service

NOTE

this check is compatible with MySql server 3.22 and later.

HAProxy can perform a simple MySql check, either checking the MySql Handshake packet or testing a full Client Authentication test:

  • In the backend section:

    • option mysql-check [user <username>] [post-41] : Turns on mysql check

      • without the user option, a MySql Handshake is performed
      • user <username> (optional) : perform a Client Authentication check, using <username>

      This requires an update into the MySql servers, like shown below, using mysql client software:

      USE mysql;
INSERT INTO user (Host,User) values ('<ip_of_haproxy>','<username>');
FLUSH PRIVILEGES;
      • post-41 (optional) : Send checks compatible with MySql server 4.1 and later

NOTE

If checks over a MySql server are too frequent and/or if there is too much traffic. MySql server might block HAProxy. Issue a FLUSH HOSTSon the mysql server to clear it up.

  • on the server line:
    • check : enables health checking
    • a TCP port on configured on the server IP or the port parameter

The check is valid if the server response contains a successful resultCode.

NOTE

For more complicated and advanced MySql checks, please read the the section generic Send/Expect below.

Examples:

  1. Check a MySql service:

    backend be_myapp
 [...]
 option mysql-check
 server srv1 10.0.0.1:3306 check
 server srv2 10.0.0.2:3306 check

  2. Check a MySql service using the Client Authentication method for a MySql 5.1 server:

    backend be_myapp
 [...]
 option mysql-check user haproxy post-41
 server srv1 10.0.0.1:3306 check
 server srv2 10.0.0.2:3306 check

Checking a PgSQL service

HAProxy can perform a simple PostgreSQL check, by sending a StartupMessage.

  • In the backend section:
    • option pgsql-check [user <username>] : Turns on PostgreSQL check
      • user <username> (optional) : perform a Client Authentication check, using <username>
  • on the server line:
    • check : enables health checking
    • a TCP port on configured on the server IP or the port parameter

The check is valid if the server response contains a Authentication request.

NOTE

For more complicated and advanced PgSQL checks, please read the the section generic Send/Expect below.

Examples:

  1. Check a PgSQL service:

    backend be_pgsql
 [...]
 option pgsql-check
 server srv1 10.0.0.1:5432 check
 server srv2 10.0.0.2:5432 check

  2. Check a PgSql service using the Client Authentication method:

    backend be_pgsql
 [...]
 option pgsql-check user haproxy
 server srv1 10.0.0.1:5432 check
 server srv2 10.0.0.2:5432 check

Checking a redis service

HAProxy can perform a redis health check by sending the PING command:

  • In the backend section:
    • option redis-check : Turns on redis check
  • on the server line:
    • check : enables health checking
    • a TCP port on configured on the server IP or the port parameter

The check is valid if the server response contains the string +PONG.

NOTE

For more complicated and advanced redis checks, please read the the section generic Send/Expect below.

Examples:

  1. Check a redis service:

    backend be_redis
 [...]
 option redis-check
 server srv1 10.0.0.1:6379 check
 server srv2 10.0.0.2:6379 check

Checking a SMTP service

HAProxy can perform a SMTP health check:

  • In the backend section:
    • option smtpchk [HELO|EHLO] [<domain>] : Turns on smtp check
      • without any options, a HELO localhost is sent to the server.
      • HELO or EHLO (optional, requires <domain> ) : tupe of smtp hello command to use: HELO for SMTP and EHLO for ESTMP. Any other value here with failback to HELO
      • <domain> (optional, requires HELO or EHLO ) : domain name to present to the server
  • on the server line:
    • check : enables health checking
    • a TCP port on configured on the server IP or the port parameter

The check is valid if the server response code starts by ‘2’.

NOTE

For more complicated and advanced SMTP checks, please read the the section generic Send/Expect below.

Examples:

  1. Check a SMTP service:

    backend be_smtp
 [...]
 option smtp-check
 server srv1 10.0.0.1:25 check
 server srv2 10.0.0.2:25 check

  2. Check a ESMTP service, using our haproxytest.check domain

    backend be_smtp
 [...]
 option smtp-check EHLO haproxytest.check
 server srv1 10.0.0.1:25 check
 server srv2 10.0.0.2:25 check

  3. Check a postfix farm where the proxy-protocol is used:

    backend be_smtp
 [...]
 option smtp-check
 server srv1 10.0.0.1:25 check send-proxy
 server srv2 10.0.0.2:25 check send-proxy

Checking any service

HAProxy is able to run health checks in a send / expect way, allowing as many send and expect as required to validate a service state. It makes health check scriptable.

This makes HAProxy very flexible and can be used to match almost any protocol, or to enhanced existing hardocded checks with features not available.

  • The following directives are available in the backend section:

    • option tcp-check : enables and allows tcp-check connect / send / send-binary / expect sequences

    • tcp-check connect [port <port>] [send-proxy] [ssl] : establishes a new TCP connection

      • port <port> (optional) : TCP port to which we establish the TCP connection. If not set, server’s line port is used if set, otherwise, the configured server port.
      • send-proxy (optional) : force sending the proxy protocol over the new connection
      • ssl (optional) : enble encryption of the connection

    • tcp-check expect [!] <match> <pattern> : method to analyze the data sent by the server:

      • [!] : negate the match result
      • <match> : matching method used to look for <pattern> in the response. Accepted match method are:
        • binary : raw binary string match in the response’s buffer
        • string : raw sting match in the response’s buffer
        • rstring : regex match in the response’s buffer
      • <pattern> : raw string or regex used by the <match> method. Spaces must be backslashed. For binary sting matching,<pattern> is passed as a serie of hexadecimal digits in a even number. Each one represents a byte.

    • tcp-check send <data> : specifies the request string to be sent to the server

    • tcp-check send-binary <hexadata> : specifies the request string in hexadecimal to be sent to the server

      NOTE

      multiple tcp-check send or tcp-check send-binary in a row are concatened together before being sent over the wire.

  • on the server line:

    • check : enables health checking
    • a TCP port on configured on the server IP or the port parameter

Examples:

  1. Check a POP3 service:

    backend be_pop
 [...]
 option tcp-check
 tcp-check connect 110
 tcp-check expect string +OK
 server srv1 10.0.0.1:110 check
 server srv2 10.0.0.2:110 check

  2. Check POP3 and POP3s services in a farm where both of them are load-balanced:

    backend be_pop
 [...]
 option tcp-check
 tcp-check connect 110
 tcp-check expect string +OK
 tcp-check connect 995
 tcp-check expect string +OK
 server srv1 10.0.0.1 check
 server srv2 10.0.0.2 check

  3. Check in a redis farm, which one is the master and select it as valid:

    backend be_redis
 [...]
 option tcp-check
 tcp-check send PING\r\n
 tcp-check expect string +PONG
 tcp-check send info\ replication\r\n
 tcp-check expect string role:master
 tcp-check send QUIT\r\n
 tcp-check expect string +OK
 server srv1 10.0.0.1:6379 check
 server srv2 10.0.0.2:6379 check

  4. Check in a HTTP service answers a status code 200 and have the string “service up” in the body:

    backend be_myapp
 [...]
 option tcp-check
 tcp-check send GET\ /check HTTP/1.0\r\n
 tcp-check send Host:\ www.mydomain.com\r\n
 tcp-check send \r\n
 tcp-check expect rstring ^HTTP/1.1\ 200\ Ok
 tcp-check expect string service\ up
 server srv1 10.0.0.1:8080 check
 server srv2 10.0.0.2:8080 check

  5. Forge a binary string to check php-fpm server status:

First, enable the following statements in your php-fpm configuration:

ping.path = /ping
ping.response = pong

Then configruation HAProxy to forge a fast-cgi query to reach the /ping url and look for the pong keyword in the response:

backend be_phpfpm
 [...]
 option tcp-check
 # FCGI_BEGIN_REQUEST
 tcp-check send-binary   01 # version
 tcp-check send-binary   01 # FCGI_BEGIN_REQUEST
 tcp-check send-binary 0001 # request id
 tcp-check send-binary 0008 # content length
 tcp-check send-binary   00 # padding length
 tcp-check send-binary   00 #
 tcp-check send-binary 0001 # FCGI responder
 tcp-check send-binary 0000 # flags
 tcp-check send-binary 0000 #
 tcp-check send-binary 0000 #
 # FCGI_PARAMS
 tcp-check send-binary   01 # version
 tcp-check send-binary   04 # FCGI_PARAMS
 tcp-check send-binary 0001 # request id
 tcp-check send-binary 0045 # content length
 tcp-check send-binary   03 # padding length: padding for content % 8 = 0
 tcp-check send-binary   00 #
 tcp-check send-binary 0e03524551554553545f4d4554484f44474554 # REQUEST_METHOD = GET
 tcp-check send-binary 0b055343524950545f4e414d452f70696e67   # SCRIPT_NAME = /ping
 tcp-check send-binary 0f055343524950545f46494c454e414d452f70696e67 # SCRIPT_FILENAME = /ping
 tcp-check send-binary 040455534552524F4F54 # USER = ROOT
 tcp-check send-binary 000000 # padding
 # FCGI_PARAMS
 tcp-check send-binary   01 # version
 tcp-check send-binary   04 # FCGI_PARAMS
 tcp-check send-binary 0001 # request id
 tcp-check send-binary 0000 # content length
 tcp-check send-binary   00 # padding length: padding for content % 8 = 0
 tcp-check send-binary   00 #

 tcp-check expect binary 706f6e67 # pong

NOTE

the whole binary string could be setup in one line, but being written like this make it clearer to update later.

  • 1 Người dùng thấy hướng dẫn này hữu ích
Hướng dẫn này có hữu ích?

Những hướng dẫn liên quan

Haproxy-SSL/TLS layouts

By design, HAProxy is a proxy. It means that it maintains 2 types of connections:...

Haproxy-Handling SSL/TLS

Table of Contents Before starting Managing TLS/SSL certificates Configuring HAProxy...