Mirajv1.0
EN
EnglishFrançais

23. miraj-proxy: load balancing across nodes

miraj-proxy sits between applications and the nodes of a MIRAJ cluster (chapter 17). It directs each connection to the right node, with no change on the application side:

  • on the write port (rw), a connection always goes to the current primary, including after a switchover: the application no longer needs to know the primary;
  • on the read port (ro), a connection goes to the least loaded healthy node, skipping secondaries that lag too far behind.

The proxy relays bytes without reading the protocol. TLS encryption and authentication pass through end to end: the proxy has neither a certificate nor the passwords of application accounts, and clients connect with their usual tools and drivers, as to a MIRAJ server.

The node is chosen when each connection is opened. An open connection stays on its node until it is closed; the proxy does not distribute the statements of a single connection.

miraj-proxy ships with the Cluster edition. It also works in front of a single server, outside a cluster: it then serves as a single entry point, which prepares a later move to a cluster.

23.1 Setup#

  1. Nodes. Start each node with the proxy's address in --proxy-protocol-from (or the proxy_protocol_from key of miraj_config.xml):

     miraj-server.exe --root D:\donnees --log --cluster-config cluster.toml --proxy-protocol-from 10.0.0.9

    Connections coming from this address must begin with the PROXY header that gives the real client's address (see 23.5). Other connections are unchanged.

  1. Monitoring account. On the primary (accounts are relayed to the other nodes):

     CREATE USER 'proxy_monitor'@'localhost' IDENTIFIED BY '…';
 GRANT PROCESS, REPLICATION CLIENT ON *.* TO 'proxy_monitor'@'localhost';

    Monitoring connections arrive without a client address: the node sees the proxy's address there. Name the account's host after this address (localhost if the proxy runs on a node, 10.0.0.9 otherwise). PROCESS is used to count the active sessions of all accounts; REPLICATION CLIENT to read the replication lag.

  1. proxy.toml, next to miraj-proxy.exe (or --config <file>): see 23.2.

  1. Startup:

     miraj-proxy.exe --log

    The proxy probes each node, announces the primary, then accepts connections. Applications connect to the write port to write, and to the read port for reads that tolerate a slight replication lag.

23.2 The proxy.toml file#

[proxy]
rw = "0.0.0.0:7007"          # writes and reads: always the current primary
ro = "0.0.0.0:7017"          # reads: the least loaded healthy node (optional)
admin = "127.0.0.1:7117"     # proxy state, read by miraj-proxy status (optional)
language = "fr"              # language of the errors sent to clients (en by default)
log_file = "proxy.log"       # journal written with --log, relative to the folder of this file
source = "10.0.0.9"          # local address of the connections to the nodes (optional)

[nodes]                      # node identifier (cluster.toml, [node] id) = client address
n1 = "10.0.0.1:7007"
n2 = "10.0.0.2:7007"
n3 = "10.0.0.3:7007"

[monitor]                    # monitoring account
user = "proxy_monitor"
password = "…"               # or the MIRAJ_PROXY_PASSWORD environment variable
tls = "preferred"            # preferred | required | disabled
tls_ca = "cluster-ca.pem"    # authority of the nodes' certificates (implies required)
protocol = "auto"            # auto | miraj | postgresql: protocol of the nodes
period_ms = 1000             # one probe per node and per period
timeout_ms = 2000            # connection to a node and reply to a probe
unhealthy_after = 3          # consecutive failed probes before discarding a node

[routing]
primary_in_ro = false        # the primary also receives read connections
max_lag_bytes = 16_777_216   # secondary excluded from reads beyond this (0: no limit)
max_lag_seconds = 5          # same, in seconds (0: no limit)
active_weight = 4            # weight of an active session relative to an open connection
connection_reserve = 5       # slots left free under a node's max_connections
close_rw_on_primary_change = true  # closes write connections to the former primary
proxy_protocol = true        # PROXY header: nodes see the real client's address

Only rw, the [nodes] table, and user are required. Relative paths are read from the folder of proxy.toml. A syntax error, an unknown key, or an out-of-range value stops the proxy at startup (exit code 2), with the number of the offending line (proxy.toml, ligne 3 : clé « port » inconnue dans [proxy]).

KeyDefaultBounds and rules
rw, ro, adminro and admin: none"host:port", IPv6 in square brackets ([::1]:7007); two listeners on the same address are refused.
languageenLanguage of the errors sent to clients.
log_fileproxy.logFolder created if needed.
sourcechosen by the systemIP address; an address unusable on the machine stops the proxy at startup.
[nodes]—Identifiers: letters, digits, _ and $, at most 64 characters, no duplicates (case ignored); port from 1 to 65535.
tlspreferred (required with tls_ca)preferred and disabled cannot be combined with tls_ca.
protocolautoauto, miraj or postgresql.
period_ms1000100 to 3,600,000.
timeout_ms2000100 to 600,000.
unhealthy_after31 to 1000.
max_lag_bytes16777216 (16 MiB)0 (no limit) or more.
max_lag_seconds50 to 31,536,000.
active_weight40 to 1,000,000.
connection_reserve50 to 100,000.
primary_in_ro, close_rw_on_primary_change, proxy_protocolfalse, true, trueBooleans (true or false).

The identifiers in [nodes] are those of the nodes (@@cluster_node_id): on each probe the proxy checks that the node reached does announce this identifier, and discards one that announces another (wrongly copied address). The addresses are those of the client port (7007 by default), not of the inter-node port.

Protect proxy.toml: it contains the monitoring account's password in clear text, unless it is given through MIRAJ_PROXY_PASSWORD.

protocol key of [monitor]: auto (default, the nodes' protocol is recognized when the probe connects), miraj or postgresql. The proxy relays bytes without reading them: it serves nodes in the PostgreSQL language just the same (psql, Npgsql, pgjdbc clients...). A client refused for lack of a node receives error 9043 in its own protocol: an error packet of the legacy protocol, or, for a PostgreSQL client (recognized by its startup request, within 300 ms), N in reply to its TLS request and then an ErrorResponse FATAL 08004.

23.3 Node selection#

23.3.1 Write port#

The primary is the healthy node that announces the PRIMARY role, accepts writes, and carries the highest epoch (@@cluster_epoch). A former isolated primary that still believes it is primary is therefore ignored as soon as the new one is elected. Two primaries at the same epoch (erroneous manual configuration): the proxy chooses neither, refuses write connections, and reports it in its log (plusieurs primaires à l'époque 7 (n1, n3) : écritures refusées). A single server outside a cluster (role NONE), the only node in [nodes], is its own primary; several servers outside a cluster in [nodes] give none. An isolated (FENCED) or read-only node is never primary.

23.3.2 Read port#

A node receives read connections if it is healthy and:

  • a streaming secondary as seen from the primary (STATE = CONNECTED), within the limits max_lag_bytes and max_lag_seconds;
  • or primary, if primary_in_ro = true;
  • or a server outside a cluster;
  • and if it has more than connection_reserve slots left under its max_connections.

Among these nodes, the proxy chooses the one with the lowest load:

load = connections opened by the proxy to this node + active_weight × active sessions

Open connections are counted instantly, as soon as the choice is made: a burst of connections is distributed immediately, without waiting for the next probe. Active sessions (executing a command, excluding Sleep) come from the last probe and also count the work of clients connected directly to the node. At equal load, nodes are served in turn.

If no node is eligible (secondaries all lagging or unreachable), the read connection goes to the primary rather than being refused.

23.3.3 Unreachable node#

A node for which unhealthy_after consecutive probes fail is discarded; the next successful probe reinstates it. If opening the connection to the chosen node fails, the proxy tries the next one before having transmitted a single byte: the client sees nothing. With no node at all, the client receives error 9043 ("No node available for RW connections" or "RO") in place of the handshake.

23.4 Primary switchover#

When the primary changes (election in raft mode, promotion in manual mode):

  • new write connections go to the new primary as soon as a probe has seen it, in general in less than one probe period;
  • with close_rw_on_primary_change = true (default), the proxy closes the write connections still open to the former primary. The application receives a lost-connection error and reconnects through the same port, to the new primary. Without this closing, its writes would receive error 1290 from a secondary;
  • open read connections continue: a secondary remains readable.

During an election, no node is primary: write connections receive error 9043 for the duration of the election. An application must therefore know how to reconnect, as in front of a restarted server.

23.5 Client address: the PROXY header#

Without precaution, each node would see all connections coming from the proxy's address. Accounts defined by host ('compta'@'10.0.1.%') would no longer match, and a single client that gets the password wrong enough times would get the proxy's address blocked, and thus all clients (max_connect_errors, see 16.5.2).

With proxy_protocol = true (default), the proxy writes at the head of each connection a PROXY header (version 2) that gives the real client's address. The node uses it to choose the account, for blocking after failures, and displays it in SHOW PROCESSLIST. Its log notes connection from <client> via relay <proxy>.

A client announced by the header is never localhost, even if its address is a loopback (127.x, ::1): a program launched on the proxy's machine must not pass for a local client of the node. Its host is its address (127.0.0.3), displayed as is by USER() and SHOW PROCESSLIST; it opens the accounts 'u'@'127.0.0.3', 'u'@'127.%', or 'u'@'%', but neither the @localhost accounts (root included) nor those of the hosts 127.0.0.1 and ::1, reserved for direct connections. The proxy's own connections (probes, without a client address) keep the proxy's real address: localhost if it runs on the node, hence the account 'proxy_monitor'@'localhost' of 23.1.

The node only reads this header on connections coming from the addresses in --proxy-protocol-from, and requires it from them: another host cannot pass itself off as an arbitrary client, and a connection from the proxy without a header is closed. Two consequences:

  • give the address of the proxy's machine as the nodes see it (the one chosen by source if you set it);
  • from the proxy's machine, an administrator cannot connect directly to a node (their connection would come from a trusted address, without a header): they go through a proxy port. On a machine with several addresses, source reserves one address for the proxy's connections and leaves the others ordinary.

If the nodes are not configured, they close the probe connection at the handshake and the proxy's log says so (the node must accept the PROXY header from this proxy). A node the proxy cannot reach at all is discarded with the cause of the failure (TCP connection to … impossible: …, unusable source address), a refused authentication or a TLS failure with the error returned by the node, without this reminder. proxy_protocol = false removes the header; the proxy accepts it, with a warning at startup.

23.6 Monitoring#

miraj-proxy status reads the state on the admin port of the running proxy:

MIRAJ 1.0.0 - proxy, démarré depuis 3605 s
Primaire : n1
NŒUD  ADRESSE        RÔLE       ÉPOQUE  SAIN  CONNEXIONS  ACTIVES  RETARD  RW  RO  SCORE  LECTURES
n1    10.0.0.1:7007  PRIMARY    7       oui   42/151      3        0       38  0   50     primaire (primary_in_ro = false)
n2    10.0.0.2:7007  SECONDARY  7       oui   23/151      1        0       0   21  25     oui
n3    10.0.0.3:7007  SECONDARY  7       oui   22/151      0        0       0   22  22     oui
Connexions relayées depuis le démarrage : RW 1250 (refusées 3), RO 8410 (refusées 0)

CONNEXIONS (connections): the node's Threads_connected out of its max_connections; ACTIVES (active): active sessions; RETARD (lag): replication lag in bytes; RW, RO: connections opened by the proxy; SCORE: load of the node (23.3.2); LECTURES (reads): oui (yes), or the reason the node does not receive reads, followed by the last probe error. Possible reasons: injoignable (unreachable), plein (N connexions sur M) (full), isolé (FENCED), ancien primaire (former primary), primaire (primary_in_ro = false), pas en flux (BOOTSTRAPPING) (or another state seen from the primary; not streaming), inconnu du primaire (unknown to the primary), retard de N octets (lag of N bytes), retard de N s (lag of N s). A node not yet probed shows ?. Without a primary, the Primaire line is aucun (none), followed by the conflict, if any.

The admin port requires no authentication: keep it on the loopback (the example's value).

Without an admin key, miraj-proxy status answers pas de port admin ([proxy] admin) ; état indisponible (code 2); with the proxy stopped, Proxy injoignable sur … : … (code 1).

Events go to standard output, problems to standard error. With --log, both are also written, timestamped, to log_file: startup, each node discarded or reinstated, each role or epoch change, each primary change and the write connections closed, unreachable nodes. An ordinary connection leaves no line there. Examples:

MIRAJ 1.0.0 - proxy : écritures sur 0.0.0.0:7007, lectures sur 0.0.0.0:7017, état sur 127.0.0.1:7117 ; nœuds : n1 (10.0.0.1:7007), n2 (10.0.0.2:7007), n3 (10.0.0.3:7007).
Proxy : nœud n1 joint (PRIMARY époque 7).
Proxy : nœud n3 écarté : connexion TCP à 10.0.0.3:7007 impossible : ….
Proxy : primaire n1 → n2 (époque 8).
Proxy : 38 connexion(s) d'écriture vers n1 fermée(s).
Proxy : nœud n3 rétabli (SECONDARY époque 8).

A node that announces an identifier other than the one in [nodes] is discarded with le nœud joint s'annonce « n2 » (vérifiez l'adresse de n3 dans [nodes]).

Other commands: miraj-proxy --help displays help, miraj-proxy --version the version.

23.7 Automatic startup#

miraj-proxy stops with the window that launched it. To start it with Windows, create a scheduled task, as for the server (see 2.2):

schtasks /Create /TN "MIRAJ - proxy" /SC ONSTART /RU SYSTEM /RL HIGHEST ^
  /TR "\"C:\Program Files\MIRAJ\miraj-proxy.exe\" --log"

For availability, install two proxies on two machines, behind a virtual address or a DNS resolution with two entries: they share no state, each probes the nodes.

23.8 Limitations#

  • Choice at open only. A connection stays on its node. An application that mixes reads and writes on a single connection must use the write port.
  • No node discovery. The [nodes] list is fixed: adding a node requires editing proxy.toml and restarting the proxy.
  • Read after write. A read through the read port may not see a very recent write made through the write port (asynchronous replication). To guarantee it, write with @@cluster_sync_commit = APPLIED (§17.8), or read through the write port.
  • One thread per direction and per connection. The proxy is sized for a few thousand simultaneous connections, not for tens of thousands.
  • KILL and connection identifiers. CONNECTION_ID() and SHOW PROCESSLIST give the identifiers of the node reached: a KILL must be sent to the same node.