Mirajv1.0
EN
EnglishFrançais

22. miraj-proxy: load balancing across nodes

miraj-proxy sits between applications and the nodes of a MIRAJ cluster (chapter 12). It directs each connection to the right node, with no change on the application side:

  • on the write port (rw), a connection always goes to the current primary, including after a switchover: the application no longer needs to know the primary;
  • on the read port (ro), a connection goes to the least loaded healthy node, skipping secondaries that lag too far behind.

The proxy relays bytes without reading the protocol. TLS encryption and authentication pass through end to end: the proxy has neither a certificate nor the passwords of application accounts, and clients connect with their usual tools and drivers, as to a MIRAJ server.

The node is chosen when each connection is opened. An open connection stays on its node until it is closed; the proxy does not distribute the statements of a single connection.

miraj-proxy ships with the Cluster edition. It also works in front of a single server, outside a cluster: it then serves as a single entry point, which prepares a later move to a cluster.

22.1 Setup#

  1. Nodes. Start each node with the proxy's address in --proxy-protocol-from (or the proxy_protocol_from key of miraj_config.xml):

     miraj-server.exe --root D:\donnees --log --cluster-config cluster.toml --proxy-protocol-from 10.0.0.9

    Connections coming from this address must begin with the PROXY header that gives the real client's address (see 22.5). Other connections are unchanged.

  1. Monitoring account. On the primary (accounts are relayed to the other nodes):

     CREATE USER 'proxy_monitor'@'localhost' IDENTIFIED BY '…';
 GRANT PROCESS, REPLICATION CLIENT ON *.* TO 'proxy_monitor'@'localhost';

    Monitoring connections arrive without a client address: the node sees the proxy's address there. Name the account's host after this address (localhost if the proxy runs on a node, 10.0.0.9 otherwise). PROCESS is used to count the active sessions of all accounts; REPLICATION CLIENT to read the replication lag.

  1. proxy.toml, next to miraj-proxy.exe (or --config <file>): see 22.2.

  1. Startup:

     miraj-proxy.exe --log

    The proxy probes each node, announces the primary, then accepts connections. Applications connect to the write port to write, and to the read port for reads that tolerate a slight replication lag.

22.2 The proxy.toml file#

[proxy]
rw = "0.0.0.0:7007"          # écritures et lectures : toujours le primaire courant
ro = "0.0.0.0:7017"          # lectures : le nœud sain le moins chargé (facultatif)
admin = "127.0.0.1:7117"     # état du proxy, lu par miraj-proxy status (facultatif)
language = "fr"              # langue des erreurs envoyées aux clients (en par défaut)
log_file = "proxy.log"       # journal écrit avec --log, relatif au dossier de ce fichier
source = "10.0.0.9"          # adresse locale des connexions vers les nœuds (facultatif)

[nodes]                      # identifiant du nœud (cluster.toml, [node] id) = adresse client
n1 = "10.0.0.1:7007"
n2 = "10.0.0.2:7007"
n3 = "10.0.0.3:7007"

[monitor]                    # compte de supervision
user = "proxy_monitor"
password = "…"               # ou la variable d'environnement MIRAJ_PROXY_PASSWORD
tls = "preferred"            # preferred | required | disabled
tls_ca = "cluster-ca.pem"    # autorité des certificats des nœuds (implique required)
period_ms = 1000             # une sonde par nœud et par période
timeout_ms = 2000            # connexion à un nœud et réponse d'une sonde
unhealthy_after = 3          # sondes ratées d'affilée avant d'écarter un nœud

[routing]
primary_in_ro = false        # le primaire reçoit aussi des connexions de lecture
max_lag_bytes = 16_777_216   # secondaire écarté des lectures au-delà (0 : sans limite)
max_lag_seconds = 5          # idem, en secondes (0 : sans limite)
active_weight = 4            # poids d'une session active face à une connexion ouverte
connection_reserve = 5       # places laissées libres sous max_connections d'un nœud
close_rw_on_primary_change = true  # coupe les connexions d'écriture vers l'ancien primaire
proxy_protocol = true        # en-tête PROXY : les nœuds voient l'adresse du vrai client

Only rw, the [nodes] table, and user are required. Relative paths are read from the folder of proxy.toml. A syntax error, an unknown key, or an out-of-range value stops the proxy at startup, with the number of the offending line.

The identifiers in [nodes] are those of the nodes (@@cluster_node_id): on each probe the proxy checks that the node reached does announce this identifier, and discards one that announces another (wrongly copied address). The addresses are those of the client port (7007 by default), not of the inter-node port.

Protect proxy.toml: it contains the monitoring account's password in clear text, unless it is given through MIRAJ_PROXY_PASSWORD.

22.3 Node selection#

22.3.1 Write port#

The primary is the healthy node that announces the PRIMARY role, accepts writes, and carries the highest epoch (@@cluster_epoch). A former isolated primary that still believes it is primary is therefore ignored as soon as the new one is elected. Two primaries at the same epoch (erroneous manual configuration): the proxy chooses neither, refuses write connections, and reports it in its log. A single server outside a cluster (role NONE) is its own primary.

22.3.2 Read port#

A node receives read connections if it is healthy and:

  • a streaming secondary as seen from the primary (STATE = CONNECTED), within the limits max_lag_bytes and max_lag_seconds;
  • or primary, if primary_in_ro = true;
  • or a server outside a cluster;
  • and if it has more than connection_reserve slots left under its max_connections.

Among these nodes, the proxy chooses the one with the lowest load:

load = connections opened by the proxy to this node + active_weight × active sessions

Open connections are counted instantly, as soon as the choice is made: a burst of connections is distributed immediately, without waiting for the next probe. Active sessions (executing a command, excluding Sleep) come from the last probe and also count the work of clients connected directly to the node. At equal load, nodes are served in turn.

If no node is eligible (secondaries all lagging or unreachable), the read connection goes to the primary rather than being refused.

22.3.3 Unreachable node#

A node for which unhealthy_after consecutive probes fail is discarded; the next successful probe reinstates it. If opening the connection to the chosen node fails, the proxy tries the next one before having transmitted a single byte: the client sees nothing. With no node at all, the client receives error 9043 ("No node available for RW connections" or "RO") in place of the handshake.

22.4 Primary switchover#

When the primary changes (election in raft mode, promotion in manual mode):

  • new write connections go to the new primary as soon as a probe has seen it, in general in less than one probe period;
  • with close_rw_on_primary_change = true (default), the proxy closes the write connections still open to the former primary. The application receives a lost-connection error and reconnects through the same port, to the new primary. Without this closing, its writes would receive error 1290 from a secondary;
  • open read connections continue: a secondary remains readable.

During an election, no node is primary: write connections receive error 9043 for the duration of the election. An application must therefore know how to reconnect, as in front of a restarted server.

22.5 Client address: the PROXY header#

Without precaution, each node would see all connections coming from the proxy's address. Accounts defined by host ('compta'@'10.0.1.%') would no longer match, and a single client that gets the password wrong enough times would get the proxy's address blocked, and thus all clients (max_connect_errors, see 11.5.2).

With proxy_protocol = true (default), the proxy writes at the head of each connection a PROXY header (version 2) that gives the real client's address. The node uses it to choose the account, for blocking after failures, and displays it in SHOW PROCESSLIST. Its log notes connection from <client> via relay <proxy>.

A client announced by the header is never localhost, even if its address is a loopback (127.x, ::1): a program launched on the proxy's machine must not pass for a local client of the node. Its host is its address (127.0.0.3), displayed as is by USER() and SHOW PROCESSLIST; it opens the accounts 'u'@'127.0.0.3', 'u'@'127.%', or 'u'@'%', but neither the @localhost accounts (root included) nor those of the hosts 127.0.0.1 and ::1, reserved for direct connections. The proxy's own connections (probes, without a client address) keep the proxy's real address: localhost if it runs on the node, hence the account 'proxy_monitor'@'localhost' of 22.1.

The node only reads this header on connections coming from the addresses in --proxy-protocol-from, and requires it from them: another host cannot pass itself off as an arbitrary client, and a connection from the proxy without a header is closed. Two consequences:

  • give the address of the proxy's machine as the nodes see it (the one chosen by source if you set it);
  • from the proxy's machine, an administrator cannot connect directly to a node (their connection would come from a trusted address, without a header): they go through a proxy port. On a machine with several addresses, source reserves one address for the proxy's connections and leaves the others ordinary.

If the nodes are not configured, they close the probe connection at the handshake and the proxy's log says so (the node must accept the PROXY header from this proxy). A node the proxy cannot reach at all is discarded with the cause of the failure (TCP connection to … impossible: …, unusable source address), a refused authentication or a TLS failure with the error returned by the node, without this reminder. proxy_protocol = false removes the header; the proxy accepts it, with a warning at startup.

22.6 Monitoring#

miraj-proxy status reads the state on the admin port of the running proxy:

MIRAJ 1.0.0 - proxy, démarré depuis 3605 s
Primaire : n1
NŒUD  ADRESSE        RÔLE       ÉPOQUE  SAIN  CONNEXIONS  ACTIVES  RETARD  RW  RO  SCORE  LECTURES
n1    10.0.0.1:7007  PRIMARY    7       oui   42/151      3        0       38  0   50     primaire (primary_in_ro = false)
n2    10.0.0.2:7007  SECONDARY  7       oui   23/151      1        0       0   21  25     oui
n3    10.0.0.3:7007  SECONDARY  7       oui   22/151      0        0       0   22  22     oui
Connexions relayées depuis le démarrage : RW 1250 (refusées 3), RO 8410 (refusées 0)

CONNEXIONS (connections): the node's Threads_connected out of its max_connections; ACTIVES (active): active sessions; RW, RO: connections opened by the proxy; LECTURES (reads): oui (yes), or the reason the node does not receive reads, followed by the last probe error.

The admin port requires no authentication: keep it on the loopback (the example's value).

With --log, the log_file log keeps, timestamped: startup, each node discarded or reinstated, each role or epoch change, each primary change and the write connections closed, unreachable nodes. An ordinary connection leaves no line there.

22.7 Automatic startup#

miraj-proxy stops with the window that launched it. To start it with Windows, create a scheduled task, as for the server (see 2.2):

schtasks /Create /TN "MIRAJ - proxy" /SC ONSTART /RU SYSTEM /RL HIGHEST ^
  /TR "\"C:\Program Files\MIRAJ\miraj-proxy.exe\" --log"

For availability, install two proxies on two machines, behind a virtual address or a DNS resolution with two entries: they share no state, each probes the nodes.

22.8 Limitations#

  • Choice at open only. A connection stays on its node. An application that mixes reads and writes on a single connection must use the write port.
  • No node discovery. The [nodes] list is fixed: adding a node requires editing proxy.toml and restarting the proxy.
  • Read after write. A read through the read port may not see a very recent write made through the write port (asynchronous replication). To guarantee it, write with @@cluster_sync_commit = APPLIED (§12.8), or read through the write port.
  • One thread per direction and per connection. The proxy is sized for a few thousand simultaneous connections, not for tens of thousands.
  • KILL and connection identifiers. CONNECTION_ID() and SHOW PROCESSLIST give the identifiers of the node reached: a KILL must be sent to the same node.