shannondata/rapid_classifier

ShannonBase Rapid Workload Classifier

A LightGBM binary classifier, exported to ONNX, that ShannonBase uses to decide whether a query should run on the primary MySQL (InnoDB / OLTP) engine or be offloaded to the Rapid secondary engine (OLAP).

It is consumed at query-prepare time by Query_arbitrator (storage/rapid_engine/ml/query_arbitrator.cpp) via ONNX Runtime, and ships with the server as extra/llm-models/shannon_rapid_classifier.onnx.

Model I/O

	Name	Type	Shape
Input	float_input	float32	[N, 18]
Output 0	probabilities	float32	[N, 2]
Output 1	label	int64	[N]

Class 1 = OLAP (offload to Rapid), class 0 = OLTP (keep on the primary). The engine reads output 0 and takes probabilities[1] as the offload score.

Note: output 0 must be a plain float tensor. Export with zipmap=False — a ZipMap output makes the tensor read fail and every query falls back to the primary engine.

Decision rule

The score is compared against a threshold of 0.5. For queries with at least 4 of the OLAP-shaped features set (has_group_by, has_having, has_aggregation, has_order_by, has_subquery), the threshold is scaled by 0.6 to bias toward offloading. A score above the effective threshold routes the query to Rapid.

Feature vector

Order is significant and must match training exactly.

#	Feature	Description
0	mysql_total_ts_nrows	Rows scanned by non-index-ref table scans
1	mysql_cost	Estimated primary-engine cost
2	count_all_base_tables	Number of base tables
3	count_ref_index_ts	Table scans served by an index ref
4	base_table_sum_nrows	Sum of base-table cardinalities
5	are_all_ts_index_ref	All scans are index refs (0/1)
6	table_count	Tables in the query block
7	has_having	HAVING clause present (0/1)
8	has_group_by	GROUP BY present (0/1)
9	has_rollup	ROLLUP present (0/1)
10	has_order_by	ORDER BY present (0/1)
11	has_limit	LIMIT present (0/1)
12	has_join	More than one table (0/1)
13	has_subquery	Subquery present (0/1)
14	has_aggregation	Aggregate function present (0/1)
15	select_list_size	Number of select-list items
16	where_condition_count	Number of top-level WHERE conditions
17	estimated_rows	Estimated result cardinality

Usage

hf download shannondata/rapid_classifier shannon_rapid_classifier.onnx \
  --local-dir extra/llm-models/

import numpy as np, onnxruntime as ort

sess = ort.InferenceSession("shannon_rapid_classifier.onnx")
x = np.zeros((1, 18), dtype=np.float32)
probs = sess.run(None, {"float_input": x})[0]
print("offload score:", probs[0][1])

Training

Trained and exported by Shannon-Data/ShannonBase-Tools. To roll out a new model, replace extra/llm-models/shannon_rapid_classifier.onnx in the ShannonBase tree.