Methodology & submission guide

Overview

Hundreds of SOC estimation methods are published every year, each evaluated on different data, drive profiles and error metrics — which makes them impossible to compare. This tool fixes the data and the test: everyone parameterises or trains on the same open data, and every submission is scored on the same blinded data by the same evaluator. Results are directly comparable across authors, methods and years.

Blinding matters. The m448 cell is never released, and the standard UDDS / HWFET / LA92 / US06 cycles plus one HWCUST and one HWGRADE cycle per cell and temperature are withheld. An algorithm cannot be tuned to the answer key.

The blinded test cases

Each test case is an average of the per-cycle RMSE (in % SOC) over a subset of the evaluation runs: the 144 blinded drive cycles (36 per cell: six cycle types at six temperatures) plus the robustness runs — test 4 uses charging profiles, test 10 restarts three cycles from a wrong initial SOC (90 / 60 / 30 %), test 11 re-runs three cycles with a constant current-sensor offset (±0.05, ±0.1, ±0.3 A) — 195 runs in all. The three headline groups are estimation accuracy, operating conditions and model robustness.

Metrics & weighted score

For every blinded cycle the evaluator reports RMSE, MAE and maximum error (all in % SOC), and returns the predicted and actual SOC time series. Each test case above is the mean RMSE over its cycles.

The leaderboard ranks by weighted error: the weighted sum of the test-case values using the weights in the tables above. Weights are equal per test type (0.1 each), split evenly where a type has several cases (loads, temperatures), and test 1 is weighted 0 because every other test is a subset of it. Weights sum to 1, so the score is itself a percentage-point SOC error.

Two further columns appear on the leaderboard: Max error (worst instantaneous error anywhere) and a Complexity classification from 1 (trivial) to 10 (extreme, ±1) reflecting the computational cost of the model in the evaluator.

Submission format

A submission is a single .zip file with everything at the top level — no sub-folders. It must contain the estimator; author, affiliation and model name come from your account and the submission form:

• Model.m, Model.p or Model.py — the estimator function, named exactly Model. Use p-code if you need to protect MATLAB source. Python models get the same Model(X, z) contract (return (Y_est, z)) and run in an environment with numpy and scipy only — ship trained weights as arrays and implement inference with numpy.
• Any other files the model needs (parameter .mat files, lookup tables). Toolboxes are not available — implement network layers and filters yourself.

Function signature

The evaluator calls the model once per sample at 1 Hz, mimicking a BMS:

1[SOC(1), z] = Model(X(1, :));        % initial call, no z
2for i = 2:T
3    [SOC(i), z] = Model(X(i, :), z);   % iterative call with state
4end

• X is a 1×3 row: Current [A] (negative = discharge, positive = charge), Voltage [V], Battery temperature [°C].
• Y is the 1×1 SOC estimate on 0 … 1.
• z is free-form memory returned to you on the next call: filter states, hidden states, input history, parameters. Detect the first call with nargin < 2 and initialise there. Avoid load() inside the loop.

Minimal example

1% SOC Estimation Example V2 — online Coulomb counter
2function [Y_est, z] = Model(X, z)
3    Current  = X(1);      % A, negative = discharging
4    % Voltage  = X(2);    % V   (unused here)
5    % Temp     = X(3);    % °C  (unused here)
6    Capacity = 4.6;       % Ah
7
8    if nargin == 1        % first sample: initialise memory z
9        SOC = 1;          % assume fully charged
10    else
11        SOC = z + Current*(1/3600)/Capacity;
12    end
13    z = SOC;              % memory returned to the evaluator
14    Y_est = SOC;          % 0 … 1
15end

Complete EKF, FNN and LSTM packages — including how to unpack trained network weights into a step function — are walked through on the Examples page with schematics and annotated source.

Testing before you submit

You don't need MATLAB or any local tool. On the Submit page, Test your package first runs your zip through the production evaluator on one public drive cycle (m80, REORDERED1 at 25 °C, first two hours of the open data): the same +0.3 A validation pass, then the cycle with the standard one-hour padding. Within a minute or so you see whether the package loads and runs, the error message if it doesn't, its RMSE on that cycle, and its complexity bin.

A test run never touches the blinded data, is not scored, does not appear on any leaderboard and does not count against contest limits. It is rate-limited to five per hour per account. Package checks (archive layout, file names, function signature) also run instantly on every upload.

How evaluation runs

1. Your package is stored and a job is queued. The queue is processed in order by the evaluation worker.
2. An evaluation may use up to 6 hours of compute (the reference LSTM takes minutes); test runs are limited to 10 minutes. A run that exceeds the limit fails with a timeout.
3. The model is loaded and iterated over every blinded cycle for all four cells at all six temperatures, then over the charging profiles and the robustness variants (initial SOC of 90 / 60 / 30 %; current offsets of ±0.1 A and ±0.3 A).
4. Per-cycle errors, test-case averages, the weighted score and down-sampled time series are written to your submission page; you receive an email.
5. The uploaded package is deleted as soon as the evaluation finishes. Source is never displayed on the site.

If a model throws, returns NaN, or exceeds the runtime budget, the submission is marked failed with the evaluator's message and can be re-queued after you fix it.

Citation

Please cite both the paper and the dataset when you publish results from the tool:

P. J. Kollmeyer, M. Naguib, F. Khanum and A. Emadi, “A Blind Modeling Tool for Standardized Evaluation of Battery State of Charge Estimation Algorithms,” 2022 IEEE Transportation Electrification Conference & Expo (ITEC), pp. 243–248, 2022, doi: 10.1109/ITEC53557.2022.9813996.

P. J. Kollmeyer, F. Khanum, M. Naguib and A. Emadi, “Tesla Model 3 2170 Li-ion Cell Dataset and Battery SOC Estimation Blind Modeling Tool,” Borealis, V2, doi: 10.5683/SP3/ZVTR4B.