Building a quadrature encoder water meter with ESPHome and magnetometers

Understanding how meter works is very important, there is an excellent YouTube video about this:

Mine happened to be a Badger Meter as well. What matters here is that inside the chamber there is a magnet wheel, driven by the nutating disk beneath it. As the disk nutates, it spins the magnet wheel, which produces a multi-pole magnetic field. This rotating field is what causes the register mounted on top to advance its readings — essentially translating the mechanical motion of water flow into the numbers you see on your meter face.

In principle, we use the magnetometer sensors (QMC5883L) that function the same as the register (the city gives to us), when we detect a magnetic field change similar to a sin wave, we count certain volume pass through the meter. That is the physics foundation of the project (also used by tronikos/esphome-magnetometer-water-gas-meter).

As you can probably guess, with a single sensor, we can only count how many the pole change event happened, what quadrature encoder helps is that it can also give us the direction by using 2 sensors with phase shift, essentially 2 sensors will produce sin/cos waves where you can decode into direction. This will give bidirectional counting — knowing not just that water flows, but which direction. Why would you care directions? Isn't that you would only consume water from the city? That is true, you will see later why we need direction as well.

Looks straightforward — so let’s get started.

both tronikos/esphome-magnetometer-water-gas-meter and espleak just encoding the ESP32 C++ code in YAML config file directly. Well, I really do not like writing C++ source code in YAML. The tronikos project, like most ESPHome configs, puts all the logic inside lambda: blocks. This works, but you lose syntax highlighting, proper compilation errors, and any semblance of code organization once your logic exceeds 20 lines.

ESPHome gives you two escape hatches here. The first is includes:, where you list .h and .cpp files that get copied into the platformio build:

esphome:
  includes:
    - sensor_update.h
    - sensor_update.cpp

This works great locally with esphome compile, but falls apart on Home Assistant's ESPHome builder because paths get expanded to absolute during the YAML fusion step. /config/esphome/sensor_update.h is where it ends up looking, and of course your files aren't there if you are using packages.

The 2nd approach is external components, which is what I ended up using:

components/
  sensor_update/
     __init__.py
     sensor_update.cpp
     sensor_update.h
     sensor_update_component.h  # empty component

The trick is that platformio automatically compiles all C++ files under your component directory. The sensor_update_component.h is a dummy — ESPHome's component system requires it, but the actual logic lives in sensor_update.cpp and gets called from a thin lambda in the YAML. For deployment to Home Assistant, you switch the source from type: local to type: git and point it at your repo. Problem solved.

 cw:  0 → 1 → 3 → 2 → 0
ccw:  0 → 2 → 3 → 1 → 0

static const int qdec[16] = {
   0,  1, -1,  2,
  -1,  0,  2,  1,
   1,  2,  0, -1,
   2, -1,  1,  0
};

int curr = (q[sensor] << sensor)  + (q[1-sensor] << (1-sensor));
int prev = (qlast << sensor)      + (q[1-sensor] << (1-sensor));
int mod  = qdec[prev * 4 + curr];

With that in mind, I started the experimentation setup, 3D printed a spinning wheel with 2 magnet attached to it and wiring up my ESP32 with QMC5883ls offset by 90 degrees and started debugging. The first thing I need to deal with is the hysteresis sampling. Why? It would be nice if we get perfect sin/cos wave from the meter but in reality we get a very noisy signals, in the end we will want to get a somewhat clean box wave to represent 0 and 1s, which our software developers are familiar with. There are multiple way doing this, people in electrical engineering dealt with this long time ago through schmitt trigger (which is quite elegant and efficient). Here we use tare + hysteresis to produce box waves.

Without going too much into details, the logic is basically (But again, I would like to point out this potentially 8 CPU instructions can be implemented by a simple circuits).

float tare       = (tmax + tmin) / 2.0f;
float hysteresis = (tmax - tmin) * 0.2f;
bool  high       = signal > tare + hysteresis;
bool  low        = signal < tare + hysteresis;

Once I did the experimentation on the test setup, confirm that I can get consistent direction reading when I rotate the magnets, time to move to the real thing.

Here is where things got interesting. I mounted two QMC5883L sensors at 90 degrees physical separation on a Badger Model 25 water meter, ran some water, and got garbage. The state transitions were chattered — random CW/CCW with net zero movement. The serial log looked like this:

sensor 0 at 755414, (49.59, -28.82, 77.47) => (0, 1 -> 1)
sensor 1 at 755626, (-12.05, -93.19, 17.02) => (1, 3 -> 1)
sensor 0 at 755633, (-3.40, -28.80, 77.44) => (3, 2 -> 1)
sensor 0 at 755925, (48.02, -29.42, 76.97) => (2, 3 -> -1)
sensor 1 at 755939, (-62.28, -93.16, 16.99) => (3, 1 -> -1)
sensor 0 at 756146, (0.64, -29.40, 76.94) => (1, 0 -> -1)
sensor 1 at 756165, (0.28, -93.14, 16.97) => (0, 2 -> -1)
sensor 1 at 756452, (-62.65, -93.11, 16.94) => (2, 0 -> 1)
sensor 0 at 756456, (49.22, -29.37, 76.91) => (0, 1 -> 1)
sensor 1 at 756665, (-13.34, -93.09, 16.92) => (1, 3 -> 1)
sensor 0 at 756677, (-3.04, -29.35, 76.89) => (3, 2 -> 1)
sensor 0 at 756963, (48.02, -29.33, 76.86) => (2, 3 -> -1)

Directions flipping back and forth within milliseconds. Something was fundamentally wrong with the phase relationship.

I added CSV logging to dump the raw field strength from both sensors, and the plot told me a very different story.

When sensor 0 peaked, sensor 1 troughed, and vice versa. That is 180 degrees — anti-phase, not the 90 degrees needed for quadrature. Both sensors cross their thresholds at nearly the same time. Which one crosses "first" depends on noise and sampling jitter, so the decoder sees random CW/CCW. Initially I was still in doubt where is the bug in my code until I use an LEDs to help me confirm the behavior.

The culprit: the meter has a 4-pole magnet inside the register and my test setup uses only 2 magnets. Multi-pole magnets are common in residential water meters for higher pulse resolution and stronger near-field coupling.

The solution is then moving the second sensor to approximately 45 degrees, finger crossed and turned on the water tap.

Ah ha, it is working in real-life!

And then after is of course the boring software part of deal, creating template sensors in esphome, computing volumes based on rotations, etc. The state of the project is at xeechou/esphome-magnetometer-water-gas-meter on the quadrature-meter branch if you are interested.

Programming for micro-controllers is much more different than pure software development really. This is some C++ code running at 200kHz on a chip with only 4Mb memory, every-time we make changes in the code, we need to flash to the chip to test out. (Same story is true for GPU shaders, but drivers insulate us too well for this).

Debugging is very non-trivial either. You can't simply step through the code to figure out your mistakes, we need real world magnet field readings. Logging is not a reliable way either, the chips are very weak can easily throttled in this way.

This project is not complete either. The flow monitoring and leak trigger is not setup yet. There are also some unanswered question I have but it is to a state where I can leave it for running for a while. Have fun.

• tronikos/esphome-magnetometer-water-gas-meter — the original single-sensor project this was forked from.
• dnschneid/espleak — flow monitoring and leak detection implementation that inspired the burstmon and upcoming flowmon features.
• SparkFun: How to use a quadrature encoder (PDF) — the Gray-code lookup table reference.