Category: Open Energy Monitor

Updating EmonPi to Continuous Monitoring

Posted on by Adam

Continuous monitoring with the Open Energy Monitor EmonPi hardare us now possible (since Feb 2023), see: https://docs.openenergymonitor.org/emonpi/firmware.html. This gives a more accurate measurement of power than sampling at time intervals.

Unfortunately, the process is incompletely documented there and the forum contains several threads which contain errors, distractions, and diversions alongside the information to make the firmware update work. I certainly had to spend time disentangling things to arrive at what is actually a simple procedure, although the forum was a source of some trepidation. Also: do not be tempted into the compilation process described on that firmware web page.

NB: what follows is based on my EmonPi setup. I believe some details will be different for EmonTX + EmonBase setup.

First, ensure that your system is up to date using Setup > Admin > Update: Full Update. This does not change the firmware, but will change what you see available in the Update Firmware Only section. Once updated, you can select options in the drop-down lists in the Update Firmware Only. My hardware is an emonPi and I opted for the “RFM68 LowPowerLabs” radio format (as recommended). Upon hitting the “Update Firmware” button, various messages will appear in the log.

Once it has completed, you will see that there are no updates in Setup > Inputs or Setup > EmonHub. This is because the baud rate and message format have changed. In Setup > EmonHub, use the “Edit config” button to fix this. Changes to two sections are required.

Under “[interfacers]” will be an entry which seems to have gone under different names at different times and/or different hardware configurations. If you have an emonPi, look for the section which contains “com_baud = 38400”, “pubchannels = ToEmonCMS”, and “baseid = 5”. Mine is identified as [[RFM2Pi]]. If you do not have an emonPi then a different baseid will apply. I only changed the com_baud from 38400 to 115200. Leave the rest alone!

Scroll down and find, under “[nodes]”, a section “[[5]]” (matches the baseid above). Leave the “nodename” entry as it is but modify the lines under “[[[rx]]]” to read:

names = Msg, power1,power2,power1pluspower2,vrms,t1,t2,t3,t4,t5,t6,pulse1count,pulse2count,E1,E2
datacodes = L, h, h, h, h, h, h, h, h, h, h, L, L, l, l
scales = 1, 1,1,1, 0.01, 0.01,0.01,0.01,0.01,0.01,0.01, 1, 1, 1,1
units = n,W,W,W, V, C,C,C,C,C,C, p, p, Wh,Wh

Note that the scales line can be used to adjust the power values and Vrms which the emonPi records to accomodate systematic errors due to component tolerances; the text quoted above assumes that this calibration has not been done.

I found that simply saving the config and then using the “View log” button showed this worked. Some people state that EmonHub should be restarted, but I did not find this was needed.

Make a final check that things are working by visiting Setup > Inputs and looking at graphs, visualisation, your app etc. At this point, you might note that there are now two new attributes in the input data: E1 and E2. Unfortunately (again, argh!), the meaning of these (vs the pre-existing power1 and power2) is not properly documented and mentions in the forums are often roundly uninformative.

Using Websockets with EmonPi MQTT Broker to Create a “Live Feed” Dashboard

Posted on by Adam

This is not a complete description of the background tech; there is plenty of info on the web about websockets, mqtt, and the javascript libraries.

The motivating idea behind this experiment is to be able to have a live-updating dashboard with the minimum set of dependencies of the “install X” kind. Since the EmonPi aleady has a MQTT broker, it provides the basis for feeding data to the dashboard. Websockets allows a “web page” (it can be a file viewed in your web browser) to send and receive data, and update the page, using JavaScript. This is not particularly difficult, but there are several steps which took a couple of evenings to research and put into practice, so here are some notes (for myself in the future, and anyone else who finds it!).

If you are looking for some “homework reading”, a decent place to start is Steve’s Internet Guide.

Make Mosquitto Listen for Websockets Connections

Mosquitto is the MQTT broker on the EmonPi. It is configured to listen for connections which employ the “mqtt:” protocol. It is possible to add a websockets listener (“ws:” protocol), with the conventional port 9001 assignment as follows:

  • SSH onto the EmonPi and navigate to /etc/mosquitto.
  • Modify the mosquitto.conf file to add “listener 9001” and “protocol websockets”, see below. I have also added explicit lines for the default mqtt listener on port 1883 in the interest of clarity, although I believe they are not required. You will need to use “sudo”. Alternatively, it should be possible to add a file to the “include_dir”.
  • Restart mosquitto using “sudo systemctl restart mosquitto.service”
pid_file /var/run/mosquitto.pid
persistence false
persistence_location /var/lib/mosquitto/
log_dest file /var/log/mosquitto/mosquitto.log
include_dir /etc/mosquitto/conf.d

listener 1883
protocol mqtt

listener 9001
protocol websockets

allow_anonymous false
password_file /etc/mosquitto/passwd
log_type error

It should now be possible to test two things: that the existing mqtt protocol listener, which is relied on to service data to EmonCMS, and that the websockets listener us “up”. I used “MQTT Explorer”, a free and simple client, which should be set up to not validate a certificate and to have an empty “Basepath” (it defaults to “ws”).

Create the Websockets Dashboard with HTML and JavaScript

My primary aim for this experiment is to be able to co-opt EmonPi to broker air quality data from a home-brew particulate matter, VOC, NOx, CO2 sensor combo, but I’m using the existing emon data to demonstrate the concept, which boils down to “guages” using the Google Charts toolkit, and scrolling line charts using Chart.js.

Here is the code to hack about with, based on snippets from various places, with modifications and updating to a recent. It should just live in a plain text file with a “.html” extension, and can be opened in your web browser. It is not beautiful but demonstrates the concept. There is some logging to the “console”, which is where error messages will also appear. Hit F12 on Firefox or Edge (or Chrome too I think) to find the console.

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <!-- This helps with viewing on mobile devices -->
    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
    <title>Real-Time Charts</title>
    <!-- Google charts -->
    <script type="text/javascript" src="https://www.gstatic.com/charts/loader.js"></script>
    <!-- Chart.js -->
    <script src="https://cdnjs.cloudflare.com/ajax/libs/Chart.js/3.9.1/chart.min.js" integrity="sha512-ElRFoEQdI5Ht6kZvyzXhYG9NqjtkmlkfYk0wr6wHxU9JEHakS7UJZNeml5ALk+8IKlU6jDgMabC3vkumRokgJA==" crossorigin="anonymous" referrerpolicy="no-referrer"></script>
    <!-- The Paho Javascript for MQTT over Websockets -->
    <script src="https://cdnjs.cloudflare.com/ajax/libs/paho-mqtt/1.0.1/mqttws31.min.js" type="text/javascript"></script>
    <!-- Bootstrap CSS - can be removed but will help with the styling -->
    <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/bootstrap/5.2.3/css/bootstrap.min.css" integrity="sha512-SbiR/eusphKoMVVXysTKG/7VseWii+Y3FdHrt0EpKgpToZeemhqHeZeLWLhJutz/2ut2Vw1uQEj2MbRF+TVBUA==" crossorigin="anonymous" referrerpolicy="no-referrer" />
    <link rel="stylesheet" type="text/css" href="{{ url_for('static', filename='main.css') }}">
</head>
<body>
<script type="text/javascript">
    // load the google charts stuff and only after it is loaded should we start to set up the data acquisition
    // otherwise, we find the charts library is being called upon before it exists!
    google.charts.load('current', {'packages':['gauge']});
    google.charts.setOnLoadCallback(doWhenReady);

    function doWhenReady() {
        // Create a client instance. NB clientid SHOULD BE DIFFERENT between browser clients; the following should work fine in a home environment.
        var clientId = "client-" + Date.now().toString();
        client = new Paho.MQTT.Client("192.168.1.1", 9001, clientId);
        // set callback handlers
        client.onConnectionLost = onConnectionLost;
        client.onMessageArrived = onMessageArrived;
        // connect the client
        client.connect({onSuccess:onConnect, userName: "emonpi", password: "emonpimqtt2016"});

        // called when the client connects
        function onConnect() {
          // Once a connection has been made, make subscriptions
          console.log("onConnect");
          client.subscribe("emon/emonpi/vrms");
          client.subscribe("emon/emonpi/power1");
        }

        // called when the client loses its connection
        function onConnectionLost(responseObject) {
          if (responseObject.errorCode !== 0) {
            console.log("onConnectionLost: " + responseObject.errorMessage);
            document.getElementById("status").innerText = "Lost connection due to: " + responseObject.errorMessage;
          }
        }

        // called when a message arrives. Note that the full topic name is aka "destinationName"
        function onMessageArrived(message) {
          document.getElementById("status").innerHTML = "";
          console.log("onMessageArrived: " + message.destinationName + " : " + message.payloadString);
          if (message.destinationName == "emon/emonpi/vrms") {
            // "+" converts string to numeric, which is then rounded by toFixed(), which returns a string!
            var vrms = (+message.payloadString).toFixed(2);
            // Google chart guage
            vrmsGaugeData.setValue(0, 1, +vrms);
            vrms_gauge.draw(vrmsGaugeData, vrmsOptions);
            // scrolling line chart.js
            if (vrmsChartData.data.labels.length === 20) {
                vrmsChartData.data.labels.shift();
                vrmsChartData.data.datasets[0].data.shift();
            }
            // we don't have the time in the mqtt message!
            var timestamp = new Date().toLocaleTimeString();
            vrmsChartData.data.labels.push(timestamp);
            vrmsChartData.data.datasets[0].data.push(vrms);
            lineChartVrms.update();
          } else if (message.destinationName == "emon/emonpi/power1") {
            var power1 = message.payloadString;
            // Google
            power1GaugeData.setValue(0, 1, +power1);
            power1_gauge.draw(power1GaugeData, power1Options);
          }
        }

        // ------------- Google Charts -----------
        var vrmsGaugeData = new google.visualization.arrayToDataTable([
            ['Label', 'Value'],
            ['Vrms', 240]
        ]);
        var power1GaugeData = google.visualization.arrayToDataTable([
            ['Label', 'Value'],
            ['Power', 0]
        ]);
        var vrmsOptions = {
            min        : 200,
            max        : 260,
            minorTicks : 5,
            greenFrom  : 230,
            greenTo    : 250,
            majorTicks : ['200', '210', '220', '230', '240', '250', '260']
        };
        var power1Options = {
            min        : 0,
            max        : 5000,
            minorTicks : 5,
            majorTicks : ['0', '1kW', '2kW', '3kW', '4kW', '5kW']
        };

        var vrms_gauge = new google.visualization.Gauge(document.getElementById('vrms-gauge'));
        var power1_gauge = new google.visualization.Gauge(document.getElementById('power1-gauge'));
        vrms_gauge.draw(vrmsGaugeData, vrmsOptions);
        power1_gauge.draw(power1GaugeData, power1Options);

        // chart.js
        var chartOptions = {
            responsive: true,
            tooltips: {
                mode: 'index',
                intersect: false,
            },
            hover: {
                mode: 'nearest',
                intersect: true
            },
            scales: {
                xAxes: {
                    display: true,
                    scaleLabel: {
                        display: true,
                        labelString: 'Time'
                    }
                },
                yAxes: {
                    display: true,
                    scaleLabel: {
                        display: true,
                        labelString: 'Value'
                    }
                }
            }
        };

        // this allows us to have common chart options, while varying the scale in each chart
        var vrmsChartOptions = structuredClone(chartOptions);
        // use sensible range, which will be expanded if the data goes outside "suggested"
        vrmsChartOptions.scales.yAxes.suggestedMin = 230;
        vrmsChartOptions.scales.yAxes.suggestedMax = 250;

        var vrmsChartData = {
        type : 'line',
        data : {
            labels: [],
            datasets : [{
            label : 'Vrms',
            backgroundColor : 'rgba(255, 136, 0, 0.5)',
            borderColor : 'rgba(255, 136, 0, 1.0)',
            fill: true,
            data : []
            }]
        },
        options : vrmsChartOptions
        };

        const lineChartVrms = new Chart('vrmsChart', vrmsChartData);
    }
</script>

<h1>EmonPi Live Feed</h1>
<div class="container">
    <div class="row">
        <div id="vrms-gauge" style="width: 120px; height: 120px;"></div>
        <div id="power1-gauge" style="width: 120px; height: 120px;"></div>
    </div>
    <div class="row" id="status">loading data...</div>
    <div class="col-10">
    <div class="card">
        <div class="card-body">
            <canvas id="vrmsChart"></canvas>
        </div>
    </div>
</div>
</div>
</body>
</html>

Don’t forget to change the IP address for the MQTT broker!

Serving the Dashboard

While the above HTML + JavaScript works fine as a file on your PC (so long as you have a network connection to acquire all those JavaScript libraries from), it can also be placed on the EmonPi.

I have opted to create a separate area from EmonCMS in the interest of avoiding too much risk of cock-up, future muddle, etc… The files in this separate area will all be “static” in the sense that they are just served to users as they are (no PHP etc). The dashboard page above IS static in this sense, since the JavaScript executes on your PC inside the web browser; all the EmonPi does is send it.

EmonCMS uses the Apache2 web server, so it is easy to make it listen on a different port (EmonCMS uses port 80, and I have chosen port 8001) using the following commands (in a SSH).

Create the place where the HTML file will live

cd /var/www
mkdir static

Check the directory ownership is correct using “ls -la”, which should show “drwxr-xr-x 2 pi pi 4096 Nov 27 23:02 static”.

If necessary “chown pi:pi static”.

Place the HTML + JavaScript file here (I suggest using the scp command, for example “scp dashboard.html pi@192.168.1.1:/var/www/static”).

Change the Apache2 config

There are two things to change, first to make apache2 listen on port 8001, and secondly to tell it to use the files in /var/www/static in connection with port 8001 requests.

cd /etc/apache2
sudo nano ports.conf

Add a single line “Listen 8001”, then save and exit.

For the second step, enter the “sites-enabled” directory. I chose to copy the existing emoncms.conf file, naming the copy “static.conf” and editing it to contain:

<VirtualHost *:8001>
  ServerName localhost
  ServerAdmin webmaster@localhost
  DocumentRoot /var/www/static

  # Virtual Host specific error log
  ErrorLog /var/log/static/apache2-error.log

  <Directory /var/www/static>
    Options FollowSymLinks Indexes
    AllowOverride All
    DirectoryIndex index.html
    Order allow,deny
    Allow from all
  </Directory>
</VirtualHost>

The changes are fairly obvious, but note that I have also added “Indexes” against “Options”. This makes apache give a listing of all the files in “static” when I use the URL “http://192.168.1.1:8001”. If you only have one dashboard, then simply name it “index.html” and it will appear when that URL is used.

Make a directory for logs

Otherwise apache2 will not restart.

cd /var/log
mkdir static

Restart Apache2

To load the new config.

sudo systemctl restart apache2

Make sure EmonCMS is still working, then visit your new site! It should work adequately on a mobile phone, once rotated to landscape format.

 

Using MQTT in Open Energy Monitor to Capture External Device Data

Posted on by Adam

I struggled to find clear and well-organised information about this using web searches, so here is a condensed “how to”. The scenario I have is using a home-brew ESP8266 based device attached to my solar PV inverter which I want to relay definitive power output to my Open Energy Monitor via MQTT. This seems quite simple in principle, and is simple in practice, but seemingly not well documented.

First thing is to publish the data to an MQTT broker on the emonpi with a topic which starts “emon/{source}/{key}”, replacing {source} with a suitable name for the data source and {key} with the attribute name for the data being sent. In my case, I used “emon/solis/power” as the data source is the output power for my Solis PV inverter. The message payload is simply the data value. This immediately makes the published data appear on the “Inputs” screen of EmonCMS.

Two refinements/possibilities:

Send JSON

Rather than just a single value, send several in the same message. There are two ways to do this:

a) use a topic of form “emon/{source}”

b) use a topic of form “emon/{source}/{key}”

If the message payload sent to the MQTT topic “emon/solis/power” looks like {“ac”: 90, “dc”: 120}, option (a) creates EmonCMS inputs “ac” and “dc” under “solis”, while option (b) creates inputs called “power_ac” and “power_dc”.

Include a timestamp

To do this, simply include an extra field in the JSON called “time”, with a value which is the Unix time. If you are testing, the Unix time needs to be fairly close to the actual time (ignoring summer time) otherwise EmonCMS will indicate “inactive”, but it still captures the data.

Aside: I used the VSMqtt plugin for VSCode as the MQTT client, as I’m using the PlatformIO plugin to develop my ESP8266 code (using the Arduino libraries). Update: these days I’ve changed to using MQTT Explorer as my go-to software for viewing MQTT messages.