The Unspoken, Gaps in Data

Over the years we have come across a number of problems when it comes to receiving data. One of the common problems is where there is are gaps in data. This can cause problems for the energy management system that you may be using, and make your job even more difficult and time-consuming than it already is.

When there is no data. What do you do?

The most used and straight forward method is to carry forward values. Meaning you’re taking a previous value and bringing it forward to fill the gap. This can work when you have a single value missing and it may be sufficient depending on the scope of your needs.

Unfortunately, there is no right or wrong method of filling gaps, as it is always a compromise. The method you choose is dictated by the quality of the data you require versus the effect to attain it and the importance of the impact that the choice of the method makes to your results.

Using the MCE (Meniscus Calculation Engine) we have come up with a system that allows us to extract the data from your Energy Management System (or any other system that you may use) and pull it into MCE. This is where we set up calculations that automatically identify the data gaps.

There are various different methods filling gaps in data and there is no right or wrong method of filling gaps, as it is always a compromise. The following methods are just some of the ways you could fill the gaps in data:

• Carry values forward. Meaning you’re taking the previous value and bringing it forward to fill the gap.

• You could use data that was received at the same time. Data from a day, week or year before, and use that to fill the gap.

Other more complicated methods involve the following:

• Taking averages of points in previous years for the same point. Or doing more with season + day type averages. Each day has an associated day type (weekend + weekday for a simple example) and season of the year. This allows averages to be built that are more reflective of seasonal variations and usage and are less prone to suffering from the impact of more extreme individual values.

• Back-filling data by Interpolation using incrementing data sets. For example, if your incremental counter ended on 100 units at the start of the gap and started on 180 units at the end of the gap that means we used 80 units. And, 80 units is over a 4-day period, so therefore it’s 20 units per day. This we feel to be the most accurate method as the data is there and is accurate as it is actually what has been utilised.

Using MCE (Meniscus Calculation Engine), we extract the data into our systems where the calculations are set up to automatically identify the data. E-mailed Management reports are generated by MCE and the information is sent to you or directly to the Data Collector to ensure that the gaps are filled with the correct data.