Q: I am seeking clarification regarding the evaluation of R² and CV(RMSE) in the M&V guidelines. Specifically, is the evaluation of these metrics intended to be performed on the full training dataset, or should it incorporate the testing dataset or cross-validation techniques?

In machine learning practice, there are well-established reasons to avoid relying solely on the full training dataset for model evaluation, as it can lead to overfitting and does not necessarily reflect the model's performance on unseen data. This approach contrasts with my current understanding of the IPVMP guidelines, which seem to imply an evaluation on the training set alone.

Could you please provide guidance on this matter to align our model evaluation process with the best practices in the field? Thank you for your time and expertise.

A: The evaluation of R2 and CV(RMSE) is based on ordinary least squares regression models, and their values should be calculated and checked for the training period data sets from which they are developed. This is a minimum set of criteria for evaluating models.

IPMVP recognizes that there are additional ways to check model validity, including out-of-sample testing and cross-validation. If done properly, these methods provide additional information about a model’s prediction capabilities and validity. IPMVP currently has no recommendations for how these methods should be performed or what the criteria would be. However, IPMVP does not restrict M&V practitioners from using more advanced methods to demonstrate the validity of models used in the M&V analysis.

IPMVP agrees with your point about machine learning algorithms and encourages you to address this concern using more advanced methods to demonstrate model validity when using such modeling methods. IPMVP is working to assess and include these advanced model validity methods in future guidance documents.

Q: I want to know how to calculation of specific product energy savings. Can you provide data on the formula for calculating energy savings for individual items?

A: Per your question, if the “specific product” or “individual items” is referring to an EEM, please refer to retrofit isolation Options (option A or B) in IPMVP Core Concepts for associated formulas.

If a “specific product” or “individual items” are production outputs from an industrial process or facility with EEM’s implemented, an appropriate response to the first question could be whole facility ala Option C.   Perhaps an energy use indexing type approach could  be appropriate?   One calculation for energy savings to share from BPA’s M&V Energy Use Indexing Protocol version 2.0, July 2018 (available for download here), is:

 Energy Savings (kWh) = ((kWh/units)base – (kWh/units)post) * units

 where:

Energy savings refers to savings occurring during the reporting period.

Units is the normalizing variable, such as square feet, widgets produced, etc. The number of units must be measured over the same time period (and at the same intervals) as the corresponding energy use.

Each parameter (kWh, units) in the ratio, kWh/units, is either a total or an average aggregated over a period of time. Alternatively, the ratio may be the slope of a regression with units as the independent variable and kWh (or other energy use measure) as the dependent variable.

 Two other formula’s:

Percentage Savings (%) = ((kWh/units)base – (kWh/units)post) / (kWh/units)base

Per-Unit Savings = (kWh/units)base – (kWh/units)post

 With data collected during baseline (base) and post-project (post) time periods, one could develop an energy savings calculation for each “specific product” or an “individual item” output from the process or facility.

Q: "MEASUREMENT&VERIFICATION-ISSUES AND EXAMPLES" 2019 Is the material up to date? If you have document about option examples, please give me that material!

A: The examples in the 2019 document referenced are the latest published examples.  There will be additional, updated examples that will be released in upcoming IPMVP application guides.

Q: Can you please confirm which IPMVP Option most closely aligns with the following approach for both single-ECM and multi-ECM projects?

A: Overall – an IPMVP Option approach consists of both a baseline method and a corresponding post-EEM (reporting period) measurement method.

Q: Perform Baseline Energy Metering - Energy meters are installed to the circuits feeding power to the addressed equipment/systems to isolate those systems from the rest of the building and capture pre-ECM(s) energy use. The baseline metering period for a given project is determined based on site local weather variations, the operating characteristics of the addressed equipment/system, and the proposed scope of work. These meters will remain installed during implementation of the ECMs.

A: [i]As stated, this Baseline approach seems to be focused on isolation of the retrofit/EEM with the installed meters. Given this information, this baseline approach for retrofit isolation with potential of all parameter measurement via the electric meters, would align with an Option B method of baseline determination.
[*][/i] [b][/b]

Q: Generate the Energy Baseline - The subhourly baseline period energy meter data in concert with coincident local weather data for the baseline period are used to derive the weather and time-of-week normalized baseline model for isolated systems at the building.

A: Same as above – appears to be stating retrofit isolation via energy meters – Option B – again, assuming the metering is capturing all parameters required for energy consumption and demand.

Q: Perform Post-ECM(s) Energy Metering - The energy meters that were installed for the baseline energy metering will be used to monitor and track the energy use of the addressed equipment/systems post-implementation of the ECMs. The post-ECM(s) energy metering period is one calendar year after the owner's acceptance of the implemented ECMs.

A: Same as above – retrofit isolation via meters which are capturing total retrofit energy use (all parameters) – used during reporting period for EEM savings determination – Option B.

Q: I would like to confirm the IPMVP Option that best aligns with our M&V approach. We provide fully financed, turnkey energy efficiency solutions to our commercial and industrial customers. These solutions can be deployed as either portfolio programs across multiple sites throughout the US or single site projects. We bill our customers based on the post-install energy savings for each "system" we install at a given site, where a "system" in our contracts is defined as the all of the equipment we install, retrofit, or optimize at a given customer site (e.g., replacement chillers, boilers, pumps, and air handling units installed at a single site can constitute a "system" in our contracts). For each contract "system", we install submeters to monitor and track the energy use of the equipment included in that "system" both before and after that equipment is optimized, retrofitted, or replaced. The pre-install metering period can vary system to system based on a variety of factors including the what equipment will be addressed and how local climate and/or site operations impact the energy use of that equipment. To measure the energy savings, we aggregate the post-install submeter interval data for the "system" and compare it to the baseline regression model established using OpenEEmeter and the pre-install submeter data. Internally, we have interpreted our M&V approach as Option C because we measuring savings at the whole building level using our submeter interval data to isolate the contract "system" from the rest of the energy using equipment at a given customer site. Is this interpretation correct? If not, which IPMVP Option best aligns with our standard M&V approach?

A: Based on the above description of your M&V strategy, since you are describing the measurement boundary for the sub-meters at the EEM equipment specific level, ("using our submeter interval data to isolate the contract "system" from the rest of the energy using equipment at a given customer site"), this approach would conceptually align with an IPMVP Option B Retrofit Isolation approach - assuming all parameters are being measured.
Please refer to the current IPMVP Core Concepts Chapter 8 for more guidance in selecting IPMVP Options as well as Chapter 9.2 regarding Option B applications and methodology.

Q: I have a question about which IPMVP Option most closely aligns with our M&V strategy. To perform M&V at our customer's sites, we install submeters to monitor and track the energy use of the addressed equipment/systems and aggregate that interval data to measure the energy savings. The percentage of whole building energy use captured by our submeters can vary building-to-building based on the impact of the addressed equipment/systems on whole building energy use. Rarely does the estimated energy savings of our projects fall below 5% of baseline whole building energy use. Based on our interpretation of IPMVP, we believe our M&V strategy most closely aligns with Option C because we are performing site-level M&V calculations using our aggregated submetering interval data. Is this interpretation correct? If not, which IPMVP option would most closely align with our M&V strategy?

A: Based on the above description of your M&V strategy, since you are describing the measurement boundary for the sub-meters NOT specifically at the EEM level, this approach would conceptually align with an IPMVP Option C Whole Facility approach - assuming your submeters will also be measuring non-EEM level energy consumption and demand, and associated potential changes, which must be accounted for in the details of the savings calculation methodology.
Please refer to the current IPMVP Core Concepts Chapter 8 for more guidance in selecting IPMVP Options as well as Chapter 9.3 regarding Option C applications and methodology.

Q:  I have a question about which IPMVP Option best aligns with our M&V strategy. We provide turnkey energy efficiency services to our commercial and industrial customers across their real estate portfolios. Our M&V strategy is to install submeters at each building to monitor and track the energy use of equipment/systems we address during our projects. For example, if we are only replacing 3 of the 5 packaged rooftop units at a retail building, then we will install submeters to only those 3 packaged rooftop units we replaced. To measure energy savings we aggregate our interval submeter data to eliminate potential impacts the non-addresses equipment/systems may have on our measured savings. Based on our interpretation of IPMVP, we believe our M&V strategy most closely aligns with Option C because we are using our aggregated submeter data to measure savings at a building-level. Is this interpretation correct? If not, which IPMVP Option most closely aligns with our M&V strategy?

A: Based on the above description of your M&V strategy, since you are describing the measurement boundary for the sub-meters at the EEM equipment specific level, this approach would conceptually align with an IPMVP Option B Retrofit Isolation approach - assuming all parameters are being measured.
Please refer to the current IPMVP Core Concepts Chapter 8 for more guidance in selecting IPMVP Options as well as Chapter 9.2 regarding Option B applications and methodology.

Q: I am from the UK and I am having difficulties in obtaining HDD and I wondered if there was a good source of information from this? The only information on the Government website seems to be for average HDD over the entire country and not for each relevant weather station.

A:  Yes, http://www.degreedays.net is a good source for airport weather stations in the UK. 

Q: I'm a graduate student at Texas A&M University.
I'm wondering if there is any acceptable calibration tolerance in recent IPMVP publications (for example, MBE & CV-RMSE values on a monthly & hourly basis).
I think the ASHRAE G14-2014 & 2015 FEMP provided these criteria, but I should be back to the 2002 IPMVP to get this criteria.

A:  HI, you are correct. Recommend continuing to refer to ASHRAE, FEMP and other Industry publications regarding acceptable calibration tolerance guidance for different applications. 

Q: I have a question regarding the choice of option A or option B. In my project, I will not directly measure all forces on the equipment, but I can calculate indirectly through engineering theories. The project involves the exchange of a mineral oil for a synthetic one in a reducer of a steel plate laminator. I'm not going to measure the forces directly on the laminating rollers, but by the reduction and material, I can calculate. The electric current will be measured directly in the electric motor. In a case like this, should I select option A or option B?

Another question is whether you certify/approve/review a project like this (obviously following the IPMVP to the letter) and how I apply it.

A: Based on the information provided, this appears to be an Option A. As described, if the motor kW is the one and only energy parameter, measuring kW of the motor pre and post exchange of the mineral oil, this could be considered an Option B under the following conditions:

Q: I have to do a regression for gas consumption in a mall in Romania under the whole facility approach.
I have 12-month data for total gas consumption and tenant consumption. I have subtracted both to get common area gas consumption. However, for the months May to Sep, the subtracted result of the gas consumption is zero. So, end result, I have 12 months of common area gas consumption, but, zero consumption for 5 months. In this scenario, is the regression done on the remaining 7-month data or should I consider the whole 12 months? (ASHRAE says no missing data in the baseline period, but is the calculated zero consumption considered as missing data?)

Also, HDD base 12.5 gives me a CV of 0.21. is that acceptable? My gas savings is 15%.

Kindly advise how to proceed. I have attached the file.

A: First - the calculated zero consumption should not be treated as missing data since it represents no usage. Second - the situation you describe is one that does occur for certain building types/climate zones. For example, when looking at a lot of gas data for a project in California about why natural gas savings projects are not participating in NMEC programs (which uses Option C). The issue is exactly this – low gas use during the warmer months causes the CV(RMSE) to blow up – because the average energy use of the baseline year is in the denominator of the CV(RMSE).

To get around this, we are recommending that the low use period data be separated from the high use period data, and separate baseline models developed from them. It would be useful to develop some sort of average temperature that defines low use from high use periods, e.g. when the average monthly temp is below X, then there is low usage. This would be helpful when making predictions of adjusted baseline use under reporting period conditions – knowing when to apply the low use model or the high-use model – described next:

Make a model using the low use period only, calculate the CV(RMSE) and NMBE, R2, etc. and check t-statistics of the model coefficients to make sure you have a good model. Repeat for the high use period. (Don’t rely too much on R2 as a criteria though).

There are some modeling algorithms that are piecewise linear such as ASHRAE’s change-point models. The 3-parameter model has two segments, a sloped portion and a flat portion. I would think that different software packages would find the temperature (or HDD) where these segments come together. Software packages that run change-point model exist:

Energy Explorer from Prof. Kelly Kissock https://academic.udayton.edu/kissock/http/research/energysoftware.htm

Energy Charting and Metrics tool (an Excel add-in, free) https://sbwconsulting.com/ecam/ (probably the best option for Excel users)

nmecr (R code implementation of multiple modeling algorithms) https://github.com/kW-Labs/nmecr (also free and for users of R software)