[FEATURE] *Robust state estimation algorithms*

[IrenaDk]

Describe the feature request

The Weighted Least Squares (WLS) algorithms currently implemented in PGM for state estimation rely on the assumption of a Gaussian distribution of the measurement error. However, in practice this may not always be the case, especially with the use of pseudo-measurements. There are robust WLS methods who reduce the influence of bad-data, outliers and non-Gaussians errors. Users of PGM, like DSA, will greatly benefit from robust WLS algorithms. This includes (not exhaustive list) : Least Absolute Value, Huber estimator, Iteratively Reweighted Least Squares, Hampel etimator, Schweppe-Huber, GM-Tukey's biweight estimator, Quadratic-Linear and Quadratic-Constant. Some of these methods are implemented in other opensource engines, such as[ panda-power]

Community Meeting

• I wish to discuss this feature in the bi-monthly PGM community meeting. Please join the PGM community meeting.

[nitbharambe]

From a theoretical perspective some of these algorithms indeed benefit in case of pseudo measurements.
Implementation effort of some of these would be significant.
Maybe some of these algorithms build on top of other.

I imagine a scoped analysis is required for DSA's requirements, advantages, and if switching to a particular algorithm leads to a significant improvement in the desired state variable. PGM team then also needs to do an investigation on implementation effort.

[bferanmi806-sketch]

I looked through the current NR-SE path and wanted to check the scope before coding.

Would Huber M-estimation via IRLS for Newton-Raphson SE be a useful first slice of #1543?

It seems to fit the current solver well since NR-SE already rebuilds its gain/Jacobian each iteration. The main details I found are:

• apply Huber weights on top of the existing inverse-variance weighting;
• handle bus injections through the augmented R block rather than the direct HᵀWH path;
• keep zero-variance constraints untouched;
• retain the variance-normalization factor so standardized residuals still use the original statistical scale.

I’d keep the first implementation to NR-SE only, preserve WLS as default, support symmetric/asymmetric cases, and operate on PGM’s existing aggregated measurements.

Would that be a sensible first direction?

[petersalemink95]

Hi @bferanmi806-sketch,

Awesome to hear you're interested to contribute on this topic!

For us a great first step would be if you could provide a script to show your ideas.
That way we can verify the numerical side together.

Would you be willing to make a PoC repo on your GitHub in which you showcase your proposed solution?
Both Python, as well as C++ would be fine; main goal is to easily understand your proposal.

[bferanmi806-sketch]

Absolutely a PoC repo sounds like a good next step.

[bferanmi806-sketch]

@petersalemink95 I put together the PoC here:

https://github.com/bferanmi806-sketch/pgm-huber-irls-poc

It includes a small AC WLS baseline, Huber-IRLS, exact zero-injection constraints, and the variance-scaling handling we discussed.

One useful finding: Huber works well for the contaminated pseudo-measurement case, but a bad branch-flow measurement exposes a leverage-like failure where plain residual-based Huber does not protect the estimate. I documented both cases rather than hiding that limitation.

Would be interested in your thoughts on the numerical setup and whether comparing this against a leverage-aware approach such as SHGM would be the right next step.

[Jerry-Jinfeng-Guo]

Hi @bferanmi806-sketch , thank you for the PoC! We will take a deeper look once we are back in capacity. We will provide an answer as soon as we are able to have a discussion about it.

[Jerry-Jinfeng-Guo]

Hi @bferanmi806-sketch , thank you again for thinking along! After internal discussion, we have concluded that the scope of this feature request should be made in such a way that a thorough workout is to be carried out and the knowledge of specific choice of methods are to be well documented. For that reason, we intend to shift the context of this feature request to a more academic setting. Your input here is greatly appreciated!

[bferanmi806-sketch]

Thanks @Jerry-Jinfeng-Guo! That direction makes sense, especially given the numerical trade-offs we observed in the PoC.

I'd be happy to stay involved in the discussion and contribute to the investigation as the scope develops.

In the meantime, if there are any other issues or roadmap items where the team could use an extra hand, I'd be glad to help!