No predicted postings for most transactions

[sim642]

I have manually labelled a lot of data to train from, e.g.

2015-10-03 * "AS EESTI TELEKOM" ""
  Assets:Swedbank:Main  -5.75 EUR
  Expenses:Phone

2015-10-16 * "TARTU ÜLIKOOL" "Õppetoetus"
  Assets:Swedbank:Main  240.00 EUR
  Income:Uni-Tartu:Stipends

For new unlabeled data, I have an importer that gives

2016-01-04 * "AS EESTI TELEKOM" ""
  Assets:Swedbank:Main  -5.60 EUR

2016-01-07 * "TARTU ÜLIKOOL" "Õppetoetus"
  Assets:Swedbank:Main  240.00 EUR

If I also apply the PredictPostings hook to that and enable debug logging, it prints

DEBUG:smart_importer.predictor:Loaded training data with 1057 transactions, filtered from 1743 total transactions
DEBUG:smart_importer.predictor:Trained the machine learning model.
DEBUG:smart_importer.predictor:Apply predictions with pipeline
DEBUG:smart_importer.predictor:Added predictions to 187 transactions

The result is

2016-01-04 * "AS EESTI TELEKOM" ""
  Assets:Swedbank:Main  -5.60 EUR
  Expenses:Phone

2016-01-07 * "TARTU ÜLIKOOL" "Õppetoetus"
  Assets:Swedbank:Main  240.00 EUR

Actually over 90% of transactions are not labeled, even though smart_importer claims to add to all 187 transactions in the file.
Also, it's correctly set up because it does work for a handful of transactions, so it's not completely broken.

[sim642]

Looks like this might be the issue described here: beancount/beangulp#159 (comment). Beangulp is putting the new entries into existing_entries while deduplicating, which causes the new unlabeled transactions to also be used for training. This is utter nonsense: smart_importer is being trained on the new data and then called to predict on the same new data. No wonder it doesn't work if it just learned to predict nothing...

Apparently that has been the behavior for 2 years since beancount/beangulp#114. I'm surprised smart_importer has done anything for anyone for all this time.

[andreasgerstmayr]

Apparently that has been the behavior for 2 years since beancount/beangulp#114. I'm surprised smart_importer has done anything for anyone for all this time.

As a workaround, you can wrap the individual importers instead of using beangulp hooks.

[tarioch]

Is this really still the case with latest version? I'm using this with simple beangulp hook all the time and I'm not running into this issue.

[andreasgerstmayr]

Is this really still the case with latest version? I'm using this with simple beangulp hook all the time and I'm not running into this issue.

I'm using smart_importer 1.1 and beangulp 0.2.0 and hit this issue (with PredictPostings()). Some transactions are predicted also with the simple beangulp hook, but if I have many transactions to import it doesn't work well. According to beancount/beangulp#159 (comment) the set of new entries merged with existing entries is passed as training data to smart_importer - that would explain why the issue gets worse with a higher number of new transactions to be imported.

[tarioch]

Ok, I think I understand it now, I'm probably not hitting this issue, because the number of new entries are only a few vs the training data. If I see the linked issue in beangulp, is there anything we could do in smart importer to work around this?

[andreasgerstmayr]

If I see the linked issue in beangulp, is there anything we could do in smart importer to work around this?

For the PredictPostings() hook, I think a possible workaround would be to ignore transactions with a single posting from the training data. Basically we need to make sure that the set of new entries does not become part of the training data.

[tarioch]

Just added a filter for the training data to ignore single posting trx, which should hopefully fix this.

[andreasgerstmayr]

Just added a filter for the training data to ignore single posting trx, which should hopefully fix this.

Thank you very much @tarioch! It's working for me now with the beangulp hook method.