Allowing non-fixed length generation

[sirmammingtonham]

A quick hack for generating longer than 1024 tokens (feeds a fraction of output as context until a truncation token is reached). Not the most efficient method but it gets the job done.

Non-fixed length generation works when length is None or 0, and the fraction of output used for context is specified by split_context

Hopefully this closes #2

[minimaxir]

oo

Will need to test this but it sounds neat!

[sirmammingtonham]

Thanks! I did some more testing and realized that there were a lot issues with my first commit, namely that the generation would use the same context for all samples in a batch and that generation would stop completely when one sample reached truncation for batch_size larger than 1. There was also an issue with it repeating the context twice in the output.

I just updated it with a new commit which should have fixed all the issues above and also allows for generation larger than 1024 tokens without a truncation term (it will stop generation when the output has reached that length).

This version hopefully runs faster as well (i realized that i could just take a slice of the out array as context instead of having to decode and re-encode it lol).

[saippuakauppias]

Any updates or ideas?