Support for chrono (#39) and jiff

[l-7-l]

#39
chrono version>= 0.4:

• NaiveTime
• NaiveDate
• NaiveDateTime
• DateTime<Utc>
• DateTime<FixedOffset>

jiff (version >= 0.2):

• Date
• Offset // offset was not added to the fuzz tests because offset doesn't implement serde
• Time
• Timestamp
• Zoned

crate version 0.6.9 -> 0.7.0
lz4_flex 0.11.2(yanked) -> 0.13.0

[l-7-l]

CC: @finnbear, @caibear 中秋快乐 happy Moon Festival🥮

[l-7-l]

🫣

[dsegovia90]

This would be lovely to have, nice work 👍🏻

[finnbear]

Sorry for the delay. I took a cursory look and found lots of unwrap, unchecked math, and unwrap_or that look suspiciously like they could trigger crashes or ignore malformed input. bitcode isn't supposed to crash, regardless of the input, or ignore malformed input.

I don't work with date libraries very often (I always use Unix time in my code) so I could be wrong about the existence or scope of the problem here.

ConvertFrom is generally best for infallible stuff (bijective mapping between "from" and "to"), whereas a custom decoder may be required to handle data with validity constraints. Remember: bitcode always validates data up-front while decoding, and then often does unwrap_unchecked to decode individual items. One way to validate your implementation is by adding the new types to the fuzzer.

TODO for us

It might be a good idea to introduce TryConvertFrom to make it easier, albeit slower, to add support for data with validity constraints.

Finally, for code style, we would prefer if each date library used its own date types (Second, Nanosecond, etc.), rather than factoring those into common code. Each library should have a self-contained implementation in ext/library_name.rs).

[l-7-l]

@finnbear

Sorry for the delay. I took a cursory look and found lots of unwrap, unchecked math, and unwrap_or that look suspiciously like they could trigger crashes or ignore malformed input. bitcode isn't supposed to crash, regardless of the input, or ignore malformed input.

I don't work with date libraries very often (I always use Unix time in my code) so I could be wrong about the existence or scope of the problem here.

ConvertFrom is generally best for infallible stuff (bijective mapping between "from" and "to"), whereas a custom decoder may be required to handle data with validity constraints. Remember: bitcode always validates data up-front while decoding, and then often does unwrap_unchecked to decode individual items. One way to validate your implementation is by adding the new types to the fuzzer.
TODO for us

It might be a good idea to introduce TryConvertFrom to make it easier, albeit slower, to add support for data with validity constraints.

Finally, for code style, we would prefer if each date library used its own date types (Second, Nanosecond, etc.), rather than factoring those into common code. Each library should have a self-contained implementation in ext/library_name.rs).

Sorry for the delay.

thank you for your patient and excellent review. It was very helpful to me.

Here is what I need to do:

1. Remove all uses of unwrap as much as possible.
2. Use ConvertFrom wherever possible.
3. Use the crate’s own types instead of common code (however, types such as Hour, Minute, Second, and Month are essentially the same across any time library. Thus, once it is confirmed that they will not change, I still recommend using common code—though keeping them separate does facilitate maintenance. Additionally, regarding support for the time library, I only changed types like year, month, day, hour, minute, and second to use common types; the rest of the code remains unchanged. Support for chrono was implemented by referring to the approach used for the time library.)

Is my understanding correct?

Best regards

[finnbear]

Remove all uses of unwrap as much as possible.

To be clear, the unwrap function (and unchecked math) is only permissible if it can never trigger a crash, regardless of the input. Whatever you do, please add the new types to the fuzzer in fuzz/ to validate this. Rather than unwrap_or(default), an error should be returned.

Use ConvertFrom wherever possible.

What I meant was ConvertFrom is probably not appropriate for date types, because they have validity constraints. The alternative is to write a new View/Decoder implementation that checks constraints and decodes in populate, storing the items in a CowSlice to return one by one in decode. We could make this easier by creating a new TryConvertFrom trait and decoder that abstracts away this complexity.

Use the crate’s own types instead of common code

Yes, please use separate types for the crate (whether from the crate or new types in bitcode). While it might seem unnecessary, it is the only way to guarantee compatibility with future time libraries.

[l-7-l]

@finnbear It seems that if we want to use TryFrom, both Decoder and Decode need to be changed…

impl<'a, F: TryConvertFrom<T>, T: Decode<'a>> TryDecoder<'a, F> for TryConvertFromDecoder<'a, T> {
    #[inline(always)]
    fn try_decode(&mut self) -> Result<F, crate::Error> {
        F::try_convert_from(self.0.decode())
    }
}

It seems that, if we don’t consider bit flips or data tampering, using unwrap is safe, since we’re just decoding data that we ourselves encoded. Also, implementing it in chrono
seems more convenient. Finally, chrono seems to be deprecated
, so I’m hesitating whether to continue with this PR. However, even if we don’t support chrono, supporting TryConvertFrom in the future would still be meaningful for converting other third-party library types into Option.

[finnbear]

if we don’t consider bit flips or data tampering

bitcode may return an error if the data is corrupt. it may not crash. data tampering may occur on a client, a potential counter example to "we're just decoding data that we ourselves encoded," and should not cause a server to crash. crashes on invalid data could be exploited to conduct a denial of service attack.

supporting TryConvertFrom in the future would still be meaningful

I agree

[l-7-l]

@finnbear It seems like I don't need ConvertTryFrom anymore

collapsed legacy code

```rust use crate::{ coder::{Decoder, View}, convert::{impl_convert, ConvertFrom, ConvertIntoEncoder}, error::error, int::{ranged_int, IntDecoder}, Decode, Encode, }; use jiff::{ civil::{DateTime, Time}, tz::TimeZone, Timestamp, };

// The value is guaranteed to be in the range 0..=23.
ranged_int!(Hour, u8, 0, 23);
// The value is guaranteed to be in the range 0..=59.
ranged_int!(Minute, u8, 0, 59);
// The value is guaranteed to be in the range 0..=59.
ranged_int!(Second, u8, 0, 59);
// The value is guaranteed to be in the range 0..=999_999_999
ranged_int!(Nanosecond, u32, 0, 999_999_999);

type TimeEncode = (u8, u8, u8, u32);
type TimeDecode = (Hour, Minute, Second, Nanosecond);

impl ConvertFrom<&Time> for TimeEncode {
fn convert_from(value: &Time) -> Self {
(
value.hour() as u8,
value.minute() as u8,
value.second() as u8,
value.subsec_nanosecond() as u32,
)
}
}

impl ConvertFrom for Time {
fn convert_from(value: TimeDecode) -> Self {
Time::constant(
value.0.into_inner() as i8,
value.1.into_inner() as i8,
value.2.into_inner() as i8,
value.3.into_inner() as i32,
)
}
}

type TimeStampEncode = (i64, i32);

impl ConvertFrom<&Timestamp> for TimeStampEncode {
fn convert_from(value: &Timestamp) -> Self {
(value.as_second(), value.subsec_nanosecond())
}
}

impl Encode for Timestamp {
type Encoder = ConvertIntoEncoder;
}

#[derive(Default)]
pub struct TimestampDecoder {
timestamp: Timestamp,
}
// Can't derive since it would bound T: Default.
// impl<'a> Default for TimestampDecoder<'a> {
// fn default() -> Self {
// Self {
// seconds: IntDecoder::default(),
// nanos: IntDecoder::default(),
// }
// }
// }

impl<'a> View<'a> for TimestampDecoder {
fn populate(&mut self, input: &mut &'a [u8], length: usize) -> Result<(), crate::Error> {
println!("---- length: {length} ----");

    // if self.seconds.is_none() {
    let mut decoder = IntDecoder::<i64>::default();
    decoder.populate(input, length)?;

    let seconds: i64 = decoder.decode();
    println!("---- seconds : {seconds} -----");
    // return Ok(());
    // }
    let mut decoder = IntDecoder::<i32>::default();
    decoder.populate(input, length)?;

    let nanos: i32 = decoder.decode();

    println!("---- seconds : {nanos} -----");
    match jiff::Timestamp::new(seconds, nanos) {
        Ok(x) => {
            self.timestamp = x;
        }
        Err(_) => {
            return Err(error("decode jiff::Timestamp failed: {}"));
        }
    }

    Ok(())
}

}

impl<'a> Decoder<'a, Timestamp> for TimestampDecoder {
fn decode(&mut self) -> Timestamp {
self.timestamp
}
}

impl<'a> Decode<'a> for Timestamp {
type Decoder = TimestampDecoder;
}

#[cfg(test)]
mod tests {
#[test]
fn test() {
// assert_eq!(timestamps, decoded);

    // 合法：正常秒和纳秒
    let bytes = bitcode::encode(&(0i64, 0i32));
    let ts: Timestamp = bitcode::decode(&bytes).unwrap();
    assert_eq!(ts, Timestamp::new(0, 0).unwrap());

    // 合法：秒最小值，纳秒 0
    let bytes = bitcode::encode(&(UNIX_SECONDS_MIN, 0i32));
    let ts: Timestamp = bitcode::decode(&bytes).unwrap();
    assert_eq!(ts, Timestamp::new(UNIX_SECONDS_MIN, 0).unwrap());

    // 合法：秒最大值，纳秒最大值
    let bytes = bitcode::encode(&(UNIX_SECONDS_MAX, NANOS_MAX));
    let ts: Timestamp = bitcode::decode(&bytes).unwrap();
    assert_eq!(ts, Timestamp::new(UNIX_SECONDS_MAX, NANOS_MAX).unwrap());

    // 非法：秒小于最小值
    let bytes = bitcode::encode(&(UNIX_SECONDS_MIN - 1, 0i32));
    let result: Result<Timestamp, _> = bitcode::decode(&bytes);
    assert!(result.is_err());

    // 非法：秒大于最大值
    let bytes = bitcode::encode(&(UNIX_SECONDS_MAX + 1, 0i32));
    let result: Result<Timestamp, _> = bitcode::decode(&bytes);
    assert!(result.is_err());

    // 非法：纳秒小于最小值（-1_000_000_000）
    let bytes = bitcode::encode(&(0i64, -1_000_000_000i32));
    let result: Result<Timestamp, _> = bitcode::decode(&bytes);
    assert!(result.is_err());

    // 非法：纳秒大于最大值（1_000_000_000）
    let bytes = bitcode::encode(&(0i64, 1_000_000_000i32));
    let result: Result<Timestamp, _> = bitcode::decode(&bytes);
    assert!(result.is_err());

    // 非法：秒为最小值，纳秒为负（组合约束）
    let bytes = bitcode::encode(&(UNIX_SECONDS_MIN, -1i32));
    let result: Result<Timestamp, _> = bitcode::decode(&bytes);
    assert!(result.is_err());

    // 合法：秒为最小值+1，纳秒为负
    let bytes = bitcode::encode(&(UNIX_SECONDS_MIN + 1, -500i32));
    let ts: Timestamp = bitcode::decode(&bytes).unwrap();
    assert_eq!(ts, Timestamp::new(UNIX_SECONDS_MIN + 1, -500).unwrap());

    // 合法：秒远大于最小值，纳秒为负
    let bytes = bitcode::encode(&(1000i64, NANOS_MIN));
    let ts: Timestamp = bitcode::decode(&bytes).unwrap();
    assert_eq!(ts, Timestamp::new(1000, -999_999_999).unwrap());

    // 合法：秒为最小值，纳秒为 0（已测）
    // 合法：秒为最小值，纳秒为正
    let bytes = bitcode::encode(&(UNIX_SECONDS_MIN, 500i32));
    let ts: Timestamp = bitcode::decode(&bytes).unwrap();
    assert_eq!(ts, Timestamp::new(UNIX_SECONDS_MIN, 500).unwrap());

    assert!(crate::decode::<Timestamp>(&crate::encode(&Timestamp::now())).is_ok());
}

use alloc::vec::Vec;
use jiff::Timestamp;

const UNIX_SECONDS_MIN: i64 = -377705023201;
const UNIX_SECONDS_MAX: i64 = 253402207200;
const NANOS_MIN: i32 = -999_999_999;
const NANOS_MAX: i32 = 999_999_999;

fn bench_data() -> Vec<Timestamp> {
    crate::random_data(1000)
        .into_iter()
        .map(|(s, n): (i64, i32)| loop {
            Timestamp::new(s % UNIX_SECONDS_MAX, n % NANOS_MAX).unwrap();
        })
        .collect()
}
crate::bench_encode_decode!(duration_vec: Vec<_>);

}

test result:
```sh
running 3 tests
---- length: 1 ----
---- seconds : 0 -----
---- seconds : 0 -----
---- length: 1 ----
---- seconds : -377705023201 -----
---- seconds : 0 -----
---- length: 1 ----
---- seconds : 253402207200 -----
---- seconds : 999999999 -----
---- length: 1 ----
---- seconds : -377705023202 -----
---- seconds : 0 -----
---- length: 1 ----
---- seconds : 253402207201 -----
---- seconds : 0 -----
---- length: 1 ----
---- seconds : 0 -----
---- seconds : -1000000000 -----
---- length: 1 ----
---- seconds : 0 -----
---- seconds : 1000000000 -----
---- length: 1 ----
---- seconds : -377705023201 -----
---- seconds : -1 -----
---- length: 1 ----
---- seconds : -377705023200 -----
---- seconds : -500 -----
---- length: 1 ----
---- seconds : 1000 -----
---- seconds : -999999999 -----
---- length: 1 ----
---- seconds : -377705023201 -----
---- seconds : 500 -----
---- length: 1 ----
---- seconds : 1773764525 -----
---- seconds : 530222042 -----
test ext::jiff::tests::test ... ok
test ext::jiff::tests::bench_duration_vec_decode has been running for over 60 seconds
test ext::jiff::tests::bench_duration_vec_encode has been running for over 60 seconds

------------------------------------------------ ** UPDATE ** ------------------------------------------------------------

use crate::{
    coder::{Decoder, View},
    convert::{ConvertFrom, ConvertIntoEncoder},
    error::error,
    int::{ranged_int, IntDecoder},
    Decode, Encode,
};
use jiff::{civil::Time, Timestamp};

// The value is guaranteed to be in the range `0..=23`.
ranged_int!(Hour, u8, 0, 23);
// The value is guaranteed to be in the range `0..=59`.
ranged_int!(Minute, u8, 0, 59);
// The value is guaranteed to be in the range `0..=59`.
ranged_int!(Second, u8, 0, 59);
// The value is guaranteed to be in the range `0..=999_999_999`
ranged_int!(Nanosecond, u32, 0, 999_999_999);

type TimeEncode = (u8, u8, u8, u32);
type TimeDecode = (Hour, Minute, Second, Nanosecond);

impl ConvertFrom<&Time> for TimeEncode {
    fn convert_from(value: &Time) -> Self {
        (
            value.hour() as u8,
            value.minute() as u8,
            value.second() as u8,
            value.subsec_nanosecond() as u32,
        )
    }
}

impl ConvertFrom<TimeDecode> for Time {
    fn convert_from(value: TimeDecode) -> Self {
        Time::constant(
            value.0.into_inner() as i8,
            value.1.into_inner() as i8,
            value.2.into_inner() as i8,
            value.3.into_inner() as i32,
        )
    }
}

type TimeStampEncode = (i64, i32);

impl ConvertFrom<&Timestamp> for TimeStampEncode {
    fn convert_from(value: &Timestamp) -> Self {
        (value.as_second(), value.subsec_nanosecond())
    }
}

impl Encode for Timestamp {
    type Encoder = ConvertIntoEncoder<TimeStampEncode>;
}

#[derive(Default)]
pub struct TimestampDecoder {
    seconds: Vec<i64>,
    nano: Vec<i32>,
    pos: usize,
}

impl<'a> View<'a> for TimestampDecoder {
    fn populate(&mut self, input: &mut &'a [u8], length: usize) -> Result<(), crate::Error> {
        self.seconds.clear();
        self.nano.clear();
        self.pos = 0;

        let mut sec_decoder = IntDecoder::<i64>::default();
        sec_decoder.populate(input, length)?;

        let mut nano_decoder = IntDecoder::<i32>::default();
        nano_decoder.populate(input, length)?;

        const SEC_MIN: i64 = -377705023201;
        const SEC_MAX: i64 = 253402207200;
        const NANOS_MIN: i32 = -999_999_999;
        const NANOS_MAX: i32 = 999_999_999;

        for _ in 0..length {
            let sec: i64 = sec_decoder.decode();
            let nano: i32 = nano_decoder.decode();
            if !(SEC_MIN..=SEC_MAX).contains(&sec) {
                return Err(error("seconds out of range"));
            }
            if !(NANOS_MIN..=NANOS_MAX).contains(&nano) {
                return Err(error("nanoseconds out of range"));
            }
            if sec == SEC_MIN && nano < 0 {
                return Err(error("nanoseconds must be >=0 when seconds are minimal"));
            }
            self.seconds.push(sec);
            self.nano.push(nano);
        }

        // self.seconds = seconds;
        // self.nano = nanos;

        Ok(())
    }
}

impl<'a> Decoder<'a, Timestamp> for TimestampDecoder {
    fn decode(&mut self) -> Timestamp {
        let sec = self.seconds[self.pos];
        let nano = self.nano[self.pos];
        self.pos += 1;

        Timestamp::constant(sec, nano)
    }
}

impl<'a> Decode<'a> for Timestamp {
    type Decoder = TimestampDecoder;
}

#[cfg(test)]
mod tests {
    #[test]
    fn test() {
        // 合法：正常秒和纳秒
        let bytes = bitcode::encode(&(0i64, 0i32));
        let ts: Timestamp = bitcode::decode(&bytes).unwrap();
        assert_eq!(ts, Timestamp::new(0, 0).unwrap());

        // 合法：秒最小值，纳秒 0
        let bytes = bitcode::encode(&(UNIX_SECONDS_MIN, 0i32));
        let ts: Timestamp = bitcode::decode(&bytes).unwrap();
        assert_eq!(ts, Timestamp::new(UNIX_SECONDS_MIN, 0).unwrap());

        // 合法：秒最大值，纳秒最大值
        let bytes = bitcode::encode(&(UNIX_SECONDS_MAX, NANOS_MAX));
        let ts: Timestamp = bitcode::decode(&bytes).unwrap();
        assert_eq!(ts, Timestamp::new(UNIX_SECONDS_MAX, NANOS_MAX).unwrap());

        // 非法：秒小于最小值
        let bytes = bitcode::encode(&(UNIX_SECONDS_MIN - 1, 0i32));
        let result: Result<Timestamp, _> = bitcode::decode(&bytes);
        assert!(result.is_err());

        // 非法：秒大于最大值
        let bytes = bitcode::encode(&(UNIX_SECONDS_MAX + 1, 0i32));
        let result: Result<Timestamp, _> = bitcode::decode(&bytes);
        assert!(result.is_err());

        // 非法：纳秒小于最小值（-1_000_000_000）
        let bytes = bitcode::encode(&(0i64, -1_000_000_000i32));
        let result: Result<Timestamp, _> = bitcode::decode(&bytes);
        assert!(result.is_err());

        // 非法：纳秒大于最大值（1_000_000_000）
        let bytes = bitcode::encode(&(0i64, 1_000_000_000i32));
        let result: Result<Timestamp, _> = bitcode::decode(&bytes);
        assert!(result.is_err());

        // 非法：秒为最小值，纳秒为负（组合约束）
        let bytes = bitcode::encode(&(UNIX_SECONDS_MIN, -1i32));
        let result: Result<Timestamp, _> = bitcode::decode(&bytes);
        assert!(result.is_err());

        // 合法：秒为最小值+1，纳秒为负
        let bytes = bitcode::encode(&(UNIX_SECONDS_MIN + 1, -500i32));
        let ts: Timestamp = bitcode::decode(&bytes).unwrap();
        assert_eq!(ts, Timestamp::new(UNIX_SECONDS_MIN + 1, -500).unwrap());

        // 合法：秒远大于最小值，纳秒为负
        let bytes = bitcode::encode(&(1000i64, NANOS_MIN));
        let ts: Timestamp = bitcode::decode(&bytes).unwrap();
        assert_eq!(ts, Timestamp::new(1000, -999_999_999).unwrap());

        // 合法：秒为最小值，纳秒为 0（已测）
        // 合法：秒为最小值，纳秒为正
        let bytes = bitcode::encode(&(UNIX_SECONDS_MIN, 500i32));
        let ts: Timestamp = bitcode::decode(&bytes).unwrap();
        assert_eq!(ts, Timestamp::new(UNIX_SECONDS_MIN, 500).unwrap());

        assert!(crate::decode::<Timestamp>(&crate::encode(&Timestamp::now())).is_ok());
    }

    use alloc::vec::Vec;
    use jiff::Timestamp;

    const UNIX_SECONDS_MIN: i64 = -377705023201;
    const UNIX_SECONDS_MAX: i64 = 253402207200;
    const NANOS_MIN: i32 = -999_999_999;
    const NANOS_MAX: i32 = 999_999_999;

    fn bench_data() -> Vec<Timestamp> {
        crate::random_data(1000)
            .into_iter()
            .map(|(s, n): (i64, i32)| Timestamp::new(s % UNIX_SECONDS_MAX, n % NANOS_MAX).unwrap())
            .collect()
    }
    crate::bench_encode_decode!(duration_vec: Vec<_>);
}

unning 3 tests
test ext::jiff::tests::test ... ok
test ext::jiff::tests::bench_duration_vec_encode ... ok
test ext::jiff::tests::bench_duration_vec_decode ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 205 filtered out; finished in 0.00s

 *  Terminal will be reused by tasks, press any key to close it. 

Can I submit a PR for Jiff(crates.io: Github:)?
Chrono is now in a soft-deprecated state, and the author of Chrono recommends using Jiff instead(link: ).
Downloads:

[l-7-l]

@finnbear cc

[finnbear]

Sorry for the delay. My review comments are above. Thanks very much for this PR!

[l-7-l]

@finnbear Greetings to you, my friend far away.
When would you like to merge this PR? (Please don't hesitate to let me know if anything needs to be fixed.)

May you be blessed with peace, health, and joy.

[finnbear]

Thanks again! Your changes look good to me. Before merging, @caibear and I will need to review more deeply, and I'm not sure when we'll have time to do that (could be weeks/months from now). If any further changes are required, such as not bumping the major version here, there is a good chance I'd personally make them during review, rather than requesting further changes.

Until then, I hope you can get some value from this PR using a patch in your Cargo.toml:

[patch.crates-io]
yew = { git = "https://github.com/l-7-l/bitcode.git", branch = "support_for_chrono#39" }

(this doesn't work if you're making a library that you plan to publish on Crates.io, but otherwise it should work)

[l-7-l]

Thank you, I will change the 0.70 version back to 0.69.

[TellerBewitch]

Thanks for the detailed breakdown of the required changes! It looks like you're making great progress on updating the chrono and jiff support. If you need help with the fuzz tests or any other aspect, feel free to reach out!