we should limit the `n` argument for EVP_EncodeBlock() to some sane value

[Sashan]

The EVP_EncodeBlock(3ossl) description reads as follows

int EVP_EncodeBlock(unsigned char *t, const unsigned char *f, int n);

     EVP_EncodeBlock() encodes a full block of input data in f and of length n
     and stores it in t. For every 3 bytes of input provided 4 bytes of output
     data will be produced. If n is not divisible by 3 then the block is
     encoded as a final block of data and the output is padded such that it is
     always divisible by 4. Additionally a NUL terminator character will be
     added. For example if 16 bytes of input data is provided then 24 bytes of
     encoded data is created plus 1 byte for a NUL terminator (i.e. 25 bytes
     in total). The length of the data generated without the NUL terminator is
     returned from the function.

     EVP_EncodeBlock() returns the number of bytes encoded excluding the NUL
     terminator.

The function may behave insanely when n appraches 2GB limit (0x7fffffff bytes).

There are two implementations in OpenSSL the standard variant without AVX2
optimizations (evp_encodeblock_int()) and variant optimized for AVX2 CPUs:
encode_base64_avx2()

The snippet here comes from encode_base64_avx2()

480 int encode_base64_avx2(EVP_ENCODE_CTX *ctx, unsigned char *dst,
481     const unsigned char *src, int srclen, int ctx_length,
482     int *final_wrap_cnt)
...
495     /* Process 96 bytes at a time */
496     for (; i + 100 <= srclen; i += 96) {
497         _mm_prefetch((const char *)(input + i + 192), _MM_HINT_T0);

I think the condition at line 496 deserves some attention w.r.t. signed int overflow, which
happens as srclen approaches 2GB limit. The non-AVX2 variant suffers from
a similar error at line 219:

126
127 int evp_encodeblock_int(EVP_ENCODE_CTX *ctx, unsigned char *t,
128     const unsigned char *f, int dlen, int *wrap_cnt)
129 {
...
218     } else {
219         for (i = 0; i + 2 < dlen; i += 3) {
220
221             t1 = f[i];

Use Makefile here to build test program:

main: main.c 2gigs.txt base64.txt
	$(CC) -g -O0 $(CFLAGS) \
	    -I${OPENSSL_HEADERS} -L${OPENSSL_LIB_PATH} -lcrypto -lssl \
	    -o main main.c

2gigs.txt:
	dd if=2gigs.txt bs=1m count=4000

base64.txt:
	dd if=base64.txt bs=1m count=4000

clean:
	rm -f main
	rm -f 2gigs.txt
	rm -f base64.txt

The test program below maps 2GB (0x7fffffff bytes) from file. It also maps output buffer to output file.

#include <stddef.h>
#include <stdio.h>
#include <stdint.h>
#include <fcntl.h>
#include <unistd.h>
#include <err.h>

#include <openssl/evp.h>

#include <sys/mman.h>

int
main(int argc, const char *argv[])
{
	int in, out;
	unsigned char *inbuf, *outbuf;
	size_t in_sz = 0x7fffffff;
	size_t out_sz = in_sz + (in_sz >> 1);
	int encoded;

	in = open("2gigs.txt", O_RDWR);
	if (in == -1)
		err(1, "2gigs:");

	out = open("base64.txt", O_RDWR|O_APPEND);
	if (out == -1)
		err(1, "base64:");

	/*
	 * request 2GB to mmap
	 */
	inbuf = mmap(NULL, in_sz, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FILE, in, 0);
	printf("%d @ %p\n", outbuf[in_sz - 1], &inbuf[in_sz -1]);
	if (inbuf == MAP_FAILED)
		err(1, "inbuf:");

	outbuf = mmap(NULL, out_sz, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FILE, out, 0);
	printf("%d @ %p\n", outbuf[out_sz - 1], &inbuf[out_sz -1]);
	if (outbuf == MAP_FAILED)
		err(1, "outbuf");

	encoded = EVP_EncodeBlock(outbuf, inbuf, in_sz);
	printf("in: %zu\tencoded: %d\n", in_sz, encoded);

	close(in);
	close(out);

	return (0);
}

the program crashes with stack as follows:

(gdb) r
Starting program: /home/sashan/scratch/main 
0 @ 0xcb770749ffe
0 @ 0xcb7b0749ffd

Program received signal SIGSEGV, Segmentation fault.
0x00000cb6e829e06b in evp_encodeblock_int (ctx=0x0, t=0xcb81b1f4aa8 "", f=0xcb6f074a000 "", dlen=2147483647, wrap_cnt=0x7990a63e6308) at crypto/evp/enc_b64_scalar.c:223
223                 t3 = f[i + 2];
(gdb) bt
#0  0x00000cb6e829e06b in evp_encodeblock_int (ctx=0x0, t=0xcb81b1f4aa8 "", f=0xcb6f074a000 "", dlen=2147483647, wrap_cnt=0x7990a63e6308) at crypto/evp/enc_b64_scalar.c:223
#1  0x00000cb6e829ea45 in EVP_EncodeBlock (t=0xcb77074a000 'A' <repeats 200 times>..., f=0xcb6f074a000 "", dlen=2147483647) at crypto/evp/encode.c:481
#2  0x00000cb3f18d0c2f in main (argc=<optimized out>, argv=<optimized out>) at main.c:42
$1 = 2147483646
(gdb) p (i + 2)
$2 = -2147483648

To avoid the crash one has to fix calll to EVP_EncodeBlock(). Tweak below avoids crash for non-avx variant:

encoded = EVP_EncodeBlock(outbuf, inbuf, in_sz - 2); 

This is the output of fixed program:

(gdb) r
Starting program: /home/sashan/scratch/main 
0 @ 0xde4b6b1fffe
0 @ 0xde4f6b1fffd
in: 2147483647  encoded: -1431655768
[Inferior 1 (process 14963) exited normally]

I suspect in_sz - 100 will be needed for CPUs with AVX instruction where call is dipatched to encode_base64_avx2().