Segmentation fault when deserializing malformed data

[michaelkl]

$ python -VV
Python 3.12.1 (main, Jan 11 2024, 10:03:43) [GCC 12.2.0]

$ pip install pyroaring
Collecting pyroaring
  Downloading pyroaring-1.0.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (10 kB)
Downloading pyroaring-1.0.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 6.1 MB/s eta 0:00:00
Installing collected packages: pyroaring
Successfully installed pyroaring-1.0.0

Hello,

I'm using bitmaps for packing large arrays of integers when communicating with other services. In other words, I'm receiving serialized bitmaps from a 3rd party and I need a way to tolerate malformed inputs.

But when I try to deserialize bytes that were manually corrupted, I get segmentation fault.

>>> import base64
>>> import pyroaring

>>> input_b64='OjAAAAEAAAAAAAkAEAAAADIAMwA0ADUANgA3ADgAOgA7ADwA'
>>> bitmap_bytes: bytes = base64.b64decode(input_b64)
>>> bitmap = pyroaring.FrozenBitMap.deserialize(bitmap_bytes)
>>> bitmap
FrozenBitMap([50, 51, 52, 53, 54, 55, 56, 58, 59, 60])

>>> input_b64='AjAAAAEAAAAAAAkAEAAAADIAMwA0ADUANgA3ADgAOgA7ADwA'
>>> bitmap_bytes: bytes = base64.b64decode(input_b64)
>>> bitmap = pyroaring.FrozenBitMap.deserialize(bitmap_bytes)
>>> bitmap
Segmentation fault (core dumped)

[lemire]

The issue is likely here:

cdef croaring.roaring_bitmap_t *deserialize_ptr(char *buff):
    cdef croaring.roaring_bitmap_t *ptr
    ptr = croaring.roaring_bitmap_portable_deserialize(buff)
    return ptr

The roaring_bitmap_portable_deserialize function may return NULL. (And so can roaring_bitmap_frozen_view.)

[lemire]

Here is how to more safely deserialize:

    t = roaring_bitmap_portable_deserialize_safe(serializedbytes, expectedsize);
    if(t == NULL) {
        // handle error
    }
    const char *reason_failure = NULL;
    if (!roaring_bitmap_internal_validate(t, &reason_failure)) {
        printf("safely deserialized invalid bitmap: %s\n", reason_failure);
        // handle error
    }

[lemire]

@michaelkl This being said, if you are deserializing data from untrusted sources, you should use secured hashing. Even if the software does not crash, it is bad practice to load data from untrusted sources without verification of the content.

[michaelkl]

Hi @lemire! Thanks!

Well, the issue here is not that I'm communication with untrusted sources. They are all internal microservices, after all. Besides, I'm not trying to send something important, just a list of integers. :-)

I just don't want my Python service to go into segmentaion fault every time someone on the other side makes an error or sends a wrong piece of data. Especially considering "the other side" is not written in Python. 🤯

I'd expected it to throw some kind of exception in that case.

[lemire]

@michaelkl In this instance, I think that @Ezibenroc can make the lib much safer with little work, following my comments above. I would still recommend hashing for binary data in general. E.g., your payload should include a hash value + the binary data.

[Ezibenroc]

Hello,
Thank you both, I will try to have a look (unless you want to make a pull request of course, this would be even better !)

[lemire]

See https://github.com/Ezibenroc/PyRoaringBitMap/pull/125/files

[michaelkl]

Hey guys! I was wondering if there are any plans on a new release incorporating this fix any time soon. 1.0.1 maybe? 🤗

[Ezibenroc]

Hello!
Yes, sorry, I was trying to fix the release workflow and did not find much time for that (I suspect the new armv7l architecture we added since previous release).
I will try to have a look this week (perhaps temporarily removing this architecture).

[Ezibenroc]

Hey @michaelkl, I have just released 1.0.1 on pypi 😉

[michaelkl]

@Ezibenroc Awesome news! Thanks so much!