gh-112014: correct buffer protocol support in ctypes (use native formats)

[skirpichev]

This allows better interoperation with the memoryview, e.g. support for ctypes arrays. Just like NumPy arrays, they also don't specify endianness:

>>> import numpy as np
>>> memoryview(np.ndarray(3, dtype=np.float16)).format
'e'

But keep specified byteorder for byte-swapped types.

• Issue: Allow memoryview objects to support ctypes Arrays #112014

[vstinner]

I think that I can understand the rationale for this change, but it seems like @encukou has a different opinion on how the issue should be addressed: #112014 (comment).

[vstinner]

From this NEWS entry, it's uneasy to understand the rationale for the change. According to #112014 (comment), I understand that before, it was not possible to modify a ctypes type using memoryview, since the buffer format was rejected by memoryview when attempting to modify the view.

[skirpichev]

it's uneasy to understand the rationale for the change

It's a bug :-) Memoryview support for ctypes buffers is a side effect of the bugfix.

[skirpichev]

I think that I can understand the rationale for this change, but it seems like @encukou has a different opinion on how the issue should be addressed: #112014 (comment).

Well, obviously it's a breaking change. We should decide whether the current behavior is a bug or not. My arguments are in the issue thread: #112014 (comment)

[encukou]

As is, currently, this PR is wrong.
I have a 64-bit machine where long has 8-bytes:

>>> import ctypes
>>> ctypes.sizeof(ctypes.c_long)
8

Per struct docs, an array of native longs should use the format l, but with this PR, it uses q:

>>> memoryview((ctypes.c_long*8)()).format
'q'
>>> 

(Without this PR, it correctly uses <q, which means “8 bytes, little endian”.)

[bedevere-app]

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

[skirpichev]

You are right, probably I should use _ctypes_alloc_format_string_for_type() helper for swapped types. And just copy format code per default.

Unfortunately, ctypes and struct use "slightly different format codes" (c).

[skirpichev]

@encukou, this probably needs some more work to adjust and/or add tests. But I think it's ready for review as a concept.

[skirpichev]

I have made the requested changes; please review again.

[bedevere-app]

Thanks for making the requested changes!

@encukou: please review the changes made to this pull request.

[encukou]

OK, I'll review.

As I said in the issue: in anything that can be used for storage/interchange, I'd prefer to keep describing data using explicit sizes rather than platform-specific types.

While there are use cases where this change would help, it makes things less explicit. I don't think it is worth the behaviour change.

Feel free to discuss the concept in the issue.