py/map: Convert map implementation to preserve insertion order.

[andrewleech]

Summary

py/map: Convert map implementation to preserve insertion order.

Testing

TBD

Trade-offs and Alternatives

TBD

[github-actions]

Code size report:

Reference:  docs/esp32: Replace 'esptool.py' by 'esptool' in command line example. [e4920d6]
Comparison: tests: Add ordered dict tests and benchmarks. [merge of 2197cfe]
  mpy-cross:   +64 +0.017% 
   bare-arm:   +36 +0.064% 
minimal x86:  +128 +0.068% 
   unix x64:  +696 +0.081% standard
      stm32: +7440 +1.886% PYBV10[incl +396(bss)]
      esp32:  +356 +0.020% ESP32_GENERIC[incl -32(data)]
     mimxrt:  +320 +0.083% TEENSY40
        rp2:  +544 +0.059% RPI_PICO_W
       samd:  +320 +0.116% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:  +606 +0.133% VIRT_RV32

[andrewleech]

/review

[mpy-reviewer]

Several blocking issues need addressing before this can land. The most critical are: an out-of-bounds read in mp_map_init_copy that ASAN catches on dict_fixed.py, a memory leak from m_del being called with the wrong byte size on every map dealloc/rehash, and heap-locked allocation failures in settrace_stackless (heapalloc.py, extreme_exc.py, heapalloc_super.py crash with MemoryError: memory allocation failed, heap is locked). DEVLOG.md should not be committed to the repository. Also missing: a MICROPY_PY_MAP_ORDERED compile-time toggle to select between implementations (noted as TODO in the DEVLOG itself).

CI Issues: sanitize_address fails on basics/dict_fixed.py (out-of-bounds read in mp_map_init_copy). settrace_stackless fails on micropython/heapalloc.py, micropython/extreme_exc.py, and micropython/heapalloc_super.py — all crash with MemoryError: memory allocation failed, heap is locked, indicating the new implementation triggers heap allocation during previously allocation-free operations.

[mpy-reviewer]

Don't commit this to the repo.

[mpy-reviewer]

src->table for an ordered map (including ROM maps) has only src->alloc * sizeof(mp_map_elem_t) bytes — there are no hash indices appended. Copying MP_MAP_TABLE_BYTE_SIZE(src->alloc) bytes is an out-of-bounds read. This is what ASAN catches in dict_fixed.py. Need:

size_t n = src->is_ordered
    ? src->alloc * sizeof(mp_map_elem_t)
    : MP_MAP_TABLE_BYTE_SIZE(src->alloc);

[mpy-reviewer]

m_del(mp_map_elem_t, old_table, old_alloc) frees old_alloc * sizeof(mp_map_elem_t) bytes, but the allocation was MP_MAP_TABLE_BYTE_SIZE(old_alloc) bytes. Memory leak of MP_MAP_INDEX_SIZE(old_alloc) * old_alloc bytes on every rehash. Same bug in mp_map_compact and in mp_map_deinit (which uses the pre-existing m_del call that now also has the wrong size). Need m_free(old_table, MP_MAP_TABLE_BYTE_SIZE(old_alloc)) at all three sites.

[mpy-reviewer]

After mp_map_compact(), map->used == map->filled and &map->table[map->used] is the next insertion slot. Storing the deleted value there with key = MP_OBJ_NULL is fragile — any reentrant lookup that inserts will clobber that slot before the caller reads slot->value. Pass the value out via an out-param or save it before calling compact.

[mpy-reviewer]

mp_map_new and mp_map_free are declared here but not defined anywhere in this diff.

[mpy-reviewer]

Should be < 256. With 1-based indexing, alloc=255 stores max index 255 which fits in uint8_t.

Suggested change

	#define MP_MAP_IS_UINT8(alloc) ((alloc) < 255)
	#define MP_MAP_IS_UINT8(alloc) ((alloc) < 256)

[mpy-reviewer]

Should default to 0 — most embedded targets won't need >65535-element dicts, and enabling this by default adds the uint32_t index paths to all builds.

[mpy-reviewer]

Adding a full size_t filled costs +4 or +8 bytes per mp_map_t. Every module dict, class dict, and instance dict pays this. Is there a way to track it more cheaply, e.g., derive it from used minus the tombstone count?