Rewrite symbol management system

CHANGELOG.md

@@ -1,5 +1,39 @@
 # splat Release Notes
 
+### 0.50.0
+
+* We have skipped to version 0.50.0, as this version may introduce several breaking changes and require some extra care when migrating
+* BREAKING CHANGES:
+  * Declaring an overlay symbol but not specifying its `segment:` or `rom:` in symbol_addrs is no loger accepted. A warning will be emitted and the symbol won't be used.
+  * Associating a symbol with a segment that is not within the Vram or Rom address space for that segment is no longer accepted.
+    * Use the new prioritized segment system to tell splat to use a symbol from a specific segment.
+  * Declaring a symbol without an associated segment and outside the Vram space of any global segments is no longer accepted.
+    * Use the new absoulte symbols system to declare symbols that aren't part of any real segment.
+  * `splat.utils.symbols.all_symbols_dict` and `splat.utils.symbols.all_symbols_ranges` have been removed.
+  * The algorithm for checking vram overlaps between global segments and overlays have been changed and it is more strict now. This means it is more likely to report conflicts, which may result in errors.
+    * A common source of errors is giving a vram address (usually `vram: 0xA4000040`) for the ipl3 segment on n64 projects. A recommened workaround is to either remove the vram for this segment or tag it with its own `exclusive_ram_id` (i.e. `exclusive_ram_id: ipl3`).
+  * Autogenerated symbol names may have changed
+    * Previously, splat failed to properly set some attributes required for autogenerated symbol naming and thus symbolized them incorrectly. For example, it only used the vram address (like `D_{VRAM}`) when it should have been using both the vram and rom addresses (`D_{VRAM}_{ROM}`) for naming overlay symbols. These attributes are now correctly set, resulting in improved symbol names. This may break existing symbols in C code, so please keep this in mind when migrating.
+* Rewrite symbol management system.
+  * Completely change how the internal symbol tracking system works, aiming to fix issues with overlapping addresses in overlays.
+  * Ideally this should fix most issues related to using a symbol from overlay A in overlay B even when both are marked with the same `exclusive_ram_id`.
+  * Because of this, splat is a lot more strict about user-declared symbols. From now on, splat will warn and discard symbols that it can't figure out the corresponding segment on its own, instead of just trying to use them whenever they fit. This is mostly an issue for declaring symbols for overalys with overlapping addresses (segments declared with `exclusive_ram_id`).
+* Add prioritized segment system.
+  * Allows to tell which segments should be prioritized when looking up for symbol references.
+  * This way a user can choose how to disambiguate references to multiple segments with overlapping addresses, for example, referencing a an address shared by multiple overlays (say A, B and C) by telling the segment to lookup symbols from segment B first instead of looking into any other arbitrarily.
+  * Also allows global segments to reference symbols from specific overlays.
+  * Each segment can have their own list of prioritized segments. This list can be declared with the `prioritized_segments` yaml attribute on the segment level.
+  * This attribute is only accepted for top-level segments, subsegments silently ignore this attribute.
+* Add absolute symbols system.
+  * Allows declaring symbols that are not part of any segment of the ROM.
+  * Declared with attribute symbol `absolute:True`.
+  * Symbols declared this way always take priority over everything else when looking up for symbol references.
+  * This should not be used to declare symbols that are part of the actual address space of the rom.
+* `alabel` is recognized as a symbol type.
+  * An alternative entrypoint for functions.
+  * Works similarly to branch labels, in the way that it is used to declare symbols in the middle of functions.
+* `spimdisasm` 1.42.1 or above is now required.
+
 ### 0.41.1
 
 * Mark `__romPos` as `HIDDEN`.
@@ -28,8 +62,8 @@
 
 ### 0.40.0
 
-* Append an `-include` directive to the generated elf dependency file.  
-  * Includes the dependency files for each object needed to build the target, simplifying dependency management on the build system.  
+* Append an `-include` directive to the generated elf dependency file.
+  * Includes the dependency files for each object needed to build the target, simplifying dependency management on the build system.
   * This behavior is enabled by default, and can be disabled with the new option `ld_dependencies_include`.
 
 ### 0.39.1

README.md

@@ -21,7 +21,7 @@ The brackets corresponds to the optional dependencies to install while installin
 If you use a `requirements.txt` file in your repository, then you can add this library with the following line:
 
 ```txt
-splat64[mips]>=0.41.1,<1.0.0
+splat64[mips]>=0.50.0,<1.0.0
 ```
 
 ### Optional dependencies

docs/Adding-Symbols.md

@@ -1,10 +1,13 @@
+# Declaring symbols
+
 Symbols (i.e. labelling a function or variable) are controlled by the `symbols_addrs.txt` file.
 
 The format for defining symbols is:
 
 ```ini
 symbol = address; // option1:value1 option2:value2
 ```
+
 e.g.
 ```ini
 osInitialize = 0x801378C0; // type:func
@@ -30,10 +33,12 @@ An optional `key:pair` list of settings, note that each option should be separat
 ### `type`
 
 Override splat's automatic type detection, possible values are:
+
 - `func`: Functions
 - `jtbl`: Jumptables
 - `jtbl_label`: Jumptables labels (inside functions)
 - `label`: Branch labels (inside functions)
+- `alabel`: Alternative function entrypoints (inside functions).
 - `s8`, `u8`: To specify data/rodata to be disassembled as `.byte`s
 - `s16`, `u16`: To specify data/rodata to be disassembled as `.short`s
 - `s32`, `u32`: To specify data/rodata to be disassembled as `.word`s (the default)
@@ -61,7 +66,9 @@ RawHuffmanTable = 0x8022E0E0; // type:symbol size:0x100
 
 ### `rom`
 
-The ROM offset for the symbol, useful (potentially mandatory) for symbols in overlays where multiple symbols could share the same VRAM address.
+The ROM offset for the symbol.
+
+It is mandatory to either specify at least this or [segment](#segment) for symbols in overlays where multiple symbols could share the same VRAM address. If both are missing, the symbol is assumed to be part of any of the non-overlay segments.
 
 **Example:**
 ```ini
@@ -70,13 +77,32 @@ create_particle_effect = 0x802D5F4C; // type:func rom:0x6E75FC
 
 ### `segment`
 
-Allows specifying to which specific segment this symbol belongs to, useful to disambiguate symbols from segments that share the same VRAM address. This name must be the same as the name of a segment listed in the yaml.
+Allows specifying to which specific segment this symbol belongs to, useful to disambiguate symbols from segments that share the same VRAM address.
+
+This name must be the same as the name of a segment listed in the yaml.
+The segment's Vram and Rom address must include the addresses of the given symbol.
+
+It is mandatory to either specify at least this or [rom](#rom) for symbols in overlays where multiple symbols could share the same VRAM address. If both are missing, the symbol is assumed to be part of any of the non-overlay segments.
 
 **Example:**
 ```ini
 sMenuTexture = 0x06004040; // segment:menu_assets
 ```
 
+### `absolute`
+
+Declares this symbol as being associated to no segment.
+
+Some symbols are special in the way they exist outside the address space of the ROM. Instead they are provided by the OS, hardware, etc. `absolute` can be used to name these kind of symbols.
+
+`absolute` symbols also have the property of being prioritized over everything else when looking up for address references.
+
+**Example:**
+
+```ini
+osTvType = 0x80000300; // absolute:True
+```
+
 ### `name_end`
 
 Emits a symbol after the end of the data of the current symbol. Useful to reference the end of an assembly symbol, like RSP data.

docs/Segments.md

@@ -642,3 +642,35 @@ This value expects the name of the other segment that should be paired to the cu
       - [0x2C6B0, .rodata, libultra/audio/init_15550]
       # -- snip --
 ```
+
+### `prioritized_segments`
+
+A list of top-level segments.
+
+This segment is allowed to "see" the symbols from the segments on that list, even if it shouldn't given their `exclusive_ram_id`.
+
+This also allows control over how to disambiguate references over multiple overlay segments with overlaping addresses.
+
+Note the visibility is unidirectional; if you want two segments to see each other, you need to add a `prioritized_segments` on both, listing each other's names.
+
+```yaml
+  - type: code
+    name: volcano_assets
+    dir: volcano
+    start: 0x326C10
+    vram: 0x800FFF90
+    bss_size: 0x20
+    exclusive_ram_id: unk800FFF90
+
+  # -- SNIP --
+
+  - name: volcano_code
+    dir: volcano
+    type: code
+    start: 0x7272E0
+    vram: 0x802D60E0
+    bss_size: 0x170
+    exclusive_ram_id: level
+    prioritized_segments:
+      - volcano_assets
+```

pyproject.toml

@@ -1,7 +1,7 @@
 [project]
 name = "splat64"
 # Should be synced with src/splat/__init__.py
-version = "0.41.1"
+version = "0.50.0"
 description = "A binary splitting tool to assist with decompilation and modding projects"
 readme = "README.md"
 license = {file = "LICENSE"}
@@ -20,7 +20,7 @@ dependencies = [
 
 [project.optional-dependencies]
 mips = [
-    "spimdisasm>=1.41.0,<2.0.0", # This value should be keep in sync with the version listed on disassembler/spimdisasm_disassembler.py
+    "spimdisasm>=1.42.1,<2.0.0", # This value should be keep in sync with the version listed on disassembler/spimdisasm_disassembler.py
     "rabbitizer>=1.12.0,<2.0.0",
     "pygfxd>=1.0.5",
     "n64img>=0.3.3",

requirements.txt

@@ -4,7 +4,7 @@ tqdm==4.67.1
 intervaltree==3.1.0
 colorama==0.4.6
 # This value should be keep in sync with the version listed on disassembler/spimdisasm_disassembler.py and pyproject.toml
-spimdisasm>=1.41.0
+spimdisasm>=1.42.1
 rabbitizer>=1.10.0
 pygfxd>=1.0.5
 n64img>=0.1.4

src/splat/__init__.py

@@ -1,7 +1,7 @@
 __package_name__ = __name__
 
 # Should be synced with pyproject.toml
-__version__ = "0.41.1"
+__version__ = "0.50.0"
 __author__ = "ethteck"
 
 from . import util as util

src/splat/disassembler/spimdisasm_disassembler.py

@@ -7,7 +7,7 @@
 
 class SpimdisasmDisassembler(disassembler.Disassembler):
     # This value should be kept in sync with the version listed on requirements.txt and pyproject.toml
-    SPIMDISASM_MIN = (1, 41, 0)
+    SPIMDISASM_MIN = (1, 42, 1)
 
     def configure(self):
         # Configure spimdisasm

src/splat/scripts/split.py

@@ -3,17 +3,24 @@
 import argparse
 import hashlib
 import importlib
-from typing import Any, Dict, List, Optional, Set, Tuple, Union
+from typing import Any, Dict, List, Optional, Tuple, Union
 from pathlib import Path
 
 from collections import defaultdict, deque
 
 from .. import __package_name__, __version__
 from ..disassembler import disassembler_instance
-from ..util import cache_handler, progress_bar, vram_classes, statistics, file_presets
+from ..util import (
+    cache_handler,
+    progress_bar,
+    vram_classes,
+    statistics,
+    file_presets,
+    metadata,
+)
 
 from colorama import Style
-from intervaltree import Interval, IntervalTree
+from intervaltree import IntervalTree
 import sys
 
 from ..segtypes.linker_entry import (
@@ -168,29 +175,6 @@ def initialize_segments(config_segments: Union[dict, list]) -> List[Segment]:
     return ret
 
 
-def assign_symbols_to_segments():
-    for symbol in symbols.all_symbols:
-        if symbol.segment:
-            continue
-
-        if symbol.rom:
-            cands: Set[Interval] = segment_roms[symbol.rom]
-            if len(cands) > 1:
-                log.error("multiple segments rom overlap symbol", symbol)
-            elif len(cands) == 0:
-                log.error("no segment rom overlaps symbol", symbol)
-            else:
-                cand: Interval = cands.pop()
-                seg: Segment = cand.data
-                seg.add_symbol(symbol)
-        else:
-            cands = segment_rams[symbol.vram_start]
-            segs: List[Segment] = [cand.data for cand in cands]
-            for seg in segs:
-                if not seg.get_exclusive_ram_id():
-                    seg.add_symbol(symbol)
-
-
 def brief_seg_name(seg: Segment, limit: int, ellipsis="…") -> str:
     s = seg.name.strip()
     if len(s) > limit:
@@ -294,11 +278,10 @@ def initialize_all_symbols(all_segments: List[Segment]):
     symbols.initialize(all_segments)
     relocs.initialize()
 
-    # Assign symbols to segments
-    assign_symbols_to_segments()
+    metadata.segment_manager.initialize(all_segments, symbols.all_symbols)
 
     if options.opts.is_mode_active("code"):
-        symbols.initialize_spim_context(all_segments)
+        symbols.initialize_spim_context(metadata.segment_manager.manager)
         relocs.initialize_spim_context()
 
 
@@ -482,7 +465,7 @@ def write_undefined_funcs_auto():
         to_write = [
             s
             for s in symbols.all_symbols
-            if s.referenced and not s.defined and s.type == "func"
+            if (s.referenced or s.user_declared) and not s.defined and s.type == "func"
         ]
         to_write.sort(key=lambda x: x.vram_start)
 
@@ -494,9 +477,9 @@ def write_undefined_syms_auto():
         to_write = [
             s
             for s in symbols.all_symbols
-            if s.referenced
+            if (s.referenced or s.user_declared)
             and not s.defined
-            and s.type not in {"func", "label", "jtbl_label"}
+            and s.type not in {"func", "label", "jtbl_label", "alabel"}
         ]
         to_write.sort(key=lambda x: x.vram_start)
 

src/splat/segtypes/common/bss.py

@@ -118,7 +118,10 @@ def disassemble_data(self, rom_bytes: bytes):
 
         for spim_sym in self.spim_section.get_section().symbolList:
             symbols.create_symbol_from_spim_symbol(
-                self.get_most_parent(), spim_sym.contextSym, force_in_segment=True
+                self.get_most_parent(),
+                self,
+                spim_sym.contextSym,
+                force_in_segment=True,
             )
 
     def should_scan(self) -> bool:

src/splat/segtypes/common/c.py

@@ -269,7 +269,9 @@ def split(self, rom_bytes: bytes):
                             or options.opts.disassemble_all
                         ):
                             rodata_sym = self.get_symbol(
-                                spim_rodata_sym.vram, in_segment=True, local_only=True
+                                spim_rodata_sym.vram,
+                                in_segment=True,
+                                local_only=True,
                             )
                             assert rodata_sym is not None
 
@@ -470,7 +472,9 @@ def create_c_file(
             else:
                 for spim_rodata_sym in entry.rodataSyms:
                     rodata_sym = self.get_symbol(
-                        spim_rodata_sym.vram, in_segment=True, local_only=True
+                        spim_rodata_sym.vram,
+                        in_segment=True,
+                        local_only=True,
                     )
                     assert rodata_sym is not None