From c6f77d08818e2fb8a0a525047d56e32d4bcfa632 Mon Sep 17 00:00:00 2001 From: Filipe Rodrigues Date: Thu, 5 Oct 2023 20:43:40 +0100 Subject: [PATCH] Added some extra documentation for the project. --- .vscode/settings.json | 3 ++ README.md | 33 +++++++++++++++ docs/exec.md | 97 +++++++++++++++++++++++++++++++++++++++++++ docs/index.md | 16 +++++++ docs/tim.md | 69 ++++++++++++++++++++++++++++++ 5 files changed, 218 insertions(+) create mode 100644 docs/exec.md create mode 100644 docs/index.md create mode 100644 docs/tim.md diff --git a/.vscode/settings.json b/.vscode/settings.json index c29b0013a..401027659 100644 --- a/.vscode/settings.json +++ b/.vscode/settings.json @@ -84,6 +84,8 @@ "Seraphimon", "SkullGreymon", "sllv", + "spritesheet", + "spritesheets", "SPRT", "srav", "srlv", @@ -93,6 +95,7 @@ "submode", "Taomon", "thiserror", + "tims", "Veemon", "WarGreymon", "WarGrowlmon" diff --git a/README.md b/README.md index b77da3a17..64e2a1e44 100644 --- a/README.md +++ b/README.md @@ -4,6 +4,34 @@ This is a full decompilation project of the PS1 video games "Digimon World 3" / The name stands for "Decompilation - Digimon World 3/2003" +# Status + +Currently the following is done: + +- ✅: Done +- +- 🚧: In work +- 🛑: Not started + +| Project | Status | Comments | +| ------------------ | ------ | --------------------------------------------------------------------------------------------------------------------- | +| Digimon world 3 | 🛑 | For now, 2003 is receiving the main focus. It should be easy to make it catch up to it, but it will require some time | +| Digimon world 2003 | 🚧 | See [this section](##2003) | + +## 2003 + +| Project | Status | Comments | +| ------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| Executable (and dylibs) | 🚧 | Fully disassembled in `asm/`. Some parts have been organized, but it's mostly still just machine-generated. | +| Images (`tim`s) | ✅ (🚧) | All found `tim`s have been fully converted to pngs with their `clut`s as separate png files in `tim/`. Further work should be done here to convert them to a single file | +| Localization files (lang files) | ✅ (🚧) | All found localization files are in `lang_file/`. Some escape codes are currently unknown and need to be documented | +| Video (`str`s) | 🛑 | They remain in their original `str` file format. In the future we would want to convert them to another file format that would support lossless conversion to `str`. | +| Other files | 🛑 | Many other files remain in `dw2003/` that are simply being copied over. These need to be reverse-engineered. | + +# Documentation + +See [`docs/index.MD`]() for an introduction to the repository. + # Building The project is built using [`zbuild`](https://github.com/Zenithsiz/zbuild). (zbuild 0.1.1). @@ -17,7 +45,12 @@ You can run `zbuild` (without arguments) to build the game's bin/cue files. The > of the licenses required (they should be placed in the `licenses/` directory): > > - `psexe-eu.dat` / `psexe-na.dat`: PS-EXE executable licenses for Europe / North America, respectively. +> +> These are just text files with the text `Sony Computer Entertainment Inc. for ` (no newline). +> > - `psx-bin.dat`: ISO Sony license +> +> This is a binary file, of length 0x6d80, which `mkpsxiso` expects. You can also use `zbuild compare` to build all game files and check their checksum. diff --git a/docs/exec.md b/docs/exec.md new file mode 100644 index 000000000..47c1e9b6e --- /dev/null +++ b/docs/exec.md @@ -0,0 +1,97 @@ +# Executable + +The games feature an executable and various dylibs (dynamic libraries) that are loaded at runtime. + +The dynamic library loading is still mostly unknown. + +## File format + +The executable is a `psexe`, which is simply a header following by machine instructions (called `text` section). + +The header has the following structure: + +| Name | Offset | Type / Size (bytes) | Description | +| ------------- | ------ | ------------------- | --------------------------------------------------------------- | +| Start address | 0x0 | `u32` (0x4) | The address at which the cpu will start executing instructions. | +| Text base | 0x4 | `u32` (0x4) | The location to place the `text` section at | +| Text size | 0x8 | `u32` (0x4) | The size of the `text` section | +| Stack pointer | 0x8 | `u32` (0x4) | What to initialize the stack pointer to | +| License | 0xc | `[u8]` (0x7b4) | License | + +In total, it has `0x800` bytes. + +Meanwhile, the dynamic libraries are called `raw_exe`. i.e. they are simply machine instructions. + +## Usage + +### `psexe` Executables + +Executables are built from elf files, by using their `text` section as the `text` section of the psexe. All fields are automatically filled. + +> TODO: The stack pointer can't be currently customized, should we allow it? Either way the game sets the stack pointer itself, so it shouldn't matter. + +They are built by `ddw3-mkpsexe` from `yaml` header files, with the following structure: + +```yaml +# The elf file +elf: + +# License +license: + +# If the text section should be resized truncated / filled with zeroes +# (Optional) +resize_text: +``` + +Then using `build/tools/ddw3-mkpsexe --output --license --resize-text ` +(The `zbuild` file automatically reads the yaml and passes the correct arguments) + +> TODO: Although this _can_ be done in zbuild, maybe we should use an external python script, since it would yield better errors on missing fields, as well as be faster and simpler by not relying on invocations of `yq` and `tools/process_path.py`. + +### `raw_exe` Dynamic libraries + +Dynamic libraries are also built from elf files, by using their `text` section, similarly to `psexe`s. However, they do not have a header, so no extra input is required. + +They are built using `objcopy --dump-section=.text= `. + +### Elf + +Elf files are created from a list of object files. Each object files will have multiple sections that will be flattened into `text` using the provided order (with a linker script). + +They are built by `ddw3-mkpsexe` from `yaml` header files, with the following structure: + +```yaml +entry: +start_addr: +objs: +sections: +``` + +Then using `tools/ld_from_yaml.py -o --linker-script-output --ld-bin `. + +### Object files (assembly) + +Assembly object files are created from a single assembly file using `as -o -EL -mabi=32 -mips1 -march=r3000 -msoft-float -g -Iasm/ -I./ -I/ --strip-local-absolute` + +### Object files (rust) + +Rust files are created from rust crates. + +They are built using `cargo build --release --package= -Zunstable-options` (inside `rust/`). + +This will create the library in `rust/target/mipsel-sony-psx/release/lib.a`. + +## Organization + +The `psexe` headers are located in `/psexe`. + +The `raw-exe` headers are in `raw_exe`. + +The assembly sources are in `asm/`. + +The rust sources are in `rust/`. + +## Notes + +When creating `elf`s, we can't use the `.a` archives that rust produces for some reason. You can fix this by replacing it with `.o` instead, which is a file that `zbuild` creates from the `.a` to fix this problem. diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 000000000..f4cfaaac1 --- /dev/null +++ b/docs/index.md @@ -0,0 +1,16 @@ +# Documentation + +This directory contains several important documentation files for how the project works. + +## Topics + +- [Image files (tims)](tim.md) +- [Executables](exec.md) + +> TODO: Pack files and lang files. + +## General notes + +All `yaml` header files use absolute paths to mean root of project. You can use a double absolute path `//` to encode a truly absolute path. This however should only be done locally. The repo shouldn't require any absolute paths. + +> Note for windows users: I'm not sure how exactly this works under windows, you would need to test it. diff --git a/docs/tim.md b/docs/tim.md new file mode 100644 index 000000000..9c2afab80 --- /dev/null +++ b/docs/tim.md @@ -0,0 +1,69 @@ +# Image files (`tim`) + +The games use the `tim` file format to store all images. + +## File format + +Supports either: + +- Indexed images (4-bit or 8-bit indexes with 16-bit (`R5G5B5A1`) colors) +- Color images (16-bit (`R5G5B5A1`) or 24-bit (`R8G8B8`)) + +They also store an x, y position to place the image at. If they are indexed, the clut (Color LookUp Table, or palette) also has an x, y position to place. + + + +Color images / `clut`s are mapped directly to VRAM at their respective position, so they mustn't overlap. + +## Usage + +They are built by `ddw3-mktim` from `yaml` header files, with the following structure: + +```yaml +bpp: Indexed4 | Indexed8 | Color16 | Color24 + +# Palette data +# Note: Only used if `Indexed4` or `Indexed8` +clut: + # Position to store the clut at + pos: + - + - + + # Kind of palette: + # - `include`: Uses the clut to generate the indices + # for the image, and will include the clut. + # - `external`: Uses the clut to generate the indices + # for the image, but will *not* include the clut. + # - `auto`. Automatically calculates the clut from + # the given image, and includes it. + kind: include + path: 0.clut.png + +# Image data +img: + # Note: Required even when using indexed images + pos: + - + - + + # Path to the image + # Note: Even when indexed, this should be a png. + # The indices will be calculated from the clut. + path: + +``` + +Then using `build/tools/ddw3-mktim --output ` + +## Organization + +The header files are stored in `tim/`, alongside the image files themselves. + +> TODO: Maybe they should be split? Especially so we can create them from files other than png without cluttering `tim/`. For example when we want to make a spritesheet / animation, it might be worth it to have a `img/` directory for status images, `spritesheet/` for spritesheets and `anim/` for animations or something like that. + +# Notes + +The games only ever seem to use 4-bit indexed images, always with their clut. + +Using `auto` for the clut kind is currently unsupported. This should be fixed.