Area Collision Data - Remove || Swap

[BlueMue]

This is a tutorial on how to remove the area collision data of any area of Shenmue, or, how to swap that data between two areas.

Video demonstration (of the collision removal):

phpBB [video]

Steps:
- Open with a hex-editor the MAPINFO.BIN file of the area you want. In our case, we will work with the file of Dobuita, found in folder "SCENE/0X/D000" (0X = 01 or 02 or 03 ...).

- Search (CTRL+F) for "COLS" and find the first reference. Start highlighting from COLS and go all the way down ...



- ... until you find the first reference of "REGD". Stop exactly before REGD (that is not select any part from REGD or after it).



- ... With all these lines being highlighted, press the Delete (DEL) button in your keyboard.

- Save the modified file. [Proceed to the necessary steps to create a self-bootable image file.]

About Swapping the clipping information between two MAPINFO.BIN files of two different areas:
- The only difference from the above steps, is to not Delete the highlighted part of the file, but Cut (CTRL+X) it and move it to the other file that you want to swap the clipping information with.
- That means you have to replace the "COLS ... (until before REGD)" lines of the one file, with the "COLS ... (until before REGD)" lines of the other file.

===

Things to improve/fix:
- Programmable camera movements and angles of a specific area has been removed. That is, for example, when you go to a parking lot to practice or to a vending machine to buy a drink, you will not see the usual camera movements and angles. FIXED!
- Wrap points has been removed. You will not be able to enter another area (unless, for example, if you open the door to enter the Hazuki Residence), or enter the Tomato Convenience Store, the You Arcade et cetera. FIXED!
- The vertical collision data is also removed, thus gravity is off and, in areas like Yamanose, the height of the player (Ryo) is not consistent with the height of the ground. FIXED!
- You can still not pass through some objects/models, like: humans, bikes, shutters (when the shops close) et cetera.

Notes:

- Even as a "young'n", newbie to Shenmue modding/hacking, while working on that I learned some things: go for the *crazy* way, try different (crazy, all and any) ways, trust your hunches, pay really close attention to details, think like the programmer (how they could have put things in a file et cetera).

- My thought process (wall-of-text): First, I thought to use the MAPINFO.BIN file of the folder 0000. That file doesn't contain any clipping information. I compared it visually with other mapinfo files and noticed some parts look similar (like the one from REGD and below). First I replaced in the mapinfo of Dobuita (D000, that was the area to make all my experiments) all the lines from the top until before REGD, with the ones from the mapinfo of 0000 folder, that is the lines from the top until before REGD. The end result was this. As you can see, a lot of textures, models et cetera has been removed. Then I thought about mapinfo swapping, like when you do it to access the beta versions of some areas (like Yamanose's and Sakuragaoka's). There you have to copy from after the ECAM part until the end. And thus I thought that before the ECAM part, there is all the information about textures, models et cetera. Combined with the thought that the part from REGD until the end remains the same (like with the case when I swapped D000's with 0000's), I end up removing the part from ECAM (including ECAM) until before REGD. [I removed ECAM, too, because I thought it is a command and so when it is triggered it tries to load something from the memory, which it can't find, when it has been removed from the file...] But end up doing that after several tries; first I removed the part after ECAM until before COLS, then from COLS util before REGD, and other tries. Also, I just removed the COLS, COLI et cetera parts, to see if that will do the job, too, but with no success. So, after several tries, I figured that you need to remove the lines only from COLS until before REGD (and so the part from ECAM until before COLS can remain intact). And, here's the general story of how it happened...

[BlueMue]

That is an impressive first step into the hacking territory! Really nice to see the back side of the hot dog truck.

It is not possible to buy drinks from the vending machine at all because of the missing camera motion or does it just happen without the motion? That would be really interesting to see. A whole new "view" on the vending machine!

[Giorgio]

It is not possible to buy drinks from the vending machine at all because of the missing camera motion or does it just happen without the motion? That would be really interesting to see. A whole new "view" on the vending machine! :D

Haha. But now that I checked it again, the camera movements and angles are working! Probably because of the last method applied (that is to not remove the "ECAM ... (until before COLS)" lines). Before indeed they didn't work. However, in both cases the vending machine animations work and you can buy drinks. [Maybe if you alter the motion files for the vending machines or remove a specific relevant event trigger, could make the vending machines to appear but not being functional (or, as you said, to buy drinks without the motions; indeed, it's crazy).]

[Giorgio]

By the way: Did you ever want to fight around the streets of Dobuita? Now you can! [Well, sort of... there isn't anyone to beat them up, hehe!]

phpBB [video]

Made thanks to the collision data removal hack. ;)

[BlueMue]

However, in both cases the vending machine animations work and you can buy drinks.

Would've loved to see how buying a drink looks that way. Anyway, the new video shows something great again.
I imagine it being such a pleasure to train your moves without the camera constantly circling around Ryo, switching around the controlls. Would be so great if you indeed can practice one move constantly without having to look out how to do the directional commands.

[shengoro86]

Thanks for all your hard work on this Giorgio!

[BlueMue]

UPDATE

How to retain the vertical collision(?) (or the height) data (and everything else, except the horizontal collision data):

- With a hex editor (for this example, the hex editor named HxD has been used), open the file where the collision data is stored (usually the MAPINFO.BIN file)

- Search (CTRL+F) for the keyword COLS

- Look for the keyword COLI (which is located some bytes after the COLS)

- Take note of the four bytes after COLI. You will see something like e.g.: D0 54 00 00. These four bytes (or word) are in the Little Endian format. Convert them to the Big Endian format (in the previous example, the value D0 54 00 00 will become 00 00 54 D0 when converted to the Big Endian format) and take note of that value.

- Place the cursor eight bytes after the offset of COLI. Press CTRL+E to select a block of bytes. Select the radio button of "Length" and in the relevant text area write the aforementioned four bytes (as they are converted in the Big Endian format), then press OK. Press the DEL button.

- Overwrite the aforementioned four bytes from whatever value they are to the new value: 00 00 00 00.

- Find how many bytes the COLS data section is. There are two ways to do this:
- - (1) Take note of the four bytes after COLS. You will see something like e.g.: 1C 55 01 00. These four bytes (or word) are in the Little Endian format. Convert them to the Big Endian format (in the previous example, the value 1C 55 01 00 will become 00 01 55 1C when converted to the Big Endian format) and take note of that value. Open a calculator which does hexadecimal computations. Subtract the value of the four bytes after COLS with the value of the four bytes after COLI [but first, both byte values should be converted in the Big Endian format; for the previous examples, you would need to substract 00 01 55 1C with 00 00 54 D0, and the result would be 00 01 00 4C (that is 00 01 55 1C minus 00 00 54 D0 equals 00 01 00 4C)]. You need to convert the result of the previous substraction to the Little Endian format (for the previous example, the value 00 01 00 4C will become 4C 00 01 00 when converted to the Little Endian format). Overwrite the four bytes after COLS with the converted to the Little Endian format value that you found as the result from the substraction.
- - (2) Place the cursor at the offset of COLS, continuously press down the left mouse button from that offset and move the cursor towards the bottom of the file until you reach the offset of REGD. Look for the "Length:" text at the bottom-center of the hex editor's window; it reports the length of the bytes you have selected. The value of the length is in the Big Endian format, so it needs to be converted in the Little Endian format [as explained how to do this previously]. Overwrite the four bytes after COLS with the converted to the Little Endian format value that you saw at the "Length:" text.

[shengoro86]

UPDATE

How to retain the vertical collision(?) (or the height) data (and everything else, except the horizontal collision data):

- With a hex editor (for this example, the hex editor named HxD has been used), open the file where the collision data is stored (usually the MAPINFO.BIN file)

- Search (CTRL+F) for the keyword COLS

- Look for the keyword COLI (which is located some bytes after the COLS)

- Take note of the four bytes after COLI. You will see something like e.g.: D0 54 00 00. These four bytes (or word) are in the Little Endian format. Convert them to the Big Endian format (in the previous example, the value D0 54 00 00 will become 00 00 54 D0 when converted to the Big Endian format) and take note of that value.

- Place the cursor eight bytes after the offset of COLI. Press CTRL+E to select a block of bytes. Select the radio button of "Length" and in the relevant text area write the aforementioned four bytes (as they are converted in the Big Endian format), then press OK. Press the DEL button.

- Overwrite the aforementioned four bytes from whatever value they are to the new value: 00 00 00 00.

- Find how many bytes the COLS data section is. There are two ways to do this:
- - (1) Take note of the four bytes after COLS. You will see something like e.g.: 1C 55 01 00. These four bytes (or word) are in the Little Endian format. Convert them to the Big Endian format (in the previous example, the value 1C 55 01 00 will become 00 01 55 1C when converted to the Big Endian format) and take note of that value. Open a calculator which does hexadecimal computations. Subtract the value of the four bytes after COLS with the value of the four bytes after COLI [but first, both byte values should be converted in the Big Endian format; for the previous examples, you would need to substract 00 01 55 1C with 00 00 54 D0, and the result would be 00 01 00 4C (that is 00 01 55 1C minus 00 00 54 D0 equals 00 01 00 4C)]. You need to convert the result of the previous substraction to the Little Endian format (for the previous example, the value 00 01 00 4C will become 4C 00 01 00 when converted to the Little Endian format). Overwrite the four bytes after COLS with the converted to the Little Endian format value that you found as the result from the substraction.
- - (2) Place the cursor at the offset of COLS, continuously press down the left mouse button from that offset and move the cursor towards the bottom of the file until you reach the offset of REGD. Look for the "Length:" text at the bottom-center of the hex editor's window; it reports the length of the bytes you have selected. The value of the length is in the Big Endian format, so it needs to be converted in the Little Endian format [as explained how to do this previously]. Overwrite the four bytes after COLS with the converted to the Little Endian format value that you saw at the "Length:" text.

OH HELL YES THANK YOU GIORGIO!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

I may have to update the explore mod.

[scarless landi]

By the way: Did you ever want to fight around the streets of Dobuita? Now you can! [Well, sort of... there isn't anyone to beat them up, hehe!]

phpBB [video]

Made thanks to the collision data removal hack. ;)

I laughed when you were in the actual streets. Was waiting for a car but I guess it was stripped in order to get this working.

Great job guys!

[Giorgio]

Was waiting for a car but I guess it was stripped in order to get this working.

Cars, NPCs etc are absent because, when you enter the Practice mode then, the program turns them off.

Some more raw observations (inside the COLS section data):

EVNT => Wrap points (they are used to change area, insert buildings like Tomato Convience Store and YOU Arcade).
UNDU => Height...
SOND => It messes with the height of the building's staircases...
PROP => It messes with the height of the building's staircases and with the walking/running animation on stairs.
@coli => Removes the horizontal collision data from everywhere, except from the 2nd floor of buildings (after you've climbed up their stairs).

By removing data sections (and carefully update the relevant values about the pointers, sizes etc), one can find what they are about for.

[BlueMue]

You are a genius, Giorgio! This was the last missing piece of the puzzle to make the no-clip exploration really perfectly easy and enjoyable.