Shenmue Files, Emulation & Ripping Compendium
Preface
The following article is intended to be a compendium trying to explain the characteristics of the files encountered in Shenmue and their handling. It is based on facts and assumptions gathered by fans of the series. The article may not be complete or perfect. If you find any error or have an idea how to rewrite/correct parts of it then just let us know.
Chapter 1: File types encountered
The following is a listing of the file types encountered in Shenmue and what these files contain. Also file types not directly found on the disc, but related to Shenmue are mentioned.
DSF (PSF)
Dreamcast Sound Format is the proposed title of a digital music format for encoding Dreamcast music, similar to PlayStation Sound Format (PSF, also "Portable Sound Format). So far, such a format has not been publicly released.
The Header starts with "PSF". Dreamcast games don't contain DSP/PSF files, if such files are found they usually originate from someone who ripped the sounds already.
A DSF file contains a raw ARM7 sound program image which is loaded and copied to a specified address inside the program. The ARM7 then resets and executes the image.
DSF files can be loaded into a Winamp plugin called Highly Theoretical.
There is also the SSF, the "Saturn Sound Format".
DSF Files are usually manually (or in batch processes) created from MLT files.
A good way to convert DSF into MP3 or wav is using either vgmstream or a plugin for Foobar.
Good source of tools:
http://snesmusic.org/hoot/kingshriek/ssf/
DTPK
DTPK is an AM2 sound file. Usually uses the AICA sound driver.
There are some python scripts to extract DTPK sound files and the aica/am2 driver. Files start with "DTPK" if viewed in hex-Editor
SND
Just the common extension for DTPK sound files. The files are called like sound.snd. A Hex-Editor reveals the true nature of the file. Check hex header for "DTPK".
SPSD
Dreamcast digital sound file. Files start with "SPSD" in hex Editor. When extracted from afs their file ending in Shenmue is usually "*.STR". can be extracted with VGMstream, a nice utility to extract loads of different game music formats. For mono files it is required to add a Generic header (GENH) in order to rip them with VGMStream. VGMStream requires files to be renamed to "filename.SPSD" in order to extract these.
Newest built usually available at HCS64 site:
http://hcs64.com/vgmstream.html
A tool surfaced that does all of this automatically. It also eliminated all the problems that occured with VGMStream.
AFS
Sega container file. Starts with "AFS" if viewed in a Hex-Editor. This is comparable to archives like "zip" and usually just contain several other files in different formats.
Extract with AFSUtils.
IDX
Index files. These are related to an ASF file of the same name and give information about the contents (filenames) of the AFS archive. Hex-header starts with "IDX"
ADX
CRI Middleware compressed Audio format. Used in many SEGA games. Usually collected in AFS archives. There are plenty tools to extract or play these.
AHX
Basically the same as ADX, but it seems to be optimized for speech.
PVR
Power VR Texture/Image file. The only format Shenmue 1 and 2 use for Textures.
SFD
Movie file. Although an official format by CRI Middleware this format does not seem to be used in Shenmue. It seems all cut-scenes, the intro and other cinematics are done in real-time with the engine.
GENH
File with generic header. Shenmue does not have such files by default, it is a header someone added to make the file usable with VGMStream and other DSF players.
MT5/MT7
AM2's generic 3D model format. Partly directly on the disc, partly grouped in AFS files. Although some french guy found a way to interpret the model format there is no known tool released to convert them. So the only way to get the 3D models is ripping them out of the emulator (More about this in the next chapter)
DDS
DDS files don't directly appear in Shenmue. But if you rip models with 3D Ripper DX the texture format will be DDS. If you have a 3D program that can not handle DDS files as textures you will need to convert the to other formats like jpg or png.
Chapter 2: Extracting, Ripping, Converting and Emulating
There are several approaches to get content from Shenmue. To make things easier i try to group them by the corresponding media types.
Music and sound
Most music and sound files are stored as DTPK files (typically as filename.snd), AM2's proprietary file format that is probably best described as a kind of Sequenced tracks including the samples.
A good tool to convert the proprietary DTPK files to the easier to handle DSF format is the python script by kingshriek:
http://snesmusic.org/hoot/kingshriek/ssf/
Other files (especially speech and sound effects) come in SPSD format. SPSD files are not normally encountered alone on the Shenmue discs, they usually come in large bundles, stored in a single monolithic file with the AFS extension (accompanied by the IDX index files). most AFS Extractors will name the files with a .str extension (file.str). There is a nice batch related tool that can mass convert str files to wave from audio which can be played with all media players.
Textures
All textures come as PVR format files. Most are embedded in AFS bundles. There is a tool called PVR-dissector which can be used to mass-extract PVR files directly into PNG images. Note that they usually come mirrored and upside-down.
When you rip textures from an emulator the textures come in DDS format which you may need to convert to jpg or png in order to use.
3D Models
The 3D Models come in MT5 and MT7 formats, which are proprietary formats by AM2. A french guy has found out how these files are built and successfully managed to completely read the files.
The current method of ripping the 3D models is to rip the whole scene from the current view with 3D Ripper DX in NullDC Emulator. To get the models in 3D you need to use a special stereoscopic plugin for NullDC.
More about Emulation in Chapter 4 - Emulation. Video tutorials are available that demonstrate the whole process of ripping.
Chapter 3: usefull Links and tools
Info about ripping Sound and Music from Dreamcast games:
http://hcs64.com/vgm_ripping.html
DSF collection (spares the pain to create DSF ourselves)
http://dsf.hcs64.com/s/
DDS Converter 2.1 (Mass Converts DDS to jpg or png)
http://www.tankraider.com/userup/1264411639.zip
MT5 Extractor, GUI (requires mono) and sources
http://abdessel.iiens.net/dreamcast/mt5/
MT5 Extractor Mass extraction GUI (Basically same as above but doesnt need mono)
http://www.tankraider.com/userup/1261175783.zip
Flashless GUI & most recent mt5 extractor:
http://www.tankraider.com/userup/1265893559.zip
VB6 mass extractor Source (VB6):
http://www.tankraider.com/userup/1265915914.zip
Null DC 1.00B16 with Stereoscopic Plugin (a must for ripping from the game)
http://www.tankraider.com/userup/1265047692.zip
3D Ripper DX 1.7 (Rip 3D Models while playing the game)
http://www.tankraider.com/userup/1265076820.zip
DSF Plugins for Foobar2000 and Winamp
http://www.tankraider.com/userup/1266756862.zip
Big Package of extraction tools
http://www.tankraider.com/userup/1266735863.zip
AFS Utility V2 - extract/view/manipulate AFS archives
http://www.tankraider.com/userup/1266741951.rar
Shenmue CInematics director - lets you change subtitles
http://www.tankraider.com/userup/1266677884.rar
PVR Dissector - Good tool to extract PVR files from various other resources
http://www.tankraider.com/userup/1266748225.zip
AM2 AHX Extractor Tool
http://www.tankraider.com/userup/1266690927.rar
STR/DTPK mass extraction tool
http://www.tankraider.com/userup/1266713241.rar
SND2WAV - as it says it extracts SND files
http://www.tankraider.com/userup/1266770332.rar
MAPINFO Patcher - you need this if you wanna swap maps in the game
http://www.tankraider.com/userup/1266721825.zip
Shenmue Font hacking pack - a tutorial and tools to change the fonts of the subtitles
http://www.tankraider.com/userup/1266727432.rar
Selfboot Data pack. If you need to make a new selboot image from your edited files this will help.
http://www.tankraider.com/userup/1266772850.rar
Humans dissector - extracts Humans.afs
http://www.tankraider.com/userup/1266757966.rar
Photoshop DDS Plugins - allows editing of DDS files in PS
http://www.tankraider.com/userup/1266733003.zip
AiO Cinematics Translator - good cinematics subtitle editor for shenmue
http://www.tankraider.com/userup/1266725831.zip
Shenmue Free Quest subtitle editor
http://www.tankraider.com/userup/1266737565.zip
Shenmue test environment - To easily test out how your changes will look
http://www.tankraider.com/userup/1266707434.zip
IDX Creator - recreates IDX index for your modified afs
http://www.tankraider.com/userup/1266733513.zip
MT5/MT7 Texture editor - edits the texture of the model
http://www.tankraider.com/userup/1266773066.zip
SPR Editor - Edit textures/graphics in SPR files
http://www.tankraider.com/userup/1266700546.zip
PVR2PNG - Extracts PVR to PNG while keeping transparency information
http://www.tankraider.com/userup/1266679211.zip
SEGA PVR Tools package - all tools SEGA used themselves
http://www.tankraider.com/userup/1266747838.zip
CRI Middleware 2008 - AFS, AHX, ADX tools
http://www.tankraider.com/userup/1271176534.zip
AFS Explorer 3.2
http://www.tankraider.com/userup/1271607190.zip
GDI2Data - Extract GDI images
http://www.tankraider.com/userup/1271623975.zip
KyoDTPK - mass converter i made to convert dtpk files
http://www.tankraider.com/userup/1365733086.zip
Chapter 4: Emulation
There are 2 major emulators available that are able to run commercial games like Shenmue: NullDC and Chankast. The more compatible and generally prefered one is NullDC. Games usually come as .cdi files which represents a Disc Juggler image file. As the original GD-Rom Game discs can not be used for reading in a normal PC CD-Rom drive the only easy solution is to download the CDI files from the internet. A small search in google for "Shenmue" and "torrent" or "Shenmue" and "Rapidshare" should reveal some sources. Transforming the GD-Rom discs into an image is unfortunately quite complicated and requires at least a Dreamcast coder's cable (Dreamcast to serial port PC) or the Broadband adaptor (BBA) while the later is ridiculous expensive and the first one is hard to get.