Since mediatomb doesn't care much about what it serves, it's not that hard to have it serve dvd movies from an ISO file. A PlayStation 3 however is rather picky when it comes to video formats, so getting the three to play along can get interesting.

First of all, you need to tell mediatomb that anything with a .iso extensions is something it should look into:

<map from="iso" to="video/dvdiso"/>

Then tell it to transcode this particular mimetype:

<transcode mimetype="video/dvdiso" using="mencoder-iso"/>

All you need then is a transcoding profile which handles the iso files.

<profile name="mencoder-iso" enabled="yes" type="external">
        <mimetype>video/mpeg</mimetype>
        <accept-url>yes</accept-url>
        <first-resource>yes</first-resource>
        <accept-ogg-theora>yes</accept-ogg-theora>
        <agent command="/usr/local/bin/mediatomb-mencoder-iso" arguments="%in %out"/>
        <buffer size="1000000" chunk-size="512000" fill-size="20480"/>
</profile>

If you're unfamiliar with the mediatomb config.xml structure, my full config file is up here. Finally, here's the helper script that does the actual transcoding of the dvd iso (with subtitles!) to something the PS3 knows and likes:

#!/bin/bash
CHAPTER=`lsdvd "$1" | grep Longest | sed 's/.* //'`

exec mencoder -dvd-device "$1" \
dvd://$CHAPTER -slang nl,en -oac copy -ovc lavc -of mpeg \
-lavcopts vcodec=mpeg2video:keyint=1:vbitrate=200000:vrc_maxrate=12000:vrc_buf_size=1835 \
-mpegopts muxrate=12000 -vf harddup,scale=720:-2 \
-o "$2" &>/dev/null

This script uses "lsdvd" to find the longest chapter of the dvd first - we'll have to assume there that the longest chapter is indeed the one with the main movie. A dvd iso containing two movies, or containing various episodes of a series for an example, won't work too well with this script - which is fine by me, since I don't have that kind of isos. This script Works For Me™ but feel free to change mencoder parameters to suit your needs. If you're not interested in subtitles, you could try without the -slang nl,en and using -ovc copy, essentially remuxing the media stream without actually transcoding it. YMMV.

So there you have it - iso files show up in the PS3's menu, and they get transcoded on the fly with dutch subtitles, or english subtitles if dutch ones aren't available in the iso. And you don't even need mediatomb 0.12 for it to work.