Persisting a MUD state with plain binary serialization

9 messages in this thread from mud-dev2 in 2009-01

  1.   Tiago <tiago.matias@gm...com> 01-16 12:51
  2.   Chris White <ckw4y@cs...edu> 01-25 02:40
  3.   Mike Oxford <szii@sz...com> 01-27 18:44
  4.   Jon Mayo <jon.mayo@gm...com> 02-03 17:38
  5.   Jeffrey Kesselman <jeffpk@gm...com> 02-03 18:33
  6.   Jeffrey Kesselman <jeffpk@gm...com> 02-07 18:04
  7.   Mike Oxford <szii@sz...com> 02-08 05:20
  8.   Tiago.matias@gm...com <tiago.matias@gm...com> 02-10 19:25
  9.   Tiago <tiago.matias@gm...com> 02-05 16:01

Tiago <tiago.matias@gm...com>

2009-01-16 12:51:23
Hello all!

I'm building a (classical) MUD server as hobby. I would like to have a fully
persistent game state and thought of using simple serialization to dump the
entire object graph to disk periodically.

Thing is, binary serialization has a few drawbacks, in terms of versioning
and it's a pain to control which parts of the graph are serialized or not
(it's perfect to serialize everything tough).

So, I would like to hear if someone here has had any (bad) experiences with
serialization (applied to this particular domain) and if they recommend it
or not.

Thanks!

tiago matias
_______________________________________________
MUD-Dev2 mailing list
MUD-Dev2@mu...com
http://my.binhost.com/lists/listinfo/mud-dev2

Chris White <ckw4y@cs...edu>

2009-01-25 02:40:44
I've used binary XML trees on a commercial MMO I worked on.  Each 
persistent object type had at most 20 nodes or so, so it wasn't a big 
hit to have to save the entire tree any time one of the nodes changed.  
My only real regret about the system was that it was kindof awkward to 
convert from the binary XML to human readable XML, which made it harder 
to debug.  If this is just a hobby project, the path of least resistance 
is probably just to use XML.  There are tons of freely available 
utilities and you're unlikely to need super high performance out of a 
hobby MUD server.

Tiago wrote:
> Hello all!
>
> I'm building a (classical) MUD server as hobby. I would like to have a fully
> persistent game state and thought of using simple serialization to dump the
> entire object graph to disk periodically.
>
> Thing is, binary serialization has a few drawbacks, in terms of versioning
> and it's a pain to control which parts of the graph are serialized or not
> (it's perfect to serialize everything tough).
>
> So, I would like to hear if someone here has had any (bad) experiences with
> serialization (applied to this particular domain) and if they recommend it
> or not.
>
> Thanks!
>
> tiago matias
> _______________________________________________
> MUD-Dev2 mailing list
> MUD-Dev2@mu...com
> http://my.binhost.com/lists/listinfo/mud-dev2
_______________________________________________
MUD-Dev2 mailing list
MUD-Dev2@mu...com
http://my.binhost.com/lists/listinfo/mud-dev2

Mike Oxford <szii@sz...com>

2009-01-27 18:44:48
Tiago wrote:
> Thing is, binary serialization has a few drawbacks, in terms of versioning
> and it's a pain to control which parts of the graph are serialized or not
> (it's perfect to serialize everything tough).
Not sure what you mean by "binary serialization."  Just copying out of 
memory won't work as I'm sure you're aware.

Couple thoughts here.

1)  Version every single object.  Every object knows how to serialize 
and deserialize itself, with a version header.  You then pass around 
your "save context" and the object knows how to get itself out of it 
(switch() statement calling, say, DeserializeV1()) or put itself in.  If 
you never have more than one "saved graph" you can even go with 
"Deserialize() and DeserializeFromPrevious()" or some such.
2)  You just version the entire graph.  Every time you touch a 
meta-element (schema) ANYWHERE you roll the version up one.  This leads 
to "version 1418437113" but it's an option.  There is no "upgrade path" 
between versions.

The problem with versioning "subtrees/subgraphs" means that the 
interactions across boundaries becomes an issue unless you encapsulated 
everything very very very well and don't pass structs and such.

I would recommend #1 with good encapsulation/data hiding practices and 
virtual function calls.  It's a bit more work but for any project of any 
size you'll more than make up for the time in maintenance.  Also, #1 
gives you the ability to save everything out, stop the game, fire up a 
newer version of the game (with the correctly versioned deserialization) 
and everything will load.

-Mike


_______________________________________________
MUD-Dev2 mailing list
MUD-Dev2@mu...com
http://my.binhost.com/lists/listinfo/mud-dev2
On Fri, Jan 16, 2009 at 4:51 AM, Tiago <tiago.matias@gm...com> wrote:
> Hello all!
>
> I'm building a (classical) MUD server as hobby. I would like to have a fully
> persistent game state and thought of using simple serialization to dump the
> entire object graph to disk periodically.
>
> Thing is, binary serialization has a few drawbacks, in terms of versioning
> and it's a pain to control which parts of the graph are serialized or not
> (it's perfect to serialize everything tough).
>
> So, I would like to hear if someone here has had any (bad) experiences with
> serialization (applied to this particular domain) and if they recommend it
> or not.
>
> Thanks!
>
> tiago matias
I just append my binary blobs to the end of a journal file and update
hash table entries to point to the new offset. I write the entire
record out to the journal so book keeping of the blobs is relatively
simple. When I boot and reply the journal I immediately rewrite a
compacted journal to a new file (and rename the old file to a backup).
Compacting can be done as often as you like, but in my system it means
you're writing the entire graph to disk in one go. Which causes a
moderate amount of pausing for players depending on disk performance
and size of the graph.

I keep a generation number in every object, but I don't use it for
anything. It turns out I didn't need it. And accessors for setting all
the fields of an object also set a dirty flag so I know that it needs
to be serialized.

My binary serialization uses a printf style string to describe the
format. It was easier than my other experiments using ASN.1 to
generate routines or doing it by hand with a (large) hand rolled
function for every type of object. I think if I were to do it again
I'd write an XDR parser (as in Sun RPC) and my own code generator, the
printf one was neat in that I could get gcc to warn if I passed the
wrong thing, but with many many fields to dump I either needed a
function that took many arguments or I needed to call the routine and
update an buffer offset each time to do the serialization process.

-- 
Jon Mayo
<jon.mayo@gm...com>
_______________________________________________
MUD-Dev2 mailing list
MUD-Dev2@mu...com
http://my.binhost.com/lists/listinfo/mud-dev2

Jeffrey Kesselman <jeffpk@gm...com>

2009-02-03 18:33:51
On Fri, Jan 16, 2009 at 7:51 AM, Tiago <tiago.matias@gm...com> wrote:

> I'm building a (classical) MUD server as hobby. I would like to have a
> fullypersistent game state and thought of using simple serialization to
> dump theentire object graph to disk periodically.
> 
> Thing is, binary serialization has a few drawbacks, in
> terms of versioningand it's a pain to control which parts of the graph are
> serialized or not(it's perfect to serialize everything tough).
> 
> So, I would like to hear if someone here has had any (bad)
> experiences withserialization (applied to this particular domain) and if
> they recommend itor not.
> 
> Thanks!
> 
> tiago matias 
You could just let Project Darkstar do it all for you.

www.projectdarkstar.com

_______________________________________________
MUD-Dev2 mailing list
MUD-Dev2@mu...com
http://my.binhost.com/lists/listinfo/mud-dev2

Jeffrey Kesselman <jeffpk@gm...com>

2009-02-07 18:04:16
On Thu, Feb 5, 2009 at 11:01 AM, Tiago <tiago.matias@gm...com> wrote:
>> Tiago wrote:
>>
>> Not sure what you mean by "binary serialization."  Just copying out of
>> memory won't work as I'm sure you're aware.
>>
>
>
> Well, by binary serialization, I was in fact refering to a facility
> whithin.NET apps that simplify dumping an entire object graph do disk.
> My biggestinterest in this was just the simplicity it offers. Just
> callSerialize(World, filestream) and it's done.
>
> Problem is, like you might imagine, it's all or nothing. Anything
> besidesthat very easily becomes a pain to implement. Another problem is
> theversioning and upgrades (like you said). It effectively renders
> datamigrations almost impossible.
>
> I'm considering manually serializing to XML some big game blocks like,
> themap, the players, the mob's and the positions of these things in the
> world.Then it would just be a matter of fixing the references as these
> things aredeserialized back from the disk into memory
>
> The generated XML would obviously be human readable and writable and
> wouldmake upgrading possible.
>
> Disadvantes: A lot more to code when what I would really want to be
> doingwas programming MOB behaviour...
If you dont want to use a pre-built solution, have considered using a
real RDBMS like mysql?

Not that hard to use really (at least from Java thanks to JDBC) and very
very flexible.


_______________________________________________
MUD-Dev2 mailing list
MUD-Dev2@mu...com
http://my.binhost.com/lists/listinfo/mud-dev2

Mike Oxford <szii@sz...com>

2009-02-08 05:20:07
Jeffrey Kesselman wrote:
> On Thu, Feb 5, 2009 at 11:01 AM, Tiago <tiago.matias@gm...com> wrote:
>   
>>> Tiago wrote:
>>>
>>> Not sure what you mean by "binary serialization."  Just copying out of
>>> memory won't work as I'm sure you're aware.
>>>
>>>       
>> Well, by binary serialization, I was in fact refering to a facility
>> whithin.NET apps that simplify dumping an entire object graph do disk.
>> My biggestinterest in this was just the simplicity it offers. Just
>> callSerialize(World, filestream) and it's done.
>>
>> Problem is, like you might imagine, it's all or nothing. Anything
>> besidesthat very easily becomes a pain to implement. Another problem is
>> theversioning and upgrades (like you said). It effectively renders
>> datamigrations almost impossible.
>>
>> I'm considering manually serializing to XML some big game blocks like,
>> themap, the players, the mob's and the positions of these things in the
>> world.Then it would just be a matter of fixing the references as these
>> things aredeserialized back from the disk into memory
>>
>> The generated XML would obviously be human readable and writable and
>> wouldmake upgrading possible.
>>
>> Disadvantes: A lot more to code when what I would really want to be
>> doingwas programming MOB behaviour...
>>     
>
>
> If you dont want to use a pre-built solution, have considered using a
> real RDBMS like mysql?
>
> Not that hard to use really (at least from Java thanks to JDBC) and very
> very flexible.
Along the same lines, it may be worth it to look at OODBMS as well.  
Again, though, you have to
deal with the upgrade path yourself. 

-mox

_______________________________________________
MUD-Dev2 mailing list
MUD-Dev2@mu...com
http://my.binhost.com/lists/listinfo/mud-dev2

Tiago.matias@gm...com <tiago.matias@gm...com>

2009-02-10 19:25:55
On 2009/02/07, at 18:04, "Jeffrey Kesselman" <jeffpk@gm...com> wrote:
> On Thu, Feb 5, 2009 at 11:01 AM, Tiago <tiago.matias@gm...com> wrote:
>>> Tiago wrote:
>>>
<snip> 

> If you dont want to use a pre-built solution, have considered using a
> real RDBMS like mysql?
>
> Not that hard to use really (at least from Java thanks to JDBC) and  
> very very flexible.
Perhaps... In reality it's all a matter of how much code you have to
write for data access/storage. Since I need all the objects in memory at
all times, xml,sql,binary will all do the trick. 

Perhaps a sql database is the fastest way to implement what i need :)

Thanks !

_______________________________________________
MUD-Dev2 mailing list
MUD-Dev2@mu...com
http://my.binhost.com/lists/listinfo/mud-dev2

Tiago <tiago.matias@gm...com>

2009-02-05 16:01:10
> Tiago wrote:
>
> Not sure what you mean by "binary serialization."  Just copying out of
> memory won't work as I'm sure you're aware.
Well, by binary serialization, I was in fact refering to a facility
whithin.NET apps that simplify dumping an entire object graph do disk.
My biggestinterest in this was just the simplicity it offers. Just
callSerialize(World, filestream) and it's done.

Problem is, like you might imagine, it's all or nothing. Anything
besidesthat very easily becomes a pain to implement. Another problem is
theversioning and upgrades (like you said). It effectively renders
datamigrations almost impossible.

I'm considering manually serializing to XML some big game blocks like,
themap, the players, the mob's and the positions of these things in the
world.Then it would just be a matter of fixing the references as these
things aredeserialized back from the disk into memory

The generated XML would obviously be human readable and writable and
wouldmake upgrading possible.

Disadvantes: A lot more to code when what I would really want to be
doingwas programming MOB behaviour...

_______________________________________________
MUD-Dev2 mailing list
MUD-Dev2@mu...com
http://my.binhost.com/lists/listinfo/mud-dev2

9 messages in this thread from mud-dev2 in 2009-01