11 months, briefly.  

A lot happened this year.  I learned a lot of stuff, but not really that much if you consider how much there is to learn.  I want to write down a bit about everything I’ve been doing..

Maybe I’ll work backwards.  So most recently, I was finally allowed to make maps for Scourge of War.  So I dug out my dusty random map generator that I started back in August 2011, and worked on it basically nonstop for about two weeks.  Get up at 7, work on it til midnight (with breaks to go to class) and repeat next day.  The result?  We have a bunch of cool new maps to play Scourge of War on.  As I’ve said before, I’m a terrain junkie.  A lot of those hours I spent were wasted (?) flying around my maps just looking at them, the way I look at terrain while riding in a car or on a plane or in a train.  Terrain is cool.

It was nice to have classes during these two weeks, since class time — at least when it’s boring — is when I have a lot of my ideas.  Because I’m stuck in a chair with paper and pencil, and I think of algorithms for generating creeks and roads and such.  The map generator code is weird.  Most of it is C#, and then there are a lot of supporting python scripts and batch files to combine everything together.  The C# is broken into different modules, for instance there’s a heightmap generator that makes hills and creeks, and it has to return where the creeks are so the terrain and texture generators can put blue there — and the heightmap generator takes the locations of the roads so it can make them sunken.  So each module has an input data set and an output data set.  Much of the code is generalized stuff — functions that apply geometry to make stuff on what are basically bitmaps of varying sizes.

I didn’t do each piece with performance as a priority.  I have a log file that gives me detailed info on how long everything takes, so when I have time to work on performance, I pick the biggest thing in the logfile and address that.  The biggest one was generating a bunch of 4096×4096 (2d) perlin noise maps.  Those took a few seconds each if I remember right.  So I made a little cache wrapper thing that stores them in files, and just grabs as many as I need from a set of two or three times as many as I need, so they don’t have to be generated on the fly.  No one’s gonna know if the random greenish texture splattered on one 5×5 mile map is in the same layout as the brownish texture splattered on another map.  The important thing is that they look good, and on a given map they don’t repeat which really makes it look more natural.

The current performance issue is the generation of creeks, which is not a simple problem in general.  My current implementation was one of the first things I did for the generator, and the results are OK, but not ideal.  If I get time I want to redo the creeks to be more realistic, and make sure performance is good next time.  But tracing creek routes on a 7.5^2 mile heightmap just isn’t something that can be done instantly, I don’t think.   (I need a bigger heightmap than the area used so the creeks can flow in and out along the edges.)  It only takes a few seconds so it’s not that bad.

Then there’s roads.  My first road algorithm made unique-looking road networks but they were not very useful in game since there were a lot of isolated tree-like road areas.  And there was no easy way to align the fields with the roads.  So I redid that as a randomly noisy grid, that makes it easy to split up the inside areas into fields, and makes the roads useful in-game.  I’m not happy with it, but it looks good and plays well.

(one of the random maps)

 

Before the random map generator, I decided to do something in C++.  I rarely do anything in C++ because I have all this library code I’ve written in C# and so many projects already released that I can improve with minimal resistance.  But I wanted to practice C++.  I sort of know how memory works in C#, and C++ is relatively straightforward in comparison since you have to do everything yourself.  You can’t really understand your C++ code without knowing how you’re using memory.

So to start with I made something with minimal effort that was useful — a TCP client/server app where the server runs on my brothers’ minecraft server, and we can connect using the client and send it messages to, for instance, teleport our character around, or swap our inventory between different worlds.  I made it do just one thing, sort of hackily at first, but it worked.  And at that point, if I wanted to improve it, I had to write C++ since it would take more work to rewrite in C# than to continue writing C++.  So I implemented my GCSV parser and ini-file handler in C++.  (These two things are particularly important to me and are used in all of my C# projects, since I like to keep as much of my data in external files as possible, and I always like to change things from hardcoded to options when possible.)

The GCSV parser in particular forced me to decide how to handle memory.  I’ll start from the beginning.  A GCSV file looks like this:

~myfile
col1,col2
a,field_for_a
b,field_for_b

So when I read the file, it detects ‘~’ as the start of a new “GcsvTable” and uses col1 and col2 as the column names.  So I can say something like

var gcsv = Gcsv.ReadFromFile("myfile.csv");
foreach(var line in gcsv) {
    var field1 = line["col1"];
    var field2 = line["col2"];
    Assert.IsTrue(field2 == "field_for_"+field1);
}

So I can easily add columns and change the order of columns in my csv file without breaking my program.  My C# code for this is available in the GCSV library in my open sourced Utilities.dll code.  It’s awesome for me, I don’t expect anyone else to use it.  That code I’m still happy with even though I wrote it two years ago.  Oh, and since I githubbed this minecraft service I’m talking about as well, the C++ version (which is relatively barebones in comparison) is also available.

GCSV also supports a default or specified index for a Table, where I could say gcsv["a"] and get the first line in my example file.  This is lazy-instantiated since I usually don’t need it.  And there’s other cool stuff, like a GCSV file can have multiple Tables in it which is useful for instance in GCM where I have like fifty tables each representing an option when creating a battle.  It’s reasonably optimized in C# so the overhead of loading a typical Table is very little more than just the text itself.

Where was I.. oh yeah, the GCSV code in C++ forced me to decide how to handle memory.  I sort of knew how smart pointers worked, but hadn’t used them in anything important where I really wanted to know exactly how they worked.  With this, I did.  See, the GCSV file gets read into a Table object, which has a Header and a bunch of Lines.  Each Line has a reference to the Header — so you can pass a Line around and use it like a read-only associative array, even though all it is is a simple array of strings and a pointer to the header.  But the Line shouldn’t know about the Table, because in theory a Line could be passed off somewhere, and if it’s the only Line my code knows about anymore, the Table and the rest of the lines should be freed.  Likewise, the Header doesn’t know about the Table.  Anyway, shared_ptr is great and this all is pretty straightforward and it works and makes me happy.

With the GCSV code in C++ it was simple to set up the TCP server with a data file listing the minecraft worlds and the teleports for each world in gcsv files.  Anyway, I worked feverishly on this for a couple weeks, and we used it, it worked, great.  We keep a vanilla minecraft server since we don’t play enough to justify keeping a set of mods compatible with the game as it is updated.  Since my service just accesses the world files, and these almost never change, my stuff doesn’t really get out of date.  (The specific files I access haven’t fundamentally changed ever since we started our world in minecraft multiplayer beta.)

 

Before that, let’s see.. I spent the summer in New York, as an intern at Liquidnet.  It was a great internship.  Good people, minimal bureaucracy, a code base which — while big like they always are — was less of a nightmare than I would have expected.  (Granted, as an intern I wasn’t working with all or most of it, but I was exposed to a lot of it.)  Build/test infrastructure was pretty good I thought.  I was able to work on stuff for real people that I think might actually be used — my work was in UAT when I left, to ship when the in-progress system it was built for ships.  Informal work environment which I don’t imagine is typical of finance.  Lots of food.  Aside from students and professors it’s the first time I’ve worked directly with other programmers, and that was good.  I was happy.

I liked living in the city.  First of all, there are tons of people, so whatever you want to do you can find people doing it.  You don’t have to get in a car and drive several miles to find food or people.  I fully intend to live somewhere more densely populated when I graduate.

(the view from my window at Columbia)

During the summer I got some more exposure to F#, which I had dabbled in before, and finally actually used for “production” code.  I rewrote and expanded on a chunk of my GCM code base in F#.  A lot of that code really maps well to being purely functional, and the size of the code that I rewrote cut to less than one half what it was before.  F# is really good.  (It’s OCaml with .Net integration).

 

Before the summer I worked a lot on GCM.  Added a singleplayer campaign that by current count, 125 people have played, with 1000 battles recorded so far.  This was interesting because I had to take my GCM multiplayer campaign, and basically have a clone of it for each singleplayer campaign.  So everything I had written with “just one of these” in mind became “n of these”.  It wasn’t that difficult really, since I had the idea of “a campaign” right from the start, and just had to allow more of these.  So each campaign has its own index of regimental histories, which is one topic I want to talk about some time since it involves packing lots of bits of data (hundreds of thousands of “this regiment fought in this battle under this commander” entries) into a small amount of space with fast access for the one common access pattern — which is “show me all the battles this regiment fought in.”

 

I have a few other blog posts that are sitting as drafts, that I’ve been working on.  One of them is the history of my perspective of the Scourge of War / GCM scene.  Basically a narrative of the challenges the game faced in the multiplayer scene, and how they were addressed over the past two years.  But that story continues to get longer, perhaps faster than I write the history.  Either I’ll speed up, or just post a part 1, or it’ll never get posted.

Leave a comment

Hurrah  

I’m finally “off of” php.

The new site, www.sowmp.com, has gone live.  I thought it would be a lot longer.  Early on in rewriting I had had that awful feeling that “I’ve got to redo all that stuff I already did…” which was countered by the knowledge that if I don’t do it now, I’ll never get the chance.  The bulk of it only took three weeks.  I started testing it live early this week, and after a couple days it was obvious the old one wasn’t going to be used again, so I made the official switch where I redirected the old links to the new site and archived the old one.  The migration has gone smoothly, so far 50 people have logged in to the new one.  I kept the old accounts so no one had to re-register — very important as people don’t like registering.  (at least I don’t)

It’s all C#/Javascript now.  The code was faster to write, is much nicer to read… and is shorter.  Because of all this, I’ve done a lot of things that would never have been worth the ugliness in php, but are clean and simple in C#.  The folks using it are happy too, though they won’t know why I didn’t add any of this stuff before.

Leave a comment

 

As of last week, I reached the “top of the hill” in the project I was working on.  I had finally got the architecture in place allowing me to begin rapidly filling in the code.  Designing and building the architecture is worse not because it’s harder, but because it’s scary.  And it sometimes it gets discouraging if I’m not sure I’ll finish the project.  It’s sort of like going up a mountain.  On the way back down, I have gravity on my side, and it will get done.

The project is to replace the first website I ever made, which was written in PHP, with one written in C#, using MVC 3.  (It also involves replacing the desktop app, but 3/4 of that code is being reused directly since it’s still C#.)

www.sow.philipmcg.com —> www.philipmcgarvey.com/gcm

The original website works reliably enough, but it’s no fun to change in its current state.  Part of that is because it’s PHP, and I don’t like PHP.  In fact, most of that is because it’s PHP.

Because it is PHP:
It is much harder to refactor than all of my C# code.
Because it is harder to refactor, some of the code hasn’t been touched since it was written in my first few days of web development.
I couldn’t share code between the website and the C# desktop app that goes with it.  With ASP.NET, code can go in a shared project, which means with good web <-> desktop communication abstractions which I have done, and which weren’t hard to do, I can pass real C# objects directly between the two.

And mainly, it’s just a pain to write good code in PHP.  Perhaps this is all my personal bias — but I do think there are good, inherent reasons that PHP is worse than C# for my purposes.

I don’t regret having done it in PHP the first time.  It was very easy to get started, and it mostly it just worked.  It’s served its purpose for 14 months now — a lot of use has been squeezed out of it.

So back to the topic.  Being over the top of the hill, I’ve been able to work morning ’til night filling and draining the todo list, and it’s very satisfying.  The project uses Entity Framework Code-First.  Now that the site is live (in beta), I’m not regenerating the database, so when I add columns I do it in both the code and the database, but this isn’t that bad and I haven’t had to do it very much.

I want to get this project to a state where it can replace the old one before school starts in two weeks.  I’ll get back to work now.

Leave a comment

Terrain  

I really like terrain, and games that let me make terrain maps to play on.  One of the big reasons I made the HDN mod was that I loved making terrain for it, and the game needed to be fun or no one would ever enjoy that terrain.

Terrain crafting is an art.  Here are some of my favorite works, still fought over in HDN battles.  The full list of HDN maps used by the Hawks is here.

Some time ago, I also made a few maps for Company of Heroes.  The closeup detail here was great.

When Scourge of War finally lets us make maps some day, if I have time for it, there will doubtless be some pictures coming here.  (Edit: and now you do)

Leave a comment

 

Last night I picked up a Finnish war novel.  On page 9:

[For context, Makila is the quartermaster]

    The kind of emergency that would make Makila voluntarily surrender an article of equipment was unlikely ever to arise.  As heir to a big farm in Laihia he often received parcels from home.  These he would furtively carry to the storeroom to avoid having to share them with the other NCOs.  Once, however, the mail arrived so late that Makila had already gone to bed, and a messenger brought a parcel addressed to him to the NCO barracks, creating a most awkward situation for Makila.  He could not very well sneak the parcel away, and to keep it in the barracks meant having to share it.  He temporized by muttering something about sharing it in the morning and hiding the parcel under his pillow.

    In the night a faint sound of rustling paper came from Makila’s bed.  Instantly the light went on and Hietanen’s voice boomed through the room:

    “Let’s go, boys!  Makila’s sharing his parcel.”

Leave a comment

 

I put together a solid proof of concept “game” from the stripped down Org I mentioned last.  I left out the live networking, because without NAT punch-through which requires a live TCP/IP server, hosts behind routers have to port forward, and people don’t want to do that for good reason.

So I took a different approach.  I wanted to make a turn based strategy game with vague similarity to Plot & Conquer.  It had to be easier to use though.  In the end, I came up with a responsive system for making a turn based game.

My ASP.NET server handles authentication and holding the list of games and players, and all that.  You log in to the game on your computer, and you can start a new game or load one from the list that your player is in.

It’s a simultaneous turn based system, similar to Medieval Total War (the first one), where all players give orders at once, and then they are executed together.  So when you load the game, it knows what turn it is, and if you haven’t finished the turn yet, you can give orders.  My proof of concept “game” is just a multi-user map editor for Org.  You click the hexes to change their terrain, and when you’re happy you click Finish.  When all players in the game have clicked Finish, the Watch Turn button appears, and clicking that executes all of the orders.  If you’re the host, it also sends the savegame back to the server after executing the orders, which is how the server can record the game.

After watching the turn, you can immediately start giving orders for the next one.  It works pretty well.  It also has live, well, five-times-per-second-polling chat, to make it feel more like you’re playing the game with other people.

In the process, I learned a lot about ASP.NET MVC 3.  I tried the database-first, model-first and code-first methods.  I ended up liking code-first best, it was the cleanest and didn’t leave a bunch of ugly generated ORM code.  I made a really clean interface for passing objects between the desktop and server using xml serialization.  It’s very nice to be able to share entire C# projects between server and desktop, which is a luxury I didn’t have when using php for the website.

Oh yeah and tonight I installed an Ubuntu VirtualBox and ran some C# on it.  Glad to finally see mono work, all I had ’til now was rumors.

Leave a comment

 

Last week during a boring class I had an idea for a way to turn Org into a war game of sorts.  I had tried a few things before and discarded them because they took attention away from the main focus which is exploring, finding the best routes across the map and setting up efficient networks between the settlements and cities.  Anything the player had to do other than this made the game too hard to play.

The idea I had was that the military side will happen on its own.  You build the infrastructure, and your cities generate armies that go attack other cities, and you don’t control it at all.  It wasn’t difficult to add, and it is neat to watch.  Org is still normally played without military — the military is an experimental thing.

 

Working with my evolving C# game engine again was a pleasure.  Since Org itself is fun in its own right, and I can’t add much to the gameplay without ruining it, I decided to try doing something new with the engine.  So I’ve copied it again, got the copy in a separate SVN location, and cut out a lot of things leaving me with a stripped down but working game to mess around with.

The graphics engine, data file manager, network code and utilities library make up maybe 2/3 of the code.  From the remaining 1/3 which is the stuff specific to Org, I stripped out the code for constructibles (cities/settlements etc), roads, the AI implementation, and market.  I still have the hex map engine, terrain, random map generator, “players”, moveable objects, pathfinding, and so on.  We’ll see if I come up with anything interesting from this base.

Leave a comment

 

I finally got around to getting an ASP.NET site going.  I’ve wanted to try it out for a while, as an alternative to php.  I expected that using C# for websites would be more productive and more fun, and so far I have not been disappointed.  I had wanted to have Org games recorded to the web, but had put it off since I didn’t want to use php.  Yesterday, I did that little project for my trial run of MVC 3, and here’s the result.  Nothing really fancy, but it fits nicely with the existing site.  This framework is really impressive.

Leave a comment

 

Been a while since I posted.

To follow up the last post, here are the map playing rates for SOW since the random maps were added:

Random Map      66
DevilsDen       12
CulpsHill        7
McPherson        6
Gettysburg       3
PPT              1
EastCavalryField 1
Alpine           1

Beyond that, there’s currently little more I can do with Scourge of War, so I’ve moved on to working on other things.  I spent last week’s free time coding for a Mount & Blade mod.  Mount & Blade took a very good approach to letting their game support mods.  The bulk of the gameplay code is in external scripts and open for modding.  The scripts are “written in” python.  Actually, the scripts are written as lists of tuples in python, and python is used to process these into smaller files for the game engine to read.  It’s a somewhat cumbersome way to write code, but like any form of coding one can get used to it.  The advantage of having all of this separate from the compiled game code is that because so much can be changed, the game drew in lots of modders right from the early beta versions.  Now, years later, this modding community is massive.  Almost all of the multiplayer community plays mods.  It’s clear that the game has benefited enormously from this.

I had been modding Mount & Blade in 2008, up until I discovered what I could do with the American Conquest engine and started making HDN.  In 2008 I had written a bunch of gameplay balancing scripts and released them in a mod of my own.  This time around, I brought these scripts (and a bunch of new ones) into an existing “total conversion” mod someone else had made.  As I was doing this I came up with a simple scheme for adding a form of macros in the Mount & Blade python scripts.  This helped remove some of the drudgery from the coding and made it a bit more interesting.

I’m not sure what I’ll work on next.  Whatever it is it will have to wait for time to be available!

Leave a comment

 

I have a keen interest in terrain and maps, and also in procedural generation of stuff that interests me.  There is evidence for this, like ORG which is a game focused on procedurally generated terrain maps, and GCM whose core is procedural generation of American Civil War armies.  When I had just started programming, my interest in terrain took shape of hand-crafted battlefield terrain in various games, HDN in particular.

I really like writing code to generate things that feel realistic.  In some multiplayer games, which are played over and over again, variety is extremely important.  However, the flavor of realism is still important.  This is where random generation shines.

GCM is, in part, my attempt to reduce repetition as much as possible in the multiplayer of Scourge of War – Gettysburg.  As much as possible I want each battle to feel new.  The huge hindrance here has been the battle maps.  The game originally had five, and over a year after release it has eight.  Two of these are barely worth playing.  In 2011, these were the amount of times each map was played:

PPT             145
DevilsDen        91
EastCavalryField 87
McPherson        81
CulpsHill        77
Gettysburg       47
Kansas           21
Alpine            8

GCM did still randomly generate the scenarios, so the objectives and armies would be in different places each time, but you still knew at the start of a battle what ground to race for.  Due to some logic that I haven’t fathomed, putting certain arbitrary restraints on a modding community gives some perceived benefit.  The restraint in this case is that we are intentionally prevented from making new maps for the game or modifying existing ones.  The apparent reason is so the company can sell maps because people wouldn’t buy them if they could get maps for free from modders.  Perhaps some time I will write a thorough explanation of the problems I see with this reasoning.

Anyway, the result is that the multiplayer community has dealt with a lot of needless repetition, and the modding community has produced far less than its potential if unrestricted.  I find it a very sad story.  Only sad because the game is so good otherwise.  If the game was nothing remarkable to begin with, I wouldn’t care about it.

With the ability to make maps, the battle variety shoots way up.  Basically, every battle you have to figure out new terrain.  There’s a kind of realism here that can’t exist in historically accurate maps of the real thing.  Lee hadn’t fought the battle of Gettysburg 100 times when he decided not to “move around to the right.”

Now, a partial move toward this has been made.  One important part of the maps can be modified, the part which says what ground is forest, what is wheat field, what is stone wall, and so one.  What can’t be changed is the elevations, and the textures on the ground. And of course, since we can’t add new maps, I have to overwrite existing ones with the modifications.

The good news is, last Friday I started a random map generator.  I could have done this far earlier if real map modding was allowed, but oh well.  For now, we can have random terrain on an existing heightmap with existing roads and creeks.  Still way better than before.  The locations of forests and stone walls are critical.

This is multiplayer, of course, so it’s necessary for all players to have the same map.  My auto updating system comes in handy here.  I run a script that runs the map generator, and deploys it as an update all at once.  Then all the players run their updater to get the map, and we’re ready to play.

I’ve also made a heightmap generator which will sit idle until such time as the map making restriction is lifted.

1 Comment