Further thinking about stats and skills

We are still thinking about how we want to handle the customization. On one side the option to give full customization is tempting, as it would really allows game owner/designers to change the rules as they want. On the other side it will for sure increase the difficulty for new comers to approach the system as they may need to write some code to change the rules.

This can be of course mitigated by the fact we offer some default content, documentation and tutorials but it may scare some people.

Does it then make sense to take this road or would it be better to offer mainly a form which need to be tweaked?

Let's take a possible example: our "Life" or "HP" stat:

// Example of the script behind the HP stat.
// Function returning the currently maximum allowed value.
function MaxValue()
{
    return API.GetStat('Level')*20;
}
// Function returning the currently minimum allowed value.
function MinValue()
{
    return 0;
}
// Function run every time the value is modified
function ValueChanged(newValue)
{
    if(newValue < 1)
    {
        API.Teleport(0,0,0,0);
        API.SetStat('HP',1);
    }
    if(newValue > MaxValue())
    {
        API.SetStat('HP',MaxValue());
    }
    UI.UpdateStatBar();
}

In this example the API and UI are functions offered by the engine. While the functions defined in the stat code will be called from the engine at some points. As you see this could open the door to multiple features like a "berserk" skill which could trigger once you reach 10% of life.

To implement this, I will first have to develop the full parser / runtime of this language, however if I do it now, it will be possible to use it later on for plug-ins development which would mean we would have a single language used in multiple places.

Also having such language could be used on the objects themselves for example a potion could restore a stat or kill you.

Again it's a question of balancing flexibility to user friendliness (easy to use) and here I'm a bit unsure which road we should take.

Stats points, skills, make the engine flexible

To make the engine an engine and not just a fixed  game which would offer little flexibility the engine should let you define the stats points and the skills.

As we want to let the game owners create the games they want, being vampires, gangs or medieval, the engine must let you create whatever stats points you want as well as define any skill you want.

Stats would be like your current value on something, being the money, your life, or experience. Stats could recover themselves have triggers when reach some value (you die if you reach 0 life for example) or have a maximum level (energy for example). All that's is just matter of letting you define a name, and some fields. But what if a stat maximum depends of a "level" or depends on what you wear? Here the things starts to be a bit more complex and require some sort of logic built on the stats which must be defined by the game owner.

Of course we must offer standard stats with a standard logic such that new comers don't spend 10 years learning our tool but can start with something and later on tweak it.

To build all that, it will require also tools to view / modify those information, and store / retrieve them from the database. The rules must be defined by the owner and stored per game, and for each player a set of current stats must be stored as well.

All this require quiet some work which will not directly be visible, but it's a must to make the engine an engine.

Initially I plan to offer really few default stats, and we may increase the default offered later on:

• HP / Life
• Money
• Experience
• Level

As skill, I want to offer those one:

• Attack
• Defense

As you see it's really really limited, but if we manage to create a good system, adding further one in the default setup will be a piece of cake.

Hopefully we will not already need a full scripting language to support those stats / skills but it may end up nearly the same, with complex formula and actions to trigger when the values changes.

Coordinate transformations

Any graphical game will at some point transform coordinate from one kind to another. For example, for a grid map, you will need to transform the map to screen coordinate, which at first is really easy: screenX=mapX*tileWidth and the same for the Y coordinate.

On the other side, what if you want to center the screen around your player? Already there there is a bit more transformations, with some offsetX and Y for the top left corner for example. What if your maps are split in different areas? Like an area is a 100x100 like in my case, as soon as you goes out of this area you need to change the area index and restart the X,Y coordinate on the map.

You slowly see how I'm heading? The more features the more complexity for your coordinate system. And guess what? What you do in one direction you will most likely need to have in the other, what if I click on the screen and need to know on which cell of my map I'm? And here you are with the reverse of the previous calculation.

Therefore, it's smarter and certainly safer to have those transformation stored in two functions, and then always call those function. That will allows to debug only once and then be assured it will always work (or at least so it should be).

Example of my screen to map coordinate transformation:

public ScreenToMap(x: number, y: number): RenderScreenCoordinate
{
    var pos = $("#gameCanvas").position();
    var tileWidth = game.World.tileSetDefinition.background.width;
    var tileHeight = game.World.tileSetDefinition.background.height;
    var orx = Math.abs(this.offsetX) % tileWidth * (this.offsetX < 0 ? -1 : 1);
    var ory = Math.abs(this.offsetY) % tileHeight * (this.offsetY < 0 ? -1 : 1);
    var x = (x - pos.left) + this.offsetX;
    var y = (y - pos.top) + this.offsetY;
    var ox = x % tileWidth;
    var oy = y % tileHeight;
    x = Math.floor(x / tileWidth);
    y = Math.floor(y / tileHeight);
    var cx = this.areaX + Math.floor(x / this.world.areaWidth);
    var cy = this.areaY + Math.floor(y / this.world.areaHeight);
    var tx = x;
    var ty = y;
    if (tx < 0)
        tx = (this.world.areaWidth - 1) - (Math.abs(tx + 1) % this.world.areaWidth);
    else        tx %= this.world.areaWidth;
    if (ty < 0)
        ty = (this.world.areaHeight - 1) - (Math.abs(ty + 1) % this.world.areaHeight);
    else        ty %= this.world.areaHeight
    var rx = tx + (cx - this.areaX) * (this.world.areaWidth - ((cx - this.areaX) < 0 ? 1 : 0));
    var ry = ty + (cy - this.areaY) * (this.world.areaHeight - ((cy - this.areaY) < 0 ? 1 : 0));
    return { TileX: tx, TileY: ty, AreaX: cx, AreaY: cy, RelativeX: rx, RelativeY: ry, OffsetX: ox, OffsetY: oy };
}

As you see, not really a 2 line function. Yes some comments would also help to read what's going on, but that's more for an example than anything else.

Path solving

Path solving in game programming is something you will mostly use at some point. This kind of algorithm let you find ideally the shortest / quickest way between 2 points, either by using connections between points or cells on a grid map.

The most well known and the best one to use in most case is called A*.

A* is nothing else than a code which will try all possible routes and use the shortest one when it find it. At start it will goes in all the possible directions, add those as new starting point, and repeat. As optimizations you could sort the list of path to try by using the nearest one to the goal first. Of course don't test twice the same node or cell of the map.

Path solving may have some drawbacks: it may take quite some time to solve, so you may want to avoid testing it every single time. Second, if you work at a pixel level it may end up doing WAY too many checks, maybe use a grid on top to make things faster could help you. If blocking elements move you may need to rework you path, to avoid having to do it all the time, you may do it only when you get blocked while traveling the path.

As possible example of a running A* algorithm:
http://bgrins.github.io/javascript-astar/demo/

The same kind of algorithm will work in mazes or with weight on the cost of travel (for example swimming could be slower than walking).

On my own I have a couple of more issues to solve, like for example as my maps are split in "areas" crossing an area change the way you need to handle the path.

Another issue is that maybe I don't want that you can solve a maze simply by clicking the goal, as it would spoil the game. To solve this, I may simply introduce a limit on the number of steps my path solver will try to go through.

Having a good / smart path solver will actually increase your game experience a lot therefore don't spend too little time on it.

Async callback nightmare

One of the main complain I can have with node.js is its way to handle the "asynchronous" calls. Basically some calls take time to run, for example querying a database or connecting to a remote host. Instead of blocking the single thread on which your node.js code runs, those function will call you back once the operation is completed.

At a first thought you may think: great, I don't have anymore blocking calls and at the same time I don't need to deal with multi-threading and possible locks / semaphores.

Indeed this model where every part of your code runs within one thread and you are called when there is something for you is great to solve multi-threading issues. It is also great that you don't have to deal with shared variables or hoping some code is not interrupted in some nasty areas.

Yet, many tasks do require a cascade of events like:

1. Connect to a database
2. Execute a select
3. For each elements of the result update a value
4. Close the connection & free up
5. Return the values

As you see, there is no way you can run on parallel these tasks, and you really need the result of the previous step to do the next one.

In node.js such code could be written like that (it's more pseudo-code than a real API):

db.connect(function (err,conn)
{
   conn.executeQuery("select * from users",function (err2, results)
   {
       for(var i=0;i < results.length;i++)
       {
         conn.executeQuery("update users set gold=gold+10 where id="+results[i].id,function(err3,results2)
        {
        });
     });
   });
});

Ooo wait! There is a bug! You can't call executeQuery to update within the loop, as the queries will be sent all in parallel and may actually be an issue if you have a limit of the number of queries to run at the same time. Would be better to run then one after the other. Also this is by far not readable if you end up in the 10th callback function.

So how can we solve that?

There is some work around found on the net, for example: https://github.com/yortus/asyncawait

I didn't tested them yet, but it should solve exactly this kind of figure where you need to do the operations in sequence (and believe me it's more frequent than to run them in parallel). This library don't actually block node.js after your await call, instead it will call you back transparently once the function you await completed.

As said that needs all to be tested to further understand how that works and if it works well, but hopefully I can clean up the mess created by an actually poorly thought framework. Why am I so aggressive against node.js? Because those problems should be solved at a language level and not via some 3rd party library. And if so many developers have the same issues as me it means that should be actually be a problem solved at the root.

Bugs... stupid bugs...

When you start developing you fight mainly with the language and the framework. While your knowledge grow you still fight but usually you fight for an algorithm.... or stupid bugs which resists you.

Today I fought about 3 hours to find a single little number which was wrong:

area.actors.splice(i, 0);

Instead of

area.actors.splice(i, 1);

Result? The first code do... nothing, while the second actually removes an item from an array.

You could think that his kind of bugs can be solved in no time, sadly the bigger the code base the harder it is to find such bugs. I had to isolate the area which produced this error, try to make sure it wasn't somewhere else, and at the same time I discovered a few other things which didn't really made sense either. As said... 3 hours for this.

Anyhow now I have rats walking around my world:

The movement of the rat is mainly random, it doesn't head yet toward the player. However I implemented collisions with background tiles, and for example water is non walk-able.

I will have to implement collisions with objects as well, yet for this part I need to think how I want it. Will it be a circular area round the ground position? Or could I choose the collision shape? All open questions.

I will need also to create monster spawner, which will let the game owner place monster where he/she want and not simply randomly scattered around the map.

Monster handling

Code design (or game design) is sometimes more subtle than you may at first thing. In this case I'm talking about where to place some piece of code such that it is both logic and can handle the situation in a smart way.

My current problem is to make the monsters walk around the map, and under certain condition toward the player such to attack him later on. The first idea you would get is that a monster is an object by itself and therefore there is a class "monster" which will have all the logic. But is the logic of the monster not somewhat shared by the player as well? Will the monster and the player not somehow walk around on the same world? You see already that maybe both monsters and the player will have a parent class let's say "actor" which may contains the x,y position and some other information. Yet the monster will have a different logic as the player, as the player will be handled all by the guy/girl behind the screen while the monsters should walk alone. So some part will be specific for sure.

Also, how many monsters will we deal with? The whole infinite world? Or shall we keep the monsters only for the areas we have currently in memory? Quite certainly this second case, so monsters should be kept with their area, and when we destroy an area we will stop handling them as well. Freshly loaded or generated area should bring new monsters and therefore we will handle only a limited set of monsters at a given time. Infinite number of object handling is simply not possible so that seems a good option... yet what if a monster shall cross an area border? Then it need to be placed on the new area or.... killed if it goes out of the currently handled areas.

Remains also to think about the rendering, ideally the render part of the game engine should deal with players, monsters and NPC basically in the same way.

All that makes the whole idea a bit fuzzy / complex and therefore maybe need some further refinement.

What if we introduce "pets" or "friendly monsters"? What if we want to have monsters acting as a group? And so on... so I clearly don't have yet all the answers, and will try to clarify my mind while working on it.

New phone received => not much work done

Today I received a new phone, and therefore wasn't really so productive. I must also say I didn't slept well, and that doesn't help either.

Anyhow, just to make a short story really short, I spend my time moving stuff from my old phone to the new, configure the new one and sell the old one to somebody. Really productive isn't it?

What's good is that with all the "cloud" hosting of your settings most of it is actually pretty automatic, while the remaining parts are handled by a nice little touch of the phone maker: Sony in this case. I ended up having all my data on the new phone without having to re-type anything. Cool!

JSON Schema

Today JSON files replaces slowly but surely XML files. JSON certainly is more compact and is easier to work with in JS as they are nothing else than JS objects (JS Object Notation).

However even if JSON files are convenient for many things, being smaller, being faster to write, they had a huge drawback in my opinion over XML. The lack of schema definition which would allow to test if a JSON file is valid or not and makes the IDE aware of what we could write within.

Yet some IDE starts to support JSON Schema definitions ( http://json-schema.org/ ) for example Visual Studio (which actually is the one which interest me).

That finally close the issue I have with JSON and let me enforce a bit more strength inside my JSON configuration files.

Let's start with a small JSON file:

{
  "stores": [
    {
      "name": "My Little Book Shop",
      "owner": "Me Myself",
      "location": {
        "address": "Av. Somewhere 29",
        "city": "Someplace",
        "postal_code": "5000",
        "country": "MyCountry"      }
    }
  ],
  "books": {
    "How To Code for Dummies": {
      "isbn": "12389298732",
      "price": 53.12
    },
    "I love ponies": {
      "isbn": "98727356128",
      "price": 23.99
    }
  }
}

This file contains book shops and a list of books we can find.

Now so far nothing special, however how to ensure that the content of this JSON file is correct and that the next time we type something in we don't make mistakes? Quick answer: by using a JSON Schema!

To do so in our JSON file we need to add a property (usually the first one): "$schema": "book.schema.json"

This tells the IDE that for this JSON file we will use the file book.schema.json as schema. Of course this file is now missing so let's create it:

{
  "$schema": "http://json-schema.org/draft-04/schema",
  "properties": {
    "$schema": { },
    "stores": { },
    "books": { }
  },
  "additionalProperties": false}

This small schema file defines that the JSON file can contains ONLY 3 properties: $schema, stores and books. Everything else would be prohibited. That's already a first step, but let's define what in the stores can be placed as value:

{
  "$schema": "http://json-schema.org/draft-04/schema",
  "properties": {
    "$schema": { "type": "string" },
    "stores": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "name": { "type": "string" },
          "owner": { "type": "string" },
          "location": {
            "type": "object",
            "properties": {
              "address": { "type": "string" },
              "city": { "type": "string" },
              "postal_code": { "type": "string" },
              "country": { "type": "string" }
            },
            "additionalProperties": false          }
        },
        "additionalProperties": false      }
    },
    "books": { }
  },
  "additionalProperties": false}

The same can be done for the "books" property, yet this one is a key / value pair and therefore need to be defined as "additionalProperties" while defining the type stored:

{
  "$schema": "http://json-schema.org/draft-04/schema",
  "properties": {
    "$schema": { "type": "string" },
    "stores": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "name": { "type": "string" },
          "owner": { "type": "string" },
          "location": {
            "type": "object",
            "properties": {
              "address": { "type": "string" },
              "city": { "type": "string" },
              "postal_code": { "type": "string" },
              "country": { "type": "string" }
            },
            "additionalProperties": false          }
        },
        "additionalProperties": false      }
    },
    "books": {
      "type": "object",
      "additionalProperties": {
        "type": "object",
        "properties": {
          "isbn": { "type": "string" },
          "price": { "type": "number" }
        },
        "additionalProperties": false      }
    }
  },
  "additionalProperties": false}

The schema is now complete for our JSON file, you could then check if something is valid or not, and also the IDE should propose you what kind of properties a given object should have. To further improve the schema I would strongly suggest to provide a description of the properties as well and therefore further help you while entering data within the JSON file:

{
  "$schema": "http://json-schema.org/draft-04/schema",
  "properties": {
    "$schema": { "type": "string" },
    "stores": {
      "type": "array",
      "description": "Lists all the shop in the francise.",
      "items": {
        "type": "object",
        "properties": {
          "name": {
            "type": "string",
            "description": "Name of the shop."          },
          "owner": {
            "type": "string",
            "description": "Name of the owner of the shop."          },
          "location": {
            "type": "object",
            "description": "Location of the shop.",
            "properties": {
              "address": {
                "type": "string",
                "description": "Address of the shop"              },
              "city": {
                "type": "string",
                "description": "City where the shop is."              },
              "postal_code": {
                "type": "string",
                "description": "Postal code."              },
              "country": {
                "type": "string",
                "description": "Country of the shop."              }
            },
            "additionalProperties": false          }
        },
        "additionalProperties": false      }
    },
    "books": {
      "type": "object",
      "description": "List of all the books the shops can sell.",
      "additionalProperties": {
        "type": "object",
        "properties": {
          "isbn": {
            "type": "string",
            "description": "Unique ID linked to a book. The ISBN is usually printed on the book itself."          },
          "price": {
            "type": "number",
            "description": "Usual price the book will be sold at."          }
        },
        "additionalProperties": false      }
    }
  },
  "additionalProperties": false}

Finally you can test the schema and the JSON provided here on an online validator like:
http://www.jsonschemavalidator.net/

The validator will not show the description while you type but will ensure the properties are acceptable for example.

After all this work, if you have a good IDE, you should have a result like this while working on your JSON file:

Map compression

Work on the map saving & restore went further and in order to decrease the amount of data stored on the server I decided to compress the map while keeping the data as string (and therefore don't have issues with JS). If would accept to keep the data as binary I could use a GZip or something similar.

The first step would be to store numbers in another format to be able to reduce the size. 0-9 cannot be reduced if you want to keep one them split and not merge them into a single byte. However bigger numbers like 2112 (just a number picked randomly) takes 4 bytes to write as string yet could be stored in a more compact form. Either being in HEX or... by storing it to a more compact form. For example using a-zA-Z as base (0 being a, 1 b and so on).

Let's write a small JS function for that:

var numberCompressionPossibleChars = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ";
// Numbers must be positive!
function NumberToString(source, nbChar)
{
var result = "";
var rest = source;
for (var i = 0; i < nbChar; i++)
{
result += numberCompressionPossibleChars.charAt(rest % numberCompressionPossibleChars.length);
rest = Math.floor(rest / numberCompressionPossibleChars.length);
}
return result;
}

 A JS Fiddle has been created for it: https://jsfiddle.net/68konLxw/

To transform this back into a number:

function StringToNumber(source, position, nbChar)
{
var result = 0;
for (var i = 0; i < nbChar; i++)
{
var c = source.charAt(i + position);
result += numberCompressionPossibleChars.indexOf(c) * Math.pow(numberCompressionPossibleChars.length, i);
}
return result;
}

And the JS Fiddle here allows to go from one and back: https://jsfiddle.net/gLwtwo73/

Now that's just the work of a single number, and we don't gain much unless the numbers are big. But what can we do on an array of numbers? Well, the next step is to implement the Run Length Encoding which counts the number of times the same piece comes:

// Numbers must be positive!
function NumberToString(source, nbChar)
{
var result = "";
var rest = source;
for (var i = 0; i < nbChar; i++)
{
result += numberCompressionPossibleChars.charAt(rest % numberCompressionPossibleChars.length);
rest = Math.floor(rest / numberCompressionPossibleChars.length);
}
return result;
}
function StringToNumber(source, position, nbChar)
{
var result = 0;
for (var i = 0; i < nbChar; i++)
{
var c = source.charAt(i + position);
result += numberCompressionPossibleChars.indexOf(c) * Math.pow(numberCompressionPossibleChars.length, i);
}
return result;
}

JS Fiddle of the array compression: https://jsfiddle.net/gfpyq7qo/

The first few characters of the compressed array contains the number of character used to compress the number.

And the last piece of our work, decompress this array:

function StringToArray(source)
{
var result = [];
var strNb = "";
var i = 0;
for (; i < source.length; i++)
{
var c = source.charAt(i);
if (c == "-")
break;
strNb += c;
}
i++;
var nbChar = parseInt(strNb);
strNb = "";
for (; i < source.length; i++)
{
var k = source.charCodeAt(i);
if (k >= 48 && k <= 57)
strNb += source.charAt(i);
else
{
var nb = StringToNumber(source, i, nbChar);
i += nbChar - 1;
if (strNb == "")
result.push(nb);
else
{
var n = parseInt(strNb);
for (var j = 0; j < n; j++)
result.push(nb);
strNb = "";
}
}
}
return result;
}

And the last Fiddle of this post: https://jsfiddle.net/p3o0rems/

Load & Save

Persistence in a game is something mandatory but is often not enough thought. At least in my opinion.

Also persistence is split between player state and world persistence. In many games the player had little to no influence to the world or at least the influence is only temporary. For example you break a lamp save quit the game and reload and the lamp will be as new. What you wear and your stats are usually saved and kept.

Nothing wrong with this, I mean, you have to think about how much it would take to store every little changes you do on the world.

So how could the persistence be implemented for a web game? First you need to think about what you want to do with the data. For example, if we work like me with a 2D grid of tiles, the server may not need to even handle them, and I quite certainly don't want to make odd queries on those data. Therefore a straight forward serialization like JSON.stringify will do the trick.

On the other side, if you would like to make reports on some data, for example find the best players, or those which has more money in bank, serializing the player class and storing it "as is" is not a good idea. Better then to store the values in a format you can then query correctly.

Another thing to think of is if you want to compress your data somehow or you agree to use all the space as needed. Compressing can either be done with libraries like ZIP, GZIP or such or simply do run length encoding (RLE for short). RLE is a simple method which counts how many times you have the same thing coming, and then having a number plus the thing. For example the string "AAAAABBBBCAAAD" could be written like "5A4B1C3A1D", reducing the string from 14 character to 10 which means a compression of 28% less space. Some times it works some times it doesn't specially if each character is used only once or very few times.

The compression could be made on the client such that the data would also take less space over the network and ideally be faster, or could be done on the server if it is something too complex to do on the client side and what you want is just save space on the your server disk.

Backend of my hobby project => Node.JS

My hobby project is progressing, but before I can really go on, I need to start working a bit on the back-end allowing to store data, for example maps, as well as having a bit of access control.

As you may have saw from my previous posts, I currently decided to use Node.JS simply because it allows me to more easily host my project on an existing Linux VPS I already rent.

Node.JS has many advantages like being able to develop using a single language between the front-end and the back-end, having cool features like socket.io which handles the web sockets for you, and much more.

However Node.JS do have drawbacks (at least in my opinion). All (or most) works using asynchronous call, for example, you connect to a database (async) and make a query (yet another async). You end up easily to have loads of nested anonymous functions just to handle such cases.

This of course makes the code harder to read and to write. But you have also issues about error handling. If an exception is fired, you may simply fail to be able to catch it, which means normally to kill your server, and requires a restart (sure it can be automated). This means whatever you had in memory is lost.

Let me show you an example:

app.post('/backend/OwnerExists', function (req, res, next)
{
    if (!req.body.user)
    {
        res.writeHead(500, { 'Content-Type': 'text/json' });
        res.write(JSON.stringify({ error: "parameter 'user' is missing." }));
        res.end();
        return;
    }
    var connection = getConnection();
    if (!connection)
    {
        res.writeHead(500, { 'Content-Type': 'text/json' });
        res.write(JSON.stringify({ error: "connection failed." }));
        res.end();
        return;
    }
    connection.connect(function (err)
    {
        if (err != null)
        {
            connection.end();
            console.log(err);
            res.writeHead(500, { 'Content-Type': 'text/json' });
            res.write(JSON.stringify({ error: "error with database." }));
            res.end();
            return;
        }
        connection.query('select id from game_owners where name = ?', [req.body.user], function (err1, r1)
        {
            connection.end();
            if (err1 != null)
            {
                console.log(err1);
                res.writeHead(500, { 'Content-Type': 'text/json' });
                res.write(JSON.stringify({ error: "error with database." }));
                res.end();
                return;
            }
            // Not yet registered            if (r1.length == 0)
            {
                res.writeHead(200, { 'Content-Type': 'text/json' });
                res.write(JSON.stringify({ result: false }));
                res.end();
                return;
            }
            else            {
                res.writeHead(200, { 'Content-Type': 'text/json' });
                res.write(JSON.stringify({ result: true }));
                res.end();
                return;
            }
        });
    });
});

This connects to a MySQL database and makes a single query. You see already how many nested function I have.

Sure there is ways to improve the situation using 3rd party libs or working a bit differently, still at the end of the day the same will happen => each connection / query will require a new function callback.

I really wonder how you can handle big projects written with such framework.

Map editor is progressing

The map editor of my hobby project is progressing. Beside having finally plants and trees placed on the grass by the world generator, the map editor has by default a grid display to make it a bit easier to see where the tiles are. You can disable it if needed.

Also I worked on the first "functions" of the map editor which now let you choose either by painting tiles or placing / removing objects.

To make sure the render engine is not slowed down by searching the objects to render, a grid cache has been implemented which means objects are placed in a grid of the same size of tiles.

The drawback of such optimization means each time you change the objects or change the position of an object you will need to update the grid cache.

To make the look more natural objects are not placed on a grid either, they are freely to be placed anywhere you want on the map.

Finally, to make my life easier to deal with the JSON art definition I improved the typescript interface which let me directly access from within Typescript the the JSON:

interface TilesetInformation
{
    background: TilesetMap;
    objects: TilesetObject;
}
interface TilesetMap
{
    file: string;
    width: number;
    height: number;
    types: TilesetType;
    mainType: string;
    transitions?: TilesetTransition[];
    levels?: TilesetLevels[];
}
interface TilesetType
{
    [s: string]: number[];
}
interface TilesetTransition
{
    from: string;
    to: string;
    transition: number[];
}
interface TilesetLevels
{
    type: string;
    maxLevel: number;
}
interface TilesetObject
{
    [s: string]: TilsetObjectDetails;
}
interface TilsetObjectDetails
{
    file: string;
    width: number;
    height: number;
    groundX?: number;
    groundY?: number;
    x: number;
    y: number;
    frequency?: number;
    placeOn?: string[];
}

Having a well defined JSON file ensures I don't mess around while trying to access information.

Perlin Noise and world generation

In the video game world, there is one formula which is used a lot while trying to generate procedural worlds: Perlin Noise.

Perlin Noise is basically function which returns a value between -1 and 1 for a given X,Y coordinate. Whatever coordinate you will give it will always return a value and if you call it back later on it should return the same value again for a given set of coordinate.

This function allows to generate terrains, where from -1 to 0 for example it would be the sea and whatever is above 0 will have an terrain height.

It works for 3D worlds but for 2D worlds it works just as fine, as you could change the tile based on the value the perlin noise returns.

I'm surprised how few developers I know actually know what this function is, it doesn't mean it is not know, actually it's one of the most used one in the game dev and you can find loads of tutorials and articles like this nice one here:

http://devmag.org.za/2009/04/25/perlin-noise/

Guess what? I will use perlin noise as well for my hobby project Stay tuned to see the results ;)

Node.JS and Visual Studio

With Visual Studio 2015 came the "official" support of Node.JS within Visual Studio. A great news for me which already works mainly with Visual Studio and switching from one IDE to the other is always a pain being for key bindings or just general productivity.

Keep in mind I'm not selling Visual Studio, and that I would not pretend it's the best solution for all. Simply for me it's a commodity to have yet this other possibility within VS.

For my hobby project having it written in .NET doesn't seems all that of a good choice specially for the cost of the hosting, and therefore I tried this weekend to create a new project using Node.JS, the result is actually far from being bad, and after some hard work (to understand a few things, import some definitions and so on), I have it mostly how I want it.

Basically I don't want to code directly in JavaScript and instead develop fully in Typescript which adds really the few tiny bits that makes life easier for developers. I therefore develop both the back-end and the front-end via Typescript and all that within the IDE I use every days.

One piece is missing for me and it's the deployment, too bad there is no SFTP integrated within Visual Studio and I didn't found any extension doing it. It's not a show stopper but it's again something which could improve the productivity.

Let's hope in future this support is even more developed and improved. Let's hope that we could then have native LESS compilation. Let's hope that the Typescript definition could be downloaded like NPM or NUGET packages...

We shall see how it goes. So far I'm already quite happy.

MVC and separation of concerns

There is a couple of "hype" words in the development world which somehow hurts my feeling of old monkey developer. One is "Test Driven Development" and the other is "MVC" framework.

https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller

MVC is an old (1970s) design pattern which could split the responsibilities of a process in 3 pieces. This design pattern evolved a lot since then, and I would say it's not till the last 3-5 years that we really started to hear it for web development.

In an ideal implementation the MVC pattern should really be split in 3 parts, where the model contains just the data (and is maybe responsible of storing / retrieving them from the DB), the controller which received the input of the user and applies the modification to the model and finally the view which should present the data from the model to the user.

MVC pattern as implemented by Microsoft somehow mix parts together, where the controller also serves the view, and the view has more than a little bit of logic inside.

Overall this can be a design choice and can make sense specially in web development, however I tend to find difficult to call those implementations true MVC.

MVC for me has a few points which makes it interesting:
- Separation of concerns (which means each piece of code works only on its main task not mix the tasks of another piece).
- Possibility to change the view (and therefore have multiple views) without having to change neither the controller neither the model. For example to offer multiple interfaces to the same data.

However with those standard implementations, you hardly achieve those 2 goals due to the mixing of the concerns.

Thanks to the new web techs like AJAX which let you now make a Single Page Application, you could actually separate the view on the client, while keeping the controller and the model on the server. This would then allows to have multiple different views sharing the same back-end, and even offering full 3rd party API.

I would personally push others to really think a bit more out of the box and see what their needs are before jumping on the latest framework. Even if you develop with .NET as I do, it doesn't mean you MUST follow all what Microsoft offers. You can certainly use some of the ideas or the tools offered but use them with what fits your project best.

User Interface (UI) testing

While unit testing and integration tests usually focus on the back-end of your code, what your customer will still see is the user interface, or at least for most of our development that's it.

Therefore having tests on the interface is not useless while usually being quite hard to implement.

As I develop mainly web applications and therefore I will concentrate here on this particular kind of UI.

Depending how you developed your software, if it is a single page application (SPA) you could actually develop tests all in JavaScript. It's by far not as crazy as it seems, the issue is that you will need a full browser to run it, and you can't really fire it from a build server.

The other option is independent of how your page is developed and can (and will) run on a build server: WebDriver, Selenium & PhantomJS

If you are like me developing in .NET simply grab PhantomJS and Selenium.WebDriver having that let you have an headless Chrome, with all the binding to pilot it from your test functions.

Initializing the web driver is matter of:

DriverService = PhantomJSDriverService.CreateDefaultService();
DriverService.HideCommandPromptWindow = true;
DriverService.IgnoreSslErrors = true;
browser = new PhantomJSDriver(DriverService, new PhantomJSOptions());

Now that you have your "browser" you can issue your command like:

browser.Navigate().GoToUrl("http://myurl.com");

Finding an element on the page is done via:

browser.FindElement(By.Id(id))

With the IWebElement received you can check the content (.Text) or "click" it (.Click()) or type to it (.SendKeys("xxx"))

All that let you build your automated web UI tests, and finally check those interface to see that a change don't break everything down.

Don't be fooled however: it will be work to really test your UI, but I'm sure that once you have your UI covered with a good number of tests you will see how useful it is.

I did myself a smallish framework to help me writing those tests and at the end a test can be written like:

[TestMethod]
public void QuickSearch_LabelSearch()
{
    Url("/#action=ListBootPc");
    WaitElementStartsWith("contentArea", "Boot PC");
    WaitElement("quickSearch").SendKeys("CR12");
    WaitVisible("quickSearchResult");
    Assert.AreEqual(10, WaitElement("quickSearchResult").FindElements(By.TagName("a")).Count());
}

As you see it's really seen from .NET as a normal Unit Test and therefore can be understood by TFS and run like once a day.

Team Foundation Server Update 2

I'm actually astonished by the speed the development team at Microsoft is working and it may actually be more difficult to get the news than actually get an update.

Today while I was a bit browsing one of the MS blog I found this:
https://www.visualstudio.com/en-us/news/tfs2015-update2-vs.aspx

Of course I wanted to see it in action, and since our Team Foundation Server (TFS for short) server is now running in a standard way, being part of the domain as well as the DB being part of the domain, I thought it should not be such a big deal to update. And so it was.

I still took the precaution to backup the DB as well as taking a snapshot of the VM running our TFS installation, as last time I installed an update I lost quite some time trying to recover from a bad situation.

Anyhow, this time all the precautions have not been used which is even better. I installed the update and after all its automated work I got the TFS running again with the brand new shiny features up and running. Great job Microsoft!

What's clear is that we must dig a bit more in the Release Management part of TFS to see how we could use it.

File re-organized

It took me about 4 hours to re-organize the front end files, I went from a structure where all the LESS files was in a single directory, all the templates in another and finally the Typescript in a Typescript structure.

While this first structure was good to see all the LESS files in one place and so on, it was clearly a pain while you was working on a single feature like an admin panel you would like to have the template, LESS and logic all in one directory such that it would be easier to work with.

Therefore, I took my time, and loads of patience, and moved things around to reflect my wished layout.

It is now possible to simply browse per feature and find all the needed files. I will still need to edit the main.less file to point to the feature less file, but once this is done, you will work simply with that single directory.

This shows how important it is to organize well your files as it actually has quite an impact on your productivity.

Code cleanup and tests

Code is somewhat odd, it should stay as is, it's digital afterward, right? However it seems like a fruit, it rotten over time. With that I mean that if your don't spend time on your code it will slowly but constantly degrade, specially if you continue to add features on it.

The only parade to this is to continue to cleanup your code, improve things, and have the courage to refactor / change pieces even if that was always working.

The drawback is that the more you work on a code the more bugs you may introduce, which means you will for sure introduce bugs while doing so. The cure for it is automated tests, to ensure that at least a good chunk of your code base still work after the face lift you did.

After the Entity Framework conversion, I wrote a small piece of code which would check that at least all my DB tables can be accessed. It doesn't mean the data is correct but it means the ORM doesn't have troubles accessing them (for example wrong table name).

using (var ctx = new IV4.Backend.Model.InventoryConnection())
{
    foreach (var prop in ctx.GetType().GetProperties().Where(p => typeof(IQueryable).IsAssignableFrom(p.GetGetMethod().ReturnType)))
    {
        IQueryable table = (IQueryable)prop.GetValue(ctx, null);
        var rowNb = 0;
        // Read the first 10 rows (just to see if we get data)        foreach (var row in table)
        {
            rowNb++;
            if (rowNb >= 10)
                break;
        }
    }
}

My next step is re-organize the way the code is stored, instead of a directory "template" and a directory "typescript" I will try to put all the things together in "modules", with template, LESS files and typescript all in one directory. The advantage is that when you need to edit one feature you will find all the files inside a single directory instead of browsing 3 different directories.

For the end user again this has no impact (in the best case), but for the developer like me it saves time, which means I can develop faster what the customer want.