As part of an initiative to start looking at various technologies and methodologies that will make revolutionise video game development, we were looking at various programming languages that would help scalability in servers. Multi threaded C/C++ code, design patterns, libraries, frameworks, and finally programming languages themselves fall under this division.

Subsequent to my earlier blog post on Erlang (refer: http://blogs.sierraatlantic.com/2010/10/erlang-and-experiments-with-scalable-servers/) we whipped up a quick server to test its scalability. Srini (a fresh graduate from University of Hyd) and Vijayender worked on the server side of the game while Chaitanya (not me) worked on writing a client using the Blender API.

It was in many ways an experiment on the productivity of the programmers. Vijayender, with his vast experience, was mostly involved in design while Srini and Chaitanya were responsible for actual implementation.  Keep in mind that Chaitanya and Srini have each just about 2 months of experience and I was already demanding them to write multi threaded code. However the results definitely surprised me.

We planned on making a very simple server that would cater to multiple incoming connections. The overall idea is that we would use one thread to run the networking related code, one thread to do the event management and one thread to perform collisions and the like. The reason for this was straightforward since this is a very simple form of threading (using the producer consumer pattern).

So how long did it take for us the finish this application? 2 months. Yes, that’s right. Just two months with a brand new pair of hands who have no experience writing multi threaded code.  The number of bugs that were reported? Less than 5 and all of them were deviations from the specification. None were technology related bugs.

The following graph depicts the throughput. X-Axis shows the number of clients while the Y-Axis shows the average time it took for a single update (in milliseconds)

The point to notice here is not how long each update took to perform. It is how the server scales automatically when we throw more cores at it. The performance is bound to improve if we switch all of the collision and mathematical calculations to a C/C++ interop (maybe even CUDA/OpenCL… hmmm).

Notice how the 800 clients an 8 core processor gives more than 30 fps consistent. Every time we figured an increased load on the server, all we had to do was toss more cores at it and the performance pretty much improved linearly.

Having worked extensively on multi-threaded code I have come to believe that the future lies in languages that are designed with threading in mind. Multi-core architectures are not going anytime soon and this is exactly why writing code that scales well is of extreme importance. I believe that it is better to toss more hardware at a problem than tossing programmers and Erlang seems to be a good choice.

Notes:

Here is a list of things that are left:

• Mathematics interop in C/C++
• Checking the numbers with Hipe VM
• Checking the numbers on multiple operating systems
• Hopefully try it out on the cloud
• Persistent store (database integration)

Here is the current setup:

• Windows 7
• Intel i7 CPU (8 hardware threads)
• 6 GB RAM (although we were never limited by the RAM)
• Erlang R14B 5.8.11

The problem:

They say music is the language of the universe. That sounds (pun intended) good but what about written languages? Since I can read/write English I have been lucky enough to be able to visit most of the websites. There are many others who do not and when they visit websites, the default language is English and that is when things go awry. Some websites have a localised version of the webpage but how does the user know how he can view the page in the language of his choice, without actually knowing that he has to spot the text “Language”? This assumes that the user can read/write English. Here are a few fixes that I propose. Mind you that some of them are already in use.

Possible solutions:

IP based:

We can map geo-ip to a language since the ips allocated to countries are well known. However this threw a problem for me when I was in South Korea. Initially I could not read/write Korean and I had no idea how I could change the language to English on MSN.com . I finally used a proxy based out of UK to get the pixel location of the term “language” and mapped it to the Korean version of the page. This hack does not work all the time since the webpage layout might be different for different territories (not to mention the browsers).

Login based:

When a user logs into the website we can set the language of the pages based on his preference at the time of the account creation. This is by far the best solution. It requires pain for the user only at the initial account creation. However the problem moves to the default webpage that displays the login screen.

Scrolling/cycling text:

A ticker that scrolls on the top right portion of the screen that says “language” in multiple languages is the solution that I love the most. Since humans (being predators) are attracted to movement this solution relies on grabbing a user’s focus to the most important piece of information for the page. Obviously the user will ignore all the text that is incomprehensive and latches onto the text that is in familiar language. A drop down just below it should hold a list of all languages.

Flags based:

A slight variation on the above is to use flags instead of text, especially where the locales might not be properly supported. The only problem with this is when a single country has multiple languages (ex: India).

Conclusion:

In my humble opinion, I’d go for the scrolling/cycling text solution since it is easy to manage and ensures that all the possible languages are covered. I’d love to hear any interesting solutions that people have come across.

Unless one of these (or any new) solutions will be incorporated, the problem will only grow manifold as the internet age reaches out to every nook and corner of the world. The information revolution cannot be fully realized unless web content in localized languages become a reality.

Majority of the game engine makers have now taken “real” 3D seriously and I have seen a lot of their implementations. I must say that with a slight bit of variation they all have a bright future. While stereoscopic rendering is nothing new, it has implications on the framerate. The most basic form of stereoscopic rendering is to render the same scene twice twice with a slight offset. This is designed to emulate the way humans see. Where rendering differs from the human is with the composition. When humans look at things around them, there are two full frequency images projected onto the retina. The brain produces a composite image from these two images. A stereoscopic renderer  The two renders are composited into a single scene by using filters (the red and the green glasses or the polarised glass that people put on).

A little introduction to Erlang. Erlang is a language that was designed around concurrency as a core feature. It is a functional language and supports threading as a first class primitive. This is one of the features that set it apart from most other languages, primarily C/C++ and Java. These languages rely on libraries to provide concurrency primities (threads, processes, locks and syncs). Since Erlang had this designed into the language, threading and associated issues have been reduced to  minimum.

Why Erlang?

Games today need to scale smoothly. The advent of multiple cores on each processor has resulted in an almost a feverish rush toward the parallel processing paradigm. However anyone who has written large chunks of code will appreciate the problems that would naturally arise from using threads. Sprinkling a lot of locks around the threads hardly improves performance, the very reason we add threads. At the same time failure to keep synchronisation and race conditions in mind will result in a lot of late nights debugging the code. I faced this exact problem a few years ago and out of purely academic interest I started searching for languages that could make things better (for me atleast). This was when I discovered into Erlang.

Erlang uses message passing to communicate between processes. While this is not a new thing, having it as a construct within the langauge ensures that code written in Erlang is inherently concurrent. Since messages passed between processes give up ownership of data, it prevents locking and thus makes IPC faster. At the same time processes in Erlang are faster than native OS threads (hence even OS processes).

The biggest advantage, in my opinion, is loading and replacing code at runtime. Service Oriented Architecture is one area that will greatly benefit from this. It means that the downtime is extremely minimal. Who enjoys disturbing a gamer while he is playing only for “scheduled downtime”? It also encourages prototyping and rolling out faster updates. This is of great importance especially when gameplay balancing comes into picture.

Being a functional language code written in Erlang is generally shorter than equivalent code written in C/C++ or Java. This results in easier verification of the correctness of the code and fewer bugs creep up because of this. Code reviews are also faster to perform.

What are we up to?

As part of our casual game development division we’ve been at work trying to find various architectures and schemes that make scalability as smooth as possible. Some solutions we have implemented are based on the cloud while others like the Erlang initiative are projects that take a different approach for solving the problems.

We currently are working on a nice little multiplayer game that has a flash frontend and runs an Erlang server in the backend. Erlang is good with IPC hence we spawn a new process for each of these dungeons and for each entity in the game. Purely client side effects are cosmetic effects while physics, entity spawn and destruction and scores are simulated on the server. Since Erlang messages are asynchronous we have to perform some double checks, especially during scoring. Doing this was extremely easy since we decided to only trust the server and force the client to reflect these changes. Essentially the client presents a view of the world for the player. Since Erlang is good with concurrent code, we’ve had a chance to simulate cool particles on the server side. However we shifted all of this to the client side since particles are only cosmetic we decided to move this over to the client. It also resulted in saved network bandwidth.

Work in progress:

We are working on coming up with metrics that will compare existing frameworks with what we are implementing, for the same set of features. This will give us a better idea as to where we are heading and where we need to be in the coming few weeks.

A test automation that will reduce the workload of testers is definitely something we are desperate for. The usual click based testing frameworks help, but I am hoping that my team here will come up with something that is far superior to anything I’ve seen till date. Again, I would be interested in seeing if we can develop a load/stress testing suite that can use Erlang underneath.

Integrating 3rd party APIs, especially some of the payment gateways. This will be a huge leap ahead for us, especially since micro transactions are a popular way to monetise in casual games.

Metrics to check the load bearing characteristics of the servers.

Metrics for the kind of bugs we reported on the servers.

Challenges:

The biggest challenge we face is with respect to the number of people who are aware of Erlang as a programming language. This has resulted in some training time for the members of the project.

The next challenge we have is integration with 3rd party libraries. While Erlang has bridges for running C code (even Python ) these need to be extensively tested before we can start using them.

Integer performance seems to be better in C/C++. I am guessing that this would be because the CPUs nowadays are really good with predictable data. I am not sure how we can tackle this in Erlang.

Having worked in Erlang, I can say with a certain amount of confidence that it has definitely proven itself on the scalability end. We need to see what are the limits to it’s execution and this is where Sierra Atlantic’s expertise in testing and test automation will help me in the coming few weeks.

I hope to post more cool stuff in the coming weeks. Till then, tah.

A long time ago, in a kingdom far far away HPC was restricted to the likes of Cray. There was also a time when GPUs were merely specialised silicon that could draw pixels a little faster than the CPU could. Around 1998 nVIDIA release the Riva TNT and things began to change, albeit slowly. Then came hardware T&L support from nVIDIA’s Geforce 256. Games did not look the same after that. And time moved on… . GPUs had a fixed function pipeline for most of their existence till programmable shaders entered the picture. This allowed programmers to treat the GPU like another programmable chip. The one thing to notice here is that all these GPUs were specialised processors to make fast games render faster.

Around this time, GPGPU (General Purpose GPU)  initiatives by the GPU manufacturers saw GPUs being used for not only rendering but also general purpose computing. One drawback was that GPUs still liked textures and matrices and disliked strings.  However with the introduction of CUDA by nVIDIA, OpenCL by Khronos group and CAL/CTM from AMD/ATi GPUs changed the terrain in favour of programmers. To explain the difference between a CPU and GPU’s processing capabilities, I’ll use the following video: http://www.youtube.com/watch?v=fKK933KK6Gg . The Mythbusters showed exactly what it is that the GPUs are good at – They are good at doing a designated task incredibly fast only when it is performed in parallel. Using that same video as an example, the GPU would have a hard time drawing the smiley, one dot at a time. However if you were to build a new parallel gun that has the smiley imprinted on it, the drawing happens in an instant.

If you were to harness the power of the GPU, you would need to write a program in a way that it can execute instructions in large chunks. The size of these chunks depends on the GPU (or the CPU). The operating model for these programs is based on the SIMD model – Single Instruction Multiple Data. What this means is that you have a single instruction being executed simultaneously across multiple chunks of data. An example for where SIMD would be a good fit is calculating the magnitude of a vector. magnitude_square = x*x + y*y + z*z.  The three multiplies could be performed in a single go.