Subsequent to my earlier blog post on Erlang (refer: http://blogs.sierraatlantic.com/2010/10/erlang-and-experiments-with-scalable-servers/) we whipped up a quick server to test its scalability. Srini (a fresh graduate from University of Hyd) and Vijayender worked on the server side of the game while Chaitanya (not me) worked on writing a client using the Blender API.

It was in many ways an experiment on the productivity of the programmers. Vijayender, with his vast experience, was mostly involved in design while Srini and Chaitanya were responsible for actual implementation.  Keep in mind that Chaitanya and Srini have each just about 2 months of experience and I was already demanding them to write multi threaded code. However the results definitely surprised me.

We planned on making a very simple server that would cater to multiple incoming connections. The overall idea is that we would use one thread to run the networking related code, one thread to do the event management and one thread to perform collisions and the like. The reason for this was straightforward since this is a very simple form of threading (using the producer consumer pattern).

So how long did it take for us the finish this application? 2 months. Yes, that’s right. Just two months with a brand new pair of hands who have no experience writing multi threaded code.  The number of bugs that were reported? Less than 5 and all of them were deviations from the specification. None were technology related bugs.

The following graph depicts the throughput. X-Axis shows the number of clients while the Y-Axis shows the average time it took for a single update (in milliseconds)

The point to notice here is not how long each update took to perform. It is how the server scales automatically when we throw more cores at it. The performance is bound to improve if we switch all of the collision and mathematical calculations to a C/C++ interop (maybe even CUDA/OpenCL… hmmm).

Notice how the 800 clients an 8 core processor gives more than 30 fps consistent. Every time we figured an increased load on the server, all we had to do was toss more cores at it and the performance pretty much improved linearly.

Having worked extensively on multi-threaded code I have come to believe that the future lies in languages that are designed with threading in mind. Multi-core architectures are not going anytime soon and this is exactly why writing code that scales well is of extreme importance. I believe that it is better to toss more hardware at a problem than tossing programmers and Erlang seems to be a good choice.

Notes:

Here is a list of things that are left:

• Mathematics interop in C/C++
• Checking the numbers with Hipe VM
• Checking the numbers on multiple operating systems
• Hopefully try it out on the cloud
• Persistent store (database integration)

Here is the current setup:

• Windows 7
• Intel i7 CPU (8 hardware threads)
• 6 GB RAM (although we were never limited by the RAM)
• Erlang R14B 5.8.11

Why Erlang?

Games today need to scale smoothly. The advent of multiple cores on each processor has resulted in an almost a feverish rush toward the parallel processing paradigm. However anyone who has written large chunks of code will appreciate the problems that would naturally arise from using threads. Sprinkling a lot of locks around the threads hardly improves performance, the very reason we add threads. At the same time failure to keep synchronisation and race conditions in mind will result in a lot of late nights debugging the code. I faced this exact problem a few years ago and out of purely academic interest I started searching for languages that could make things better (for me atleast). This was when I discovered into Erlang.

Erlang uses message passing to communicate between processes. While this is not a new thing, having it as a construct within the langauge ensures that code written in Erlang is inherently concurrent. Since messages passed between processes give up ownership of data, it prevents locking and thus makes IPC faster. At the same time processes in Erlang are faster than native OS threads (hence even OS processes).

The biggest advantage, in my opinion, is loading and replacing code at runtime. Service Oriented Architecture is one area that will greatly benefit from this. It means that the downtime is extremely minimal. Who enjoys disturbing a gamer while he is playing only for “scheduled downtime”? It also encourages prototyping and rolling out faster updates. This is of great importance especially when gameplay balancing comes into picture.

Being a functional language code written in Erlang is generally shorter than equivalent code written in C/C++ or Java. This results in easier verification of the correctness of the code and fewer bugs creep up because of this. Code reviews are also faster to perform.

What are we up to?

As part of our casual game development division we’ve been at work trying to find various architectures and schemes that make scalability as smooth as possible. Some solutions we have implemented are based on the cloud while others like the Erlang initiative are projects that take a different approach for solving the problems.

We currently are working on a nice little multiplayer game that has a flash frontend and runs an Erlang server in the backend. Erlang is good with IPC hence we spawn a new process for each of these dungeons and for each entity in the game. Purely client side effects are cosmetic effects while physics, entity spawn and destruction and scores are simulated on the server. Since Erlang messages are asynchronous we have to perform some double checks, especially during scoring. Doing this was extremely easy since we decided to only trust the server and force the client to reflect these changes. Essentially the client presents a view of the world for the player. Since Erlang is good with concurrent code, we’ve had a chance to simulate cool particles on the server side. However we shifted all of this to the client side since particles are only cosmetic we decided to move this over to the client. It also resulted in saved network bandwidth.

Work in progress:

We are working on coming up with metrics that will compare existing frameworks with what we are implementing, for the same set of features. This will give us a better idea as to where we are heading and where we need to be in the coming few weeks.

A test automation that will reduce the workload of testers is definitely something we are desperate for. The usual click based testing frameworks help, but I am hoping that my team here will come up with something that is far superior to anything I’ve seen till date. Again, I would be interested in seeing if we can develop a load/stress testing suite that can use Erlang underneath.

Integrating 3rd party APIs, especially some of the payment gateways. This will be a huge leap ahead for us, especially since micro transactions are a popular way to monetise in casual games.

Metrics to check the load bearing characteristics of the servers.

Metrics for the kind of bugs we reported on the servers.

Challenges:

The biggest challenge we face is with respect to the number of people who are aware of Erlang as a programming language. This has resulted in some training time for the members of the project.

The next challenge we have is integration with 3rd party libraries. While Erlang has bridges for running C code (even Python ) these need to be extensively tested before we can start using them.

Integer performance seems to be better in C/C++. I am guessing that this would be because the CPUs nowadays are really good with predictable data. I am not sure how we can tackle this in Erlang.

Having worked in Erlang, I can say with a certain amount of confidence that it has definitely proven itself on the scalability end. We need to see what are the limits to it’s execution and this is where Sierra Atlantic’s expertise in testing and test automation will help me in the coming few weeks.

I hope to post more cool stuff in the coming weeks. Till then, tah.

Around this time, GPGPU (General Purpose GPU)  initiatives by the GPU manufacturers saw GPUs being used for not only rendering but also general purpose computing. One drawback was that GPUs still liked textures and matrices and disliked strings.  However with the introduction of CUDA by nVIDIA, OpenCL by Khronos group and CAL/CTM from AMD/ATi GPUs changed the terrain in favour of programmers. To explain the difference between a CPU and GPU’s processing capabilities, I’ll use the following video: http://www.youtube.com/watch?v=fKK933KK6Gg . The Mythbusters showed exactly what it is that the GPUs are good at – They are good at doing a designated task incredibly fast only when it is performed in parallel. Using that same video as an example, the GPU would have a hard time drawing the smiley, one dot at a time. However if you were to build a new parallel gun that has the smiley imprinted on it, the drawing happens in an instant.

If you were to harness the power of the GPU, you would need to write a program in a way that it can execute instructions in large chunks. The size of these chunks depends on the GPU (or the CPU). The operating model for these programs is based on the SIMD model – Single Instruction Multiple Data. What this means is that you have a single instruction being executed simultaneously across multiple chunks of data. An example for where SIMD would be a good fit is calculating the magnitude of a vector. magnitude_square = x*x + y*y + z*z.  The three multiplies could be performed in a single go.

Common sense would tell you to take your message where the majority of the people are gathering and paying attention, which just happens to be Web 2.0(Shift in consumer habits: From expert driven content to User driven content) social mediums. The data is growing every single day. The number of connection points to the DATA is becoming increasingly available. Realizing what information is available about the business relationships becomes more critical as dollar value increases. Every dollar and every minute spend on marketing tactics is for one purpose: Business Growth!  Let me share this interesting quote from Tony Hsieh, Zappos CEO – “People relate to people, not companies”, Social media provides us an opportunity to humanize the brand and engage the companies with consumers at personal level, which can provide substantial results in long-term like decreased customer support costs, customer feedback management, good PR and marketing channel to engage with target market.  Social Media is for every company that genuinely wants to engage with their customers, it doesn’t matter whether you are fortune brand or Local restaurant. As Sandy Carter, Marketing Head at IBM quoted, “I think the primary difference here is not in big or small company. But the difference is in how close you are to your customers”. I personally find that it is very interesting and exciting “Net Space” to be in and also rewarding if used in the right manner.

The first Video Game I ever saw and played, which I still remember to this day, was the Freeway on Atari. The first computer game I ever saw and played was Paratrooper. They were so much fun, no matter how blobby the graphics look and monotonic the sound effects sound now. Pacman infact still has cult following.

Video Games, these wonderful illusions provide an audio visual experience, tricking one into thinking they are inside a fantastical world, moving, jumping, racing and doing what all not when actually staring at a physically stationary screen. Fascinated with this thought, one of the first “gamey” things I tried to program in Pascal was a “walk through a tunnel” simulation, where you would walk through a tunnel and use the arrow keys to, accelerate, decelerate, turn left and right.

Games continue to evolve, rapidly, their hunger for hardware pushing growth in that segment as well. Let me try and take you through this evolution as I have witnessed.

Shooting Games…

I’ve been a big fan of shooting games, so I’ll start with this genre. Wolfenstien 3d, good old days those, the game would fit on a 3.5 inch floppy. This game was a trend setter and pretty much gave birth to first person shooting games. Shooting down Nazis and their rogue dogs, changing weapons, seeing the protagonist’s face change with his health was a lot of fun, a new experience. The PC was now being looked at as an entertainment machine as well.

The popularity of the first person shooters saw a lot of hugely popular titles like the Doom series, Duke Nukem series, Redneck Rampage, Blood and the revolutionary Quake Series. Quake hit it big time with its online game play over the internet.

On consoles the Halo series has created much more than a stir, contributing significantly to XBOX sales. Half Life added the sci-fi angle to first person shooters. Unreal is another extremely popular example.

Lara Croft, the protagonist of Tomb Raider came in and changed this genre by adding a new dimension to it. This was the beginning of Third Person Shooters. With the protagonist visible on the screen, albeit from behind or from over the shoulder, the protagonist suddenly started getting a lot more attention, literally. MDK became popular with it’s really intelligent enemies and the sniper mode. Max Payne with his story of vendetta being brought out in a comic book like story line, killing the bad guys is one of the most popular Third Person Shooting games.

And can I go ahead without mentioning Contra, the Nintendo console game when talking about shooting games? I still remember the button combination that allowed 30 lives.

Combat Games…

“Finish Him”. If you know that phrase, there’s a good chance you have Carpal Tunnel Syndrome setting in your wrists. Yes the fatalities did take a toll on those overworked nerves and tendons along with the Mortal Combat characters.

Street Fighter II was another immensely popular arcade combat game, where you had to perfect those hard to remember and time special moves. And then you had the Capcom Vs. Namco games as well.

Racing Games…

One of the most popular ones, this genre appeals to varied demographic groups.

The relatively easy learning curve attracts a lot of new/non gamers.

I fell in love with racing games on the computer when I played Test Drive eons ago.

I was thrilled with the concept of being able to upgrade my car with the race money I won. Bikes, cars, I could race them all. Pole Position was also a popular game.

With new versions, one could see increasing amounts of realism being thrown into the game. You could honk, switch on your lights, and operate your vipers to increase visibily if it were raining. The asphalt on the game road started looking truer with better textures being used. The trees you whizzed past also assumed a more life like form as compared to mere triangles that would barely catch the corner of your eye earlier.

One could tweak various parameters in the car to suit personal preferences or the track/weather requirements. One could tune the suspension to suit the track, change the tyres from slicks to treads to suit the weather conditions. The game could be made more challenging by switching from automatic to manual transmission, the features were increasing at a break-neck speed. The physics in these games kept getting better. The damage sustained by your vehicle now also started to impact you vehicle’s performance if you so wanted.

“I feel the need, the Need for Speed”, the iconic dialog from the blockbuster movie Top Gun possibly gave birth to the title of one of the most popular racing games ever, the Need for Speed series.

Some racing games like Carmageddon and Road rash started endorsing violence, hung around for a while and yes, gained popularity.

Adventure Games…

Breathtaking backgrounds, an interactive story with an element of mystery and solving a lot of puzzles, now we’re talking Adventure Games.

The Legend of Kyrandia, that’s the first name that comes to my mind in this genre. Of course there were adventure video games (starting with text based adventure games) before that, but this was the first one I ever played.

Leisure Suite Larry, ok, let me just skip this one.

Games like Space Quest, the Monkey Island series also became popular. Myst took this genre world by storm.

The number of people following such games is dwindling and this is attributed to the other genres gaining popularity because they are not as “slow”.

RPG s…

Now this is one genre that has everybody talking. And by everybody I mean millions of hard core gamers, people who take games very seriously spend a lot of money and time on them.

The origin of this genre dates back to the invention of games like Chainmail and Dungeons & Dragons. These games were played by a group of people sitting across a table, using dice and pen and paper (we’re talking pre compter era days here ), assuming the roles of characters of fantasy, leading armies to conquer kingdoms, set out on adventues, honing their skill points with each adventure, becoming stronger.

The popularity of these games gave way to computer RPGs (also referred to as CRPGs).

CRPG’s like The Final Fantasy series, Elder Scrolls, The Legends of Zelda and Baldur’s Gates are some of the most followed ones in this genre.

Strategy Games…

The two genres of RPGs and Strategy Games often overlap, the line between the two being very thin if not non-existant. Games like World of Warcraft, Civilization, Heroes of Might and Magic, Commandos, Command and Conquer, Sim City, Dune, Age of Empires, Starcraft could easily fit into either of the two genres, because of the combination of strong Role Play and Strategy. Turn Based and Simultaneous Strategy games are both popular.

Casual Games…

I have seen more than one serious gamer calling the term “Casual Games” an oxymoron with a lot many more unmentionable adjectives thrown in. Come to think of it, the term serious gamer is also an oxymoron! Anyway, hard core gamers scoff at this genre, hard core game programmers probably want to stay away from programming these, but does that take away anything from the humungous fan base that these games have managed to build? I am sure all your facebook friends can be categorized into two buckets,

1.) Who play (or have played) these games a lot. Yes I’m talking about your friends whose crops have withered, or the Russian job that has to be carried out

2.) Who totally detest these games

Love them or hate them, but you can’t ignore them. They have a user base spanning all demographics, from kids to grannies. These games have spawned a whole new parallel economy of virtual goods and the numbers, both monetary and in terms of user base are gigantic in proportion.

Where are we headed?

Better graphics? More intriguing game play? MMO FPS games?

I feel it is the controllers that are going through a phase of revolution right now and the trend will continue for quite a while. The Nintendo Wii with its moderate processing hardware and just about ok graphics became immensely popular with its accelerometer and gyroscope driven Wii mote. You feel you’re in the game.

I’m also sure everybody is looking forward to this holiday season with the Microsoft’s Project Natal aka Kinect. If the demos and blogs are anything to go by, you might never see a controller again. The camera which is smart enough to recognize 31 body parts, 6 different players, with gesture and facial expression recognition is being touted as the future.

So what next? Gaming Hardware that detects eyeball movement? Maybe a game that reads your mind and doesn’t require even a camera.

Game for a brain implant for a game?

Computer games take from one to three years to develop (depending on scale). Testing begins late in the development process, sometimes from halfway to 75% into development (it starts so late because, until then, there is little to nothing to play or test). Testers get new builds from the developers on a schedule (daily/weekly) and each version must be uniquely identified in order to map errors to versions.

Once the testers get a version, they begin playing the game. Testers must carefully note any errors they uncover. These may range from bugs to art glitches to logic errors and level bugs. Some bugs are easy to document but many are hard to describe and may take several paragraphs to describe so a developer can replicate or find the bug. On a large-scale game with numerous testers, a tester must first determine whether the bug has already been reported before they can log the bugs themselves. Once a bug has been reported as fixed, the tester has to go back and verify that the fix works – and occasionally return to verify that is has not reappeared.

Evaluation of game rules: Game rules adequately explain operation of all components of the game including features, free games etc. Consistency with game rules and pay tables

UI and Functional test: Games outcome and data are correctly shown when games are played. Intermediate and final game outcomes are displayed for a reasonable time.

Verification the Integration points: Game win determination aligns with game rules.

Reviewing gaming procedures: procedures will be reviewed by System management, player account management, tournaments and promotions.

Infrastructure and security review: Secure and reliable operation for example time synchronization and OS reliability and security

You can follow few tips

• Need to identify the algorithm from Source code to identify issues
• RNG seeding checked in source code.
• Need to verify the source code appropriate use of random numbers, error handling, no unused variable etc…
• Need to verify the RNG mapping to make sure the output of game symbols, cards & other outcomes
• Need to test validate and evaluate the game predefined rules and its consistency
• Need to Make sure the indecent or offensive content/material display
• Irrespective of Single/Multi player games we need to validate bandwidth and client software.
• Reports need to validated with data accuracy Different date and time format, larger win, total money played, total no. of games etc.
• Infrastructure plays a vital role need to verify the network architecture and device configuration, OS reliability & security. Need to verify the gaming procedures with change control mechanism, game management and player management.

Keep all these tips and regardless of background experience in game programming and testing, anyone can become a game tester successfully.

Here’s my take on this:

- Perfrect for networking and online games. Think about the architectural possibilities now that all the client side and server side information is on the server – the cloud. Of course the client side now becomes a video decoder.

- 5 Mbps for an HDTV output is asking for a little too much. It is not always possible for everyone to have this sort of a connection consistently. I know for sure that South Korea, Japan and Singapore can march ahead in this area, especially in the wireless devices space.

- We will slowly be able to move away from a GPU+CPU architecture to one that is purely CPU (hmm or may just be GPU) architecture. Aren’t we tending towards software renderers here? Ray tracing, although years away, should become a natural progression.

- A more consistent “Minimum Requirements” spec for playing games.

- We definitely need to consider the possibility that having a vector graphics processor will lower bandwidth requirements.

- Fatter bills for data transfer

I know for sure that this will not make consoles disappear but it will make PC gaming and mobile gaming obtain better market penetration. A few years ago Steam changed the way I played games. I hope a service like Gaikai or Onlive will also change the landscape.

Better tools make people more efficient. That way they can concentrate on making a better quality game instead of fighting to meet the deadlines.  To make this work one needs to slowly chip away at the hurdles that slow the pace of development.

Programmers already have a wide array of tools at their disposal and the tools are getting better. Compilers are getting better so are machines and the processors. Profilers significantly improved over the generations for both consoles and for the PC. Multi threading however is an area that needs significant bump in the tools.  I also think that a different language that supports threading as a core paradigm is better fit than C++. Of course thread profilers and threading libraries (Boost, TBB from Intel) are of great help, especially when combined with some of the design patterns that play well with threading (ex: Producer consumer, command pattern).

Artists, who compose at least half the development team, definitely need better tools. We need tools that revolutionise the way art is generated. The areas where I see significant improvements in the tools are:

1)      Procedural content generation – Great if we want to reduce the time to generate the content, reduce the size of the game (think downloadable content). The 64K competition is definitely a trailblazer in this segment

2)      Gameplay logic – Wouldn’t it be better if the gameplay designer could just create his own gameplay in the game inside of trying to convince a programmer and the team? I am talking about Kismet (Unreal Engine 3)

3)      Lip sync – It is time that we don’t have lip flapping and see good lip syncing especially for non English languages

4)      Automation tools – Tools that will give the artists an early red flag when content they add to a level causes a drop in the frame rate or pushes the memory limits over the boundaries

It is important to understand that a majority of the tools, including the level editor and modeling software are useful for both programmers and artists. Improving these will definitely make things better for artists and for programmers.

I can dream of a day when character rigging happens procedurally but for the next five years, I hope to see and build a lot more of these tools. Tools for creating content have been improving ever since humans desired to create art. From the cave paintings and the crop circles all the way to today’s video games and movies, they show an improvement in the kind of content we create. It thus makes me believe that improving the tool chain opens up new avenues for creating art.