Subsequent to my earlier blog post on Erlang (refer: http://blogs.sierraatlantic.com/2010/10/erlang-and-experiments-with-scalable-servers/) we whipped up a quick server to test its scalability. Srini (a fresh graduate from University of Hyd) and Vijayender worked on the server side of the game while Chaitanya (not me) worked on writing a client using the Blender API.

It was in many ways an experiment on the productivity of the programmers. Vijayender, with his vast experience, was mostly involved in design while Srini and Chaitanya were responsible for actual implementation.  Keep in mind that Chaitanya and Srini have each just about 2 months of experience and I was already demanding them to write multi threaded code. However the results definitely surprised me.

We planned on making a very simple server that would cater to multiple incoming connections. The overall idea is that we would use one thread to run the networking related code, one thread to do the event management and one thread to perform collisions and the like. The reason for this was straightforward since this is a very simple form of threading (using the producer consumer pattern).

So how long did it take for us the finish this application? 2 months. Yes, that’s right. Just two months with a brand new pair of hands who have no experience writing multi threaded code.  The number of bugs that were reported? Less than 5 and all of them were deviations from the specification. None were technology related bugs.

The following graph depicts the throughput. X-Axis shows the number of clients while the Y-Axis shows the average time it took for a single update (in milliseconds)

The point to notice here is not how long each update took to perform. It is how the server scales automatically when we throw more cores at it. The performance is bound to improve if we switch all of the collision and mathematical calculations to a C/C++ interop (maybe even CUDA/OpenCL… hmmm).

Notice how the 800 clients an 8 core processor gives more than 30 fps consistent. Every time we figured an increased load on the server, all we had to do was toss more cores at it and the performance pretty much improved linearly.

Having worked extensively on multi-threaded code I have come to believe that the future lies in languages that are designed with threading in mind. Multi-core architectures are not going anytime soon and this is exactly why writing code that scales well is of extreme importance. I believe that it is better to toss more hardware at a problem than tossing programmers and Erlang seems to be a good choice.

Notes:

Here is a list of things that are left:

• Mathematics interop in C/C++
• Checking the numbers with Hipe VM
• Checking the numbers on multiple operating systems
• Hopefully try it out on the cloud
• Persistent store (database integration)

Here is the current setup:

• Windows 7
• Intel i7 CPU (8 hardware threads)
• 6 GB RAM (although we were never limited by the RAM)
• Erlang R14B 5.8.11

Nowadays, several hosting companies are assuring to provide high-end infrastructure and application services over the cloud computing environment.  This environment contains number of virtual machines running on one or more physical systems.  With the introduction of cloud computing, revolutionary changes have been taken place in hardware and in data center models with exceptional online storage and backup systems.   Security, reliability and availability are major challenges for cloud hosting providers and they are effectively monitored in accordance to the SLAs defined with the users.  Cloud environment is denoted as public or private based on the availability and accessibility of the environment at specific premises with necessary privileges.

Fig: 1   Imaginary picture of private and public cloud environments

Public cloud

In general, everyone can access public cloud environment with appropriate subscription from service providers.  In public cloud computing environment, users never know hardware details of the server on which their application is hosted; because server scales up to tens of thousands of computers automatically on need basis.  Infrastructure as a service (IaaS) is industry vernacular for the public cloud computing services as it offers access to virtual servers, desktop computers or remote storage locations on usage basis.

Private Cloud

Private cloud is similar to public cloud environment, but access is restricted to limited number of users within the company firewalls.   Private cloud computing can be tightly integrated with public cloud computing environments which are running on high bandwidth of 24 X 7 availability.   There are several wrong perceptions about private cloud computing with apprehensive calculations of hardware and software costs.  In reality, it can be set-up in much cheaper way and wouldn’t consume much time to install the software.   At minimum, private cloud environment can be set-off with $4000 approximately.

Cloud Service Providers

Microsoft, Amazon, Google, Sales force…etc are the leading public cloud service providers in today’s market. In general, all of them would provide more or less same type of services with their innovative business ideas.

Cloud Users

Cloud services are mainly beneficial for the technical people or hosting companies who wish to use cloud as future platform for their business.  It’s less significant for the end users who browse the application from the cloud environment.

If you’ve a question like, who are the users for the cloud environment?

It depends on the type of service subscribed from the cloud service provider.  For example, a development team can create a web application that will be hosted in cloud environment.  Similarly tester can validate same web application on cloud staging environment.  A hosting company might want to host their web sites on public cloud environment to reduce costs on hardware.  A gaming company could deploy online games in cloud environment to make use of hardware acceleration and performance.  All these users must subscribe appropriate services from the public cloud environment before start using it.

Web Application Development and Deployment on Windows Azure

You can look at the media files which demonstrate the subscription, development and deployment process of the public cloud computing environment on Windows Azure.  I’ve built these movies based on the sample scenario that was explained as objective in 1^st video.   Essentially these movies would provide answers to the following 3 questions…

In Microsoft’s perspective, cloud computing affects 3 important areas / models…

• Data Center / Hardware model;
• Application model;
• Operation model.

Data Center / Hardware Models: Unlike traditional monolithic data centers, pre-assembled/modular data centers are emerging in today’s market; especially in the cloud computing environment and hardware has enormous capacity to produce extremely high performance and backup at nominal cost.  IaaS (Infrastructure as a Service) provides a platform for virtualization where you can subscribe virtual servers, desktops and other infrastructure related services on usage basis.

Application Model:  In cloud environment, we can easily and quickly deliver applications that are targeted for international audience from any device with rich end-user experience.   A simple browser with internet connectivity would be sufficient to deal with business critical applications.  SaaS (Software as a Service) offers solutions to the wide variety of applications on consumption wise.

Operation Model:  Services from the cloud environment would be available on 24X7 basis.  You’ll be covered under pre-defined or customized SLAs when you subscribe services from the public cloud environment; it provides incredible online storage with strong back-up and recovery mechanism.

We have to change the competitive landscape by being not just better than our competitors but by taking quality to a whole new level.  We have to make the quality so special, valuable and important to the success of our clients that we should become their real value choice. Clients are changing the Wexelblat’s scheduling Algorithm from “Choose two: Good, Fast, and cheap” To “Choose ALL: Good, Fast, and cheap.”

Billions of dollars in business are lost annually because companies and software vendors fail to adequately test their software systems and products. These kind of business losses are expected to continue as long as testing is considered just another check mark on a To-Do list or a task given to resources who are on the bench. Testing is, in fact a professional role that requires technical skills and mindset that encourages the early discovery of the problems that represent real business risks.

Technology, Methodology, Customer, business expectation, business complexities, globalization all these are more and more business visibility. The final skill on my list is the ability to see the larger picture of a company’s overall business strategy. This enables a great software tester to actively participate at a level higher than just an individual contributor—instead of merely finding and reporting a Priority 2 / Severity 2 bug, a great software tester can identify strategic strengths and weaknesses of a software system that can ultimately lead to a business competitive advantage. I think its time for us to be different not just as efficient!

Here are few tips from my end – What you can do?

# Tip 1:  Start with thanking them when they help the developer to fix the critical anomalies. It sounds impossibly obvious, but it’s surprising how many QA managers neglect this simple action.

# Tip 2: Bring innovative problem-solving and team building techniques to help people become more create and productive

# Tip 3: Test lead and QA manager need to communicate and give detailed instructions and provide specific instructions to test team to work better.

#Tip 4: Test lead should avoid asking people to do the same boring tasks all the time. A stimulating environment creates enthusiasm and the opportunity for “big picture” thinking.

#Tip 5: Arrange some testing events, quiz for team. Start appreciating the team for their achievements.

The main objective is here to share the practical experiences gained while working for different teams and explains why motivation is required? Where motivation plays a key role? And how motivation makes a team successful? So please share/post your tips with us.

Methodologies followed to perform appliance testing vary according to the class of the appliance. Broadly, these follow the following pattern:

• Gather Baseline numbers – this can be picked off a manual or rule book if the test is an adherence to a standard kind of test. These numbers can be arrived at using statistical evaluations based on a large set of data from representative usage runs. Rule of thumb can also be a driver in arriving at the baseline data. When the testing is done to calculate the effectiveness of an appliance, a load test is done without the appliance and values of identified parameters are recorded as baseline.
• Identify usage pattern and environment
• Design lab and application flows on pattern, environment identified above
• Subject the appliance to ‘load’ and measure parameters
• Compare / Publish / Certify results

Quick Tip

• Have a panel of experts certify the important choices made: The parameters selected to evaluate performance, the flow identified to exercise load on the appliance, the ‘load’ scenario that the appliance will be subjected to and the lab setup. You are sure to have a few plus or minus. Having only the in-house expert certify these is bound to leave you open to ‘bias’ claims

Why Erlang?

Games today need to scale smoothly. The advent of multiple cores on each processor has resulted in an almost a feverish rush toward the parallel processing paradigm. However anyone who has written large chunks of code will appreciate the problems that would naturally arise from using threads. Sprinkling a lot of locks around the threads hardly improves performance, the very reason we add threads. At the same time failure to keep synchronisation and race conditions in mind will result in a lot of late nights debugging the code. I faced this exact problem a few years ago and out of purely academic interest I started searching for languages that could make things better (for me atleast). This was when I discovered into Erlang.

Erlang uses message passing to communicate between processes. While this is not a new thing, having it as a construct within the langauge ensures that code written in Erlang is inherently concurrent. Since messages passed between processes give up ownership of data, it prevents locking and thus makes IPC faster. At the same time processes in Erlang are faster than native OS threads (hence even OS processes).

The biggest advantage, in my opinion, is loading and replacing code at runtime. Service Oriented Architecture is one area that will greatly benefit from this. It means that the downtime is extremely minimal. Who enjoys disturbing a gamer while he is playing only for “scheduled downtime”? It also encourages prototyping and rolling out faster updates. This is of great importance especially when gameplay balancing comes into picture.

Being a functional language code written in Erlang is generally shorter than equivalent code written in C/C++ or Java. This results in easier verification of the correctness of the code and fewer bugs creep up because of this. Code reviews are also faster to perform.

What are we up to?

As part of our casual game development division we’ve been at work trying to find various architectures and schemes that make scalability as smooth as possible. Some solutions we have implemented are based on the cloud while others like the Erlang initiative are projects that take a different approach for solving the problems.

We currently are working on a nice little multiplayer game that has a flash frontend and runs an Erlang server in the backend. Erlang is good with IPC hence we spawn a new process for each of these dungeons and for each entity in the game. Purely client side effects are cosmetic effects while physics, entity spawn and destruction and scores are simulated on the server. Since Erlang messages are asynchronous we have to perform some double checks, especially during scoring. Doing this was extremely easy since we decided to only trust the server and force the client to reflect these changes. Essentially the client presents a view of the world for the player. Since Erlang is good with concurrent code, we’ve had a chance to simulate cool particles on the server side. However we shifted all of this to the client side since particles are only cosmetic we decided to move this over to the client. It also resulted in saved network bandwidth.

Work in progress:

We are working on coming up with metrics that will compare existing frameworks with what we are implementing, for the same set of features. This will give us a better idea as to where we are heading and where we need to be in the coming few weeks.

A test automation that will reduce the workload of testers is definitely something we are desperate for. The usual click based testing frameworks help, but I am hoping that my team here will come up with something that is far superior to anything I’ve seen till date. Again, I would be interested in seeing if we can develop a load/stress testing suite that can use Erlang underneath.

Integrating 3rd party APIs, especially some of the payment gateways. This will be a huge leap ahead for us, especially since micro transactions are a popular way to monetise in casual games.

Metrics to check the load bearing characteristics of the servers.

Metrics for the kind of bugs we reported on the servers.

Challenges:

The biggest challenge we face is with respect to the number of people who are aware of Erlang as a programming language. This has resulted in some training time for the members of the project.

The next challenge we have is integration with 3rd party libraries. While Erlang has bridges for running C code (even Python ) these need to be extensively tested before we can start using them.

Integer performance seems to be better in C/C++. I am guessing that this would be because the CPUs nowadays are really good with predictable data. I am not sure how we can tackle this in Erlang.

Having worked in Erlang, I can say with a certain amount of confidence that it has definitely proven itself on the scalability end. We need to see what are the limits to it’s execution and this is where Sierra Atlantic’s expertise in testing and test automation will help me in the coming few weeks.

I hope to post more cool stuff in the coming weeks. Till then, tah.

Multi-skilling is currently the big buzzword. The common belief is that a multi-skilled workforce can avoid retrenchment and can lead to increased productivity for the organization in a time of increased severity, which can directly impact the company’s bottom line. In leaner times, a multi-skilled workforce helps bring in operational efficiency and increased productivity, since a smaller workforce is required to cater to the scaled down demand. But even when economic conditions are on the upswing, a multi-skilled workforce helps in addressing customer demand faster and better,”

IT sector might be on the down slope due to the global financial crisis and recession in the world economy but the “Software Testing Industry” is growing rapidly.  The incessant pressure on increasing the “Performance” of various software is ensuring that the Software Testing market is getting bigger. The business potential in Software testing is also at an all time high and continues to grow, considering the growing complexity of software products and increasing expectations from the end-users demanding the best quality product. The size of the Software Testing industry globally is $13billion. At present to ensure the competitive advantage, testing phase which is typically 30% of total software development is critical to ensure on time release within budget and with the agreed level of confidence.  And more over 300 automated test tools in the market, a new testing role emerged that is responsible for identifying the right tool for the right testing. These growth rates are forcing Software test engineers to be well equipped to meet the day to day challenges.

Some testing tasks may require specific skills within the technical or subject-matter area. For example, a tester who is experienced, or at least familiar, with security testing techniques should be responsible for security testing. A tester who is not skilled in this area can only guess what makes an application secured. Similarly, in the case of localization testing, an English speaker can only guess regarding the correct translation of an application into another language. A more effective localization tester would be a native speaker of the language into which the application has been translated.

Individual skill and knowledge contributes in business failure and success. The individual skill and knowledge contributes in business failure and success. if a resource enhances his skill sets, albeit in a phased manner, he will be far more valuable to his company and the company will be a more potent force in the market. Not only this is relevant in a recession, but also in a booming economy where companies compete to attract and retain the best business. “Forced Multi-Skilling” by companies might lead to the employee losing interest in job and finally walking out. This apart, employees’ work-life balance may be negatively impacted due to time constraints. “Multi-skilling needs to be nurtured and ‘sold’ to employees. It should not be thrust randomly

Around this time, GPGPU (General Purpose GPU)  initiatives by the GPU manufacturers saw GPUs being used for not only rendering but also general purpose computing. One drawback was that GPUs still liked textures and matrices and disliked strings.  However with the introduction of CUDA by nVIDIA, OpenCL by Khronos group and CAL/CTM from AMD/ATi GPUs changed the terrain in favour of programmers. To explain the difference between a CPU and GPU’s processing capabilities, I’ll use the following video: http://www.youtube.com/watch?v=fKK933KK6Gg . The Mythbusters showed exactly what it is that the GPUs are good at – They are good at doing a designated task incredibly fast only when it is performed in parallel. Using that same video as an example, the GPU would have a hard time drawing the smiley, one dot at a time. However if you were to build a new parallel gun that has the smiley imprinted on it, the drawing happens in an instant.

If you were to harness the power of the GPU, you would need to write a program in a way that it can execute instructions in large chunks. The size of these chunks depends on the GPU (or the CPU). The operating model for these programs is based on the SIMD model – Single Instruction Multiple Data. What this means is that you have a single instruction being executed simultaneously across multiple chunks of data. An example for where SIMD would be a good fit is calculating the magnitude of a vector. magnitude_square = x*x + y*y + z*z.  The three multiplies could be performed in a single go.