Appliance performance is performed for various reasons. There are laws that govern domestic appliances and their performance. Appliance testing is done to confirm adherence to these laws. Along similar lines are ratings tests. These are pseudo legal requirements in that the adherence to standards is strict but the objective is to use the rating to gain market acceptance. Appliance testing is also done to ensure that the appliance is accepted in the market. Appliance testing is also done to evaluate the effectiveness of the appliance. This entails calculating the improvement (or otherwise) factor that can be expected when the appliance is deployed in a typical environment as against when the appliance is not part of the environment. In all these cases, the results of the test are used to alter the appliance or reconfigure it.

Methodologies followed to perform appliance testing vary according to the class of the appliance. Broadly, these follow the following pattern:

• Gather Baseline numbers – this can be picked off a manual or rule book if the test is an adherence to a standard kind of test. These numbers can be arrived at using statistical evaluations based on a large set of data from representative usage runs. Rule of thumb can also be a driver in arriving at the baseline data. When the testing is done to calculate the effectiveness of an appliance, a load test is done without the appliance and values of identified parameters are recorded as baseline.
• Identify usage pattern and environment
• Design lab and application flows on pattern, environment identified above
• Subject the appliance to ‘load’ and measure parameters
• Compare / Publish / Certify results

Quick Tip

• Have a panel of experts certify the important choices made: The parameters selected to evaluate performance, the flow identified to exercise load on the appliance, the ‘load’ scenario that the appliance will be subjected to and the lab setup. You are sure to have a few plus or minus. Having only the in-house expert certify these is bound to leave you open to ‘bias’ claims

A little introduction to Erlang. Erlang is a language that was designed around concurrency as a core feature. It is a functional language and supports threading as a first class primitive. This is one of the features that set it apart from most other languages, primarily C/C++ and Java. These languages rely on libraries to provide concurrency primities (threads, processes, locks and syncs). Since Erlang had this designed into the language, threading and associated issues have been reduced to  minimum.

Why Erlang?

Games today need to scale smoothly. The advent of multiple cores on each processor has resulted in an almost a feverish rush toward the parallel processing paradigm. However anyone who has written large chunks of code will appreciate the problems that would naturally arise from using threads. Sprinkling a lot of locks around the threads hardly improves performance, the very reason we add threads. At the same time failure to keep synchronisation and race conditions in mind will result in a lot of late nights debugging the code. I faced this exact problem a few years ago and out of purely academic interest I started searching for languages that could make things better (for me atleast). This was when I discovered into Erlang.

Erlang uses message passing to communicate between processes. While this is not a new thing, having it as a construct within the langauge ensures that code written in Erlang is inherently concurrent. Since messages passed between processes give up ownership of data, it prevents locking and thus makes IPC faster. At the same time processes in Erlang are faster than native OS threads (hence even OS processes).

The biggest advantage, in my opinion, is loading and replacing code at runtime. Service Oriented Architecture is one area that will greatly benefit from this. It means that the downtime is extremely minimal. Who enjoys disturbing a gamer while he is playing only for “scheduled downtime”? It also encourages prototyping and rolling out faster updates. This is of great importance especially when gameplay balancing comes into picture.

Being a functional language code written in Erlang is generally shorter than equivalent code written in C/C++ or Java. This results in easier verification of the correctness of the code and fewer bugs creep up because of this. Code reviews are also faster to perform.

What are we up to?

As part of our casual game development division we’ve been at work trying to find various architectures and schemes that make scalability as smooth as possible. Some solutions we have implemented are based on the cloud while others like the Erlang initiative are projects that take a different approach for solving the problems.

We currently are working on a nice little multiplayer game that has a flash frontend and runs an Erlang server in the backend. Erlang is good with IPC hence we spawn a new process for each of these dungeons and for each entity in the game. Purely client side effects are cosmetic effects while physics, entity spawn and destruction and scores are simulated on the server. Since Erlang messages are asynchronous we have to perform some double checks, especially during scoring. Doing this was extremely easy since we decided to only trust the server and force the client to reflect these changes. Essentially the client presents a view of the world for the player. Since Erlang is good with concurrent code, we’ve had a chance to simulate cool particles on the server side. However we shifted all of this to the client side since particles are only cosmetic we decided to move this over to the client. It also resulted in saved network bandwidth.

Work in progress:

We are working on coming up with metrics that will compare existing frameworks with what we are implementing, for the same set of features. This will give us a better idea as to where we are heading and where we need to be in the coming few weeks.

A test automation that will reduce the workload of testers is definitely something we are desperate for. The usual click based testing frameworks help, but I am hoping that my team here will come up with something that is far superior to anything I’ve seen till date. Again, I would be interested in seeing if we can develop a load/stress testing suite that can use Erlang underneath.

Integrating 3rd party APIs, especially some of the payment gateways. This will be a huge leap ahead for us, especially since micro transactions are a popular way to monetise in casual games.

Metrics to check the load bearing characteristics of the servers.

Metrics for the kind of bugs we reported on the servers.

Challenges:

The biggest challenge we face is with respect to the number of people who are aware of Erlang as a programming language. This has resulted in some training time for the members of the project.

The next challenge we have is integration with 3rd party libraries. While Erlang has bridges for running C code (even Python ) these need to be extensively tested before we can start using them.

Integer performance seems to be better in C/C++. I am guessing that this would be because the CPUs nowadays are really good with predictable data. I am not sure how we can tackle this in Erlang.

Having worked in Erlang, I can say with a certain amount of confidence that it has definitely proven itself on the scalability end. We need to see what are the limits to it’s execution and this is where Sierra Atlantic’s expertise in testing and test automation will help me in the coming few weeks.

I hope to post more cool stuff in the coming weeks. Till then, tah.

Sipping a cup of hot coffee and re looking at the good old days, Science and Technology has indeed changed and evolved for good. There were many new innovations and the new decade started.  Our new generation is living in an era of virtual information explosion. There is unimaginable amount of content that is available on any topic just at a stroke of a keyboard or click of a mouse. Thinking about my profession, I wonder whether my profession, Software Testing, has also completed a decade. Wow! it was a successful year and a boom time for software testing industry and the complexity of software products increased thus rising the end user expectations. It has also increased the pressure on software testers to have multiple skills to gain the customer confidence.

Multi-skilling is currently the big buzzword. The common belief is that a multi-skilled workforce can avoid retrenchment and can lead to increased productivity for the organization in a time of increased severity, which can directly impact the company’s bottom line. In leaner times, a multi-skilled workforce helps bring in operational efficiency and increased productivity, since a smaller workforce is required to cater to the scaled down demand. But even when economic conditions are on the upswing, a multi-skilled workforce helps in addressing customer demand faster and better,”

IT sector might be on the down slope due to the global financial crisis and recession in the world economy but the “Software Testing Industry” is growing rapidly.  The incessant pressure on increasing the “Performance” of various software is ensuring that the Software Testing market is getting bigger. The business potential in Software testing is also at an all time high and continues to grow, considering the growing complexity of software products and increasing expectations from the end-users demanding the best quality product. The size of the Software Testing industry globally is $13billion. At present to ensure the competitive advantage, testing phase which is typically 30% of total software development is critical to ensure on time release within budget and with the agreed level of confidence.  And more over 300 automated test tools in the market, a new testing role emerged that is responsible for identifying the right tool for the right testing. These growth rates are forcing Software test engineers to be well equipped to meet the day to day challenges.

There are specific testing-related measures against which a tester can be evaluated. For example, all testers must be detail oriented and possess analytical skills, independent of whether they are technical testers, subject-matter experts, security testers, performance tester or usability testers. A software tester should requires at least complete knowledge of software testing  tools, logical and analytical ability along with quality approach to the software testing processes. Tester should focus and improve all his skills on” understanding customer requirements and writing appropriately”, “comprehensive test cases”, “performing and executing test” and “preparing a detailed self-explanatory test report”. Testing complex business requirements will require extraordinary level of experience and knowledge to keep the budget and time constraint for performing testing without compromising with the coverage or quality.

Some testing tasks may require specific skills within the technical or subject-matter area. For example, a tester who is experienced, or at least familiar, with security testing techniques should be responsible for security testing. A tester who is not skilled in this area can only guess what makes an application secured. Similarly, in the case of localization testing, an English speaker can only guess regarding the correct translation of an application into another language. A more effective localization tester would be a native speaker of the language into which the application has been translated.

Individual skill and knowledge contributes in business failure and success. The individual skill and knowledge contributes in business failure and success. if a resource enhances his skill sets, albeit in a phased manner, he will be far more valuable to his company and the company will be a more potent force in the market. Not only this is relevant in a recession, but also in a booming economy where companies compete to attract and retain the best business. “Forced Multi-Skilling” by companies might lead to the employee losing interest in job and finally walking out. This apart, employees’ work-life balance may be negatively impacted due to time constraints. “Multi-skilling needs to be nurtured and ‘sold’ to employees. It should not be thrust randomly

A long time ago, in a kingdom far far away HPC was restricted to the likes of Cray. There was also a time when GPUs were merely specialised silicon that could draw pixels a little faster than the CPU could. Around 1998 nVIDIA release the Riva TNT and things began to change, albeit slowly. Then came hardware T&L support from nVIDIA’s Geforce 256. Games did not look the same after that. And time moved on… . GPUs had a fixed function pipeline for most of their existence till programmable shaders entered the picture. This allowed programmers to treat the GPU like another programmable chip. The one thing to notice here is that all these GPUs were specialised processors to make fast games render faster.

Around this time, GPGPU (General Purpose GPU)  initiatives by the GPU manufacturers saw GPUs being used for not only rendering but also general purpose computing. One drawback was that GPUs still liked textures and matrices and disliked strings.  However with the introduction of CUDA by nVIDIA, OpenCL by Khronos group and CAL/CTM from AMD/ATi GPUs changed the terrain in favour of programmers. To explain the difference between a CPU and GPU’s processing capabilities, I’ll use the following video: http://www.youtube.com/watch?v=fKK933KK6Gg . The Mythbusters showed exactly what it is that the GPUs are good at – They are good at doing a designated task incredibly fast only when it is performed in parallel. Using that same video as an example, the GPU would have a hard time drawing the smiley, one dot at a time. However if you were to build a new parallel gun that has the smiley imprinted on it, the drawing happens in an instant.

If you were to harness the power of the GPU, you would need to write a program in a way that it can execute instructions in large chunks. The size of these chunks depends on the GPU (or the CPU). The operating model for these programs is based on the SIMD model – Single Instruction Multiple Data. What this means is that you have a single instruction being executed simultaneously across multiple chunks of data. An example for where SIMD would be a good fit is calculating the magnitude of a vector. magnitude_square = x*x + y*y + z*z.  The three multiplies could be performed in a single go.