2.1 Related Work

In the past, it was already clear the need to simulate the processes of SE, as it is presented by Lin et al. [³]. Their work examines trade-offs of cost, schedule, and functionality, and to test the implications of different managerial policies on a project’s outcome and enable software managers to gain a better understanding of the dynamics of software project development and provides a learning environment through simulation where the implications of different policies on a project can be studied, and insight can be gained into the causes of project dynamics.

In this same direction, Kellner et al. [⁴] focused their work on simulation; it offers a set of simulations that can be selected and serve as guides to practical application, so it uses a set of working hypotheses through models. There are also works that show how to simulate processes in requirements engineering [⁵, ⁶]. They show, as the major goals behind the modeling and simulation effort are to assess the issues associated with the social and behavioral aspects of the EasyWinWin process, and to explore how these issues affect the overall outcome of the process. Pfahl et al. [⁷] present concepts of a computer-based training (CBT) and argue that university education needs to provide to their computer science and SE students not only technology-related skills but also a basic understanding of typical phenomena occurring in industrial (and academic), software projects. Thus, in a recent study, Chen and Chong [⁸] show that some aspects in SE education can be taught through simulations by offering comprehensive training that explores subjects such as collaborative software development, team projects and the social aspects of software development. It allows students to experiment real problems rather than mere academic exercises.

Wangenheim et al. [⁹] identified the potential to teach SCRUM in concrete situations through an educational game and complement the learning process through theoretical lectures. According to authors, SCRUMIA was created based on experience in class and using instructional design. This game has been applied several times in two undergraduate project management courses. They also evaluated the competency, understanding and motivation. SCRUMIA explores a specific situation through a case study.

Other authors such as Qin et al. [¹⁰], Barros and Araújo [¹¹] and Alvarez et al. [¹²] Carried out some experiments in which they taught SE with games in small software projects. Bollin et al. [¹³] highlight the importance of using real (usually large) projects to teach SE in addition to soft skills such as economic planning responsibility, ethic and project evaluation. These issues should be better taught by means of simulations or games, which caught partial situations of real projects.

For example, the AMEISE system [¹³] allows students to exercise their software project management skills and reflect about their decisions. These decisions are based on incomplete and uncertain information and feedback, which are provided by the system. Certainly, GBL could be used efficiently to supply and simulate real processes.

In the same way, GBL (of board, cards and computer), simulators and others are being explored to support teaching in different areas of knowledge [¹⁴, ¹⁵, ¹⁶].

These approaches mainly support the idea that GBL should be combined with aspects such as motivating the students, being enjoyable and providing software, as well as providing an environment where students do not feel like they are learning [¹⁴]. GBL also is an important didactic proposal to explore collaboration. For instance, Alvarez et al. [¹²] show the benefits of including GBL tools in the classroom to foster collaborative learning and active student participation. The authors believe that games can be used as a tool supporting teaching methods that are effective towards the educational objectives.

Ebner and Holzingerb [¹⁴] suggest that there is evidence, which shows the learning result of using games is at least equivalent to the results from learning using the traditional method. For that reason, it is important to consider games as a modern and useful method for learning as they do not disturb or not offer disadvantage for the learners. Quite the contrary, the study carried out points out that GBL is one of the most preferred modes of learning from the point of view of students.

There are others approaches to teach SE by means of the implementation of computer game-development course. The students perform different roles in different areas of SE, the aim is to improve abilities, skills and prepare software engineers for industry demands in an interesting way as they use game-inspired exercises [¹⁷]. In addition, Qin and Mooney [¹⁰] proposed to use game-oriented projects as a promising choice to make the learning environment as close to the real-world software development environment as possible.

Claypool and Claypool [¹⁸] suggested building today’s entertainment applications, which use computer games, coupled with the software engineering discipline, arguing that it presents an opportunity to use computer games as a means to better train software engineers. Project-based modules can be used to illustrate all aspects of the software process, tapping into a broad range of SE disciplines as is required to build current applications while enticing students to grasp and apply software engineering to such disciplines by using games as a powerful motivator.

There are quite a lot of papers providing evidence that games and SE have attracted attention of software engineer’s educators [¹⁹, ²⁰, ²¹]. Hainey et al. [²²] Describe problems that are associated with traditional approaches to teaching requirements elicitation and analysis, with the authors suggesting that this is very difficult for a traditional course teaches students some of the skills that are required of professional software developers, and that traditional lectures do not adequately tackle the software development process.

Nevertheless, there are advantages and disadvantages of GBL as compared to traditional teaching as listed by Hainey et al. [²²]. For instance: lectures have as advantages control of the learning experience and provide models of how to address problems; as disadvantages there are no mechanisms to ensure that students are intellectually engaged, it is possible to lost the attention after a short time, and it is assumed that students are at the same level of understanding even if they are not engaged.

Just like GBL has advantages providing a safe environment to increase practice experience, GBL helps the students can naturally transfer what they have experienced in class to real life because they receive immediate feedback. Nevertheless, there are disadvantages: it has to be well planned, monitored and in some cases, it could put pressure on learners and could result in embarrassment. On the other hand, the authors also contribute to the empirical evidence in the context of teaching requirements elicitation and analysis, as they report on a five evaluation experiments comparing a GBL approach to a role-playing and paper-based approach. This evidence is collected thought of evaluations took place in Higher Education (HE) and Further Education (FE). Moreover, the results showed a significant increase in knowledge at both FE and HE level after the experiments had been applied. They also believe that the GBL approach to teach requirements elicitation and analysis may be more suitable to HE level than FE level. Finally, they suggest that initial knowledge should be taken into account when considering a GBL approach for different educational levels.

Paraskeva et al. [²³] describe online computer games and its teaching potential, highlighting some studies that have been focused on classifying types of teaching, which are supported by games. The authors also show the potential to draw together players from different contexts to communicate and collaborate. More than that, they suggest that with GBL it is possible to address ways that the student does strategies, hypothesis testing, or problem solving, preferably with higher order thinking rather than rote memorization or simple comprehension.

Paraskeva et al.’s work shows that the successful games have to be related to characteristics like rules, goals and objectives, outcomes and feedback, conflict (and/or competition, challenge, opposition), interaction, and representation of story. Finally, this work also examines the educational value of games doing. Along of a literature review this value is described, showing the games’ potential for player engagement, justifying with research reports on factors such as game use (frequency of game use, gender differences, identification with the characters, game preferences), and some psychosocial factors that may influence learning (academic performance, self-esteem, computer self-efficacy).

Boyle et al. [²⁴] and Connolly et al. [¹] present two systematic review of empirical research. They examined different aspects related to engaging and enjoyable activities in games. On the one hand, Boyle et al. focused in showing aspects of experiences with entertainment games like motivation for playing games; game usage and time spent playing games and the impact of playing on life satisfaction. Boyle et al. show that understanding game usability has had priority over understanding game enjoyment, cognitive and emotional involvement. The authors describe that subject, as real world dissociation, challenge and control are aspects little explored.

On the other hand, Connolly et al. [¹] focused on computer games and serious games showing empirical evidence about the positive impact of learning and how they are still growing, as noted earlier. They also quote modern theories that suggest that learning is most effective when is active, experiential, situated, problem-based, and provide immediate feedback. Connolly et al. [¹] show in their related work how games are integrated into the learning experience indicating aspects as feedback, strategies for varying difficult level, and availability of support memory that are keys to the success of the games-based approach. For that reason, the authors suggest that to encourage the use of games in learning it is important to develop a better understanding of the tasks, activities, skills and operations and, to be fair, examine how these might match desired learning outcomes.

Although authors like Caulfield et al. [²⁵] expressed that more evidence and research is required to prove the efficacy of games in teaching SE, we believe that teachers should consider using games as part of their courses because they could become useful and interesting add-ons.

2.2 Using Game Tools in SE Education

Given this context of GBL, we will focus on SE education on a game named SimulES and its evolution with different versions until it becomes a digital version named SimulES-W.

We agree with Liu et al. [²⁶] that if students learn computational problem solving with games they will be more likely to perceive a flow learning experience than in traditional lectures. They also confirm that there is a close association between the students’ learning experience and their problem solving strategies.

Our study with SimulES-W, analyzed feedback, examined the differences between the game and traditional lecture and problem solving behaviors. We found that these behaviors are strongly related to: a) understanding the problem, b) devising a solution plan and c) testing the plan as was mentioned in Liu et al. [²⁶]

Fully understanding the game is important to meet learning aims because perception of the game gives players “the true game” [²⁷]. In other words, when a player has a much better understanding of the game than the other player, the first can enjoy more with less effort and consequently achieve aims in an appropriate way.

The results presented in Hanaki et al. [²⁷] show that players might feel good without fully understanding highly complex strategic environments. The authors thought their results suggested that players might have a good understanding about the dynamic of the game, but it is necessary to maintain a level of mystery enveloping the game strategies as to live space for discovery. In the same way, it is necessary that players have a reasonable knowledge about their playoffs and opponents’ playoffs.

In the case of SimulES-W, payoffs can be higher if the game is in a more advanced stage when players have more artifacts, such that players can improve their game and damage opponents’ games, despite the SimulES-W randomness provided by throwing dice. It is interesting to notice how the players enjoy and feel satisfaction with the game’s payoff and how they use them.

There are simulators to teach software engineering such as SESAM [²⁸], SimVBSE [²⁰], SIMSE [²¹], and others presented in Table 1.

SESAM [²⁸] works as a simulation system that is able to execute models. It is focused on teaching software management. The basic idea of the game is to create a software process model with particular data, which is then simulated by the system. Quantitative data is generated based on the user selections for the specific project. As a result of the simulation, it is possible to analyze the process and the user choices.

SimVBSE [²⁰] is a game for students, which help them to better understand value-based software engineering. It departs with what the stakeholders have defined as critical factor of its success and the preference values, this last one is a concept predefined in the game and that allows to be analyzed in detail. The users (students) should identify what stakeholders regard as critical success factor (choose this concept) and determine a strategy to balance the critical factors with the other preference values.

SIMSE [²¹] is an interactive educational software that is used with a single player and simulates a software engineering process. Therefore, it guides students through the different software processes, in which students have to deal with budget, project time and other difficulties that arise when the simulation is running.

At the same time, students must make decisions that could affect positively or negatively the project. The idea in this game is to finish the project within the stipulated time and budget. These games are similar to SimulES because they were evolutions of the game "Problems and Programmers", also known as PnP [²⁹].

Approaching these games from the point of view of how they were specified or modeled, we observed the following. (a) SESAM game does not have any pre-defined method related to documentation on specification, design and architecture. (b) SimVBSE was built by prototyping method based on the different evolutions of itself.

And (c) SIMSE has been modeled for each one of its versions, i.e. each version of the game has its own specification, modeling and architecture with an individual approach to each. PnP was the source of inspiration for the different versions of SimulES up to SimulES-W.

3.1 SimulES-W Development

The development process of SimulES-W was performed by six steps as illustrated by the SADT diagram in Fig.. The first step was based on a review of the literature related to educational games as explained in [³⁸]. In addition, we studied the available documentation about the previous versions to gain an initial knowledge about the game. This theoretical knowledge was allied with the practice of playing the game, still in its manual version was used when playing the game.

As such, we gained a new understanding about the game and the intentional model of the game was useful for that [³²]. Intentional modeling, with i* [³⁶] for instance, was chosen because it represents the interaction among players; other representation approaches researched did not consider this interaction. The modeling language used in [³²] follows the ERi*c method [⁴⁰] which helps to create i* models.

During the elicitation process we used i* models to represent the interaction among players, and to show the game dynamics.

For the creation of these models we adopted the ERi*c method [⁴⁰] that builds these models using six steps, interconnected by means of requirements baseline. These steps are: (i) goal and actor elicitation, (ii) SDsituations identification, (iii) goal modeling for each actor, (iv) rationale modeling for each actor, (v) SDsituations specification, and (vi) analysis of SD and SR models as described by Monsalve et al. [³⁸].

Each round of the game is represented in SDsituation, which represents in a structured diagram the situations of strategic interdependence between actors sharing a goal common within an organizational context. Monsalve et al. portray the round Play Round to Start in [³⁸]. In Figure 2, the oval-shaped represents a goal that must be met and in which two actors interact, such as: Player in turn interacts with SimulES.

Fig. 1 SADT diagram - The process to build SimulES-W 

Fig. 2 SDsituation: Play round to start 

In the syntax i* this means that the actor Player depends on the actor SimulES (the system) to Game be started, the system available resources, publishes movements. In like manner SimulES depends on Administrator (role) to Entry of player be closed and type of cards be chosen. Also, Adversary depends on Player in turn to Project be accepted, as Player in turn depends on SE to be hired. In addition, the box-shaped figure represents resource; they may be read as follows: Player in turn depends on SimulES to available Dice. In Figure 2, the cloud-shaped represents a soft goal or a non-functional requirement and can be read as follows: Player in turn depends on SimulES to the game is availability.

Figure 3 presents the description about the model view controller architecture resuming the heuristics used to identify the elements of modeling that were subsequently used in the implementation. Summarizing, the behavior found in each SDsituation, presented in Figure 8, was implemented as user interfaces or related modules.

Fig. 3 General heuristics 

Fig. 4 Partial UML Class diagram of SimulES-W 

Fig. 5 SimulES-W packages 

Fig. 6 Main Page of SimulES-W 

Fig. 7 Example of an individual board 

Fig. 8 Example of Project Card 

Symbols in Lexicon, which are a starting point to describe the vocabulary of the application, were modeled in i* diagrams as resources or actors and then they were mapped to the diagram class (Figure 4).

Goals and tasks represented in i* diagrams which were categorized as verb type symbols were implemented as methods.

Simultaneously, as we were using these models to develop SimulES-W, we did one experience with an interdisciplinary group what had interacted with SimulES. We used elicitation techniques, such as the observation of the players and questionnaires, to evolve the requirements.

SimulES-W was developed mainly in Java with the support of other software tools and frameworks such as MySQL, Hibernate, and Javascript. The base used for the implementation was the ERi*c’s models which were mapped to a general MVC (Model View Controller) architecture [⁴¹] and the code was instrumented with scenarios describing i* tasks. This process is detailed in [³⁸].

The development environment used was NetBeans version 6.5. Figure 5, shows the packages that separated each component of the MVC pattern (Control, Model, and View).

In addition, two additional packages were created in the group Servlets and Util Classes. The project was written on Java and Javascript.

As source code metrics, we presented Code lines by Packages: View: 18034, Control: 4292, Model: 1806, Util: 506, Servlet: 367.

Classes by packages: SimulES: 79, Model: 22, Control: 18, Util: 8, Servlet: 3. Total methods: 1326. Classes with the biggest number of methods: Index: 96, SourceCardsPage: 85, ProblemsByPlayerPage: 71, SubmitProblems Page: 66, SessionBean1: 59. Frameworks used jMaki Ajax, Visual Web JavaServer Faces, JavaServer Faces and Hibernate.

This code was documented using scenarios [³⁵] and we used MySQL as a database server.

Figure 6 shows the main page of the game. This page exhibits the messages exchanged between players and system messages according to moves made in the game. These last messages are displayed when a player makes some moves in the game.

Furthermore, the chosen project and its modules are also displayed. Apart from that, a list of moves accepted between players is displayed in the bottom area of the screen as well as players that are online who are already registered in the game.

The version of SimulES-W presented in [³⁸] is being packed and will be released as soon as open source software. Being open software allows anyone to evolve it.

Furthermore, it will work as a web application and therefore it will not require downloads, installations or special configurations. However, the main advantage of SimulES-W is that it can be easily customized. That means, categorization can be incorporated; it will happen if the instructor chooses to approach the issue from a different angle or just want to stress specific topics.

For example, if the instructor identifies that the topic, which should be used in the training, is Requirements Engineering Management or Software Verification and Validation, then the instructor could produce cards to address the topic, and consequently the discussion generated when the players are interacting with the game will be the targeted one. In fact, the customization allows cards to be edited and stored in named files, so the game can be played with different emphasis as well as with different philosophies.

Other customization may be enacted, for example: changing the rate of white and gray cards or different calculations for time points, and analyze different players and tune the game accordingly with the observations found. Another advantage of SimulES-W is that concept cards and problems cards do handle links to web material, thus making possible direct access to bibliography or other resources. Having a direct access to supporting material is supposed to improve student performance, since the bibliography would be easily accessed in the right context.

This enables an even more targeted education, with a combination of cards and proper educational material.

3.2 Game Components

The most important idea in SimulES-W is that it simulates the SE process where each player takes the role of a software project manager in his own game, consequently he has to: deal with budget, deal with SE employment, build different software artifacts and control its quality by inspecting them before submitting the final product.

During the game, other players, named adversaries, such that new problems may be assigned to disrupt the player´s situation in the game, can establish some situations.

It is also the case that the player analyses his own game and the others player’s games to create a strategy of development, which allows one’s player game to become a better game than of his adversaries.

This strategy is based on the analysis of the player’s resources (problems cards and concept cards), the project budget and the skill that its software engineers have.

If he has a good and big team he can develop quickly and inspect all the artifacts latter or he can mix constructing artifacts and inspecting them.

The resources used during the game are: the main board (Figure 6), the individual boards (Figure 7), the project cards (Figure 8), the software engineer cards (Figure 9), the concept cards (Figure 10), the problem cards (Fig. ), the white and the gray artifact cards (Figure 7) and the dice.

Fig. 9 Graphical interface to manage of software engineers cards 

Fig. 10 Graphical interface to manage of Concept Cards 

The Individual Board is the base for playing, this is the area of the game in which each player places their Software Engineers in the columns and the artifacts in the rows.

Each project requires constructing artifacts of the following types: Requirements, Design, Code, Traceability and Help. Figure 11 shows the Individual Board in a game scenario with three Software Engineers.

Fig. 11 Graphical interface to manage of Problems Cards 

The artifacts are placed in cells of the board, below the software engineer who produced them and in rows for their types. These types represent the same type of real software project artifacts that must be built. At first when the game starts, the project has to be established and it will be available for all players.

This definition is made randomly so all players have to roll the dice and the player who got the highest dice result is the one who execute the action to get the project card and starts the game. This movement is important so all the players start to get involved in the game and starts to know SimulES-W interface and how the project characteristics will influence their moves. Figure 12 exemplifies a Project Card and its features, like description, complexity, size, quality, and budget and modules descriptions.

Fig. 12 SimuES-W SDsituations - the rounds in the game [³⁹] 

The project complexity is related to the white cards value, so the white cards will be worth the value, which is indicated in the complexity, and gray cards will be worth half of this value. The size denotes the number of modules to be built in each project and the quality shows the amount of error free modules that the final product must have to win the game.

The budget is the total project cash and defines the amount of money available to be spent.

Finally, the modules description describes the quantity and the type of modules to build on the individual board.

There is a round where each player throws the dice once. The dice result allows the player to draw white cards, gray cards and software engineers, if the dice upshot is less than three, then he will draw only concepts and problems cards according to this result. On the other hand, if the dice upshot is more than three he will draw concepts and problems cards and software-engineer cards.

The quantity of software engineer cards will be the difference between three and the dice result (dice upshot - 3). As an illustration, if Mary threw the dice and its result was 2, then Mary would draw 2 cards (problems and concepts). 1px solid blacktheless, if Mary threw the dice and its result was 4, then Mary would draw 3 cards (problems and concepts) and 1 software engineer card, these cards will be kept until that player needs them to make his move in the game, namely the player applies concepts or submits problems or places a software engineer card on the individual board.

Placing a software engineer (Figure 13), on the board (Figure 11) means that the player has hired a software engineer, but that action is limited to the project budget, since each hiring subtracts the salary of a software engineer from the total budget, so a new hiring is dependent on the number of software engineers already on the board and their salaries.

Fig. 13 Graphical interface to manage Individual Board 

Fig. portrays a characteristic software engineer card and its features like name, description of his personality, salary, which is related to the budget, an ability, which is related to the project complexity, and a maturity, which is occasionally used in problem cards conditions.

A software engineer can execute one or more actions in the individual board (Figure 11) such as build artifacts, inspect artifacts, correct defects and integrate artifacts in the module. He does these actions according to his ability, a number (see Fig. ), which determines how many moves he can do in each action and is related to the project complexity. It is also known as time points. For example, if the software engineer has an ability level of 2 and the project complexity is 2 he can build in each round one white card artifact.

The ability is the number of points of time (productivity) that the software engineer has to spend on each round, so it defines the number of white and gray artifacts cards that can be produced by him. In Figure 12, white artifact cards costs 2 and gray cards costs 1, so Karen (Figure 13) with the ability of 2 points of time and with the project complexity (2) can build 1 white artifact card or 2 gray artifact cards. Therefore, if the player has engineers with higher ability, then the player will have more productivity (will build more artifacts, and as such may finish earlier).

Each artifact is built either with a white or gray card. As shown in Figure 12, the quality attribute determines the number of modules that will have to be bug free, and the budget attribute defines the amount of money available to be spent in hiring software engineers.

Concept Cards and Problem Cards have a name, description and a reference. The reference shows the concept source and explains it in detail justifying its use. Thus, if the players are interested in some issues, then they could research more about it. The description in both type cards (Concept Cards and Problem Cards) describes how cards have to be used, some of them may also have a cost associated with it.

As a result, the player can use the Concept Card not only to block a Problem Card, but he can also use it to improve the player´s performance. Hence, both types of cards have the sufficient information to present a concept or highlight a common problem.

3.3 The Game Rounds

SimulES-W has different types rounds where players execute their moves such as: Start, Concept and Manage problems, Actions (Build, Inspect or Correct artifacts and Integrate Artifacts into a Module), and Submit product. Figure 12, an SDSituation diagram of the ERi*c method [⁴⁰] illustrates these rounds in a time-oriented flow chart.

When the game starts, one project must be chosen from those available [T1] in Figure 12. The dice is rolled and the one who gets the highest dice result chooses the project and starts the game. Furthermore, the information about the project is displayed to all players.

After that, each player assembles an individual board and picks up one software engineer in the stack of SE cards.

In the “play round to actions” [T2], each player with the information of his/her software engineers (ability and salary) and the information in the project card (size, complexity and budget) uses a software engineer to: build artifacts, Inspect artifacts, correct artifacts and Integrate artifacts in a module (see [T2] in Figure 12). In the action build artifact, if the player builds with white artifact cards, he/she will spend the points of time as per the complexity in the project card, but if he builds with gray cards then he/she will spend half of the points of time.

However, white artifact cards (5 cards to 1 defect) have a lesser defect rate than gray artifacts cards (3 cards to 2 defects). Inspect artifact is an action of turning up an artifact card under the responsibility of a software engineer, disclosing its quality status (with or without a bug - see (Figure 7). The cost of inspection is fixed by 1 point of time per card if it is performed by the same software engineer that built the artifact and 2 if it is performed by another software engineer. Correct defect action has to be performed when the software engineer inspects an artifact card and finds a defect (“bug”).

By correcting a defect, he/she spends 1 point of time if it is performed by the same software engineer that built the artifact and 2 if it is performed by another software engineer. Integrate Artifacts in a Module action has to be performed before the player submits the product. This situation happens when the player has built all types of artifacts required in a module (Figure 8).

The player can choose the artifacts that are available in his/her individual board, considering the artifacts types described in the project card to compose a module.

The artifacts can be originated from different software engineers (columns in the individual board).

In the “play round to concepts” [T3] in Figure 12, each player rolls the dice once. The dice result allows the player to draw concept/problem cards. These cards (concepts and problems) are shuffled together and piled upside down in the main board. If the dice shows a number greater or equal to 3, then software engineering cards may also be drawn. The quantity of software engineer cards will be the difference between three and the dice result (roll of the dice - 3). Thus, the greater the result from rolling the dice, the more resources the player will have.Here is where luck comes into play.

At this point, the player has to think about team composition: the number of software engineers is limited to the overall budget (see Figure 4), that is the sum of the salaries of the software engineers posed in the Individual Board. This implies the possibility of hiring and firing software engineers (project management skills). Note that there is an educational purpose of making students deal with real world issues (hiring/firing) by means of simulation.

In [T3] the player uses concept and problem cards. So during the game, the player can receive problem cards from the other players. These cards, when received, are to be used in the next round. The objective is to damage the game of the other. However, if the player has one card, which invalidates some problem cards (a concept card), the player will be able to use it and the action described in the problem card will not affect his game. Then he must discard both cards. On the other hand, if he does not have any card that invalidates the problem cards, this problem will be applied to his game. At this point in the game, the educational goal is that the players discuss both problem and concept cards claimed to pair. A player using a concept card has to build an argument as why that card neutralizes the problem card.

This argument can be discussed, but it will only take effect if all players agree. As mentioned before, this discussion can be mediated by an instructor. The “submit round” [T4] can be performed in the beginning of the player turn. When the player integrates all his modules, he can submit the product.

Then the other players have to inspect those artifacts that are not inspected (faced up). The product will be accepted, and wins the game, if there are n number of modules free of bugs, whereas n is indicated by the Quality attribute in the Project card Figure 8.

Figure 13, shows two different situations in The Individual Board: a) Build Artifact and b) Inspect Artifact. At the top the Software Engineers who is employed by the player. The Individual Board (Figure 13 a) portraits the White artifact Cards and the Gray artifact Cards. It illustrates cards, which have not been inspected yet. Alike, the part b of Figure 13 shows when artifacts have already been inspected and also the result of the inspection. This figure also shows when the artifact has a defect (bug) or not. This result is chosen randomly by SimulES-W and is based on the rate of defects mentioned before. Both sides part a and b at the bottom of this figure present the different operations, described above, that can be executed by the player based on his own board.

Each player would use his knowledge about SE in Play Round to Concepts and Manage Problems. This type of round is where he can throw problem cards (Figure 12) to another player, treat problems and improve his game if possible.

This round allows the player to use Concept Cards and Problem Cards which describe a typical process, good practice or problem in a specific area of SE.

Figure 10, portrays some typical processes and tasks of SE. These cards can be used by player to improve his game or block some problem that he has. After that the player analyzes his own game, looks if he has problems to deal with that have been submitted by other players and if some card could be used to treat or invalidate these problems. In similar fashion there are concept cards that can be used at any time and that can improve his game giving advantage over his adversaries. To illustrate it, there are concepts which enable to increase the budget, solve problems related to inspection, solve a specific problem and improve the ability of engineers to better build the software product that is asked in a more efficient way.

Problems Cards, shown in Figure 11, represent typical problems in SE process. This is the view of the player when his adversaries have submitted problems to him. The objective is to damage the game of the other and to try to win the game. [T3] above describes the mechanics of this type of round.

By the end, if player understands what the cards means and has the opportunity to use them, then he will apply the Problems Cards to halt the adversaries’ progress.

For this reason, the proper use of these cards could lead to the creation of a good strategy game that is reflected in the good performance of the player. This could lead to winning, but it is clear that there is a component related to the fate of the player and he has to have the right cards and know how to use them at the right time.

5.1 A Proposed Process to Teach with SimulES-W Didactic Concepts

As described in [³⁷] the proposed educational of SimulES-W is based on collaborative experience. This means that teachers address experimental learning to a specific issue and trainees or students should be able to learn through discussions and decisions making techniques which are represented by the game rounds. All the time, the players are interacting with each other. In the same way, these actions will be reflected in the final result of each student’s game. We strongly recommend teachers or trainers to offer a previous training about the game dynamics and how to use the software.

Alike, when the experience has finished, evaluations should be applied, one to evaluate the software acceptance and other to evaluate concepts that were learnt.

The teacher is the one who decides these contents according to what issue is necessary to teach. In Figure 14, we propose a didactical process that we established in order to apply SimulES-W as an education tool. The following parts were identified: Prepare Train, Execute, Feedback and Analyze.

Fig. 14 A SADT model of the process we propose to use SimulES-W 

5.2 An Assessment Scenario

In this section, we present a comparison of two education strategies for a part of a SE undergraduate course. The focus of our assessment relies on the part of the course that deals with Software Risk Management (SRM). One approach was just based on lectures, and the other was the use of the SimulES-W game to present the SRM material together with lectures. We report on the information collected and lessons learned from this experience.

This activity took place in the second semester of 2014; at the Universidade do Estado do Rio de Janeiro, Brazil. 37 students attended this activity divided into two groups. The division was made alphabetically. The first group (14 students) had one class using SimulES-W.

After that, in another class they answered a survey and attended to a SRM lecture. Another group (23 Students) only attended the SRM lecture. Each activity lasted 90 minutes. Both groups were tested by a software engineering exam with one question on SRM.

At first, to prepare the SRM contents in SimulES-W, two co-authors read [⁴²] Chapter 5 and studied how SRM involves identifying risks and developing plans to minimize the risk effect on software projects, stressing the tasks of risk analysis, risk planning, risk identification and monitoring. These tasks involve other activities like: identify project, product and business risks, evaluate the probability and consequences of risk, prioritized list of risks and also if risk could be known, predictable, unpredictable; risk type like project, product and business and related to technology, people, organizational, tools, requirements and estimation.

Once knowledge had been obtained on the subject, we divided the activities to create problem cards and concept cards to SimulES-W based on software engineering contents. Three people (one teacher and two graduate students) were responsible for the activity. The students made the problems and concept cards to the game; these cards were created and separated by subject in the book. After that, the teacher according to his experience and the reference book revised and validated the cards. They were refined and a final version was available. Finally, these contents were incorporated in the SimulES-W database.

Then the contents were analyzed by means of tests. The SimulES-W was ready to the activity with the students.

5.3 Data Collection (Feedback)

To assess the results of the experience a survey was designed using closed and opened questions to measure perception. In the first group of questions (closed) the students were asked to give a grade to some issues about SimulES-W usability, motivation, role in learning and if SimulES-W shows the dynamic of software development process.

Table 3 summarizes the answers by giving the statistical numbers and Figure 15, shows in a graph how the answers were distributed to the different questions. Questions 1, 2 and 3 used a Linkert scale and question 4 was a binary question.

Fig. 15 Answers of Closed Questions Related to the use of SimulES-W by UERJ Students 

For the first question proposed to the students: how do you perceive SimulES-W usability? As presented in Figure 15, the most of them report that SimulES-W is perceived as easy to use (57%), 22% of them claim that they find it easy to use. A lesser proportion of students claim that it is reasonable (15%) while 7% declare that it is hard to use. The results suggest that the game according to the appreciation of the players can become usable therefore; this could be a proposal to support software engineering class.

The statistics of Table 3 suggest positive results, given the median, the mean and the standard deviation, so: second question of Table 3 is related to motivation, that is, if the student found motivating or tiring the game. Figure 15 shows that most of them find SimulES-W interesting (72%), motivating (21%) and 7% of them report that it is tiring. Even with the positive result, we believe that it is necessary to better understand this question. For that reason, new studies are necessary. In a similar fashion, the median and the mean confirm the trend, thus, median with value 2 points to the answer Interesting, the mean shows the inclination for that answer (1,929) and standard deviation shows how far the lie from the mean.

Question 3 (Table 3) portrays the result related to how students perceived the role of the game, if it is oriented to teaching or otherwise if it disturbs. The result shows that most of the students agree that SimulES-W (Figure 15) teaches (50%) or informs (43%). Nevertheless, there are 7% who think that SimulES-W is neutral.

We think that this experience with SimulES-W was positive, and the result supports this thought. In a similar fashion, the median with value 1.5 refers to middle value between teaches and informs also mean shows the intermediate trend for those answers (1.571) and finally the value 0.646 as standard deviation shows how far the values lie from the mean. Finally, Question 4 (Table 3) enables us to identify if students recognize the dynamic of a software project in SimulES-W. The majority of the answers shows that the students recognized the existence of software process as the basis for the game (Figure 11), (79%), followed by a group who reports that they do not identify the notion of process in SimulES-W (22%). For this question we have interviewed some of the respondents and found out that of those who answered yes, found that the notion of a software project is clear, as well as the process for producing artifacts; for those who answered no, there was a consensus that the notions are still provided in a superficial way and it gives just a rough idea.

The second set of questions (open) was designed to gather student’s opinion in a free style text. With these questions we collected information related to problem and concept cards, strengths and weaknesses of SimulES-W and information about aspects of navigation, appearance and task execution. These questions are shown in Table 4.

In the fifth question, “What do you think about problem cards?” most of respondents answered that the goal is clear and that they represent real problems in software engineering.

Even thought, it is necessary to increase the number of problems cards so the game will be more interesting.

In the sixth question, “What do you think about concept cards?” most of respondents think that the goal is clearer than problem cards. “Concept cards are clear, when they did not apply to specific problem, they were useful to exemplify situations.” On the other hand, some students claimed that the game was too fast, and that for that reason it was impossible that some of them could recognize the cards quality and their details, as well as claims that some cards were hard to understand.

There are many positive aspects that were collected with question number seven, “In your opinion, what are the strengths of SimulES-W” Some of them were: “SimulES-W encourages me to learn software engineering.”, “it was a challenge that SimulES-W gets to simulate a software engineering environmental.”

“With SimulES-W it was possible to play in group and discussions are brought up.”, “it is possible to recognize the value of quality and the quality of software engineers in the software project.”, “this game tries to show stages in a software project.”, “with this kind of game is possible that class in software engineering becomes more practical.”

The eighth question, “In your opinion, what are the weaknesses of SimulES-W?” allowed us to identify aspects to improve in SimulES-W. Naming only a few: “there are many actions that are based on good luck”, students think that many actions in game should not depend on the dice, that it is required to improve performance and response time of the software, students suggest developing some strategy to improve the time that students wait between each round. Students propose to spend more time in training so they would understand better the software. It is necessary to improve the software navigation, interface and layout, create more alert messages being activated when users execute some action.

Students have also been asked to tell us more about contents and learning elements that were passed on SimulES-W (Table 5).

The goal with this part of the survey was to target a more depth on student´s understanding that is if SimulES-W communicated the contents planned.

In the first question, most of the respondents agree that concepts were taught related to typical problems in software development project combined with requirements, prototyping, and risk analysis. In a similar fashion, respondents think that SimulES-W teaches how the different tasks in software construction are important and some techniques to approach these tasks. It is also the case that students understand the importance of software engineers to the projects. One student claimed that he did not remember what was thought and other said that SimulES-W does not achieve the goal proposed.

In the second question, most of the respondents answered that the concepts were passed through the concept and problem cards, and by the Project manager tasks of hiring and budgeting.

The third question, “According to your own opinion, what conditions lead to lose or win the game?” most of respondents said that luck was the main condition to win or lose the game, other students said that it was necessary more time to end the game. In addition, students also believe that inexperience, shallow knowledge about project management and knowledge on decision making were important factors.

The fourth question, “According to your own opinion, does SimulES-W simulate a real situation?” most of respondents said yes (64%), furthermore they complemented the answer saying that it is necessary in software projects some kinds of strategy and to be aware that problems can occur. They also mentioned that the tasks of software engineers are clear, and also that some of the problems that may occur in real projects are presented, as in the case of dealing with personnel and budgeting. Students who answered negatively (29%) had problems with the problem cards, either because they failed to apply them or because they did not see them fit in their game. They also pointed out that the game is superficial since they did not use the concept or problem cards as expected, they consider that the game does not provide a real SE situation.

5.4 The Software Engineering Exam

For the sake of assessment of using SimulES-W we compared two groups of students, one using SimulES-W and the other with just regular lectures. An exam was used to test students of both groups.

As such, in this part, we included an analysis related to the exam applied after SimulES-W activity. The idea with this analysis was to look through the exams and find useful information related to the usage of SimulES-W. That means, our analysis is basically centered on identifying the student´s performance on questions related mainly to risk analysis, which was the emphasis given in the SimulES-W activity as well as their general performance. Finally, we compared the performance between students who attended the activity (playing with SimulES-W) with those who did not participate.

The exam had five questions and each question had different values.

Although just the last question was geared towards SRM, questions two and four are topics dealt by the dynamics of SimulES-W with type of artifacts and verification by inspection. The exam was a general one, but will be used in our assessment since part of its questions was dealt by the game. It is also the case that we are not doing a direct comparison of learning with or without the game.

We understand this as an advantage of our assessment, since we using the exam as a confirmation of the survey result and not as the prime result, which would be required a more complex assessment structure.

As such the hypothesis we are trying to check with the exam is if the students who used SimulES-W had at least equivalent learning results as the ones that used only traditional methods [¹⁴].

So, if there are no disadvantages for the learners who used the game, then we may assume that using SimulES-W does not influence learning in a negatively way.

All the students in the class took the exam. As noted earlier, the exam was composed by five questions and one question was about SMR, the target issue of this SimulES-W cards.

A total of 37 students took the exam, in which 23 had not played the game and 14 who did. Figure 16, a) (SimulES=Y) shows the success rate of the risk management question for the students who played the game. The figure portraits that five of the fourteen students achieved 100% of the answer this represents 36% of this group, one achieved 85% and this represents 7%, one achieved 80% this represents 7%, two students achieved 75% and this represents 14%, four students achieved 60% and this represents 29%, finally one student achieved 50% and this represents 7%. So, the majority of students got at least 60% of the questions.

Fig. 16 Results related to Exam Applied after SimulES-W Activity (SimulES=Y) 

Figure 17, part a (SimulES=N) shows the success rate of the risk management question for the students who did not play the game; of those (23), thirteen achieved 100% of the question and this represents 57% of them, alike two students achieved 75% and 70% this represents 9%, four students achieved 60% and this represented 17%, one student achieved 40% and this represented 4%, finally three students failed to adequately answer the question (13%).

Fig. 17 Result related to Exam Applied after SimulES-W Activity (SimulES=N) 

Comparing the mean of the two groups we see that students who played the game got a little better degree than students who did not play the game because 1px solid black of them get lower than the average (50%).

However, the mean number are very close (7,893 and 7,435) although the standard deviation has great differences from 1,873 to 3.703, showing grades that were more consistent within the group that played the game.

1px solid black of the students who played the game had a performance below the average (5.0) in risk management in software development. In contrast, the other group had four students below the average which represents 17% of the group. This is an important observation since not only SimulES-W did not negatively interfere on the learning process, but, it also showed that the players of the game had better grades overall.

If the whole exam is taken into account, Figure 16, part b (SimulES=Y) shows that most of the students (64%) who played with SimulES-W got a satisfactory score (Grade more than 7). However, 28% of them got a medium score (5.5 and 6) and only one student who did not answer the question of risk management in an adequate way (50%) got a grade (4.5) representing 7% of the sample.

In the same way, the Figure 17, part b (SimulES=N) shows that only 47% of the students got satisfactory score (Grade more than 7), 9% of the students got medium scores (6.9 and 6.8), 17% students got grades in the limit of the average (5) and 26% of the students fail the exam getting scores lower than five.

Comparing the mean of the two groups we see that students who played with SimulES-W got a little better degree than students who did not participate because the mean numbers for each group were 7,257 and 6,326 and the standard deviation have great differences from 1,308 to 2,325. From this data, it is clear that there was no disadvantage for the students who used the game.

In Figure 18 we present the results of a paired-samples t-test with a Confidence Interval of 95% (α=0.025). The t-test was conducted to compare the hypothesis that the two means for the correct answer of the Risk Management question and the

Fig. 18 Result related to Exam Applied after SimulES-W Activity 

Exam results were different for the two groups of students (students who played SimulES-W and students who only attended to traditional classes). The first result is that both groups can be considered to have the same distributions scores for the answer to the question about Management Risk because the t-test in a confidence range of 95% proved the hypothesis that the two groups have the same average and so there is no difference between the values answer questions of these two groups because the t-Ratio is equal 0,498 that should be less than 0,689. Otherwise considering the Grades of the Exam t-test rejects the hypothesis of having the same means for the two groups, the t-ratio 1,558 is greater than 0,936. So, this experiment evidences that students Exam Grades that used the SimulES-W had greater grades then the ones who did not use the game.