1 Introduction

Industry 4.0 is becoming more and more popular as smart devices and Internet of Things (IoT) technologies proliferate. By merging the current manufacturing system with the industry technology system, this new paradigm aims to improve the production system [¹⁷]. Industry 4.0 gives birth to various technologies, such as the Internet of Robotic Things (IoRT). It combines IoT and robotics into one technology, which is called the IoRT. ABI Research [²⁹] came up with the idea of the IoRT, which defines it as a collection of intelligent, disparate devices that can manage events and massive amounts of data.

Numerous researchers aim to leverage IoRT paradigm to incorporate it into other domains (e.g., Business Process (BP), etc.). Integrating IoRT technology into the traditional BP creates the IoRT-aware BP generation [⁴], which strives to increase productivity, automate operations, etc. However, executing an IoRT-aware BP within the organization could be expensive due to the energy and resources consumed, among others. To bridge these gaps, the Business Process Outsourcing strategy (BPO) remains the best solution by executing some parts of the entire process outside the enterprise [¹⁰].

Numerous environments can be considered to perform process outsourcing, for instance, the Cloud and Fog. The Cloud is defined as hardware resources and software services available on the internet [³²]. It has a large storage capacity compared to other surroundings [¹]. Furthermore, it enables businesses to increase their services in response to client demand gradually.

Using the Cloud during the execution of IoRT-aware BPs allows the business managers to meet their process’s computational and availability requirements. Despite its benefits, the Cloud fails to support sensitive applications due to the distance between the user devices and the Cloud data center. Therefore, the Fog environment appeared.

It is a paradigm that brings computational resources and services to the network edge near user devices, lowering latency and connecting with Cloud resources [²]. It is characterized by its ability to perform latency-sensitive applications regarding its proximity to the user’s devices (e.g., IoT, robots, etc.).

To achieve the outsourcing of an IoRT-aware BP into Cloud and Fog environments, a business manager should allocate the appropriate resources of (Cloudi, Fogj where i,j ∈ [1..n]). The Resource Allocation (RA) is defined as the groundwork for BP outsourcing.

It must match the demand of the instances of processes running with the resources available for a specific business objective (e.g., minimizing process cost, maximizing process availability, etc.) [²⁸]. Thus, the RA will allow the business managers to ensure the effectiveness of their processes. Additionally, it enables businesses to increase production, realize an equitable distribution of responsibilities, and decrease the need for human intervention.

Back to the literature, several approaches addressed RA for the business process. In general, these approaches focus on the Cloud resources [¹¹], or Fog ones [¹⁹, ³¹]. However, considering both environments enables business managers to take advantage of both. Furthermore, most of these approaches deal with process cost [²³, ²⁵, ²²] and execution time [¹⁵, ¹²].

Nonetheless, reducing the energy consumption within the enterprise and reducing latency can improve the selection of resources. Furthermore, diverse algorithms were applied for supporting RA in the BPs field, including exact approaches [¹¹], meta-heuristic methods (e.g., [¹²]), and machine learning [³]. However, these algorithms demonstrate serious overhead and the need for more accuracy in their evaluation.

Therefore, to close the gaps in the literature-based RA approach, we take advantage of Reinforcement Learning (RL) as an Artificial Intelligence (AI) algorithm to perform resource allocation with less overhead and scattered errors [¹⁹]. We aim through this paper to propose a Reinforcement Learning-based approach for the Fog and/or Cloud RA to achieve an optimal execution of the IoRT-aware BP. Our proposal aims to satisfy a set of different RA goals. Therefore, it intends to minimize the RA cost and reduce the energy consumed.

In addition, our proposal intends to reduce the execution time and the latency value. To accomplish these objectives, we extended the ifogSim tool to estimate the execution cost, consumed energy, execution time, and latency of an IoRT-aware BP using Fog and Cloud resources. The obtained results are promising. Furthermore, compared to other RA methodologies, the experimental results prove the efficacy of the proposed RL-based RA approach by offering a significant benefit in optimizing RA performance. The remainder of this paper is structured as follows. Section 2 presents a background that briefly describes the IoRT-aware BP and RL algorithm. Section 3 overviews the recently published approaches that deal with the RA question. Section 4 details the proposed reinforcement learning-based approach for RA to ensure the adequate execution of an IoRT-aware BP. The evaluation and discussion of the proposed approach are presented in Section 5. Finally, Section 6 summarizes our work and highlights its future directions.

2 Background

This section presents some relevant concepts to ensure the execution of an IoRT-aware BP using suppliers’ resources. In this setting, we intend to give an overview of an IoRT-aware BP. Then, we present a brief description of the RL algorithm.

2.2 Reinforcement Learning

Reinforcement Learning (RL) is an AI model that allows algorithms to learn from their trials and errors. Intending to understand how the algorithm makes the right decision, the RL has been confronted with a set of decisions [³⁰].

Therefore, whether it has taken the wrong decision, it is penalized, while, in the case that the right decision is taken, the algorithm gains a reward. The problems in the RL are frequently formulated as a Markov Decision Process (MDP). The MDP is defined as a mathematical framework for solving decision-making problems [³³].

It is used to present the environment model and dynamicity [¹⁴] (see Figure 1). The MDP model usually has a transition function to transform the environment from one state to another after applying an action using an agent. According to [³³], the RL has five main components which are detailed in what follows:

• – S: Gives the set environment states.

• – A: Presents the possible actions taken by the agent to change the environment state.

• – R(A) : S * A : Indicates the reward’s current scale.

• – f: Designates the state transition function. f(s,a,s’) = P(s’| s,a) is the probability that state s transits to s′ after performing an action a.

• – γ: Is a value in the range [0, 1]. It is among the MDP parameters. It defines a discount factor that is intended to reduce the impact of future rewards on the present.

Fig. 1 Reinforcement learning process [³,¹⁴] 

There are two main types of reinforcement models:

• – Model-free: Means that the model is based on optimal policies and value functions of the obtained data regarding the agent’s interaction with the environment. For the model-free, the agent learns with trial and error from experiencing explicit [³³].

• – Model-based: Refers to learning optimal behavior from a model of the environment, taking actions, and observing the outcomes that include the next state and the immediate reward. The policy of a model-based can be discovered using various planning techniques [³³].

In our work, we deal with the model-free as it does not require a model from the environment to make its decisions compared to the model-based.

In fact, the model-free is characterized by its simple implementation compared to the model-based that requires a lot of memory and computation.

Moreover, the model-free learns more quickly as it does not require a large amount of data to learn an optimal policy.

3 Related Work

In this section, we provide an overview of some recently published approaches that deal with the RA for a BP. These approaches are examined while relying on a set of relevant criteria, which are listed below:

This criterion distinguishes the most considered simulation tool.

Tables 1 and 2 classify the studied approaches regarding the selected criteria. Referring to table 1, we remark that most of the recent approaches, for instance, [³⁵], [¹⁵], [¹²], [¹¹], [²³], [⁷], and [²⁰] address the RA for the classic BP. To our knowledge, no work has been proposed for allocating resources to accomplish IoRT-aware BP execution.

Moreover, from the same table, we note that some of the studied approaches, for instance, [¹¹] deal with the Cloud resources for process execution. However, we noticed a lack of approaches that used both Cloud and Fog resources to execute the process. Furthermore, according to table 1, we denote that few works cope with the AI algorithms to support the RA. Indeed, AI has become fundamentally ingrained within numerous fields, for instance, the BP, and so on.

Referring to table 2, we note that most of the existing approaches (e.g., [¹²], [²⁵], and [²²]) deal with cost as a primary criterion to select the appropriate resources. Nevertheless, several other criteria can impact the selection of adequate resources. Therefore, the resource’s energy consumption can be tackled for the RA decision, where the business manager always seeks to execute their process using resources that consume less energy. Additionally, the execution time is among the relevant criteria that can impact the RA decision, where the business managers attempt to gain time by using resources that require less time to execute their processes. Furthermore, the latency can be considered when identifying suitable resources.

Our literature study points out that several approaches (e.g., [²³, ¹⁶, ²⁵, ²²]) ignore the process workflow during the RA decision-making. However, to ensure the correct process execution, it seems crucial to capture dependency between its tasks. In the same setting, we outlined from our literature review that different methods address the RA decision using process task, such as [³⁵, ¹⁵, ¹², ¹¹, ²³, ¹⁶, ²⁵, ²²] rather than the sub-process.

However, addressing resource allocation for a sub-process ensures the preservation of task workflow integrity. To bridge the distinguished RA gaps, we introduce an innovative AI-based RA methodology to proficiently execute an IoRT-aware BP leveraging Cloud and Fog resources. Our proposed approach tackles the IoRT-aware BP by dividing it into Single Entry Single Exit (SESE) fragments, which are blocks comprising one or more tasks.

Moreover, it manages process workflow to uphold task dependency execution integrity. This methodology is rooted in Reinforcement Learning (RL), capitalizing on its advantages to optimize performance and adaptability. The proposed approach defines several goals including reducing the RA cost, consumed energy, execution time, and latency.

4 Reinforcement Learning Based Resource Allocation Approach

Achieving an optimal RA using hand-coded heuristics and fixed strategies is difficult due to the heterogeneity of processes and resource requirements. In this research work, we design and implement a decision-making approach based on an artificial intelligence algorithm to accomplish the selection of adequate Fog and Cloud resources.

The proposed approach is called Reinforcement Learning-based Fog and Cloud Resource Allocation, which is referred to as (RL-FCRA). Figure 2 illustrates an overview of the suggested RL-FCRA.

Fig. 2 Proposed approach for fog and cloud resources allocation 

It is based on the RL algorithm, and it has several advantages that have proven worthwhile in the RA field. It does not require a large data set compared to the other AI algorithms. Furthermore, RL does not require training as it can automatically adapt to new environments.

In this work, we adopted Q-learning as an RL model-free algorithm. It handles issues with stochastic transitions and reward values. In the following, we describe the proposed approach steps.

4.2 Exploration

In the RL, exploration involves trying different actions to discover which ones lead to the best reward. It is considered one of the relevant RL phases, allowing the agent to achieve their goals with the best reward.

We model the RA problem using the MDP which is considered the most popular mathematical framework for solving RL problems. The MDP defines the tuple {S, A, P, R}, where:

• – S: Presents the environment states. In our work, we deal with tasks of the SESE fragments as a set of environment states (see equation 5):

S={T1,…,Tn}. (5)

• – A: Depicts the actions that can be applied to the environment. In our work, we deal with selecting the resource as an action. Consequently, we can perform the selection of a Fog resource (see equation 6) and/ or Cloud resource (see equation 7):

A1={Fog1,…,Fogf}, (6)

A2={Cloud1,…,Cloudc}. (7)

• – P: Sets forward the policy that is considered for an RL algorithm. To achieve the Fog and Cloud RA goal, we propose our policy that allows the agent to receive an SESE coarse-grained decision D generated by the Multi-Criteria Decision Method (MCDM) approach presented in [⁶]. The coarse-grained decision gives the selected environment (e.g., Fog, Cloud, Fog&Cloud, in-house).

  • If the decision D equals Fog, the agent will choose actions from the set of Fog resources A1. In the case where the selected decision D is equal to Cloud, the agent will target the Cloud resources A2 to choose the adequate resources for the SESE tasks. However, where the decision D equals Fog&Cloud, the agent will select the resources from both resource sets. Finally, where the in-house gives as the coarse-grained, the agent refrains from selecting any resources.

  • After identifying the set of resources, the agent initializes its Q-table values and applies algorithm 2 to both update the Q-table and select the appropriate resource. Based on algorithm 2, the agent selects the first task for each SESE and distinguishes the adequate resource according to the estimated reward RT value. After that, it performs the other SESE tasks, which aim to optimize the reward value as detailed in equation 8.

  • Hence, it is relevant to note that to achieve the RA goal, the agent takes into account the workflow patterns between tasks. As required by the process’s functional requirements, workflow patterns give the execution dependencies of the process tasks. During this work, We rely on the de facto BPMN standard [²⁴] to consider the process workflow patterns (e.g., sequence, parallel, loop, etc.). After each iteration, the agent updates the Q-table values. The job repeats until the number of episodes (processes) is reached (see algorithm 1).

• – R: Gives the reward function of the proposed RL-FCRA. Our proposed reward function (see equation 8) relies on the cost C, energy E, execution time ET, and latency L. When selecting the resource, we aim to reduce its costs and consumed energy. Moreover, we intend to minimize the latency and execution time to speed up the resource selection:

R=2∗C+2∗E+(ET+L)∗(1/2). (8)

Algorithm 1 Reinforcement learning-based algorithm for optimal Fog and Cloud resource allocation 

Algorithm 2 Resource 

5 Evaluations and Discussion

The goals of our experimental evaluations are two-fold:

To conduct these experiments, we implemented the suggested approach using the Eclipse tool. It is defined as a free and Java-based development platform that allows developers to implement systems and applications within different fields.

In our experiments, we considered a dataset of IoRT-aware BPs in the agriculture field. These processes were defined in the frame of the PRECIMED project [²⁶] and are developed under the Eclipse Modeling Framework (EMF) using the extended BPMN 2.0 modeler plug-in.

In Figure 3, we give an example of the developed IoRT-aware BPs, where this process presents a smart irrigation management system that aims to boost water-use efficiency and nutrients.

Fig. 3 IoRT-aware business process on agriculture field composed by fourteen SESE fragments 

We divided the process into various SESE blocks using the Refined Process Structure Tree (RPST) technique. Each SESE has its requirements regarding the million instructions per second (MIPS), random access memory (RAM), size, and bandwidth (BW) (see Table 3).

In the first experiment, we aim to estimate the required time to select adequate resources for each SESE tasks. In this setting, we measure the response time to allocate a resource regarding the number of process fragments and the number of considered processes.

Considering the results presented in Figures 4 and 5, we notice that the proposed RL-FCRA depends (i) on the considering SESE fragments of each IoRT-aware BP and (ii) the considered episodes (number of the IoRT-aware BPs). Therefore, we observe that the response time is directly proportional to the size of the involved fragments and episodes.

Fig. 4 Response time of the RL-FCRA according to the number of the considered SESE fragments 

Fig. 5 Response time of the RL-FCRA according to episode number 

The obtained results are justified by the limited capacities (e.g., MIPS, RAM, etc.) of the selected resource, which can result in a longer response time, where the agent struggles to keep up with the requirements of the SESE fragment.

Moreover, we assess the response time of the RL-FCRA by varying the available resources provided by the Fog and Cloud. In this setting, we consider the process presented in Figure 3, which comprises fourteen SESE fragments.

The considered Fog and Cloud resources are presented in table 4. These resources are generated using the ifogSim. The result of this experimentation is shown in Figure 6. Figure 6 shows that the response time depends on the number of available Fog and/or Cloud resources, where the time increases regarding the considered resources.

Fig. 6 Response time of the RL-FCRA considering various fog and cloud resources 

This is due to the exploitation phase ensured by the RL agent which should estimate the reward for each resource and then select the suitable one. In the context of the RL-FCRA evaluation, we conducted another experiment that aims to compare the rewarded cost, energy, execution time, and latency of our proposal with other RA approaches. The considered approaches are described in the following:

To be able to make this comparison in relevant conditions, the proposed RL-FCRA, RLRAM, and FIFO approaches were executed using the same input configuration of BPs, Fog, and Cloud features. In this setting, we take into account the process (see Figure 5) as an IoRT-aware BP that grouped fourteen SESE fragments, where each SESE has its characteristics (see table ??). Moreover, we consider a set of twenty Fog resources (Fogi where i ∈[1..20]), and twenty Cloud resources (Cloudj where j ∈[1..20]). Table 5 gives some examples of the resource features regarding their mips, ram, up bw, down bw, rate per mips, busy power, and idle power values.

Table 6 shows the results of the set forward experiment. It shows that the RL-FCRA outperforms FIFO and RLRAM regarding the consumed energy during the execution of the IoRT-aware BP instance (see table 6). Moreover, we denote from table 6 that our approach requires 1.10s to achieve the execution of the process while the FIFO requires 1.45s to execute it. Furthermore, the estimated latency for the RL-FCRA is around 0.04s. This value is relatively reduced compared to the RLRAM approach which is estimated to be 0.89s. However, for the cost parameter, we note that the proposed approach is less than FIFO.

The experimental results indicate that the RL-FCRA offers a substantial advantage in optimizing the performance of RA in BPM, which confirms the usefulness of our approach and demonstrates its relevance compared to RLRAM and FIFO approaches.

6 Conclusion and Future Work

With the proliferation of Industry 4.0, business managers nowadays seek to benefit from IoT and robotics that bring them enormous change in production systems, especially in shorter lead times, flexibility in manufacturing, etc. This led to IoRT-aware BPs. Nonetheless, executing such a process inside the enterprise may be costly due to the consumed resources, the need for computational capacity, etc.

To close these gaps, the business process outsourcing (BPO) strategy can be carried out to externalize the IoRT-aware BP for Cloud and Fog environments. Towards this objective, we proposed a RL-based approach to achieve an optimal allocation of the Fog and Cloud resources for executing an IoRT-aware BP, where this approach addressed the cost, consumed energy, execution time, and latency.

In future directions, we will address some of the current study limitations such as Fog devices’ security and mobility issues. Furthermore, in the future, we aim to enhance our proposal by scheduling the RA for the different process fragments.