Author(s: Yalda Zdeh. An illustration of our Expansive Setting. [ Image by author ] — There is a set (or agents) and an environment, which are two distinct entities. Agent(s) aim to accomplish a specific goal within a sequence of decision-making tasks, while environmental constraints limit their ability. They may have an effect on the environment individually or collectively. However, the agents may interact with the environment to make external changes. The agent(s), whether they are working together or competing, may also affect it internally. Agent(s), or a combination of them, would decide to take actions. The environment then would provide signals that would help to guide the agents. If the signals are clear, learning occurs through reinforcement. Otherwise, it happens through some form or other of entanglement, which might require initiating. Both the agent and environment have a coding of past experiences in the form states at any given moment. If the agent(s), can observe all of the states in the environment, then they will copy these to their state. The agent may add a part of their history to augment the state, however, if the environment’s conditions are not fully visible. The entire sequence of uncharting of the environment’s functions through state observation, actions and signals can only be done if the agent does not have a mathematical model or mathematically solveable model. Sukrit Shashi Shankar The narrative presents an expansive setting, with multiple paradigms, related around the theme of Reinforcement Learning(RL). This setting is believed to help readers gain a wider view of RL and understand its underlying assumptions. It also allows them to see unexplored potential research avenues. The first article in the series, “A Discourse On Reinforcement Learning”, is here. We begin with a comprehensive overview of RL and an expansive setting. Part I — A vast setting Part II — Algorithmic methods & variations Part III — Future avenues and advanced topics The Agent (s) and Environment Agent(s) & Environment The agent(s), or set of agents, is a learner who attempts to make a decision by interfacing with the environment. The mutual dynamics among agents is not possible in single agent settings. However, multi-agent situations may require the [mutual-agents interplays] to be taken into consideration. [mutual-agents interplays] Agents may work together, compete, or in a mixture of competition and collaboration to reach decisions. Sometimes agents can have inaccuracies among themselves. Agents may sometimes suffer from imperfect information amongst themselves, i.e. On the other hand, the environment can be understood as something that may contain multiple objects. The agent’s actions or inactions, along with how they interact with it, determine the state of the environment at any given [time]. instance. [time] We can use the concept of time because we’re talking about sequential decision-making. The environment can have a significant impact on the ability of agents to make decision. As we will see, Agents may have no direct interaction with objects in the environment. It may happen that agents only interact with a small number of objects. This can influence their behavior and affect the environment. The environment may be described as: # Partially Observable# Fully Observable# Unobservable Agents make decisions based on what is presented to them (or not). These are the states of the environment. The decisions made by the agents in response to such communications can be referred to as their [action(s)]. [action(s)] It is possible that an agent does not take any action, but that is still a valid decision. The actions taken may not be immediately in response. Some actions could be dependent upon past state observations, or more generally, on future environmental conditions. b. The states We have already mentioned that the environments roughly correspond to their situations. Are we able to have the state(s) of agents as well? Are we able to have states for the environment objects as well? These states can tell us what they mean, and when is it possible to say something is a “state”. This discussion will continue from the perspective of the concept of a condition. A situation in an environment is a reflection of what happened to it in the past. This can be due to the actions of objects, agents, or the initial conditions the environment existed in. The environment’s future will be affected by the current situation. This philosophy states that a state can refer to any entity’s past and may have an impact on its future. [Reference] D. Silver. Google DeepMind 1. 1 (2015). Now we can see that states may result from past actions, observations and inputs. This is due to the agent’s interaction with the environment. The objects’ states may also be formed from their past behaviours and external inputs due interactivity. Remark: The objects can be considered part of an environment. The environment is at the top of the hierarchy. However, objects are located at a lower place in that hierarchy. However, the agent(s) is at the top in a different hierarchy, as they are not related to the environment. We won’t normally talk about entities as states. They aren’t at the top of any hierarchy. When referring to the state of the environment, most texts refer to them as states. Because agents can observe all aspects of the environment and make decisions based on that information, it is not necessary for them to create their own state. The environment is the state they are in. The agent would have to create their states if the observation was not complete or accurate at any given moment. They may be required to supplement the state (or null) communicated by environment with some of their history of actions and observations. To carry out future/ subsequent actions. Except where otherwise stated, states of an environment are also referred to as states. c. Environmental Functionality and State The state of an environment reflects its functionality. Is it possible to learn about an environment only by examining its states? It seems like it would take a long time, similar to taking pictures of a sequence at regular intervals and then uncharting the sequence step-by-step. What if we could instead have a mathematical-physical
