In the previous post we introduced one of the most important concepts in Sciops: how differences in structural compression, and the resulting differences in resolution across structural components, can shape the quality of conscious experience.

Perhaps the most important examples of this, as the foundation of our sensory experiences, are space and time. Both space and time have an aspect of linear order. Elements and objects in space are adjacent, they exist next to each other. Elements and objects in time are successive: they exist after each other. Both can be mathematically described as containing a linear order in this sense (ie there is an order present with both adjacency and succession). However space requires a unified manifold or canvas: a linear order of elements/objects that is perceived together, at one point in time. Time on the contrary requires persistence of an object or element across changes, or a unification of elements/objects across time. Sciops hypothesizes that these qualitative differences arise from differences in the causal roles of linearly ordered elements/objects, resulting in different levels of information loss and redundancies that can be compressed away. Of course we would also expect these differences in information loss to be consistent with which information is actually useful for navigating real world environments.

We briefly discussed in the last post that in order for the conscious experience of an emergent structure to contain a clear linear order between elements or objects, that order must play a causal role, and moreover, for that order to be experienced in high resolution, downstream computations or actions must be conditioned on that specific order with high conditional probability (ie based on the output of downstream computation Y, what are the conditional probabilities over all possible orders of the causal inputs to Y).

In other words, if we have a set of elements, for example A through F, in order for them to be experienced both together and with a linear order, all elements A through F must play a causal role on the downstream convergent computation Y such that reshuffling the elements in a different order produces different downstream effects (instantiate different emergent structures), and from the specific downstream effects, the resolution of the order can be calculated using conditional probabilities (ie from said event, what is the probability of its cause being equal to a set of elements of each specific order).

We can imagine that there could be many different ways in which order could be retained with high resolution from the perspective of Bayesian Inference. For example a computational system could use positional encoding, where the position of an element in the order is encoded with some sort of signal or value which then can be utilized by downstream computations. In this case, in order to retain the order, the value of the positional encoding vector would need to be reconstructable from Y with a high degree of certainty. Transformers for example utilize this mechanism, though they are not guaranteed to retain this information in high resolution.

Alternatively the position could be built into the connectivity itself, with architectures such as a Convolutional Neural Network restricting information flow locally. For example CNNs apply a particular convolution to local regions and then slide these transformations across an image for example, creating these positionally overlapping local transformations. Such an architecture is designed to bias and retain local information and structure, but again is not guaranteed to retain this order in high resolution all the way through to the output.

The final mechanism could potentially result simply from compression, even if order is not strictly built into the connectivity or signaling. Imagine that, for example, we are dealing with a visual neural network where position is not so strictly encoded but the objects can be represented in their most compressed and simple form if a linear order is implied / emerges. In other words, if given a certain assumed linear order (across every iteration of the network) our visual objects can be cleanly modeled as lines, curves, circles, squares, etc, and without such an assumption far more information is needed to retain all the information of each (more random looking) shape, then the structure could perhaps instantiate such a high resolution order via compression alone. And since Sciops hypothesizes that the conscious experience of the structure should match the most compressed form of the structure, this could be a legitimate way of instantiating the phenomenal experience of order.

Of course what we have described so far only specifies how a linear order can be instantiated structurally but has yet to differentiate the linear order in space compared to time. Before diving into the structures of our conscious experience of space and time, it’s worth clarifying the difference between our phenomenal experience of space and time, and how the terms space and time are used in physics to describe events in the natural world. Let’s term the variables of space and time in physics: physics space and physics time.

One thing that is abundantly clear is that we cannot make a short cut in our thinking and attempt to simply inherit or smuggle in any qualities of physics time or physics space into our experience of conscious time or conscious space. For example neural computations which occur at different places in physics time may be experienced in one moment of conscious time. Similarly, neural computations that occur in different physical neurons at different points in physics space could be experienced at one point in conscious space. We discussed the core reason for this with a simple example using logical expressions, where for example (A OR B) OR (B OR C) = A OR B OR C, which could compress multiple neuronal computations into one expression and an experience that could occur at one moment. Compression can take multiple events that are distinct in physics space or physics time and merge them into one emergent structure and one corresponding conscious experience that loses these distinctions.

IIT correctly makes similar arguments regarding our conscious experience of space and time (Haun & Tononi 2019). But of course this distinction between physical space time and our conscious representation of space and time has been discussed for centuries (Kant’s Critique of Pure Reason being perhaps the most famous example). And since we cannot simply define conscious time or conscious space using properties of physics time or physics space, Sciops would hypothesize that the burden is fully on us to define conscious time and space mathematically based on the emergent mathematical structures that arise in the brain, such that their phenomenal qualities arise from that structure alone.

2.2.1 Differences in Linear Order Compression Between Space and Time

If we describe conscious space and conscious time (hereon referred to simply as space and time) both using a linear ordered set of objects (space being adjacent and time being successive) then after careful examination, we can clearly see a vast difference in resolution across each ordered set of objects.

Linear Order in Space

Let’s use a visual example where we have a scene full of visual objects (though a more full understanding of vision will be a topic for the next post). It’s clear that we have some level of non-negligible resolution into all objects across a visual scene and that we also have resolution into the approximate order (and extendedness and distances across that order) between objects. We can even imagine an example where all of the visual objects are approximately the same structure: picture a work of abstract art with red polka dots spread evenly across a canvas: despite there being many instances of virtually the same object (a red circle), they are not compressed into one object, but instead remain extended and their individuality (and therefore plurality) as well as their order/location is preserved in relatively high resolution.

The implication of this is that spatially separated objects each play a unique causal role on downstream emergent structures, and the individual differences between them such as their locations and plurality are also causal, such that this information is not redundant and cannot simply be compressed away. And since space appears congruent and relatively continuous (there are not large obvious gaps in our visual space) this seems to imply that generally all spatial objects play a causal role. The exact nuances of how this can be computationally feasible will be discussed in subsequent posts and may practically require recurrent connections to maximize resolution – however it’s worth noting now that our entire spatial scene is surely not entirely high resolution at every single moment – change blindness studies show that there are in fact gaps in our vision that go unnoticed and our periphery is clearly very blurry, but nonetheless as we shift our attention and vision around we experience a consistently full and linearly ordered world of objects.

From the perspective of Bayesian Inference, Sciops would therefore hypothesize that downstream emergent structures that are experienced spatially (ex visual space) are conditioned on a linearly ordered set of elements and objects such that the individual spatially separated objects are not redundant and cannot be compressed away, and that these binding emergent structures are also conditioned on the order of objects across space. Furthermore, as we’ll discuss below, we will predict that not only are spatial objects not redundant (and fully compressible), but they are also not compressible to a lower dimension and less extended space.

Of course one can ask the question how such convergent emergent structures (which are conditioned on spatial objects) emerge in the first place, and as discussed in the previous post. To quickly recap, Sciops defines bound emergent structures as compressed mathematical structures that converge multiple spatial objects into a smaller set of non-redundant downstream actions. For example if you are reading the word “Tesla” on this page, the computation associated with reading Tesla, and it’s downstream effects are both 1) conditioned on detecting the sufficient set of lines and curves in a specific 2D linear order such that they match the word Tesla and 2) produces a set of downstream effects (thoughts, actions, etc) that are likely always (or often) triggered upon reading that word, thus allowing them to be compressed into one emergent reference frame – ie essentially a logical gate that triggers a specific set of consistent effects conditioned on this set of visual shapes.

Thus the key distinction with space is that the linearly ordered elements and objects that are experienced spatially have individual and extended qualities (each object’s order, size, distance, etc) that are not compressed away and their information is not lost. Downstream scene understanding seems to be conditioned on nearly the full set of linearly ordered objects in some high resolution region of space. And this makes sense: in order to navigate the world correctly you need to be able to see all visual objects, including their sub-objects (lines, curves, dots, etc). If similar looking objects that were actually physically distinct just collapsed into one spatial object, then we would have serious difficulties navigating the world.

Linear Order in Time

We can model time as a linear ordered set of objects in succession, where the first place in our linear order refers to an object (or set of objects) at time T1, and the second place in our linear order corresponds to an object (or set of objects) at time T2, and so on. Of course when imagining this we may imagine a visual movement over time where objects are also spatially ordered within each moment of time (ie there could be a spatial order within T1, etc), but for this discussion we are only referring to their order across time, ie from T1 to T2, etc (one could imagine a sequence of musical notes as perhaps a better example).

With time we clearly see that the individuality / plurality of objects across such an order is not cleanly preserved. For example, if we imagine a movie as a set of images (frames) that rapidly flow in quick succession: the human visual experience does not experience each individual frame: we clearly cannot easily count how many frames were played, and we certainly cannot mentally detect the distance (in terms of number of frames) between two distant frames. The idea of frames in a movie is a useful analogy but shouldn’t be taken overly literally: the same statements can be made about our eyes detecting the real world: when we see a basketball fly towards a hoop we cannot decipher how many different basketball images/locations our retina detected along its flight path, or how many distinct feed-forward passes of this information through our visual cortex occurred, or easily estimate any sort of distance between them.

Instead what seems to happen is that, if structurally similar enough, multiple objects that are linearly ordered across time (ball at T1, ball at T2, etc) are experienced as the same object, compressing away redundancies such that they can be reduced to one non-extended mathematical object. And within each object some differences are preserved and experienced as change over time, while others are non-causal and compressed away (ie things that we don’t notice changing to objects as a scene evolves).

Thus, unlike spatial structures, where events are conditioned on a full set of linearly ordered objects, including their individual plurality and extended location, temporal structures seem to be conditioned on simply detecting a single object and how it changes over time, rather than being conditioned on each individual instance of that object at each time. And again we can reason over why this may be practically: in order to navigate the real world we need to understand which objects are present, how they are changing, and what their current state is, rather than preserving detailed information about each individual past state along the way.

To better understand this difference we can walk through a toy example with a linear ordered set of elements:

Imagine we have a set of twenty objects which we can label O1 through O20. Each object has a magnitude value (which is in itself a relation between elements) which describes how many of the same element is present in each object (please reference the last post as a refresher of how multiple instances of an element can be compressed to a magnitude corresponding to that object).

Now imagine that our magnitude distribution across objects looks something like this:

In the graph above we’ve displayed each object as a spatially separate visual object, so your brain will naturally see seventeen individual bars separated spatially. However we can also notice a patter where objects O2-O5, O8-O11, and O13-O16 all have similar distributions. One could imagine that the brain may group these three sets of four bars into one combined parent object: resulting in three very similar looking objects, and one smaller distinct object (O18-O20). But the experience would still be spatial: we would see four distinct spatial objects separated spatially and all presented together at one moment in time.

Now we can imagine the temporal example which requires losing information by compressing redundancies. For example, if we had our object O2-O5 at time T1, then we could describe O8-O11 as simply the same object at time T2, but having grown slightly (specifically at the O10 location). Similarly, we could describe the same object at time T3 having grown a but more (modeled by O13-O16). In these examples we have one base object (O2-O5) that changes over time as opposed to three distinct objects. And again, Sciops would hypothesize that this can occur if information such as the number of distinct objects, their distance from each other, etc is not recoverable in any reasonable resolution from the perspective of downstream scene understanding and action selection computations, thus allowing the three instances of our object to be compressed to one (thereby reducing redundancies) and then preserving any high resolution causal differences in the object by experiencing a change in the object over time.

Thus we can see in the example above that the same underlying information (some linearly ordered set of objects) could be experienced either spatially or temporally based on the causal role of the information on downstream computations, and how redundancies can be eliminated to reach the most compressed form of the structure. So far we’ve discussed this idea theoretically, but to understand how neural networks can practically instantiate such a structure, we need to revisit the idea of Higher Order Relations.

2.2.2 The Role of Higher Order Relations in Time

The example above implements two types of First Order Relations (FORs) between our elements: order and magnitude. However to fully explain how neural networks can actually instantiate time and space we need to discuss their Higher Order Relations.

To remind the reader, Higher Order Relations (HORs) can be defined as a relation between a set of relations. For example if as linear order changes across a structure, magnitude values change according to some structured way, then this implies a relation between linear order and magnitude (two FORs) across this structure. As one can imagine, HORs are incredibly useful for compression: rather than modeling all FOR values for every single element in a structure, much of this information often has a predictable pattern which can be exploited by compression via HORs.

We asserted above that with temporal experiences that we retain high resolution into a single object and how it changes across iterations (ie linearly ordered objects over time) without retaining resolution into the details of the object in each iteration (ex in each visual frame). We can model this as a combination of 1) a high resolution description of a single object at a single iteration (for example the current one) in addition to 2) an HOR that is activated when spatial changes to that object follow a specific structure over the temporal linear order.

For example in the case of a ball falling: rather than downstream computations retaining all of the information about the ball across each iteration (ie frame), these downstream computations could simply preserve the spatial and visual features of one single instance of the ball, along with a set of neurons which detects something like: an object falling at ~9.8m/s^2. This type of computation, detecting a specific kind of change to an object across the linear order of time, is precisely an example of a Detection HOR: ie a computation that detects a specific higher order relation (in this cause the relationship between the objects location and how that changes across the linear order of time according to ~9.8m/s^2), and then conditionally causes some downstream action if this higher order relation is detected. And to be very clear: the higher order relation here is really a second order relation between the spatial linear order at time T1, and the spatial linear order at time Tn.

It’s clear how such an HOR could produce a time-like experience, as all of the individual information about each specific iteration of the ball (ie each object in our temporal linear order) is either detected by the HOR (and therefore experienced as change over time) or is lost and therefore can be compressed away. And the information needed in order to experience the ball falling spatially (as many separate balls separated spatially across a linear order) such as the count of objects, their linear order distance from each other, etc is not preserved. As a result, the most compressed structure will not preserve each iteration’s individuality, and rather than experiencing plurality of the object across space, we experience one persistent object across time.

If we were to describe time only with detection HORs however, our model may be too passive – the detection HOR by itself is subject to the whims of the inputs, as it only detects structures when they are found in the earlier layers but doesn’t actively instantiate such structures. The brain in reality likely plays a more active role in predicting the future, especially regarding short term changes over time. This is where we may expect Transformation HORs to also play a role in temporal experience. Whereas a detection HOR simply detects a relation between two or more FORs and then causes some computation as a result, a transformation HOR acts like a function: it may take some structure (with all of its elements and relations) as an input, and transform those relations in a structured way.

For example we could imagine that since gravity is ubiquitous, and we would expect objects that are falling at 9.8m/s^2 to continue to do so at this acceleration, the brain could contain transformation HORs that would take some visual object at time T1, and apply a transformation of its place in the spatial linear order such that it predicts a new location at time T2 consistent with gravity. This then allows the brain to perform recurrent processing / predictive coding, where it compares its predicted transformation of the object with the next wave of input data.

Mathematically, a perfection resolution detection HOR for two FORs y = f(x) can be described as:

Whereas a transformation HOR can be described more simply as:

It’s reasonable to imagine how useful it may be for detection HORs and transformation HORs to work in tandem to improve structural accuracy, resolution and binding. For example transformation HORs could modulate the threshold / sensitivity of detection HORs based on which structures are likely to occur at subsequent times. Similarly, detection HORs could modulate the activity of transformation HORs which could refine noisy inputs, transforming them to into more structured FORs. And we could also imagine detection HORs that detect particular sequences of subsequent transformation HORs.

Crucially, we would predict that a typical series of transformation HORs across iterations in itself would not be sufficient for creating time without detection HORs if the sequence of transformations in isolation could simply be compressed to the equivalent one-step transformation. If so the most compressed structure would be equal to the single transformation and would destroy any sense of time across each transformation.

In other words given a sequence of transformation HORs:

Then:

Which can potentially be compressed to:

Interestingly, detection HORs and transformation HORs will not always correspond to the same information loss and experience. Generally, we think of detection HORs as being invertible and often containing a bijective mapping (for each element across one FOR, the HOR defines a single value for the other FOR). As a result, the resolution into each FOR would typically feel symmetrical.

On the contrary, we may expect transformation HORs to have far more non-invertible and many-to-one mappings. And if we have downstream computations which are conditioned on the output of a transformation HOR, then their resolution into the input of the HOR would be subject to this information loss.

Experientially, we may expect a detection HOR to feel more as if a particular higher order relation (ex a shape or path) gives us resolution into the all FOR values symmetrically: for example the relationship between the x and y values on a grid and the equation for a circle create the experience of the shape of a circle. With a transformation HOR, we may feel more that the HOR allows us to experience one FOR (or lower level HOR) through its asymmetrical relationship with the other – for example we have some sense of the possible past states of an object, given the current state and how the current state was reached via a transformation.

To better understand how both FORs and HORs can alter the resolution of the experience, let’s walk through a few scenarios in depth, all from the perspective of an object changing over time. All of these scenarios could potentially really occur in the brain based on how downstream computations and actions are conditioned.

Scenario 1 – Current Object in High Resolution, Detection HOR in High Resolution

In order to understand each scenario here rigorously, we need to understand which additional information each object and relationship is telling us and how it can play a causal role in downstream computations. A detection HOR can be thought of as a function that tells us, for any given x, this is the corresponding y (where both x and y represent FOR values such as order, magnitude, etc). What a detection HOR does not necessarily tell you, depending on how it is configured in a neural network, is which actual x values were detected in the current object. Thus we can have an HOR that may carry information such as: the detected relationship is such that for every x, there is a defined y according to some function, and based on the threshold of the neural network there is at least some minimum number n of x, but the exact number and location of each x is unknown.

This matches our experience of time: for example we may know the general path of change (how an object is moving, stretching, spinning, etc) without experiencing how many frames occurred or the exact spatial qualities of each frame relative to each other. But if we bind together this path of change over time with a high resolution spatial experience of the current object, then we get something that seemingly resembles time: ie we see an object, and experienced some sense of how that object changed over the recent past.

Scenario 2 – Current Object in Low Resolution, Detection HOR in High Resolution

This scenario is less common but also happens in every day life: imagine if a movement is quick enough we may experience the nature of the change of the object over time (ex it’s path or trajectory) without actually seeing what the object is in high resolution.

Scenario 3 – Current Object in High Resolution, Detection HOR in Low Resolution

If we experience the current object in high resolution, but it’s detection HOR over time in low resolution, this implies that we can visually understand the current state of the object, but that the object is changing in such a way that we cannot understand or describe how it changed over time clearly (it’s path is unclear, it’s changing shape in strange ways, etc). Alternatively, perhaps the object is simply not changing, thereby not triggering a detection HOR.

Scenario 4 – Current Object in High Resolution, Transformation HOR in High Resolution

Transformation HORs are performing a very different computation than Detection HORs. Whereas a Detection HOR essentially asks the question: given a set of x,y pairs, is there a specific function that can predict y based on x (and vice versa), the Transformation HOR simply takes a given object x (in each iteration for example) and transforms it to it’s corresponding object y based on some relation.

Thus if we bind together the experience of a current object in high resolution, with a transformation HOR that acted on that object, rather than getting a possible path of past objects and how they changed, we simply get some sort of sense of the specific pre-transformation object, via the transformation itself. And again, if we are examining a sequence of transformation HORs over time, then the new information gained by the transformation HOR itself can potentially be collapsed to a single compressed transformation (ie collapse a multi-step functional transformation to the equivalent single step transformation).

As a result, the experience of a high resolution current object + a high resolution transformation HOR (or sequence of them) across many iterations would in itself likely not be experienced as time, would be experienced as some sort of knowledge regarding the initial state of the object, and the current state of the object, and some transformation of that object which essentially caused this transition.

For example, if we have a falling ball, whereas the detection HOR may better describe the path of this ball as it falls, the transformation HOR would give us a better sense of: the ball is currently in midair as a result of being higher and then having gravity applied to it, and we predict in the future will be on the ground as a result of a similar transformation.

Scenario 5 – Current Object in Low Resolution, Transformation HOR in High Resolution

In this scenario we see another contrast between transformation HORs and detection HORs. Even without the current object, a detection HOR can still tell us the path (change over time) of some object, because it can still tell us that there was some minimum number of xy pairs that followed a specific functional mapping. However without the current object, the Transformation HOR cannot tells us what the pre-transformed object was at all. In fact, the only information gained by a high resolution transformation HOR without the current object, is that a certain type of transformation occurred. For example perhaps a transformation HOR associated with predicting falling objects could, if activated with high certainty, communicate to downstream computations that something fell, or gravity acted on something, without giving any more information about the object of path.

Thus where a detection HOR in isolation may tell us about the shape of change (or path), a transformation HOR in isolation may tell us more about the type of force (or causation) that occurred, without any information about which object this force was applied to.

Scenario 6 – Current Object in High Resolution, Transformation and Detection HORs in High Resolution

Finally we can put all of the building blocks together, and we begin to see a structure that seems to more closely match our real experience of time. If we have high resolution into the current object (ex a visual object), as well as a detection HOR describing how it has changed across iterations, as well as a transformation HOR which alters and predicts the details of those changes, then our experience would essentially be: We see the current object in high resolution, we understand it’s prior state and what causal forces caused its state to change, and we understand the path that it took between the prior state and the current state.

It’s also worth noting, that though our structure of time for the most part feels like a lossy compression of space – meaning that it is more compressed than space but loses information in the process – that actually there are some structural aspects that we seem to have better resolution into with time than space. For example, imagine the visual object of a clock face, and that there are three instances of this visual object, each encoding very similar but slightly different times (ex 3:14, 3:15, 3:16). If these three clock faces were experienced as three visual objects over space (ie simultaneously but at different points in space) it would likely be harder to spot the differences between them, then if the three clock faces were experienced in succession over time. With the latter, we would likely more easily spot difference in the clock hands via their motion. And this makes sense if time utilizes detection HORs specifically designed to look for such changes, and the activation of these HORs can feed downstream computations associated with recognizing such changes. On the contrary, spatial HORs are typically more concerned with identifying individual objects rather than spotting their differences.

2.2.3 The Role of HORs in Space

Unlike time, where HORs seems to play a crucial role in instantiating a structure of change across a persistent object, we could imagine a spatial structure that doesn’t necessarily rely on higher order relations at all. Instead we could architect a network where inputs are linearly ordered, and we use AND/OR gate like logic to detect shapes across that order. If we are ignoring practical computational restraints, we could for example imagine a single AND gate conditioned on each possible spatial object or combination of spatial objects (ie one for every possible visual scene that our model can experience). As long as the order information is either encoded, or can be instantiated via compression, Sciops would hypothesize that we can create a bound spatial experience this way.

However it’s also obviously true that detecting shapes can be made far more efficient, if rather than being conditioned on an exact layout of points (ex all the points on a circle), instead downstream computations are conditioned on a higher order relation between the different spatial dimensions (for example how the color at each point in y space changes as one moves in x space). Similarly, even without the explicit use of detection HORs, shapes can be defined in a more compressed way using HORs than individual points (for example the equation for a circle is far more compressed than a sum of the location of every point on a circle). And since the most compressed structure is what is experienced, we therefore would expect our experience of space and shapes in space to typically be defined by HORs.

This idea has very interesting implications for the long standing debate as to whether or not our conscious experiences (ex that of space and time) are continuous or discrete. Based on the theory that our conscious experience matches the most compressed structure, Sciops would predict that both types of experiences are possible, and discrete experiences can occur when the most compressed structure is one defined by a discrete number of elements and a discrete set of relations between them, and that continuous experiences can arise when those structures are compressed to higher order relations that encode a continuous relation between structures.

One final point on higher order relations in space: We discussed in section 2.2.2 how detection HORs describe the relation between multiple dimensions (ex x and y) without necessarily defining which xy pairs were detected in any particular moment. If we consider this spatially, the resulting experience would be that of a geometric shape, but such that how “full” the shape is (ie how many points on the shape were actually detected) is blurry.

At least in our foveal vision, this doesn’t seem to be a common occurrence. Instead we typically experience our shapes as nice, clean, full shapes, even if the underlying light input information is far more noisy or incomplete. Thus our visual networks seems to instantiate some sort of bias into its detection HORs such that if a relationship is detected between xy pairs, the structure seems to almost assume that all xy pairs are present, ie the shape is full. Structurally we may say that these detection HORs have a very high threshold in terms of the number of xy pairs detected, and somehow the system is biased towards reaching those thresholds. The exact mechanism for this requires further discussion and research but it seems either transformations HORs (which modify the structure to be more full) or recurrent connections may be at play here. More on this in future posts.

2.2.4 Extendedness in Space

Some of the classic phenomenal properties of space include that space contains order (adjacency), it is extended, it is unified and simultaneous into one shared spatial field, and it contains a plurality of objects within this field with distance and directional relations between them. We’ve discussed at length already how all of these properties rise from the emergent structure of space except for extendedness, which we will do now in depth. And our understanding of HORs and time will be very useful in this discussion.

If we are examining any set set of ordered elements, when fully compressed to the least redundant mathematical structure, this set of elements must either:

1. Retain no extendedness within each element or the relation between each element, resulting in infinitely small spatial width, such that adding more elements (with an order relation) does not extend the set of points in any direction, and the set of points can grow in number while remaining dimensionless point-like entities.
2. Retain some extendedness or spatial width (that exists within the relation between each element or in the element itself), such that adding more elements/relations grows the extendedness or width of space

We must examine both of these possibilities from the perspective of compression and information resolution.

In scenario 1, if our ordered set of elements act as dimensionless points/relations, but their order is causal and retained in high resolution, then it follows that this order must be retained in some phenomenal aspect of the experience. If not, we have a mismatch between the emergent mathematical structure and the quality of the experience, which Sciops would hypothesize is not possible. Therefore, if the order is not experienced as space, or extendedness, then it must be experienced in some other way.

Time was one example of this: rather than ordered elements being experienced as next to each other in one unified simultaneous space-field, adjacent points are experienced in succession. But this is of course only possible because we discarded information about each ordered element (or object) which cannot be structured temporally (individuality, distance/direction, etc).

Similarly, we can hypothesize (and will discuss in future posts) that there could be other ways of encoding order without extendedness. For example, a computational system could assign a positional encoding variable and use a number to encode position and distance, and this number may be experienced in its most compressed form as a magnitude rather than extendedness in space.

In the subsequent posts we will discuss how both color and pitch have aspects of order. And of course we will also need to consider other possible structures which contain non-spatial order that humans do not even experience, but are structurally possible.

So what determines if an ordered set of elements or objects is experienced spatially, with extendedness, or if it this order is encoded with some other structure such as time, or pitch or color? The astute reader will already anticipate the answer: compression. Sciops hypothesizes that the experience will always match the maximally compressed structure.

This idea is particularly useful for understanding extendedness. In fact we can describe extendedness as a structure in which multiple objects are experienced both A) together: in one unified and simultaneous spatial field, and also B) separate: each object has its own distinct location in which it can uniquely display its other structural properties without those properties blending in with its neighbor. An ordered set of dimensionless points lacks the second property: separation.

And separation in a sense fights against compression: the most compressed structure commonly converges objects to maximally reduce redundancies. Thus our conscious experience, like a ruthlessly efficient ruler, is constantly fighting to destroy extendedness, to destroy space, and to represent structures in a more compressed form. But extendedness cannot be compressed away if it is necessary to preserve information that plays a significant causal role in triggering downstream emergent structures, and therefore is experienced in high resolution by those structures.

We can use visual space as an example: each unique point in visual space encodes a very large amount of information. Not only does it encode color and greyscale values, but also the brightness/magnitude of each color, and it does so across at least two spatial dimensions. Furthermore spatial objects are structured based on HORs across all of these variables along both dimensions. Thus Sciops would predict that all of this information simply cannot be retained in the necessary degree of resolution without extendedness: ie without retaining a separate position for each object so that all of its properties can be distinctly retained. In other words, the high resolution information at each point in visual space cannot be retained while collapsing the structure to a dimensionless set of points. The resulting blending of information would create incongruent information loss.

Using similar reasoning we could suspect that two spatial dimensions corresponds to the minimum required of object separation via extendedness to represent all of the information in the conscious experience a visual image, and that if there was even more information retained (ex multiple colors for each point in space, other structural elements, etc) that even more spatial dimensions may be required.

To conclude on the topic of extendedness, it seems that the existence of extendedness in our conscious experience very much depends on the combinatory mechanics of the information encoded in each object in the ordered set. Extendedness is likely only necessary when 1) the ordered objects cannot be combined or collapsed into a dimensionless representation, and must be kept separate, but also 2) the objects must be bound into a unified and simultaneous experience and cannot be separated with a non-spatial structure like time (ie temporal separation or succession) or other more compressed structures.

2.2.5 Neuroscience and the HORs of Time and Space

There is extensive research that supports the hypothesis that neural systems in the brain are selective for higher order relations, both spatially and temporally. We won’t go into great depth into each study here, but it’s worth noting research that has shown several important computational building blocks for HORs and linear orders, such as selective excitatory and inhibitory signaling, dendritic discrimination of temporal sequences / order (Branco et al 2010), multiple non-linear computational subunits in each neuron (Polsky et al 2004, Gidon et al 2020) and the connection with dendritic spikes and the higher order relationship of spatial orientation (Smith et al 2013), gain modulation which can affect detection (Fergusen & Cardin 2020) and predictive coding (Shipp 2023), just to name a few. The important take away here is that the neural evidence supports the hypothesis that neurons and neural circuits in the brain can be, and are, selective for non-linear and complex higher order relations between inputs.

Research has also discovered much of the specific selectivity when measuring neural responses in the visual cortex to the higher order relations of space and time. It has been known for a long time that the early visual cortex is selective for specific lines, edges and curves in V1 (Hubel & Wiesel 1968), combinations of these edges in V2 (Anzai et al 2007), and complex shapes in V4 (Pasupathy & Connor 2001). Importantly, visual representations become more stable and temporally persistent as information flows to deeper and more hierarchical regions, supporting the idea that individual object instances are largely discarded in favor of persistet objects over time (Piasini et al 2021). These findings are in line with the downstream convergence of objects and more complex spatial HORs that Sciops predicts.

Sciops also predicts that with space, extendedness is preserved when the individuality and plurality of objects is retained in high resolution, and that this is less present over temporal order. Several studies confirm the presence of neurons and neural circuits selective for the quantity of objects and groups of objects in a visual scene, including neurons fine tuned to be maximally responsive to specific quantities (Nieder & Miller 2004, Nieder & Merten 2007, Viswanathan & Nieder 2013). Furthermore this high resolution understanding of quantity can be compositional, allowing for quantities both within groups and between groups to contribute to our sense of numerosity (Moscoso et al 2021).

There is also neuroscientific evidence for neural circuits that respond selectively to specific changes over time which could correspond with temporal HORs. Early direction specific detectors are present (Albright et al 1984, Euler et al 2002), with later circuits in v5 showing selectivity for complex objection direction changes independent of orientation (Rust et al 2006) or location (Graziano et al 1994), providing evidence that some HORs can detect specific kinds of changes while discarding other information about the individual objects. We even have evidence of neural areas that respond to object persistence, independent of change/transformations over time, which can be key in binding together detection HORs with the persistent object that they are detecting change over (Rust & DiCarlo 2012). Other regions of visual processing show selectivity to specific more complicated kinds of movement, such as radial, spiraling and circular optic flow, providing more evidence of specific temporal HORs (Duffy & Wurtz 1991).

Things get particularly interesting when we dive into the research on various frequencies: temporal frequency, spatial frequency and frame rate frequency. Substantial research shows that different neurons can be independently selective to spatial frequency, temporal frequency, or speed (temporal frequency / spatial frequency), supporting our hypothesis that we can have HORs tuned to spatial distributions, temporal changes in one area, or temporal changes to a persistent object (Priebe et al 2003, Lui et al 2007). Interestingly, even at very high frame rates that are not consciously perceivable, a neural response can be seen in terms of phase-locking (Williams et al 2004), and Sciops would predict that since the count of objects and distance of objects across frames is not consciously perceived, that either there is no downstream selectivity for this structural property (ie difference in phase-locking don’t causally influence down stream frame rate detectors) or that if there is downstream selectivity for frame-rate that this information is compressed away in scene binding. Similarly, even when frame rate is slow enough to be noticeable consciously (in the form of a lag or even a flicker) we would not expect the actual count or speed of frames to be easily decipherable (in stark contrast to intuitive numerosity with vision) unless so slow that the differences across scenes no longer are perceived as persistent objects. All of these hypothesis could be tested experimentally by Sciops Labs.

These studies are key in connecting the mathematical structures that are proposed by Sciops simply using the principles of compression with known neural structures. And there seems to be strong agreement there. What is less clear from the existing research is the exact flow of information to deeper computations and how information is lost and retained from some of the above circuits to the areas of the brain responsible for visual reasoning and decision making. This flow of information, for example, is crucial in binding individual detection HORs to the broader visual scene, visual reasoning and action. Though some of the information selectivity has been discovered already (detecting certain spatial / temporal components while ignoring others) further research is needed here. We will propose some new studies later in this post.

2.2.6 Movement and The Arrow of Time

One useful (though not always exactly precise) way of visualizing the different phenomenal qualities of our conscious experience is that space gives us a relatively complete picture of a structure, without significant loss of structural information. As a result we see the structure in full, in a sense. This is not true of our other qualia, such as not only time, but as we will see in future posts it is also true of qualia like color, pitch, pain and more. For these experiences we lose information about the structure, and this information is hidden from our bound emergent reference frames in different ways, resulting in different qualias.

In the case of time specifically, we proposed that time results from a full resolution into only part of the structure (objects in the current moment) and a loss of information into the rest of the structure (full detail of the same objects in all previous moments). However, as discussed above, temporal structures do have some resolution into the past sequence of objects, specifically regarding their changes, triggered by transformation HORs and detected by detection HORs. And this information must not only be experienced somehow, but must be experienced in the most compressed possible form.

It should be very intuitive to see that if the information contained in the changes to an object over time are 1) small compared to the information contained in the object itself and 2) are relatively smooth over time, that the most efficient way to experience the information encoded by these changes is movement.

For example, imagine we have a complex visual object (ex a specific human face) at time T1, and this face is positioned in our spatial visual field at the spatial coordinates 21,14,7 (imagine in this example that visual space can be described via coordinates which is an oversimplification). Now imagine that at time T2 the spatial coordinates change to 24,13,8. To communicate this change in location in full resolution, without duplicating any of the information in the face itself (which is far more structural information than is encoded by the change of location), the only way it seems to do so without introducing redundancies is to use movement: ie to move the face from position 21,14,7 to 24,13,8.

We can also imagine that movement might not be the best way always to compress such an experience in the case that the information encoded if the change between objects is greater than the information in the current object itself. For example, if we just flash random images on the screen, the experience is not of one object moving to morph into another object, but of two separate objects. The experience can still feel temporal, perhaps only because other visual objects in this example are persistent and do change smoothly over time. Otherwise, as Kant points out, without persistence we would likely not experience something time-like, but instead just a sequence of disconnected experiences popping into and out of existence. In practice, masking and flash experiments seem to show that incongruent visual objects that are shown in quick succession can often just fail to be bound together into one experience.

The Arrow of Time

The discussion above makes the argument why certain structures must be experienced as movement over time, but it doesn’t make any strong assertions yet regarding the direction of that movement, and why the arrow of time always seems to flow in the same direction.

One could hypothesize that this occurs simply given the nature of binding in the brain: that for example a detection HOR that defines a particular movement/path sequence is only bound to the full resolution current object, when that object perfectly matches the final position in its path (ie the final xy pair in the functional mapping between FORs). However this doesn’t feel like a particularly practical or satisfying answer, as it would seem reasonable for a detection HOR to be triggered and bound mid-path, while still defining the future trajectory (especially when transformation HORs may help in shaping that future trajectory).

A better hypothesis likely has to do with the asymmetry of transformation HORs. If transformation HORs play a crucial role in binding, shaping the structure of the current object based on the predicted movement of that object, and if that transformation is always non-invertible, then this could perhaps explain the constant arrow of time. And we could imagine that this transformation could be non-invertible if there is a many-to-one mapping that destroys information, such that the transformation is more predictive and more compressive if run forward than backwards. We can also consider the possibility that the transformation may be non-Markovian in one direction, meaning, in order to predict the subsequent state we need more than the current state, while remaining Markovian in the other direction. If true, this unidirectional non-Markovian nature would make the reversing the arrow of time rather strange, as it would imply that the Tn-1 state would not only predict the Tn state, but also the Tn+1 state, etc – which may be inconsistent with a feeling of the present.

These ideas of many-to-one and non-Markovian mappings seem especially relevant for transformation HORs which surely utilize saved information about the past (memory) to predict the next state of an object. From this perspective, as soon as a neural network moves beyond passively detecting patterns over time, and begins to predict or anticipate such patterns based on past information, the direction of time seems in a sense inevitable. And of course humans can also take actions based on past information, thereby influencing the environment and of course any predictions over future states. However the issue is perhaps not so simple, as we could ask the question: well what if we imagine the universe ran in reverse? Could the information in a future brain state not also predict some past actions or neural computations?

Ultimately if we want a truly satisfactory and rigorous answer to the arrow of time however we would need to begin a discussion that is beyond that scope of this post. Such an analysis would require removing the assumption that any emergent structure in the brain (for example any neural computation, not just those encoding transformation HORs) can only be conditioned on past events and opening the possibility of conditioning on future information as well. Perhaps it’s possible for example that there are some emergent structures that are best compressed by defining reference frames conditioned on future information.

A truly rigorous analysis of this question would likely need to take a God’s eye view / block-universe perspective and then ask about the full scope of emergent structures based on maximal compression. Ultimately these abstract and challenging questions are highly related to entropy, and the differences in entropy between sub-systems in the universe and how this relates to compression. They are great topics for future research. What we can say confidently for now, is that compression and emergent structures can introduce structural asymmetries, and such asymmetries when combined with time/movement can result in directions of time/movement.

2.2.7 Conscious Space & Time as a Representation of Physics Space-Time

It has long been remarked that space and time as we know them are merely our conscious representations of the “real” physical world. Sciops makes this concept abundantly clear and more mathematically rigorous: space as we experience it is a type of emergent structure, resulting from high resolution, bound, linearly ordered and extended (non-collapsable) objects that arise computationally in our brain.

Consequently, we cannot assume that space as we know it is a property of lower level mathematical descriptions of the universe like quantum mechanics or general relativity. Perhaps order itself is a necessary ingredient for any meaningful mathematical structure, including those defined in physics, but order in itself is not enough to create the experience of space, as we’ve discussed at length in this post. And of course, it’s not a coincidence that the laws of physics and our conscious experience of space both contain order, as clearly the purpose of our conscious structures is to represent and model the physical world to an extent that we can successfully navigate our physical environment.

An area ripe for further research would be to examine the compressibility of linear orders in the equations of physics, and from which reference frames this linear order must retain both unity and separation (and thus extendedness), and from which references frames the order can be collapsed to a more compressed structure. Performing such an analysis for example on the Schrödinger Equation, from the context of various measurements in real experimental set-ups, would be fascinating.

Similarly, conscious time is the result of an emergent structure that compresses many instances of structurally similar objects into one persistent object that changes over time. And the structure of conscious time, even more so than space, feels very specific. If much more information was preserved across iterations then the experience would likely be spatial, and if less information was preserved then each moment of time could just collapse down to a single step. Therefore we may suspect that conscious time may be an even more rare emergent structure in the universe than space. For example we may expect the deterministic transformations defined by many physical laws to collapse many times into one step in their most compressed form, where QM of course provides perhaps a very fascinating complication to this idea with measurements and observers.

To conclude, the long-standing assertions that time and space are conscious representations, distinct from physics time and physics space, is well supported by Sciops and our mathematical framework for defining space and time as emergent structures. And the opportunity for future research, investigating how emergence can play a role in physics and our understanding of physics time and space, is incredibly exciting and promising.

2.2.8 Epistemic Heuristics and Future Research

When self-reflecting on our work thus far on space and time in the context of our 10 epistemic heuristics, we’ll be both satisfied with our progress thus far but also excited for future research opportunities.

The heuristics that are most obviously satisfied in our work already outlined in this post are that of 2) Neural Structures (see section 2.2.5), 3) Simplicity (we’ve derived all of this from the principle of compression) as well as 4) Mathematical Structure, 7) Fundamental and Non-Arbitrariness, as well as the very Sciops specific heuristics 8) Rigorous Boundaries, 9) Binding via Convergence and 10)Defining Qualia via Compression.

Thus we will frame the bulk of our concluding discussion in this post on heuristics 5) Phenomenal Specificity and 6) Structural Specificity to self-critique which areas have been explained with satisfaction and which require further research, as well as briefly introducing some experimental predictions that will be further detailed in future posts (Heuristic 1: Experimental Falsifiability).

If our definitions of space and time are perfectly phenomenally specific, then our proposed structures for space and time should be clearly incongruent with any other conscious experience.

If our definitions of space and time are perfectly structurally specific, then we should be able to show that no other structural description other than ours could possibly explain the experience of space and time (unless it was mathematically equivalent). In other words, we would need to show that our precise structural definitions for space and time are necessary and sufficient for the experience of space and time.

Heuristic 5: Phenomenal Specificity

If one is to accept the core Sciops premise, that consciousness contains no redundancies, and matches the most compressed structure instantiated by the neural network of the brain – and one also accepts our model for binding of objects via convergence from this compression (detailed in the last chapter), then we believe we have now in this chapter supplied a sufficient structural explanation for classic properties of space: order, extendedness, unity, simultaneity, relations defined by distance and direction, etc. It’s clear that other non-spatial human conscious experiences (time, pitch, pain, etc) do not possess all of these properties, and therefore our proposed structure for space cannot mistakenly describe them. In this sense, our proposed structure for space is phenomenally specific when it comes to human experience.

The same can be said for time and its properties of persistence, temporal order, change and movement, temporal unity, etc. We believe the structure proposed here for time clearly contains these properties based on information resolution, and therefore cannot be applied to other conscious properties of known human experiences (an image with no movement, any instantaneous sensory experience, etc) that are not temporal.

However it is possible that other types of conscious experiences could exist that are not spatial or temporal, and that humans simply have not experienced them. Thus it’s possible that our structures here actually define a space-like bucket of experiences and time-like bucket of experiences, rather than only exactly our space and time. Therefore this is an area for further research: examining other possible conscious experiences that could match the properties defined here for space and time.

Just as one example: we described extendedness with the properties of unity, simultaneity and separation, and these properties don’t seem to match any other human conscious experiences, but perhaps they could match other non-human and yet non-spatial experiences.

Heuristic 6: Structural Specificity

It is a much stronger claim to assert that the structures defined in this post are a necessary and sufficient explanation of conscious time and space, such that no other structural explanation is possible.

We will not make such a claim here, and instead believe that the structural components defined in this post are perhaps necessary for the experience of space and time (with further experimental evidence needed to confirm this), and that they are only possibly sufficient, with further research needed to explore other possible requisite criteria.

To better understand this, let’s examine all of the definite and possible qualities of conscious space and time and discuss whether or not the Sciops structures are necessary and/or sufficient to satisfy these properties.

Space

Extendedness

If one is to accept the core premise of Sciops, that our consciousness matches the most compressed structure, then our mathematical description of extendedness does in fact seem necessary for the experience of extendedness, simply because if a structure can compress away the separation that is present in extendedness without losing information that can be retained via Bayesian Inference from downstream structures, then this compression must occur. And it doesn’t seem we can have extendedness without both separation and unity. Without unity, we cannot have a cohesive spatial field, and without separation, different elements/objects cannot occupy different points on that field.

The proposed structure however is not necessarily sufficient for explaining the extendedness of space. For example, perhaps the ideas of unity, simultaneity and separation are too broad to be in themselves sufficient for space, and as mentioned above, maybe they could also instantiate other non-spatial or non-extended types of experiences. Imagine for example if we could experience dozens or hundreds of pitches at the same time, while retaining clarity into the separation of each one, while retaining a unified experience. This is not true of pitch as we know it (beyond a few pitches the objects get muddled together), but we could imagine some sort of other pitch-like experience could be possible. Thus there could be yet another criteria or structural property necessary to define only space, and not other structures that may contain the structure we’ve detailed in this post.

This opens the door for both further Sciops research, but also applying other theories of consciousness (IIT, RPT, NR, GWT, etc) on top of Sciops to further define the full set of requirements for space. For example perhaps recurrent processing is necessary to specifically define space and only space. Or perhaps the specific nature of space is crafted from the emergence of a global workspace reference frame. Such claims are not supported at the moment and are just examples, but could warrant further research.

And of course we will always leave open the possibility that emergent structures themselves are only necessary but not sufficient for conscious experience of any kind, a prediction that other theories such as IIT and RPT may agree with. Thus it is implied in every post that any Sciops structure might be necessary but not sufficient for any experience.

Unity and Simultaneity of Space and Plurality of Spatial Objects

In the previous post we already discussed how one emergent structure can be conditioned on multiple non-redundant structures, thereby binding them together while retaining resolution into each via conditional probabilities. With space we now simply need to add the conditions that this binding occurs with multiple objects such that the objects are ordered (resolution into their order is retained via conditional probabilities), extended (despite being bound together their individuality is retained and they remained separated), and simultaneous (their individuality cannot be compressed to one object that changes over time).

And we would hypothesize (though this central claim is certainly not absolutely proven) that one cannot have unity or bound plurality without compression. Sciops claims that unity cannot arise without compression: reducing redundant structures into one unified structure. And without a unified structure, Sciops would also hypothesize that there cannot be conditioning on a discrete number of unique causal objects (bound plurality). It doesn’t seem, for example, that the continuous laws of physics would give us these properties without compression.

Again, if you accept the Sciops hypothesis of compression, these seem like necessary conditions of space: if any of them are compressed away, we lose a required property of space. However, this doesn’t imply that they are sufficient for the Unity and Plurality in space. For example, to actually recognize and identify multiple unique objects, this may require further downstream processing, which we will discuss in the next post with vision. And as mentioned previously, the ability to bind together many spatially distinct objects with any reasonable degree of resolution may require recurrent connections or other sophisticated computational techniques.

One way to think about it: Sciops seems to provide much of the necessary foundation for precisely describing extendedness, unity, simultaneity and plurality, but may be missing some additional necessary building blocks to actually construct a full human spatial experience. It’s also difficult to imagine a full spatial experience without discussing concepts like color and other visual properties, which we will do in the next post.

Relative Location, Direction & Distance

This is an area that requires further research and discussion. Though we have defined order and extendedness, and when applying this concept to a 2D or 3D space, we can derive a sense of order, direction and distance, we have not properly defined how the mind can navigate a full spatial scene, bouncing between different sets of bound objects and shifting attention and resolution.

For example, just because a downstream computation may be conditioned on a specific order of objects does not mean it can automatically extract distances and perform geometric calculations. The exact mechanisms in which the brain performs some of these tasks and actually practically binds together so many objects while nearly perfectly preserving their geometric relations is speculative and requires more research. These questions are highly relevant for the tasks of computer vision and visual reasoning, which are fields currently lagging behind purely text based ML tasks. Thus, this area of research remains a huge and exciting focus for Sciops Labs.

Continuity & Divisibility

It is clear and non-debatable that at least visual space generally appears to be non-gappy and can be divisible, meaning that larger spatial regions can be divisible into smaller regions. What is less introspectively clear is if space is fully continuous and infinitely divisible, as opposed to discrete and finitely divisible.

The concept of divisibility in general arises from the layering of objects, where emergent structures can be conditioned on multiple non-redundant elements, unifying them into an object, and then this process can repeat, unifying multiple objects together and created nesting objects. However this does not necessarily imply infinite divisibility.

It’s also not obvious that all spatial experiences must be non-gappy. For example a migraine could wipe out large areas of our vision. In every day experiences, our spatial experience of sound doesn’t fill all of space and is in a sense gappy. These examples are not perfect as there are other mechanisms at play here, but they serve as examples that allow us to at least visualize a gappy space that is not self-contradictory.

In regards to fully continuous and infinitely divisible spatial structures, as mentioned above, HORs can be one explanation for how this might be possible. Simply put, some spatial objects could be described in a more compressed way via a continuous HOR than they can be as a discrete set of points. However we cannot claim that this mechanism is either sufficient or necessary for fully continuous space, or that full continuity is even definitely a property of space. Perhaps space only appears continuous because our resolution is not high enough to distinguish the underlying discrete elements for example.

Another concept that we will discuss in future posts is the idea of overlap: that there could be overlapping orders between bound objects, and the way these overlapping redundancies are resolved could play a role in the experience of space in regards to continuity and divisibility. Overlapping orders are very common for example in CNN architectures and in the early visual cortex. This idea of overlapping spatial regions seems central to IITs formation of space via connection, fusion and inclusion for example. It’s certainly an area for further research for both Sciops and other theories.

Boundaries & Regions

To some extent the idea of distinct and distinguishable boundaries arises from the ideas of extended objects, however just because objects are separated and extended does not mean that their relative positions need to be high resolution and not blurry. We see this with peripheral and distant vision for example. Thus the idea of distinguishable regions and boundaries has more to do with maximizing resolution over extended objects. We will discuss this idea more in the next chapter.

The other interesting boundary is the idea of the edges of a bound space field itself, and how are for example the edges of our visual experience seems to fade away rather than having a distinguishable boundary. This could perhaps just be explained using resolution: ie the peripheries of our vision are so low resolution that we cannot distinguish any “end” to our visual field. However this explanation may not be fully sufficient to explain this property and further mathematical investigation is needed to say for sure.

Time

Persistence

If one accepts the core Sciops hypothesis of compression, then it follows that compression itself is necessary for any sort of persistent emergent structure, and therefore for persistence across time.

Nonetheless the argument could still be made that compression alone, and specifically converging multiple instances of an object into one persistent object, is not sufficient for persistence over time, and that perhaps some other structure like recurrent processing is required. But further research would need to develop grounds for such a claim mathematically and why it could be necessary, which is currently not clear.

Of course persistence, and persistence over time are not the same thing, as the latter requires other properties of time such as succession.

Temporal Order / Succession & Motion

One can perhaps imagine succession without persistence, with disconnected objects popping into and out of existence while preserving some knowledge of their order. But to have succession as we know it, with time smoothly flowing from one moment to the next, objects changing and moving smoothly, our structure needs persistence and smooth changes to persistent objects.

To have persistent objects, our neural network must compress physically spatially and temporally distinct computations (object at time T1, object at time T2) into one structure, and in order to experience motion or smooth succession, the network must retain resolution into how this unified structure changes over this linear order. If all information regarding change is compressed away, then we have objects with no change, and therefore no sense of succession or time. But if too many differences between each iteration of our persistent object are preserved, it may no longer be compressible into one persistent object (for example if conflicting or incongruent differences are present or if relations that require individuality are present in high resolution, thus requiring something like extendedness).

Thus it seems that the proper balance of compression is necessary for time: enough to create persistence, but not enough to destroy change and motion. However further research, both experimental and mathematical, needs to be done confirm if the exact mathematical structures of detection HORs and transformation HORs are necessary or sufficient to instantiate this precise structure. Further researchers could also speculate whether persistence + change over linear order is sufficient for defining exactly time, or if other conscious experiences could share this structure, and if some further structure is needed to narrow down this change to be exactly motion as we experience it.

The Present & Memory of Recent Past

Our work on “the present” and the arrow of time is promising but incomplete, requiring further research to precisely define directionality with maximal compression. For example, most conditional probabilities are calculated based on assuming causality moves forward in time, but to the extent to which our universal laws may be invertible, this is not necessarily a safe assumption to make. Thus Sciops has yet to precisely define the arrow of time and the present structurally.

Duration

Our structure of time establishes the requirements for persistence, change and motion, and a bound unified experience over the path of change (via detection HORs). However none of these structures necessarily define the duration of each path in terms of the amount of time that unfolds. In other words, we have defined for example how some structure y must change as x (time) changes, but not the duration of x.

The answer to this question is outside of the scope of this chapter and involves complicated questions regarding the boundaries of personal identity and the Degree of Existence (DoE) of different emergent structures. These topics will be discussed in length in future posts and are extremely consequential topics for the fields of ethics, metaphysics, and more.

Heuristic 1: Falsifiable Experimental Predictions

An entire future post will be dedicated to future experiments to be conducted by Sciops Labs in regards to space, time, vision and sound, however we can introduce a few ideas here especially relevant to space and time just to give the reader an idea of the direction we are heading.

In section 2.2.5 we discussed frame rates and hypothesized that neurons in the brain are likely much more selective to spatial object numerosity than temporal numerosity despite early neuronal changes (phase locking) to temporal numerosity. The latter can be investigating by looking for any downstream neurons (V4, V5, etc) that are selective to input frame rate. We would not expect to find strong selectivity here, and if we did we would not expect this signal to be propagated in high resolution to deeper layers. Thus a search for such a structure could provide counter evidence to our hypothesis.

Neural Decoding research in general can give us insight into the mathematical structures in the brain, especially in regards to their information content. It’s worth prefacing before detailing any decoding research however that full brain neural decoding allows the model to in a sense cheat: meaning the model can potentially access any information in the brain, including information that is lost to deeper emergent reference frames. And of course decoding relies on low resolution imagine techniques which also introduce their own error in terms of level of detail. So these caveats must be carefully considered always.

A 2021 experiment (King & Wyart 2021) tested the neural decode-ability of a sequence of visual stimuli and generally found that images could be decoded for up to approximately one second, even if each image was flashed at a shorter time scale and in succession with other images, but that doing so required decoding a different signal at each time stamp (ie the signal encoding past images remained decodable but changed over time). In this experiment, they used a sequence of persistent but changing image objects (ex vertical lines) with random rotation transformations applied to it over time (to create subsequent images). Since the same type of image object was used, Sciops would predict the experience created here would largely be of one persistent object that changes over time but according to a series of separate detection HORs (since the rotations are random and not cohesive).

A future Sciops study that would be especially revealing regarding space and time would be to test what happens if 1) the sequence of images has structured motion-based transformation in addition to random transformation (rotations, shifts, etc consistent with a movement) and 2) the sequence of images uses different and non-consistent objects (ex different shapes or visual scenes).

Sciops would make several interesting predictions here, which we will dive into great detail in a future post, but a few notable ones are:

Prediction 1: Scenario A will be more likely to trigger one simple detection HOR than Scenario B (which may require multiple detection HORs) leading to a cleaner and easier to decode signal. Furthermore we would expect Scenario A to potentially trigger a transformation HOR and not scenario B. The same might be true of C and D, especially from motion detectors invariant to object detail. Thus decoding signals could be used here to identify the locations of detection and transformation HORs.

Prediction 2: If shapes are different enough, Scenarios C and D will not be reconstructable from temporal detection HORs (which can help reconstruct how an object changes over time when paired with the current object), but the images could still be decoded via working memory, and therefore decoding would be more dependent on these regions (IPS, PFC, IT, LOC, etc) and the reconstruction accuracy curve would more so match the time scale of working memory. Blinding the decoding of these areas could make them much harder to reconstruct than the images in Scenarios A and B. And we would expect for persistent object transformations (Scenario A especially) that reconstruction accuracy would drop more in line with detection HOR time scales, which we would also predict to match the timescales of bound temporal experiences.

Prediction 3: Object invariant temporal transformations (ex object invariant rotations, spatial shifts, etc) should be detectable across scenarios A and C better than B and D, but differences between A and C may be relatively small. However object specific temporal transformations (stretching, bending, morphing, defecting, etc) should be far more reconstructable in A than C. We would predict this because we would expect A and C to trigger single HORs describing their object invariant movement, but only A to trigger shape specific detection HORs.

Prediction 4: Accuracy of object recognition tasks (performed by subjects) would likely also follow the above predictions. For example, object specific transformations should be most detectable consciously by participants in Scenario A.

Illustrative example of predicted decode-ability of a sequence of shapes in Scenario D, where “Current shape” represents the most recently flashed image in the sequence, “Trajectory / HOR” refers to object invariant HORs, and “Exact prior shape” refers to reconstructing the full past shapes including object specific transformations:

Importantly, in Scenario A we would expect the green line accuracy to remain much higher for longer.

Sciops Labs will also conduct various behavioral experiments related to space and time. For example, visual scenes can be manipulated such that a single scene is presented either E) together as an image or F) in separated chunks over time as several images, and a set of visual frames representing a scene evolving over time can be presented as G) a movie showing the frames over time or H) a comic book layout showing each frame at once and spatially separated.

Sciops can then make various predictions about scene detection accuracy such as: G will better detect changes to persistent objects (ex rotation, movement, stretch, etc) than H, where as the quantity of transformations applied (number of frames) will be better estimated in Scenario H. Similarly, E should have better accuracy over the distances between objects and overall count of objects, whereas F may have better accuracy in identifying which objects are identical, which are different, and which are transformed versions of other objects.

Scenarios E and F are particularly interesting because as they have been described above, F can be thought of as filling out a full coherent scene over time, thus allowing the full scene to actually be perhaps experienced fully spatially (if shown in quick enough succession) or partially spatially via working visual memory (meaning that we may experience a full unified scene but with lower resolution pockets based on information loss across working memory). Another variable we could modify to test these relationships is whether or not the sequence of images in F actually starts to contain some contradictory information not congruent with a single unified visual scene. Adding in these components, we would predict, would not deteriorate performance on temporal HOR related tasks (detecting change) but would deteriorate performance on spatial tasks (object distances and directions, object quantities, etc). These predictions may be particularly non obvious and revealing.