Abstract - TLDR

This project will develop a system that emulates intuitive scientific discovery to discover causal models of everyday devices. The system will drive a real robot to experiment with real-world devices in order to build models of those devices. This project introduces several complexities over the original Autumn project, and will demand advances in perception, representation, and reasoning.

Background

In order to carry out everyday tasks, humans have to learn to interact with a wide variety of devices — light switches, plug-sockets, locks, traffic lights, microwaves, ATMs, telephones, and so on. To do this successfully relies on using mental models, which allow us to make predictions, infer latent causes, assess counterfactuals, and ultimately zmake decisions and act upon them. Many of the mechanisms we encounter are novel to us, and hence learning through observation, experimentation, and instruction is necessary.

The process by which children and adults alike discover models of the mechanisms in their environments resembles the procedures that scientists use to discover the laws and properties of phenomena in the world. Just like scientists, all people construct hypotheses, engage in active experimentation, try to discern causal links from spurious correlations, revise beliefs in light of evidence, gravitate towards explaining conflicting or contradictory facts, and seek coherence between theories. In recent years, there have been several important strides in framing the process of intuitive model discovery in concrete computational terms A compelling paradigm is to draw from the framework of generative modelling and formulate model discovery as a form of probabilistic inference. the space over which inference is performed in scientific discovery are spaces of theories or models themselves. And yet, there remains a large gap between a three-year-old’s capability to infer causal intuitive models of the world, and state-of-the-art methods. While some of these gaps will be addressed with more effective search and inference procedures within the paradigms of model-program discovery, there remain some fundamental shifts to be bridged.

The primary gap that this project seeks to resolve is that human knowledge of phenomena takes multiple multiple forms and representation. Human knowledge is broad, spanning a multitude of different domains, and deep, describing phenomena from multiple perspectives and degrees of abstraction. Human reasoning is able to span both this breadth and depth. For example, an epidemiologist may predict how a mutation in SARS-CoV-2’s morphology will likely affect the border restrictions or long term economic growth of a country. In contrast, formal computational models, including probabilistic programs, tend to be narrow and shallow, focusing on only one domain in isolation to a single degree of abstraction. This is the primary foundational gap between the current paradigms of modelling and inference, and machines that could, for example, infer that the assassination of Franz Ferdinand was the probable cause of World War I; that an experiment for X should control for Y; or that an unreliable water supply is ultimately caused by chronic government underinvestment.

Rather than a single isolated model, this project aims to construct representations of interconnected networks of models, whereby multiple models of the same thing can both coexist and interact by virtue of sharing structure. Such polystructural models will be incrementally built, incorporating more and more types and sources of knowledge.

Objectives

In short, the aim of this project is to develop a system that drives a robot to experiment with a real world physical device in order to build a general understanding of that device in the form of a polystructural model. and to formulate and solve the variety of different problems necessary to achieve this within a general framework of causal probabilistic programming.

This project focuses on different areas that are key to advancing automated scientific discovery. It aims to develop a holistic system capable of automating the process of scientific discovery by integrating improvements in the following areas:

• Polystructural Model Representations: A significant goal is to design and implement polystructural models. These representations aim to encapsulate phenomena at various abstraction levels. We plan to develop probabilistic reasoning systems that can synthesize these models, and active inference strategies that mirror scientific processes to generate data required for model induction.
• Reasoning: This project seeks to address diverse reasoning types, each defined by the data and model modalities and their computational demands and execution speed. Our aim is to unify these different types of reasoning within a single probabilistic programming system. The reasoning process is broken down into:
  • Perception: The objective here is to develop systems capable of accurate and efficient environment perception and interpretation, a crucial foundation for subsequent stages of model synthesis and active inference.
  • Model Synthesis as Program Discovery: The project targets the creation of probabilistic reasoning systems proficient in synthesizing models, treating model synthesis akin to program discovery. Such systems will be designed to formulate and refine hypotheses, mirroring the process scientists use to discover and enhance scientific theories.
  • Active Inference: This part of the project deals with developing systems for active inference. Using principles of Bayesian optimal experiment design, we plan to control a robot to conduct experiments and gather data. This process will allow the system to learn about the world in a manner similar to how scientists carry out experiments to test their theories and build their understanding of the world.
• Evaluation: Constructing effective evaluation metrics is a critical and nontrivial part of this project. This is both a scientific project in of it self (what does it mean to have the right model?) as well a design and engineering challenge.

By developing and integrating these elements, the project aims to create a system that can emulate the process of scientific discovery, providing a solid base for subsequent projects aimed at fully or partially automated scientific discovery.

Scope

Describe the boundaries of the project. What are the limitations and constraints of the project? What is not included in the project scope?

The scope of this project encompasses the design and development of a new programming language and synthesis algorithm, the selection of a suitable real-world testing environment, and the creation of an evaluation methodology.

Justification

Theoretical Impact

The central organizational goal for Basis is to help solve intractable scientific and societal problems. Emulating intuitive science, as a precursor to future projects potential projects that implement actual automated or semi-automated scientific discovery. real science, furthers that aim in the sense that

The main technical contributions to core technology that solving this project will demand include:

• Language features for representation of polystructural models: Many of the domains we would like to perform reasoning have the feature that they contain extremely complex phenomena that can and have beem modeled in a variety of different ways, often by different teams of experts making different assumptions. While there are domain-specific mechanisms in use to combine such models, such as in coupled climate models, there are no general systems that apply to several domains.
• Formulating the problem of scientific discovery: As elaborated in Autumn: Formalize General Problem Statement and Semantic Priors, progress on formalizing scientific discovery as a computational problem will be a major advance that is necessitated by this project.
• Probabilistic reasoning over programs and structured objects in general: Programs, when viewed purely as objects of syntax, share many properties with other structured objects, such as graphs and X. Probabilistic inference over these objects has historically been very challenging. See Autari: Probabilistic Inference
• Unified Heterogeneous Reasoning: This project involves various types of reasoning, differentiated by the types of objects involved and the speed at which inference needs to be executed. For example, perception and active inference (like controlling a robot arm)
  demand real-time decisions, while the formulation of higher-level experiments or inference processes related to discovering causal laws can happen at a slower pace. Unifying these disparate forms of reasoning within a single system will be a challenging but important part of this project.
• Nested inference and nesting of inference and optimisation: Several components of this project, such as the active experimentation and even the basic, passive, synthesis problem itself are most faithfully represented as computational problems that mix, and nest inference queries. For instance, active experimentation can be modelled within the framework of Bayesian pptimal experiment design, which requires the optimization of a distributional quantity (like information gain) conditional on the data (posterior inference). There are a wide of problems in other domains that currently are challenging for probabilistic programming systems to both represent and solve.

The following sections flesh out the areas of theoretical impact in more detail.

A Language for Polystructural Dynamical Models

See: Polystructural Models

A primary objective is to construct Autumn 2 (see Autumn 2 Language Design Document (WIP)— a language designed to represent models that the system synthesizes from phenomena in the world. A hallmark of these phenomena is their capacity to be understood at varying levels of abstraction. As an illustration, consider a light switch: one could represent it merely as an on/off binary system or delve into a more nuanced representation of its internal circuitry. The aim of Autumn 2 is to provide the flexibility and expressive power to capture such multiplicity of abstraction.

Understanding and developing systems for polystructural modelling extends beyond this project. Possible uses cases include automated model refinement, a process that requires a computational system to understand what its current model represents about reality and consequently revise it. Another use case in supporting the incremental and distributed construction of large libraries or repositories of models, generalizing the state of the art for example in climate modelling.

By polystructural, we mean a language that is capable of representing multiple models of the same phenomena, the relationship between such models, and the relationship between each model and the phenomena that it is a model of.

Perception

Perception, in the context of this project, refers to the task of inferring the nature of objects and their properties from sensory input, primarily high-resolution image data from RGB, mono, and depth cameras.

Basic Perception

In a simple form of this perception task, the problem the goal is to identify objects and their properties over time. This is similar to the task in Autumn 1, where the aim was to map observational data to 2D sprites, except now we deal with a more complex 3D space.

We can formulate this as a problem of probabilistic inference. Namely, given a prior over scenes, construct There are both computational and conceptual challenges:

1. sim-to-real gap: a discrepancy between the simulated or idealized models and the actual real-world data. Our model’s representation of reality may be more impoverished than the real world, so we must account for this in our inference processes.
2. Sources of prior knowledge: Currently, it’s unclear how to arrive at meaningful priors over scene structures. Specifying manually seems arbitrary, and learning from data is challenging due to the fact that one does not directly collect data from these latent scene variables.

Advanced Perception

The basic formulation has several shortcomings:

1. There’s an cut between what counts as perception and the rest of the synthesis process that is based on intuition and not justified
2. Being prescriptive about exactly what the representation of what the target representation should be may (i) lead to sub-optimal models, (ii) miss something fundamental about the modelling process. For example, if we presuppose the existence of objects, will this prevent us from modelling fluids well, short of us going in to make a language change to accommodate.

Synthesizing Polystructural Causal Dynamics

(see Autari: Probabilistic Inference)

Active Exploration and Control

Active exploration is a key aspect of this project, where the robot should actively interact with the world to learn about its underlying structures and mechanisms. This includes deciding on high-level experiments (such as pressing a button) as well as low-level motor commands to perform the action.

This task presents significant challenges due to the need for accurate perception, control, and learning capabilities.

Challenges in Active Exploration

1. Model-Experiment Coupling: There’s a necessary coupling between the current model of the world and possible experiments. For instance, the robot needs to hypothesize or assume the existence of buttons before proposing an experiment to see what the next button does.
2. Robotic Control: Controlling a robot to perform precise actions in the real world is a non-trivial problem due to the complexities in perception, planning, and actuation.
3. Bayesian Experiment Design: Although Bayesian formulations of experimental design exist, they often make simplifying assumptions about the model and the world that may not hold in practice.

Control and Bayesian Optimal Experiment Design

Control plays a crucial role in active exploration, with Bayesian optimal experiment design offering a potential approach. However, these methodologies have their shortcomings, primarily due to their simplifying assumptions and computational complexity.

Hardware for Active Exploration

To perform active exploration in the real world, we will use Panda Robots from Franka Emika. The Franka Emika Panda robot is a 7-axis robot arm that offers a 3 kg payload and 850 mm of reach, with high precision and advanced torque control capabilities.

However, there might be challenges with the robot’s default gripper’s precision, so we might want to consider alternatives such as the Valve Knuckles or the Trossen ALOHA KIT.

The Trossen ALOHA KIT offers benefits for bimanual teleoperation, which can be helpful for passive data collection. Passive data collection involves a human interacting with the device through the robot, which can be beneficial for gathering initial observational data.

Some considerations

• Active and Passive Data Collection: While the ultimate goal is for the robot to actively experiment and learn a model for the world, we will initially need to start with passive data collection. This involves using human-controlled interactions to gather data.
• Potential Challenges and Considerations: One potential challenge is that many domains with interesting polystructure are difficult or dangerous to manipulate, such as circuit boards, hardware tools, and chemistry sets.
• Distraction: Additionally, there may be a concern about the potential distraction of focusing too much on the robotics aspect, diverting from the core problem of active model discovery. Therefore, it’s essential to strike a balance between these intertwined aspects to ensure progress in all areas.

Social impact: Would addressing the challenge be significant, useful, or important? Who stands to gain from success? Who, if anyone, could lose? Is there a clear mechanistic link between technical improvements and project success?

Team fit: Are we and our collaborators able to execute? Are we essential to the success of the project? Do our collaborators have sufficient resources and bandwidth? Are there clear project leaders/points of contact on both sides?

Risk: Would taking on the challenge incur too great or too little technical or organizational risk in the context of our portfolio? Is success overly tied to a single theoretical development or platform feature? Is it a prudent commitment of scarce Basis resources given the opportunity costs?

Growth potential: Could the challenge start small and over time translate progressively larger amounts of resources like money, data, headcount or compute into growing project impact? Would the solution organize and align large amounts of previously disparate knowledge and activity in qualitatively new ways?

Strategy

Connection to current Basis projects and core tech work

Brief description of related Basis projects and core tech work, referencing entries in the where applicable.

Objectives Formalized

The core objectives of this project can be formalized as follows:

1. Autumn 2 Development: Create a programming language that can accurately represent the intricacy and polystructural nature of real-world physical environments.
2. Algorithm Development: Develop an algorithm capable of learning from observations and interactions within these complex environments, thus facilitating the automatic discovery of polystructural models.
3. Environment Selection: Identify a suitable real-world environment that is complex enough to test the capacities of the newly developed tools but not so convoluted as to overwhelm the model discovery process.
4. Evaluation Mechanism and Benchmarking: Design a systematic, quantifiable method of assessing the success of the project, taking into account the lack of a definitive ‘ground truth’ in real-world contexts.

Methodology

Explain the approach and methodology that will be used to achieve the project objectives. What research methods will be used? What data collection and analysis techniques will be used?

Autumn 2 Development: This process will involve a deep exploration of modern video game engines, studying their principles and methodologies in order to extract the best features for Autumn 2. The language will be designed to capture continuous time and space and represent polystructural models, enabling the accurate reflection of real-world dynamics.

Algorithm Development: The creation of a new synthesis algorithm, inspired by the principles behind AutumnSynth, will involve bridging disparate techniques in program synthesis. The algorithm will be designed to attribute each observed change in the environment to other elements or actions, thereby defining a functional mapping and unearthing causal dependencies.

Environment Selection: We will consider environments that offer polystructural elements while ensuring the manipulation task is not excessively challenging. We’ll draw inspiration from concepts like the NIST-Board, while aiming for a higher degree of polystructural complexity.

Evaluation Mechanism and Benchmarking: As there’s no direct ground truth in real-world contexts, we will need to design a unique and efficient evaluation mechanism that allows for an objective assessment of the project’s success.

For evaluation, one inspiration to emulate are NIST. NIST prototype Assembly Task Boards (ATB) are standardized boards designed to test robotic manipulation in different areas.

What do we want from the benchmarks:

• Relatively easy manipulation: Advancing the state of robotic manipulation is not a goal of this project. Hence, we should design manipulation tasks that are within the reach of current methods.
• Clearly differentiated levels of abstractions: Different use cases require different levels

Deliverables

1. Autumn 2 Programming Language: A fully developed language with its compiler capable of capturing complex, polystructural models.
2. Synthesis Algorithm: A novel algorithm designed for automated theory discovery in real-world physical environments.
3. Selected Testing Environment: An appropriate real-world environment where the tools will be tested.
4. Evaluation Methodology: A mechanism to quantify and assess the success of the project in the absence of a definitive ground truth.

Resources

The project will require a blend of human resources and technical tools. Our collaborators at Columbia, MIT, and Stanford will play integral roles. Software tools for programming language design and development, data analysis techniques for the synthesis algorithm, and appropriate physical equipment for setting up the real-world environment will be needed.

Personnel:

Needed Expertise

• Compilation / jit compilation
• Deep knowledge of physics simulation models

Risk Mitigation

Given the pioneering nature of this project, potential risks include unforeseen challenges in language and algorithm development, or difficulties in model discovery within the selected environment. To mitigate these, we will carry out extensive research, iterative development, and testing. Regular check-ins and updates will be scheduled among team members to ensure a timely response to any unforeseen issues.

Evaluation

Despite the lack of a definitive ground truth, we will devise an evaluation mechanism that quantifies the effectiveness of the project. This could include measures such as the accuracy and consistency of the discovered models, the ability of the models to predict outcomes, and the response of the models to changes in the environment. Feedback from all stakeholders will be an integral part of this evaluation process.