Lifeboat Foundation

How to use causal influence diagrams to recognize the hidden incentives that shape an AI agent’s behavior.

There is rightfully a lot of concern about the fairness and safety of advanced Machine Learning systems. To attack the root of the problem, researchers can analyze the incentives posed by a learning algorithm using causal influence diagrams (CIDs). Among others, DeepMind Safety Research has written about their research on CIDs, and I have written before about how they can be used to avoid reward tampering. However, while there is some writing on the types of incentives that can be found using CIDs, I haven’t seen a succinct write up of the graphical criteria used to identify such incentives. To fill this gap, this post will summarize the incentive concepts and their corresponding graphical criteria, which were originally defined in the paper Agent Incentives: A Causal Perspective.

A causal influence diagram is a directed acyclic graph where different types of nodes represent different elements of an optimization problem. Decision nodes represent values that an agent can influence, utility nodes represent the optimization objective, and structural nodes (also called change nodes) represent the remaining variables such as the state. The arrows show how the nodes are causally related with dotted arrows indicating the information that an agent uses to make a decision. Below is the CID of a Markov Decision Process, with decision nodes in blue and utility nodes in yellow:

Continue reading “Spotting Unfair or Unsafe AI using Graphical Criteria” »

The liver appears to play a significant role in regulating feeding behaviors in mice.

A new generation of scientists is growing up with CRISPR technology. Here’s how some high school students learn to edit genes.

Click on photo to start video.

Click on photo to start video.

Linear scaling is often difficult to achieve because the communication required to coordinate the work of the cluster nodes eats into the gains from paralleliza… See more.

A new distributed-training library achieves near-linear efficiency in scaling from tens to hundreds of GPUs.

A Google engineer’s claim that the LaMDA program is sentient underscores an urgent need to demystify the human condition.

Click on photo to start video.

The ability to change your mind is a key part of being a social creature. Reasoning by oneself is a shallow endeavor.