Experiment Notebook: Model Validation

Experiment Notebook: Model Validation#

Table of Contents#

Experiment Summary#

The purpose of this notebook is to recreate selected scenario analyses from the (widely acknowledged) Hoban/Borgers Ethereum 2.0 Economic Model using the CADLabs model, and to compare the results.

Analysis 1, “Profit Yields Across Validator Environments”, plots the average profitability of validators across different validator environments in two different adoption (i.e. total ETH staked) scenarios and a wide range of ETH prices. A description of the different validator environments and respective assumptions can be found in the assumptions document.

Analysis 2, “Network Yields and Network Inflation”, combines the simulation of average (i.e. not validator-environment specific) Revenue Yields, Profit Yields (across two illustrative ETH price levels), with the associated overall network inflation.

Analysis 3, “Revenue/Profit Yield Spread “Three Region Analysis””, compares the spread between average (i.e. not validator-environment specific) Revenue Yields and Profit Yields across a wide range of ETH prices, and across the two adoption scenarios seen earlier in this notebook.

These analyses and illustrative insights will be described in further detail in their corresponding sections.

Experiment Assumptions#

Our model adopts a range of assumptions from the Hoban/Borgers Ethereum 2.0 Economic Model (notably all validator cost assumptions across validator environments). However, because the Hoban/Borgers Model was published pre-Altair and the CADLabs model post-Altair, some assumptions differ slightly (notably validator incentive parameters, and rewards such as the new sync committee reward). Hence, rather than a perfect match of simulation results, we expect a very close match, which will serve the purpose of a sanity check.

See assumptions document for further details.

Experiment Setup#

We begin with several experiment-notebook-level preparatory setup operations:

  • Import relevant dependencies

  • Import relevant experiment templates

  • Create copies of experiments

  • Configure and customize experiments

Analysis-specific setup operations are handled in their respective notebook sections.

import setup

import copy
import logging
import numpy as np
from radcad import Engine

import experiments.notebooks.visualizations as visualizations
from experiments.run import run
from model.types import Stage

# Enable/disable logging
logger = logging.getLogger()
logger.disabled = True

# Import experiment templates
from experiments.default_experiment import experiment, TIMESTEPS, DELTA_TIME

# Create a copy of the simulation
simulation = copy.deepcopy(experiment.simulations[0])

# Override default-experiment System Parameters
# using assumptions from Hoban/Borgers Economic Report
simulation.model.params.update({
    "stage": [Stage.BEACON_CHAIN],
    "daily_pow_issuance": [0],  # ETH
    # Combine the validator internet (99.9%), power (99.9%), and technical (98.2%) uptime
    # from Hoban/Borgers Report
    "validator_uptime_process": [lambda _run, _timestep: 0.999 * 0.999 * 0.982],  # Percentage (unitless)
})

# Override default-experiment State Variables
# using assumptions from Hoban/Borgers Economic Report
simulation.model.initial_state.update({
    "eth_supply": 112_000_000,  # ETH
    "eth_price": 25,  # USD/ETH
    "eth_staked": 524_288,  # ETH
    "number_of_validators": 16_384,  # Unitless
})

# Set runs to number of ETH price / staked samples
simulation.runs = 50
# Run single timestep, set unit of time `dt` to multiple epochs
# (see 0_README.ipynb for further details)
simulation.timesteps = 1
simulation.model.params.update({"dt": [TIMESTEPS * DELTA_TIME]})

# Drop state history substeps to improve performance
# (see 0_README.ipynb for further details)
simulation.engine = Engine(drop_substeps=True)

Analysis 1: Profit Yields Across Validator Environments#

The below analysis from the Hoban/Borgers Ethereum 2.0 Economic Model simulates how the average annual-validator profitability varies across validator environments (deployment type) and ETH price ranges. The first analysis simulates the original Beacon Chain minimum requirement of 524,288 ETH staked, the second analysis a much higher adoption level at 33,6m ETH staked. Insights include:

  • The average annual-validator profitability across all validator environments is much lower in the high-adoption scenario due to systematically lower revenue yields

  • As ETH approaches very low price levels, a “profitability cliff” exists for all non-StaaS (Staking-as-a-Service) validator environments in both adoption scenarios (assumes that StaaS providers offer constant ETH returns)

  • Average annual-validator profitability between validator environments converge as adoption and ETH price grow, due to decreasing relevance of operational costs

In a next step, we will recreate the below analysis using the CADLabs model and compare the results.

Annualized Model - Profit Yields of Validator Environments at 524_288 ETH Staked | Annualized Model - Profit Yields of Validator Environments at 33_600_000 ETH Staked

  • | - | alt | alt |

Configuration#

# Create a copy of the simulation
simulation_1 = copy.deepcopy(simulation)

# Create a range of 50 discrete ETH price values starting at
# 25 USD/ETH and ending at 1500 USD/ETH
# Assumption adopted from Hoban/Borgers Economic Report
eth_price_samples = np.linspace(
    start=25,
    stop=1500,
    num=50
)

parameter_overrides = {
    "eth_price_process": [
        # Sample the ETH price values using the run as the index
        lambda run, timestep: eth_price_samples[run - 1]
    ],
    "eth_staked_process": [
        # A sweep of two fixed ETH staked points
        # Assumption adopted from Hoban/Borgers Economic Report
        lambda _run, _timestep: 524_288,  # Beacon Chain genesis ETH staked requirement
        lambda _run, _timestep: 33_600_000,  # 30% of the total ETH supply
    ],
}

# Override default experiment parameters
simulation_1.model.params.update(parameter_overrides)

Execution#

df_1, exceptions = run(simulation_1)

Output Preparation#

df_1
stage timestamp eth_price eth_supply eth_staked supply_inflation network_issuance pow_issuance number_of_validators_in_activation_queue average_effective_balance ... target_reward_eth head_reward_eth block_proposer_reward_eth sync_reward_eth whistleblower_rewards_eth amount_slashed_eth daily_revenue_yields_pct cumulative_revenue_yields_pct daily_profit_yields_pct cumulative_profit_yields_pct
1 2.0 2024-08-14 12:27:49.754941 25.000000 1.121123e+08 524288 0.001017 112258.485611 0.0 0 3.200000e+10 ... 46314.294319 24938.466172 14540.830968 3635.207742 5.0625 40.5 21.419332 21.419332 14.032775 14.032775
3 2.0 2024-08-14 12:27:49.754941 25.000000 1.128986e+08 33600000 0.008140 898570.386846 0.0 0 3.200000e+10 ... 370619.295081 199564.235813 116359.594865 29089.898716 5.0625 40.5 2.674437 2.674437 -3.699895 -3.699895
5 2.0 2024-08-14 12:27:49.754941 55.102041 1.121123e+08 524288 0.001017 112258.485611 0.0 0 3.200000e+10 ... 46314.294319 24938.466172 14540.830968 3635.207742 5.0625 40.5 21.419332 42.838663 17.436153 31.468929
7 2.0 2024-08-14 12:27:49.754941 55.102041 1.128986e+08 33600000 0.008140 898570.386846 0.0 0 3.200000e+10 ... 370619.295081 199564.235813 116359.594865 29089.898716 5.0625 40.5 2.674437 5.348874 -0.296517 -3.996412
9 2.0 2024-08-14 12:27:49.754941 85.204082 1.121123e+08 524288 0.001017 112258.485611 0.0 0 3.200000e+10 ... 46314.294319 24938.466172 14540.830968 3635.207742 5.0625 40.5 21.419332 64.257995 18.434749 49.903678
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
191 2.0 2024-08-14 12:27:49.754941 1439.795918 1.128986e+08 33600000 0.008140 898570.386846 0.0 0 3.200000e+10 ... 370619.295081 199564.235813 116359.594865 29089.898716 5.0625 40.5 2.674437 128.372984 2.421844 96.735391
193 2.0 2024-08-14 12:27:49.754941 1469.897959 1.121123e+08 524288 0.001017 112258.485611 0.0 0 3.200000e+10 ... 46314.294319 24938.466172 14540.830968 3635.207742 5.0625 40.5 21.419332 1049.547252 20.156730 968.060290
195 2.0 2024-08-14 12:27:49.754941 1469.897959 1.128986e+08 33600000 0.008140 898570.386846 0.0 0 3.200000e+10 ... 370619.295081 199564.235813 116359.594865 29089.898716 5.0625 40.5 2.674437 131.047421 2.424059 99.159450
197 2.0 2024-08-14 12:27:49.754941 1500.000000 1.121123e+08 524288 0.001017 112258.485611 0.0 0 3.200000e+10 ... 46314.294319 24938.466172 14540.830968 3635.207742 5.0625 40.5 21.419332 1070.966583 20.158856 988.219146
199 2.0 2024-08-14 12:27:49.754941 1500.000000 1.128986e+08 33600000 0.008140 898570.386846 0.0 0 3.200000e+10 ... 370619.295081 199564.235813 116359.594865 29089.898716 5.0625 40.5 2.674437 133.721858 2.426186 101.585636

100 rows × 157 columns

Analysis Results#

The below plots recreate Hoban/Borgers’ analysis using the same validator adoption levels (524,288 ETH to 33,6m ETH) and cost assumptions. The profit yields across validator environments and adoption levels match very closely (e.g. at the 524,288 ETH staked adoption level and assuming an ETH price of 500 USD/ETH, profit yields for a StaaS validator are shown at around 18-19%; those were profitable days. We conclude that the model is valid for this specific analysis.

visualizations.plot_validator_environment_yields(df_1.copy())

Analysis 2: Network Yields and Network Inflation#

The below analysis from the Hoban/Borgers Ethereum 2.0 Economic Model combines the simulation of average (i.e. not validator-environment specific) Revenue Yields, Profit Yields (across two illutrative ETH price levels), with the associated overall network inflation. Illustrative insights include:

  • Both Revenue Yields and Profit Yields decrease systematically as adoption grows (base-reward decreases at square root of number of validators, hence yields decrease)

  • Profit Yields at the 25 USD/ETH price level are lower (in fact negative as adoption grows) than at the 1500 USD/ETH price level (decreasing relevance of operational costs)

  • Network issuance is expected to stay below 1% per year (and in fact turn negative after EIP-1559 implementation)

In a next step, we will recreate the below analysis using the CADLabs model and compare the results.

profit_yield_at_1500_dollars = 20.48
profit_yield_at_25_dollars = 14.39

# Difference between profit yields for each ETH price scenario
profit_yield_at_1500_dollars - profit_yield_at_25_dollars

6.09

Annualized Model – Rev Yields vs. Network Inflation ../../_images/experiment_model_validation_scenario_2.png

Configuration#

# Create a copy of the simulation
simulation_2 = copy.deepcopy(simulation)

# Create a range of 50 discrete ETH staked values starting at
# the Beacon Chain genesis requirement of 524_288 ETH staked and ending at
# 30% of the ETH supply (33_600_000 ETH staked at time of report)
# Assumption adopted from Hoban/Borgers Economic Report
eth_staked_samples = np.linspace(
    524_288,
    33_600_000,
    50
)

parameter_overrides = {
    "eth_staked_process": [
        # Sample the ETH staked values using the run as the index
        lambda run, timestep: eth_staked_samples[run - 1],
    ],
    "eth_price_process": [
        # A sweep of two fixed ETH price points
        # Assumption adopted from Hoban/Borgers Economic Report
        lambda _run, _timestep: 25,
        lambda _run, _timestep: 1500,
    ],
}

# Override default experiment parameters
simulation_2.model.params.update(parameter_overrides)

Execution#

df_2, exceptions = run(simulation_2)

Output Preparation#

df_2
stage timestamp eth_price eth_supply eth_staked supply_inflation network_issuance pow_issuance number_of_validators_in_activation_queue average_effective_balance ... target_reward_eth head_reward_eth block_proposer_reward_eth sync_reward_eth whistleblower_rewards_eth amount_slashed_eth daily_revenue_yields_pct cumulative_revenue_yields_pct daily_profit_yields_pct cumulative_profit_yields_pct
1 2.0 2024-08-14 12:27:49.754941 25 1.121123e+08 5.242880e+05 0.001017 112258.485611 0.0 0 3.200000e+10 ... 46314.294319 24938.466172 14540.830968 3635.207742 5.062500 40.500000 21.419332 21.419332 14.032775 14.032775
3 2.0 2024-08-14 12:27:49.754941 1500 1.121123e+08 5.242880e+05 0.001017 112258.485611 0.0 0 3.200000e+10 ... 46314.294319 24938.466172 14540.830968 3635.207742 5.062500 40.500000 21.419332 21.419332 20.158856 20.158856
5 2.0 2024-08-14 12:27:49.754941 25 1.121698e+08 1.199303e+06 0.001538 169801.784246 0.0 0 3.200000e+10 ... 70047.344115 37717.800677 21992.056784 5498.014196 5.062500 40.500000 14.161755 35.581087 7.167142 21.199917
7 2.0 2024-08-14 12:27:49.754941 1500 1.121698e+08 1.199303e+06 0.001538 169801.784246 0.0 0 3.200000e+10 ... 70047.344115 37717.800677 21992.056784 5498.014196 5.062500 40.500000 14.161755 35.581087 13.293189 33.452045
9 2.0 2024-08-14 12:27:49.754941 25 1.122122e+08 1.874317e+06 0.001923 212249.407439 0.0 0 3.200000e+10 ... 87554.361942 47144.656430 27488.558258 6872.139564 5.062500 40.500000 11.326254 46.907340 4.484767 25.684684
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
191 2.0 2024-08-14 12:27:49.754941 1500 1.128798e+08 3.224997e+07 0.007969 879764.701591 0.0 0 3.199999e+10 ... 362863.113639 195387.830421 113924.464957 28481.116239 5.062498 40.499982 2.728081 240.663228 2.476932 222.683485
193 2.0 2024-08-14 12:27:49.754941 25 1.128881e+08 3.292499e+07 0.008045 888087.049492 0.0 0 3.199999e+10 ... 366295.567148 197236.074618 115002.117699 28750.529425 5.062499 40.499991 2.697427 243.360655 -3.678148 -75.046477
195 2.0 2024-08-14 12:27:49.754941 1500 1.128881e+08 3.292499e+07 0.008045 888087.049492 0.0 0 3.199999e+10 ... 366295.567148 197236.074618 115002.117699 28750.529425 5.062499 40.499991 2.697427 243.360655 2.447934 225.131419
197 2.0 2024-08-14 12:27:49.754941 25 1.128986e+08 3.360000e+07 0.008140 898570.386846 0.0 0 3.200000e+10 ... 370619.295081 199564.235813 116359.594865 29089.898716 5.062500 40.500000 2.674437 246.035092 -3.699895 -78.746372
199 2.0 2024-08-14 12:27:49.754941 1500 1.128986e+08 3.360000e+07 0.008140 898570.386846 0.0 0 3.200000e+10 ... 370619.295081 199564.235813 116359.594865 29089.898716 5.062500 40.500000 2.674437 246.035092 2.426186 227.557605

100 rows × 157 columns

Analysis Results#

The below plot recreates Hoban/Borgers’ analysis. The Profit Yields in each ETH price scenario differ slightly between the Hoban/Borgers and the CADLabs model – likely due to the Altair updates – whereas the annualized inflation rates match very closely, within 0.01 of a percent. We conclude that the model is valid for this specific scenario analysis.

visualizations.plot_revenue_yields_vs_network_inflation(df_2)

# Difference between profit yields for each ETH price scenario
df_2.query('subset == 1').iloc[0]['total_profit_yields_pct'] - df_2.query('subset == 0').iloc[0]['total_profit_yields_pct']

6.2153247625

# Minimum and maximum annualized inflation rate
(df_2.query('subset == 0')['supply_inflation'] * 100).describe()

count    50.000000
mean      0.547942
std       0.191176
min       0.101691
25%       0.416338
50%       0.579630
75%       0.705919
max       0.813983
Name: supply_inflation, dtype: float64

Analysis 3: Revenue/Profit Yield Spread (“Three Region Analysis”)#

The below “Three Region Analysis” from the Hoban/Borgers Ethereum 2.0 Economic Model compares the spread between average (i.e. not validator-environment specific) Revenue Yields and Profit Yields across a wide range of ETH prices, and across the two adoption scenarios seen earlier in this notebook. Illustrative insights include:

  • As ETH approaches very low price levels, a “profitability cliff” exists in both adoption scenarios, as operational costs squeeze validators’ margins

  • The Revenue/Profit Yield Spread is smaller in the high-adoption scenario as Revenue decreases, but costs remain constant in dollars

In a next step, we will recreate the below analysis using the CADLabs model and compare the results.

Annualized Model - Revenue/Profit Yield Spread at 524_288 ETH Staked | Annualized Model - Revenue/Profit Yield Spread at 33_600_000 ETH Staked

  • | - | alt | alt |

Configuration#

# Create a copy of the simulation
simulation_3 = copy.deepcopy(simulation)

# Create a range of 50 discrete ETH price values starting at
# 25 USD/ETH and ending at 1500 USD/ETH
# Assumption adopted from Hoban/Borgers Economic Report
eth_price_samples = np.linspace(
    start=25,
    stop=1500,
    num=50
)

parameter_overrides = {
    "eth_price_process": [
        # Sample the ETH price values using the run as the index
        lambda run, _timestep: eth_price_samples[run - 1]
    ],
    "eth_staked_process": [
        # A sweep of two fixed ETH staked points
        # Assumption adopted from Hoban/Borgers Economic Report
        lambda _run, _timestep: 524_288,  # Beacon Chain genesis ETH staked requirement
        lambda _run, _timestep: 33_600_000,  # 30% of the total ETH supply
    ],
}

# Override default experiment parameters
simulation_3.model.params.update(parameter_overrides)

Execution#

df_3, exceptions = run(simulation_3)

Output Preparation#

df_3
stage timestamp eth_price eth_supply eth_staked supply_inflation network_issuance pow_issuance number_of_validators_in_activation_queue average_effective_balance ... target_reward_eth head_reward_eth block_proposer_reward_eth sync_reward_eth whistleblower_rewards_eth amount_slashed_eth daily_revenue_yields_pct cumulative_revenue_yields_pct daily_profit_yields_pct cumulative_profit_yields_pct
1 2.0 2024-08-14 12:27:49.754941 25.000000 1.121123e+08 524288 0.001017 112258.485611 0.0 0 3.200000e+10 ... 46314.294319 24938.466172 14540.830968 3635.207742 5.0625 40.5 21.419332 21.419332 14.032775 14.032775
3 2.0 2024-08-14 12:27:49.754941 25.000000 1.128986e+08 33600000 0.008140 898570.386846 0.0 0 3.200000e+10 ... 370619.295081 199564.235813 116359.594865 29089.898716 5.0625 40.5 2.674437 2.674437 -3.699895 -3.699895
5 2.0 2024-08-14 12:27:49.754941 55.102041 1.121123e+08 524288 0.001017 112258.485611 0.0 0 3.200000e+10 ... 46314.294319 24938.466172 14540.830968 3635.207742 5.0625 40.5 21.419332 42.838663 17.436153 31.468929
7 2.0 2024-08-14 12:27:49.754941 55.102041 1.128986e+08 33600000 0.008140 898570.386846 0.0 0 3.200000e+10 ... 370619.295081 199564.235813 116359.594865 29089.898716 5.0625 40.5 2.674437 5.348874 -0.296517 -3.996412
9 2.0 2024-08-14 12:27:49.754941 85.204082 1.121123e+08 524288 0.001017 112258.485611 0.0 0 3.200000e+10 ... 46314.294319 24938.466172 14540.830968 3635.207742 5.0625 40.5 21.419332 64.257995 18.434749 49.903678
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
191 2.0 2024-08-14 12:27:49.754941 1439.795918 1.128986e+08 33600000 0.008140 898570.386846 0.0 0 3.200000e+10 ... 370619.295081 199564.235813 116359.594865 29089.898716 5.0625 40.5 2.674437 128.372984 2.421844 96.735391
193 2.0 2024-08-14 12:27:49.754941 1469.897959 1.121123e+08 524288 0.001017 112258.485611 0.0 0 3.200000e+10 ... 46314.294319 24938.466172 14540.830968 3635.207742 5.0625 40.5 21.419332 1049.547252 20.156730 968.060290
195 2.0 2024-08-14 12:27:49.754941 1469.897959 1.128986e+08 33600000 0.008140 898570.386846 0.0 0 3.200000e+10 ... 370619.295081 199564.235813 116359.594865 29089.898716 5.0625 40.5 2.674437 131.047421 2.424059 99.159450
197 2.0 2024-08-14 12:27:49.754941 1500.000000 1.121123e+08 524288 0.001017 112258.485611 0.0 0 3.200000e+10 ... 46314.294319 24938.466172 14540.830968 3635.207742 5.0625 40.5 21.419332 1070.966583 20.158856 988.219146
199 2.0 2024-08-14 12:27:49.754941 1500.000000 1.128986e+08 33600000 0.008140 898570.386846 0.0 0 3.200000e+10 ... 370619.295081 199564.235813 116359.594865 29089.898716 5.0625 40.5 2.674437 133.721858 2.426186 101.585636

100 rows × 157 columns

Analysis Results#

The below plots recreate Hoban/Borgers’ analysis. The Revenue/Profit Yield Spread for our model closely matches the Hoban/Borgers Ethereum 2.0 Economic Model for both scenarios. We can conclude that the model is valid for this specific scenario analysis.

visualizations.plot_three_region_yield_analysis(df_3.query('subset == 0').copy())

visualizations.plot_three_region_yield_analysis(df_3.query('subset == 1').copy())
Contents