# Arm System Characterization Tool

The Arm System Characterization Tool (ASCT) is a standalone command-line utility for running low-level benchmarks, diagnostic scripts, and system tests to analyze and debug performance on Arm-based platforms.

ASCT provides a standardized environment for evaluating key hardware characteristics, such as memory latency and bandwidth, and is especially suited for platform bring-up, system tuning, and architectural comparison tasks. It helps developers and system architects gain early and repeatable insights into performance-critical subsystems.

Current capabilities include:

- System report. Generate a structured inventory of CPU, caches, NUMA topology, operating system, kernel, and tooling state. You can also generate Arm Coherent Mesh Network (CMN) topology and configuration reports on demand. For more information, see System report.

- Memory latency and bandwidth benchmarks. Measure how latency and bandwidth change across cache levels and DRAM. Characterize NUMA effects, including idle latency, loaded latency, core-to-core latency, and peak bandwidth. For more information, see Memory characterization.

- Storage latency and bandwidth benchmarks. Run controlled fio-based parameter sweeps. Vary request size, queue depth, process count, and access pattern. Measure bandwidth, IOPS, latency, and CPU utilization under input and output load. For more information, see Storage characterization.

- System diff. Compare one or more ASCT run output directories. Highlight configuration differences and benchmark differences. Generate CSV and JSON output and plots for supported benchmarks. For more information, see Compare run results.

Planned features include:

- Hardware register inspection and configuration for debug and bring-up use

- Input and output performance testing

- Locking and synchronization stress tests

- Floating-point compute performance measurement in FLOPS

## Documentation

For detailed guidance, see:

- Installation

- Getting started

- System report

- Memory characterization

- Storage characterization

- Compare run results

## Requirements

- ASCT is supported on Linux systems
- Python 3.10 or later. You need `pip` if you want to use a Python virtual environment (options 1 and 2).
- Build tools such as:
  - `gcc`
  - `make`
- If you want to install ASCT as a system-wide tool (option 3), you need:
  - `uv`. Install `uv` from: https://docs.astral.sh/uv/getting-started/installation/.
  - `sudo` privileges
  - Permission to create:
    - `/opt/uv/tools`
    - `/usr/local/bin`

## Download ASCT

1. Download the latest ASCT release bundle from: artifacts.tools.arm.com.

   For users within the Arm network, early-access versions might be available at: artifacts.internal.tools.arm.com.

2. Locate the downloaded file on your host machine. Move it to the target machine where you want to install and run the tool.

3. Extract the release bundle:

       tar -xvzf asct-<version>-release.tar.gz

   This process creates a directory:

       asct-<version>/

   The directory contains the following files:

   - `asct-<version>.tar.gz`: The Python source distribution (the package to install)
   - `asct_docs/`: Generated HTML documentation. Open `asct_docs/index.html` in a web browser.
   - `install.sh`: An optional installer helper (tested on Ubuntu 24.04)
   - `README.txt`: A release bundle summary and quick-start notes

## Install ASCT

You can install ASCT in one of the following ways:

- Option 1 (recommended): Install into a Python virtual environment using the `install.sh` helper script.

- Option 2: Install manually using `pip` into a Python environment.

- Option 3: Install system-wide using `uv` (for shared machines or CI systems).

### Option 1: Install ASCT with the helper script

This is the simplest installation method. It creates a dedicated Python virtual environment and installs ASCT into it.

#### Steps

From inside the extracted `asct-<version>/` directory:

        ./install.sh --install-dir /path/to/install/directory

If `--install-dir` is not specified, the script installs into the directory containing `install.sh`.

The `install.sh` script performs the following tasks:

- Creates a Python virtual environment
- Installs ASCT into that environment using `pip`
- Optionally builds `fio` if required

By default, the virtual environment is created at `<install-dir>/asct_venv`.

After installation, follow the instructions printed by the script to activate the virtual environment before running ASCT.

### Option 2: Install ASCT manually using pip

Use this method in the following cases:

- You prefer to manage Python environments manually

- You are installing into an existing virtual environment

- You automate installation in your own scripts

You can install ASCT into any active Python environment.

#### Install into a new virtual environment

1. Create a virtual environment:

       python3 -m venv asct_venv

2. Activate it:

       source asct_venv/bin/activate

3. Install ASCT from the bundled source distribution:

       python3 -m pip install ./asct-<version>.tar.gz

   ASCT is now installed inside the virtual environment.

#### Install into an existing environment

    python3 -m pip install ./asct-<version>.tar.gz

ASCT is installed into the currently active Python environment.

### Option 3: Install ASCT as a system-wide tool with uv

This method installs ASCT as a system-wide tool available on the system `PATH`.

Use this option only for shared systems, managed environments, or CI environments.

#### Steps

1. Create the required directories:

       sudo mkdir -p /opt/uv/tools /usr/local/bin

2. Install ASCT with `uv` and place the executable in `/usr/local/bin`:

       sudo UV_TOOL_DIR=/opt/uv/tools UV_TOOL_BIN_DIR=/usr/local/bin uv tool install ./asct-<version>.tar.gz

   After installation, ASCT is available on the system `PATH`.

## Runtime tools

Some workloads require additional system tools, such as:

- `numactl`
- `fio` 3.36 or later

If tools are missing, ASCT reports this at runtime.

## Run ASCT

- If you installed ASCT in a virtual environment, activate the environment first.
- If you installed ASCT with `uv`, no activation is required.

To verify the installation, run:

    asct help

You can also open the offline documentation at: `asct_docs/index.html`.

## Uninstall ASCT

Uninstall ASCT as follows.

### Option 1: If you installed ASCT using the helper script

To uninstall ASCT, remove that virtual environment directory:

    rm -rf <install-dir>/asct_venv

### Option 2: If you manually installed ASCT using pip

If you installed ASCT into a dedicated virtual environment, you can remove that environment directory.

Alternatively, activate the relevant Python environment and uninstall ASCT:

    python3 -m pip uninstall asct

### Option 3: If you installed ASCT system-wide using uv

Remove ASCT using `uv`:

    sudo UV_TOOL_DIR=/opt/uv/tools UV_TOOL_BIN_DIR=/usr/local/bin uv tool uninstall asct

# Getting started

You can run ASCT with several commands that follow this pattern:

    asct [command] [arguments]

Command | Description | More information
--- | --- | ---
`asct cmn` | Collects information about the Arm Coherent Mesh Network (CMN) interconnect when it is present on your system | See System report
`asct diff` | Compares results from previous ASCT runs | See Compare run results
`asct list` | Lists available benchmarks | See asct list command
`asct run` | Runs a list of benchmarks based on a provided list of keywords | See asct run command
`asct system-info` | Displays system information and exits. ASCT also includes this information by default with every benchmark run. | See System report

For more information on a specific command, run:

    asct [command] --help

To display the ASCT version, run:

    asct version

To get started, run the default set of benchmarks:

    sudo asct run

> [!IMPORTANT] Some benchmarks require `sudo` or root privileges to configure huge pages and access certain system information. You can run ASCT without `sudo`, but some benchmarks might be unavailable or have limited functionality.

### `asct run` command

The `asct run` command runs one or more benchmarks.

#### Default behavior

    sudo asct run

Running the `asct run` command with no arguments executes a default set of benchmarks and displays the results in the terminal.

Each time you run `asct run`, ASCT collects system information and includes it with the benchmark results. ASCT also saves detailed output in a directory under the current working directory.

The output can include data in multiple formats, such as CSV and JSON. Some benchmarks also generate additional artifacts, such as plots or raw data dumps.

By default, ASCT creates an output directory named: `data.<YYYYMMDD_HHMMSS_microseconds>`.

You can change the output location using the `--output-dir` flag.

#### Select benchmarks

To run all available benchmarks:

    sudo asct run all

To run specific benchmarks, pass a list of benchmark names as arguments to the `asct run` command.

For example, to run the `latency-sweep` and `idle-latency` benchmarks:

    sudo asct run latency-sweep idle-latency

Alternatively, each benchmark also has associated keywords that you can use to select multiple benchmarks that match those keywords.

For example, to run all benchmarks tagged with the `memory` keyword:

    sudo asct run memory

You can exclude benchmarks by prefixing the name or keyword with a caret (`^`).

For example, to run all benchmarks but exclude all of those tagged with the `bandwidth` keyword:

    sudo asct run all ^bandwidth

> [!NOTE] If a benchmark is a dependency for running another benchmark, excluding it has no effect. For example, if you use `^latency-sweep` to exclude the `latency-sweep` benchmark, but another benchmark depends on `latency-sweep`, the `latency-sweep` benchmark still runs.

To view the list of available benchmarks and their keywords, run the following command:

    asct list

#### Output options

The `run` command also supports the following arguments:

- `--format`, `-f` `[stdout, csv, json]`

  - Specify the output format, either to the terminal with `stdout` (default), individual CSV files (`benchmark-name.csv`), or a single combined JSON file (`report.json`)

    For example: `asct run idle-latency --format=json`

- `--log-level`, `-L` `[debug,info,warning,error,critical]`
  - Set the verbosity level (`info` by default).
- `--log-file` `LOG_FILE`
  - Specify the output file for logs, based on the `--log-level` setting. ASCT writes logs to this file and also prints them to standard error `stderr`.
  - If no log messages are generated, ASCT does not create the log file.
- `--output-dir`, `-o` `OUTPUT_DIR`
  - Set the directory for output files. By default, ASCT saves output files in a directory named `data.<YYYYMMDD_HHMMSS_microseconds>` in the current working directory. Use this flag to specify a different directory. You can specify an absolute or a relative path.
- `--force`
  - When the output directory specified by `--output-dir` already exists, ASCT displays an error and quits to avoid overwriting data. Use the `--force` flag to overwrite the output directory if it exists.
- `--quiet`, `-q`
  - Disable all output to `stdout` and `stderr`, including critical errors and log messages. Use `--log-file` to capture and view logs.
- `--no-progress-bar`
  - Disable the animated progress bar. Single-line update messages display instead.

### `asct list` command

Run the following command to get the list of available benchmarks and their associated keywords:

    asct list

# System report

This section describes the system reporting commands in ASCT. Use these commands to view hardware, software, and platform configuration details.

ASCT collects system information each time you run `asct run`. Arm Coherent Mesh Network (CMN) reporting is available as a separate command.

Use these commands:

- `asct system-info`: View hardware, operating system, and network information
- `asct cmn`: View CMN topology and configuration when present

## View system information

Use the `asct system-info` command to generate a report that summarizes the hardware and software configuration of the system.

The report captures CPU, memory, interconnect, operating system, and related platform characteristics. Use this information to understand the benchmark environment and interpret performance results.

### What `asct system-info` collects

The `asct system-info` report provides a structured snapshot of your system. It includes the following information:

- CPU architecture, core count, and CPU model
- Cache hierarchy and cache line size
- NUMA topology
- Interconnect identification
- Memory capacity and theoretical peak bandwidth
- Operating system and kernel details
- Performance tooling and kernel feature checks
- Vulnerability status from kernel reporting

### Run `asct system-info`

Run this command:

    sudo asct system-info

If you run the command without `sudo`, ASCT collects only a limited set of information. ASCT cannot access Desktop Management Interface (DMI) platform and memory details without elevated privileges.

### Use `asct system-info` options

`asct system-info` supports the common ASCT output and logging options.

Example:

    sudo asct system-info --format=json --output-dir data --force --log-level=error --log-file data/asct.log --quiet

### View `asct system-info` output

ASCT supports these output formats:

- `--format=stdout` (default): Prints the report to the terminal
- `--format=csv`: Writes a key-value CSV to the output directory
- `--format=json`: Writes a combined JSON file to the output directory

ASCT writes these files to the output directory:

- `system_info.csv`
- `report.json`

### Example output

Example output for `asct system-info`:

    System Information ------------------------------------------------------------

    System feature report:
      Collected:           2025-07-17 08:16:18.729527
      ASCT version:        0.4.2
      Running as root:     True

    System hardware:
      Architecture:        ARMv8.2
      CPUs:                128
      Interconnect:        CMN-600 x 1

    ...

    Network Information ------------------------------------------------------------

    Local IPv4:      192.0.2.10
    Local IPv6:      2001:db8::10
    Interfaces:
      eth0: UP mtu=1500 ipv4=192.0.2.10

    ...

## View CMN information

Use the `asct cmn` command to inspect the CMN topology and configuration on supported systems.

The command shows CPU-to-crosspoint mapping. It can also provide topology or register-level detail. Use this information to analyze interconnect layout and potential performance impact.

### What `asct cmn` collects

Use `asct cmn` to collect information about the CMN interconnect when it is present on your system. `asct cmn` requires root access to gather CMN information.

Depending on the options that you select, the output includes:

- A high-level topology summary
- An ASCII topology diagram
- A register dump view

### Run `asct cmn`

Before you run `asct cmn` for the first time, run:

    asct cmn --detect

The `--detect` option performs CMN discovery and CPU mapping. This step can take several minutes.

Then run:

    asct cmn

### Use `asct cmn` options

Use these optional flags:

- `--standard`: Prints an ASCII topology diagram
- `--verbose`: Prints additional detail, including registers
- `--detect`: Performs CMN discovery and CPU mapping before reporting

Examples:

    asct cmn --standard
    asct cmn --verbose
    asct cmn --detect --verbose

### View `asct cmn` output

ASCT supports these output formats:

- `--format=stdout` (default): Prints the CMN summary or diagram to the terminal
- `--format=csv`: Writes CMN information as a CSV file to the output directory
- `--format=json`: Writes CMN information to the combined JSON report

ASCT writes these files to the output directory:

- `cmn.csv`
- `report.json`

### Example output

The examples in this section are truncated to improve readability.

#### Default output

In the CPU grid layout, each parenthesized group lists the Linux logical CPU IDs that map to a CMN crosspoint (XP) at that grid position. An empty `()` indicates that no CPUs map to that XP.

#### Output with –standard

#### Output with –verbose

# Memory characterization

ASCT includes benchmarks that measure memory latency and bandwidth.

## Latency

The following latency benchmarks are available:

- Latency sweep
- Idle latency
- Loaded latency
- Core-to-core latency

### Latency sweep

- Use this benchmark to measure memory latency across data sizes from 128 bytes to 1 GiB. The results include the average access latency for each size.

- The results show how memory access latency changes across cache levels and dynamic random-access memory (DRAM). These changes indicate transitions in the cache hierarchy.

- Randomized linked lists prevent hardware prefetching from influencing the latency measurements.

- The benchmark uses 1 GiB huge pages. Using huge pages reduces the effect of page table walks and translation lookaside buffer lookups on latency measurements. Note that on some systems, huge pages are referred to as large pages.

- The benchmark calculates:

  - Lower and upper bounds for each cache level

  - Average latencies

  - The optimal data size for L1, L2, last-level cache (LLC), and DRAM. Other memory benchmarks like Idle Latency and Loaded Latency reuse these sizes to improve precision.

        Latencies at different levels of cache
        --------------------------------------
        Level Lower bound Upper bound Optimum data size Latency [ns]
        L1           128         64K         32.0625K          1.5
        L2           64K        512K             288K          5.2
        LLC           1M          8M             4.5M         48.1
        DRAM         32M          1G             528M        107.6

- The benchmark also generates a line graph and saves it as `latency-sweep.png` in the output directory.

Latency sweep line chart of average memory latency against data size. Latency stays low (~1.3 ns) up to 32 KiB, then increases in steps at 128 KiB, 1 MiB, and 16 MiB, reaching ~120 ns at 64 MiB and above.

### Idle latency

- Use this benchmark in a non-uniform memory access (NUMA) system to measure memory access latency. The benchmark measures latency from the last core on each node to the local and remote memory of that node.

- To characterize the system accurately, ensure it is idle. Close all applications and background processes except the test itself. ASCT imposes only the minimal load needed for measurement, but it cannot control other background activity that might affect test results.

- The benchmark produces a matrix of size n by n, with n equal to the number of NUMA nodes.

      Latencies of random memory access at idle (in nanoseconds)
      ----------------------------------------------------------
              Node 0 Node 1
      Node 0  113.3  266.9
      Node 1  266.5  115.7

Note: ASCT derives the data size used to target DRAM from the `latency-sweep` benchmark. If you do not select this benchmark manually, ASCT automatically includes `latency-sweep` as a dependency.

### Loaded latency

- Use this benchmark to measure the memory latency of the last core on the first NUMA node to its local memory, while other cores generate increasing memory traffic.

- To vary memory pressure, ASCT interleaves memory reads with different numbers of no-operation (NOP) instructions.

        Loaded latency with background memory activity
        ------------------------------------------------
         Injected NOPs  Loaded latency [ns]  Bandwidth [GB/s]
                  3000                115.5              10.7
                   900                115.8              35.2
                   500                117.6              61.4
                   180                122.2             175.3
                   100                129.2             306.5
                    80                134.1             385.0
                    70                138.3             441.8
                    50                158.8             603.9
                    40                193.9             753.7
                    30                250.7             912.7
                    20                266.5             918.3
                    10                281.0             918.3
                     0                305.4             920.7

Note: ASCT derives the data size used to target DRAM from the `latency-sweep` benchmark. If you do not select this benchmark manually, ASCT automatically includes `latency-sweep` as a dependency.

### Core-to-core latency

- Use this benchmark to measure the latency of cache line transfers between pairs of CPU cores across the system.

- The benchmark uses a ping-pong microbenchmark that alternates shared memory access and modification between 2 threads pinned to different cores.

- The shared memory region is a randomized linked list. The benchmark pointer-chases this list so that each memory access depends on the result of the previous one. This approach prevents hardware prefetchers from interfering with results.

- The benchmark binds the memory region to a specific NUMA node. This approach enables you to measure core-to-core latency both locally on the same node and remotely across nodes. This distinction helps you understand the impact of NUMA topology on inter-core communication.

- The benchmark calculates:

  - Node-to-node median latency matrices that summarize inter-node and intra-node communication costs.

  - Highest-latency core pairs that highlight outliers and potential bottlenecks.

  - Asymmetry in latency between different directions, for example A to B versus B to A.

- The benchmark presents results as:

  - Tables that show node-to-node and core-to-core latencies.

  - Heatmaps that show latency patterns across all core pairs.

  - CSV and JSON files that support further analysis.

        Core-to-Core Latency Summary (ns): Data Address @ Local Numa Node
        =================================================================
        Node-to-Node Median Latency Matrix (ns):
        ----------------------------------------
                Node0   Node1
        Node0   31.91  152.86
        Node1  153.10   32.00

        Latency Statistics (ns):
        ------------------------
        Min       : 23.81
        Max       : 161.65
        Mean      : 92.87
        Median    : 140.66

        Top Latency Core Pairs with Median Latency
        ------------------------------------------
        CPUA   CPUB    Latency (ns)
        ---------------------------
        186       91         161.65
        187       83         161.37
         95      186         161.29
         91      190         161.12
        178       90         161.02
        186       83         160.50
         82      157         160.44
         83      176         160.42
         90      187         160.42
         90      159         160.41

        Node-to-Node Latency Statistics (ns):
        -------------------------------------

        Node0 → Node0:
        Min:    24.68 ns
        Max:    42.24 ns
        Mean:   32.16 ns
        Median: 31.91 ns

        Node0 → Node1:
        Min:    112.43 ns
        Max:    161.29 ns
        Mean:   152.67 ns
        Median: 152.86 ns

        Node1 → Node0:
        Min:    141.72 ns
        Max:    161.65 ns
        Mean:   153.14 ns
        Median: 153.10 ns

        Node1 → Node1:
        Min:    23.81 ns
        Max:    41.44 ns
        Mean:   32.26 ns
        Median: 32.00 ns

Square core-to-core latency heatmap with green diagonal quadrants (low latency within clusters) and orange off-diagonal quadrants (higher latency between clusters). A vertical color scale indicates latency from green (low) to red (high).

#### Ping-pong microbenchmark

- Use this benchmark to measure latency between 2 CPU cores by forcing cache line transfers between them.

- The benchmark pins 2 threads on the target cores.

- Each thread alternates between accessing and modifying the shared data structure. During its turn, a thread performs a complete pointer chase on the structure.

- The benchmark uses cache invalidation and data dependency chains to estimate latency.

- When one thread writes to the shared data structure, the corresponding cache line in the other core is evicted.

A diagram titled Inter-Core Ping-Pong Synchronization showing communication between two threads on separate CPUs. Thread A on CPU_1 waits for a spin signal from Thread B on CPU_2, then follows a pointer chain with a write operation, and finally signals Thread B to proceed. A note below explains that threads A and B alternate modifying and reading cache lines, with writes on one causing cache invalidation on the other.

#### NUMA and memory binding

- The benchmark distinguishes between local and remote cache line transfers by binding the memory region to a specific NUMA node. Local transfers, where the cores and memory are on the same node, typically show lower latency. Remote transfers, where the cores and memory are on different nodes, show the cost of traversing the NUMA interconnect.

## Bandwidth

The following bandwidth benchmarks are available:

- Bandwidth sweep
- Cross-NUMA bandwidth
- Peak bandwidth

### Bandwidth sweep

- Use this benchmark to measure the average memory bandwidth achieved by a single core at each level of the memory hierarchy (L1, L2, LLC, and DRAM).

        Bandwidth at different levels of cache
        --------------------------------------
        Data size used Level Bandwidth [GB/s]
             32.0625K    L1            126.5
                 288K    L2             74.5
                 4.5M   LLC             35.1
                 528M  DRAM             15.8

- The benchmark also generates a bandwidth-by-size graph and saves it as `bandwidth.png` in the output directory.

Bandwidth sweep chart showing memory bandwidth (in GiB/s) vs. data sizes. Bandwidth remains near peak (~149 GiB/s) for small sizes, then drops sharply around 128 KiB, and continues declining gradually as data size increases

Note: ASCT derives the data size used to target each cache level from the results of the `latency-sweep` benchmark. If you do not select this benchmark manually, ASCT automatically includes `latency-sweep` as a dependency.

### Cross-NUMA bandwidth

- This benchmark measures the maximum aggregate memory bandwidth for all cores in a NUMA node. The cores access either local memory or memory from a remote NUMA node.

- The benchmark produces an n × n matrix, where n is the number of NUMA nodes.

        Cross-NUMA bandwidths for the system (in GB/s)
        ----------------------------------------------
               Node 0 Node 1
        Node 0  459.1   78.3
        Node 1   78.8  459.2

Note: ASCT derives the data size used to target DRAM from the `latency-sweep` benchmark. If you do not select this benchmark manually, ASCT automatically includes `latency-sweep` as a dependency.

### Peak bandwidth

- This benchmark makes full use of all cores on all NUMA nodes to measure the maximum achievable memory bandwidth of the system.

- To examine how different memory access patterns affect maximum usage, the benchmark tests multiple traffic types. Each pattern yields a corresponding peak bandwidth value.

- If available, compare these results with the theoretical peak bandwidth reported in the system information output.

        Peak memory bandwidth
        ---------------------
                    Traffic type  Peak BW [GB/s]
                       All Reads           918.9
                3:1 Reads-Writes           868.8
                2:1 Reads-Writes           859.5
                1:1 Reads-Writes           842.7
        2:1 Rd-Wr (Non-Temporal)           652.1

Note: ASCT derives the data size used to target DRAM from the `latency-sweep` benchmark. If you do not select this benchmark manually, ASCT automatically includes `latency-sweep` as a dependency.

# Storage characterization

This section describes the storage benchmarks in ASCT. These tests evaluate the performance characteristics of block devices and filesystems using `fio` under controlled sweep parameters.

## What ASCT storage benchmarking measures

ASCT evaluates storage performance by sweeping parameters and measuring their impact on input/output (I/O) performance.

- Sustained bandwidth
- Input/Output Operations Per Second (IOPS) and latency
- Host CPU utilization during I/O

CPU utilization is reported as a breakdown of time spent in user space, kernel code, I/O wait, and hardware and software interrupt handling. This information helps identify whether performance is limited by the application, the kernel, or the storage device.

## Sweeps included

ASCT automates multiple types of parameter sweeps, each designed to isolate one dimension of I/O behavior:

- Request Size Sweep (4 KiB → 128 KiB) Reveals throughput scaling as well as controller or device saturation points
- I/O Queue Depth Sweep (QD=1 → QD=128, powers of two) Capture parallelism gains and concurrent access behavior
- Concurrent Process Count Sweep (1 → 16 concurrent processes, powers of two) Show scaling across CPU cores
- Access Pattern Sweep Sequential vs random, and read/write/mixed ratios (default: 70% reads). Demonstrates workload-specific differences. Compare sequential and random I/O, as well as read-only, write-only, and mixed workloads (default 70% reads). Use this sweep to show how performance varies by workload.

## How ASCT runs parameter sweeps

ASCT uses fio to perform controlled parameter sweeps. For each sweep, ASCT changes one parameter at a time while keeping all other parameters fixed. It then repeats the sweep for each configured parameter set.

Sweep type | Example values
--- | ---
Request size | 4K, 8K, 16K, 32K, 64K, 128K
Queue depth | 1, 2, 4, 8, 16, 32, 64, 128
Jobs | 1, 2, 4, 8, 16
Access pattern | seq read, seq write, rand read, rand write, seq mixed, rand mixed

For each sweep point, ASCT records the following key metrics:

- From `fio`: bandwidth, I/O rate, average latency, latency distribution
- From `mpstat`: CPU utilization breakdowns (user/system/iowait/hard irq/soft irq)

## Run storage benchmarking

Use the `run` command with storage keywords and optional `--user-config` parameters to select a device or create a temporary file.

    # Run a specific sweep (in order of Request Size Sweep, I/O Depth Sweep, Concurrent Process Count Sweep, Access Pattern Sweep)
    sudo asct run storage-request-size-sweep ...
    sudo asct run storage-io-depth-sweep ...
    sudo asct run storage-process-count-sweep ...
    sudo asct run storage-access-pattern-sweep ...

Alternatively, use the short-form alias:

    # Run a specific sweep (short form in order of Request Size Sweep, I/O Depth Sweep, Concurrent Process Count Sweep, Access Pattern Sweep)
    sudo asct run srss ...
    sudo asct run sids ...
    sudo asct run spcs ...
    sudo asct run saps ...

ASCT requires a target device or file. CAUTION: tests might overwrite file content.

    # Provide a target file (ASCT will use /tmp/mytest.dat for test, user needs to create file upfront)
    sudo asct run srss --user-config srss.file_names=/tmp/mytest.dat
    # Provide a target device (ASCT will use the device /dev/nvme07, user needs to ensure device exists)
    sudo asct run srss --user-config srss.file_names=/dev/nvme07

ASCT exposes several user overrides, including the following examples:

- Create a temporary file for file benchmarking so that you do not need to provide the device name:

    # Auto-create temporary file for file I/O
    sudo asct run srss --user-config srss.create_temp_file=1

- Change the read-write mix ratio:

    # Change read write mix ratio for access pattern sweep to use 20% read instead of default of 70%
    sudo asct run saps --user-config saps.rwmixread=20

- Change the sweep steps:

    # Use linear iodepth sweep step between 4 and 8 for more details
    sudo asct run sids --user-config sids.iodepth_sweep_steps=4,5,6,7,8

See the ASTC help for a full list of user overrides.

## Outputs generated

Storage benchmarking in ASCT produces output similar to memory benchmarking:

- Text summary A text summary either printed to the console or exported as CSV/JSON.
  Each sweep point generates a row with the following metrics:
  - Bandwidth (MB/s)
  - I/O rate (kops)
  - Average latency (µs)
  - CPU utilization (%)
- Graphical plots:
  - Bandwidth vs sweep parameter → `storage_io_bandwidth.png`
  - I/O rate vs sweep parameter → `storage_io_io_rate.png`
  - CPU utilization vs sweep parameter → `storage_io_cpu_utilization.png`
  - Latency cumulative distribution (CDF) line per sweep parameter →
    - `storage_io_read_latency_distribution.png`
    - `storage_io_write_latency_distribution.png`

## Per-sweep sample outputs

This section describes sample outputs for each sweep.

### Request Size Sweep

This sweep varies the I/O request size while keeping the rest of parameter constant. It highlights how throughput scales with larger transfers and shows when the device reaches saturation.

#### Console summary

BlockSize | Read BW (MB/s) | Write BW (MB/s) | Total BW (MB/s) | Read Thruput (kops) | Write Thruput (kops) | Thruput (kops) | Read Lat. (us) | Write Lat. (us) | Lat. (us) | CPU usr (%) | CPU sys (%) | CPU iowait (%)
--- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---
4K | 11.9 | 0.0 | 11.9 | 3.0 | 0.0 | 3.0 | 1311.7 | 0.0 | 1311.7 | 0.1 | 0.1 | 99.8
8K | 23.8 | 0.0 | 23.8 | 3.0 | 0.0 | 3.0 | 1311.3 | 0.0 | 1311.3 | 0.1 | 0.2 | 99.6
16K | 47.7 | 0.0 | 47.7 | 3.0 | 0.0 | 3.0 | 1311.3 | 0.0 | 1311.3 | 0.1 | 0.2 | 99.5
32K | 95.4 | 0.0 | 95.4 | 3.1 | 0.0 | 3.1 | 1310.1 | 0.0 | 1310.1 | 0.1 | 0.3 | 99.6
64K | 127.1 | 0.0 | 127.1 | 2.0 | 0.0 | 2.0 | 1965.9 | 0.0 | 1965.9 | 0.1 | 0.2 | 99.7
128K | 127.2 | 0.0 | 127.2 | 1.0 | 0.0 | 1.0 | 3932.0 | 0.0 | 3932.0 | 0.0 | 0.2 | 99.8

#### Plots for details

Bandwidth | I/O Rate
--- | ---

CPU Utilization | Read Latency CDF
--- | ---

### I/O Depth Sweep

This sweep varies the I/O queue depth (the number of outstanding requests) while keeping the request size and process count constant.
It measures how effectively the device exploits parallelism.

#### Console summary

IODepth | Read BW (MB/s) | Write BW (MB/s) | Total BW (MB/s) | Read Thruput (kops) | Write Thruput (kops) | Thruput (kops) | Read Lat. (us) | Write Lat. (us) | Lat. (us) | CPU usr (%) | CPU sys (%) | CPU iowait (%)
--- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---
1 | 11.9 | 0.0 | 11.9 | 3.0 | 0.0 | 3.0 | 1311.8 | 0.0 | 1311.8 | 0.1 | 0.1 | 99.7
2 | 11.9 | 0.0 | 11.9 | 3.0 | 0.0 | 3.0 | 1311.3 | 0.0 | 1311.3 | 0.1 | 0.1 | 99.7
4 | 11.9 | 0.0 | 11.9 | 3.0 | 0.0 | 3.0 | 1311.3 | 0.0 | 1311.3 | 0.4 | 0.2 | 98.4
8 | 11.9 | 0.0 | 11.9 | 3.0 | 0.0 | 3.0 | 1311.4 | 0.0 | 1311.4 | 0.5 | 0.2 | 98.5
16 | 11.9 | 0.0 | 11.9 | 3.0 | 0.0 | 3.0 | 1311.4 | 0.0 | 1311.4 | 0.1 | 0.1 | 99.8
32 | 11.9 | 0.0 | 11.9 | 3.0 | 0.0 | 3.0 | 1311.3 | 0.0 | 1311.3 | 0.1 | 0.1 | 99.8
64 | 11.9 | 0.0 | 11.9 | 3.0 | 0.0 | 3.0 | 1311.3 | 0.0 | 1311.3 | 0.1 | 0.1 | 99.7
128 | 11.9 | 0.0 | 11.9 | 3.0 | 0.0 | 3.0 | 1311.4 | 0.0 | 1311.4 | 0.7 | 0.2 | 98.7

#### Plots for details

Bandwidth | I/O Rate
--- | ---

CPU Utilization | Read Latency CDF
--- | ---

### Concurrent Process Count Sweep

This sweep varies the number of concurrent processes issuing I/O.
It evaluates scaling across CPU cores and submission queues, and highlights the point at which performance no longer improves.

#### Console summary

ProcessCount | Read BW (MB/s) | Write BW (MB/s) | Total BW (MB/s) | Read Thruput (kops) | Write Thruput (kops) | Thruput (kops) | Read Lat. (us) | Write Lat. (us) | Lat. (us) | CPU usr (%) | CPU sys (%) | CPU iowait (%)
--- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---
1 | 7.0 | 0.0 | 7.0 | 1.8 | 0.0 | 1.8 | 552.7 | 0.0 | 552.7 | 0.1 | 0.1 | 24.9
2 | 11.9 | 0.0 | 11.9 | 3.0 | 0.0 | 3.0 | 655.5 | 0.0 | 655.5 | 0.1 | 0.2 | 49.7
4 | 11.9 | 0.0 | 11.9 | 3.0 | 0.0 | 3.0 | 1311.3 | 0.0 | 1311.3 | 0.1 | 0.2 | 99.7
8 | 11.9 | 0.0 | 11.9 | 3.0 | 0.0 | 3.0 | 2622.3 | 0.0 | 2622.3 | 0.1 | 0.3 | 99.6
16 | 11.9 | 0.0 | 11.9 | 3.0 | 0.0 | 3.0 | 5245.5 | 0.0 | 5245.5 | 0.1 | 0.5 | 99.3

#### Plots for details

Bandwidth | I/O Rate
--- | ---

CPU Utilization | Read Latency CDF
--- | ---

### Access Pattern Sweep

This sweep evaluates different workload profiles, including sequential and random access patterns, and the mix of read and write operations. It highlights the workload sensitivity of the device.

#### Console summary

AccessPattern | Read BW (MB/s) | Write BW (MB/s) | Total BW (MB/s) | Read Thruput (kops) | Write Thruput (kops) | Thruput (kops) | Read Lat. (us) | Write Lat. (us) | Lat. (us) | CPU usr (%) | CPU sys (%) | CPU iowait (%)
--- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---
read | 11.9 | 0.0 | 11.9 | 3.0 | 0.0 | 3.0 | 1311.8 | 0.0 | 1311.8 | 0.1 | 0.1 | 99.8
write | 0.0 | 11.9 | 11.9 | 0.0 | 3.0 | 3.0 | 0.0 | 1311.4 | 1311.4 | 0.0 | 0.4 | 99.4
randread | 11.9 | 0.0 | 11.9 | 3.0 | 0.0 | 3.0 | 1311.3 | 0.0 | 1311.3 | 0.1 | 0.2 | 99.7
randwrite | 0.0 | 11.9 | 11.9 | 0.0 | 3.0 | 3.0 | 0.0 | 1311.3 | 1311.3 | 0.0 | 0.4 | 99.6
rw | 8.3 | 3.6 | 11.9 | 2.1 | 0.9 | 3.0 | 1235.1 | 1488.3 | 1311.4 | 0.1 | 0.3 | 99.7
randrw | 8.3 | 3.6 | 11.9 | 2.1 | 0.9 | 3.0 | 1237.9 | 1481.8 | 1311.4 | 0.1 | 0.3 | 99.4

#### Plots for details

Bandwidth | I/O Rate | CPU Utilization
--- | --- | ---

Read Latency CDF | Write Latency CDF
--- | ---

## Additional notes

- Run with elevated privileges if direct I/O or device binding requires it.
- Ensure that results directories are on a stable path. ASCT saves data files under the run-specific `data.*` directory.