Hudi

A small PHP client for Apache Hudi table metadata: read hoodie.properties and list the commit timeline of a table.

Metadata only. This driver never reads or writes Hudi data files (Parquet base files or Avro log files). Use it to inspect table properties, table names and commit instants; hand query execution to Spark, Trino, Flink or another engine.

Quick start

use Erikwang2013\BigData\BigData;

$table = BigData::driver('hudi', ['base_path' => '/warehouse/orders']);

$table->tableName();     // "orders", from hoodie.table.name
$table->properties();    // ['hoodie.table.name' => 'orders', 'hoodie.table.type' => 'MOR', ...]
$table->timeline();      // [['timestamp' => '20240101000000', 'action' => 'commit', 'state' => null], ...]
$table->latestCommit();  // "20240101000000"

The factory opens .hoodie/hoodie.properties and lists .hoodie/timeline once, through the selected storage, and caches both in memory.

Direct instantiation

use Erikwang2013\BigData\Driver\Hudi\HudiTable;
use Erikwang2013\BigData\Driver\Hudi\LocalStorage;

$table = HudiTable::open('/warehouse/orders', new LocalStorage());

Storage

HudiStorage has two methods — list(string $path) and read(string $path) — and three implementations: local, WebHDFS and S3.

Local filesystem

use Erikwang2013\BigData\Driver\Hudi\HudiTable;
use Erikwang2013\BigData\Driver\Hudi\LocalStorage;

$table = HudiTable::open('/warehouse/orders', new LocalStorage());

WebHDFS

Through the unified driver:

$table = BigData::driver('hudi', [
    'base_path' => '/warehouse/orders',
    'storage' => 'webhdfs',
    'host' => 'namenode.local',
    'port' => 9870,
    'user' => 'php',
]);

Direct instantiation:

use Erikwang2013\BigData\Driver\Hudi\HudiTable;
use Erikwang2013\BigData\Driver\Hudi\WebHdfsConfig;
use Erikwang2013\BigData\Driver\Hudi\WebHdfsStorage;

$storage = new WebHdfsStorage(WebHdfsConfig::fromArray([
    'host' => 'namenode.local',
    'port' => 9870,
    'user' => 'php',
]));

$table = HudiTable::open('/warehouse/orders', $storage);

WebHdfsConfig::fromArray() accepts:

WebHdfsStorage uses op=LISTSTATUS to list a directory and op=OPEN to read a file, following the DataNode redirect manually. Client-level default headers and auth are not forwarded to the DataNode hop. An optional PSR-3 logger can be passed as the third constructor argument; every request (including redirect hops) is logged at debug level with the method and URI only.

S3-compatible object storage

storage => 's3' reads from an S3 bucket through aws/aws-sdk-php, which is an optional dependency:

composer require aws/aws-sdk-php

Through the unified driver:

$table = BigData::driver('hudi', [
    'base_path' => 'warehouse/orders',
    'storage' => 's3',
    'bucket' => 'my-bucket',
    'region' => 'eu-west-1',
    'prefix' => 'lakes',
]);

S3 keys:

base_path may be relative (resolved under prefix) or absolute (s3://bucket/warehouse/orders); an absolute path must name the configured bucket.

Direct instantiation:

use Erikwang2013\BigData\Driver\Hudi\HudiTable;
use Erikwang2013\BigData\Driver\Hudi\S3Storage;

$storage = S3Storage::fromConfig([
    'bucket' => 'my-bucket',
    'region' => 'eu-west-1',
    'prefix' => 'lakes',
]);

$table = HudiTable::open('warehouse/orders', $storage);
// An existing Aws\S3\S3Client can be injected: new S3Storage($client, 'my-bucket', 'lakes')

Timeline

Timeline entries are instant files named {timestamp}.{action} for completed instants and {timestamp}.{action}.{state} for in-flight ones:

The timeline is sorted by timestamp. latestCommit() ignores requested and inflight instants and returns the highest completed timestamp, or null when the table has no completed instants yet.

Errors

Metadata problems raise Erikwang2013\BigData\Driver\Hudi\Exception\HudiException (extends Erikwang2013\BigData\Exception\BigDataException), for example a missing or unreadable metadata file, or a missing S3 bucket / SDK. WebHDFS transport errors use the shared Erikwang2013\BigData\Exception\ConnectionException / Erikwang2013\BigData\Exception\RequestException. Errors raised by the AWS SDK on S3 reads are not wrapped.

Notes

• Read-only; data and log files are never touched.
• properties and timeline are read once and cached per HudiTable instance.
• No automatic retries are performed.