bigdata-php documentation

Requirements

Framework adapter versions and minimum PHP versions:

A framework's own minimum PHP version may be higher than this package's 8.0; the framework requirement takes precedence.

Installation

Standard installation (available on Packagist):

composer require erikwang2013/bigdata-php:^2.0

The package is declared as yii2-extension so Yii2 can auto-register its component; Laravel, ThinkPHP, Webman, Hyperf, Yii3, and non-Yii2 projects still install it as a normal Composer library. See section 6 for framework registration.

Install optional dependencies as needed:

composer require apache/thrift           # Hive / Spark Thrift Server
composer require mroosz/php-cassandra    # Cassandra

Doris / StarRocks need no extra Composer package, but the pdo_mysql extension must be enabled in php.ini.

Quick Start

use Erikwang2013\BigData\BigData;

$trino = BigData::driver('trino', [
    'host' => '127.0.0.1',
    'port' => 8080,
    'user' => 'php',
    'catalog' => 'hive',
    'schema' => 'default',
]);

$result = $trino->query('SELECT 1 AS id');

foreach ($result->rows() as $row) {
    echo $row[0], PHP_EOL;
}

Common ResultSet methods:

$result->columns(); // Column names or column descriptions; exact structure is driver-defined
$result->rows();    // Row arrays; null values are preserved as-is
$result->count();   // Number of rows
$result->raw();     // Raw driver response, useful for debugging protocol details

Scheduler-driver example:

$airflow = BigData::driver('airflow', [
    'baseUrl' => 'http://127.0.0.1:8081',
    'username' => 'admin',
    'password' => 'admin',
]);

$airflow->triggerDag('daily_etl', ['date' => '2026-10-29']);

Optional-driver mode:

BigData::drivers();   // All 21 driver keys
BigData::available(); // Driver keys whose optional dependencies are present in the current environment

try {
    $hive = BigData::driver('hive', ['host' => '127.0.0.1']);
} catch (\Erikwang2013\BigData\Exception\DriverUnavailableException $e) {
    echo $e->getMessage(); // Contains a composer require apache/thrift hint
}

Drivers can also be instantiated directly. Direct instantiation bypasses the optional dependency check; a missing extension or class fails at runtime:

use Erikwang2013\BigData\Driver\Trino\TrinoClient;
use Erikwang2013\BigData\Driver\Trino\TrinoConfig;

$trino = new TrinoClient(TrinoConfig::fromArray([
    'host' => '127.0.0.1',
]));

Framework Integration

The core package depends on no framework. The adapters in src/Integration do only two things: read the unified default / drivers configuration and bind Erikwang2013\BigData\Integration\Manager to the framework container.

Each adapter has a dedicated guide: Laravel, ThinkPHP, Webman, Hyperf, Yii2, Yii3.

On first boot the adapter copies the config template to the framework's conventional path: config/bigdata.php for Laravel, ThinkPHP, Webman, and Yii2, and config/autoload/bigdata.php for Hyperf. Yii3 needs no copy because yiisoft/config merges the package config automatically. Existing files are never overwritten.

After composer require, registration follows each framework's own rule:

Unified configuration structure (identical across frameworks):

// config/bigdata.php
return [
    'default' => 'trino',
    'drivers' => [
        'trino' => [
            'host' => '127.0.0.1',
            'port' => 8080,
            'user' => 'php',
            'catalog' => 'hive',
            'schema' => 'default',
        ],
        'airflow' => [
            'baseUrl' => 'http://127.0.0.1:8081',
            'username' => 'admin',
            'password' => 'admin',
        ],
    ],
];

Public behavior of Manager:

• driver() uses default; driver('airflow') uses an explicit name; instances with the same name are cached.
• drivers() returns the driver names configured under drivers.
• available() returns the configured driver names whose optional dependencies are all present.
• Only drivers configured in drivers can be created; an unknown name throws InvalidArgumentException.

Laravel

The package declares the provider under extra.laravel.providers, so Laravel discovers and registers it automatically after composer require; no config/app.php or bootstrap/providers.php edit is needed. The provider writes config/bigdata.php on first boot and never overwrites an existing file. Publish explicitly when preferred:

php artisan vendor:publish --tag=bigdata-config

Usage:

$trino = app(\Erikwang2013\BigData\Integration\Manager::class)->driver('trino');

// The provider also registers the "bigdata" alias
$airflow = app('bigdata')->driver('airflow');

ThinkPHP

The package declares the service under extra.think.services. With the topthink/think skeleton, the post-autoload-dump script runs php think service:discover after composer require; run it once by hand only for older or custom skeletons without that script. The service writes config/bigdata.php on first boot, so no template copy is needed.

$manager = app('bigdata');
$trino = $manager->driver('trino');

Webman

The package ships a Webman plugin installer (src/Install.php). On composer require, the Webman skeleton's post-package-install hook calls it and writes config/plugin/erikwang2013/bigdata-php/app.php and bootstrap.php; Webman loads that config and runs Bootstrap at start-up. The adapter writes config/bigdata.php on the first worker start, so no manual config/bootstrap.php edit is needed.

If the plugin hook is disabled, register the bootstrap manually:

// config/bootstrap.php
return [
    Erikwang2013\BigData\Integration\Webman\Bootstrap::class,
];

The adapter uses lazy definitions, and each worker process has its own Manager:

$manager = \support\Container::get(\Erikwang2013\BigData\Integration\Manager::class);
$trino = $manager->driver('trino');

Hyperf

The package declares ConfigProvider under extra.hyperf.config, so Hyperf merges it automatically after composer require; no config_providers edit is needed. The provider writes config/autoload/bigdata.php on first boot. Publish explicitly when preferred:

php bin/hyperf.php vendor:publish erikwang2013/bigdata-php

Hyperf 3 requires PHP 8.1 and Swoole 5:

$manager = $container->get(\Erikwang2013\BigData\Integration\Manager::class);
$trino = $manager->driver('trino');

Yii2

The package is declared as type=yii2-extension and registers BigDataBootstrap under extra.bootstrap. yiisoft/yii2-composer adds the component automatically, so no config/web.php or config/console.php edit is needed; the component writes config/bigdata.php on first use. Non-Yii2 projects still install it as a normal library through Composer's default installer.

If Yii2's Composer plugin is disabled, register the component manually:

// config/web.php
return [
    'components' => [
        'bigdata' => Erikwang2013\BigData\Integration\Yii2\BigDataComponent::class,
    ],
];

Driver config is read from Yii::$app->params['bigdata'] when set; otherwise the auto-written config/bigdata.php is used:

$trino = Yii::$app->bigdata->driver('trino');

Yii3

The package declares params and di under extra.config-plugin; the yiisoft/config Composer plugin merges them on install or update. The application must allow the plugin:

composer config allow-plugins.yiisoft/config true

No file is copied; override the erikwang2013/bigdata-php section in the application's config/common/params.php:

// config/common/params.php
return [
    'erikwang2013/bigdata-php' => [
        'default' => 'trino',
        'drivers' => [
            'trino' => ['host' => '127.0.0.1'],
        ],
    ],
];

Manager is already registered in the container, so it can be injected or fetched directly:

final class DashboardController
{
    public function __construct(
        private \Erikwang2013\BigData\Integration\Manager $bigdata,
    ) {
    }
}

Or fetch it directly from the container:

$manager = $container->get(\Erikwang2013\BigData\Integration\Manager::class);
$trino = $manager->driver('trino');

Driver Matrix

19 platforms, 21 driver keys. — means no extra dependency.

The full namespace prefix for driver keys and client classes is Erikwang2013\BigData\. For example, the fully qualified class name of Driver\Trino\TrinoClient is Erikwang2013\BigData\Driver\Trino\TrinoClient.

When used through BigData::driver(), Hudi / Paimon must be given base_path, with optional storage (local or webhdfs) and WebHDFS host / port / user:

$table = BigData::driver('hudi', ['base_path' => '/warehouse/orders']);
$latest = $table->latestCommit();

Architecture

Call path:

1. Application layer: business code calls BigData::driver() directly, or obtains Integration\Manager from the container through a framework adapter.
2. Framework adapter layer (optional): the Laravel, ThinkPHP, Webman, Hyperf, Yii2, and Yii3 adapters read the default / drivers configuration and bind Manager to the container; they introduce no ORM, query builder, or extra abstraction.
3. Manager / Registry: Manager handles configuration resolution and instance caching; BigData internally maintains a static map from the 21 driver keys to factory closures, and both unknown keys and missing optional dependencies are intercepted here.
4. Driver layer: one directory per platform, containing XxxConfig, XxxClient (some drivers also have a dedicated ResultSet or exceptions). Shared HTTP, result-set, config, and exception code lives in src/Support/ and src/Exception/; a driver depends only on its own configuration and the injected HTTP / log instances.
5. External systems: HTTP / JSON (Guzzle 7), Thrift TCLIService, MySQL protocol (PDO), CQL, local filesystem, or WebHDFS.

Key constraints: drivers are created on demand and cached by name; Manager never connects to any external system during construction; the HTTP client and logger can be injected from above, making it easy to apply uniform timeout, proxy, and logging settings.

Functional Design

The code is divided into five layers:

• Core: the BigData facade (driver / drivers / available), Integration\Manager, the driver registry, and the optional dependency gate. The core layer only does name resolution, config merging, and factory invocation.
• Support: shared Support\Http (a thin Guzzle 7 wrapper), Support\ResultSet, Support\Config, and an exception hierarchy rooted at BigDataException; connection and request failures land in ConnectionException and RequestException respectively. Hive, Cassandra, Doris, and StarRocks keep dedicated hierarchies, while Hudi/Paimon/Delta Lake metadata errors keep HudiException / PaimonException / DeltaLakeException.
• Driver: 19 platform-isolated directories, grouped into five categories: SQL / OLAP, Hadoop / compute, Schedulers, Lakehouse, and NoSQL. Drivers do not share client instances, and there is no cross-driver unified SQL layer.
• Optional dependencies: ext-pdo_mysql (Doris/StarRocks SQL only), apache/thrift, mroosz/php-cassandra, aws/aws-sdk-php (S3 only). Hive and Cassandra are checked when the driver is created, PDO when SQL is invoked, and S3 when storage is read.
• Tests: unit tests use mock HTTP, local fixtures, and a fake server; integration tests form a separate group and connect to real services on demand.

The goal of this structure is "a unified entry point without unified semantics": configuration, result sets, and exception naming stay consistent, but each driver still works according to the target system's protocol and error model.

Request Lifecycle

A typical driver call goes through 8 steps:

1. Read configuration: the application or framework adapter reads default and drivers; standalone scripts pass the config array directly to BigData::driver().
2. Resolve the driver name: Manager uses the explicit name or the default name; when calling BigData directly, the caller passes the name.
3. Validate the driver key and optional dependencies: the registry checks the name; doris / starrocks check pdo_mysql, hive checks apache/thrift, and cassandra checks mroosz/php-cassandra.
4. Create config and client: the factory calls XxxConfig::fromArray(), then constructs XxxClient; the Guzzle client and PSR-3 logger can be injected.
5. Authenticate (if needed): HTTP Basic / Bearer / token, Hive SASL PLAIN or LDAP, Doris / StarRocks PDO login, Cassandra CQL authentication.
6. Send the transport request: HTTP/JSON, Thrift TCLIService, PDO MySQL, CQL, local or WebHDFS reads; paginated requests may issue multiple calls at this step.
7. Parse the response: handle JSON and application-level errors under HTTP 200, follow nextUri, result tokens, or offsets, and parse Thrift row batches, PDO rows, or Base64-encoded HBase cells.
8. Return the result: query drivers return ResultSet; scheduler / catalog drivers return driver-specific arrays; Hive and Cassandra sessions are explicitly closed by the caller with close().

Error mapping: transport failures, timeouts, and connection errors throw ConnectionException; HTTP non-2xx responses or application-level errors throw RequestException. No driver retries automatically; the caller decides whether to retry based on idempotency.

Lifecycle

Optional dependencies have an independent lifecycle: Hive/Cassandra check their class when the driver is created, ext-pdo_mysql is checked on the first Doris/StarRocks SQL call, and S3 storage is checked on the first read; a missing dependency raises an exception with an install hint. Directly new-ing a client bypasses the creation-time check.

Directory Structure

bigdata-php/
├── src/
│   ├── BigData.php              # Driver registry, factories, and optional dependency checks
│   ├── Install.php              # Webman plugin installer
│   ├── Driver/
│   │   ├── Trino/
│   │   ├── Hadoop/
│   │   ├── Hive/
│   │   ├── Spark/
│   │   ├── Flink/
│   │   ├── Doris/
│   │   ├── StarRocks/
│   │   ├── Kylin/
│   │   ├── Druid/
│   │   ├── Iceberg/
│   │   ├── Hudi/
│   │   ├── Paimon/
│   │   ├── DeltaLake/
│   │   ├── DolphinScheduler/
│   │   ├── Airflow/
│   │   ├── Azkaban/
│   │   ├── Oozie/
│   │   ├── HBase/
│   │   └── Cassandra/
│   ├── Integration/             # Framework adapters and Manager
│   │   ├── Manager.php
│   │   ├── Laravel/
│   │   ├── ThinkPHP/
│   │   ├── Webman/
│   │   ├── Hyperf/
│   │   ├── Yii2/
│   │   └── Yii3/
│   ├── Support/                 # Shared HTTP, ResultSet, and config helpers
│   └── Exception/
│       ├── BigDataException.php
│       ├── ConnectionException.php
│       ├── RequestException.php
│       └── DriverUnavailableException.php
├── config/                      # Framework config samples
├── docs/
│   ├── diagrams/                # Chinese/English architecture, functional, request lifecycle, and lifecycle SVGs
│   ├── drivers/                 # Detailed docs for the 19 platforms (en/ for English)
│   ├── frameworks/              # Six framework guides (en/ for English)
│   ├── pet.svg / pet-expressions/  # Project pet and 12 animated expressions
│   └── site/                    # Site content source, build and deployment config
├── scripts/                     # Site build / check scripts
├── site/                        # Local site build output (gitignored, not packaged in releases)
├── tests/
│   ├── Driver/                  # Unit tests and fixtures
│   └── Integration/             # Integration tests, run on demand
├── demo/                        # Trino / Flink / Airflow / Cassandra demos
├── composer.json
├── phpstan.neon
├── phpunit.xml / phpunit-8.0.xml
├── ASSETS-LICENSE.md
├── CHANGELOG.md
├── README.md
└── README_EN.md

Tests

composer test              # Unit tests, no real cluster required
composer test-integration  # Integration tests, require available external services

Unit tests use a mock HTTP client, local fixtures, and a fake server, covering 296 tests and 994 assertions, plus 14 integration tests. The test suite is split into the unit and integration groups in phpunit.xml; the default composer test runs only unit and excludes tests/Integration.

Integration tests run on demand—Cassandra, for example, requires a reachable instance; demo/ provides Docker Compose configurations for Trino, Flink, Airflow, and Cassandra as a starting point for local verification.

Limitations

• Lakehouse metadata is read-only: Iceberg reads the REST Catalog and table metadata; Hudi reads the .hoodie timeline; Paimon reads schema / snapshot; Delta Sharing returns Parquet file manifests. None of them read Parquet row data or perform predicate pushdown or query execution.
• Delta Sharing returns no rows: DeltaSharingClient::query() returns file manifests and metadata; the caller must read the actual data according to the manifest.
• Hive authentication scope: supports NONE, NOSASL, SASL PLAIN, and LDAP (QOP auth); NONE/PLAIN/LDAP use SASL PLAIN and NOSASL disables it; Kerberos / GSSAPI are not supported. Binary transport can use TLS via tls => true; HTTP transport uses scheme => https.
• Cassandra is optional and may collide on names: the driver wraps mroosz/php-cassandra, which shares the Cassandra\ namespace with ext-cassandra; loading both at the same time may conflict.
• No automatic retries: connection, HTTP, and application-level errors are thrown directly as ConnectionException / RequestException; retry policies, backoff, and idempotency decisions are the caller's responsibility.
• No unified SQL layer or ORM: drivers only adapt protocols; they do not perform SQL dialect translation, cross-source JOINs, transaction orchestration, connection pooling, or model mapping.
• Optional dependency checks happen only in BigData::driver(): directly instantiating a client skips the check, and a missing extension or class surfaces as a PHP error at runtime.
• Driver capabilities are asymmetric: each platform covers only part of its public API; refer to the docs under docs/drivers/en/ for specific methods, config fields, and limitations.

License

MIT. See LICENSE for details.