Almeria Industries

AWS S3-Compatible Object Storage Server with Web UI

Server Information

Server Identity

Property	Value
Service Name	s3-storage
Version	1.0
API Compatibility	AWS S3 API
Default Port	8080
Protocol	HTTP/HTTPS
Default Region	us-east-1

Deployment Endpoints

Environment	Endpoint	Web UI
Production	https://s3-storage.almeriaindustries.com	https://s3-storage.almeriaindustries.com/ui
Local Development	http://localhost:8080	http://localhost:8080/ui

S3 API Compatibility

Supported S3 Operations

Operation	Method	Path	Status
ListBuckets	GET	/	✓ Supported
CreateBucket	PUT	/{bucket}	✓ Supported
DeleteBucket	DELETE	/{bucket}	✓ Supported
HeadBucket	HEAD	/{bucket}	✓ Supported
ListObjectsV2	GET	/{bucket}?list-type=2	✓ Supported
PutObject	PUT	/{bucket}/{key}	✓ Supported
GetObject	GET	/{bucket}/{key}	✓ Supported
DeleteObject	DELETE	/{bucket}/{key}	✓ Supported
HeadObject	HEAD	/{bucket}/{key}	✓ Supported
Multipart Upload	Various	/{bucket}/{key}?uploadId=...	✓ Supported
Object Versioning	Various	/{bucket}/{key}?versionId=...	✓ Supported
GetBucketReplication	GET	/{bucket}?replication	✓ Supported
PutBucketReplication	PUT	/{bucket}?replication	✓ Supported
DeleteBucketReplication	DELETE	/{bucket}?replication	✓ Supported

Multipart Upload Support

The server now supports AWS S3 multipart upload for large files:

Operation	Description	Status
CreateMultipartUpload	Initiate a multipart upload session	✓ Supported
UploadPart	Upload individual parts (5MB-100MB each)	✓ Supported
CompleteMultipartUpload	Finalize and assemble all parts into final object	✓ Supported
AbortMultipartUpload	Cancel upload and clean up parts	✓ Supported
ListParts	List uploaded parts for an in-progress upload	✓ Supported
ListMultipartUploads	List all in-progress multipart uploads for a bucket	✓ Supported

Multipart Upload Limits

Parameter	Value	Notes
Minimum Part Size	5 MB	All parts except the last must be at least 5MB
Maximum Part Size	100 MB	Cloudflare upload limit per request (bypass Cloudflare for larger parts)
Maximum Parts	10,000	Maximum number of parts per upload
Maximum Object Size	~976 GB	10,000 parts × 100MB (limited by part count and size)

Quota Enforcement

Storage quota is enforced progressively during multipart uploads:

• Progressive Checking - Quota is validated before each part upload, preventing wasted bandwidth and resources
• Fail Fast - If quota would be exceeded, the upload fails immediately before transferring data
• Final Validation - Quota is also checked during CompleteMultipartUpload for defense in depth
• Synchronous Stats - Bucket statistics are updated synchronously to prevent race conditions

Query Parameters

Supported query parameters for ListObjectsV2:

• prefix - Filter objects by prefix
• delimiter - Group objects by prefix (folder simulation)
• max-keys - Maximum number of keys to return (default: 1000)
• continuation-token - Token for pagination
• start-after - Start listing after this key

Authentication Methods

AWS Signature Version 4 (SigV4)

Component	Details
Algorithm	AWS4-HMAC-SHA256
Header Format	Authorization: AWS4-HMAC-SHA256 Credential=..., SignedHeaders=..., Signature=...
Timestamp Header	X-Amz-Date (ISO 8601 format: 20240115T103000Z)
Content Hash	X-Amz-Content-SHA256 (SHA256 hex of request body or UNSIGNED-PAYLOAD)
Max Request Age	15 minutes
Credential Scope	YYYYMMDD/region/s3/aws4_request

Web UI Session Authentication

Property	Value
Method	Cookie-based sessions
Login Path	/ui/login
Session TTL	24 hours (configurable via SESSION_TTL)
CSRF Protection	Enabled (token in meta tag)

Per-User Credentials

Each user can have their own S3 credentials configured by administrators:

• Access Key ID - Unique identifier (e.g., AKIAIOSFODNN7EXAMPLE)
• Secret Access Key - Secret key for signature calculation
• Bootstrap Admin - Uses global credentials from environment variables (S3_ACCESS_KEY_ID, S3_SECRET_ACCESS_KEY)
• Regular Users - Credentials set by admin through Web UI

Azure Active Directory (AAD) Authentication

S3-Storage supports Azure AD as an identity provider for Web UI login. When enabled, users authenticate via the standard OAuth 2.0 authorization code flow.

Variable	Description
AZURE_AD_ENABLED	Set to true to enable Azure AD authentication
AZURE_AD_TENANT_ID	Azure AD tenant ID (GUID)
AZURE_AD_CLIENT_ID	Azure AD application (client) ID
AZURE_AD_CLIENT_SECRET	Azure AD client secret

Register the following redirect URI in your Azure AD app registration:

{REQUEST_BASE_URL}/ui/auth/callback
# Example:
https://s3-storage.almeriaindustries.com/ui/auth/callback

Access Control

User Roles

Role	Permissions
admin	Full access to all buckets and objects Create, read, update, delete users Set S3 credentials for users Configure bucket ACLs View system statistics
user	Access owned buckets Access buckets with explicit read/write permissions Upload, download, delete own objects Generate share links for accessible objects

Bucket ACLs (Access Control Lists)

Admins can configure per-bucket permissions for users:

• Read Permission - List objects, get object metadata, download objects
• Write Permission - Upload objects, delete objects
• Owner - Automatically has read/write permissions

Storage Features

Content-Type Detection

Feature	Details
Detection Engine	Magika (Google's ML-based file type detection)
Model Version	standard_v3_3
Supported Types	100+ file types including images, documents, videos, archives, code files
Fallback	application/octet-stream if detection fails

Object Metadata

Each object stores the following metadata in xl.meta files (MessagePack format):

• Size - Object size in bytes
• ETag - SHA256 hash of the object content (64 characters hex)
• LastModified - Unix timestamp of last modification
• ContentType - MIME type detected by Magika
• DiskUUID - Storage disk identifier
• Parts - Object parts (currently single part only)

Storage Limits

Limit	Value	Notes
Max Object Size (Single PUT)	100 MB	Cloudflare upload limit for single PUT requests
Max Object Size (Multipart)	~976 GB	Using multipart upload (10,000 parts × 100MB). Use direct endpoint to bypass Cloudflare 100MB part limit
Storage Quota	Configurable	Set via STORAGE_QUOTA_BYTES (0 = unlimited). Enforced progressively during multipart uploads
Max Keys per List	1000	Can be reduced with max-keys parameter
Bucket Name Length	3-63 characters	Lowercase, numbers, hyphens only

Storage Layout

Objects are stored on disk using the following structure:

/data/
├── .users              # User database (JSON)
├── .sharelinks         # Share links database (JSON)
├── bucket-name/        # Bucket directory
│   ├── .bucket.meta    # Bucket metadata
│   └── object-key/     # Object directory (key as folder)
│       ├── xl.meta     # Object metadata (MessagePack)
│       └── part.1      # Object data (binary)

Bucket Replication

Replication Targets

A replication target describes a remote S3-compatible bucket that objects can be replicated to. Targets are registered via a MinIO-admin-compatible API (used by the mc CLI) and identified by a generated ARN.

Operation	Method	Path
Register target	PUT	/minio/admin/v1/set-target?bucket={bucket}
List targets	GET	/minio/admin/v1/list-targets
Remove target	DELETE	/minio/admin/v1/remove-target?arn={arn}

Each target stores the remote endpoint URL, destination bucket, access/secret keys, and region (defaults to us-east-1). The ARN format is arn:minio:replication::{uuid}:{bucket}. Targets and per-bucket rules are persisted atomically to a .replication file in the storage directory.

Replication Rules (S3 API)

Once a target is registered, replication rules for a bucket are managed with the standard S3 bucket-replication subresource:

Operation	Method	Path
GetBucketReplication	GET	/{bucket}?replication
PutBucketReplication	PUT	/{bucket}?replication
DeleteBucketReplication	DELETE	/{bucket}?replication

Rules follow the standard ReplicationConfiguration XML shape — each <Rule> has an ID, priority, status (Enabled/Disabled), an optional key-prefix filter, and a destination referencing a registered target ARN:

<ReplicationConfiguration>
  <Role>arn:aws:iam::replication-role</Role>
  <Rule>
    <ID>replicate-all</ID>
    <Priority>1</Priority>
    <Status>Enabled</Status>
    <Filter><Prefix></Prefix></Filter>
    <Destination>
      <Bucket>arn:minio:replication::<uuid>:destbucket</Bucket>
    </Destination>
  </Rule>
</ReplicationConfiguration>

Before a rule is accepted, the server validates compatibility with the target (reachability, versioning state, and encryption settings) so misconfigured replicas are rejected up front rather than failing silently in the background.

• Multiple rules per bucket - Priority-ordered, each with its own target and optional prefix filter
• Prefix filtering - Replicate only a subset of keys (empty prefix = replicate everything)
• Multipart-aware - Objects written via multipart upload participate in replication the same as single-PUT objects
• Delete propagation - Object deletes are replicated to targets, not just creates/updates

JetStream Events & Monitoring

Object and bucket write events are published to NATS JetStream (when enabled) and consumed by the async replicator, decoupling object writes from the replication path. Replication attempts (success and failure) are recorded to an event log for auditing and troubleshooting:

Operation	Method	Path
All recent replication events	GET	/api/replication/events
Events for one bucket	GET	/api/replication/events/{bucket}

The Web UI surfaces replication status on the bucket list and dashboard, and includes a dedicated Replication page for managing targets, rules, and viewing event history.

Restore from a Replica

Objects can be restored back from a configured replica into the source bucket — useful for disaster recovery if the primary copy is lost or corrupted. A restore runs asynchronously as a background job; progress is streamed so large restores can be monitored without blocking.

• Selective or full restore - Restore specific object keys, or omit keys to restore everything available on the replica
• Async job model - Starting a restore returns a job ID immediately; progress (objects processed, bytes restored, failures) is reported per object with a final completion event
• Web UI - Available from the Buckets page for any bucket with a configured replication target

Web UI Features

Dashboard

• Storage statistics (used space, total buckets, total objects)
• Storage quota visualization
• Recent activity

Bucket Management

• Create buckets (with owner assignment)
• Delete buckets (with cascade option for objects)
• View bucket statistics (size, object count)
• Configure bucket ACLs (read/write permissions per user)
• Search and filter buckets

Object Management

• Upload - Drag & drop or click to select files
• Download - Direct download via browser
• Delete - Single object delete with confirmation
• Bulk Delete - Select multiple objects and delete in one action
• Browse - Navigate folder-like prefixes (simulated directories)
• Search - Filter objects by name
• Sort - Sort by name, size, type, or date
• Pagination - Configurable objects per page (default: 50, max: 500)
• View Signature - Display SHA256 hash (truncated with click-to-copy full hash)

Share Links

• Generate Links - Create temporary URLs for objects
• Expiration Options - 1 hour, 1 day, 1 week, 30 days, or never expires
• Anonymous Access - Share links work without authentication
• Token-Based - Secure random tokens (not guessable)
• Management - View, delete share links

User Management (Admin Only)

• Create users (username, password, roles)
• Delete users
• Set S3 credentials (Access Key ID + Secret Access Key)
• Assign roles (admin/user)
• View user list with credentials status
• Password reset

Configuration

Environment Variables

Variable	Default	Description
STORAGE_DIRECTORY	/data	Directory for storing buckets and objects
S3_AUTH_ENABLED	false	Enable AWS SigV4 authentication
S3_AUTH_REGION	us-east-1	AWS region for signature verification
S3_ACCESS_KEY_ID	(empty)	Bootstrap admin access key ID
S3_SECRET_ACCESS_KEY	(empty)	Bootstrap admin secret access key
WEB_UI_ENABLED	false	Enable web user interface
WEB_UI_PREFIX	/ui	URL prefix for web UI
LOCAL_AUTH_ENABLED	true	Enable local authentication
LOCAL_AUTH_USERNAME	admin	Bootstrap admin username
LOCAL_AUTH_PASSWORD	changeme	Bootstrap admin password
SESSION_SECRET	(random)	Secret key for session cookies (32 bytes)
SESSION_TTL	86400	Session time-to-live in seconds (24h)
STORAGE_QUOTA_BYTES	0	Storage quota in bytes (0 = unlimited)
MAGIKA_ASSETS_DIR	/opt/magika/assets	Path to Magika ML model assets
REQUEST_BASE_URL	http://localhost:8080	Base URL for share links
JAEGER_ENDPOINT	(empty)	Jaeger telemetry endpoint
AZURE_AD_ENABLED	false	Enable Azure Active Directory authentication
AZURE_AD_TENANT_ID	(empty)	Azure AD tenant ID (GUID)
AZURE_AD_CLIENT_ID	(empty)	Azure AD application (client) ID
AZURE_AD_CLIENT_SECRET	(empty)	Azure AD client secret
KES_ENABLED	false	Enable KES server-side encryption integration
KES_ENDPOINT	(empty)	KES server URL (e.g. https://kes.internal:7373)
KES_DEFAULT_KEY	s3-master-key	Master key name used when a bucket has no specific key configured
KES_INSECURE_TLS	false	Skip TLS certificate verification (dev/self-signed only)
KES_AUTH_MODE	api-key	Authentication mode: api-key or mtls
KES_API_KEY	(empty)	API key token (kes:v1:...) — api-key mode only
KES_CLIENT_CERT	(empty)	Path to PEM client certificate — mtls mode only
KES_CLIENT_KEY	(empty)	Path to PEM client private key — mtls mode only
KES_CA_CERT	(empty)	Path to PEM CA certificate for KES server TLS verification (optional)
VERSIONS_CLEANUP_INTERVAL_SEC	86400	Cleanup interval for versioned objects in seconds (24h)
NATS_JET_STREAM_ENABLED	false	Enable NATS JetStream for replication event notifications
NATS_CLUSTER_ENDPOINTS	nats://orangegrove.almeriaindustries.com:4222	Comma-separated list of NATS cluster endpoints
NATS_JET_STREAM_NAME	s3	JetStream stream name used for bucket/object events
NATS_JET_STREAM_SUBJECT	s3.events	JetStream subject used for bucket/object events
NATS_JET_STREAM_AUTH_METHOD	userpass	Authentication method for the NATS connection
NATS_JET_STREAM_USER	(empty)	NATS username — userpass auth method only
NATS_JET_STREAM_PASSWORD	(empty)	NATS password — userpass auth method only

Command-Line Flags

Flag	Short	Default	Description
--port	-p	8080	HTTP server port
--timeout	-t	60	Graceful shutdown timeout (seconds)
--swagger	-s	(disabled)	Enable Swagger UI (dev only)
--devel	-d	(disabled)	Development mode (dev only)
--telemetry	-r	(empty)	Enable telemetry: local or remote
--pprof	-o	(disabled)	Enable pprof profiling endpoints

Client Examples

AWS CLI

# Configure
aws configure set aws_access_key_id AKIAIOSFODNN7EXAMPLE
aws configure set aws_secret_access_key wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
aws configure set default.region us-east-1

# Create bucket
aws --endpoint-url=https://s3-storage.almeriaindustries.com s3 mb s3://my-bucket

# Upload file
aws --endpoint-url=https://s3-storage.almeriaindustries.com s3 cp file.txt s3://my-bucket/

# List objects
aws --endpoint-url=https://s3-storage.almeriaindustries.com s3 ls s3://my-bucket/

# Download file
aws --endpoint-url=https://s3-storage.almeriaindustries.com s3 cp s3://my-bucket/file.txt ./

# Upload large files with multipart upload
# Set the threshold for when multipart kicks in (default is 8MB)

aws configure set default.s3.multipart_threshold 100MB

# Set the chunk size for each part
aws configure set default.s3.multipart_chunksize 64MB

# Set the number of parallel upload threads
aws configure set default.s3.max_concurrent_requests 10

# Upload large file using multipart (for uploads bypassing Cloudflare use upload subdomain)
aws s3 cp large-file.mp4 s3://my-bucket/large-file.mp4 --endpoint-url=https://s3-storage.almeriaindustries.com

Python (boto3)

import boto3

s3 = boto3.client('s3',
    endpoint_url='https://s3-storage.almeriaindustries.com',
    aws_access_key_id='AKIAIOSFODNN7EXAMPLE',
    aws_secret_access_key='wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY',
    region_name='us-east-1'
)

# Create bucket
s3.create_bucket(Bucket='my-bucket')

# Upload file
with open('file.txt', 'rb') as f:
    s3.put_object(Bucket='my-bucket', Key='file.txt', Body=f)

# List objects
response = s3.list_objects_v2(Bucket='my-bucket')
for obj in response.get('Contents', []):
    print(obj['Key'])

cURL (Direct API)

# Create bucket (requires SigV4 signature)
curl -X PUT https://s3-storage.almeriaindustries.com/my-bucket \
  -H "Authorization: AWS4-HMAC-SHA256 Credential=..."

# Get object (with pre-signed URL)
curl -O https://s3-storage.almeriaindustries.com/my-bucket/file.txt?X-Amz-Algorithm=...

Monitoring & Observability

Telemetry

• OpenTelemetry - Traces for all operations
• Jaeger Integration - Distributed tracing backend
• Span Attributes - Bucket name, object key, user ID, operation type
• Error Recording - Failed operations recorded with error details

Metrics (via Dashboard)

• Total storage used (bytes)
• Storage quota usage percentage
• Number of buckets
• Number of objects
• Per-bucket statistics

Logging

• Structured Logging - JSON format with correlation IDs
• Log Levels - Debug, Info, Warn, Error
• Correlation IDs - Trace requests across log entries

Security Considerations

Signature Validation

• Timestamp Verification - Rejects requests older than 15 minutes
• HMAC-SHA256 - Cryptographically secure signature algorithm
• Payload Hash - Validates request body integrity (conditional for proxied requests)
• Constant-Time Comparison - Prevents timing attacks

Proxy Detection

The server detects reverse proxies (Cloudflare, Traefik, nginx) via headers:

• CF-Ray - Cloudflare request ID
• X-Forwarded-For - Generic proxy header
• X-Real-IP - Traefik/nginx real client IP

When detected, the server adjusts validation (skips strict payload hash checks) to accommodate proxy modifications.

Session Security

• Secure Cookies - HttpOnly, SameSite flags
• CSRF Protection - Token validation for state-changing operations
• bcrypt Passwords - Hashed with cost factor 10
• Session Expiration - Configurable TTL (default 24h)

Rate Limiting

• Configurable events per second (MAX_EVENTS_PER_SEC, default: 100)
• Burst size limit (MAX_BURST_SIZE, default: 120)
• Per-IP rate limiting

Server-Side Encryption (KES)

S3-Storage supports per-bucket server-side encryption (SSE-KMS) via a KES-compatible key management server (e.g. go-vkms-kes backed by HashiCorp Vault Transit). Encryption is toggled per bucket through the Web UI (Buckets page → Encryption Settings).

Authentication Mode: API Key (Default)

The server authenticates with KES using an Authorization: ApiKey <token> header over TLS.

export KES_AUTH_MODE="api-key"
export KES_API_KEY="kes:v1:ABCdef..."

# Optional: CA certificate to verify the KES server's TLS cert
export KES_CA_CERT="/etc/ssl/kes-ca.pem"

Authentication Mode: mTLS

The server authenticates with KES by presenting a client TLS certificate. Identity is derived from the certificate by KES — no API key header is used.

export KES_AUTH_MODE="mtls"
export KES_CLIENT_CERT="/etc/ssl/kes-client.pem"   # PEM client certificate
export KES_CLIENT_KEY="/etc/ssl/kes-client-key.pem" # PEM client private key

# Optional: CA certificate to verify the KES server's TLS cert
export KES_CA_CERT="/etc/ssl/kes-ca.pem"

KES Server Setup Requirements

Before enabling encryption, the following must be configured on the KES server:

1. Create the master key (via KES API or go-vkms-kes UI):

   curl -X POST https://kes.internal:7373/v1/key/create/s3-master-key \
     -H "Authorization: ApiKey kes:v1:..."

2. Create a policy granting generate and decrypt permissions on the key:

   {
     "allow": [
       "/v1/key/generate/s3-master-key",
       "/v1/key/decrypt/s3-master-key"
     ]
   }

3. Assign an identity (API key fingerprint or mTLS cert fingerprint) to the policy so s3-storage is authorised to use the key.

Encryption Behaviour

• Encryption is enabled per bucket through the Web UI
• New objects uploaded to an encrypted bucket are automatically encrypted on disk
• Existing objects in the bucket before encryption was enabled remain unencrypted and are served as-is
• When encryption is disabled for a bucket, previously encrypted objects remain encrypted on disk and are still transparently decrypted on read — only new uploads are stored unencrypted
• The dashboard displays the KES server status (Online/Offline), endpoint, authentication mode, and key store backend type

KES Environment Variables Summary

Variable	Default	Description
KES_ENABLED	false	Enable KES integration
KES_ENDPOINT	(empty)	KES server URL (e.g. https://kes.internal:7373)
KES_DEFAULT_KEY	s3-master-key	Master key name used when a bucket has no specific key configured
KES_INSECURE_TLS	false	Skip TLS cert verification (dev only — do not use in production)
KES_AUTH_MODE	api-key	Authentication mode: api-key or mtls
KES_API_KEY	(empty)	API key token (kes:v1:...) — api-key mode only
KES_CLIENT_CERT	(empty)	Path to PEM client certificate — mtls mode only
KES_CLIENT_KEY	(empty)	Path to PEM client private key — mtls mode only
KES_CA_CERT	(empty)	Path to PEM CA cert for KES server TLS verification (optional)

Troubleshooting

Common Issues

Issue	Cause	Solution
SignatureDoesNotMatch	Incorrect credentials or timestamp skew	Verify credentials, check system clock, ensure region matches (us-east-1)
Upload fails with 502 error	File exceeds Cloudflare limit (100MB)	Use direct connection, Cloudflare Workers, or paid plan
Cannot login to Web UI	Wrong username/password	Check LOCAL_AUTH_USERNAME and LOCAL_AUTH_PASSWORD
Magika model not found	MAGIKA_ASSETS_DIR incorrect	Verify path points to magika/assets directory
Request time too skewed	Client/server clock difference > 15min	Sync system clocks using NTP

Debug Mode

Enable debug logging and Swagger UI for development:

./s3-storage -d -s -r local -p 8080

• -d - Development mode (verbose logging)
• -s - Enable Swagger UI at /swagger/index.html
• -r local - Local telemetry endpoint

API Documentation

Swagger/OpenAPI

Interactive API documentation is available at:

• Local Development: http://localhost:8080/swagger/index.html
• Requires: Start server with -s or -d flag
• Format: OpenAPI 3.0 specification

Additional Tools

meta-decoder

Command-line tool for inspecting metadata files:

# Decode bucket metadata
meta-decoder -meta /data/.meta

# Decode object metadata
meta-decoder -xl /data/bucket/object/xl.meta

# Decode share links
meta-decoder -share /data/.sharelinks

# JSON output only
meta-decoder -meta /data/.meta -json | jq '.'

See src/test/tools/meta-decoder/Readme.md for full documentation.

System Requirements

Runtime Dependencies

Dependency	Version	Purpose
ONNXRUNTIME	1.23.2	Machine learning inference engine (for Magika)
Magika Assets	standard_v3_3	ML model for content-type detection

Supported Platforms

• macOS Intel (x86_64)
• macOS Apple Silicon (arm64)
• Linux x64 (x86_64)
• Linux ARM64 (aarch64)

Recommended Resources

Resource	Minimum	Recommended
CPU	1 core	2+ cores
RAM	512 MB	2 GB+
Disk	100 MB (app)	Depends on storage needs