AWS S3-Compatible Object Storage Server with Web UI

Server Information

Server Identity

Property	Value
Service Name	s3-storage
Version	1.0
API Compatibility	AWS S3 API
Default Port	8080
Protocol	HTTP/HTTPS
Default Region	us-east-1

Deployment Endpoints

Environment	Endpoint	Web UI
Production	https://s3-storage.almeriaindustries.com	https://s3-storage.almeriaindustries.com/ui
Local Development	http://localhost:8080	http://localhost:8080/ui

S3 API Compatibility

Supported S3 Operations

Operation	Method	Path	Status
ListBuckets	GET	/	✓ Supported
CreateBucket	PUT	/{bucket}	✓ Supported
DeleteBucket	DELETE	/{bucket}	✓ Supported
HeadBucket	HEAD	/{bucket}	✓ Supported
ListObjectsV2	GET	/{bucket}?list-type=2	✓ Supported
PutObject	PUT	/{bucket}/{key}	✓ Supported
GetObject	GET	/{bucket}/{key}	✓ Supported
DeleteObject	DELETE	/{bucket}/{key}	✓ Supported
HeadObject	HEAD	/{bucket}/{key}	✓ Supported
Multipart Upload	Various	/{bucket}/{key}?uploadId=...	✓ Supported
Object Versioning	Various	/{bucket}/{key}?versionId=...	✗ Not Supported

Multipart Upload Support

The server now supports AWS S3 multipart upload for large files:

Operation	Description	Status
CreateMultipartUpload	Initiate a multipart upload session	✓ Supported
UploadPart	Upload individual parts (5MB-100MB each)	✓ Supported
CompleteMultipartUpload	Finalize and assemble all parts into final object	✓ Supported
AbortMultipartUpload	Cancel upload and clean up parts	✓ Supported
ListParts	List uploaded parts for an in-progress upload	✓ Supported
ListMultipartUploads	List all in-progress multipart uploads for a bucket	✓ Supported

Multipart Upload Limits

Parameter	Value	Notes
Minimum Part Size	5 MB	All parts except the last must be at least 5MB
Maximum Part Size	100 MB	Cloudflare upload limit per request (bypass Cloudflare for larger parts)
Maximum Parts	10,000	Maximum number of parts per upload
Maximum Object Size	~976 GB	10,000 parts × 100MB (limited by part count and size)

Quota Enforcement

Storage quota is enforced progressively during multipart uploads:

• Progressive Checking - Quota is validated before each part upload, preventing wasted bandwidth and resources
• Fail Fast - If quota would be exceeded, the upload fails immediately before transferring data
• Final Validation - Quota is also checked during CompleteMultipartUpload for defense in depth
• Synchronous Stats - Bucket statistics are updated synchronously to prevent race conditions

Query Parameters

Supported query parameters for ListObjectsV2:

• prefix - Filter objects by prefix
• delimiter - Group objects by prefix (folder simulation)
• max-keys - Maximum number of keys to return (default: 1000)
• continuation-token - Token for pagination
• start-after - Start listing after this key

Authentication Methods

AWS Signature Version 4 (SigV4)

Component	Details
Algorithm	AWS4-HMAC-SHA256
Header Format	Authorization: AWS4-HMAC-SHA256 Credential=..., SignedHeaders=..., Signature=...
Timestamp Header	X-Amz-Date (ISO 8601 format: 20240115T103000Z)
Content Hash	X-Amz-Content-SHA256 (SHA256 hex of request body or UNSIGNED-PAYLOAD)
Max Request Age	15 minutes
Credential Scope	YYYYMMDD/region/s3/aws4_request

Web UI Session Authentication

Property	Value
Method	Cookie-based sessions
Login Path	/ui/login
Session TTL	24 hours (configurable via SESSION_TTL)
CSRF Protection	Enabled (token in meta tag)

Per-User Credentials

Each user can have their own S3 credentials configured by administrators:

• Access Key ID - Unique identifier (e.g., AKIAIOSFODNN7EXAMPLE)
• Secret Access Key - Secret key for signature calculation
• Bootstrap Admin - Uses global credentials from environment variables (S3_ACCESS_KEY_ID, S3_SECRET_ACCESS_KEY)
• Regular Users - Credentials set by admin through Web UI

Access Control

User Roles

Role	Permissions
admin	Full access to all buckets and objects Create, read, update, delete users Set S3 credentials for users Configure bucket ACLs View system statistics
user	Access owned buckets Access buckets with explicit read/write permissions Upload, download, delete own objects Generate share links for accessible objects

Bucket ACLs (Access Control Lists)

Admins can configure per-bucket permissions for users:

• Read Permission - List objects, get object metadata, download objects
• Write Permission - Upload objects, delete objects
• Owner - Automatically has read/write permissions

Storage Features

Content-Type Detection

Feature	Details
Detection Engine	Magika (Google's ML-based file type detection)
Model Version	standard_v3_3
Supported Types	100+ file types including images, documents, videos, archives, code files
Fallback	application/octet-stream if detection fails

Object Metadata

Each object stores the following metadata in xl.meta files (MessagePack format):

• Size - Object size in bytes
• ETag - SHA256 hash of the object content (64 characters hex)
• LastModified - Unix timestamp of last modification
• ContentType - MIME type detected by Magika
• DiskUUID - Storage disk identifier
• Parts - Object parts (currently single part only)

Storage Limits

Limit	Value	Notes
Max Object Size (Single PUT)	100 MB	Cloudflare upload limit for single PUT requests
Max Object Size (Multipart)	~976 GB	Using multipart upload (10,000 parts × 100MB). Use direct endpoint to bypass Cloudflare 100MB part limit
Storage Quota	Configurable	Set via STORAGE_QUOTA_BYTES (0 = unlimited). Enforced progressively during multipart uploads
Max Keys per List	1000	Can be reduced with max-keys parameter
Bucket Name Length	3-63 characters	Lowercase, numbers, hyphens only

Storage Layout

Objects are stored on disk using the following structure:

/data/
├── .users              # User database (JSON)
├── .sharelinks         # Share links database (JSON)
├── bucket-name/        # Bucket directory
│   ├── .bucket.meta    # Bucket metadata
│   └── object-key/     # Object directory (key as folder)
│       ├── xl.meta     # Object metadata (MessagePack)
│       └── part.1      # Object data (binary)

Web UI Features

Dashboard

• Storage statistics (used space, total buckets, total objects)
• Storage quota visualization
• Recent activity

Bucket Management

• Create buckets (with owner assignment)
• Delete buckets (with cascade option for objects)
• View bucket statistics (size, object count)
• Configure bucket ACLs (read/write permissions per user)
• Search and filter buckets

Object Management

• Upload - Drag & drop or click to select files
• Download - Direct download via browser
• Delete - Single object delete with confirmation
• Bulk Delete - Select multiple objects and delete in one action
• Browse - Navigate folder-like prefixes (simulated directories)
• Search - Filter objects by name
• Sort - Sort by name, size, type, or date
• Pagination - Configurable objects per page (default: 50, max: 500)
• View Signature - Display SHA256 hash (truncated with click-to-copy full hash)

Share Links

• Generate Links - Create temporary URLs for objects
• Expiration Options - 1 hour, 1 day, 1 week, 30 days, or never expires
• Anonymous Access - Share links work without authentication
• Token-Based - Secure random tokens (not guessable)
• Management - View, delete share links

User Management (Admin Only)

• Create users (username, password, roles)
• Delete users
• Set S3 credentials (Access Key ID + Secret Access Key)
• Assign roles (admin/user)
• View user list with credentials status
• Password reset

Configuration

Environment Variables

Variable	Default	Description
STORAGE_DIRECTORY	/data	Directory for storing buckets and objects
S3_AUTH_ENABLED	false	Enable AWS SigV4 authentication
S3_AUTH_REGION	us-east-1	AWS region for signature verification
S3_ACCESS_KEY_ID	(empty)	Bootstrap admin access key ID
S3_SECRET_ACCESS_KEY	(empty)	Bootstrap admin secret access key
WEB_UI_ENABLED	false	Enable web user interface
WEB_UI_PREFIX	/ui	URL prefix for web UI
LOCAL_AUTH_ENABLED	true	Enable local authentication
LOCAL_AUTH_USERNAME	admin	Bootstrap admin username
LOCAL_AUTH_PASSWORD	changeme	Bootstrap admin password
SESSION_SECRET	(random)	Secret key for session cookies (32 bytes)
SESSION_TTL	86400	Session time-to-live in seconds (24h)
STORAGE_QUOTA_BYTES	0	Storage quota in bytes (0 = unlimited)
MAGIKA_ASSETS_DIR	/opt/magika/assets	Path to Magika ML model assets
REQUEST_BASE_URL	http://localhost:8080	Base URL for share links
JAEGER_ENDPOINT	(empty)	Jaeger telemetry endpoint

Command-Line Flags

Flag	Short	Default	Description
--port	-p	8080	HTTP server port
--timeout	-t	60	Graceful shutdown timeout (seconds)
--swagger	-s	(disabled)	Enable Swagger UI (dev only)
--devel	-d	(disabled)	Development mode (dev only)
--telemetry	-r	(empty)	Enable telemetry: local or remote
--pprof	-o	(disabled)	Enable pprof profiling endpoints

Client Examples

AWS CLI

# Configure
aws configure set aws_access_key_id AKIAIOSFODNN7EXAMPLE
aws configure set aws_secret_access_key wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
aws configure set default.region us-east-1

# Create bucket
aws --endpoint-url=https://s3-storage.almeriaindustries.com s3 mb s3://my-bucket

# Upload file
aws --endpoint-url=https://s3-storage.almeriaindustries.com s3 cp file.txt s3://my-bucket/

# List objects
aws --endpoint-url=https://s3-storage.almeriaindustries.com s3 ls s3://my-bucket/

# Download file
aws --endpoint-url=https://s3-storage.almeriaindustries.com s3 cp s3://my-bucket/file.txt ./

# Upload large files with multipart upload
# Set the threshold for when multipart kicks in (default is 8MB)

aws configure set default.s3.multipart_threshold 100MB

# Set the chunk size for each part
aws configure set default.s3.multipart_chunksize 64MB

# Set the number of parallel upload threads
aws configure set default.s3.max_concurrent_requests 10

# Upload large file using multipart (for uploads bypassing Cloudflare use upload subdomain)
aws s3 cp large-file.mp4 s3://my-bucket/large-file.mp4 --endpoint-url=https://s3-storage.almeriaindustries.com

Python (boto3)

import boto3

s3 = boto3.client('s3',
    endpoint_url='https://s3-storage.almeriaindustries.com',
    aws_access_key_id='AKIAIOSFODNN7EXAMPLE',
    aws_secret_access_key='wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY',
    region_name='us-east-1'
)

# Create bucket
s3.create_bucket(Bucket='my-bucket')

# Upload file
with open('file.txt', 'rb') as f:
    s3.put_object(Bucket='my-bucket', Key='file.txt', Body=f)

# List objects
response = s3.list_objects_v2(Bucket='my-bucket')
for obj in response.get('Contents', []):
    print(obj['Key'])

cURL (Direct API)

# Create bucket (requires SigV4 signature)
curl -X PUT https://s3-storage.almeriaindustries.com/my-bucket \
  -H "Authorization: AWS4-HMAC-SHA256 Credential=..."

# Get object (with pre-signed URL)
curl -O https://s3-storage.almeriaindustries.com/my-bucket/file.txt?X-Amz-Algorithm=...

Monitoring & Observability

Telemetry

• OpenTelemetry - Traces for all operations
• Jaeger Integration - Distributed tracing backend
• Span Attributes - Bucket name, object key, user ID, operation type
• Error Recording - Failed operations recorded with error details

Metrics (via Dashboard)

• Total storage used (bytes)
• Storage quota usage percentage
• Number of buckets
• Number of objects
• Per-bucket statistics

Logging

• Structured Logging - JSON format with correlation IDs
• Log Levels - Debug, Info, Warn, Error
• Correlation IDs - Trace requests across log entries

Security Considerations

Signature Validation

• Timestamp Verification - Rejects requests older than 15 minutes
• HMAC-SHA256 - Cryptographically secure signature algorithm
• Payload Hash - Validates request body integrity (conditional for proxied requests)
• Constant-Time Comparison - Prevents timing attacks

Proxy Detection

The server detects reverse proxies (Cloudflare, Traefik, nginx) via headers:

• CF-Ray - Cloudflare request ID
• X-Forwarded-For - Generic proxy header
• X-Real-IP - Traefik/nginx real client IP

When detected, the server adjusts validation (skips strict payload hash checks) to accommodate proxy modifications.

Session Security

• Secure Cookies - HttpOnly, SameSite flags
• CSRF Protection - Token validation for state-changing operations
• bcrypt Passwords - Hashed with cost factor 10
• Session Expiration - Configurable TTL (default 24h)

Rate Limiting

• Configurable events per second (MAX_EVENTS_PER_SEC, default: 100)
• Burst size limit (MAX_BURST_SIZE, default: 120)
• Per-IP rate limiting

Troubleshooting

Common Issues

Issue	Cause	Solution
SignatureDoesNotMatch	Incorrect credentials or timestamp skew	Verify credentials, check system clock, ensure region matches (us-east-1)
Upload fails with 502 error	File exceeds Cloudflare limit (100MB)	Use direct connection, Cloudflare Workers, or paid plan
Cannot login to Web UI	Wrong username/password	Check LOCAL_AUTH_USERNAME and LOCAL_AUTH_PASSWORD
Magika model not found	MAGIKA_ASSETS_DIR incorrect	Verify path points to magika/assets directory
Request time too skewed	Client/server clock difference > 15min	Sync system clocks using NTP

Debug Mode

Enable debug logging and Swagger UI for development:

./s3-storage -d -s -r local -p 8080

• -d - Development mode (verbose logging)
• -s - Enable Swagger UI at /swagger/index.html
• -r local - Local telemetry endpoint

API Documentation

Swagger/OpenAPI

Interactive API documentation is available at:

• Local Development: http://localhost:8080/swagger/index.html
• Requires: Start server with -s or -d flag
• Format: OpenAPI 3.0 specification

Additional Tools

meta-decoder

Command-line tool for inspecting metadata files:

# Decode bucket metadata
meta-decoder -meta /data/.meta

# Decode object metadata
meta-decoder -xl /data/bucket/object/xl.meta

# Decode share links
meta-decoder -share /data/.sharelinks

# JSON output only
meta-decoder -meta /data/.meta -json | jq '.'

See src/test/tools/meta-decoder/Readme.md for full documentation.

System Requirements

Runtime Dependencies

Dependency	Version	Purpose
ONNXRUNTIME	1.23.2	Machine learning inference engine (for Magika)
Magika Assets	standard_v3_3	ML model for content-type detection

Supported Platforms

• macOS Intel (x86_64)
• macOS Apple Silicon (arm64)
• Linux x64 (x86_64)
• Linux ARM64 (aarch64)

Recommended Resources

Resource	Minimum	Recommended
CPU	1 core	2+ cores
RAM	512 MB	2 GB+
Disk	100 MB (app)	Depends on storage needs