Deployment

Step-by-step

Step 1: Log in to your AWS account and navigate to the AWS Marketplace.

Step 2: Search for "sparkpipe" or go directly to the SparkPipe AWS Marketplace page.

Step 3: Click on Continue to Subscribe.

Step 4: Click on Continue to Configuration.

Step 5: Select the desired region and software version (latest recommended), then click on Continue to Launch.

Step 6: Select the desired EC2 instance type, subnet settings, security group (if unsure, keep the defaults) and a pre-existing SSH key pair. Then click Launch.

Step 7: Navigate to the EC2 dashboard of your selected region and securely access the instance using EC2 Instance Connect.

The Configuration section will guide you on how to configure the SparkPipe service and its modules.

AWS CLI

After subscribing to the product as shown in the section above (steps 1-3), you can deploy SparkPipe using the following commands:

1. Create a new security group and to allow inbound SSH access :

aws ec2 create-security-group \
    --group-name sparkpipe-sg \
    --description "SparkPipe"

aws ec2 authorize-security-group-ingress \
    --group-name sparkpipe-sg \
    --protocol tcp \
    --port 22 \
    --cidr 0.0.0.0/0

2. Retrieve the latest AMI available in your region:

aws ec2 describe-images \
    --owners aws-marketplace \
    --filters "Name=product-code,Values=your-product-code" \
              "Name=state,Values=available" \
    --query 'Images | sort_by(@, &CreationDate) | [-1].ImageId' \
    --output text

3. Launch an instance using the latest SparkPipe AMI:

aws ec2 run-instances \
    --image-id <AMI_ID> \
    --count 1 \
    --instance-type m6g.medium \
    --key-name my-key-pair \
    --security-groups sparkpipe-sg

4. SSH into the instance using EC2 Instance Connect:

aws ec2-instance-connect ssh --instance-id <INSTANCE_ID>