Deployment

Step-by-step

This section provides step-by-step instructions for deploying the SparkPipe AMI from the AWS Marketplace.

Step 1: Log in to your AWS account, then open the SparkPipe product page on the AWS Marketplace by clicking the following link→ https://aws.amazon.com/marketplace/pp/prodview-xiiaocgyyc3z2

Step 2: Click Try for free to start your 31-day trial.

Step 3: After subscribing to the product, click on Launch your software.

Step 4: Select Amazon EC2 as the service, then choose the “One-click launch from AWS Marketplace” launch method (recommended for simplicity). Next, select your target Region and Instance type.

Step 5: Create or select an existing VPC, Subnet, Security group and Key pair. Then click Launch to deploy your instance.

Step 6: After launch is successful, Click View instance on EC2.

Step 7: Connect to your EC2 instance using either an SSH client or the web-based EC2 Instance Connect service.

Note

When connecting to the EC2 instance, make sure to use ec2-user as the username.

Step 8: After successfully connecting to the instance, navigate to the /etc/sparkpipe directory to configure the service.

Note

The Configuration section provides detailed instructions on how to configure the SparkPipe service and its modules.

AWS CLI

After subscribing to the product in the AWS Marketplace, you can deploy SparkPipe by running the following commands using the AWS CLI:

Step 1: Create a new security group and allow inbound SSH access:

aws ec2 create-security-group \
    --group-name sparkpipe-sg \
    --description "SparkPipe"

aws ec2 authorize-security-group-ingress \
    --group-name sparkpipe-sg \
    --protocol tcp \
    --port 22 \
    --cidr 0.0.0.0/0

Step 2: Retrieve the latest SparkPipe AMI available in your region:

aws ec2 describe-images \
    --owners aws-marketplace \
    --filters "Name=product-code,Values=your-product-code" \
              "Name=state,Values=available" \
    --query 'Images | sort_by(@, &CreationDate) | [-1].ImageId' \
    --output text

Step 3: Launch an instance using the latest SparkPipe AMI:

aws ec2 run-instances \
    --image-id <AMI_ID> \
    --count 1 \
    --instance-type m6g.medium \
    --key-name my-key-pair \
    --security-groups sparkpipe-sg

Step 4: SSH into the instance using EC2 Instance Connect:

aws ec2-instance-connect ssh --instance-id <INSTANCE_ID>