Create the missing Kinesis Data Firehose
The Analytics Team has complained that even though the application is running, there is no data being sent to their data lake’s staging area. Let’s take another look at the Alien Attack architecture diagram:
According to this diagram, the Kinesis Data Stream sends the data through a Kinesis Data Firehose to a bucket in S3. The Analytics Team has appended the following information: “We have already created the S3 bucket to store this data in our AWS account. It has the suffix ‘raw’”.
Action Item: before we move on, try to navigate to your S3 console and find the bucket that the Analytics Team is referring to. Click the checkbox to the left of the bucket, select Copy Bucket ARN and paste the ARN to a separate text editor file.
The team said that this fix should be pretty simple: “All you need to do is connect the Kinesis Data Firehose to our existing Kinesis Data Stream. If the Kinesis Data Firehose doesn’t exist, create one! Grant us access to the environment and we can help, or call us if you need”.
Let’s get started.
What are we fixing? Fix or create a Kinesis Data Firehose so that it is properly sending data from our Kinesis Data Stream to the Analytics Team’s S3 bucket. We are currently missing a mechanism to do this within our AWS architecture.
Hint: Click here to see a diagram of your broken architecture.
- From your AWS Management Console, navigate to the Amazon Kinesis Console (note that this is separate from the Kinesis Video console). Make sure you are still in the same region you chose at the beginning of this workshop.
- Inside of the Amazon Kinesis dashboard, you’ll see a panel for Kinesis data streams on the left and a panel for Kinesis Firehose delivery streams on the right.
Do you see any Kinesis Firehose Delivery Streams for your Alien Attack environment?
It looks like our application doesn’t have any Kinesis Firehose Delivery Streams built, so let’s create one:
- In Kinesis Firehose Delivery streams panel, click Create Delivery Stream.
- Configure your delivery stream name and source:
- Delivery stream name: YourEnvironmentName_Firehose (If our environment name is alienenv123456, then the stream name should be ALIENENV123456Firehose. Note the capitalization.)
- Source: Kinesis Data Stream
- Choose Kinesis stream: YourEnvironmentName_InputStream
- Click Next. Decide how your delivery stream will process records:
- Record transformation: Disabled
- Record format conversion: Disabled
- Click Next. Let’s choose a destination for our delivery stream. In this case, our destination is the raw data S3 bucket created by our Analytics Team.
- Destination: Amazon S3.
- S3 bucket: Select the bucket attached to your application. The name will have the form .raw.
- Click Next. Configure your settings:
- Buffer size:
- Buffer interval:
- S3 compression: GZIP
- S3 encryption: Disabled
- Error logging: Enabled
- IAM Role: Click on the button Create new or choose. An IAM configuration page will open.
- IAMRole: Leave the option Create a new IAM Role selected.
- Role Name:
YourEnvironmentName_FirehoseRole (Replace “YourEnvironmentName” with the Environment Name you’ve been using.)
- Select Allow. You will be taken to the previous page.
- Click Next. Review your configuration. Click Create delivery stream.
If the stream was properly created, you should see it appear in your Kinesis Dashboard.
Stuck? Click here for a Fast Fix
- From the Cloud9 Console, navigate to the cdk folder inside of alienattack.workshop and search all .ts (typescript) files for “MISSING KINESIS FIREHOSE” by right-clicking on the cdk folder and selecting Search in this Folder. Follow the commented out guidance in each file to adjust your code.
- Save all files that you changed and run
cdk diff -c envname=$envname -c sessionparameter=true -c kinesisintegration=true -c firehose=true in the terminal window. This will show you what will be changed in your environment.
- Confirm these changes and run
cdk deploy -c envname=$envname -c sessionparameter=true -c kinesisintegration=true -c firehose=true to deploy the changes.