Koroid Corporation Joins Google Cloud Startup Program


In this comprehensive guide, we will walk you through the process of setting up an AWS ECS (Elastic Container Service) environment with essential components such as CloudWatch Logs, CloudTrail, S3 Buckets, and KMS (Key Management Service) integration. By following this step-by-step tutorial, you will be able to configure a robust ECS setup that incorporates advanced logging, auditing, secure storage, and encryption capabilities.

## Prerequisites

1. Install or update AWS CLI to the latest version by following the guide [here](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html).
2. Verify that the AWS CLI is installed correctly by running `aws --version` in your terminal.

## Let’s Go…

To set up ECS Exec with your current ECS setup using Fargate, you'll need to perform the following steps:

1. **Install and Configure the SSM Agent**: ECS Exec requires the AWS Systems Manager (SSM) agent to be installed on the ECS instances. The SSM agent allows you to run commands against the instances using the ECS Exec functionality. You can install and configure the SSM agent manually on your ECS instances or use an automation tool like AWS Systems Manager Run Command or AWS Systems Manager State Manager.

   1. Verify that the SSM plugin is installed correctly by running `aws ssm help` in your terminal.

      [Install the Session Manager plugin for the AWS CLI - AWS Systems Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html)

2. **Set up KMS**: ECS Exec requires an AWS Key Management Service (KMS) key for encryption. If you don't have one already, create a KMS key through the AWS Management Console or using the AWS CLI.

   1. **Note:** It may be helpful to create a user and role for your KMS keys with no policies at this point.

   2. **KMS Key Configuration**: Make sure you have a KMS key created. If you haven't created one yet, follow the AWS documentation to create a KMS key. Once created, note down the KMS key ARN.

      1. To add a KMS key to your ECS cluster, you need to update the task execution role associated (**`ecsTaskExecutionRole`**) with your cluster. Here's how you can do it:

         1. Open the AWS Management Console and navigate to the Amazon ECS service.
         2. Select your ECS cluster from the list.
         3. In the left navigation pane, click on "Settings".
         4. Under the "Permissions" section, locate the "Task execution IAM role" and click on the role name.
         5. This will take you to the IAM console, specifically to the IAM role associated with your task execution role.
         6. In the IAM role details page, click on "Add inline policy" under the "Permissions" tab.
         7. On the "Create Policy" page, select the "JSON" tab and enter the following JSON policy, replacing **`your-kms-key-arn`** with the ARN of your KMS key:

            ```json
            {
              "Version": "2012-10-17",
              "Statement": [
                {
                  "Effect": "Allow",
                  "Action": ["kms:Decrypt", "kms:Encrypt", "kms:GenerateDataKey"],
                  "Resource": "your-kms-key-arn"
                }
              ]
            }
            ```

3. **Update Task Definitions**: Update your task definitions to include the necessary configurations for ECS Exec. You'll need to modify the task definition to add the `FargatePlatformConfiguration` section and set the `platformVersion` to `1.4.0` or later. Also, ensure that your task execution role has the necessary permissions to interact with S3, CloudTrail, and KMS.

   1. Add `Task role` for execution on container (must be combined with elastic cache or other AWS programmatic access permissions).

      ```json
      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Action": [
              "ssmmessages:CreateControlChannel",
              "ssmmessages:CreateDataChannel",
              "ssmmessages:OpenControlChannel",
              "ssmmessages:OpenDataChannel"
            ],
            "Resource": "*"
          }
        ]
      }
      ```

      ```json
      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Action": ["kms:Decrypt"],
            "Resource": "your-kms-key-arn"
          }
        ]
      }
      ```

      ```json
      				{
                  "Effect": "Allow",
                  "Action": "s3:PutObject",
                  "Resource": "arn:aws:s3:::<ECS_EXEC_BUCKET_NAME>/*"
              },
              {
                  "Effect": "Allow",
                  "Action": "s3:GetEncryptionConfiguration",
                  "Resource": "arn:aws:s3:::<ECS_EXEC_BUCKET_NAME>"
              }
      ```

      ```json
      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Action": ["logs:DescribeLogGroups"],
            "Resource": "*"
          },
          {
            "Effect": "Allow",
            "Action": ["logs:CreateLogStream", "logs:DescribeLogStreams", "logs:PutLogEvents"],
            "Resource": "arn:aws:logs:<region>:<account-id>:log-group:<path>:*"
          }
        ]
      }
      ```

4. **Configure Audit Logging**: ECS Exec requires audit logs to be enabled in CloudTrail. If you haven't done so already, enable CloudTrail and ensure it's configured to log API events for ECS.

   1. **Audit Logging Configuration**: Enable audit logging for your ECS cluster using CloudTrail. Ensure that CloudTrail is properly set up and configured to capture ECS API events.

      1. This policy allows the CloudTrail service (**`cloudtrail.amazonaws.com`**) to perform the **`kms:GenerateDataKey`** and **`kms:Decrypt`** actions on the specified KMS key (**`your-kms-key-arn`**).
      2. Remember to review and adjust the policy based on your specific requirements and resource configurations. Additionally, you may need to include other statements or conditions depending on your use case and desired access control.
      3. Might need to add s3 bucket to `Key Policy`
      4. Add the following to the `Key Policy` for you new KMS key

         ```json
         {
           "Version": "2012-10-17",
           "Id": "key-policy-id",
           "Statement": [
             {
               "Sid": "Allow CloudTrail to GenerateDataKey and Decrypt",
               "Effect": "Allow",
               "Principal": {
                 "Service": "cloudtrail.amazonaws.com"
               },
               "Action": ["kms:GenerateDataKey", "kms:Decrypt"],
               "Resource": "your-kms-key-arn"
             }
           ]
         }
         ```

5. **Create an S3 Bucket for Container Logs**: Set up an S3 bucket to store the container logs generated by ECS Exec. You can create a new bucket using the AWS Management Console or the AWS CLI. Make sure to configure appropriate permissions and access controls for the bucket.
6. **Launch or Update Your ECS Cluster**: Launch or update your ECS cluster with the updated task definitions. This will ensure that new tasks launched on your cluster have the necessary configurations for ECS Exec.

Once you've completed these steps, you should be able to use ECS Exec locally by following the official documentation provided by AWS. This typically involves configuring the AWS CLI, setting up session tokens, and running the `ecs exec` command to establish a session with the container running in your ECS task.

Remember to refer to the AWS documentation for detailed instructions on setting up ECS Exec and configuring the necessary components for your specific environment.

Run https://github.com/aws-containers/amazon-ecs-exec-checker to validate setup (may need to run `brew install jq` on mac)

https://github.com/aws-containers/amazon-ecs-exec-checker

Please note that it may take some time for the ECS service to propagate the configuration changes and for the **`amazon-ecs-exec-checker`** tool to reflect the updated status.

Update task to enable service, example below:

```bash
aws ecs update-service \
    --cluster <cluster> \
    --service <service> \
    --task-definition <task-definition> \
    --enable-execute-command \
    --force-new-deployment
```

## Testing the exec command.

We can use either `sh` or `bash` in most Docker instances, provided we are running Linux, to test our exec command. If you are logged into the container shell, it worked!

Note: Container name can be found in the AWS ECS cluster via the task section.

```bash
aws ecs execute-command \
    --cluster <cluster> \
    --task <task-id> \
    --container <container> \
    --interactive \
    --command "sh"
```

![Screenshot 2023-05-31 at 3.43.11 PM.png](/news-page/articles/set_up_ecs_exec_custom_logging_with_kms/Success-1.png)

Initial Success! But we still need to update our configuration to use the KMS key.

## Update Task Role Permissions

Update the cluster to allow for the new logging configuration with KMS to provision.

```bash
aws ecs update-cluster \
    --cluster <cluster> \
    --region <region> \
    --configuration executeCommandConfiguration="{logging=OVERRIDE,\
        kmsKeyId='<your-kms-key-arn>',\
        logConfiguration={cloudWatchLogGroupName="/ecs/<path>",\
                        s3BucketName='<S3-bucket>',\
                        s3KeyPrefix=exec-output}}"

```

![Screenshot 2023-05-31 at 10.17.21 PM.png](/news-page/articles/set_up_ecs_exec_custom_logging_with_kms/Success-2.png)

Configuring CloudTrail

1. Go to → Dashboard → Create Trail
2. Add a trail name
3. Use the existing S3 bucket that you created in the previous step to store logs
4. Keep Log file encryption on using KMS
5. Select the existing KMS key used for your set up
6. Enable cloudwatch logs
   1. Use an existing log group name of `/ECS/<container-name>`
   2. Create a new role for your cloud trail `CloudTrailRoleForCloudWatchLogs-<container-name>`
7. Create logger

For reference the cloud trail role for cloud watch logs policy is below

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AWSCloudTrailCreateLogStream2014110",
      "Effect": "Allow",
      "Action": ["logs:CreateLogStream"],
      "Resource": [
        "arn:aws:logs:<REGION>:<ACCOUNT-ID>:log-group:/ecs/<PROJECT>:log-stream:<CLOUDTRAIL-RESOURCE>*"
      ]
    },
    {
      "Sid": "AWSCloudTrailPutLogEvents20141101",
      "Effect": "Allow",
      "Action": ["logs:PutLogEvents"],
      "Resource": [
        "arn:aws:logs:<REGION>:<ACCOUNT-ID>:log-group:/ecs/<PROJECT>:log-stream:<CLOUDTRAIL-RESOURCE>*"
      ]
    }
  ]
}
```

## Sweet Success!

By following this guide, you have successfully set up an AWS ECS environment with comprehensive logging using CloudWatch Logs, auditing capabilities with CloudTrail, secure storage using S3 buckets, and encryption through KMS integration. This setup ensures enhanced visibility, compliance, and security for your ECS containerized applications. Feel free to explore additional features and advanced configurations to further optimize your ECS deployment.

Remember to refer to the official AWS documentation for detailed instructions and best practices, as well as to keep your setup up to date with the latest security recommendations and enhancements.

#### Guides / More info

[Using Amazon ECS Exec for debugging - Amazon Elastic Container Service](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-exec.html)

[Configure AWS KMS key policies for CloudTrail - AWS CloudTrail](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/create-kms-key-policy-for-cloudtrail.html?icmpid=docs_cloudtrail_console)

[Getting Started with ECS Exec for ECS Fargate.](https://deniskolpakov.com/posts/ecs-exec/)

[Use AWS ECS execute command to access a container on AWS Fargate and EC2](https://towardsthecloud.com/amazon-ecs-execute-command-access-container)

[Using the Amazon ECS command line interface - Amazon Elastic Container Service](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_CLI.html)


How to set up AWS Elastic Container Service (ECS) with s3 logging and KMS token


# Introduction:

In an era defined by rapid technological advancement and an increasing reliance on data-driven decisions, harnessing the power of machine learning for predictive analytics is not just advantageous—it’s essential. Many organizations are turning to robust cloud platforms to process vast amounts of data to forecast future trends accurately. Among these platforms, Amazon Web Services (AWS) stands out as a leader in providing accessible, efficient, and powerful tools that democratize machine learning capabilities for companies of various sizes and sectors. This article delves into our organization's recent experience with AWS Forecast, detailing how we seamlessly integrated this tool into our operations to enhance our predictive analytics capabilities. Our journey exemplifies how even complex technologies can be rapidly deployed to meet immediate business needs and improve decision-making processes at all levels.

## Implementing AWS Forecast: Enhancing Decision-Making with Predictive Analytics

As organizations delve into the world of big data, the ability to anticipate future trends becomes a significant asset. With Amazon Web Services’ (AWS) machine learning tools, this capacity for predictive analytics is no longer a distant dream, but a present reality. Our organization recently embarked on an ambitious journey, successfully implementing AWS Forecast into our operational model within a surprisingly short two-week period. Here’s a look at our incredible journey, the immediate benefits experienced by our end-users, and our plans for the future.

Our objective was straightforward yet profound – to incorporate accurate forecasting into our application to enhance decision-making capabilities of our users. We knew that our users could benefit greatly from predictive analytics, allowing them to optimize shift allocations and make informed, long-term organizational decisions.

We took our existing data from MongoDB and fed it into AWS Forecast. This process allowed us to generate an array of matrices based on predictive models. The simplicity and speed of this integration were truly astounding; we were able to implement this within just a week and a half, highlighting the user-friendly nature of AWS services.

The next step in our process was to save the output matrices array back into MongoDB. These arrays formed the foundation for our prediction engine, enabling us to generate insights into user availability and suitability for upcoming shifts. Every night, our system runs predictions and saves the results, forming a detailed and invaluable picture of potential future scenarios.

Moreover, we implemented a system to archive our daily predictions into an AWS S3 bucket. This not only provides a backup for our data but also serves as a growing bank of historical data points. These data points will undoubtedly be a potent resource for our future development initiatives.

The impact of this AWS Forecast implementation on our end-users was immediate and profound. Users now have access to predictive insights that help them plan better, work more efficiently, and make more informed, long-term decisions for their organizations. The feedback has been overwhelmingly positive, emphasizing the power and potential of predictive analytics.

While we have already achieved significant milestones, our journey with AWS Forecast is far from over. In our future endeavors, we aim to generate multiple predictions on the fly, using these matrices arrays. By comparing these predictions, we aim to determine the best possible outcome based on user actions. We believe this will further enhance the decision-making capabilities of our users, driving their organizations towards greater success.

In summary, our experience with AWS Forecast has been transformative. Within a short span of two weeks, we were able to enhance our application’s capabilities significantly, resulting in happy users and more efficient organizations. As we move forward, we remain excited about the possibilities that this powerful AWS service can offer. After all, the future belongs to those who can predict it!

# Conclusion:

The successful integration of AWS Forecast into our operational model marks a significant milestone in our journey towards becoming a more data-informed and efficient organization. The implementation process, although swift, laid a robust foundation for us to build upon as we continue to refine and expand our predictive analytics capabilities. Our experience has demonstrated the immense potential of machine learning tools in transforming data into actionable insights that propel organizational success. As we look to the future, we are inspired by the possibilities that lie ahead. With continued innovation and improvement in our use of AWS Forecast, we are well-positioned to remain at the forefront of our industry, leveraging the power of predictive analytics to forge a path to sustained growth and enhanced decision-making. Through this narrative, we hope to inspire other organizations to embark on their own transformative journeys, unlocking the potential of their data to predict and effectively plan for the future.


Koroid Corporation Joins Google Cloud Startup Program

chevron_left

chevron_right