跳到主要内容

Kafka AWS IAM Access Control

This page describes how to configure IAM access control when connecting to AWS MSK.

Overview​

AWS MSK is a managed Kafka service provided by Amazon. AWS provides IAM as a unified access control mechanism that allows resources within an AWS account to grant permissions to each other. BladePipe has built-in support for AWS IAM access control. This document explains how to configure it.

Procedure​

信息

When you connect to AWS MSK using IAM, the built-in ACL rules in AWS MSK do not take effect. Permissions are determined by IAM policies/roles.

Enable IAM access control on AWS MSK​

Make sure IAM authentication is enabled in the MSK cluster configuration and that the corresponding IAM role/user has the required permissions. For reference:

Get the Bootstrap Server​

In the AWS MSK console, navigate to the cluster. The client connection endpoint is the Bootstrap Server.

Configure BladePipe​

BladePipe supports both default credentials and named credentials for IAM access control.
With named credentials, you can connect to multiple MSK clusters with different identities on the same Worker.

  1. Create a Kafka / AWS MSK DataSource, and fill in:
    • Host: enter the Bootstrap Server you got previously
    • Authentication: select None
    • Extra Info: only modify the value of customClientProps to the following JSON:
信息

Parameter customClientProps: Custom properties passed to the Kafka client in JSON format. This setting takes the highest priority. If the setting duplicates another parameter, the value in customClientProps takes precedence.

{
    "security.protocol": "SASL_SSL",
    "sasl.mechanism": "AWS_MSK_IAM",
    "sasl.jaas.config": "software.amazon.msk.auth.iam.IAMLoginModule required;",
    "sasl.client.callback.handler.class": "software.amazon.msk.auth.iam.IAMClientCallbackHandler"
}
  1. Click Add DataSource.
  2. Test the connection on the DataSource list page.
信息

When you use this method, credentials are not provided or managed by BladePipe. They are handled by the AWS SDK Default Credential Provider Chain. The credential provider chain is:

  1. Environment variables: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.
  2. Java system properties: aws.accessKeyId and aws.secretKey.
  3. Web identity token credentials from the environment or container.
  4. Default credentials file, usually located in the ~/.aws/credentials (path may vary by platform), shared by AWS SDKs and the AWS CLI.
  5. Amazon ECS container credentials, loaded when AWS_CONTAINER_CREDENTIALS_RELATIVE_URI is set.
  6. EC2 instance profile credentials, provided by the Amazon EC2 metadata service.