Open in app

Sign in

Write

Sign in

AWS S3 or Minio Storage: Access Storage using SDK and Generate Signature for Direct Access without SDK

Sandeep Kumar
9 min readAug 12, 2022

Introduction

AWS provides SDKs to use with different programming languages e.g. Java, Python, NodeJs, GO etc. These can be used to access the S3 Storage as well as Minio Storage because Minio supports S3 SDKs.

Apart from the SDKs, REST based API can be used for the languages where SDKs are not available e.g. JSR223 in JMeter or in situation when SDKs cannot be used.

AWS support ‘signature 4’ version which requires to generate signature using request path, headers and payload data. HmacSHA256 algorithm is used to generate signature using the secret key of the S3 storage.

Problem

The problem is very simple, write code to access S3 or Minio Storage for download and upload file into storage.

Use SDK where SDKs are available else use REST based communication with generated AWS Signature S4.

Solution

There are different solutions for different programming languages. For S3 storage and Minio Storage when account created we will have following:

When S3 service created, an URL generated by AWS as,

https://<accountName>.<s3 domain name>private key = access key or sometimes called account namesecret key = secret keysession token = used when temporary access needed, it is optional and only used for temporary access

Apart from URL, private key, secret key we need to create a bucket in storage.

Java

To access S3 or Minio Storage, we need private key, secretkey and optional session token.

AWS Provides two SDKs in Java for accessing S3 or Minio Storage:

A. Async Client

Add maven managed dependency:

<dependencyManagement>
     <dependencies>       <dependency>
           <groupId>software.amazon.awssdk</groupId>
           <artifactId>bom</artifactIs>
           <version>2.17.210</version>
           <type>pom</type>
           <scope>import</scope>
       </dependency>     </dependencies>
</dependencyManagement>

Create Async Client:

String url = "http://localhost:9000";
String accessKey = "accessKey";
String secretKey = "secretKey";S3AsyncClient s3 = S3AsyncClient
   .builder()
   .endpointOverride(URI.create(url))
   .region(Region.AP_EAST_1)
.credentialsProvider(StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)))
.build();

Upload file with Async Client:

String conentType = "image/jpeg";
String bucketName = "bucket";
String directoryAndFilename = "core/62d63ab70789ad38270c06bb";
String localFileToBeUploaded = "1.jpg";PutObjectRequest request = PutObjectRequest
.builder()
.contentType(conentType)
.bucket(bucketName)
.key(directoryAndFilename)
.build();CompletableFuture<PutObjectResponse> cf =  s3.putObject(request,    AsyncRequestBody.fromFile(new File(localFileToBeUploaded)));try {   
   System.out.println(cf.get().eTag());  
} catch (InterruptedException | ExecutionException e) {   
   e.printStackTrace();  
}

Download file with Async Client:

String bucketName = "bucket";
String directoryAndFilename = "core/62d63ab70789ad38270c06bb";
String pathWhereFileToBeDownloaded = "1.jpg";CompletableFuture<GetObjectResponse> cf = s3.getObject(GetObjectRequest
.builder()
.bucket(bucketName)
.key(directoryAndFilename)
.build(), Paths.get(pathWhereFileToBeDownloaded));
try {   
   System.out.println(cf.get().tagCount());  
} catch (InterruptedException | ExecutionException e) {     
   e.printStackTrace();  
}

Copy file with Async Client:

String sourceBucketName = "source-bucket";
String sourceDirectoryAndFilename = "source/core/62d63ab70789ad38270c06bb";String destBucketName = "dest-bucket";
String destDirectoryAndFilename = "dest/core/62d63ab70789ad38270c06bb";CompletableFuture<CopyObjectResponse> cf = s3.copyObject(CopyObjectRequest    
.builder()    
.destinationBucket(destBucketName)    
.destinationKey(destDirectoryAndFilename)    
.sourceKey(sourceDirectoryAndFilename)    
.sourceBucket(sourceBucketName)    
.build());  
try {   
   System.out.println(cf.get());  
} catch (InterruptedException | ExecutionException e) {   
   e.printStackTrace();  
}

Example Link: https://github.com/siddhivinayak-sk/BaseTest-1/blob/localcheckout/src/main/java/aws/s3/S3MinIOTest.java

B. Sync Client

Add maven managed dependency:

<dependencyManagement>
     <dependencies>        <dependency>
          <groupId>com.amazonaws</groupId>
          <artifactId>aws-java-sdk-bom</artifactId>
          <version>1.11.79</version>
          <type>pom</type>
          <scope>import</scope>
        </dependency></dependencies>
</dependencyManagement>

Create Sync Client:

private static AmazonS3 createClient(String url, String region, String key, String secret, String token, Log4jLogger log) {
		if(null == token) {
			return AmazonS3ClientBuilder
					.standard()
					.withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials(key, secret)))
					.withEndpointConfiguration(new EndpointConfiguration(getURLWithoutTenant(url, log), region))
					.build();
		} else {
			return AmazonS3ClientBuilder
					.standard()
					.withCredentials(new AWSStaticCredentialsProvider(new BasicSessionCredentials(key, secret, token)))
					.withEndpointConfiguration(new EndpointConfiguration(getURLWithoutTenant(url, log), region))
					.build();
		}
	}

Upload file with Sync Client:

public static String uploadFile(String url, String region, String key, String secret, String token, String remoteFilePath, String localFilePath, Log4jLogger log) {
		logDetails(url, region, key, secret, token, remoteFilePath, localFilePath, log);
		try {
			PutObjectResult result = createClient(url, region, key, secret, token, log).putObject(getTenant(url, log), remoteFilePath, new File(localFilePath));
			if(null != log)
				log.info("File: " + remoteFilePath + "is has been uploaded");
			return result.getETag() + ", Status: 200";
		}
		catch(Exception ex) {
			if(null != log)
				log.error("Error while upload file: " + remoteFilePath, ex);
			ex.printStackTrace();
			return "400";
		}
	}

Download file with Sync Client:

public static String downloadFile(String url, String region, String key, String secret, String token, String remoteFilePath, String localFilePath, Log4jLogger log) {
		logDetails(url, region, key, secret, token, remoteFilePath, localFilePath, log);
		try {
			GetObjectRequest request = new GetObjectRequest(getTenant(url, log), remoteFilePath);
			createClient(url, region, key, secret, token, log).getObject(request, new File(localFilePath));
			if(null != log)
				log.info("File Created for key: " + remoteFilePath + " at " + localFilePath);
			return "200";
		} catch(Exception ex) {
			if(null != log)
				log.error("Error while download file: " + remoteFilePath, ex);
			return "400";
		}
	}

Example Link: https://raw.githubusercontent.com/siddhivinayak-sk/BaseTest-1/localcheckout/src/main/java/aws/s3/S3UploadUtil.java

Note: However, S3 SDK can be used for Minio storage management, Minio provides a separate API for storage management.

Maven Dependency:

<dependency>
<groupId>io.minio</groupId>
<artifactId>minio</artifactId>
<version>8.3.3</version>
</dependency>

Link for full code: https://raw.githubusercontent.com/siddhivinayak-sk/BaseTest-1/localcheckout/src/main/java/minio/MinioTest.java

NodeJS

Easiest way to upload, download file with S3 is to generate Signature and use with REST API call.

We must have:

  1. access key
  2. secret key
  3. storage URL
  4. bucket

To generate signature for download or upload, NPM has a API package named aws4 .

Link for NPM: https://www.npmjs.com/package/aws4

Add this into package.json :

{
  "dependencies": {
    "aws4": "^1.11.0",
    "https": "^1.0.0",
    "crypto": "^1.0.1"
  }
}

To download and install, use below command:

npm i aws4

Following parameters are used for generating signature:

  • host: host of service, mandatory
  • path: Path of the file being uploaded, mandatory = https://<host.domain>/<bucket name>/<directories>/<file name>
  • service: Service name e.g. s3
  • region: Region name e.g. us-east-1
  • method: HTTP method e.g. GET, PUT
  • accessKeyId: Access Key ID for the service, mandatory
  • secretAccessKey: Secret for access key id, mandatory
  • sessionToken: Temporary session token, optional

Use below code to upload a file:

var https = require('https')
var aws4  = require('aws4')
var crypto = require('crypto');
var fs = require('fs');

var fileBuffer = fs.readFileSync('1.jpg'); //File name from local which need to upload
var hashSum = crypto.createHash('sha256'); 
hashSum.update(fileBuffer);
var hex = hashSum.digest('hex'); //Generate SHA256 from the file


var opts = aws4.sign({
  host: '<host name for the s3 service>',
  path: '<bucket and file path in s3>',
  service: 's3',
  region: 'us-east-1',
  method: 'PUT',
  
  headers: {
    'X-Amz-Content-Sha256': hex
  },
  body: undefined
}, { 
  accessKeyId: '<access key>', 
  secretAccessKey: '<secret key>'
  sessionToken: '<session token>'
}
)

opts.path = '<complete path: https://host+bucket+filepath>';
opts.headers['Content-Type'] = 'image/jpeg'; //Content type of the file
opts.headers['User-Agent'] = 'Custom Agent - v0.0.1'; //Agent name, optional
opts.headers['Agent-Token'] = '47a8e1a0-87df-40a1-a021-f9010e3f6690'; // Agent unique token, optional
opts.headers['Content-Length'] = fileBuffer.length; //Content length of the file being uploaded

console.log(opts) //It will print generated Signature

var req = https.request(opts, function(res) {
    
  console.log('STATUS: ' + res.statusCode);
  console.log('HEADERS: ' + JSON.stringify(res.headers));
  res.setEncoding('utf8');

  res.on('data', function (chunk) {
    console.log('BODY: ' + chunk);
  });
  
});

req.on('error', function(e) {
  console.log('problem with request: ' + e.message);
});

req.write(fileBuffer);
req.end();

Note: The SHA256 is generated manually and passed into the argument before generating signature and body set as undefined in aws4.sign() method. This important when uploading a file as binary data therefore, SHA256 is generated and set before aws4.sign() method call.

This same API can be used for all different calls e.g. GET call for file download.

The sessionToken is optional as it is only required for the cases where temporary session token is generated for accessing S3 service.

This API can be used as JS with browser.

JSR223

In case of using JMeter, it is quite easy to use JSR223 Preprocessor for generating Signature.

There are two type of credential you can use while upload:

  • Service credential which are combination of: accessKey, secretKey, serviceName and region
  • Temporary credential which are combination of: accessKey, secretKey, serviceName, sessionToken and region

In both of the cases you need to add JSR223 Preprocessor under the HTTP Request Sampler.

Add the below code to JSR223 Preprocessor and provide the required details:

import javax.crypto.Mac
import javax.crypto.spec.SecretKeySpec
import java.security.InvalidKeyException
import java.security.MessageDigest
import groovy.json.JsonSlurper
import java.text.SimpleDateFormat


//Defined in User Defined Variables
def access_key = <access key>
def secret_key = <secret key>
def host = <host>
def service = "s3"
def region = "us-east-1"
def token = <session token>
def localfile = <absolute path for local file>
def fileData = readBinaryFile(localfile)

//Create SHA256 of file data
def contentSha256 = org.apache.commons.codec.digest.DigestUtils.sha256Hex(fileData)
vars.put("aws_sha256", contentSha256)
vars.put("aws_content_length", fileData.length + "")


log.info("access key: " + access_key)
log.info("secret_key: " + secret_key)
log.info("host: " + host)
log.info("service: " + service)
log.info("region: " + region)
log.info("file: " + props.get("file_1.jpg"))
log.info("file sha256: " + contentSha256)
log.info("token: " + token)
log.info("content-length: " + fileData.length)


//Obtain data form the Http Request Sampler
def method = sampler.getMethod()
def url = sampler.getUrl()
def req_path = url.getPath()
def req_query_string = orderQuery(url)
def request_parameters = '';

sampler.getArguments().each {arg ->
    request_parameters = arg.getStringValue().substring(1)
}

//Create the variable x-amz-date 
def now = new Date()
def amzFormat = new SimpleDateFormat( "yyyyMMdd'T'HHmmss'Z'" )
def stampFormat = new SimpleDateFormat( "yyyyMMdd" )
amzFormat.setTimeZone(TimeZone.getTimeZone("UTC"));  //server timezone
def amzDate = amzFormat.format(now)
def dateStamp = stampFormat.format(now)
vars.put("x_amz_date", amzDate)


//Create a Canonical Request
def canonical_uri = req_path
def canonical_querystring = req_query_string
def canonical_headers = "host:" + host + "\n" + "x-amz-date:" + amzDate + "\n"
canonical_headers = "host:" + host + "\n" + "x-amz-content-sha256:" + contentSha256 + "\n" + "x-amz-date:" + amzDate + "\n" + "x-amz-security-token:" + token + "\n"


//def signed_headers = "host;x-amz-date"
def signed_headers = "host;x-amz-content-sha256;x-amz-date;x-amz-security-token"
def payload_hash = getHexDigest(request_parameters)
def canonical_request = method + "\n" + canonical_uri + "\n" + canonical_querystring + "\n" + canonical_headers + "\n" + signed_headers + "\n" + contentSha256

log.info("---------canonical_request-------: " + canonical_request)

//Create the String to Sign
def algorithm = "AWS4-HMAC-SHA256"
def credential_scope = dateStamp + "/" + region + "/" + service + "/" + "aws4_request"
def hash_canonical_request = getHexDigest(canonical_request)
def string_to_sign = algorithm + "\n" +  amzDate + "\n" +  credential_scope + "\n" +  hash_canonical_request


log.info("------string to sign-----: " + string_to_sign)


//Calculate the String to Sign

log.info("------datestamp to sign----: "+ dateStamp)

def signing_key = getSignatureKey(secret_key, dateStamp, region, service)

log.info("------key to sign----: "+ org.apache.commons.codec.digest.DigestUtils.sha256Hex(signing_key))

def signature = hmac_sha256Hex(signing_key, string_to_sign)

//Add Signing information to Variable
def authorization_header = algorithm + " " + "Credential=" + access_key + "/" + credential_scope + ", " +  "SignedHeaders=" + signed_headers + ", " + "Signature=" + signature
vars.put("aws_authorization", authorization_header)


def hmac_sha256(secretKey, data) {
    Mac mac = Mac.getInstance("HmacSHA256")
    SecretKeySpec secretKeySpec = new SecretKeySpec(secretKey, "HmacSHA256")
    mac.init(secretKeySpec)
    byte[] digest = mac.doFinal(data.getBytes())
    return digest
}

def hmac_sha256Hex(secretKey, data) {
    def result = hmac_sha256(secretKey, data)
    return result.encodeHex()
}

def getSignatureKey(key, dateStamp, regionName, serviceName) {
    def kDate = hmac_sha256(("AWS4" + key).getBytes(), dateStamp)
    def kRegion = hmac_sha256(kDate, regionName)
    def kService = hmac_sha256(kRegion, serviceName)
    def kSigning = hmac_sha256(kService, "aws4_request")
    return kSigning
}

def getHexDigest(text) {
    def md = MessageDigest.getInstance("SHA-256")
    md.update(text.getBytes())
    return md.digest().encodeHex()
}

public static String orderQuery(URL url) throws UnsupportedEncodingException {

    def orderQueryString = "";
    Map<String, String> queryPairs = new LinkedHashMap<>();
    String queryParams = url.getQuery();

    if (queryParams != null) {
        String[] pairs = queryParams.split("&");
    
        for (String pair : pairs) {
            int idx = pair.indexOf("=");
            queryPairs.put(URLDecoder.decode(pair.substring(0, idx), "UTF-8"), URLDecoder.decode(pair.substring(idx + 1), "UTF-8"));
        }
        def orderQueryArray = new TreeMap<String, String>(queryPairs);
        orderQueryString  = urlEncodeUTF8(orderQueryArray)
    }
    return orderQueryString;
}

public static String urlEncodeUTF8(String s) {
    try {
        return URLEncoder.encode(s, "UTF-8");
    } catch (UnsupportedEncodingException e) {
        throw new UnsupportedOperationException(e);
    }
}

public static String urlEncodeUTF8(Map<?,?> map) {
    StringBuilder sb = new StringBuilder();
    for (Map.Entry<?,?> entry : map.entrySet()) {
        if (sb.length() > 0) {
            sb.append("&");
        }
        sb.append(String.format("%s=%s",
            urlEncodeUTF8(entry.getKey().toString()),
            urlEncodeUTF8(entry.getValue().toString())
        ));
    }
    return sb.toString();       
}



public static byte[] readBinaryFile(String filePath) {
    File file = new File(filePath)
    byte[] binaryContent = file.bytes
    return binaryContent
}

Provide the below parameters into the preprocessor:

def access_key = <access key>
def secret_key = <secret key>
def host = <host>
def service = "s3" //is any specific name given for service else it will be s3 by default
def region = "us-east-1" //the required region
def token = <session token>
def localfile = <absolute path for local file>

Then add the HTTP Header Manager into the HTTP Request Sampler where add the below headers:

Header Name:    Value
X-Amz-Date: ${x_amz_date}
X-Amz-Security-Token: ${sessionToken}
X-Amz-Content-Sha256: ${aws_sha256}
Authorization: ${aws_authorization}
Content-Length: ${aws_content_length}
Content-Type: <file content type>
Host: ${host}

Here, below header’s value calculated dynamically when file comes to upload by the preprocessor:

  • X-Amz-Date
  • X-Amz-Content-Sha256
  • Authorization
  • Content-Length

Now, In HTTP Request sampler set the request method (normally PUT), protocol (e.g. https), server (…s3.service.com), port (e.g. 443), path (path of the file including bucket name).

Click on File Upload tab in HTTP Request Sampler, profile the absolute path of local file which need to be uploaded and provide the mime type.

Note: Do not provide parameter name (it will be blank as file will go in raw binary format) and Use multipart/form data checkbox should be unchecked as file is going in raw format.

Just to give a little more help, you can use JSR223 Sampler for making HTTP request with different file upload / download approach. E.g. Uploading dynamically calculated data as file. Refer the link: How to pass data from JSR223 sampler to Http Request sampler in JMeter

Note: The same logic for Signature generation can be used in any programming language and use it.

Conclusion

To access S3 or Minio Storage, AWS provides SDKs for different programming languages which can be utilized based on need. We mentioned Java based SDKs available for both Sync and Async client for storage.

Apart from SDKs, AWS support HmacSHA256 based signature from the secret key provided from S3. AWS support Signature S4 which can be generated by using pre-available package.

Apart from signature generation API, we can write own code for signature generation where it is not available e.g. JMeter.

References

--

--

Sandeep Kumar

Written by Sandeep Kumar

117 Followers

Sandeep Kumar holds Master of Computer Application, working as Technical Architect having 11+ years of working experience in banking, retail, education domains.

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams