Use Powershell to Upload Data to S3 Bucket
Before reading…
- Target Audience: This article is written for developers with beginner to intermediate experience. They are familiar with the tenets of software development and PowerShell. They are new to Amazon Web Services (AWS) and the Simple Storage Service (S3).
- Scenario: Assuming the developer is new to AWS, the document uses a conversational tone to have them through an terminate-to-terminate scenario. It gets them started with the various Amazon Spider web Services and provides footstep-by-stride details for authoring the PowerShell script.
- Sources: I wrote 100% of the content, including samples and PowerShell code, without an editor providing input. The information was derived from various Amazon documentation, pulled together to be a cohesive set of stop-to-end instructions.
…and here's the sample.
The ability to script of import tasks allows IT professionals to be efficient and constructive in their tasks. Backing upwards of import files to the deject or triggering the upload of new files to your site are disquisitional functions that tin be scripted to ameliorate efficiency and make the processes resilient.
In this document, you'll learn how to upload files to Amazon Simple Storage Service (S3) using PowerShell. You lot'll create an Amazon Spider web Services (AWS) account that will allow you to manage AWS users and groups via Identity and Access Management (IAM) as well as work with S3 buckets and folders. After installing the AWS Tools for Windows PowerShell, you'll configure the prerequisites for connecting to AWS via PowerShell. And finally, you'll utilise PowerShell to recursively iterate through directories to upload files to S3.
If yous're new to Amazon S3 and need to commencement from scratch, this is a beginning-to-finish walkthrough of how to upload your files to Amazon Unproblematic Storage Service (S3) using PowerShell.
A breakdown of the process (detailed steps are farther below):
-
Create an Amazon account to access Amazon Web Services (AWS)
a. Yous can try nigh of the AWS services for complimentary for a twelvemonth.
-
Create a user and group via Amazon Identity and Admission Direction (IAM) to perform the backup/upload
-
Create a saucepan in Amazon Simple Storage Service (S3) to hold your files
-
Install AWS Tools for Windows PowerShell, which contains the modules needed to admission AWS
-
Open up PowerShell and configure prerequisite settings
-
Write a PowerShell script that copies files from your local figurer to the Amazon S3 bucket you previously created
a. The script will utilise the credentials of the backup user created.
b. The script will be a PowerShell framework script to get you started.
This tutorial will get you lot up and running with Amazon Simple Storage Service (S3) and a PowerShell script that uploads files. Once this framework script is in place, you can add to the logic making information technology as specific or circuitous as necessary for your situation.
1. Create an Amazon root business relationship
Amazon has made this uncomplicated. Navigate to Amazon AWS and select "Create a Free Account." Follow the steps outlined.
If yous already have an business relationship with Amazon for consumer purchases, yous can use this for a single logon identity. I recommend using Multi-Factor Authentication with your AWS Root account to provide an boosted layer of security.
To kick the tires on AWS without hitting your pocket book, select the costless for a year option and get some experience under your belt.
2. Create a user and grouping
To access the AWS services, create a group and user account via IAM. Go along these guidelines in listen:
- Follow IAM's Best Practices for user and grouping management
- Create a grouping to fill-in/upload files with the appropriate permissions
- Create a user for accessing S3 during the backup/upload from PowerShell
Y'all tin notice detailed instructions for how to create an account and group in the video titled "Getting Started with AWS Identity and Access Direction" at IAM'due south Getting Started. The following is a synopsis.
- Login to AWS Management Panel with your Root business relationship or an Ambassador account (if you created a separate one)
- Navigate to Amazon AWS -> select My Account in the meridian right -> select AWS Direction Console from the drop-down listing
- Create a group with access to S3 buckets called S3BackupOperators
- Prepare permissions for this grouping to AmazonS3FullAccess so that members take the necessary access and permissions to perform backups/uploads.
- Create a user for accessing the S3 buckets called backupOperator
- Add this user to the S3BackupOperators group, from which they will inherit the permissions they need to write files to the S3 buckets.
- Generate Access Keys for the user
- In order to access the Amazon S3 buckets via PowerShell, your IAM user (backupOperator) will need both an Admission Key ID and a Secret Access Key (to specify in the PowerShell script for hallmark).
- To create an access fundamental, log into the AWS Management Panel and:
- Select Users
- Select the user name previously created
- Scroll down and select Create Admission Key
- Be sure to write down or save the Access Keys created
3. Create a bucket in S3
You volition demand an Amazon S3 saucepan to hold your files, which is coordinating to a directory/binder on your local estimator. More information tin can be found at Working with Amazon S3 Buckets.
Follow the instructions at Create a Saucepan and name it something relevant, such as Backups.
Note: Because the AmazonS3FullAccess policy was applied to the S3BackupOperators grouping, members of that group have add/delete permissions to your S3 buckets. If you lot would similar to further restrict access for that grouping to only the Backups saucepan, review the documentation at Managing Access Permissions to Your Amazon S3 Resource.
Amazon has written a PowerShell module that allows you to interact with Amazon Spider web Services remotely via PowerShell scripts. Download AWS Tools for Windows PowerShell to your Windows PC and follow the installation instructions.
For further details, read Setting up the AWS Tools for Windows PowerShell.
Note: Yous volition need to enable script execution each time yous open the PowerShell prompt, which requires administrator privileges. How to do this is detailed farther below.
5. Open PowerShell and configure prerequisite settings
There are a few caveats for using PowerShell with AWS.
Select which PowerShell prompt to employ
There are two PowerShell prompts you lot can use with slightly different requirements to get started.
- Windows PowerShell that comes with Windows by default
- Windows PowerShell for AWS , which is installed with AWS Tools for Windows PowerShell
For both PowerShell prompts, you volition need to enable script execution, as outlined in Setting up the AWS Tools for Windows PowerShell. Be sure to open the prompt as an ambassador then run Set up-ExecutionPolicy RemoteSigned.
Import AWSPowerShell module
You might demand to import the AWSPowerShell modele, depending on which PowerShell prompt you use.
When using the Windows PowerShell for AWS prompt, the AWSPowerShell module is automatically imported and the Initialize-AWSDefaults cmdlet run for you, allowing you to brainstorm working with the AWS PowerShell Cmdlets immediately.
When using the default Windows PowerShell prompt, you will need to manually import the AWSPowerShell module and run the Initialize-AWSDefaultscmdlet using the following commands.
PS C:\ > Import-Module " C:\Programme Files ( x86 ) \AWS Tools\PowerShell\AWSPowerShell\AWSPowerShell.psd1 " PS C:\ > Initialize-AWSDefaults 
6. PowerShell script to upload/backup files to Amazon S3
The previous steps prepared you to copy files from your computer to Amazon S3 using PowerShell.
The full script volition exist shown at the stop of this document. Meanwhile, let's step through the sections of the script.
six.A. Prepare constant variables
Beginning, gear up the constant variables that will be used in the script.
| Variable | Description | Value (Case) |
|---|---|---|
$accessKeyID | The Access Key ID for the backupOperator user | EXAMPLEDXLAW52MZCGIA |
$secretAccessKey | The Secret Admission Key for the backupOperator user | examplekfLK2c8NCFjlhhjxvYBxJwPkli1HosK4F |
$config | AmazonS3Config object to hold configuration options, such every bit RegionEndpoint and ServiceURL | These depend on your AWS account settings. In this case, the business relationship uses the US-W-2 region endpoint. RegionEndpoint = us-due west-2 ServiceURL = https://s3-the states-west-2.amazonaws.com/ |
Note: To find out which Region your account is using, login to the AWS Management Panel and note the Region specified in the URL. The following case uses The states-WEST-two: https://us-west-ii.panel.aws.amazon.com/console/home?nc2=h_m_mc®ion=the states-w-2#
Commands to prepare the user's Key variables (using the Access Keys previously generated) and to instantiate the AmazonS3Config object to agree the RegionEndpoint and ServiceURL values:
$accessKeyID = " EXAMPLEDXLAW52MZCGIA " $secretAccessKey = " examplekfLK2c8NCFjlhhjxvYBxJwPkli1HosK4F " $config = New-Object Amazon.S3.AmazonS3Config $config . RegionEndpoint = [ Amazon.RegionEndpoint ]:: " u.s.a.-west-2 ″ $config . ServiceURL = " https://s3-us-west-two.amazonaws.com/ " 6.B. Create AmazonS3Client object
In this step, you will instantiate an AmazonS3Client object. It will use the permissions and access keys granted to the backupOperator user, based on the variables created in the previous step. The AmazonS3Client object also requires the previously created AmazonS3Config object to interact with your Amazon S3 buckets.
Instantiate the AmazonS3Client object:
$customer = [ Amazon.AWSClientFactory ]:: CreateAmazonS3Client ( $accessKeyID , $secretAccessKey , $config ) vi.C. Upload/backup files via PowerShell
With PowerShell, yous have several options for uploading your files.
- Copy specific files from a single folder or multiple directories.
- Copy all files in a directory without including subdirectories.
- Copy a directory and its subdirectories, by iterating through the subdirectories with PowerShell.
Copy specific files
To copy specific files, use the following commands.
Write-S3Object -BucketName Backups -File " C:\Documents\Business\FinancialReports.xlsx " -Fundamental " /Documents/Business organisation/FinancialReports.xlsx " Write-S3Object -BucketName Backups -File " C:\Pictures\Business Logos\logo.jpg " -Key " /Pictures/Business Logos/logo.jpg " Annotation: See the Write-S3Object Cmdlet documentation for more than information.
Copy all files in a directory
To copy all the files in a directory, utilize the post-obit command.
Write-S3Object -BucketName Backups -Folder " C:\Pictures\Family unit " -KeyPrefix " /Pictures/Family " Note: Come across the Write-S3Object Cmdlet documentation for more than information.
Re-create a directory and subdirectories
To re-create a directory and the subdirectories, utilise the following function to iterate through the subdirectories recursively.
office RecurseFolders ([ string ] $path ) { $fc = New-Object -com Scripting.FileSystemObject $binder = $fc . GetFolder ( $path ) # Iterate through subfolders foreach ( $i in $binder . SubFolders ) { $this Folder = $i . Path # Transform the local directory path to notation compatible with S3 Buckets and Folders # 1. Trim off the bulldoze letter and colon from the showtime of the Path $s3Path = $this Folder.ToString () $s3Path = $s3Path . SubString ( 2 ) # two. Replace back-slashes with forward-slashes # Escape the dorsum-slash special character with a back-slash so that information technology reads it literally, like so: "\\" $s3Path = $s3Path -replace "\\" , "/" # Upload directory to S3 Write-S3Object -BucketName Backups -Binder $this Folder -KeyPrefix $s3Path } # If subfolders exist in the current folder, and so iterate through them too foreach ( $i in $binder . subfolders ) { RecurseFolders ( $i . path ) } } 6.D. Full PowerShell script
The following is a full PowerShell script that will backup/upload a directory (including all subdirectories) from your local calculator to an Amazon S3 Bucket.
# Constants $sourceDrive = "C:\" $sourceFolder = "ImportantFiles" $sourcePath = $sourceDrive + $sourceFolder $s3Bucket = "Backups" $s3Folder = "Archive" # Constants – Amazon S3 Credentials $accessKeyID = "EXAMPLEDXLAW52MZCGIA" $secretAccessKey = "examplekfLK2c8NCFjlhhjxvYBxJwPkli1HosK4F" # Constants – Amazon S3 Configuration $config = New-Object Amazon.S3.AmazonS3Config $config . RegionEndpoint = [ Amazon.RegionEndpoint ]:: "us-west-2" $config . ServiceURL = "https://s3-us-west-2.amazonaws.com/" # Instantiate the AmazonS3Client object $client = [ Amazon.AWSClientFactory ]:: CreateAmazonS3Client ( $accessKeyID , $secretAccessKey , $config ) # Office – Iterate through subfolders and upload files to S3 function RecurseFolders ([ string ] $path ) { $fc = New-Object -com Scripting.FileSystemObject $folder = $fc . GetFolder ( $path ) foreach ( $i in $folder . SubFolders ) { $this Folder = $i . Path # Transform the local directory path to notation uniform with S3 Buckets and Folders # i. Trim off the drive alphabetic character and colon from the start of the Path $s3Path = $this Folder.ToString () $s3Path = $s3Path . SubString ( ii ) # 2. Replace back-slashes with forward-slashes # Escape the back-slash special graphic symbol with a back-slash so that it reads it literally, like and then: "\\" $s3Path = $s3Path -replace "\\" , "/" $s3Path = "/" + $s3Folder + $s3Path # Upload directory to S3 Write-S3Object -BucketName $s3Bucket -Binder $this Folder -KeyPrefix $s3Path } # If subfolders exist in the electric current folder, then iterate through them likewise foreach ( $i in $folder . subfolders ) { RecurseFolders ( $i . path ) } } # Upload root directory files to S3 $s3Path = "/" + $s3Folder + "/" + $sourceFolder Write-S3Object -BucketName $s3Bucket -Folder $sourcePath -KeyPrefix $s3Path # Upload subdirectories to S3 RecurseFolders ( $sourcePath ) Summary
As stated in the beginning of this article, the purpose of this tutorial is to get you up and running from scratch with Amazon Simple Storage Service (S3) and a PowerShell script that uploads files.
Through this process you:
- Created an Amazon AWS account
- Learned how to create and manage AWS users and groups via IAM
- Learned how to manage permissions for a group, scoping them based on their role
- Created a S3 bucket and configured permissions through an IAM grouping
- Experienced how to configure AWS Tools for Windows PowerShell
- Discovered the prerequisites for connecting to AWS via PowerShell
- Learned how to use the Write-S3Object cmdlet to upload files to S3 buckets
- Saw how to iterate through folders and subfolders recursively with PowerShell
- Saw how to alter the local folder path format to work with S3 folder format
With this framework in identify, you can expand the script to do a number of other things. Such every bit run once per twenty-four hour period with a scheduled chore (automate to improve efficiency), delete former folders/files from S3 as they're no longer used (reducing your S3 storage toll), or only upload new/changed files to S3 (reducing your data transfer cost).
Resource
- Amazon Spider web Services (AWS)
- Identity and Access Management (IAM):
- Getting Started
- IAM'south Best Practices
- Root Business relationship Credentials vs. IAM User Credentials
- User Guide for Identity and Access Management (IAM)
- AWS PowerShell:
- AWS Tools for Windows PowerShell
- Setting upwardly the AWS Tools for Windows PowerShell
- Write-S3Object Cmdlet
- Amazon S3:
- Create a Saucepan
- Working with Amazon S3 Buckets
- Managing Access Permissions to Your Amazon S3 Resources
- AmazonS3Config Object
- AmazonS3Client Object
- Getting Started with Amazon Simple Storage Service (S3)
morillovareat1954.blogspot.com
Source: http://todhilton.com/technicalwriting/upload-backup-your-files-to-amazon-s3-with-powershell/
Post a Comment for "Use Powershell to Upload Data to S3 Bucket"