142. Escape from S3

https://is1-ssl.mzstatic.com/image/thumb/Podcasts125/v4/72/e3/8f/72e38f5f-5b06-84d4-c257-780189b59763/mza_17652109777479850672.jpg/600x600bb.jpg

AWS Bites

153 episodes

1 day ago

AWS Bites is the show where we answer questions about AWS! This show is brought to you be Eoin Shanaghy and Luciano Mammino, certified AWS experts.

Technology

RSS

All content for AWS Bites is the property of AWS Bites and is served directly from their servers with no modification, redirects, or rehosting. The podcast is not affiliated with or endorsed by Podjoint in any way.

AWS Bites is the show where we answer questions about AWS! This show is brought to you be Eoin Shanaghy and Luciano Mammino, certified AWS experts.

Technology

https://d3t3ozftmdmh3i.cloudfront.net/staging/podcast_uploaded_episode/17717320/17717320-1743697686963-3f8bec6f42727.jpg

142. Escape from S3

AWS Bites

30 minutes 44 seconds

7 months ago

142. Escape from S3

We discuss common use cases and challenges for copying data between S3 buckets and S3-compatible object storage services. We share our experience building an open source Node.js CLI tool called S3-Migrate to efficiently migrate data with separate source and destination credentials. We cover performance considerations like streaming, chunk sizes, concurrency and parallelism.

AWS Bites is brought to you in association with fourTheorem. If you need a friendly partner to support you and work with you to de-risk any AWS migration or development project, check them out at ⁠⁠⁠fourtheorem.com⁠⁠⁠

In this episode, we mentioned the following resources:

s3-migrate CLI tool: https://github.com/lmammino/s3-migrate
DigitalOcean Spaces Object Storage: https://docs.digitalocean.com/products/spaces/
Cloudflare R2: https://www.cloudflare.com/en-gb/developer-platform/products/r2/
Backblaze B2: https://www.backblaze.com/cloud-storage
Wasabi Cloud Storage: https://wasabi.com/
Linode / Akamai Object Storage: https://www.linode.com/products/object-storage/
MinIO (Self-hosted S3-compatible storage): https://min.io/
Basecamp / Hey’s move away from S3: https://world.hey.com/dhh/it-s-five-grand-a-day-to-miss-our-s3-exit-b8293563
AWS re:Post - How to move objects between S3 buckets: https://repost.aws/knowledge-center/move-objects-s3-bucket
AWS Labs - Utility for S3 Migration: https://github.com/awslabs/utility-for-s3-migration
s3s3mirror (Java-based tool): https://github.com/cobbzilla/s3s3mirror
rclone S3 Support: https://rclone.org/s3/
knox-copy (Ruby-based, deprecated): https://github.com/goodeggs/knox-copy
Flexify.io (paid cloud migration service): https://flexify.io/

Do you have any AWS questions you would like us to address?