0

I have a NodeJS app that stores all of its data, even images, to MongoDB. Now I want to use an S3 to store my app's files (images, 43gb) and so, I need to transfer there all files that I have in my mongo.

I found the s3cmd tool that can transfer my files to the S3 instance (DigitalOcean space), but I can't find a way to actually access those files. I know they are stored in the fs.files and fs.chunks collections, but I can't figure out how to actually use them as input to s3cmd!

Any help will be much appreciated!!!

EDIT: This is the format the images are being stored with:

fs.files

{
  "_id" : ObjectId("5ae97922c1dabec8d2d0bdb0"),
  "filename" : "2b57455f3878d11dabc9c984da7de314_postImage.jpeg",
  "contentType" : "binary/octet-stream",
  "length" : 2291623,
  "chunkSize" : 261120,
  "uploadDate" : ISODate("2018-05-02T08:38:58.549Z"),
  "aliases" : null,
  "metadata" : null,
  "md5" : "9ad420eaa7c28a73e449199430627802"
}

fs.chunks.findOne()

{
  "_id" : ObjectId("5ae2d77f6616b4a9d93cb4b1"),
  "files_id" : ObjectId("5ae2d77f6616b4a9d93cb4b0"),
  "n" : 0,
  "data" : BinData(0,"iVBORw0KGgoAAAANSUhEUgAAAuAAAAJvCAYAAAA6OGQEAAAABGdBTUEAALGPC/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAABWWlUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNS40LjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRher38tcAAAAASUVORK5CYII=") }
1

You will have to write a script or program to retrieve files from MongoDB and copy them to S3.

Most probably files in MongoDB are stored in BSON format, so they require to be converted first.

As you have noted that you are using Parse - try these utils to extract images.

https://github.com/parse-server-modules/parse-files-utils

  • Exactly what I'm looking for! How do I convert them? – Sotiris Kaniras Dec 12 '19 at 20:53
  • Write a program to export from MongoDB to a file system - exactly how depends how you stored things in Mongo, but I doubt you'll find a pre-written script that can do this for you as few people store binary resources in a database. From there use S3 command line tools (eg aws s3 sync) to upload them to S3, or drag and drop. – Tim Dec 12 '19 at 20:56
  • That's what I don't know how to do! I mean it's not as simple as exporting a collection! I found this script, but it doesn't seem to work blog.vladimirm.com/2011/06/… Should I be looking for something like this though? – Sotiris Kaniras Dec 12 '19 at 21:11
  • Talk to the developer (of the app) – Martynas Saint Dec 12 '19 at 21:12
  • 1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.