3

I'd like some help determining whether a question I have about S3 object reindexing is on topic.

The Amazon S3 tag warns that support, functionality and configuration questions are off-topic. However, I believe my question boils down to: what is the runtime complexity for reindexing the unknown-to-me data structure underlying S3, which seems on topic.

Here is the content of the proposed question:

It seems you cannot rename an object on S3. An endless amount of answers here on SO indicate that the best you can do is copy the object, then delete the old one.

See various related questions:

According to these questions, this is true across various language APIs, including the AWS command line tools (the mv command warns that it just copies), and the online GUI (the rename action copies as well).

However, I have not found a clear technical reason for why this is the case. There are many key value data structures which allow reindexing.

In order to keep things on topic, here are some possible types of answers which I'll make up:

  1. The S3 system is backed by a ??? distributed block data structure, reindexing can't be atomic due to its decentralized nature, and multiple reindex commands even to all servers might put the system in a never ending cycle of reindexing or could violate data integrity. Aka the structure makes it impossible.
  2. The S3 system is backed by a ??? distributed block data structure, reindexing is theoretically possible, but only achieved by the following process which is a O(n^3) nightmare, making it prohibitively expensive to perform.
  3. This restriction has nothing to do with S3's technical implementation, which does allow reindexing, and AWS simply decided not to expose this functionality.
  4. Also acceptable: The S3 system is an absolute blackbox and AWS doesn't allow for its internal implementation to be known. The question cannot be answered with the information we have.
3
  • 3
    This probably amounts to the same reasons why questions about language specifications or design decisions are closed as opinion based.
    – E_net4
    Commented Nov 4, 2022 at 16:52
  • 3
    #1 is likely the truth, since we do know S3 buckets are partitioned. That said, the exact technical details aren't published and likely the only people that have a concrete answer are behind NDAs. Even if it was known, it's really an implementation detail, all we can say is S3 doesn't have a Rename API. Commented Nov 4, 2022 at 17:48
  • I wouldn't post a question which only implies a question though. Actually ask the question. Don't make it so people have to think about what you are actually asking, as you will face very rich imaginations.
    – Gimby
    Commented Nov 8, 2022 at 10:49

1 Answer 1

9

Eh... I would say that it's not on-topic.

The question boiled down to its essence is, "What technical evidence does S3 provide for not choosing to allow someone to rename an object?" The answer only lives with S3 engineers and architects.

To be blunt though, a rename really is just a copy operation of an existing object (with new name) and a delete operation. A "rename" operation is just syntactic sugar over those two steps.

5
  • I can accept the first part. However, technically, I imagine the system eventually boils down to flat files supported by some metadata structure. Imagine naively a hashmap holding references to files. There's no reason I have to move the data in order to reindex it in the metadata structure.
    – code11
    Commented Nov 4, 2022 at 16:29
  • 6
    @code11: The reason is "because AWS said so". Technically - as in technologically - you have a point, but from an objective answer perspective, "AWS said no", so that's what we're going to have to go off of.
    – Makoto
    Commented Nov 4, 2022 at 16:30
  • Would the question be better phrased as asking what datastructure underlies S3?
    – code11
    Commented Nov 4, 2022 at 16:32
  • 3
    @code11: No, since it suffers from the same injury - only S3 engineers and architects could answer the question in a practical sense, and it doesn't really change the fact that you are prohibited from doing so today.
    – Makoto
    Commented Nov 4, 2022 at 16:32
  • Hmm. I believe that means its off topic for any SE site. I know AWS has published a number of white papers about their technology, but I wasn't able to find one on this explicitly. Unfortunate. I think people would be interested in the answer if AWS ever wants to give us a peek behind the curtain.
    – code11
    Commented Nov 4, 2022 at 16:37

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .