-23

Again, my question was closed. It was, as far as I can tell, focused. I even edited it to narrow the scope. There isn't any question there's an answer to it. How can I improve?

Cheap and cheerful way to do leader election on AWS

Any of the following answers would have been perfectly good (I don't know any of these answers, and Google was very light in providing these):

  • "You can use DynamoDB as an atomic store. It is super easy to setup x, y, and z and you can use it as a leader elector (using the text description you said earlier)"
  • "You can use S3 as a store, as it provides atomic and versioned writes. Here's how you write to the S3 bucket using versions, and here's how you would query the results to get the leader."
  • "Actually, AWS Secret Manager is a great place to do leader election, as it has strongly consistent writes, and prevents overwriting."

There's nothing about a software architect or other complex items; it's just that feature (strongly consistent storage without any overwrites) that would solve it.

I understand that I could have specified further on not using Java, but my goal was not to use (as much as reasonably possible) any custom code, and just reuse an existing solution (e.g., Dynamo, S3, etc. which have many millions of person hours solving leader election).

I do want a programming solution. E.g., imagine the following (which is how I'm currently thinking about it).

Possible (great) answer:

What you could do would be to use S3 versioning as a strongly consistent way to write. For example, here's how you would do that in Python (all pseudocode, obviously):

S3BUCKET = "lock-bucket"
LOCKFILE = "lock-file"

def writeLock():
    s3.write(getHostName())

def writePeeringString(peeringString):
    s3.write(peerString)

def getAllFileVersions() -> List:
    return s3.getAllFiles(S3BUCKET, LOCKFILE)

def isThisNodeLeader() -> bool:
    # Gets all files as a list, ordered by version
    firstFile = s3.getFileVersion(S3BUCKET, LOCKFILE, 0)

    return getHostName() == firstFile.contents()

def main():
    writeLock()

    if isThisNodeLeader():
       peeringString = startServer()
       writePeeringString(peeringString)
    else:
       while:
            sleep(5)
            for f in getAllFileVersions():
                 if isHostName(f.contents()):
                      pass
                 else:
                      # The file contains a peering string, use that to start the server and then break
                      startServer(f.contents())
                      break

There are many such solutions. They require a minimum of coding, but absolutely require coding. It does not attempt to do the very complicated work of leader election; it basically relies on the S3 Versioning API to accomplish this.

I tried to ask this. Surely someone has run into a problem (and proposed a solution) similar to this.

27
  • 20
    You're asking for a solution to a problem that requires solving multiple problems and haven't given us anything else to work with. what are you expecting an answer to look like?
    – Kevin B
    May 17 at 14:39
  • 20
    It is either too broad or asking for an off-site resource All I'm looking for is an strongly consistent API to interact with. It sounds like you need a whiteboard and some architects, preferable of the software kind. That is not a service SO offers. Other sites might.
    – rene
    May 17 at 14:47
  • 14
    What further highlights the lack of focus is that after you got a suggestion that used Java, your next question was "Is there a non-java based version? Or a binary one?" Ideally you'd have specified the programming language you are using at the start itself. May 17 at 14:52
  • 3
    "All I'm looking for is an strongly consistent API to interact with." - I read this statement as you looking to us to provide an API recommendation. If that is the case then that statement would make your question out of scope. One of the key things I see, is you are asking a Java question that requires an API to achieve your end goal, and have not provided a single line of code. May 17 at 17:14
  • 1
    i'm not looking for any java/code solution ideally! just reuse (abuse?) an existing very smart external API to solve the problem. i added a bunch of context above - still really confused how to ask this question better.
    – aronchick
    May 17 at 21:46
  • 2
    hmm... does that make it not a programming related problem? it certainly seems very recommendation-like.
    – Kevin B
    May 17 at 21:57
  • Interesting question! ULTIMATELY i'll need to interact with this via a programming interface - CLI, SDK, etc - but handling the very complicated specifics of leader election is out of scope. COULD be server fault?
    – aronchick
    May 17 at 21:58
  • 1
    well, the problem is if it falls into the "recommendation-like" end of things, that's not really on topic anywhere.
    – Kevin B
    May 17 at 22:00
  • 1
    If there's no code then this isn't a programming question and is off-topic for that reason. May 17 at 22:10
  • 6
    “i'm not looking for any java/code solution ideally! just reuse (abuse?) an existing very smart external API to solve the problem.” - What you want, based solely on this description, isn’t within scope at Stack Overflow. If you don’t have a programming question, then your question, isn’t within scope at Stack Overflow. You clearly want a recommendation. May 18 at 1:37
  • I do not want a recommendation. I want a coding solution that utilizes an external API. I have mocked up a further detail above.
    – aronchick
    May 18 at 2:22
  • 1
    I have edited this into the original question. I don't want to reopen (i've already lost 20 points in reputation trying to get smarter about what I did wrong), but it suggests I should submit a new one, which ALSO feels wrong (and just an opportunity to lose a lot more reputation).
    – aronchick
    May 18 at 3:02
  • 3
    Please don't insert "EDIT"s/"UPDATE"s, just make your post the best presentation as of edit time. When is "EDIT"/"UPDATE" appropriate in a post?
    – philipxy
    May 18 at 6:01
  • 3
    I rolled back your inappropriately putting an answer into the SO question post. Please edit your post to clearly ask 1 specific researched non-duplicate on-topic question, for what you want, and to not have phrasing that seems to be asking for other things. Don't post unclear text accompanied by an example of what would be an answer to a clear question phrasing you don't give, such an example doesn't clarify, write a clear question.
    – philipxy
    May 18 at 6:19
  • 2
    Let me try to rephrase the comments here. Stack Overflow is for programming questions that are specifically scoped, your question 1) Asks about what AWS service you should use (very wide scope, this can also considered a "recommendation" question) 2) Gives no indication of programming language to be used. Even though your question might be programming related it doesn't seem to be suitable for Stack Overflow. If you had picked a specific service and a programming language and had some difficulty implementing what you want, asking that would be a more suitable question. May 18 at 12:56

0