3

There's a tag that has a lot of irrelevant questions. Almost all the questions are of purely .

I suspect the misuse is due the fact that the tag has no usage guidance. Given that spark has introduced Pandas API on Spark, it'd be great to update the tag's guidance to use it for the linked API.

1
  • Is your question what pyspark-pandas is or is your question "can someone please add a tag wiki description"?
    – TylerH
    Commented Jan 18 at 17:17

1 Answer 1

1

This has been answered on Stack Overflow itself, ironically (though about the technology itself, of course... not the tag): What is the difference between pyspark.pandas to pandas? which you have already commented on.

From the current top answer there:

PySpark is an interface for Apache Spark in Python. It allows you to write Spark applications using Python and provides the PySpark shell to analyze data in a distributed environment. Pyspark.pandas is an API that allows you to use pandas functions and operations on "spark data frames".

And from the second answer:

pyspark.pandas is an alternative to pandas, with the same api than pandas. This means you can work with pyspark exactly the same as you work with pandas

for example, to create a dataframe, you use .DataFrame same as with pandas, and use .iloc or .drop_duplicates
[...]

In short, it seems like it lets you write Pandas code (think writing jQuery for JavaScript) directly for/in Spark databases/dataframes. So... that's what the tag is for.

1
  • thank you. apologies for not being direct - i'm looking for guidance on the tag usage. however, looks like you've lightly touched on that as well.
    – samkart
    Commented Jan 18 at 18:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .