Skip to content

[SPARK-11504][SQL] API audit for distributeBy and localSort - #9470

Closed
rxin wants to merge 2 commits into
apache:masterfrom
rxin:SPARK-11504
Closed

[SPARK-11504][SQL] API audit for distributeBy and localSort#9470
rxin wants to merge 2 commits into
apache:masterfrom
rxin:SPARK-11504

Conversation

@rxin

@rxin rxin commented Nov 4, 2015

Copy link
Copy Markdown
Contributor
  1. Renamed localSort -> sortWithinPartitions to avoid ambiguity in "local"
  2. distributeBy -> repartition to match the existing repartition.

@rxin

rxin commented Nov 4, 2015

Copy link
Copy Markdown
Contributor Author

cc @nongli @yhuai

@yhuai

yhuai commented Nov 4, 2015

Copy link
Copy Markdown
Contributor

LGTM pending jenkins.

@yhuai

yhuai commented Nov 4, 2015

Copy link
Copy Markdown
Contributor

test this please

@SparkQA

SparkQA commented Nov 4, 2015

Copy link
Copy Markdown

Test build #45030 has finished for PR 9470 at commit 0c7e1cd.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):\n * public class JavaAssociationRulesExample\n * public class JavaPrefixSpanExample\n * public class JavaSimpleFPGrowth\n * class StreamInterceptor implements TransportFrameDecoder.Interceptor\n * public final class ChunkFetchSuccess extends ResponseWithBody\n * public abstract class ResponseWithBody implements ResponseMessage\n * public final class StreamFailure implements ResponseMessage\n * public final class StreamRequest implements RequestMessage\n * public final class StreamResponse extends ResponseWithBody\n * public class TransportFrameDecoder extends ChannelInboundHandlerAdapter\n

@yhuai

yhuai commented Nov 4, 2015

Copy link
Copy Markdown
Contributor

Merging to master.

@asfgit asfgit closed this in abf5e42 Nov 4, 2015
@SparkQA

SparkQA commented Nov 4, 2015

Copy link
Copy Markdown

Test build #45031 has finished for PR 9470 at commit 0c7e1cd.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):\n * public class JavaAssociationRulesExample\n * public class JavaPrefixSpanExample\n * public class JavaSimpleFPGrowth\n * class StreamInterceptor implements TransportFrameDecoder.Interceptor\n * public final class ChunkFetchSuccess extends ResponseWithBody\n * public abstract class ResponseWithBody implements ResponseMessage\n * public final class StreamFailure implements ResponseMessage\n * public final class StreamRequest implements RequestMessage\n * public final class StreamResponse extends ResponseWithBody\n * public class TransportFrameDecoder extends ChannelInboundHandlerAdapter\n

markhamstra pushed a commit to markhamstra/spark that referenced this pull request Nov 11, 2015
1. Renamed localSort -> sortWithinPartitions to avoid ambiguity in "local"
2. distributeBy -> repartition to match the existing repartition.

Author: Reynold Xin <rxin@databricks.com>

Closes apache#9470 from rxin/SPARK-11504.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rxin This causes an infinite loop, which isn't caught by the unit tests since DataFrameSuite only tests the Column* overload.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ankurdave can you create a jira?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants