Start networking and exchanging professional insights

Register now or log in to join your professional community.

Follow

Explain how is data partitioned before it is sent to the reducer if no custom partitioner is defined in Hadoop?

user-image
Question added by khageswar rao Battala , Software Engineer , Proven Technologies and Services Pvt Ltd
Date Posted: 2016/07/08
Amanul Islam Khan
by Amanul Islam Khan , Programmer Analyst , Cognizant Technology Solutions PVT LTD

Hash Partion is the default partioner in hadoop which is handled by Hadoop internally if no partioner has been defined.

Hash partitioning  is Default partition done 

Shamrooque R P
by Shamrooque R P , Apps Systems Engineer 6 , wells fargo

Hash partitioning which is the dafault partitioning in hadoop.

Ravindra Singh
by Ravindra Singh , Data Engineer , Confidential

Default Partitioner which buckets keys using a hash function is used.

Sneha Nair
by Sneha Nair , Hadoop Developer , Techdata Solutions

Hashing would be used by default else write code to make it as key and set it in mapper class

Deleted user
by Deleted user

By default Hash partioner is used for partioning the data

Reshmi KC
by Reshmi KC , lead developer , tata consultancy services

add the partition column as the key for the mapper

If there is no custom partitioner ,mapreduce by default uses hash algorithm (hash code for map key) and keys with same hashcode will send to same reducer.

Ahmed Gamil
by Ahmed Gamil , Senior System Engineer , General Authority of Civil Aviation

Map output stored in memory then spilled to the disk when it reach to the buffer threshold

the spill files are merged into a single partitioned and sorted output file

the maximum number of streams to merge at once is 10

More Questions Like This