Вопрос по – Разделение ввода Hadoop - как это работает

0

Error: User Rate Limit ExceededError: User Rate Limit Exceeded

Error: User Rate Limit Exceeded

Error: User Rate Limit Exceeded

Error: User Rate Limit Exceeded

Error: User Rate Limit Exceeded

Error: User Rate Limit ExceededError: User Rate Limit ExceededError: User Rate Limit Exceeded

Ваш Ответ

2   ответа
0

Error: User Rate Limit Exceeded

Error: User Rate Limit Exceeded

mapred.min.split.size: The minimum size chunk that map input should be split into.
mapred.max.split.size: The largest valid size inbytes for a file split. 
dfs.block.size: The default block size for new files.

Error: User Rate Limit Exceeded

Math.max("mapred.min.split.size", Math.min("mapred.max.split.size", blockSize));

Error: User Rate Limit ExceededError: User Rate Limit Exceeded.

1

Error: User Rate Limit ExceededFileInputFormatError: User Rate Limit Exceeded

Error: User Rate Limit Exceeded

  • If the input file is compressed, the input format and compression method must be splittable. Gzip for example is not splittable (you can't randomly seek to a point in the file and recover the compressed stream). BZip2 is splittable. See the specific InputFormat.isSplittable() implementation for your input format for more information
  • If the file size is less than or equal to its defined HDFS block size, then hadoop will most probably process it in a single split (this can be configured, see a later point about split size properties)
  • If the file size is greater than its defined HDFS block size, then hadoop will most probably divide up the file into splits based upon the underlying blocks (4 blocks would result in 4 splits)
  • You can configure two properties mapred.min.split.size and mapred.max.split.size which help the input format when breaking up blocks into splits. Note that the minimum size may be overriden by the input format (which may have a fixed minumum input size)

Error: User Rate Limit ExceededgetSplits()Error: User Rate Limit ExceededFileInputFormatError: User Rate Limit Exceeded

Похожие вопросы