伴随着梦想前行

标签 hive 下的文章

26/6
2019

hive调用堆栈

大数据 hive 调用堆栈 评论 

    at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:249)
    at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketForFileIdx(FileSinkOperator.java:622)
    at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:566)
    at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:675)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
    at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:88)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
    at org.apache.hadoop.hive.ql.exec.LimitOperator.process(LimitOperator.java:54)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
    at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:88)
    at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:235)
    at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:467)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:409)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)

+ MORE

27/4
2018

hive1.2报错MoveTask

大数据 hive hiverc 评论

整体环境java1.7+hadoop2.7+hive1.2；

hive执行的mr结束后，报以下错误：
Failed with exception Unable to move source hdfs://x1/hive/stagingdir_hive_2018-04-27_10-08-41_073_4546675974966003896-1/-ext-10001 to destination hdfs://x2/hive/warehouse/tag_2018042710068970_1
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask

原因：
scratchdir和warehouse在多个权限的目录下；
hdfs://x1/hive/stagingdir 和 hdfs://x2/hive/warehouse ；

解决方法：
将两个目录的权限设置一致；
set hive.exec.scratchdir=hdfs://x2/hive/scratchdir/;
set hive.metastore.warehouse.dir=hdfs://x2/hive/warehouse/;

可将参数加到~/.hiverc中；

+ MORE