Hive创建外部表,指向lzo格式文件时,无法解析出数据,报错如下:
java.io.IOException: No LZO codec found, cannot run.
hiveserver2日志报错如下:
Diagnostic Messages for this Task:
Error: java.io.IOException: java.lang.reflect.InvocationTargetException
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:265)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.<init>(HadoopShimsSecure.java:212)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:332)
at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:721)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:169)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:432)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:251)
... 11 more
Caused by: java.io.IOException: No LZO codec found, cannot run.
at com.hadoop.mapred.DeprecatedLzoLineRecordReader.<init>(DeprecatedLzoLineRecordReader.java:53)
at com.hadoop.mapred.DeprecatedLzoTextInputFormat.getRecordReader(DeprecatedLzoTextInputFormat.java:156)
at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.<init>(CombineHiveRecordReader.java:66)
... 16 more
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Stage-Stage-1: Map: 1 Reduce: 1 HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec
解决方法:
检查hadoop集群是否添加了hadoop-lzo-*.jar,该jar包是否已配置到HADOOP_CLASSPATH里
如果配置进去了,并检查core-site.xml里是否配置了如下信息:
<property>
<name>io.compression.codecs</name>
<value>
org.apache.hadoop.io.compress.GzipCodec,
org.apache.hadoop.io.compress.DefaultCodec,
org.apache.hadoop.io.compress.BZip2Codec,
org.apache.hadoop.io.compress.SnappyCodec,
com.hadoop.compression.lzo.LzoCodec,
com.hadoop.compression.lzo.LzopCodec
</value>
<description>
A comma-separated list of the compression codec classes
that can be
used for compression/decompression. In addition to any
classes
specified with this property (which take precedence), codec
classes
on the classpath are discovered using a Java ServiceLoader.
</description>
</property>
<property>
<name>io.compression.codec.lzo.class</name>
<value>com.hadoop.compression.lzo.LzoCodec</value>
</property>
如果满足这两点,应该就可以了(注意:hive安装包的lib目录不用添加hadoop-lzo-*.jar)。
相关推荐
C++直接解析注册表文件,不用注册表API.
2009-10-27更新 增加exfat支持,返回同fat 修复ntfs短文件名找不到,修复FAT下内存泄露
hive解析用户名和密码的jar包
基于 Antlr4 的 Hive SQL 解析.zip 大学生课程设计 课程设计 自己大二写的课程设计
注册表文件读写,解析HIVE文件方式 NT Registry Hive access library, constants & structures 以直接读写注册表所属文件方式读写注册表,可以避开注册表监控软件. 支持系统:win2ksp4,winxp sp2, winxp sp3
Hive内部表合并小文件Java程序
hivesql解析 支持版本<!--<hive.version>1.0.0</hive.version> <hadoop.version>2.4.1</hadoop.version> --> <hive.version>2.0.1</hive.version> <hadoop.version>2.7.2</hadoop.version>
Win10不需要Cygwin搭建大数据测试环境搭建hive的bin文件(包含官方版本中缺少的cmd文件),完美解决windows环境下配置hive时,找不到hive命令问题
// 测试用例如下 // 设置元数据服务 Delegate.getDelegate().setMetaColumnService(new IMetaColumnService() { @Override public List<String> queryMetaColumn(String dbName, String tableName) { ...
背景:由于Hive需要开启权限管理,安装网上教程,开启权限配置,重启集群后。 使用root用户登录,进入Hive...Authorization failed:No privilege 'Create' found for outputs { }. Use SHOW GRANT to get more details
适合需要使用到hive环境的人,大致有2.3.7的hive、1.8.0_231的java等
hive解析json时所需jar包。具体使用: add jar ../../../target/json-serde-1.3-jar-with-dependencies.jar; CREATE TABLE json_nested_test ( country string, languages array, religions map,array<int>>) ...
hive+hadoop配置文件hive+hadoop配置文件hive+hadoop配置文件hive+hadoop配置文件hive+hadoop配置文件hive+hadoop配置文件hive+hadoop配置文件hive+hadoop配置文件hive+hadoop配置文件hive+hadoop配置文件hive+...
HiveSQL解析原理
Hive的配置,选定一个Hive的master,其他的节点为slaves,master和slaves的配置略有不同,不是完全照搬,准确的来说,应该是只有一个节点是Hive的服务器节点负责和元数据库以及集群内部通信,其他的Hive节点其实是装...
How to create and use Hive Functions, Listing of Built-In Functions that are supported in Hive
windows平台下,hive可执行文件,完全没有问题,可以使用。 将这个bin目录覆盖hive原有的bin目录即可
hive 配置文件,包括hive-site.xml与hive-env.sh两个配置文件,可根据自己需要修改
将解压的bin覆盖目标bin文件夹即可
使用hive,hadoop,spark,datax,python,scala修改添加的配置