zlsdmx 2020-05-14
换了几个开发环境,每次都会遇到sparksql连不上元数据的情况,整理一下脱坑吧。。。。。
首先说一下几个文件吧,这些是我遇到的几个问题的解决方法,有可能你并不适用,仅供参考。
1、配置文件放在resources下面
2、下载hadoop-common-2.2.0-bin-master 这个文件,里面有hadoop必要的bin文件,以此当做hadoop的家目录
配置如下:
3.启动的时候,你可能还会遇到mysql数据元字符的问题:
我就直接贴了,把hive的元数据库改成latin1字符集,记住库和里面所有表
alter database hive character set latin1; alter table BUCKETING_COLS convert to character set latin1; alter table CDS convert to character set latin1; alter table COLUMNS_V2 convert to character set latin1; alter table DATABASE_PARAMS convert to character set latin1; alter table DBS convert to character set latin1; alter table FUNC_RU convert to character set latin1; alter table FUNCS convert to character set latin1; alter table GLOBAL_PRIVS convert to character set latin1; alter table PART_COL_STATS convert to character set latin1; alter table PARTITION_KEY_VALS convert to character set latin1; alter table PARTITIONS convert to character set latin1; alter table ROLES convert to character set latin1; alter table SDS convert to character set latin1; alter table SEQUENCE_TABLE convert to character set latin1; alter table SERDES convert to character set latin1; alter table SKEWED_STRING_LIST convert to character set latin1; alter table SKEWED_STRING_LIST_VALUES convert to character set latin1; alter table SORT_COLS convert to character set latin1; alter table TAB_COL_STATS convert to character set latin1; alter table TBLS convert to character set latin1; alter table VERSION convert to character set latin1;
4.测试
object test { def main(args: Array[String]): Unit = { val conf = new SparkConf() conf.setAppName(s"TestHive") conf.setMaster("local[4]") val spark = SparkSession.builder.config(conf).enableHiveSupport().getOrCreate() spark.sql("show databases").show } }