Sqoop配置安裝

Sqoop是一個(gè)用來(lái)將Hadoop和關(guān)系型數(shù)據(jù)庫(kù)中的數(shù)據(jù)相互轉(zhuǎn)移的工具，可以將一個(gè)關(guān)系型數(shù)據(jù)庫(kù)（例如 ： MySQL ,Oracle ,Postgres等）中的數(shù)據(jù)導(dǎo)入到Hadoop的HDFS中，也可以將HDFS的數(shù)據(jù)導(dǎo)入到關(guān)系型數(shù)據(jù)庫(kù)中。

Sqoop的User Guide地址：http://archive.cloudera.com/cdh/3/sqoop/SqoopUserGuide.html#_introduction
-
 

1:tar zxvf  sqoop-1.1.0.tar.gz

2:修改配置文件  /home/Hadoopuser/sqoop-1.1.0/conf/sqoop-site.xml

一般只需要修改如下幾個(gè)項(xiàng)：

sqoop.metastore.client.enable.autoconnect
sqoop.metastore.client.autoconnect.url
sqoop.metastore.client.autoconnect.username
sqoop.metastore.client.autoconnect.password
sqoop.metastore.server.location
sqoop.metastore.server.port

3：

 bin/sqoop help

 bin/sqoop help import

4:

[Hadoopuser@master  sqoop-1.1.0]$ bin/sqoop  import --connect jdbc:mysql://localhost/ppc --table data_ip  --username kwps -P

Enter password:

11/02/18 10:51:58 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not find appropriate Hadoop shim for 0.20.2

java.lang.RuntimeException: Could not find appropriate Hadoop shim for 0.20.2

        at com.cloudera.sqoop.shims.ShimLoader.loadShim(ShimLoader.java:190)

        at com.cloudera.sqoop.shims.ShimLoader.getHadoopShim(ShimLoader.java:109)

        at com.cloudera.sqoop.tool.BaseSqoopTool.init(BaseSqoopTool.java:173)

        at com.cloudera.sqoop.tool.ImportTool.init(ImportTool.java:81)

        at com.cloudera.sqoop.tool.ImportTool.run(ImportTool.java:411)

        at com.cloudera.sqoop.Sqoop.run(Sqoop.java:134)

        at org.apache.Hadoop.util.ToolRunner.run(ToolRunner.java:65)

        at org.apache.Hadoop.util.ToolRunner.run(ToolRunner.java:79)

        at com.cloudera.sqoop.Sqoop.runSqoop(Sqoop.java:170)

        at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:196)

        at com.cloudera.sqoop.Sqoop.main(Sqoop.java:205)

解決辦法:

默認(rèn)情況下:

./Hadoop-0.20.2/conf/hadoop-env.sh

# Extra Java runtime options.  Empty by default.
# export Hadoop_OPTS=-server

需要更改成:

export Hadoop_OPTS="-Djava.net.preferIPv4Stack=true -Dsqoop.shim.jar.dir=/home/hadoopuser/sqoop-1.1.0/shims"

特別需要注意的是：

Sqoop目前在Apache 版本的Hadoop 0.20.2上是無(wú)法使用的。

目前只支持CDH 3 beta 2版本。所以如果想使用的話，得升級(jí)到 CDH 3 beta 2版本了。

“Sqoop does not run with Apache Hadoop 0.20.2. The only supported platform is CDH 3 beta 2. It requires features of MapReduce not available in the Apache 0.20.2 release of Hadoop. You should upgrade to CDH 3 beta 2 if you want to run Sqoop 1.0.0.”

這個(gè)問(wèn)題 已經(jīng)被Cloudera 標(biāo)記為 Major Bug，希望能盡快解決吧。

服務(wù)器租用/服務(wù)器托管中國(guó)五強(qiáng)！虛擬主機(jī)域名注冊(cè)頂級(jí)提供商！15年品質(zhì)保障！--億恩科技[ENKJ.COM]