Apache Sqoop Commands

Updated: Aug 24, 2019

SQOOP IMPORT:

How to use the import tool in a variety of situations:


1.A basic import of a table named EMPLOYEES in the COMP database:

Importing an entire table is one of the most common and straightforward use cases of Sqoop. The result of this command will be a comma-separated CSV file where each row is stored in a single line.

$ sqoop import \
--connect jdbc:mysql://127.0.0.1:3306/comp
--username sqoop \
--password sqoop \
--table EMPLOYEES
--username sqoop \ 

2.Selecting specific columns from the EMPLOYEES table:

$ sqoop import \ 
--connect jdbc:mysql://127.0.0.1:3306/comp \ 
--username sqoop \ 
--password sqoop \ 
--table EMPLOYEES \ 
--columns "employee_id,first_name,last_name,job_title"                                                     

3. Controlling the import parallelism (using 8 parallel tasks):

Sqoop by default uses four concurrent map tasks to transfer data to Hadoop. Transferring bigger tables with more concurrent tasks should decrease the time required to transfer all data. You want the flexibility to change the number of map tasks used on a per-job basis.

$ sqoop import \ 
--connect jdbc:mysql://127.0.0.1:3306/comp \ 
--username sqoop \
--password sqoop \ 
--table EMPLOYEES \ -m 8

4.Specifying a Target Directory

$ sqoop import \ 
--connect jdbc:mysql://127.0.0.1:3306/comp \ 
--username sqoop \ 
--password sqoop \ 
--table EMPLOYEES \
--target-dir /hdfs/input/emp

5. Using a File Format Other Than CSV

Sqoop supports three different file formats; one of these is text, and the other two are binary. The binary formats are Avro and Hadoop’s SequenceFile. You can enable import into SequenceFile using the --as-sequencefile parameter:

$ sqoop import \ 
--connect jdbc:mysql://127.0.0.1:3306/comp \ 
--username sqoop \  
--password sqoop \ 
--table EMPLOYEES \
--as-sequencefile
“Or”
--as-avrodatafile

6. Specifying the delimiters to use in a text-mode import:

$ sqoop import \ 
--connect jdbc:mysql://127.0.0.1:3306/comp \ 
--username sqoop \ 
--password sqoop \ 
--table EMPLOYEES \
--fields-terminated-by '\t' --lines-terminated-by '\n' \
--optionally-enclosed-by '\"'

7. Importing All Your Tables

$ sqoop import-all-tables \ 
--connect jdbc:mysql://127.0.0.1:3306/sqoop \ 
--username sqoop \  
--password sqoop \ 
--table cities

Incremental Import:

Importing Only New Data


28 views

FOLLOW US ON

  • Facebook Social Icon
  • Twitter Social Icon
  • LinkedIn Social Icon

ADDRESS

Gachibowli, Hyderabad, Telangana, India

©2020  Data Science Authority