linuxgreenplumhawq

Pivotal greenplum - gpload issue with talend

When i try to run the gpload process from talend etl server.In that,I need to configure tgreenpluGPload Component first. While configuration to component it is looking for Remote Greenplum server files instead of Local windows based talend ETL files .

ENV Details

Talend server is based on - windows server 2012

Greenplum Cluster based on - centos 7

Main cause : Greenplum database server (Linux) is remote to ETL talend server (window). hence , when i am running the job from window server greenplum db server is remote for it . ALSO, i am not able to configure component tgreenplumGPload.

Screenshot tgreenplumGPload setting :

Basic setting

Advanced setting

More Detail : 1) gpfdist program is running at the Greenplum master host. 

[gpadmin@mdw ~]$ ps -A | grep gpfdist
20071 pts/0    00:00:00 gpfdist
[gpadmin@mdw ~]$

2)Checked merge operation from gpdb commandline - following process is running in greenplum server .

[gpadmin@mdw ~]$ gpload -f gpload.yml
2017-02-25 20:20:48|INFO|gpload session started 2017-02-25 20:20:48
2017-02-25 20:20:48|INFO|started gpfdist -p 8081 -P 8082 -f "/home/gpadmin/demo/gp_RevenueReport_stg0.txt" -t 30
2017-02-25 20:20:48|INFO|running time: 0.20 seconds
2017-02-25 20:20:48|INFO|rows Inserted          = 0
2017-02-25 20:20:48|INFO|rows Updated           = 3
2017-02-25 20:20:48|INFO|data formatting errors = 0
2017-02-25 20:20:48|INFO|gpload succeeded

Q1: How to set up a shared folder on Linux for Windows to access.so that, We can utilize in tgreenplumGPload setting. Or is there any alternate way to do this.

Any help would be much appreciated !

Solution

  • gpfdist will run on the ETL server, not on the Master host.

    You will have to add the ETL server ip and name to the /etc/hosts file on all of the nodes in the Greenplum cluster. You will then need to make sure the ETL server can communicate directly with the segment hosts in the private network of Greenplum. This will require connecting the 10GB private switches used by Greenplum to your 10GB LAN and create a VLAN so you can access the nodes or you can run a 10GB cables from your ETL server to open ports of the 10GB switches and assign it an IP address that doesn't conflict with the existing hosts.