Hello programmers okay?
I was programming and run the pig command returns me a syntax error in my code and I am unable to solve.
Below I will post the code and the error raised.
Apparently the error occurs in the syntax of the line 41 which is sortedResults
Code:
--Load files into relations
month1 = LOAD 'hdfs:/data/big/data/weather/201201hourly.txt' USING PigStorage(',');
month2 = LOAD 'hdfs:/data/big/data/weather/201202hourly.txt' USING PigStorage(',');
month3 = LOAD 'hdfs:/data/big/data/weather/201203hourly.txt' USING PigStorage(',');
month4 = LOAD 'hdfs:/data/big/data/weather/201204hourly.txt' USING PigStorage(',');
month5 = LOAD 'hdfs:/data/big/data/weather/201205hourly.txt' USING PigStorage(',');
month6 = LOAD 'hdfs:/data/big/data/weather/201206hourly.txt' USING PigStorage(',');
--Combine relations
months = UNION month1, month2, month3, month4, month5, month6;
/* Splitting relations
SPLIT months INTO
splitMonth1 IF SUBSTRING(date, 4, 6) == '01',
splitMonth2 IF SUBSTRING(date, 4, 6) == '02',
splitMonth3 IF SUBSTRING(date, 4, 6) == '03',
splitRest IF (SUBSTRING(date, 4, 6) == '04' OR SUBSTRING(date, 4, 6) == '04');
*/
/* Joining relations
stations = LOAD 'hdfs:/data/big/data/QCLCD201211/stations.txt' USING PigStorage() AS (id:int, name:chararray)
JOIN months BY wban, stations by id;
*/
--filter out unwanted data
clearWeather = FILTER months BY SkyCondition == 'CLR';
--Transform and shape relation
shapedWeather = FOREACH clearWeather GENERATE date, SUBSTRING(date, 0, 4) as year, SUBSTRING(date, 4, 6) as month, SUBSTRING(date, 6, 8) as day, skyCondition, dryTemp;
--Group relation specifying number of reducers
groupedMonthDay = GROUP shapedWeather BY (month, day) PARALLEL 10;
--Aggregate relation
aggedResults = FOREACH groupedByMonthDay GENERATE group as MonthDay, AVG(shapedWeather.dryTemp), MIN(shapedWeather.dryTemp), MAX(shapedWeather.dryTemp), COUNT(shapedWeather.dryTemp) PARALLEL 10;
--Sort relation
sortedResults = SORT aggedResults BY $1 DESC;
--Store results in HDFS
STORE sortedResults INTO 'hdfs:/data/big/data/weather/pigresults' USING PigStorage(':');
Here is the return I get to play the file which is contained in my code.
Error:
14/10/20 22:30:47 WARN pig.Main: Cannot write to log file: //pig_1413851447162.log
2014-10-20 22:30:47,165 [main] INFO org.apache.pig.Main - Apache Pig version 0.12.1 (r1585011) compiled Apr 05 2014, 01:41:34
2014-10-20 22:30:47,486 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /home/eduardo/.pigbootup not found
2014-10-20 22:30:47,510 [main] WARN org.apache.pig.Main - Cannot write to log file: ///weather.pig1413851447510.log
2014-10-20 22:30:47,629 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://localhost:54310
2014-10-20 22:30:47,770 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: localhost:54311
2014-10-20 22:30:48,005 [main] ERROR org.apache.pig.PigServer - exception during parsing: Error during parsing. <file home/eduardo/Documentos/pig/weather.pig, line 41, column 0> Syntax error, unexpected symbol at or near 'sortedResults'
Failed to parse: <file home/eduardo/Documentos/pig/weather.pig, line 41, column 0> Syntax error, unexpected symbol at or near 'sortedResults'
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:241)
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:179)
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1678)
at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1411)
at org.apache.pig.PigServer.parseAndBuild(PigServer.java:344)
at org.apache.pig.PigServer.executeBatch(PigServer.java:369)
at org.apache.pig.PigServer.executeBatch(PigServer.java:355)
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:140)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:202)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
at org.apache.pig.Main.run(Main.java:607)
at org.apache.pig.Main.main(Main.java:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
2014-10-20 22:30:48,008 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1200: <file home/eduardo/Documentos/pig/weather.pig, line 41, column 0> Syntax error, unexpected symbol at or near 'sortedResults'
2014-10-20 22:30:48,008 [main] WARN org.apache.pig.tools.grunt.Grunt - There is no log file to write to.
2014-10-20 22:30:48,008 [main] ERROR org.apache.pig.tools.grunt.Grunt - org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error during parsing. <file home/eduardo/Documentos/pig/weather.pig, line 41, column 0> Syntax error, unexpected symbol at or near 'sortedResults'
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1691)
at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1411)
at org.apache.pig.PigServer.parseAndBuild(PigServer.java:344)
at org.apache.pig.PigServer.executeBatch(PigServer.java:369)
at org.apache.pig.PigServer.executeBatch(PigServer.java:355)
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:140)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:202)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
at org.apache.pig.Main.run(Main.java:607)
at org.apache.pig.Main.main(Main.java:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
Caused by: Failed to parse: <file home/eduardo/Documentos/pig/weather.pig, line 41, column 0> Syntax error, unexpected symbol at or near 'sortedResults'
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:241)
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:179)
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1678)
... 15 more
Pig doesn't have SORT command. You have to try with "ORDER" command. try like this
sortedResults = ORDER aggedResults BY $1 DESC;