MaxCompute:Submit a MapReduce job

<GENERIC_OPTIONS> includes the following optional parameters:

• -conf <configuration file>: specifies the JobConf configuration file. This file contains the settings of JobConf in SDK.

  The following code shows the template of a JobConf file:

  <configuration>
     <property>
        <name>import.filename</name>
        <value>resource.txt</value>
     </property>
  </configuration>          

  In this example, a variable named import.filename whose value is resource.txt is specified in the JobConf configuration file.

  You can call the JobConf interface in MapReduce to obtain the value of this variable. You can also call the JobConf interface in SDK to obtain the variable value. For more information, see Resource samples.

  Example:

  -- Add a jar to the MaxCompute project environment and submit a jar job.
  add jar data\mapreduce-examples.jar;
  jar -resources mapreduce-examples.jar -classpath data\mapreduce-examples.jar
      org.alidata.odps.mr.examples.WordCount wc_in wc_out;
      
  -- Add files and a jar to the MaxCompute project environment and submit a jar job.  
  add file data\src.txt;
  add jar data\mapreduce-examples.jar;
  jar -resources src.txt,mapreduce-examples.jar -classpath data\mapreduce-examples.jar
      org.alidata.odps.mr.examples.WordCount wc_in wc_out;
   
  -- Create a new table test_table based on the structure of the table wc_in, add a file, the test_table and the jar to the MaxCompute project environment, then submit a jar job.  
  add file data\a.txt;
  add table wc_in as test_table;
  add jar data\work.jar;
  jar -conf odps-mapred.xml -resources a.txt,test_table,work.jar
      -classpath data\work.jar:otherlib.jar
      -D import.filename=resource.txt org.alidata.odps.mr.examples.WordCount args;

• -resources <resource_name_list>: specifies the resources that are used to run a MapReduce job. In most cases, you must specify the names of the resources that are used by the map or reduce function in the resource_name_list.

  Note

  • If the map or reduce function reads other MaxCompute resources, you must also add the names of these resources to resource_name_list.

  • Multiple resources must be separated by commas (,). If you use cross-project resources, add PROJECT/resources/ before resource_name_list. Example: -resources otherproject/resources/resfile.

  • For more information about how to use the map or reduce function to read resources, see Resource samples.

• -classpath <local_file_list>: specifies the classpath that is used to run a MapReduce job in local mode. This parameter specifies the local paths. The paths include both the relative and absolute paths of the JAR package where the main function is located.

  Package names are separated by default file delimiters. In most cases, the Windows operating system uses semicolons (;) as the default file delimiter, and Linux uses commas (,). If you run a MapReduce job on a cloud server, separate package names with commas (,).

  Note

  You may prefer to compile the main function in the same package as the map or reduce function. For more information, see WordCount samples. In this case, when you execute the sample program, mapreduce-examples.jar is included in both the -resources and -classpath options. However, the -resources option references the map or reduce function for distributed execution. The -classpath option references the main function for local execution. The specified JAR package must also be saved in a local directory.

• -D <prop_name>=<prop_value>: specifies the Java property of <mainClass> for local execution. You can specify multiple properties.

• -l: specifies that the MapReduce job is executed in local mode. This option is used for program debugging.

<MAIN_CLASS>: Main class. For example, com.example.Main.

[ARGS]: The parameters of the program (optional). These parameters are passed to the main method of the Main class.