mrjob

mrjob lets you write MapReduce jobs in Python 2.5+ and run them on several platforms. You can:

• Write multi-step MapReduce jobs in pure Python
• Test on your local machine
• Run on a Hadoop cluster
• Run in the cloud using Amazon Elastic MapReduce (EMR)

To get started, install with pip:

pip install mrjob

Guides

Basics

• Quickstart
• Concepts

Writing jobs

• Basics
• Protocols
• Writing custom protocols
• Defining command line options
• Filtering task input with shell commands
• Specifying mappers, combiners, and reducers as shell commands

Running jobs

• Runners
• Running your job programmatically
• Testing jobs

Configuration

• Config file format and location
• Options available to all runners
• Hadoop-related options
• Configuration quick reference

Cookbook

• Putting your source tree in the PYTHONPATH
• Increasing the task timeout
• Writing compressed output

Running jobs on Elastic MapReduce

• Configuring AWS credentials
• Configuring SSH credentials
• Running an EMR Job
• Choosing Type and Number of EC2 Instances
• EMR runner options
• EMR job flow management tools
• The mrjob command
• Troubleshooting
• Advanced EMR usage

Working on mrjob

• Contributing to mrjob
• Interactions between runner and job

Reference

• What’s New
  • 0.4.0
  • 0.3.5
  • 0.3.3
  • 0.3.2
  • 0.3
• Reference
  • mrjob.compat - Hadoop version compatibility
  • mrjob.conf - parse and write config files
  • mrjob.emr - run on EMR
  • mrjob.hadoop - run on your Hadoop cluster
  • mrjob.inline - alternate (debugger-friendly) local testing
  • mrjob.job
  • mrjob.local - run locally for testing
  • mrjob.parse - log parsing
  • mrjob.protocol - input and output
  • mrjob.retry - retry on transient errors
  • mrjob.runner - base class for all runners
  • mrjob.util - general utility functions
• Guides
  • Quickstart
  • Concepts
  • Writing jobs in Python
  • The mrjob command
  • Runners
  • Config file format and location
  • Options available to all runners
  • Hadoop-related options
  • Configuration quick reference
  • Cookbook
  • Testing jobs
  • Elastic MapReduce Quickstart
  • EMR runner options
  • EMR job flow management tools
  • Troubleshooting
  • Advanced EMR usage
  • Contributing to mrjob
  • Interactions between runner and job

Indices and tables

• Search Page