Pig JOIN

This execution cost me $1.76 for about 1 hour. The number of machines is the same(previous post).

X = FILTER ntriples BY (subject matches '.*business.*');
y = foreach X generate subject as subject2, predicate as predicate2, object as object2 PARALLEL 50;
j = JOIN X BY subject,y BY subject2 PARALLEL 50;
j = DISTINCT j PARALLEL 50;

Screen Shot 2014-08-26 at 8.06.23 PM

Counting the records in the file.

FILE = LOAD 'join-results';
FILE_C = GROUP FILE ALL;
FILE_COUNT = FOREACH FILE_C GENERATE COUNT(FILE);

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: