今天主要介紹下在框架中如何使用mapreduce,不涉及到mapreduce的使用講解

這邊主要的js代碼都將寫在js文件中,放在classpath下面統(tǒng)一維護(hù),修改起來也比較方便,如果直接用字符串拼接的方式在代碼中,難看又難維護(hù).

就算不用框架,就用驅(qū)動(dòng)操作mapreduce時(shí),自己也可以將js代碼寫在xml中,跟mybatis一樣,然后寫個(gè)工具類去讀取即可.

MapReduceOptions options = MapReduceOptions.options();

options.outputCollection("Article_MapReduce");

options.outputTypeReduce();

options.finalizeFunction("classpath:finalize.js");

MapReduceResults<ValueObject> reduceResults =

mongoTemplate.mapReduce("article_info", "classpath:map.js",

"classpath:reduce.js", options, ValueObject.class);

reduceResults.forEach(System.out::println);

outputCollection是指將結(jié)果輸出某個(gè)集合中

finalizeFunction是對(duì)應(yīng)的finalize的js函數(shù)代碼

mapReduce有多個(gè)重載方法,下面可以看到有不同的參數(shù),有可以指定輸入集合名稱的,也有直接傳Query的,用Query意味著可以處理符合條件的一些數(shù)據(jù),如果不指定Query,那么將處理集合中的所有數(shù)據(jù).

mongoTemplate.mapReduce(inputCollectionName, mapFunction,

reduceFunction, entityClass)

mongoTemplate.mapReduce(query, inputCollectionName,

mapFunction, reduceFunction, entityClass)

mongoTemplate.mapReduce(inputCollectionName,

mapFunction, reduceFunction, mapReduceOptions, entityClass)

mongoTemplate.mapReduce(query, inputCollectionName,

mapFunction, reduceFunction, mapReduceOptions, entityClass)

前面說到對(duì)應(yīng)的js代碼我們是寫在文件中,然后調(diào)用的時(shí)候傳這個(gè)文件的名稱,框架自己回去加載對(duì)應(yīng)的js代碼,我們從源碼中可以看到有讀取js代碼的方法.

public <T> MapReduceResults<T> mapReduce(Query query, String inputCollectionName, String mapFunction,

String reduceFunction, MapReduceOptions mapReduceOptions, Class<T> entityClass) {

String mapFunc = replaceWithResourceIfNecessary(mapFunction);

String reduceFunc = replaceWithResourceIfNecessary(reduceFunction);

DBCollection inputCollection = getCollection(inputCollectionName);

MapReduceCommand command = new MapReduceCommand(inputCollection, mapFunc, reduceFunc,

mapReduceOptions.getOutputCollection(), mapReduceOptions.getOutputType(),

query == null || query.getQueryObject() == null ? null

: queryMapper.getMappedObject(query.getQueryObject(), null));

protected String replaceWithResourceIfNecessary(String function) {

String func = function;

if (this.resourceLoader != null && ResourceUtils.isUrl(function)) {

Resource functionResource = resourceLoader.getResource(func);

if (!functionResource.exists()) {

throw new InvalidDataAccessApiUsageException(String.format("Resource %s not found!", function));

}

Scanner scanner = null;

try {

scanner = new Scanner(functionResource.getInputStream());

return scanner.useDelimiter("\\A").next();

} catch (IOException e) {

throw new InvalidDataAccessApiUsageException(String.format("Cannot read map-reduce file %s!", function), e);

} finally {

if (scanner != null) {

scanner.close();

}

}

}

return func;

}

下面貼出今天測(cè)試的js代碼,按文章的作者統(tǒng)計(jì)文章的次數(shù)

map.js

function() {

emit(this.author,1);

}

reduce.js

function(key,values) {

var sum = 0;

for (var i = 0; i < values.length; i++)

sum += values[i];

return sum;

}

finalize.js

function(key,reduce) {

return reduce;

}

finalize中沒有去格式化輸出的格式,所以輸出的格式是原始的格式

{ "_id" : "文章作者", "value" : 文章次數(shù) }

上面的調(diào)用代碼中雖然指定了輸出結(jié)果的集合名稱,但還是定義了ValueObject來接收返回值, 那么ValueObject的格式肯定也是id和value.

public class ValueObject {

private String id;

private Integer value;

public String getId() {

return id;

}

public void setId(String id) {

this.id = id;

}

public Integer getValue() {

return value;

}

public void setValue(Integer value) {

this.value = value;

}

@Override

public String toString() {

return "ValueObject [id=" + id + ", value=" + value + "]";

}

}

在庫(kù)中查詢的原始數(shù)據(jù)格式如下

> db.Article_MapReduce.find();

{ "_id" : "jason", "value" : 1 }

{ "_id" : "mk", "value" : 1 }

{ "_id" : "yinjihuan", "value" : 18 }

>

源碼下載：https://github.com/yinjihuan/cxytiandi