Spark Installation On OS X

這次的心得分享為安裝spark和使用mllib的範例，那就分兩個部分介紹吧！

安裝Spark

透過Homebrew安裝最新版本的Spark

$brew install apache-spark

$cd /usr/local/Cellar/apache-spark/1.2.0/libexec/conf

$cp log4j.properties.template log4j.properties

$vim log4j.properties

$cd /usr/local/Cellar/apache-spark/1.2.0
$./bin/spark-shell

MLlib Demo

以ALS為例，首先安裝相依套件

import org.apache.spark.mllib.recommendation.ALS
import org.apache.spark.mllib.recommendation.MatrixFactorizationModel
import org.apache.spark.mllib.recommendation.Rating

val data = sc.textFile("libexec/data/mllib/als/ydtest.data")
val ratings = data.map(_.split(',') match { case Array(user, item, rate) =>
  Rating(user.toInt, item.toInt, rate.toDouble)
})
// Build the recommendation model using ALS
val rank = 10
val numIterations = 10
val model = ALS.train(ratings, rank, numIterations, 0.01)
// Evaluate the model on rating data
val usersProducts = ratings.map { case Rating(user, product, rate) =>
  (user, product)
}
val predictions =
  model.predict(usersProducts).map { case Rating(user, product, rate) =>
    ((user, product), rate)
  }
val ratesAndPreds = ratings.map { case Rating(user, product, rate) =>
  ((user, product), rate)
}.join(predictions)
val MSE = ratesAndPreds.map { case ((user, product), (r1, r2)) =>
  val err = (r1 - r2)
  err * err
}.mean()
println("Mean Squared Error = " + MSE)