How to use Azure Machine Learning Notebook on Spark

Machine learning is a growing technology, which enables computers to learn automatically from past data. Machine learning uses various algorithms for building mathematical models and making predictions use historical data or information. We have a model defined up to some parameters, and learning is the execution of a computer program to optimize the parameters of the model using the training data or experience. The model may be predictive to make predictions in the future, or descriptive to gain knowledge from data.

The following tutorial notebook shows an example of training machine learning models on tabular data. You can import this notebook and run it yourself.

Upload the CSV into your storage

1. Find your storage and container name in the portal JSON view

   

   

2. Navigate into your primary HDI storage>container>base folder> upload the CSV

   

   

3. Log in to your cluster and open the Jupyter Notebook

   

4. Import Spark MLlib Libraries to create the pipeline

   import pyspark
   from pyspark.ml import Pipeline, PipelineModel
   from pyspark.ml.classification import LogisticRegression
   from pyspark.ml.feature import VectorAssembler, StringIndexer, IndexToString
   

   

5. Read the CSV into a Spark dataframe

   df = spark.read.("abfss:///iris_csv.csv",inferSchema=True,header=True)

6. Split the data for training and testing

   iris_train, iris_test = df.randomSplit([0.7, 0.3], seed=123)

7. Create the pipeline and train the model

   assembler = VectorAssembler(inputCols=['sepallength', 'sepalwidth', 'petallength', 'petalwidth'],outputCol="features",handleInvalid="skip")
   indexer = StringIndexer(inputCol="class", outputCol="classIndex", handleInvalid="skip")
   classifier = LogisticRegression(featuresCol="features",
                                   labelCol="classIndex",
                                   maxIter=10,
                                   regParam=0.01)
   
   pipeline = Pipeline(stages=[assembler,indexer,classifier])
   model = pipeline.fit(iris_train)
   
   # Create a test `dataframe` with predictions from the trained model
   
   test_model = model.transform(iris_test)
   
   # Taking an output from the test dataframe with predictions
   
   test_model.take(1)
   

   

8. Evaluate the model accuracy

   import pyspark.ml.evaluation as ev
   evaluator = ev.MulticlassClassificationEvaluator(labelCol='classIndex')
   
   print(evaluator.evaluate(test_model,{evaluator.metricName: 'accuracy'}))