DATABRICKS-MACHINE-LEARNING-ASSOCIATE Online Practice Questions and Answers

Correct Answer: C

The pandas API on Spark DataFrames are made up of Spark DataFrames with additional metadata. The pandas API on Spark aims to provide the pandas-like experience with the scalability and distributed nature of Spark. It allows users to

work with pandas functions on large datasets by leveraging Spark's underlying capabilities.

References:

Databricks documentation on pandas API on Spark: pandas API on Spark

Correct Answer: A

To parallelize the inference of group-specific models using the Pandas Function API in PySpark, you can use theapplyInPandasfunction. This function allows you to apply a Python function on each group of a DataFrame and return a

DataFrame, leveraging the power of pandas UDFs (user-defined functions) for better performance.

prediction_df = ( df.groupby("device_id") .applyInPandas(apply_model, schema=apply_return_schema) )

In this code:

groupby("device_id"): Groups the DataFrame by the "device_id" column. applyInPandas(apply_model, schema=apply_return_schema): Applies theapply_modelfunction to each group and specifies the schema of the return DataFrame.

References:

PySpark Pandas UDFs Documentation

Correct Answer: D

To determine the number of machine learning models that can be trained in parallel, we need to calculate the total number of combinations of hyperparameters. The given hyperparameter grid includes:

Hyperparameter 1: [2, 5, 10] (3 values)

Hyperparameter 2: [50, 100] (2 values)

The total number of combinations is the product of the number of values for each hyperparameter:3 (values of Hyperparameter 1)? (values of Hyperparameter 2)=63 (value s of Hyperparameter 1)? (values of Hyperparameter 2)=6 With 3-fold

cross-validation, each combination of hyperparameters will be evaluated 3 times. Thus, the total number of models trained will

be:6 (combinations)? (folds)=186 (combinations)? (folds)=18 However, the number of models that can be trained in parallel is equal to the number of hyperparameter combinations, not the total number of models considering cross-validation.

Therefore, 6 models can be trained in parallel.

References:

Databricks documentation on hyperparameter tuning: Hyperparameter Tuning

Correct Answer: C

To view the notebook that was run to create an MLflow run, you can click the "Source" link in the row corresponding to the run in the MLflow experiment page. The "Source" link provides a direct reference to the source notebook or script that

initiated the run, allowing you to review the code and methodology used in the experiment. This feature is particularly useful for reproducibility and for understanding the context of the experiment.

References:

MLflow Documentation (Viewing Run Sources and Notebooks).

Correct Answer: A

The suggestion not to one-hot encode categorical feature variables within the feature repository is justified because one-hot encoding can be problematic for some machine learning algorithms. Specifically, one-hot encoding increases the dimensionality of the data, which can be computationally expensive and may lead to issues such as multicollinearity and overfitting. Additionally, some algorithms, such as tree-based methods, can handle categorical variables directly without requiring one-hot encoding. References: Databricks documentation on feature engineering: Feature Engineering

Correct Answer: D

To speed up the model tuning process when dealing with a large number of model configurations, parallelizing the hyperparameter search using Hyperopt is an effective approach. Hyperopt provides tools likeSparkTrialswhich can run

hyperparameter optimization in parallel across a Spark cluster.

Example:

fromhyperoptimportfmin, tpe, hp, SparkTrials search_space = {'x': hp.uniform('x',0,1),'y':

hp.uniform('y',0,1) }defobjective(params):returnparams['x'] **2+ params['y'] **2spark_trials = SparkTrials(parallelism=4) best = fmin(fn=objective, space=search_space, algo=tpe.suggest, max_evals=100, trials=spark_trials) References:

Hyperopt Documentation

Correct Answer: C

For large datasets, Spark ML uses iterative optimization methods to distribute the training of a linear regression model. Specifically, Spark MLlib employs techniques like Stochastic Gradient Descent (SGD) and Limited-memory Broyden

Correct Answer: D

If the new solution requires that each of the three models computes a prediction for every record, the time efficiency during inference will be reduced. This is because the inference process now involves running multiple models instead of a single model, thereby increasing the overall computation time for each record. In scenarios where inference must be done by multiple models for each record, the latency accumulates, making the process less time efficient compared to using a single model. References: Model Ensemble Techniques

Correct Answer: C

To filter rows in a Spark DataFrame based on a condition, thefiltermethod is used. In this case, the condition is that the value in the "discount" column should be less than or equal to 0. The correct syntax uses thefiltermethod along with

thecolfunction from pyspark.sql.functions.

Correct code:

frompyspark.sql.functionsimportcol filtered_df = spark_df.filter(col("discount") <=0) Option A and D use Pandas syntax, which is not applicable in PySpark. Option B is closer but misses the use of thecolfunction.

References:

PySpark SQL Documentation

Correct Answer: C

The F1 score is the harmonic mean of precision and recall and is particularly useful in situations where there is a significant imbalance between positive and negative classes. When there is a class imbalance, accuracy can be misleading

because a model can achieve high accuracy by simply predicting the majority class. The F1 score, however, provides a better measure of the test's accuracy in terms of both false positives and false negatives.

Specifically, the F1 score should be used over accuracy when:

There is a significant imbalance between positive and negative classes. Avoiding false negatives is a priority, meaning recall (the ability to detect all positive instances) is crucial.

In this scenario, the F1 score balances both precision (the ability to avoid false positives) and recall, providing a more meaningful measure of a model's performance under these conditions.

References:

Databricks documentation on classification metrics: Classification Metrics