How to set up PySpark?

How do we go about setting up Apache Spark’s PySpark on Discovery?

1 Like

You can try installing pyspark via ‘pip3 install’:
module purge
module load usc
module load python/3.7.6
pip3 install --user pyspark

To check the available python packages:
pip3 list

When I try to install Python packages using pip3, I get an error saying

Did you add the --user option? This is needed to install into your home directory: pip3 install --user pyspark.

1 Like

Oh. I needed to add the “–user” option. Thank you.