How to set up PySpark?

Software and Programming Python
1

How do we go about setting up Apache Spark’s PySpark on Discovery?

1 Like
2

You can try installing pyspark via ‘pip3 install’:
module purge
module load usc
module load python/3.7.6
pip3 install --user pyspark

To check the available python packages:
pip3 list

3

When I try to install Python packages using pip3, I get an error saying

image
image964×41 3.46 KB

4

Did you add the --user option? This is needed to install into your home directory: pip3 install --user pyspark.

1 Like
5

Oh. I needed to add the “–user” option. Thank you.