ylliX - Online Advertising Network
Vertex AI - Antrophic and Mistral models: Why does it require Imegen access?

How to connect to SPARK using SPARK Connect from a client computer


I have installed Spark on a PI using the following docker compose scripts.
Everything runs fine however i am not able to connect to Spark from my laptop using pyspark and spark connect.
It seems it cannot connect to the spark server somehow, I am not an expert in docker or spark.
All the guides about spark connect is they run it on the host where spark is running which doesn’t make sense..

like: spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()s

Version: '3'

services:
spark-master:
image: apache/spark:3.5.1
container_name: spark-master
hostname: spark-master
ports:
\- "8080:8080"
\- "7077:7077"
environment:
\- SPARK_MASTER_HOST=spark-master
\- SPARK_MASTER_PORT=7077
\- SPARK_MASTER_WEBUI_PORT=8080
entrypoint:
\- "bash"
\- "-c"
\- "/opt/spark/sbin/start-master.sh && tail -f /dev/null"
volumes:
\- ./data:/opt/spark/work-dir/spark-warehouse/data:rw

spark-connect:
image: apache/spark:3.5.1
container_name: spark-connect
hostname: spark-connect
ports:
\- "4040:4040"
\- "15002:15002"
depends_on:
\- spark-master
volumes:
\- ./jars/spark-connect_2.12-3.5.1.jar:/opt/spark/jars/spark-connect_2.12-3.5.1.jar
\- ./data:/opt/spark/work-dir/spark-warehouse/data:rw
command:
\- "bash"
\- "-c"
\- "/opt/spark/sbin/start-connect-server.sh --jars /opt/spark/jars/spark-connect_2.12-3.5.1.jar && tail -f /dev/null"

spark-worker:
image: apache/spark:3.5.1
container_name: spark-worker
hostname: spark-worker
depends_on:
\- spark-master
entrypoint:
\- "bash"
\- "-c"
\- "/opt/spark/sbin/start-worker.sh spark://spark-master:7077 && tail -f /dev/null"
volumes:
\- ./data:/opt/spark/work-dir/spark-warehouse/data:rw

Does anyone has an easy guide to run spark (docker) on a seperate server and connect to it using thin client (pyspark)? thanks

I tried several solution, like setting spark_local_ip, changing docker network but without success



Source link

Leave a Reply

Your email address will not be published. Required fields are marked *