Scoopi Cluster on Kubenetes

Scoopi jobs can be deployed on Kubernetes Cluster.

Kubernetes Deployment

The sample kubernetes files are available at Scoopi Kubernetes. The scoopi.yaml configures kinds such as ClusterRole, ClusterRoleBinding, Service, PersistentVolume and PersistentVolumeClaim which are essential to run Scoopi and the scoopi jobs are defined in scoopi-job.yaml.

Download the files and apply them to k8s cluster.

kubectl apply -f scoopi.yaml
kubectl apply -f scoopi-job.yaml    

It runs /defs/examples/fin/jsoup/ex-13 on Scoopi Cluster spanned across 5 K8s pods. The output data is created in worker node /data/scoopi/output dir.

To configure Scoopi or to run your own def, create two additional directories /data/scoopi/defs and /data/scoopi/conf in all K8s nodes. From Scoopi release zip, copy defs and conf folder contents to them. Define two additional PersistentVolume and PersistentVolumeClaim in scoopi.yaml and mount the volumes in scoopi-job.yaml.

Sample files are tested on Kubernetes MiniKube and also, on real Kubernetes Cluster.

Shared Volumes

By default, the Scoopi output files are scattered across the K8s worker nodes. Similarly, the conf and defs files are to be placed in all nodes. For production setup, use shared volume feature of VM or common nfs directory and centralize conf, defs, output, logs and data folders on them.