Application Configuration
The JupyterHub Outpost uses a configuration file outpost_config.py similar to jupyterhub_config.py of JupyterHub. The Spawner configuration for the Outpost is therefore similar to the Spawner configuration in JupyterHub.
The easiest way is configure the Outpost's configuration file is via the outpostConfig key in the helm chart's values.yaml file.
Persistent database
To use a persistent database such as postgresql with JupyterHub Outpost, use extraEnvVarsSecrets in your values.yaml file. All possible values related to the database connection can be found in the source code itself.
Ensure that you have a database such postgres installed and that a JupyterHub Outpost user and database exists.
Example SQL commands for postgresql:
CREATE USER jupyterhuboutpost WITH ENCRYPTED PASSWORD '...';
CREATE DATABASE jupyterhuboutpost OWNER jupyterhuboutpost;
Create a secret in your Outpost namespace with the required values before installing JupyterHub Outpost:
kind: Secret
metadata:
name: my-db-secret
...
stringData:
SQL_TYPE: "postgresql"
SQL_USER: "jupyterhuboutpost"
SQL_PASSWORD: "..."
SQL_HOST: "postgres.database.svc"
SQL_PORT: "5432"
SQL_DATABASE: "jupyterhuboutpost"
And add the database secret to your Outpost values.yaml file:
Simple KubeSpawner
Jupyter4NFDI sends a profile to your Spawner. This is a small example which ignores the profile and always runs the Jupyter Server with the predefined setup.
values.yaml file:
outpostConfig: |
from kubespawner import KubeSpawner
c.JupyterHubOutpost.spawner_class = KubeSpawner
c.KubeSpawner.start_timeout = 600
async def profile_list(spawner):
jupyterhub_name = spawner.jupyterhub_name
spawner.log.info(f"{spawner._log_name} - Received these user_options from {jupyterhub_name}-JupyterHub: {spawner.user_options}")
slug = spawner.user_options.get("profile", "default")
default_image = "jupyter/minimal-notebook:notebook-7.0.3"
return [
{
"display_name": slug,
"slug": slug,
"kubespawner_override": {
"image": default_image
}
}
]
c.KubeSpawner.profile_list = profile_list
Update or install JupyterHub Outpost with values.yaml file:
helm upgrade --install -f values.yaml --namespace outpost outpost jupyterhub-outpost/jupyterhub-outpost
In this example we use the KubeSpawner, but you can use any JupyterHub Spawner.
Customize Logging
For the logging configuration, the Outpost offers these options (corresponding to the logging options of JupyterHub):
c.JupyterHubOutpost.log_level = ...
c.JupyterHubOutpost.log_datafmt = ...
c.JupyterHubOutpost.log_format = ...
If more customization is required, one can do this directly in the outpost_config.py file itself (possible via the outpostConfig key of the helm chart).
# Suppress /ping loggings, created by k8s livenessprobe
uvicorn_access = logging.getLogger("uvicorn.access")
class UvicornFilter(logging.Filter):
def filter(self, record):
try:
if "/ping" in record.args:
return False
except:
pass
return True
uvicorn_access.addFilter(UvicornFilter())
# Suppress missing static files by Tornado Logger
tornado_general = logging.getLogger("tornado.general")
class TornadoGeneralLoggingFilter(logging.Filter):
def filter(self, record):
# I don't want to see this log line generated by tornado
if str(record.msg).startswith("Could not open static file"):
return False
return True
tornado_general.addFilter(TornadoGeneralLoggingFilter())
import os
logged_logger_name = os.environ.get("LOGGER_NAME", "MyOutpostInstance")
c.JupyterHubOutpost.log_format = f"%(color)s[%(levelname)1.1s %(asctime)s.%(msecs).03d {logged_logger_name} %(name)s %(module)s:%(lineno)d]%(end_color)s %(message)s"
Sanitize Spawner.start response
JupyterHub Outpost relies on the start function of the configured SpawnerClass to determine where the single-user server will run. For instance, with KubeSpawner, the KubeSpawner.start() function might return a URL like http://jupyter-<id>-<user_id>:<port>, which Outpost will pass along to JupyterHub.
The OutpostSpawner then uses this information to set up an SSH port-forwarding process with the command -L 0.0.0.0:<local_jhub_port>:jupyter-<id>-<user_id>:<port>. Afterward, JupyterHub will access the single-user server at http://localhost:<local_jhub_port>.
If the response from the start function isn't correct, the OutpostSpawner and Outpost won't work together properly. However, to support most Spawners, you can customize the response that is sent to the OutpostSpawner.
# In the `outpostConfig` key of your helm values.yaml file or your outpost_config.py file:
# This may be a coroutine
def sanitize_start_response(spawner, original_response):
# ... determine the correct location for the new single-user server
return "<...>"
c.JupyterHubOutpost.sanitize_start_response = sanitize_start_response
If you don't know where your single-user server will be running at the end of the
startfunction, you have to return an empty string. In that case, JupyterHub OutpostSpawner won't create a ssh port-forwarding process. Instead, the start process of the single-user server has to send a POST request to the$JUPYTERHUB_SETUPTUNNEL_URLurl. Have a look at the API Endpoints of the OutpostSpawner (https://jupyterhub-outpostspawner.readthedocs.io/en/latest/apiendpoints.html) for more information.
Disable configuration overwrite
By default, Jupyter4NFDI can overwrite the JupyterHub Outpost configuration with the OutpostSpawner's custom_misc function. As an administrator of the JupyterHub Outpost service, you can prevent this.
# In the `outpostConfig` key of your helm values.yaml file or your outpost_config.py file:
async def allow_override(jupyterhub_name, misc):
if jupyterhub_name == "trustedhub":
return True
if list(misc.keys()) != ["image"]:
return False
return misc.get("image", "None") in ["allowed_image1", "allowed_image2"]
c.JupyterHubOutpost.allow_override = allow_override
The above example leads to the following behaviour:
- JupyterHub with credential username "trustedhub" can overwrite anything.
- If a JupyterHub (other than trustedhub) tries to overwrite anything except the image key, it will not be allowed.
- The given image must be allowed_image1 or allowed_image2.
If
custom_miscin the POST request is empty, theallow_overridefunction will not be called. Ifallow_overridereturns False, the JupyterLab will not be started. An error message will be returned to the JupyterHub OutpostSpawner and shown to the user.
Recreate ssh tunnels at startup
If your JupyterHub Outpost is used as a ssh node in the JupyterHub OutpostSpawner, all port-forwarding processes have to be recreated if the JupyterHub Outpost service was restarted. While restarting, existing ssh port-forwarding processes will fail after a few seconds and the user's single-user server would be unreachable.
By default tunnels will be recreated at JupyterHub Outpost restarts. You can disable this behaviour with the ssh_recreate_at_start key.
# In the `outpostConfig` key of your helm values.yaml file or your outpost_config.py file:
async def restart_tunnels(wrapper, jupyterhub_credential):
if jupyterhub_credential == "local_jupyterhub":
return False
return True
c.JupyterHubOutpost.ssh_recreate_at_start = restart_tunnels
# c.JupyterHubOutpost.ssh_recreate_at_start = False
JupyterHub Outpost will use the stored JupyterHub API token to recreate the port-forwarding process. If the API token is no longer valid, this will fail. The single-user server would then be unreachable and must be restarted by the user.
Flavors - manage resource access for multiple JupyterHubs
By default, all connected JupyterHubs may use all available resources. It's possible to configure "flavors" for each connected JupyterHub, offering only a part of the available resources.
For this configuration three attributes are crucial:
- flavors
- flavors_undefined_max
- flavors_update_token
Flavors
Configure different flavors, which can be used in Spawner configuration.
async def flavors(jupyterhub_name):
if jupyterhub_name == "privileged":
return {
"typea": {
"max": -1,
"weight": 10,
"display_name": "2GB RAM, 1VCPU, 5 days",
"description": "JupyterLab will run for max 5 days with 2GB RAM and 1VCPU.",
"runtime": {"days": 5},
},
}
else:
return {
"typeb": {
"max": 10,
"weight": 9,
"display_name": "4GB RAM, 1VCPUs, 2 hours",
"description": "JupyterLab will run for max 2 hours with 4GB RAM and 1VCPUs.",
"runtime": {"hours": 2},
},
}
c.JupyterHubOutpost.flavors = flavors
The connected JupyterHub "privileged" can start infinite jupyter servers. The servers will be stopped after 5 days by the JupyterHub Outpost.
Any other connected JupyterHub can start up to 10 jupyter servers (all users together for each JupyterHub, not combined for all JupyterHubs).
The according RAM / VCPUs restrictions are configured later on in the config file at c.KubeSpawner.profile_list or c.KubeSpawner.[mem_guarantee|mem_limit|cpu_guarantee|cpu_limit].
JupyterHub OutpostSpawner has to send the chosen flavor in user_options.flavor when starting a notebook server.
User specific flavors
JupyterHub Outpost supports user-specific flavors, which allows administrators to assign different resource configurations to individual users. This feature is especially useful in environments where users have varying levels of resource needs or access rights. With user-specific flavors, you can:
- Block access for specific users
- Prioritize certain users with more resources
- Allocate minimal resources for external or guest users
- Tailor resource allocation for different user groups
How It Works
The concept of "flavors" allows for the customization of resources based on user identity. By configuring these flavors, administrators can control the resource allocation and user access in a flexible manner.
Flavors are typically defined by setting parameters such as CPU, memory, disk space, and other configurations needed for running the user's server. The configuration is linked to each user, and the appropriate settings are applied when they spawn a jupyter server.
Use Cases
- Blocked Users: Administrators can configure flavors for specific users that deny access, effectively blocking them from launching any jupyter servers.
- Prioritized Users: For power users, administrators can assign more resources (e.g., higher CPU, additional memory) to ensure they have the performance needed for demanding tasks.
- External Users: For guest or external users, administrators may provide a default, minimal resource allocation to prevent overuse of the system’s resources.
Configuration
To assign a flavor to a user, you will need to update the JupyterHub Outpost configuration to define and link flavors to specific user identities.
Example configuration snippet:
resources = {
"s1": {
"node_selector": {"smallnode": "true"},
"cpu_guarantee": 0.1,
"cpu_limit": 1,
"mem_guarantee": "256M",
"mem_limit": "2048M"
},
"l1": {
"node_selector": {"bignode": "true"},
"cpu_guarantee": 1,
"cpu_limit": 4,
"mem_guarantee": "2048M",
"mem_limit": "65536M"
}
}
awesome_users = {
"l1": {
"max": 50,
"weight": 10,
"display_name": "64GB RAM, 4VCPUs, 12 hours",
"description": "JupyterLab will run for max 12 hours with 64GB RAM and 4VCPUs.",
"runtime": {"hours": 12},
},
"s1": {
"max": 20,
"weight": 11,
"display_name": "2GB RAM, 1VCPU, 4 hours",
"description": "JupyterLab will run for max 4 hours with 2GB RAM and 1VCPU.",
"runtime": {"hours": 4},
}
}
default_users = {
"s1": {
"max": 10,
"weight": 11,
"display_name": "2GB RAM, 1VCPU, 4 hours",
"description": "JupyterLab will run for max 4 hours with 2GB RAM and 1VCPU.",
"runtime": {"hours": 4},
}
}
async def userflavors(outpost, jupyterhub_name, auth_dict):
if auth_dict.get("username", "").endswith("mycomp.org"):
return awesome_users
else:
try:
with open("/mnt/blocked_users.txt", "r") as f:
blocked_users = f.read()
if auth_dict.get("username", "") in blocked_users:
return False
except:
outpost.log.exception("Could not load blocklist")
return default_users
c.JupyterHubOutpost.user_flavors = userflavors
from kubespawner import KubeSpawner
c.JupyterHubOutpost.spawner_class = KubeSpawner
async def profile_list(spawner):
jupyterhub_name = spawner.jupyterhub_name
spawner.log.info(f"{spawner._log_name} - Received these user_options from {jupyterhub_name}-JupyterHub: {spawner.user_options}")
slug = spawner.user_options.get("profile", "default")
flavor = spawner.user_options.get("flavor", "s1")
default_image = "jupyter/minimal-notebook:notebook-7.0.3"
return [
{
"display_name": slug,
"slug": slug,
"kubespawner_override": {
"image": default_image,
"node_selector": resources[flavor]["node_selector"],
"cpu_guarantee": resources[flavor]["cpu_guarantee"],
"cpu_limit": resources[flavor]["cpu_limit"],
"mem_guarantee": resources[flavor]["mem_guarantee"],
"mem_limit": resources[flavor]["mem_limit"]
}
}
]
c.KubeSpawner.profile_list = profile_list
In this example you will give users of your company the full extend of your resources, while other users get less resources.
- Pro-Tipp If you manage the flavors and resources in a ConfigMap and mount it in your Outpost, you can update flavors and resources without restarting the Outpost.
All users will have a shared counter for servers with the s1 flavor. If 10 Jupyter servers of flavor s1 are running, default users won't be able to start a Jupyter server. This allows you to reserve 10 Jupyter servers of flavor s1 to your employees, workshops, or other use cases.
Benefits
- Resource Management: Fine-grained control over resource allocation ensures efficient use of system resources.
- User Control: Administrators can easily adjust resource access based on user needs or status (e.g., external user vs internal user).
- Scalability: As your user base grows, you can easily apply different flavors to new users without major changes to the overall configuration.
Undefined Max
If JupyterHub OutpostSpawner does not send a flavor in user_options c.JupyterHubOutpost.flavors_undefined_max will be used to limit the available resources. This value is also used, if the given flavor is not part of the previously defined flavors dict. Default is -1, which allows infinite notebook servers for all unknown or unconfigured flavored notebook servers.
This example does not allow any jupyter server with a flavor, that's not defined in c.JupyterHubOutpost.flavors. Enables full control of the available resources.
Update Tokens
The JupyterHub OutpostSpawner offers an APIEndpoint, which receives or offers the current Flavors of all connected JupyterHub Outposts. With this function the Outpost will inform the connected JupyterHubs at each start/stop of a notebook server, about the current flavor situation. The corresponding URL will be given by the OutpostSpawner.
import os
async def flavors_update_token(jupyterhub_name):
token = os.environ.get(f"FLAVOR_{jupyterhub_name.upper()}_AUTH_TOKEN", "")
if not token:
raise Exception(f"Flavor auth token for {jupyterhub_name} not configured.")
return token
c.JupyterHubOutpost.flavors_update_token = flavors_update_token
In case of an exception the update is not send to JupyterHub. This will not interfere with the start of the notebook server.
Each connected JupyterHub must provide a service token with scope custom:outpostflavors:set to the Outpost administrator.