Server-related updates

Features
1

This is the root topic for discussing Datagrok server-related features and server-side computations, including scripting, clustering, deployment models, and server data access connectivity.

1 Like
2
3

Datagrok already allowed specifying Conda environments for scripts (currently for Python):

#name: TestEnvironment
#environment: EnvTest
#language: python
#output: string result
from glom import glom # glom is specified in EnvTest environment file
result = glom({'a': {'b': {'c': 'd'}}}, 'a.b.c')  # returns 'd'

In this case, environments/EnvTest.yaml is part of the package which TestEnvironment script belongs to:

name: EnvTest
channels:
  - conda-forge
dependencies:
  - python=3.6
  - glom

We’ve expanded this facility with two convenient features.

In-place environments. Now it’s possible to do this:

#name: TestEnvironment
#environment: channels: [conda-forge], dependencies: [python=3.6, glom]
#language: python
#output: string result
from glom import glom # glom is specified in EnvTest
result = glom({'a': {'b': {'c': 'd'}}}, 'a.b.c')  # returns 'd'

Simply add a body of a desired Conda-yaml as one-liner to the #environment parameter. The specified environment shall be created on the server once and be reused on the next run of scripts referencing it. This is available in any script, both from packages and ones created in the platform from the UI with New script.

Global environments. Now it’s possible to reference an environment specified in some package:

#name: TestEnvironment
#environment: GlobalEnvsPackage:EnvTest
#language: python
#output: string result
from glom import glom # glom is specified in EnvTest environment file
result = glom({'a': {'b': {'c': 'd'}}}, 'a.b.c')  # returns 'd'

Thus, any script (both in a package and ad-hoc) can reference any environment specified in any package. Once the package GlobalEnvsPackage is shared with a current user, this user will be able to run TestEnvironment re-using GlobalEnvsPackage:EnvTest. This allows setting up company-wide (global) environments in some dedicated package shared with all users involved.

Check a detailed info on the updated feature here: link.

1 Like
4