Cannot open Calculated Columns editor from some dataframes

Hey Datagrok team, I think I’ve found an issue with the current Datagrok API (or maybe I am just wrong with my assumption).

I was using your example here where everything was working well: https://dev.datagrok.ai/js/samples/data-frame/modification/calculated-columns

However, when I was trying to open the Calculated Columns editor on one of our loaded Dataframes I always got this error in the console.

dataframe.ts:2126 Uncaught TypeError: grok.functions.eval(...).prepare is not a function
    at DataFrameDialogHelper.addNewColumn (dataframe.ts:2126)
    at addCalculatedColumn (index.js:28)

After some debugging it turns out, that it is not possible to open the editor on a dataframe which was loaded with grok.shell.t (or grok.shell.tables[0]). I was also able to reproduce this in the example above and added the steps to reproduce below.

Can you please take a look into this and let us know if this is a bug or an mistake on our side?

~ Andreas

Steps to reproduce:

  1. Run script from example above
  2. Open Chrome DevTools
  3. (Optional) Verify the dataframe was added to the grok context by typing: grok.shell.t into the browser console
  4. Run following command in the browser console: grok.shell.tables[0].dialogs.addNewColumn();

Expected behaviour: I would expect to get the same function editor opened as previously with the script
Actual behaviour: Error in the console log.

Hi @andreas.gasser.novartis.com The fix will be on dev in half an hour, thanks for reporting!
With grok.shell.tables, it’s better to set the current view as follows:

let df = grok.shell.tables[0];
grok.shell.v = grok.shell.view(df.name);
df.dialogs.addNewColumn();
1 Like

Thanks for fast response, will check fixed version later (for now using alternative callback to open modal).

While playing around, I was stumbling over some additional questions and yet another issue.

Question: What is the appropriate way to listen for the calculated function create, update or delete event? Was playing around with the event listener but couldn’t find an isolated event to listen on?

Bug: Once I’ve added an event listener for the d4-dialog-showed event by using grok.events.onEvent('d4-dialog-showed') and trigger the calculated function editor, I get the error message below. Is this yet another issue related to the function editor?

js_helper.dart:1772 Uncaught TypeError: null is not a constructor
    at eval (eval at runJs$1 (command.dart:319), <anonymous>:1:17)
    at Function.call$1 (login.dart.js?35690:526)
    at Object.Primitives_applyFunctionWithPositionalArguments (js_helper.dart:1516)
    at Modal.getJsObject$0 (js_helper.dart:1489)
    at StaticClosure.dart.getWrapper (grok_api.dart:322)
    at Object.Primitives_applyFunctionWithPositionalArguments (js_helper.dart:1516)
    at dart._callDartFunctionFast (js_helper.dart:1489)
    at js_dart2js.dart:667
    at Module.toJs (wrappers_impl.ts:45)
    at toJs (wrappers.ts:8)
    at events.ts:22
    at Function.call$1 (login.dart.js?35690:526)
    at _RootZone.runUnaryGuarded$2 (zone.dart:1307)
    at _BroadcastSubscription._sendData$1 (stream_impl.dart:330)
    at _BroadcastSubscription.dart._BufferingStreamSubscription._async$_add$1 (stream_impl.dart:257)
    at _SyncBroadcastStreamController__sendData_closure.call$1 (broadcast_stream_controller.dart:387)
    at _SyncBroadcastStreamController._forEachListener$1 (broadcast_stream_controller.dart:328)
    at _SyncBroadcastStreamController._sendData$1 (broadcast_stream_controller.dart:386)
    at _SyncBroadcastStreamController.dart._BroadcastStreamController.add$1 (broadcast_stream_controller.dart:254)
    at Object.J.add$1$ax (login.dart.js?35690:41394)
    at EventBus.fire$2 (event_bus.dart:64)
    at Object.AppEvents_fire (events.dart:120)
    at Modal.show$10$backgroundColor$center$centerAt$fullScreen$height$modal$showNextTo$width$x$y (dialog.dart:332)
    at Modal.show$9$backgroundColor$center$centerAt$fullScreen$height$modal$width$x$y (login.dart.js_1.part.js?35690:65557)
    at _initJsInterop_closure191.dart._initJsInterop_closure191.call$10 (grok_api.dart:722)
    at Object.Primitives__genericApplyFunctionWithPositionalArguments (js_helper.dart:1577)
    at Object.Primitives_applyFunctionWithPositionalArguments (js_helper.dart:1543)
    at dart._callDartFunctionFast (js_helper.dart:1489)
    at js_dart2js.dart:667
    at Dialog.show (widgets.ts:500)
    at AddNewColumnDialog.<anonymous> (add-new-column.ts:70)
    at Generator.next (<anonymous>)
    at fulfilled (add-new-column.ts:4)

We do need more tests for our API, and we’ll look into this :slightly_smiling_face:
There’s an example for narrowing the onMetadataChanged event for specific tag changes, e.g., conditional coloring: https://dev.datagrok.ai/js/samples/data-frame/events/events.
And for calculated columns, it looks quite similar:

let df = DG.DataFrame.fromColumns([
  DG.Column.fromList(DG.TYPE.FLOAT, 'x', [1, 2, 3]),
  DG.Column.fromList(DG.TYPE.FLOAT, 'y', [4, 5, 6]),
  DG.Column.fromList(DG.TYPE.FLOAT, 'z', [7, 8, 9])
]);
let col = await df.columns.addNewCalculated('new', '${x}+${y}-${z}');
let view = grok.shell.addTableView(df);

// Listen to formula changes of a calculated column
df.onMetadataChanged
  .pipe(rxjs.operators.filter(data => data.args.key === DG.TAGS.FORMULA))
  .subscribe(async (data) => {
    grok.shell.info(`${data.args.change} - ${data.args.key} - ${data.args.value}`);
    if (data.args.change === 'set' && data.args.source === col) {
      // Apply a formula to the column (returns a new column instance preserving the metadata)
      col = await data.args.source.applyFormula('Avg($[x])');
    }
  });

// Change the formula either programmatically or from the interface
col.tags[DG.TAGS.FORMULA] = 'Avg($[x])';

I was also thinking to reuse onColumnsAdded & onColumnsRemoved for this purpose, too, but haven’t try it yet. There may still be some rough edges to work on.

1 Like

Thanks for this example, this works perfect!

One more question: We are also interested in the onColumnRemoved event (which is already available through the dataframe) to support the remove use case. Is there a way to identify which column was removed based on the event? At the moment, the event data of this event is actually empty or not accessible (see example below).

Event data:
1. args: Object

  1. [[Prototype]]: Object

    1. constructor: ƒ Object()
    2. hasOwnProperty: ƒ hasOwnProperty()
    3. isPrototypeOf: ƒ isPrototypeOf()
    4. propertyIsEnumerable: ƒ propertyIsEnumerable()
    5. toLocaleString: ƒ toLocaleString()
    6. toString: ƒ toString()
    7. valueOf: ƒ valueOf()
    8. __defineGetter__: ƒ __defineGetter__()
    9. __defineSetter__: ƒ __defineSetter__()
    10. __lookupGetter__: ƒ __lookupGetter__()
    11. __lookupSetter__: ƒ __lookupSetter__()
    12. get __proto__: ƒ __proto__()
    13. set __proto__: ƒ __proto__()

2. causedBy: null
3. isDefaultPrevented: false

That’s right! onColumnsRemoved, just as onColumnsAdded, is supposed to return an iterable for involved columns. We’ll figure out why it’s not propagated correctly, it seems that the columns are collected, but something is missing to convert the argument to JavaScript properly.

1 Like

Thanks for checking, looking forward to further test once this issue is fixed.

@andreas.gasser.novartis.com it’s fixed. Here’s an example for testing on dev:

let df = DG.DataFrame.fromColumns([
  DG.Column.fromList(DG.TYPE.FLOAT, 'x', [1, 2, 3]),
  DG.Column.fromList(DG.TYPE.FLOAT, 'y', [4, 5, 6]),
  DG.Column.fromList(DG.TYPE.FLOAT, 'z', [7, 8, 9])
]);
let col = await df.columns.addNewCalculated('new', '${x}+${y}-${z}');
grok.shell.addTableView(df);

df.onColumnsRemoved
  .subscribe((data) => {
    data.args.columns
      .filter((col) => col.tags.has(DG.TAGS.FORMULA))
      .forEach((col) => grok.shell.info(`Removed calculated column ${col.name}`));
});

df.columns.remove('x');
df.columns.remove('new');

That looks great, question: is this already available in a Datagrok container image (was trying 0.93.8 but no luck)?

:white_check_mark: 0.93.9 has been built.

Thanks for the new build. I did a quick test and the column remove event includes the deleted column now. However it seems now, that other meta change events on onMetadataChanged are no longer fired.

I get following behaviour on:

  • 0.93.8: 3 onMetadataChanged events, including the formula change
  • 0.93.9: 2 onMetadataChange without the formula change

This behaviour breaks our calculated column formula change detection. Could you please take a look into this?

Of course, I should have tested all the use cases, this will be fixed.

1 Like

Awesome thanks, looking forward for another Datagrok version to continue our testing phase.

@donufriienko any update on the issues mentioned above?

@andreas.gasser.novartis.com please check out 0.93.11, I have tested onColumnsAdded/onColumnsRemoved/onMetadataChanged there.

1 Like