Datagrok’s data engine enables interactive analysis of big datasets right in the browser. To achieve that, we do all sorts of optimizations that you would expect to see in a well-tuned, performance-critical C++ application and not in a web app, such as manual memory management with the column-based layout, adaptive bit storage, custom serialization codecs, and many others. This approach works great and allows us to work with datasets consisting of tens of millions of rows, or of millions of columns, however you are restricted to having columns of the pre-defined scalar types, such as int, double, string, bool, or DateTime.
Due to the complexity of business needs, sometimes having a real JavaScript object in a table cell simplifies things a lot. Previously, it was possible to do that by creating a column of type "Object’ and populating it manually - while this approach works, it uses a lot more memory and has other drawbacks.
To get the best of both worlds, we have decided to introduce a special “virtual” column type. When creating a virtual column, you would pass a function that takes an index and returns an object, and the platform would call this function whenever a table cell is accessed (for instance, when rendering a grid). This way, the data still resides in the most efficient way in the compressed columnar format, but at the same time application developers have an option to access it using custom, domain-specific objects that get constructed on the fly.
The screenshot below demonstrates the concept; the functionality is not yet released, and your feedback is appreciated.
