Datawisp DevUpdates | April 2023 - Better blocks, 20x performance boost, new tables

Here are some of the top DevUpdates for the month of April 2023 - Follow us on Twitter and LinkedIn for more DevUpdates!

April was another month where our product changed a lot, so let’s dig into it.

Better Aggregate Summarize Block

Our old Aggregate block was, admittedly, a bit inscrutable. We found in user tests it just wasn't intuitive for new users who might not know what each drop-down was supposed to do.

So, we’ve made this block way more explicit, leaving little doubt which pin does what. 

We renamed the block from “Aggregate” to “Summarize”, since that seems to be a bit more standard. Naming this block is not always easy, because, similar to GROUP BY in SQL, it needs to do two things.

In the end, it might get another rework to support more aggregation functions and our goal is to make sure that the UI doesn't overwhelm you. 

OLD:

NEW:

Better Filter Block

Simplifying the user experience is a key goal of the “embed blocks” rework, best demonstrated with the Filter block.

After this rework, certain blocks, like Filter and Summarize, can embed other blocks, like the “compare” block, making them much easier to read.

OLD:

NEW:

20x Performance Boost!

This is most certainly one of our biggest wins, but it unfortunately doesn’t have any flashy visuals. We’ve changed our query engine to be 20 times faster, both for schema evaluation and query execution. Some sheets of our users took over 30 seconds to load, which was, for the amount of data being processed, less than optimal.

Now? They load in less than 1.5 seconds, which makes working with them feel smooth again.

Of course, there’s still a lot of headroom for our engine to improve, but we definitely removed the biggest bottleneck in our engine. (The problem, if you’re nerdy enough to be curious about stuff like this, was to avoid copying entire rows of your dataset too much - previously, every time you accessed a value, e.g. to add it to another value, the entire row was copied. This is obviously not amazing, and now we’ve seriously reduced the number of copies we make, and instead access values from the buffer where possible. It adds a bit of indirection at some points, but it’s overall much faster!)

New Tables

Previously, we used a good, old, HTML <table> element to render our tables. Now, we’ve switched to glide-data-grid, an amazing library for displaying data.

The new tables perform better when scrolling through a large amount of columns / rows, are more accessible, are often easier to read, support better copying and pasting (you can now easily paste to a spreadsheet e.g. Excel), and still have all accessibility features.

Smaller Stuff

But this month was characterized by the many smaller things we added:

• Support for Microsoft SQL Server
  Okay, this one actually isn’t small, because SQL Server has some … interesting properties. Did you know there’s databases out there that don’t have boolean columns? It was surprising for some of our team members. Surprising, and perhaps even a bit frustrating. But in the end, we figured out how to make it behave, and now support it fully!

• We’ve reworked our Database import dialogue, so it’s now much easier to use.

• We’ve added support for TSV files

• We’ve improved how AI generates some sheets. It now keeps track of columns much more consistently

• Long column names don’t break our drop-downs anymore!

The Future

Of course, there’s more things we’re cooking up right now

• Support for Playfab: This one's a bit more tricky than we initially thought. Our initial plan was to release support for Playfab this month, but it seems like that’ll have to wait a bit longer.

• Better Charts: Charts have long been a pain point for us, now we’re finally addressing them. They’re going to be prettier, more customizable, and faster. They’re gonna get better in every single way!

• Calculate+: At the moment, if you want to quickly write down a formula with 3 or 4 operations, you need quite a few blocks for that, and it can become confusing quicker than we’d like.
  With Calculate+, we’ll allow you to write down an entire formula in a single block, which should simplify some of your sheets by a lot.

As always, I hope this was interesting, or at least helpful. If you have any feedback, please reach out!