Bumping d3

[etpinard]

A discussion on what modules we need in d3 v4.0.0.

[monfera]

@etpinard these are probably well known to all of us, so it's a 'just in case':

• It's currently in alpha, so it's good to think about it, but I'm not aware of a target release date which makes it hard to incorporate in a roadmap
• feels like an overall refactoring and API improvements for an incrementally better D3, rather than a revolution or leap in some direction; some favorites are better support for non-SVG (Canvas, WebGL) targets, better timer, transitions and of course the Matplotlib Viridis palettes
• .. but a real deep article from Bostock seems to be https://medium.com/@mbostock/what-makes-software-good-943557f8a488#.9lyi5ovkg
• while inclusion of D3 4.0 can be done via ES6 modules, it does not mean that payload size would be reduced: because of the method chaining style of D3, if you need the smallest thing from e.g. d3-scale, it'll pull in the entirety of d3-scale
• these are the main options of including D3 4.0 in the project- personally I'd go for
• the improvements in D3 4.0 often mean that the code needs changing, so migration of a large project takes some time

[n-riesco]

@monfera Thx for the links!

[monfera]

Ann. by Mike Bostock: D3 4.0 is now in master and a release is coming up in a month or two.

[rnons]

D3 4.0 released

[rreusser]

Observations from an aborted attempt:

• I'm sure rollup.js is a wonderful piece of software, but it's appears rather difficult to work into the existing build process because d3 imports only affect d3 through implied side-effects. 😞 Example: I tried to integrate d3-selection-multi. See: How to integrate with existing d3 build? d3/d3-selection-multi#5 for why this is difficult. The only reasonable answer seemed to be to just copy/paste code from d3 modules into the plotly codebase. A more carefully considered solution pulls off some fancy trick I'm not able to figure out related to ES6 modules and their interaction with browserified code. The suggested approach is to modify d3's main index.js to include what you need. But ya can't fork it because then what ends up in node_modules is not compiled because of the prepublish hooks. So seems like you'd need to include d3's index.js in your project, but then you have to figure out lots of module/browserify stuff.
• style({...}) -> styles({...}). Ditto attrs. This change is mostly made in the d3-upgrade branch.
• selections are no longer subclasses of arrays. This means the .selectAll(...)[0][0] indexing trick is now invalid. which is currently the primary show-stopper to pursuing this further
• .data([0]) works fine, but the style through which it's currently achieved breaks since the resulting selection doesn't actually contain the created elements. It's a simple fix. I think I got most of them.
• d3.round is gone and seems not to have a replacement. I fudged it with a quick fix, but this might require more careful consideration.

Beyond that, it's not like the logic is all that different. It's a lot of bouncing from one error to the next and translating the code. The result that I got was able to display a grid with dragging and some hoverlabels, so it's viable, just would take more work than seems appropriate in the short term.

[monfera]

@rreusser nice summary! Yes, lots of relatively superficial upgrade work even with smaller codebases. Various incompatible changes, e.g. the collapsing of the namespaces, various renamings or both (e.g. d3.layout.force -> d3.forceSimulation), or what things return (e.g. some palettes were scales but now they're arrays) or what methods need to be used.

Besides the d3.round thing, there are subtle changes, such as d3.range no longer attempting to control for numerical inaccuracy (I saw it from Irene Ros, great presentation).

I wonder if the selection[0][0] thing can be replaced with .node(). In any case the array access was never part of the API so it's a good idea to avoid it.

Regarding the promise of smaller components, maybe this new tool can help customize what's included but I haven't yet tried. The reason it might work is that there are numerous modules not used in the codebase. In any case it's an almost negligibly small fraction of the plotly.js codebase.

Based on the current way we use D3 I wouldn't expect significant user-visible improvements from a switch. It can be useful for other reasons, I suppose the community (programmers, bug fixes) move to 4.0 over time. Or if there is a big modularization attempt in plotly.js itself, e.g. now lightweight plots as in journals aren't feasible due to the load and initialization time. Some other things also came up earlier, e.g. our use of the palettes (there's duplication between plotly.js and the gl-vis packages. The force layout in 4.0 is deterministic and way faster than currently, see my three recent blocks or these two gifs from Mike Bostock.

All in all D3 4.0 doesn't seem to address a current technical problem but makes some things easier in the future. It's a tough call because most folks can leave old projects in 3.* and use 4.* on new ones but plotly.js is not a dataviz project.

[rreusser]

Haha, but array access is soo easy! Come to think of it, I'm not sure it's actually used in the code. That might just be what I use to debug in the inspector.

[rreusser]

Yeah, the other change that maybe had consequences was transitions. The current animation PR falls back to instantaneous data updates when you transition the scale and the data at the same time. The reason seems to be a race condition in which the running transitions (points moving around) don't play well with transform changes (the whole layer moving around). It's difficult to debug because of the way v3 makes it relatively difficult to group transitions (i.e. transition.select(); and specifically, difficult at least within the plotly.js codebase). v4 lets you group more precisely by simply passing a transition to a new transition.

It's ripe for debugging and refactoring, but the current priority is consistency and correctness, then handling more complicated interactions and corner cases. I think v3 will get us there. 👍

[monfera]

Good point. Transitions can be a huge thing to upgrade properly (the new is so much better, taking advantage of it may need full rethinking of the transitions). Since you wrote the animations you are the one who knows about the expected transition upgrade friction.

[etpinard]

Referencing @rreusser 's #946 (comment) about our the very common .data([0]) pattern.

[rreusser]

Makes it an extra good reason to factor it out so it's only one fix.