Vector based video player for animated content

[Saurabh Nandwana]

Greetings to all the community members!

I'm new here, I am a developer trying to get deeper into web-animations. The goal is to recreate cartoon animation on web, specifically a vector graphics based video player, which should ideally work the same way (in terms of recreation of scenes, shapes and effects), be performant and resource efficient. I know that's too much to ask and it's ambitious. That's why we are looking to get some help from the community

Our public demo: https://files.vectorly.io/demo/scene1/index.html 
Using SVGs for our version 1 | Issues: Neither performant nor efficient, loss of fps and high CPU usage

The task is to -

• Recreate smooth animations
• Achieve a good fps ~60
• Resource efficient : CPU usage in the range of 1-5% 
• Transfer computationally expensive load to GPU (Given most modern browsers/devices support this) 

 

In terms of input data, we have bunch of shapes which are simply SVG path data consisting of "d" strings  (control points), fill, stroke values and transforms (translation, scale, rotation, skew).

 

 We have explored several option thus far and 

1. SVG - Inability to perform with large number of objects at a frame level, given one frame can have a range of 100-5000 objects. We are working on bringing this number down but it has its own limitations so it can't go down beyond a certain level.   
2. Canvas. - Inability to perform well when drawing dynamic content over static content. Some sort of layering where shapes which doesn't change/move over a given timeframe are drawn on a canvas-layer and those dynamic shapes are drawn-erased-redrawn on higher (z-index) layers.
3. GSAP - Have explored morphing shapes. To be explored more.
4. WebGL - Mostly useful for 3d animations. We are still on 2d animations so probably not now. Since canvas 2d-context has fallback on this.
5. PixiJS - Doesn't have good support for SVG or Graphics in general. The latest version v5 uses WebGL2 which itself is not yet supported widely.

It would be really helpful if experts @OSUblake, @ZachSaucier, @GreenSock, @PointC from this amazing community let us learn from the years of learning that they have in this space.

 

Thanks in advance!
Saurabh,
Vectorly

[ZachSaucier]

Hey Saurabh and welcome to the GreenSock forums! 

 

The GSAP forums are for GSAP-related questions, issues, and topics, not general advice or programming help like it seems that you're asking for (though at times we stray from the straight and narrow path).

 

In terms of your request, I will just say that videos are made they way that they are for good reasons  There's a reason why no vector based video format exists. 

 

If you have GSAP questions or issues, please let us know and we're happy to help. Until then, best of luck with your project(s)! 

[Saurabh Nandwana]

Thank you so much Zach :) I appreciate you paying your attention on this.

 

The idea is to share one of the possible use cases for GSAP (and overall web-animations) with the community. Well, technically the same post can be broken down into several smaller technical know-how queries and we are actively exploring where GSAP could fit in the solution. 

I rather chose to put the "what and why" out there for the initial context. 

 

As far as the idea of a vector-based video is concerned, I completely understand your point, this is more of an RnD project. We believe given how matured h/w and modern browsers have become from the days of Flash, along with active communities/tools like GSAP and Pixi/Other HTML5 rendering engines, there is a decent potential for building a vector-based-video-codec today.

 

For a further constructive discussion, I would love to hear @OSUblake's opinion on this and what according to him should be the right set of tools to build a performant rendering engine.

[ZachSaucier]

7 minutes ago, Saurabh Nandwana said:

I would love to hear @OSUblake's opinion on this and what according to him should be the right set of tools to build a performant rendering engine.

Sounds like you want to consult with him. I believe he offers paid consulting. You can DM/email him to see if he's interested. 

[OSUblake]

Hi @Saurabh Nandwana

 

That's quite an ambitious task!

 

Performant vector rendering is a problem that people have been working on for years. 

 

Have you checked out bodymovin/lottie? It's a vector player, but it's not a perfect solution, nor is it always performant. 

https://github.com/airbnb/lottie-web

https://codepen.io/collection/nVYWZR

 

There's even a nice web component for it so you can use it just like a <video> element.

 

See the Pen JLrbYB by fernandocomet (@fernandocomet) on CodePen

 

 

8 hours ago, Saurabh Nandwana said:

Canvas. - Inability to perform well when drawing dynamic content over static content. Some sort of layering where shapes which doesn't change/move over a given timeframe are drawn on a canvas-layer and those dynamic shapes are drawn-erased-redrawn on higher (z-index) layers.

 

Having too many canvas layers can slow stuff down. I remember the developer of bodymovin/lottie saying that it uses a dirty rectangle algorithm to limit redrawing to parts of the screen that have changed instead of the entire scene.

 

In general, I would say Canvas2D is probably the fastest way to render vector graphics on the web, but it does have limits. Uncheck the Use Bitmap checkbox, and you should see the performance drop as it switches to vector rendering.

 

See the Pen a7868b452dfb5eebad6ee9ae24c2bfc3 by osublake (@osublake) on CodePen

 

8 hours ago, Saurabh Nandwana said:

GSAP - Have explored morphing shapes. To be explored more.

 

Morphing works well for some stuff, but it's probably not the best tool for a Simpsons cartoon as its hard to control the movement like you can with joints and bones.

 

8 hours ago, Saurabh Nandwana said:

WebGL - Mostly useful for 3d animations. We are still on 2d animations so probably not now. Since canvas 2d-context has fallback on this.

 

WebGL is still a nice option for 2d, but it won't automatically fallback to canvas as WebGL uses a completely different API. However, WebGL support is very good.

 

8 hours ago, Saurabh Nandwana said:

PixiJS - Doesn't have good support for SVG or Graphics in general. The latest version v5 uses WebGL2 which itself is not yet supported widely.

 

I'm pretty sure it will fallback to WebGL 1, but PixiJS was never really designed to really handle complex vector graphics. It's optimized for raster graphics. For SVG, it converts them into bitmaps, so they are static. To change a path, you would need to modify the source SVG and then get Pixi to reload it, which is a slow process. For stuff like that, it's usually faster use a Canvas2D as a texture as it's much faster to update.

 

[OSUblake]

1 hour ago, OSUblake said:

In general, I would say Canvas2D is probably the fastest way to render vector graphics on the web, but it does have limits.

 

In some cases, vector rendering in canvas can be improved by using Path2D. It caches paths, reducing the number of draw calls you need to make. Still not as fast as using drawImage, but it can definitely speed up vectoring rendering for shapes that don't change a lot. It can be polyfilled for older browser, and has some limitations in legacy Edge.

https://developer.mozilla.org/en-US/docs/Web/API/Path2D

 

 

[Sam Bhattacharyya]

Thanks so much @OSUblake and @ZachSaucier.  We'll give lottie a try for our rendering engine, as it seems pretty well supported. As @Saurabh Nandwana mentioned, one of the big issues we've run across is the large number of moving objects in popular cartoons like the Simpsons that we've tried to vectorize. Either way, I understand this thread is a bit off-topic for the forum, so maybe we can leave it here for now, and follow-up via DM. Thanks again!  

 

-Sam

(Also Vectorly)