Add interaction tests

[ericedem]

I've noticed in slate that a large majority of bugs that creep into codebase are specifically around events, interaction, cross browser, and slate-react. These bugs are often caught by people running through examples on different browsers, and happening to stumble across them.

What I wanted to do is start a discussion to talk about how we could better test for these kinds of issues, hoping someone has experience in this area. I'd like to get higher level testing, like DOM interaction and event triggers, without resorting to something like selenium which is generally error prone.

I haven't dug super deep, but a promising library that might help with this is https://github.com/kentcdodds/react-testing-library.

We could probably automate running this on something like sauce labs or browserstack, or something that has a free open source plan.

[ianstormtaylor]

Hey @ericedem thanks for opening this! I completely agree with you.

As for react-testing-library, I haven't looked into it myself. I can't figure out how they're arriving at a "real DOM". If they are using JSDOM for it, I'm not sure it will work for our case because JSDOM doesn't attempt to handle anything to do with contenteditable, or at least last I checked.

Another potential option might be using Chrome Headless? I think @zhujinxuan might have investigated this or was hoping to. It would be awesome if we can make it as low maintenance as possible, and headless Chrome might help us with this.

Either way, I totally agree. And it's great to have an issue for it!

[dmitrizzle]

FWIW it seems Slate could use end-to-end testing framework. There are some listed here. The downside, I suppose is that they are mostly browser-specific and we need to test for browser faults, meaning having one just won't cut it.

I'd say automated testing via BrowserStack would make the most sense. It probably isn't cheap, unless they'd be willing to be a "sponsor," which would make perfect sense, seeing that Slate is an excellent candidate with fantastic adoption and community.

[ericedem]

Slate could use end-to-end testing framework

I'd really prefer to avoid selenium testing if possible. They are very high overhead, and often require a full time team to maintain since there are many points of failure, and they are time consuming to debug. I think we can get most bang for our bug with CLI testing.

For cross browser testing, what I had in mind was using something like sauce labs which has an open source program. Browserstack has something similar. And using something like zuul for automating the cross browser CLI testing. Though I haven't used those in a few years, there might be something cooler and more hip these days 😄

[ianstormtaylor]

FWIW, 90% of the testing needed isn't cross-browser and just needs to test that selections work and that rendering gets updating, etc. Lots of this could be Chrome-only and still be a huge improvement over the current situation.

[ojassvi]

Yes you can use react-testing-library with headless Chrome. I am using Karma as the test runner and headless Chrome to render my components in real DOM.

I am using slate in my project and I am able to render it correctly using react-testing-libaray but not able to test the inputs so far. I can add here my findings and how I am testing it, if that is helpful.

[zhujinxuan]

@ojassvi Thank you. I did not know that react-testing-library can work with headless Chrome.

@ianstormtaylor Sorry, I did not have time to look into headless Chrome before next Jan. I think I will look into it by next Jan.

[dmitrizzle]

I've had some good success with Cypress, though I'm not 100% confident it can provide the level of testing needed for all the cases. It's easy to write tests for (i.e. go to this url, click this button, enter this text, find this text in this DOM node).

[brendancarney]

+1 for Cypress. Also, they have a nice dashboard service that's free for open source.

The Dashboard Service is free to use for non-commercial open source, public projects and has no limitations on the number of public recorded tests per month.

[ianstormtaylor]

Anyone who can help add this would be greatly appreciated!

[Jmaharman]

Some of the details in this thread may help whoever does come to work on getting this testable in Cypress

https://slate-js.slack.com/archives/C1RH7AXSS/p1579110773288500