Twitch Streaming Graph Analysis - Part 2

Introduction

This blog is divided into three parts, depending on the part of the application we are building:

• Part 1: data source and backend implementation
• Part 2: frontend implementation
• Part 3: streaming data from Kafka cluster

If you still haven't, you can read already published Part 1 and then continue reading this part. Otherwise, use already implemented backend. In this part, we are going to create React application and visualize general statistics and some interesting insights from Twitch dataset. All implementation that will be mentioned in this part of the blog you can find in frontend folder of the project.

Create React App

Let's figure out how were going to visualize all the data we have managed to collect. First we have to create React app which will work with our Flask application. We will install Node.js which will give us npx command for creating a React app. Place yourself in project root folder and run:

npm install -g [email protected] init react-app frontend --use-npmcd frontendnpm start

Now at http://localhost:3000 you can see simple React app. We will configure our React app to work well with Flask server. In package.json from frontend folder add a line at the end of the file:
"proxy": "http://localhost:5000"
This will tell React app to redirect any requests it receives on its port 3000 to port 5000, where our backend is implemented. We will use Semantic UI to build our webpage, so we have to do a few more things before dockerizing our React app. Run the following commands:

npm uninstall semantic-ui semantic-ui-cssnpm install @craco/craco @semantic-ui-react/craco-less semantic-ui-less --save-dev

After that, update your package.json with:

{  "scripts": {    "start": "craco start",    "build": "craco build",    "test": "craco test",    "eject": "craco eject"  }}

Create craco.config.js in fronted folder and paste following content in it:

module.exports = {  plugins: [{ plugin: require('@semantic-ui-react/craco-less') }],}

Create semantic-ui/site folder in src folder and then Copy the entire node_modules/semantic-ui-less/_site folder content to src/semantic-ui/site. Also create theme.config file in src/semantic-ui/ folder and then copy file node_modules/semantic-ui-less/theme.config.example to src/semantic-ui/theme.config.

Change theme.config file:

/*******************************            Folders*******************************/@themesFolder : 'themes';@siteFolder  : '../../src/semantic-ui/site';@import (multiple) "~semantic-ui-less/theme.less";@fontPath : '../../../themes/@{theme}/assets/fonts';

If it's not already installed, run:
npm install semantic-ui-react --save-dev

Now you can use Semantic UI components, such as Button (dont forget to paste all imports):

import logo from "./logo.svg";import "./App.css";import "semantic-ui-less/semantic.less";import { Button } from "semantic-ui-react";function App() {  return (    <div className="App">      <header className="App-header">        <img src={logo} className="App-logo" alt="logo" />        <p>          Edit <code>src/App.js</code> and save to reload.        </p>        <Button>Click me</Button>      </header>    </div>  );}export default App;

Dockerizing React App

We will be adding few new lines in our previously created project's docker-compose.yml file. At the end of the file add:

react-app:    build: ./frontend    volumes:      - ./frontend:/app      - /app/node_modules    ports:      - "3000:3000"    depends_on:      - twitch-app    networks:      - app-tier

You should also create Dockerfile in frontend folder like this:

# pull official base imageFROM node:14.17.5-alpine# set working directoryWORKDIR /app# add `/app/node_modules/.bin` to $PATHENV PATH /app/node_modules/.bin:$PATH# install app dependenciesCOPY package.json ./COPY package-lock.json ./RUN npm install --silentRUN npm install [email protected] -g --silent# add appCOPY . ./# start appCMD ["npm", "start"]

Here you can see we are copying package.json into the container. The best way to have all dependencies installed is to copy package.json file from here. All node modules will then be correctly installed in the container and npm start will run your React app. Node modules volume was added so that all packages don't have to be installed each time you build your project.

All that is left to do is to create .dockerignore file in frontend:

node_modulesbuild.dockerignoreDockerfile

and make a little change in package.json in proxy settings:

"proxy": "http://twitch-app:5000"

This change is made because we are running the backend as docker service on port 5000 called twitch-app. Our project structure now looks like this:

|   docker-compose.yml|   +---backend|       app.py|       Dockerfile|       requirements.txt|+---frontend|   |   .dockerignore|   |   craco.config.js|   |   Dockerfile|   |   package.json|   |   package-lock.json|   +---node_modules|   +---public|   +---src|+---memgraph|   |   |   +---import-data|   |       chatters.csv|   |       moderators.csv|   |       streamers.csv|   |       teams.csv|   |       vips.csv|   +---mg_log|   +---mg_lib|   +---mg_etc

Now you can hit docker-compose build from your root project folder and docker-compose up after that. First memgraph-mage will run and then twitch-app. After that react-app will be run. Now you can make requests from your frontend.

Frontend Implementation

Create folder components in your src file. Here you will make your components which youll use as puzzles for your web application. Let's make a little part of the puzzle, where well show the fetching of data from Memgraph. We will add node and edge counters to our webpage by making fetch requests in Counter.js. In the code below you can see that we are making a request depending on the props forwarded from the parent component.

  fetch() {    fetch("/" + this.props.count)      .then((res) => res.json())      .then(        (result) => {          this.setState({            isLoaded: true,            counter: result[this.props.count],          });        },        (error) => {          this.setState({            isLoaded: true,            error,          });        }      );  }

On the left you can see the number of nodes, and on the right the number of edges in your database.

For games statistics we are fetching top games from backend:

  fetchData(number) {    fetch("/top-games/" + number)      .then((res) => res.json())      .then(        (result) => {          this.setState({            isLoaded: true,            games: result.games,            players: result.players,          });        },        (error) => {          this.setState({            isLoaded: true,            error,          });        }      );    this.setState({      numOfGames: number,      header: "Top " + number + " games",    });  }

On the right you can see the table showing you the names of the games and number of players that are playing that game (in our dataset).

In the similar way we are fetching top teams, vips and moderators. For streamers, we are ranking them by number of followers or number of views. Because of that, you have a dropdown menu from where you can choose the way of ranking.

Lets talk about graph visualization a bit more. Here, were going to use D3.js. This is a package which we still cannot use. We need to set everything up so that we can draw our graphs using D3.js. Create a folder hooks in src folder and create useD3.js file (hooks are usually named with prefix "use").

import React from "react";import * as d3 from "d3";export const useD3 = (renderGraph) => {  const ref = React.useRef();  React.useEffect(() => {    renderGraph(d3.select(ref.current));    return () => {};  });  return ref;};

This will be your custom hook to allow D3.js to interact directly with the DOM. You can take advantage of the useRef and useEffect hook to link D3.js with the svg element that has been created, and specify when your D3.js function should be executed. Dont forget to import d3. Now we can render our graph using custom hook useD3.js. Check the Graph.js component to see how the graph can be drawn. Using that component we can get information about your favorite streamer - its teams, games and languages, like on the image below.

It is also possible to search all streamers who are playing some game in certain language. Using the same Graph.js component, you get:

Feel free to play with nodes and their forces by dragging them around. In the end, we will use powerful MAGE query modules - PageRank and Betweenness Centrality, to showcase how you can visualize your data in a pretty cool way with D3.js. For PageRank we have API get request in backend and in frontend we're fetching that data from the PageRank.js component.

  fetchData(){    fetch("/page-rank")      .then((res) => res.json())      .then(        (result) => {          this.setState({            isLoaded: true,            nodes: result.page_rank,          });        },        (error) => {          this.setState({            isLoaded: true,            error,          });        }      );  }

While PageRank results are being calculated, you will see your results loading. After the results are loaded, they are drawn with graph component GraphPR.js which uses D3.js. Next to the graph you can see the table of results with the names of the streamers and their calculated rank.

You can see the refresh button above the drawn graph. It will be used later on, in the Part 3 of the blog, when we'll show how you can stream your data using Kafka. We will stream new chatters of user BadBoyHalo and see how his rank improves with larger number of chatters in his network.
Similarly, we calculated the Betweenness Centrality and showed the results below.

Conclusion

And that's it for now! You can see all necessary code for this part of the blog post here. Hope you got everything right, but if you have any questions or want to give some feedback, feel free to join our Discord Community Server. Make sure to follow up on the last part of this blog, where you can learn how to tackle with your streaming data.