An Interest In:
Web News this Week
- April 20, 2024
- April 19, 2024
- April 18, 2024
- April 17, 2024
- April 16, 2024
- April 15, 2024
- April 14, 2024
Some of Our Sources
- Engadget
- Mashable
- Techcrunch
- Just Creative
- Pearsonified
- You The Designer
- Reencoded
- 24 Ways
- Daily Now
- TechPowerUp
Help Webnuz
Referal links:
Glitch free 1,000,000 record data processing in TypeScript with js-coroutines
Sometimes we need to process data on the front end, perhaps we are using an offline system or accessing local data. When that data gets large it can easily cause the UI to glitch. A few days ago I wrote an article demonstrating how search could be made to run at the same time as UI updates using js-coroutines. I thought I'd dive into a more powerful version in TypeScript that does more than search; it also renders the records as it goes and has a variety of progress indicators. Once done it performs a bunch of tabulations to update some charts.
Notice how you can keep typing and even start browsing the records as the searches continue. This is done using collaborative multitasking on the main thread.
Please note the tooltip supplied by Recharts doesn't work properly when this window is zoomed. See full screen version
This demo uses a new feature of js-coroutines that allows you to define a "singleton" function. Singleton functions automatically cancel the previous run if it is still underway and start again. That's exactly what you need for a search like this.
const process = singleton(function*(resolve: Function, search: string, sortColumn: string) { let yieldCounter = 0 if (!search.trim() && !sortColumn?.trim()) { resolve({ data, searching: false }) addCharts(data) return } resolve({ searching: true, data: [] }) let parts = search.toLowerCase().split(" ") let i = 0 let progress = 0 let output : Data[] = [] for (let record of data) { if ( parts.every(p => record.description .split(" ") .some(v => v.toLowerCase().startsWith(p)) ) ) { output.push(record) if (output.length === 250) { resolve({data: output}) yield sortAsync(output, (v : Data)=>v[sortColumn]) } } let nextProgress = ((i++ / data.length) * 100) | 0 if (nextProgress !== progress) resolve({ progress: nextProgress }) progress = nextProgress yield* check() } resolve({sorting: true}) yield sortAsync(output, (v : Data)=>v[sortColumn]) resolve({sorting: false}) resolve({ searching: false, data: output }) addCharts(output) function* check(fn?: Function) { yieldCounter++ if ((yieldCounter & 127) === 0) { if (fn) fn() yield } }}, {})This routine starts off by checking if we are searching for something and takes a quicker path if we aren't.
Presuming it is searching it uses a neat trick of resolving values many times to update the progress. This allows it to reveal results as soon as it has 250 records, update progress every 1% and then switch on and off searching and sorting indicators.
Calling resolve just merges some data into a standard React.useState() which redraws the UI to keep everything smoothly updating while the search progresses.
interface Components { data?: Array<Data> searching?: boolean progress?: number, sorting?: boolean, charts?: []}function UI(): JSX.Element { const [search, setSearch] = React.useState("") const [sortColumn, setSortColumn] = React.useState('') const [components, setComponents] = React.useState<Components>({}) React.useEffect(() => { setComponents({ searching: true }) // Call the singleton to process process(merge, search, sortColumn) }, [search, sortColumn]) return ( <Grid container spacing={2}> <Grid item xs={12}> <TextField fullWidth helperText="Search for names, colors, animals or countries. Separate words with spaces." InputProps={{ endAdornment: components.searching ? ( <CircularProgress color="primary" size={"1em"} /> ) : null }} variant="outlined" value={search} onChange={handleSetSearch} label="Search" /> </Grid> <Grid item xs={12} style={{visibility: components.searching ? 'visible' : 'hidden'}}> <LinearProgress variant={components.sorting ? "indeterminate": "determinate"} value={components.progress || 0} color="secondary" /> </Grid> <Grid item xs={12}> <RecordView sortColumn={sortColumn} onSetSortColumn={setSortColumn} records={components.data} /> </Grid> {components.charts} </Grid> ) function merge(update: Components): void { setComponents((prev: Components) => ({ ...prev, ...update })) } function handleSetSearch(event: React.ChangeEvent<HTMLInputElement>) { setSearch(event.currentTarget.value) }}The merge function does the work of updating things as the routine progresses, and as we've defined a "singleton" function, it is automatically stopped and restarted whenever the search or sort properties change.
The charts each individually start a calculation, and we "join" their execution to the main process so that restarting the main process will also restart the chart.
function Chart({data, column, children, cols} : {cols?: 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12, data: Array<Data>, column: (row: any)=>string, children?: any}) { const [chartData, setData] = React.useState() React.useEffect(()=>{ const promise = run(count(data, column)) // Link the lifetime of the count function to the // main process singleton process.join(promise).then((result: any)=>setData(result)) }, [data, column]) return <Grid item xs={cols || 6}> {!chartData ? <CircularProgress/> : <ResponsiveContainer width='100%' height={200}> <BarChart data={chartData}> <CartesianGrid strokeDasharray="3 3" /> <XAxis dataKey="name" /> <YAxis /> <Tooltip /> <Bar dataKey="value" fill="#8884d8"> {children ? children(chartData) : null} </Bar> </BarChart> </ResponsiveContainer>} </Grid>}Here we've use a mix of helper Async functions and generators so we have maximum control. Our final remaining generator of interest is the one that calculates the chart results:
function * count(data: Data[], column: (row: Data)=>string, forceLabelSort?: boolean) : Generator<any, Array<ChartData>, any> { const results = yield reduceAsync(data, (accumulator: any, d: Data)=>{ const value = column(d) accumulator[value] = (accumulator[value] || 0) + 1 return accumulator }, {}) let output : Array<ChartData> = [] yield forEachAsync(results, (value: number, key: string)=>{ key && output.push({name: key, value}) }) if(output.length > 20 && !forceLabelSort) { yield sortAsync(output, (v:ChartData)=>-v.value) } else { yield sortAsync(output, (v:ChartData)=>v.name) } return output}This one simply counts the labels extracted by a function and then sorts the results appropriately.
Original Link: https://dev.to/miketalbot/glitch-free-1-000-000-record-data-processing-in-typescript-with-js-coroutines-fmp
Dev To
An online community for sharing and discovering great ideas, having debates, and making friendsMore About this Source Visit Dev To