[BETA] [NEW] beta.ninjastic.space (forum search, archive and data visualization)

[joker_josue]

Yes, but which time limit? Even if it's every 24 hours, or 72 hours, rescraping a entire post can take a lot of space the way it works right now (where the entire post is saved, even if the difference between versions is a single character).

I understand. The concern is the space that the number of edited posts will take up.

Perhaps have a system, that after the second scrape that is already done, that does another scrape (without saving) and checks if there were significant changes (+x characters for example). If the number of changes is large, it saves this edit.

[LoyceV]

The problem is that right now I save another copy of the entire post every time something changes, even if it's a single letter. If I allow multiple unlimited rescrapes, storage usage can go quickly very fast...

Is this really going to be a storage problem? I can imagine only a fraction of the posts gets edited many times.

I'm curious what happens when a post is rescraped, and didn't change: do you keep track of the fact that it wasn't changed on a certain date? I think that could be valuable information later on, to estimate when a post was edited.

[ABCbits]

I thought about storing only the post diff, but then I need to rethink how my search engine works right now. Maybe there is another solution...

Have you ever considered having a manual scraping button? For example, a user would visit your page related to the post and click the button, and the scraping would be executed.

To prevent abuse, you would put a general time limit until the button is used again.

Yes, but which time limit? Even if it's every 24 hours, or 72 hours, rescraping a entire post can take a lot of space the way it works right now (where the entire post is saved, even if the difference between versions is a single character).

I also see it as possible DoS attack. Someone could make a bot to request to many threads to be re-scraped once it passed time limit.

[TryNinja]

Is this really going to be a storage problem? I can imagine only a fraction of the posts gets edited many times.

Like ABCbits said, it can be used as a DoS by spamming my bot with a bunch of rescrapes for long posts.

I'm curious what happens when a post is rescraped, and didn't change: do you keep track of the fact that it wasn't changed on a certain date? I think that could be valuable information later on, to estimate when a post was edited.

I currently don't keep track of this, but I believe I should.

[joker_josue]

Is this really going to be a storage problem? I can imagine only a fraction of the posts gets edited many times.

Like ABCbits said, it can be used as a DoS by spamming my bot with a bunch of rescrapes for long posts.

Well, not necessarily, you can create validation captchas, and create other tools to control this.

Of course, it would be practically impossible to automatically get all the editions, but it could help to create a broader registry.

I ask, just as the bot detects when there are new posts, isn't it possible to have a bot that detects edits? I know these are different things, but the forum, after an edit, leaves a record in the post that it was edited. Isn't it possible to check this information?

[LoyceV]

I ask, just as the bot detects when there are new posts, isn't it possible to have a bot that detects edits? I know these are different things, but the forum, after an edit, leaves a record in the post that it was edited. Isn't it possible to check this information?

New posts are easy to find, edited posts can be anywhere in millions of topics and tens of millions of posts. At no more than 1 page request per second, it takes months to check everything (again).

[joker_josue]

I ask, just as the bot detects when there are new posts, isn't it possible to have a bot that detects edits? I know these are different things, but the forum, after an edit, leaves a record in the post that it was edited. Isn't it possible to check this information?

New posts are easy to find, edited posts can be anywhere in millions of topics and tens of millions of posts. At no more than 1 page request per second, it takes months to check everything (again).

Well, that would require a complete overhaul. You don't have access to the database... It's a shame because you can't capture all the changes.

The solution was to make individual requests, on a case-by-case basis. But this had to be done in a way that prevented abuse. I'll think about it a bit, and try to share some ideas.

[hosemary]

I want to sort posts made in a topic by their time in descending order, but I can see only the last 20 posts in that order.

Here is an example.
https://beta.ninjastic.space/topic/5534972?sort_by=date_desc

Once you go to the second page, posts are sorted in ascending order again.

[TryNinja]

I want to sort posts made in a topic by their time in descending order, but I can see only the last 20 posts in that order.

Here is an example.
https://beta.ninjastic.space/topic/5534972?sort_by=date_desc

Once you go to the second page, posts are sorted in ascending order again.

Thanks, fixed it.

[Royal Cap]

Hey TryNinja,  I noticed one thing that earlier when we clicked on the names of those who sent or received merit in the last 24 hours on beta.ninjustice.space, all their details would be shown.
That is, who received merit from whom or who sent merit to whom. But now if we click on their names, nothing shows. Is this a bug or have you turned off this feature.

[TryNinja]

Hey TryNinja,  I noticed one thing that earlier when we clicked on the names of those who sent or received merit in the last 24 hours on beta.ninjustice.space, all their details would be shown.
That is, who received merit from whom or who sent merit to whom. But now if we click on their names, nothing shows. Is this a bug or have you turned off this feature.

It was a TODO after I rewrote the bar chart component.

I just added it back for both columns, and I will improve it in the next days.


Some changes:

- All homepage data is now displayed for the selected range (previously, it only showed the 24h data for: merit senders/receivers, most active users, most active topics)

- Added merits data modal when you click on the merit senders/receivers charts on the homepage. You can share the data by copying the URL, which is updated accordingly, for example: https://beta.ninjastic.space/merits?sender_uid=459836&date_min=2025-09-17&date_max=2025-09-24

- You can now sort the merits data table (click the header to sort by: date, sender, receiver, amount, post id, title). Shift + click multiple headers to multi-sort.

[TryNinja]

Updates


Better caching and optimizations: The pages should load faster.


New User "topics" tab/page
- Shows the topics where the user participated the most. Click the entry to open a modal with the topic page already filtered for that user and range.




- All bar charts now show the percentage of the entry when you hover it (i.e user page -> "most active boards" shows the % of posts you made on that board).




- New charts "Merits sent on boards" and "Merits received on boards" on the user's merit page.




Shallow routing: Most clickable charts now show their data (when clicked) in a modal for quick viewing, instead of routing you to the page, for better UX.


- All merit related bar charts are clickable. More to come...
- Added the missing "date" filter to the topics page.
- Improved styling all over the place (a bit).

[joker_josue]

Updates

What excellent updates. More information, more details about the information. This helps to get an overview of user activity.

I'm starting to doubt whether this version should still be called beta. 

[TryNinja]

More Updates


- Added child boards to the “Board” page: You can now select to view the aggregated data for that board with all its child boards.

For example: To see data for the entire Portuguese local board (with all child boards): https://beta.ninjastic.space/board/29?child_boards=true




- Improved board selector’s UI/UX.




- Vastly improved the topic filter on the Search page: You can now search a topic by its title (with auto complete suggestions!). Or just paste an URL/ID.




- Date ranges now syncs with the URL, allowing you to quickly share or load a page with the selected range.
    

   For example:
    
    - https://beta.ninjastic.space/board/29?date_min=2025-10-09&date_max=2025-10-10
  
    - https://beta.ninjastic.space/user/id/557798?date_min=2024-10-10&date_max=2025-10-10
    
    - https://beta.ninjastic.space/board/29?child_boards=true&date_min=2025-10-03&date_max=2025-10-10

- Added a quick overview card on the “Board” page (with the number of posts, merits, merit txs, merited and unmerited posts in the period).




- Added “Most merited posts” to the homepage.
- Added “Most merited posts” to user’s page (merit tab).
- Added “Most merited posts” to the “Board” page.
- Added “Most active topics” to the “Board” page.

[joker_josue]

More Updates


- Added child boards to the “Board” page: You can now select to view the aggregated data for that board with all its child.

Another incredible update. This information is excellent, both locally and internationally, with excellent detail.
Congratulations on your work and effort.

[TryNinja]

Another incredible update. This information is excellent, both locally and internationally, with excellent detail.
Congratulations on your work and effort.

Thanks.

Let me know if there's anything you (or anyone else) wants to see. I could work endlessly on so many things, but I have no idea if people would actually use those features. I rather focus on what you guys would use.

Next thing in my mind is a page comparing data between boards (especially local boards) on a date range (similar to the "Brief monthly overview of the local boards activity").

[joker_josue]

Let me know if there's anything you (or anyone else) wants to see. I could work endlessly on so many things, but I have no idea if people would actually use those features. I rather focus on what you guys would use.

Well, I think you've already included all the ideas I gave (or almost all of them). I have to review everything and think of other things - I'll analyze.

But, I was looking at the user page, and you could create an option or a third graph, which allows you to merge the graph of posts and merits, making a ratio between the two pieces of information.

[hosemary]

The "Go to Forum Profile" button on users profiles takes us to their merit summary on forum.
I may be wrong, but I doubt that's the intended function of the button and I think it's supposed to take us to users profiles on forum.


https://beta.ninjastic.space/user/id/3

[DYING_S0UL]

Let me know if there's anything you (or anyone else) wants to see. I could work endlessly on so many things, but I have no idea if people would actually use those features. I rather focus on what you guys would use.

In the "Search User Posts" option, can you like add a feature to search for all the address posted by that particular user? I know we have the "Addresses" feature to search it but can make it so  that we can see the complete list of addresses connected/used/posted by a user? Or did it already existed (I couldn't find it) ??

As for the rest of the things, man that is some impressive work! A total a grade job done with the site!

[TryNinja]

Cool page I'm working on: https://beta.ninjastic.space/boards/local-boards

See the monthly data of all local boards.

- Number of posts
- Number of merits
- Number of topics
- Merit/post ratio
- Merit average per transaction
- Percentage of merited posts
- Active users (1 post, 2-9 posts, 10+ posts)
- Top users by posts, per board
- Top users by merits sent, per board
- Top users by merits received, per board

... What else? (Insert your suggestion here)

I might make a topic to auto-post a small summary every first day of each month, when there's enough cool charts.


SEPTEMBER 2025

The "Go to Forum Profile" button on users profiles takes us to their merit summary on forum.

Fixed, thanks!

In the "Search User Posts" option, can you like add a feature to search for all the address posted by that particular user? I know we have the "Addresses" feature to search it but can make it so  that we can see the complete list of addresses connected/used/posted by a user? Or did it already existed (I couldn't find it) ??

There will be a page to see addresses by an user.