Search all known grids

[Tampa]

There was a discussion on IRC today about extending search to other grids as well to essentially make it metaverse-wide instead of just local. The discussion was mainly about whether this should be handled via robust, queries sent from it to all other known grids, or from the viewer, searching via its known grids from the grid manager.

I think the best approach would be to search via the viewer and simply have it and the grid exchange search urls they know to extend both known list of urls. This would prevent the local grid from becoming the chokepoint of large search requests and still provide exchange of urls to extend the search.

For starters though, and I guess to prove it's not that complicated, my suggestion is perhaps making search use all search urls from the grid manager rather than just that of the local grid. As this is just editing to extend the query in the viewer it should not be that complex to do. Whether an actual cap to exchange urls is added to OpenSim is still in the stars, but I would like to see what it would look like querying a ton of grids as I have a list of about 300 of them via my OpenGridList project.

 

[Geir Nøklebye]

Initially I think that grid search can be implemented directly as db queries in the grid database, with a few table additions to store some information that is not already there, and to generate the required indexes to speed up searches.
Depending on the database system used, one can also use full text search directly in the database, which is usually fast, however at the cost of additional storage for the indexing.

I have already made a few views that will provide the most common searches, so the viewer can use the existing (legacy) search screens for the user to provide their search criteria and get the result back in the existing forms.

Time wise, this is not the time to add that project as there are some major time consumers ahead for viewer development. Keywords are removal of Mercurial support and repositories from Bitbucket forcing every developer to either seek other solutions for their repositories, in addition to conversion to git, with the new workflows this requires.
The above will also force developers to VS2017 /Xcode 11.3 as LL will only provide future libraries with code that compiles with VS2017/Xcode 11.3. In addition there is the EOL for Windows 7 that will trigger updates from Microsoft they have been holding.

For the macOS version Xcode 11.3 development can only happen on macOS 10.14 or 10.15 and most developers, perhaps with the exception of me has old kit that cannot run these versions of macOS. Further Apple will from the beginning of February mandate signing apps with an Apple developer ID, or the application (viewer in this case) will to be able to launch at all on 10.15. The complication is of course that all new Apple HW after October 2019 cannot run anything but 10.15. So there will be a lot there to chew on.
Apple completely removing openGL support from the upcoming 10.16 this fall will not exactly make the situation better.

LL is about to launch to major updates to the viewer; reverting to in-viewer profiles and not web profiles which probably is not a massive effort to support, but more important EEP which is both complicated and disruptive for OpenSim as it is incompatible with Lightshare. Ubit is looking to remove Lightshare completely from Opensim.

So as much as I wish there was time to work on search, there isn't right now.

 

[Tampa]

Never much pressure for time from my side, I don't even know if search is working on ZW, I never really used it. I was mainly just hoping that you would be on my side regarding the approach that viewer-side implementation with a small cap to exchange potentially new urls between viewer and grid is the better approach compared to letting the robust do all the work indexing known grids as that would open it up to ddos via search and make it the chokepoint for all those queries. Proving that via a quick and dirty copy pasta to the search code to foreach the grid list rather than just the local grid, but by all means the things you mentioned are more important for now. Put this on ice for later and if you need help with the aforementioned major updates let me know, I'll see what I can do

 

[Geir Nøklebye]

Robust is not going to do much additional work except passing some data back and forth for the query. The db server will do the heavy lifting, but it is mainly CPU which database servers often have plenty of compared to real IO.

You also get completely rid of the length daily extraction+export and import runs that is used for the current search solutions.

But yes, the current search solution is clunky and needs a more modern approach. As database engines have matured dramatically with native storage types for json, xml and full text search it should be taken advantage of - particularly as most of the data needed is already stored in the existing tables.

 

[Tampa]

I wrote a proposal for handling this via viewer and grid caps, because frankly having the local grids search burdened with searching all other known grids as well to me sounds like a recipe for disaster.

Proposal for a GridInfo-Service and SearchURL Exchange CapExchange of GridIn - Pastebin.com

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

pastebin.com

I frankly find this to be the most elegant and efficient way to do it, it also gives control both in the hands of the viewer and grid alike so no party can force the system onto one another. I will talk with Ubit what he thinks and adjust accordingly, but, again, having OpenSim do it all in order to not need to change the viewer is silly and backwards and with you providing such stellar work on Dayturn I don't see why anything needs to be held back and using inferior technological approaches just because the other viewers a prime-grade a-holes about stuff like this.