I build an open-source google-like search for workplace knowledge
Content link | Reddit Link
r/selfhostedu/yuvalsteuer (339)

Comments (52)

This is unbelievably cool! Thanks for making it open source!

wow warms my heart :)

Try it on Here!

tl;dr: I built gerev - an open-source search engine for workplace pages, conversations, & docs. It's a privacy centric glean.com alternative.

I was scrolling through confluence pages trying to find ssh connection details to our jenkins second integration machine for 40 minutes straight, later I discovered my co-worker slack'ed me the ssh connection string two months ago.

so gerev.ai is a privacy centric search engine for workplace apps, it allows you to find everything from code snippets, conversations, or relevant docs.

It supports natural language queries so a query like: "how to setup test env for auth service?" yields (from a confluence page):

curl ...eu.amazonaws.com/setup_auth.sh | sh
export PYTEST_PLUGINS=auth.test_plugin.AuthPlugin pytest -v --...

WTF about privacy?!?!

gerev is completely open-source & self hosted, so no one but you should have access to internal docs.

[deleted]

It seems this is different, and haystack has their source on github: https://github.com/deepset-ai/haystack

[deleted]

Yea, I remember what you're talking about. It's also where my head went when reading the description here.

Maybe it's the Mandela effect. Travel through any universes recently?

There was a recent tool called Haystack, but the repo is gone, as is the website

Edit: The above links are related to the corporate search tool mentioned above. I signed up for updates, but now it's kaputt.

Looks cool. The website is very janky in Firefox mobile. Also it's not entirely clear what it supports.

Not worth mentioning in my eyes, as it still evolving and growing every single day.

Stay tuned on https://github.com/gerevai/gerev

Edit: Didn't mean that in a bad way, we're just adding integrations by the hour. someone just contributed bookstack..

Well, this probably wasn’t the proper response but I know why you said it. Mentioning what it supports here will only be outdated in a week, and people who discover this thread going forward will only see a lacking feature set.

That was what I meant lol :)

Fyi: Domain blocked by guest wifi in public place.

Noticed, I think it's because of .ai tld

lol sorry for the typo, built** not build**

Does this require Nvidia hardware?

Not specifically, if you’d want to serve a bunch of users, yes.

Orherwise, CPU is good enough

What does "a bunch of users" mean exactly in numbers? Can I serve 60 users with it? Can I restrict access to different resources for different user groups?

I’ll tell you that I don’t have this number exactly.

Currently, it’s self hosted - single user per container (permission wise).

By your name and the name of the project I assume this could find the missing sock in the laundry pile ? 🤭

Yep 😊

Oh nice, now I see you used it as the logo - I already love it - before even trying it. Will spin up a copy shortly.

Haha! Make sure to find me on discord if you need any help :)

BTW, in the survey, you're missing DevOps Engineer, that's the category that would match what people here in self-hosted are doing on their home labs, which sometimes it's what they do at work as well.

Bullseye 🎯

Just write a short #intro :)

Awesome project!! I’ll be giving this a shot 100%

Thanks a bunch, consider leaving a star, helps me keep the lights on!

Github Repo

"Privacy obsessed"..."Only available on discord"

Lol. Jokes aside, seems pretty cool

Cool initiative not new but cool.

Must have's in my view:

  1. PDF OCR and search. There is a a HUGE backlog of documents that are scanned or just imported that need to be OCRed and searchable. You can import from some 3rd party paperless or docspell but it has to happen to be really useful.
  2. Integrate into ELT flow. Airbyte dose data dumping form systems and I do not need another app trying to pull from it.
  3. Elastic search. I am not sure how it done now but most of my data is not in English and only system that had any work done to have not totally awful semantic performance is Elastic search.

Elastic search. I am not sure how it done now but most of my data is not in English and only system that had any work done to have not totally awful semantic performance is Elastic search.

I'm focusing on English only (I'm not a native speaker myself), but most of the users come from the US so the effort to support other languages doesn't make sense yet.

The OCR idea is awesome! I'll add that.

I won't be integrating with Airbyte they have this weird non open source license (Elastic 2.0), if that's a deal breaker for you, I'm super sorry, maybe you could add an Airbyte integration ?

I get it may be to early in the pipeline for multiple languages but while setting up architecture keep in mind you will need to add languages or leave a way to add them for contributors.

Honestly I am pulling large datasets from fairly fragile systems, so I can not afford to pull it multiple times... Airbyte seams to be main option but I have entire digestion system built out... point being I am not going to give you direct access to sharepoint give me a way to dump data in. Ideally one of the ELT or ETL tools that I probably have running.

Got it, I’ll keep my eye out for requests of this nature. Maybe it’s not unique to you

Super cool! Strong work!

Very happy to see first open source steps in the same direction as MS Copilot.

Does it use natural language processing?

Edit; yes it does.

Yep, a model we fine tuned ourselves on work style docs/pages/chats we synthesized.

Very cool. Thank you for building such a tool!

Edit: Does it support German?

While this is great for deploying to workplace. Would be interesting to see what kind of integration is added for homelab/selfhosted apps. One of the things thats still missing in the selfhosted world is a unified search application.

Where's the source for the web page so we can fix the English mistake on it?

Where’s the mistake 😂

Do you speak more than one language?

Hebrew & English

I figured. Nicely done! I was highlighting that it's not cool to talk down to someone not speaking their second language perfectly when they themselves only speak that 1 language.

I'm still trying to find the mistake, please someone help me 😂

[deleted]

We’ll try to lower the footprint. PyTorch is a bitch.

Interesting, do you plan to make money and if so how? Asking as I'm always curious about the kind of business models that could support this kind of positive efforts in the long term.

How does it work in practice? Are you enable full text search on each of the data source and if so what are you relying on? There is AI and natural language query, what model are you using? (seems like deepset/roberta-base-squad2 from the source)

IMHO if you want support for the community to help it grow write an excellent, and I mean truly helpful to a newcomer, tutorial on how to support a new datasource. There might even be connectors usable from other project or protocols, e.g WOPI or WebDAV, that instantly enables a lot of new sources.

This is great advice, and is an immediate part of our roadmap, who the fuck are you ;)
Wanna contact me on discord?

I’m not seeing any mention of the telemetry. Is there a list of what information leaves my network?

Good question, in fact given the audience I'd keep the default disabled https://github.com/GerevAI/gerev/blob/158aff9be38da41012e8a8491ad0874bcd8a3708/Dockerfile#L9 and explain clearly why it matters to hopefully get the feedback needed.

I did see a PostHog (product analytics) tracking snippet in the code...

It’s there front end and backend. I just got rid of all of it