I Built an Open Source Search Engine Position Tracker
Local Link | Content link | Reddit Link
r/selfhostedu/towfiqi (675)

Comments (71)

[deleted]

I plan to add a feature that shows the keyword search volume data using Google Search Console API. It would be great if you could contribute :)

And The goal of the project is to only track Keyword SERPs, whereas sermresh ahrefs actually are full seo suite that does a lot.

Since the goal of the project was simple, it was built with javascript to keep it simple. If we were to build a semresh alternative, we would have to be rewrite from scratch using more efficient stack.

Thanks

[deleted]

you need to set your own by setting environment variables. You can see the instruction here

Semrush does a lot more than tracking keywords.

SerpBear is an Open Source Search Engine Position Tracking App. It allows you to track your website's keyword positions in Google and get notified of their positions.

Built with Nextjs the app uses 3rd scrapers or proxy ips to scrape google search result for a certain keyword and check if your domain is present in the result.

Repo: https://github.com/towfiqi/serpbear

Docs: https://docs.serpbear.com/

​

Features

  • Unlimited Keywords: Add unlimited domains and unlimited keywords to track their SERP.
  • Email Notification: Get notified of your keyword position changes daily/weekly/monthly through email.
  • SERP API: SerpBear comes with built-in API that you can use for your marketing & data reporting tools.
  • Mobile App: Add the PWA app to your mobile for a better mobile experience.
  • Zero Cost to RUN: Run the App on mogenius.com or Fly.io for free.

​

Motivation

I own a couple of domains that ranks for certain keywords on Google. The free service I use only 10 keywords to track for free. I also needed an API to use it for data reporting tools.

Amazing work! Thank you so much for this! I will definitely be giving this a go.

Thanks :)

If this tool is half decent, you will get so many users your side project will become a job :) well done.

great work man, keep it up! giving it a try right now.

Getting hugged to death?

Thank you for sharing

Thanks man this is great stuff, clean looking UI also.

Can you autocomplete / suggest search terms?

Sorry to say you can't do that. Maybe in future.

I’m an idiot when it comes to something like SEO. What can I use the results for? To see if I need to refine my keywords?

Usually, you would do SEO(Search Engine Optimization) for some specific keywords (related to your business niche) that people use to search in google. For example if you sell web development services in your area, you would want to appear at the top of the google search when people search for "web development service Nevada".
You want to check what on position your website appear for the search result. Say you appear on 3rd page, and you do SEO both external and on site. All your hard work can only translate to a better position on google for those keywords. Hence, when you are tracking keywords, you are actually, tracking your SEO performance in a way.

Thanks for the eli5. I’ll be trying this out asap. Thank you for this.

Yes, looks like it tracks where your website shows up given specific keywords. So you know if search results are showing you lower than you'd like.

Ah, that makes sense!

Nice work! Though it's not open source until you've applied an open source license. Until then you have not specifically granted any freedoms in use/distribution/modification.

You are absolutely correct. Thanks for pointing it out. Just added MIT License.

Awesome! If this is your first Open Source project, just make sure you fully understand the implications of your license, especially if you plan to make money from this in the future. Open source licenses are generally there to protect the freedoms of the code and it's users, not the business intent of the author.

I am aware, and don't plan on monetizing the project.

That's an incredible job, a tool we all needed!

Thank You. Give it a try :)

Very cool! I can’t wait to mess around with it

Would be cool to see a user system, so users could manage specific domains they setup.

Awesome app! I followed the guide and created my first instance through fly.io, as a noob. What would be the easiest way to make a backup of my database?

Great to hear you are up and running. :)
Fly.io already backs up your data store in their volumes. But the restore feature is not public yet. They will add this feature in near future.

And also, I am planning on adding a built in backup feature that will let you backup the data to your Google Drive.

This is nice, thank you and keep it coming.

I cannot add 2 characters domain (example: xx.com). Can you take a look?

Btw, I tried with other typing way of domain (example: www.xx.com) however after that I tried to put some keywords but didn’t show up. I checked the log file, it recorded and scrapped successfully. Any idea on why it happed?

Fixed. Thanks for finding the bug.

Aren't search results all custom to the user specifically now? Won't this thing only return the results that google thinks the connecting IP will want to see?

Only if Google can personalize it. I am pretty sure for most keywords it can't. That's why there are billion dollar SEO Software companies out there and marketers are paying good money for their services.

You can learn about personalized search here

I get such terrible search results from Google, I figure it's either from those successful SEO companies tricking search engines into ranking them much higher than they should be, or it's because I block ads and tracking and Google doesn't know how to identify me very well.

Thanks for the link!

I figure it's either from those successful SEO companies tricking search engines into ranking them much higher than they should be

You are correct. Although I wouldn't call it tricking...

I used to work for a real estate web development company, we built websites for real estate agents. Our SEO division absolutely tricked search engines into ranking us on the first page, this was like 10+ years ago. I'm not sure how much of this has changed.

Its still the same, just evolved as Google figured out how to prevent these tricks(blackhat techniques).

In past it was way easier to game Google, now, it requires lots of work and money.

Man, ahrefs and semrush can eat a dick. You're the man. Good job. You should sell a cloud version of this, and you'd make a killing.

Thanks. I thought about a cloud version but since the app only handles a small section(SERP tracking only) of seo, I am not sure anyone would be interested.

Start small. $5/mo/site or something. No reason to do everything. Add a nice API. You can add more features as you go. You're already there to have a fantastic side hustle that makes money while you sleep. Do it, man. This is a good product and idea. I don't need the rest of the SEO features, really, I just care about SERP, so buying ahrefs is a way overpriced overloaded tool for me. I'd much rather just pay a few bucks and have something like this.

ScrapingAnt doesn't have proxies in my country (BE) so I tried Scraping Robot instead but the latter doesn't seem to work. Anyone else having this experience?

If you are using a country that doesn't support your country, the app automatically switches to US.

Just checked, some refactoring broke the ScrapingRobot integration. Already fixed it and will push the update shortly. Thanks for pointing it out :)

I noticed. Thanks! Great build OP.

A UX tip: If there's a successful crawl but the domain wasn't detected change the hyphen to '>100' for example. So the user knows it's actually working but wasn't in the top 100.

I'll keep following this, gj! edit: owkey, you're quick.

How's the project going? There's no update from March.

!RemindMe 1 day

I will be messaging you in 1 day on 2022-12-01 16:01:30 UTC to remind you of this link

1 OTHERS CLICKED THIS LINK to send a PM to also be reminded and to reduce spam.

^(Parent commenter can ) ^(delete this message to hide from others.)

|^(Info)|^(Custom)|^(Your Reminders)|^(Feedback)| |-|-|-|-|

Is this even useful these days when what search results you see are so personalized and dependent on context?

Only if Google can personalize it. I am pretty sure for most keywords it can't. That's why there are billion dollar SEO Software companies out there and marketers are paying good money for their services.
You can learn about personalized search here

How to install on a linux? Fly.io free is good?

Content removed in protest of Reddit blocking 3rd-party apps. I've left the site.

Yes its a bug. An issue has been opened already. Will be fixed shortly.

Hi, the tool sound awesome. I would really like it if it could connect to a discord webhook, so I can get notifications over there. Let me know if you are interested in adding a feature like that. I'm willing to work with you on the project.

Hey, it would be great if you could integrate with Automatisch (FOSS Zapper alternative)

automatisch.io

https://github.com/automatisch/automatisch

https://automatisch.io/docs/build-integrations/folder-structure

https://discord.com/channels/913926763074191381/913926763074191385

Congrats

What is the backend?

Next.js for Frontend & Backend.
Sqlite for Database.

Ok, thanks
How did you handle the hosting?
Is it expensive during spikes?

Wow, this is great stuff.. and especially thanks for making it open source.

I'm just having a problem when trying to deploy it on fly.io

--> v0 failed - Failed due to unhealthy allocations - no stable job version to auto revert to and deploying as v1

I'm on wsl (linux for windows), anyone experiencing the same issue ?

Failed due to unhealthy allocations

This happens when you have not changed the port number from 8080 to 3000 in your fly.toml file. Can you please check your fly.toml file.

I've changed the port number in the fly.toml file and it worked like a charm, thanks !

Amazing tool buddy, had it up and running in no time. Any future plans to manage backlinks?

Not really, sorry. Backlink checkers requires searching the whole internet for your domain by scanning the commoncrawl database which is huuuuge. Scanning it is quite expensive and time consuming.

Thanks for the answer 🙏🏼

I arrived from Google, was literally searching for something like this! I need to track my projects keywords and hate all the subscription based solutions!

Thank you for this! it's awesome

Is there any update?

It would be a pity if this project were abandoned.

Are there any alternatives? We are looking to track thousands of keywords for many companies and looking for a solution.

So far the closest we found was RankTracker from Link-Assistant.com running on a separate dedicated server/machine. Unfortunately on the Enterprise ver their captcha solving stops working at 180k monthly requests.

🤣 what in the backwater brew u lookin for fella... also get DonoWall'd nerd lmao

Great project, I built it in docker and have been using it for a few months now. A small problem is that some bugs are not fixed. For example, I found that it doesn't work with Cyrillic characters and I found what the error was.
encodeURI not working properly, maybe need been encodeURIComponent()