Announcing changedetection.io 0.40.0 !
Local Link | Content link | Reddit Link
r/selfhostedu/dgtlmoon123 (136)

Hi all! We are proud to announce the newest major release of our self hosted website change detection and notification app, whats new? Browser steps - Interact with the browser (login to sites, accept cookies etc) before change-detection, Dark mode - because you love it Automatically follow LDJSON price data - on websites that support it, More notification hooks - not just messaging (slack/discord/telegram etc) but also post://, get:// and others to make driving other services easier Extract data as CSV for example, watch realestate websites and easily make a CSV of the number of houses for sale, or your favourite whiskey's price! enjoy! and please help spread this app :) https://github.com/dgtlmoon/changedetection.io

https://preview.redd.it/ho3wsk6ocu4a1.png?width=1302&format=png&auto=webp&s=e5522a3f718dc38d7e52421d8605d855cdf50685

Comments (50)

One of my fav selfhosted projects. Keep up the great work! Dark theme is very welcome too. :)

Is there a better guide on how to use the JSON price checking? And it only picks up B&H and Best Buy, while ignoring Amazon and Newegg. Is this something that has to be added store-by-store?

Thanks for the kind words! Yeah so, as I wrote, it's following the LD+JSON product information, which amazon does not carry, what is LD+JSON? lets ask google

https://www.google.com/search?q=what+is+ld%2Bjson+used+for

​

newegg seems to work just fine when tested with this random page https://www.newegg.com/p/3D5-000D-001T1?Description=rtx%204080%20gaming%20pc&cm_re=rtx_4080%20gaming%20pc-_-9SIAHT3JPK5019-_-Product (you need to give it a single product page, not a listing)

Sorry, by guide I meant I am trying to follow the guide in your repo: https://github.com/dgtlmoon/changedetection.io#parse-json-embedded-in-html

Putting in json:$.price does not seem to do anything to pull the price in changedetection.

https://imgur.com/zrPqCBM

CSS/JSONPath/JQ/XPath Filters: json:$.price

I get the whole JSON output. Is this the intended behavior or should it pull only the price (249.99 like in example above)?

I improved the example here https://github.com/dgtlmoon/changedetection.io/commit/6c217cc3b6a172565172ea42b0b7e1e891d29555

Awesome! Thanks for the update.

json:$..price (the double period) made all the difference. Can confirm it works now!

Best Buy and B&H Photo work with this JSON as well. :)

Edit: Any chance of jq being installed automatically in the Docker container? It's pretty light?

I'm assuming you're talking about newegg because you didnt say otherwise

I also read the newegg html and pulled out the ld+json script stuff, and then I used google to find a validator, you can do the same thing too

I pasted the ld+json from newegg into a JSON formatter/JSONpath test selector and I can see that the query should be different, I also found those jsonpath formatters on google

Additionally I can see that the example in our README is slightly different structure in the newegg LD+JSON so it means the query has to be different slightly

Browser steps

Does this mean I can replace Distill? This is awesome, been waiting for this!

I'm hoping you can! We even have a "import from distill.io" section under the import tab :) Makes for a great distill.io alternative!

Long time user, love your work !

I use visual ping at work, but this is a wonderful alternative. Thanks for your hard work.

I was literally searching for this yesterday and stumbled on the github thread for this issue and thought to myself, "I hope this is out soon, it looks AMAZING."

Then next day, here we are. Thanks changedetection.io team!

Nice! Its an awesome project, I'm using it and will bump the version this weekend:)

Can this be used to detect changes in content made with JS?

read the README on the front page of the project :)

[deleted]

Coming soon :)

It's the best self-hosted project I have ever used. Helped me so much.

Thank you a thousands times.

<3 thanks for the kind words! you can help us in reply by spreading awareness about this app :)

What all can the site detect? Can it detect changes in azure ad?

Why? Powershell could if you wanted it too lol

For folks that want a simpler, more light-weight tool, see also this tiny shell script.

`pip install changedetection.io`

way simpler :)

Ooh, could it do what camelcamelcamel.com does?

I just took the time to read through the documentation and to answer my own question: yes, it definitely tracks prices on products (and more).

I will surely give it try.

Thanks 👍

I wonder if Amazon detects it as some sort of ddos. I manually scraped Amazon every 30 minutes and it stopped me pretty fast. I would have had to use a service to change ip address I think.

I have this question too. Can it monitor Amazon price ?

That's what I was asking, check out the camel site I mentioned above

What kind of context can it check? Can it also take a screenshot and then compare future screenshots to detect diversions? Because that would be a very nice feature to keep track of broken websites after an automated update.

If the screenshots have a visual change that Would trigger a notification to a website admin to check.

Why not just check for matched text in the response? It seems much easier than trying to analyze the difference between two images programmatically to detect changes or failures.

Because the layout breaks, not the content. If you search for eg a header title, it would still be there and not detected as a problem.

So the only reliable method is comparing a screenshot before and after an update. There are several SaaS tools tbat are build exactly for this method for agencies so you get alarms if a site design got broken after updating eg WordPress or a plugin like page builder in WordPress.

Seems like it would need a lot of tuning, major corporate sites change CTAs frequently. A better health check to me would be a request for knowns like checking for invalid status codes in static files so that the layout and content can be changed without FP alerts.

I'm sure tools exist for this. I've witnessed The Business purchasing solutions like they're the holy grail only to ignore the results because they believed the salesperson and didn't realize the amount of work to tune the tool.

Don't get me wrong, I could see significant value from a screenshot provided in an alert. I just wouldn't want to be the one who sets up an alert based on two screenshots.

Also, thank you for responding because I understand more where you are coming from. I am all about automating releases, let QA do their job during the day in a non prod environment rather than having a team trying to validate a release last minute after hours. IaC has come a long way in terms of automating the infrastructure side, but while still improving, testing still feels primitive in comparison.

I myself am in a different QA field and have seen some of the amazing things an excellent QA professional can build. I recommend heavy investments in QA and Security with the latest shiny toys (within reason) because it reaps dividends in the long run.

If you want to keep things cheap and still reliable, write many tests that can be improved and reused over time. Promote a dev culture that considers thorough tests as important as functionality. I am horrible at this and break as many things as I touch, but it makes me smile ear to ear when I request a team make a change and they ask me what values they can use to write a test to validate success.

Well it's not about sending screenshots in alerts. It's about monitoring websites for visual breaking changes. They don't have to come in alerts aling, we just need an alert that warns about visual degradation so a web dev can check that site immediate.

These problems typically come from tools and cms'es that offer some sort of auto updating like WordPress. Sometimes they run automatically and break the page builder. So all text would still be there and the response would still be a 200 ok, but the css/html is broken.

The only way to detect something like this is comparing a screenshot and search for changes in pixels I think. I know the solutions exist that do exactly this because we used them in the past, but they are extremely expensive. Something more simple that could be self hosted is my preference but so far I haven't found anything until this post on Reddit. Perhaps there could be some kind of "plugin" feature in future so more "methods" for comparing could be introduced? Or something in that line.

[deleted]

Does the visual selector feature only allow to select one part of the page to monitor at a time or can you select different parts on the same page monitor on a single monitor ? I know you can do it manually but does the visual selector allow for it ?

it will build the filter (you can see it in the CSS filters) and the software will actually return ALL results of that filter

but the visual selector only shows the FIRST option.. we wanna improve this in the future tho!

Hey there! Love your app. We're experimenting with it for doing simple e2e tests of our website.

However having some difficulty setting up slack notifications. I don't see any instructions for Slack in the dashboard either. There's only mention of discord, tgram, gets:, posts: etc..

How does one notify a slack channel?

it's pretty self-explanatory, just read the text under the 'notification URLs box', hope it helps!

This is the text I see:

Use AppRise URLs for notification to just about any service! Please read the notification services wiki here for important configuration notes.
discord:// only supports a maximum 2,000 characters of notification text, including the title.
tgram:// bots cant send messages to other bots, so you should specify chat ID of non-bot user.
tgram:// only supports very limited HTML and can fail when extra tags are sent, read more here (or use plaintext/markdown format)
gets://, posts://, puts://, deletes:// for direct API calls (or omit the "s" for non-SSL ie get://)

No mention of slack, but I see AppRise. Not familiar with AppRise, but I'm guessing that's the only method to get Slack working?

I was hoping a direct `posts://` to my slack webhook would work, but that doesn't seem to be working.

Just want to make sure I'm understanding the above text correctly.

I would just use the slack endpoint URL, as described in teh documentation https://github.com/caronc/apprise/wiki/Notify_slack

Cool thank you so much!

Hello, anyone share a sample for telegram notification please

Anyone knows how to bypass amazon captcha without proxy? I've setup changedetection.io using playwright with stealth 1, and headless set to false but still amazon websites are asking for captcha.

i recommend using a really high quality residential IP, check out oxylabs or brightdata

oxylabs

Thanks for advice, but i'll be using it very rarely and i'm searching for something free or low price.

Best "bang for your buck" is to get some mobile device that you can SSH tunnel into ;) then use that as your socks5 proxy

in very simple terms for someone very amateur, how would I do that?

well you know its a technical thing... so maybe its better to pay someone

​

its like me telling you - changing the piston rings on your car is pretty 'simple' too...