changedetection.io - The self-hosted Website Change Detection and Notification application - releases version v0.39.18
Local Link | Content link | Reddit Link
r/selfhostedu/dgtlmoon123 (416)

Jump on over to https://github.com/dgtlmoon/changedetection.io or update your containers!

screenshot

See https://github.com/caronc/apprise/wiki/Notify_homeassistant for supported HomeAssistant notifications (Control your Home Assistant when changes to web site content or a JSON API occurs)

Comments (86)

Nice, does it let you set up an alert where you only get notified if the diff matches a string? eg. maybe you point it at a changelog for a frequently updated piece of software and only want to be notified when they add a new instance of the term 'security' or 'CVE'

Absolutely! just enter the line under the "Trigger Text" field in the watch settings

Sweet! I'll have to check this out then, keep meaning to write something like this again since my old one died when a major dependency became deprecated and I do miss the functionality. Thanks for putting this out there.

You might want to mention this key feature more prominently, I didn't see it on a quick skim of the readme.md nor is it in the screenshot.

That's a great idea! I've added a few things here https://github.com/dgtlmoon/changedetection.io/blob/master/README.md#key-features does it help?

Yes definitely, it's even better than I'd hoped and ooo playwright

It looks like it has trigger text, yes.

Been using it for ages, gud shite. Best shite if you're out looking for deals.

[deleted]

That sounds like the opposite of random guesswork. It was calculated and deliberate.

I discovered that there are companies that offer an API, that tells you how much you should bump your prices by, based on your competitors availability and pricing.. horrible

Heh there is far more than that going on in the pricing market. I don't work directly in it, but my parent company offers price optimization software for manufacturers and retailers.

PM me more information :)

I've done this for a rather large Car Rental company in Europe. Well, I was building an automated scraping platform for them, which would regularly provide data to their data scientists to reverse engineer their competitors pricing and stock, to dynamically change their prices.

It was a very eye-opening job, especially on how much of time is spent spying on your competitors.

Lo,, epic username too :D thanks!!!

Ive been monitoring local cat shelters in our search for a kitten, so its been working great lol

Ive seen a few sites that don't work with it, but for the most part, it works great, and I love the detailed diff reports

That's fantastic to hear! yeah, we are slowly working towards making it more compatible with more sites, feel free to open up a discussion on our github page with the sites you're watching

[deleted]

Thanks for the kind words <3 I spent a lot of personal time on this project!

I've had good luck using Distill to generate selectors and then running the monitor on my local changedetection instance. Distill has a good UI picker "fuzzy"/wildcard selector generator that works around sites that use random strings in the CSS elements probably to try and block tools like this.

I imagine it's a bit complicated to programmatically generate these kind of selectors reliably but I'd love to see it in this app.

Could you PM me some example differences between the same selector from Distill and what we get in changedetection.io ?

Will do for sure once I get back to a computer this weekend.

The same kind of obfuscation gets in the way of the UI picker in ad blockers like ublock origin where the final element is random text (the worst since then it's basically unpickable) or somewhere in the path is random text that needs to be replaced with a wildcard.

The same kind of obfuscation gets in the way of the UI picker in ad blockers like ublock origin where the final element is random text (the worst since then it's basically unpickable) or somewhere in the path is random text that needs to be replaced with a wildcard.

I have a feeling this is why some change-detection methods use the X,Y block coordinates on the page instead.. because more than likely the text you are interested in, will always roughly be rendered in the same place

the xpath selector is usually relying on the structure of the elements more than their class/ID values, unless it can find something very specific

I love this app, been using it for a while.

Was able to order a dress my girlfriend wanted that would sell out instantly when it would come back in-stock. She's happy.

Thanks for the kind words! Please share this thread and help us grow :)

This has been one of my favorite self-hosted projects of late!

Well done!

This is pretty awesome. I've heard it mentioned plenty of times but I finally have a use for it and gave it a go. Thank you!

Fantastic! That's great to hear!

[deleted]

100% total free to run! But for a small subscription, I will run it for you on my own infrastructure, it helps pay for my time to develop it :)

Also may I suggest you open your own subreddit? Would be great to chat about changedetecion on here and exchange tips and tricks.

Like this? https://www.reddit.com/r/changedetectionio :)

Oh! Haha it’s quiet in there. You should always link to it in every post you make on selfhosted and also cross post on there for more traction ;)

Yeha you're totally right :) Ok, noted for next time :) hmmm any other ideas how to promote it?

No I think just always mentioning it with every post you make on self hosted alongside with cross posting in the subreddit will help.

Just keep encouraging conversation in the subreddit. I think it would be a great place to exchange tips and tricks about using the software.

Anyone had luck with homedepot? Some items work great, other items (using same selector) returns about 20 separate prices and they change with each time it is checked. I have used the visual selector but also looked at the xpath and it is picking the correct one

e.g https://www.homedepot.com/p/Owens-Corning-FOAMULAR-250-2-in-x-48-in-x-8-ft-R-10-Scored-Squared-Edge-Foam-Board-Insulation-Sheathing-52DD/202085962

xpath #standard-price

which looks okay when you use your browser, but changedetection gets a different result each day it checks. This is also tested using the chrome docker instance to pull the page and same result.

Nowwww I see the problem. There is a new div lazy loading at the top that also has #standard-price. I think filtering out //*[@id="root"]/div/div[2] may work. Will have to give a try this weekend.

Using the link you provided I saw the price inside:

div#eco-rebate-price

Hope that helps some.

Can you filter out other xpaths from the original selection to further isolate changes?

[deleted]

That's great to hear! :) yeah, check it out :)

Loving it so far - up until trying to get Playwright working. I'm using unRAID and have installed a separate Playwright container via dockerhub.

I can access Playwright on its own so I know it's working on the correct IP/port (I've even tried multiple versions of Playwright from dockerhub in an attempt to get this working).

I then set PLAYWRIGHT_DRIVER_URL=ws://ip_address:port/?stealth=1&--disable-web-security=true

Checking changedetection logs, I can see: ERROR:changedetectionio:Exception reached processing watch UUID:# - No module named 'playwright'

Any guidance, please? TIA

I'm having the exact same error. I'm using browserless/chrome:latest image. The env vars I'm using are all listed at https://docs.browserless.io/docs/docker.html (the ones listed by the dgtlmoon's docker-compose.yml file are obsolete.

Both the logs and the UI show No module named 'playwright' error. Please let me know if you find a solution

Glad I’m not the only one!! Unfortunately I’ve exhausted my technical abilities for now. Hoping someone else can assist!

I've reported the error upstream. Hopefully they can help

https://github.com/dgtlmoon/changedetection.io/discussions/944

hey did you ever get this sorted? u/St0lz

Yes I sorted it out by using the official Docker image ghcr.io/dgtlmoon/changedetection.io instead of Linuxserver.io one lscr.io/linuxserver/changedetection.io. Here is my current Compose file

version: '3.8'

services:

  changeDetection:
    container_name: change-detection
    environment:
      BASE_URL: https://changedetection.example.com
      PLAYWRIGHT_DRIVER_URL: ws://change-detection-fetcher:3000/?token=${TOKEN:?err}
      PGID: 1000
      PUID: 1000
    hostname: change-detection
    image: ghcr.io/dgtlmoon/changedetection.io 
    labels:
      - traefik.enable=true
      - traefik.http.routers.changeDetection.tls.certresolver=le
    networks:
      - proxy
    restart: unless-stopped
    volumes:
      - ./config:/datastore

  headlessBrowser:
    container_name: change-detection-fetcher
    environment: 
      DEBUG: '-*'
      DEFAULT_BLOCK_ADS: 'true'
      DEFAULT_IGNORE_HTTPS_ERRORS: 'true'
      DEFAULT_STEALTH: 'true'
      ENABLE_DEBUGGER: 'false'
      MAX_CONCURRENT_SESSIONS: 10
      MAX_QUEUE_LENGTH: 10
      TOKEN: ${TOKEN:?err}
      WORKSPACE_DELETE_EXPIRED: 'true'
      WORKSPACE_EXPIRE_DAYS: 7
    hostname: playwright-chrome
    image: browserless/chrome:latest
    networks:
      - proxy
    restart: unless-stopped

networks:

  proxy:
    external: true

Thanks so much. I was literally doing the same thing (using the wrong image).

playwright is not supported on arm/v6 and arm/v7 b architectures :( is that your problem perhaps?

Nah, my server is x86/x64.. are the steps that I’ve performed sounding correct? I have a feeling I’m choosing the wrong Playwright from Dockerhub…

Just follow the docker-compose.yml https://github.com/dgtlmoon/changedetection.io/blob/master/docker-compose.yml uncomment out the playwright part

​

it sounds to me like you're running this some other way

The official docs do not mention several of the variables listed by that docker-compose file, so I presume the example is obsoleted

Please help us out at https://github.com/dgtlmoon/changedetection.io/discussions/944#discussioncomment-3653312 and paste in your debug output from the commands listed

Unless I've been picking the wrong version of Playwright from Dockerhub - https://hub.docker.com/search?q=playwright

This software is awesome - it helped me and several of my friends & coworkers get covid vaccinations right after the time slots were released. Muchas gracias!

Hey thats amazing!! I need to collect these stories!

Loved the concept but never managed to get it to work properly

Wish it had login support.

Until 0.40.0 comes out with full browser interactivity, you can just use the JS steps, super simple, click **[edit]** then goto [Execute JavaScript before change detection]

document.querySelector("input[name=email]").value="xxxxx@gmail.com";
document.querySelector("input[name=pass]").value="xxxx";
document.querySelector("#loginbutton").click();

When do you think that’ll be released? I really love the idea of this software but last time I tried I couldn’t get it to work on my of the websites I wanted to monitor. All the EU “cookie pop ups” are really making it impossible to work.

Can’t wait to see future releases!

Yeah those cookies help no one, totally agreed!!

It's sort of a function of time and money :( Right now I'm donating my own time to build it, my hope is that the paid subscriptions will make it worth it, but right now that's a looooooong way away :-(

Yeah I hear you. Maybe as interest grows other talented developers may wanna chip in? It’s definitely a cool project!

Can be, my dream is to have a _lot_ of paid subscribers too, and just work on this fulltime

That would be awesome! I hope you can resist the temptation to shorthand the self hosted version, though, in order to bring in more paid subscribers. ;)

The only difference is that on our paid version you are using proxies that we have paid for

[deleted]

Until 0.40.0 comes out with full browser interactivity, you can just use the JS steps, super simple, click [edit] then goto [Execute JavaScript before change detection]

document.querySelector("input[name=email]").value="xxxxx@gmail.com"; document.querySelector("input[name=pass]").value="xxxx"; document.querySelector("#loginbutton").click();

I got it installed and running, but having trouble knowing how to track things.. documentation is very lacking and not beginner friendly.

Took me a bit to find, but there's a wiki in github. Just in case you haven't seen it

I'm not smart enough to configure it.

I got it going but man anything would set off an alert.

I just wanted monitoring of prices to change but it was hard

Take a little bit of time and learn how to filter elements using XPath. For instance, I'm monitoring a site for in stock updates and I inspected the web page found the button that says out of stock and used the css class of that button to create an XPath filter rule for the page. Only when the text in the button changes will I get notified.

xpath://*[contains(@class, 'btn product-add mini-form ')]

https://www.w3schools.com/xml/xpath_examples.asp

You can copy the page you want to monitor's source code and paste it into this testing tool below, then practice selecting the element you want to filter for/track.

http://xpather.com

If you can send me the site you're monitoring, I can send you the XPath code for tracking the price.

u/BillyDSquillions yeah drop the URL here, we can throw a couple of 'share URL's here which will pre-load the right xpath for you :)

[deleted]

Probably not what you want to hear - but It works on our paid subscriptions, we are using a really really killer proxy provider (paid), I highly recommend it

its good to use the 'visual selector' if you can get playwright talking to it (or try our subscription, it's pretty cheap!)

Fantastic for finding the BOGO deals at the local dispensaries. Get a push-notification the minute the deals go up. Love it.

I was wondering, in your Discord notifications, is there anyway to embed the URL in the title?

i.e if we specofy the URL, we get the full string and I kinda of want it masked by the title I set rather than seeing the whole thing.

We have a feature request for exactly that :) https://github.com/dgtlmoon/changedetection.io/issues/708 but it depends on time, which depends on how many paid subscribers we have :/

I'm happy with learning how to self host, is there a place where I can just make a donation?

Simply read the README, scroll all the way to the bottom https://github.com/dgtlmoon/changedetection.io thanks!

This is a fantastic app! Thank you for the hard work!

I am looking to use this to track price differences on Amazon (and other sites). Currently, I get updates when there is any change to the page when I add a particular item. Is there a way to only notify me when the price of a specific item drops below a certain dollar amount?

​

Example: iPad - Below $1300 -> Alert

​

Thanks!

Not yet, but it's a feature we'de like to add sometime soon!

I think a different way to browse the diff rather than a drop down would be more efficient - maybe just the time/date list in one column and the result on the right

Another suggestion is ability to delete individual diff items for when the site changes and causes duplicates of the same

I would contribute if I knew python :(

> I think a different way to browse the diff rather than a drop down would be more efficient - maybe just the time/date list in one column and the result on the right

I was having the same feeling, like sometimes I just want to look at different versions and dont care for the diff

Great service, managed to set up on windows without any issues and little knowhow. Even got webdriver working. Successfully bought tickets for a high demand event from Twickets monitoring sales listing feed.

Is there a way to import login cookies or something, I'd like to monitor a forum that needs a login but I have 2fa enabled

Also would be handy for logging in periodically for websites that have inactivity limits

Just copy+paste a copy of the request from chrome (including the "Cookie: xxxxx" header) into the "requests -headers" setting

Hi I'm not following sorry. I can't see that option on my changedetection.io instance and I've just opened chromedrive.exe on windows so not sure where to get info you need and where to put it

Sorry for being a noob haha

how did you get webdriver to work?

Maybe I'm a bit stupid, but I've been understanding how to make it work for days but I can't

Currently I have everything working correctly on my windows 11, but I don't know how to active webdriver

Is there a guide on how to install this on a droplet? I know there's plenty of information around but It's kinda difficult for non-tech-savvy people.

FYI I've already set a docker-compose.yml but having a problem with accessing the website. Would appreciate any help.

On Diff, how about including a screenshot as well as text to visualize change versions, ala wayback machine? Thanks for the great project.

Can this also parse the actual changed data through its API , want to monitor 3 gasstation gasoline prices and send it to homeassistant ?

sure why not, did you try it?