I have a script that utilizes the APScheduler module to run certain scripts at set intervals. These scripts are used for all kinds of purposes that make my life easier I also use a simple SQLite database to save the information and this is how I see if the data has changed, and if it has, it will send me a message using Telegram's API. Here is a basic rundown of how the script works
import sqlite3
conn = sqlite3.connect('db.db', check_same_thread=False)
from apscheduler.schedulers.background import BackgroundScheduler
sched = BackgroundScheduler()
def compareChanges(site, new_value, url, subname = ""):
cur = conn.cursor()
# Get old value (if exist)
old_value = cur.execute("SELECT Data FROM Data WHERE Name = ? AND Subname = ? LIMIT 1", (site, subname)).fetchone()
# Insert/initialize if not exist (data will be updated later in this function)
if not old_value:
old_value = "NOT SET"
cur.execute("INSERT INTO `Data` (Name, Subname, Data, Timestamp) VALUES (?, ?, ?, ?)", (site, subname, old_value, datetime.now().isoformat()))
#conn.commit()
new_value = str(new_value) # Convert to string because sometimes the data passed is a dict
if old_value != new_value:
# Log and notify of change
logger.info(f"[{site}] has new value of {new_value}")
pushMsg(f"{site} => {new_value}", url) # Send Telegram message of update
# Set change and save file
cur.execute("UPDATE `Data` SET `Data` = ?, `Timestamp` = ? WHERE Name = ?", (new_value, datetime.now().isoformat(), site))
conn.commit()
cur.close()
return True # Return True if changed
else:
#logger.info(f"{site} - No Change: {old_value}")
cur.close()
return False # Return false on no change
def Function1():
import requests
response = requests.get("https://example.com/api/endpoint").json()
compareChanges("Website Status Check", response['status'], "https://example.com")
def Bestbuy_Latest_Price():
import requests
response = requests.get("https://bestbuy.com/product/page")
# This is pseudocode but it loads products on page and compares price for each product find
for item in response:
compareChanges("Bestbuy_Latest_Price", {product: price}, "https://bestbuy.com/product", subname = sku)
sched.add_job(Function1, 'cron', hour='*')
sched.add_job(Bestbuy_Latest_Price, 'cron', hour='*')
sched.start()
The way this script works is that every hour, Function1()
will be called, pull some data, compare it to the stored value in the database, and then notify if this value was changed.
I put cur = conn.cursor()
segment inside the compareChanges()
function because otherwise I was getting errors about using a cursor inside a different thread. However now I am getting cannot start a transaction within a transaction
. Some of the functions run at the exact same time so it is a possibility that the script will attempt to update multiple values at the same time (it will never update the same row at the same time though). How do I fix my code to not start a transaction in a transaction?
The problem here is that you're reusing the same connection in both functions. Connections are not safe to use in multiple threads at once, and you're deliberately overriding that check with check_same_thread=False
. You should connecting to the database in the function separately each time:
def compareChanges(site, new_value, url, subname = ""):
with sqlite3.connect('db.db') as conn:
cur = conn.cursor()
...