Userdefaults for complications are nil while watch is working fine

I'm able to successfully send userdefaults from the phone to the watch using applicationContext and save those userdefaults to the app group I created. For the watch, I see all the userdefaults come in successfully and I'm able to use them in the WatchOS app.

Once I created the complications for the watch app, I am using the same app group ID and I added that app group in the capabilities of the watch extension but when I try to use the same userdefault that I'm using for the watch, everything is nil. I'm not understanding what I'm doing wrong. How do I share userdefaults between the watch and complication?

This is how I'm bringing in the userdefaults from the phone to the watch

        
        let defaults = UserDefaults(suiteName: K.appGroupID)
        
        if let value1 = applicationContext["lat"] as? Double {
            defaults?.setValue(value1, forKey: "lat")
        }
        if let value2 = applicationContext["lng"] as? Double {
            defaults?.setValue(value2, forKey: "lng")
        }
         ...

I am able to use this exact function in the watch app, but in the complication, the userdefault is nil.

    let formatter = DateFormatter()
    formatter.timeStyle = .short
    formatter.timeZone = .current
    
    let defaultLoad = UserDefaults.init(suiteName: K.appGroupID) 
    if defaultLoad!.bool(forKey: "timeFormat") == true {
     ...

I tried to remove and re-add the app group but that didn't work either. I also created a separate app group for the watch and extension to use alone but still same issue.